FTS starts probing as soon as fts process gets created and triggers
every minute. No reason to trigger and wait for one probe cycle on
first connection which calls initTM(). If for some reason fail to
dispatch command to segment, dispatcher will trigger probe anyways, so
why slow things down upfront.

I am not sure on usage of this GUC gp_set_read_only. When set it sets
FTS to read-only which is then used while starting any transaction and
converted to read-only. Seems better to achieve the same result using
GUC `default_transaction_read_only` or `transaction_read_only`, which
can be used at session-level as well instead of full system-wide
setting.

Previously, to avoid the leak of the gang if someone terminates
the query in the middle of gang creation, we added a global pointer
named CurrentGangCreating so the partially created gang can also be
destroyed at the end of the transaction. However, the memory context
named GangContext where CurrentGangCreating was created may be reset
before CurrentGangCreating is actually destroyed and a SIGSEGV may
occur. So this commit makes sure that CurrentGangCreating is destroyed
ahead of other created gangs and the reset of GangContext.

Allows the compile and icw tests for each platform to pass
and fail independently of other platforms.

The gate jobs were born out of necessity to handle infrastructure issues
with concourse. Now that infrastructure issues have been stabilized,
it's time to review the layout of the pipeline again.

This commit removes the icw_start_gate job that multiplexed a passing
condition from all of the compile jobs that sat in front of every icw
job. This was not desirable following a longer running
issue with the compilation on one platform, ubuntu16, that then blocked
icw tests on the remaining platforms.

Replacing the passing condition of icw_start_gate on each icw job is the
corresponding compilation job for the test, based on platform.

E.G.

This:
                    (blocks)          (blocks)
compile_gpdb_centos6  -> gate_icw_start -> icw_planner_centos6
compile_gpdb_ubuntu16 ->                -> icw_planner_ubuntu16

Is now this:

                    (blocks)
compile_gpdb_centos6  -> icw_planner_centos6
                    (blocks)
compile_gpdb_ubuntu16 -> icw_planner_ubuntu16
Signed-off-by: NTaylor Vesely <tvesely@pivotal.io>

Concurrent index builds are not supoorted in greenplum. Seems there exist GUC
gp_create_index_concurrently under which we still allow the creations. But still
with the same for bitmap indexes CONCURRENTLY cannot be supported.

This test keeps failing randomly in CI due to "WARNING: ignoring query cancel
request for synchronous replication to ensure cluster consistency." This happens
as during bitmap index creation it locally commits on each segment and then one
of the segment errors during index build with "ERROR: CONCURRENTLY is not
supported when creating bitmap indexes", issues cancelation message to other
segments.

Failures are seen in CI randomly where
checkpoint_and_wait_for_replication_replay return false, but when check on the
box later the primaries and mirrors are in sync. So, seems its timing related
failures maybe sometimes containers are running slow causing delays. This test
is for checking functionality and not performance, so avoiding flaky failures is
better with increased number of retries, which in general shouldn't affect
runtime.

For queries of the form "NOT (subselect)", planner lost the "NOT" operator
during the initial pull-up in pull_up_sublinks_qual_recurse() which resulted in
incorrect filter and hence wrong results. Inside
pull_up_sublinks_qual_recurse(), when a qual contains NOT, we check if sublink
type is any one of these: EXISTS, ANY or ALL, and invoke appropriate sublink
pull up routines. In case of qual of the form "NOT (SELECT <>)" the sublink
type is EXPR; hence we recurse into the argument of NOT; at which point we
lose the information about NOT operator.

This commit fixes the issue by returning the node unmodified when the argument
of NOT is an EXPR sublink. The EXPR sublink, later gets pulled up by
preprocess_qual_conditions() wherein, pull_up_sublinks() is invoked again to
handle sublinks in an expression.
Signed-off-by: NSambitesh Dash <sdash@pivotal.io>

The pseudocols field was added to RangeTblEntry to avoid duplicates in the
target list. When the field was added, a WRITE_NODE_FIELD was added in
_outRangeTblEntry(), but a corresponding READ_NODE_FIELD was not added in the
_readRangeTblEntry(). Additionally, this field was not added to the readfast
and outfast functions either since this field was not used after the planning
stage and did not have to be dispatched to the QE's.  Later, in
fd6741f9, a corresponding READ_NODE_FIELD was
added along with a FIXME to decide whether or not to serialize/deserialize.
After confirming that this field is not used after planning, we have decided to
remove any serialization/deserialization of the pseudocols member of
RangeTblEntry
The pseudocols member of RangeTblEntry is a list of CdbRelColumnInfo. This
structure was added in the same original commit. An _outCdbRelColumnInfo() was
added for this structure as well. Similarly, no corresponding
_readCdbRelColumnInfo() was added. This data structure is only used for making
pseudocols, so we have also removed the _outCdbRelColumnInfo() function.
Signed-off-by: NAbhijit Subramanya <asubramanya@pivotal.io>

The GPDB specific API for pgstat_report_waiting() accepts waiting
reason unlike the upstream counterpart, which accepts only a boolean
flag.  Renaming the API to gpstat_report_waiting() allows us to catch
new uses of the API introduced from upstream merges.
Co-authored-by: NAsim R P <apraveen@pivotal.io>

Without this change, ps display of postmaster child processes may get
mangled.  E.g.

postgres: 15432, gpadmin isolation2test [local] con14 cmd52 con14 cm?~??????X???

This change uses the GPDB specific function get_real_act_ps_display()
to get the ps display string before it is modified.

This change also sets Gp_role of FTS daemon process to utility instead
of the default value of dispatch.  That prevents appending "conXXX" to
FTS daemon's ps display.
Co-authored-by: NAsim R P <apraveen@pivotal.io>

Add missing sudo commands

[ci skip]

[ci skip]

To help debug the issue, PANIC incase encounter this shouldn't happen case.

Update README based on feedback from a new developer to the GPDB codebase

[ci skip]
Co-authored-by: NJim Doty <jdoty@pivotal.io>
Co-authored-by: NKevin Yeap <kyeap@pivotal.io>

Replaces lists of pip installs with a python requirements file

Before pip install would use latest version of python libraries. Now we specify the library
version found in the pythonsrc-ext repo.

I used pipenv to find a set of version numbers that have consistent
dependencies.

We have been using a very old version of paramiko, which is causing
errors in gpssh-exkeys and gpseginstall on FIPS-enabled systems, in
particular RHEL7.4.  These errors are due to the ssh daemon not
accepting deprecated cryptographic algorithms.  Upgrading paramiko fixes
these issues, but also introduces a new dependency on python-ecdsa.

Additionally, this updates the pythonsrc-ext git submodule accordingly.
Co-authored-by: NNadeem Ghani <nghani@pivotal.io>
Co-authored-by: NJamie McAtamney <jmcatamney@pivotal.io>
Co-authored-by: NJim Doty <jdoty@pivotal.io>

* doc: analyzedb - add note about removing previous snapshots to recover space.

* docs: analyzedb -add link to analyze command and analyzedb utility.

'brew install python' used to pull in python 2
Now is pulls in python 3
This commit explictly uses python 2

This reverts commit 40b3b16c. Now that CI is
bumped to concourse 3.9.2, lets see if we can live without this restriction.

By default the memory management in resource group is
implemented by tracking every memory alloc/free in memory
context.

To support external components ( such as plcontainer) which
do not use memory context, cgroup memory auditor is added
into resource group.

Note that catalog table pg_resgroup is changed.

RawColumnDefault was removed previously as part of a commit that fixed
pg_attrdef oid inconsistency between the master and primaries.  This
is not an issue anymore since we send pre-assigned oids during
dispatch since Greenplum 5X and have checks around it in the code.  We
also have an oid_consistency test in src/test/regress that passes, and
gpcheckcat doesn't complain.  It seems like we're covered here.

[ci skip]

The FIXME comment asked whether or not we wanted to remove the check
on CAC_WAITBACKUP.  It is true that we do not currently use WAITBACKUP
state, but we should continue to have the code.  It will prevent some
merge conflicts during postgres merge, and we may use it sometime in
the future if applicable to any future Greenplum features.

[ci skip]

Authored-by: NBen Christel <bchristel@pivotal.io>

Authored-by: NBen Christel <bchristel@pivotal.io>

- The CMakeLists file is for CLion IDE code intelligence only.
- We only look for sources in these directories because otherwise
  CLion might be confused by duplicated source files, e.g. for libpq
Authored-by: NBen Christel <bchristel@pivotal.io>

This follows the pattern of other scripts like make_etags.
Authored-by: NBen Christel <bchristel@pivotal.io>

Co-authored-by: NAmil Khanzada <akhanzada@pivotal.io>
Co-authored-by: NDavid Sharp <dsharp@pivotal.io>
Co-authored-by: NBen Christel <bchristel@pivotal.io>

- Use standard PMSignal mechanism to trigger FTS probes from
  dispatcher.

- Use statusVersion flag only to indicate if a probe cycle resulted in
configuration update. This was overloaded to also unblock callers
waiting for FTS probe to be triggered and finished.

- Use a separate flag "probeTick" for this purpose.
Co-authored-by: NAshwin Agrawal <aagrawal@pivotal.io>