Replaces lists of pip installs with a python requirements file

Before pip install would use latest version of python libraries. Now we specify the library
version found in the pythonsrc-ext repo.

I used pipenv to find a set of version numbers that have consistent
dependencies.

We have been using a very old version of paramiko, which is causing
errors in gpssh-exkeys and gpseginstall on FIPS-enabled systems, in
particular RHEL7.4.  These errors are due to the ssh daemon not
accepting deprecated cryptographic algorithms.  Upgrading paramiko fixes
these issues, but also introduces a new dependency on python-ecdsa.

Additionally, this updates the pythonsrc-ext git submodule accordingly.
Co-authored-by: NNadeem Ghani <nghani@pivotal.io>
Co-authored-by: NJamie McAtamney <jmcatamney@pivotal.io>
Co-authored-by: NJim Doty <jdoty@pivotal.io>

* doc: analyzedb - add note about removing previous snapshots to recover space.

* docs: analyzedb -add link to analyze command and analyzedb utility.

'brew install python' used to pull in python 2
Now is pulls in python 3
This commit explictly uses python 2

This reverts commit 40b3b16c. Now that CI is
bumped to concourse 3.9.2, lets see if we can live without this restriction.

By default the memory management in resource group is
implemented by tracking every memory alloc/free in memory
context.

To support external components ( such as plcontainer) which
do not use memory context, cgroup memory auditor is added
into resource group.

Note that catalog table pg_resgroup is changed.

RawColumnDefault was removed previously as part of a commit that fixed
pg_attrdef oid inconsistency between the master and primaries.  This
is not an issue anymore since we send pre-assigned oids during
dispatch since Greenplum 5X and have checks around it in the code.  We
also have an oid_consistency test in src/test/regress that passes, and
gpcheckcat doesn't complain.  It seems like we're covered here.

[ci skip]

The FIXME comment asked whether or not we wanted to remove the check
on CAC_WAITBACKUP.  It is true that we do not currently use WAITBACKUP
state, but we should continue to have the code.  It will prevent some
merge conflicts during postgres merge, and we may use it sometime in
the future if applicable to any future Greenplum features.

[ci skip]

Authored-by: NBen Christel <bchristel@pivotal.io>

Authored-by: NBen Christel <bchristel@pivotal.io>

- The CMakeLists file is for CLion IDE code intelligence only.
- We only look for sources in these directories because otherwise
  CLion might be confused by duplicated source files, e.g. for libpq
Authored-by: NBen Christel <bchristel@pivotal.io>

This follows the pattern of other scripts like make_etags.
Authored-by: NBen Christel <bchristel@pivotal.io>

Co-authored-by: NAmil Khanzada <akhanzada@pivotal.io>
Co-authored-by: NDavid Sharp <dsharp@pivotal.io>
Co-authored-by: NBen Christel <bchristel@pivotal.io>

- Use standard PMSignal mechanism to trigger FTS probes from
  dispatcher.

- Use statusVersion flag only to indicate if a probe cycle resulted in
configuration update. This was overloaded to also unblock callers
waiting for FTS probe to be triggered and finished.

- Use a separate flag "probeTick" for this purpose.
Co-authored-by: NAshwin Agrawal <aagrawal@pivotal.io>

While mirror is in grace period based on value of GUC
`gp_fts_mark_mirror_down_grace_period`, it should not be marked down by FTS. So,
adding coverage to validate the same. Plus modifying the test to use GUC to
speed-up marking mirror down for intentional case.

Mirror is provided with grace period before being marked as down. FTS probes
during this grace period interval, return retryRequested to probe request. Based
on `retryRequested` should retry probes to primary and incase number of retries
reach retry limit of `gp_fts_probe_retries` must skip marking such mirror down
as still primary has not given go ahead to mark it down. Only when
`retryRequested=false`, mirror must be marked down.

The fixme comment was placed here to warn that the implementation
might be backwards on Greenplum 4.X and 5.X.  The concern is not valid
for Greenplum 4.X because there is no lazy xids.  However, the concern
is indeed valid for Greenplum 5.x.  This is noted and will be changed
on the 5X_STABLE branch.

[ci skip]

Prior to GPORCA version 2.55.10, ORCA produced a plan with Redistribute
Motion on the hstore attribute(which is not a "GPDB hashable" type).
This caused queries to fail with the error like:
ERROR:  Type 40963 is not hashable.
This FIXME was added in `contrib/hstore/sql/hstore.sql` to run such
failing queries with planner. Since the issue has been fixed in ORCA,
removing it.

cache_test_acquire was incorrectly spelled in the test definition struct
making it hard to run as a single test (without using the same typo).

The code was quite confused on whether this was a switch-case or an if-else
statement..

The command tag is unfortunately not included in pg_regress output, so this
is not reflected in regression tests, even though we have ALTER EXTERNAL
TABLE commands in the test suite.

Fixes github issue #4377.

This function was made unnecessary by commit b3f300b9, which changed
GetOldestXmin() to "include" any distributed snapshots, too. rewriteheap.c
no longer changes the freezeXid it's given, so there's no need for a
function to get the new value back from it.

We had removed these earlier, when we didn't have GIN indexes. Now we do.

This argument was introduced when master/standby WAL replication was
implemented to handle persistent table cases.  Since persistent tables
have been removed, we no longer need this argument to mdtruncate.
This takes care of a GPDB_84_MERGE_FIXME.

This reverts commit a21da89e. These
changes are obsolete now that persistent tables have been removed.
Co-authored-by: NAsim R P <apraveen@pivotal.io>

When upgrading to 8.4, upstream's pg_upgrade refuses to copy old
sequence data due to a change in columns. However, that check also
prevented copying of sequence data during an 8.2->8.3 upgrade --
Postgres doesn't care about this case, but GPDB does.

Don't omit the copy if we're upgrading to 8.3, and enable the 8.2 heap
upgrade for sequence tables so that the page makes sense in the new
database.

Follow-up to 71916050. Checksums were backported to Postgres 8.3, so we
ignore them for <= 8.2.

While we're at it, switch the data_checksum_version from a false
assignment to a zero assignment, to match upstream. The actual
implementation is an unsigned int; Postgres master incorrectly uses a
bool type in the declaration.

Follow-up to e8de956e. Correct the GPDB 5 check to use Postgres 8.3's
version number, and correctly return the cached value from the GPDB 4
check.

This state was meant to prevent auxiliary processes from waiting from syncrep.
As far as we can tell, auxiliary process should not be calling
SyncRepWaitForLSN() unless it is committing a transaction.  And if it is
committing a transaction, it better wait for mirror to flush XLOG upto the
commit LSN.  One example of an auxiliary process committing a transaction is
FTS, when it updates cluster configuration on master.
Co-authored-by: NTaylor Vesely <tvesely@pivotal.io>

* docs - add pxf developer topics (release 1)

* misc edits

* add pxf dev topics to oss book subnav

* misc edits from david

docs - perms for hdfs files backing pxf ext tbls are for gp user. This closes #4692 and #4667 (#4676)

* docs - perms for hdfs files backing pxf ext tbls are for gp user

* correct pxf stop instructions

* reword pxf stop intro

* address comments from alex

* lowercase a Y

These were related to chosing the right arguments to send to GPDB's
make_agg() and cost_agg() methods for queries containing DISTINCT or set
operations.

Hash aggregation when used to implement a DISTINCT (in either form) in
the query is not related to grouping sets and thus the argments to
num_nullcols, input_grouping, grouping and rollup_gs_times should be 0.

However, since SetOp uses the upstream TupleHashTable while HashAgg uses
GPDB's HHashTable implementation, the hash table size calculations
should be computed differently. This is fixed in this commit
Signed-off-by: NSambitesh Dash <sdash@pivotal.io>

They've been unused for as long as I can see.

There was a GPDB_84_MERGE_FIXME regarding whether or not an extra lock
was needed when getting the list of child partitions during CREATE
INDEX.  We already have the lock on the root partition which prevents
any ALTER TABLE operations from changing the child partition list.
Because of that, an extra lock is not required.

[ci skip]