The list of files to clean in test/regress contained references
to files no longer present.  pg_class32 was an intermediate file
in a test for upgrades from Greenplum 3.2 to 3.3/3.4; cppudf.sql
was added in an Orca testsuite commit which seems to have never
used that file at all; gmon.out is an output file generated by
gperf and all gperf invocations have been removed from tests.
Reviewed-by: NAsim R P <apraveen@pivotal.io>
Reviewed-by: NJimmy Yih <jyih@pivotal.io>

Commit bfd1f46c used the wrong time unit (expect Ms, passed with Us)
in BackoffSweeper backend which makes it cannot re-calculate the CPU shares
in time and the normal backends will sleep more CPU ticks than before in
CHECK_FOR_INTERRUPTS and cause a performance downgrade.

It takes non-zero amount of time after a command is dispatched from a
client until it appears in pg_stat_activity.  The test must wait
before validating anything based on pg_stat_activity.  The wait logic
was already added for one instance of such validation.  This patch
addres the wait logic for the remaining instance of validation.

Also found a way to avoid creating one table, while at it.

Reviewed-by: Shaoqi Bai and Adam Berlin

This tool looks at EXPLAIN plans and recognizes the line with the
optimizer version. Recently, we added the string "(GPORCA)" to the
optimizer name.

The fix is to add parentheses to the characters we ignore in this line.

In my earlier PR #8149 I removed some typedefs using the CSpinlockOS
class that no longer exists. However, one of those typedefs, ConnectionHT
was still used in the class. It did not cause any compilation errors,
probably because this class is unused.

To be consistent and to make the code easier to read, this PR adds back
the missing typedef. In the long term we could consider removing the
entire class COptServer.

* remove reference to deleted header file
* Remove more references to CSpinlock
* Bump ORCA version to 3.56 for PR503
* Bump ORCA version to 3.57 for PR510

* docs - pxf jdbc partition range improvements

* misc edits requested by david

* clarify use of RANGE and INTERVAL (hint), edits requested by francisco

Some gpexpand behave tests verify the behaviors of the `--duration` and
`--end` arguments, they expect the gpexpand data redistribution phase to
exist after 2 seconds before all the redistribution are done.  However
internally gpexpand check for the timeout at a 5-second interval, if the
data redistribution completes within 5 seconds then the tests will fail.
This is just what is happening recently.

Improve these tests by queueing more tables for redistribution.

Add certificates & keys and test cases for contrib/sslinfo

Use `echo` + `sed` to add/remove options in postgresql.conf of
master node and standby node. This method could completely restore
the SSL related options in postgresql.conf. The imperfect point is
this way may overwrite the existing certificates and keys under data directory.

We use newly created certificates and keys, instead of certificates in src/test/ssl/ssl. Because there some fields in those certificates are missing in the test.
`sslinfo` will only be compiled and packaged when `--with-openssl` option is enabled in configuration, otherwise `sslinfo` is omitted.

After introducing sysctl settings in commit 42930ed1, gpexpand fails to
shutdown a segment with:
`'Shutdown failed! [Errno12] Cannot allocate memory'`

This is most likely due to the n1-standard-1 instance_type
having only 3.75 GB of memory, which is too small causing our python
utilities to fail when allocating multiple threads. This can be worked
around by setting the `-B` parallel_process option to a low value, or
setting the kernel parameter vm.overcommit_memory to 0. However, we
would like to test the standard sysctl settings as recommended to
customers, rather than supporting unrealistic edge cases such as
machines with very low memory.
Co-authored-by: NNikolaos Kalampalikis <nkalampalikis@pivotal.io>

Release candidates for server and clients are stored in GCS and the S3
resources were deprecated four months ago. The S3 resources were left in
the pipeline to give downstream consumers an opportunity to update their
CI pipelines to use the new GCS resources. This commit finalizes the
deprecation of the S3 resources.
Authored-by: NBradford D. Boyle <bboyle@pivotal.io>

In case of extended queries, snapshot information is shared between
reader and writer QEs using files.  The writer obtains a snapshot and
writes it to file.  This patch fixes a bug that passed incorrect address
of the array of in-progress transactions for writing to file.  The bug
caused hard to reproduce errors in production workloads that involved
extended queries (bind/execute libpq messages, declare cursor
statements, certain PL/* statements such as RETURN and EXECUTE in
pl/pgsql).

Reviewed-by: Adam Berlin and Jesse Zhang

Functions under test are dumpSharedLocalSnapshot_forCursor() and
readSharedLocalSnapshot_forCursor().  Significant amount of global state needs
to be created for the test to be able to invoke the two functions.  What we are
testing here is, whether correct snapshot information is included in what is
written to file.  Validation is performed by reading the contents of the file
and compairing them with expected values.

An implicit rule to run a unittest is defined in src/Makefile.mock.  This patch
overrides it such that the required directory for the cursor snapshot file is
created before running the test.

Reviewed-by: Adam Berlin and Jesse Zhang

Currently shareinputscan doesn't handle rescan properly, and to fully
support it, there will be a lot of code changes. After some discussions,
we decide to disable shareinputscan with outer refs for now.

The recommended sysctl settings for user clusters are out-of-date, and
after some investigation we've discovered a good minimal set of
recommended defaults.  These will be updated in the documentation, but
we also want to set them in the VMs used for our CLI Behave tests so
that we use values similar to those that users will have in their
environments, so this commit adds a section to the Concourse cluster
task file to do so.
Co-authored-by: NJamie McAtamney <jmcatamney@pivotal.io>
Co-authored-by: NKalen Krempely <kkrempely@pivotal.io>
Co-authored-by: NNikolaos Kalampalikis <nkalampalikis@pivotal.io>

Commit d0bf8563 added citext to the
default make target of 5X_STABLE, but it was never added to master.
This brings master in syn with 5X_STABLE, and sets the init_file to
use the main test/regress file.

Reported-by: Lisa Owen

Before GP6, the stats sender process works for gpperfmon and metrics
collector. If any of them enabled, stats sender should start.
From GP6 metrics collector become a standalone bgworker.
So this commit fix startup condition for stats sender, now it only
starts when gpperfmon enabled.

Previously, we did not wait for the mirrors to apply the filesystem
changes involved in a CREATE DATABASE command before 'stat'ing
relfilenodes in all the database instances (including mirrors) in the
test setup. This flakiness was introduced in PR #7792.
Co-authored-by: NDavid Kimura <dkimura@pivotal.io>

Previously, it was possible for a utility to connect to a database of
the wrong version, if you for instance had a 5 instance running and
ran a 6 utility.  There was logic in gppylib to check GPDB versions,
but it was not being used anywhere and had not been updated since GPDB
5.

This commit updates the version logic to the current version and adds
a check in initFromCatalog that will error out if it tries to connect
to a database of a different version.
Co-authored-by: NJamie McAtamney <jmcatamney@pivotal.io>
Co-authored-by: NNikolaos Kalampalikis <nkalampalikis@pivotal.io>

FTS takes responsibility for updating gp_segment_configuration, in each
fts probe cycle, FTS firstly gets a copy of current configuration, then
probe the segments based on it and finally free the copy in the end. In
the probe stage, FTS might start/close transactions many times, so FTS
should not update current copy of gp_segment_configuration when a new
transaction is started.

Auxiliary bgworkers should skip resource group assignment

bgworker_should_start_mpp() is mainly used to check whether the DTX
recovery process has recovered the distributed transactions and set
the shmDtmStarted to true. We used to let postmaster keep trying if
a DTX recovery process didn't finish and it kept postmaster in a
very high CPU usage which is not cool. Because DTX recovery process
is also a bgworker and a bgworker quiting will also force a call to
maybe_start_bgworker(), it's unnecessary to set StartWorkerNeeded to
tell postmaster keep trying.

* docs - pxf jdbc connection pooling

* edits requested by david

* use present tense in note

* misc edits requested by francisco

* docs - misc fdw- and foreign table-related doc updates

* Postgres Planner

We've hard-coded values controlled by neither our test nor the code,
this breaks on any platform with a different OpenSSL configuration than
when the author of commit 85725149 ran the test.

This reverts commit 85725149.

Seeking motivation from upstream thread
https://postgr.es/m/CAD21AoDuAYsRb3Q9aobkFZ6DZMWxsyg4HOmgkwgeWNfSkTwGxw@mail.gmail.com
which has script to detect declared but not defined functions. Using
that script found following functions on GPDB code in that category.

The script is bare minimum and flags these but the output contains
other noise too and can be modified to enhance it into tool but that's
for some other day.

Authored-by: NTingfang Bao <bbao@pivotal.io>

Authored-by: NTingfang Bao <bbao@pivotal.io>

* Add certificates & keys and test cases for contrib/sslinfo

Use `echo` + `sed` to add/remove options in postgresql.conf of
master node and standby node. This method could completely restore
the SSL related options in postgresql.conf. The imperfect point is
this way may overwrite the existing certificates and keys under data directory.

We use newly created certificates and keys, instead of certificates in src/test/ssl/ssl. Because there some fields in those certificates are missing in the test.

Previously, we cannot reconstruct the statement and dispatch it to
QEs in AlterSetting() if role and database are zero, we report an error
instead. Actually, when both role and database are not specified, it's
a ALTER USER ALL statement, this commit reconstruct it well.

Use pg_ctl to allow background worker connections during smart shutdown
and remove application_name whitelist, which was checked to allow
smart shutdown to proceed. When pg_ctl timeouts or the user decides
to Control-C , prompt user to continue shutting down in smart mode
while waiting forever, or shutdown in fast or immediate mode.
We allow the user to send a SIGINT to interrupt a long-running smart
mode shutdown and resend either a fast or immediate shutdown.
Co-authored-by: NJamie McAtamney <jmcatamney@pivotal.io>
Co-authored-by: NKalen Krempely <kkrempely@pivotal.io>
Co-authored-by: NShoai Lari <slari@pivotal.io>
Co-authored-by: NJacob Champion <pchampion@pivotal.io>
Co-authored-by: NNikolaos Kalampalikis <nkalampalikis@pivotal.io>

We add information to the TwoPhaseHeaderFile about which tablespace we're
attempting to drop. We also schedule a tablespace for deletion (similar
to CREATE TABLESPACE) on commit, which waits until the transaction
is committed to perform the filesystem deletion. This follows the
recommendation for relation files to be deleted after the commit to not
spend time deleting files during the commit.

We also modified the distributed_commit record to pass the tablespace to
be dropped to the standby. And, we modified the abort record, to help
back out of a drop on the mirrors.

- Bump the catalog for the drop tablespace xlog changes.
- Extracts hook functions to be implemented by separate components

The tests skip FTS probes during the test run to avoid FTS taking a primary
down that has a fault point injected into it.
Co-authored-by: NTaylor Vesely <tvesely@pivotal.io>
Co-authored-by: NSoumyadeep Chakraborty <sochakraborty@pivotal.io>
Co-authored-by: NAlexandra Wang <lewang@pivotal.io>

Setting a variable to itself is a no-op which can be removed. This
may have been introduced in error and instead masking a real bug,
but if it so then we have lived with it for two years so I'm opting
for removing.

Reviewed-by: Asim R P and Bhuvnesh Chaudhary

Reviewed-by: David Yozie