- also, wait for primaries to recover after panics
- also, checkpoint at the end of each test to set redo point to not leak into next text

One phase commit message change to `Distributed Commit (one-phase)`
We need to fix the new case introduced by commit #6f9368
in direct disptach answer file.

The goal of query_info_hook_test is to ensure query_info_collect_hook
are placed in porper location for emitting query execution metrics.
This test was flaky due to uncertain order of calling between QD and
QEs when the interconnect in TCP mode.
This fix simply silent all QEs from emitting messages.
This is acceptable from the scope of this test because we just want
to make sure hooks are called at correct timing for each backend.
It should not be disturbed by query dispatching between QD and QEs.

Prepare statement will bind parameters for each execution. It needs to decide to
use a cached generic plan without params or a custom plan with params. In past,
GPDB use plan cost plus re-plan cost to choose generic and custom plan. But
generic plan does not contain params which leads to it could not generate
direct dispatch plan compared with custom plan.

For non direct dispatch plan it will introduce unneccessary QEs, which still
need to go through volcano model, do two phase commit and write prepare xlog.
So the cost of failed to generate direct dispatch plan would be higher in some
case than the re-plan cost which makes custom plan runs faster than generic
plan even if it needs to re-plan for every execute.

Note that non direct dispatch cost is not considered in planner yet. Planner
treats direct dispatch as an optimization and always enable it when possible.
But for prepare statement, the case is that for generic plan it could not
generate direct dispatch plan at all. But we need to consider this cost here,
as a result, we introduce non direct dispatch cost into total cost only for
cached plans.
Co-authored-by: NNing Yu <nyu@pivotal.io>

3.58.0 corresponds to commit "Only create PropConstraint hashmap if
necessary"
3.58.1 corresponds to commit "Fix stack-use-after-scope for
`CCacheHashtableAccessor` instantiation"
Authored-by: NChris Hajas <chajas@pivotal.io>

Happened to stumble over a commit by Asim that didn't seem to use
the usual name, and sure enough. Also add a few others that I had
lying around awaiting more to make it worth committing.

The list of files to clean in test/regress contained references
to files no longer present.  pg_class32 was an intermediate file
in a test for upgrades from Greenplum 3.2 to 3.3/3.4; cppudf.sql
was added in an Orca testsuite commit which seems to have never
used that file at all; gmon.out is an output file generated by
gperf and all gperf invocations have been removed from tests.
Reviewed-by: NAsim R P <apraveen@pivotal.io>
Reviewed-by: NJimmy Yih <jyih@pivotal.io>

Commit bfd1f46c used the wrong time unit (expect Ms, passed with Us)
in BackoffSweeper backend which makes it cannot re-calculate the CPU shares
in time and the normal backends will sleep more CPU ticks than before in
CHECK_FOR_INTERRUPTS and cause a performance downgrade.

It takes non-zero amount of time after a command is dispatched from a
client until it appears in pg_stat_activity.  The test must wait
before validating anything based on pg_stat_activity.  The wait logic
was already added for one instance of such validation.  This patch
addres the wait logic for the remaining instance of validation.

Also found a way to avoid creating one table, while at it.

Reviewed-by: Shaoqi Bai and Adam Berlin

This tool looks at EXPLAIN plans and recognizes the line with the
optimizer version. Recently, we added the string "(GPORCA)" to the
optimizer name.

The fix is to add parentheses to the characters we ignore in this line.

In my earlier PR #8149 I removed some typedefs using the CSpinlockOS
class that no longer exists. However, one of those typedefs, ConnectionHT
was still used in the class. It did not cause any compilation errors,
probably because this class is unused.

To be consistent and to make the code easier to read, this PR adds back
the missing typedef. In the long term we could consider removing the
entire class COptServer.

* remove reference to deleted header file
* Remove more references to CSpinlock
* Bump ORCA version to 3.56 for PR503
* Bump ORCA version to 3.57 for PR510

* docs - pxf jdbc partition range improvements

* misc edits requested by david

* clarify use of RANGE and INTERVAL (hint), edits requested by francisco

Some gpexpand behave tests verify the behaviors of the `--duration` and
`--end` arguments, they expect the gpexpand data redistribution phase to
exist after 2 seconds before all the redistribution are done.  However
internally gpexpand check for the timeout at a 5-second interval, if the
data redistribution completes within 5 seconds then the tests will fail.
This is just what is happening recently.

Improve these tests by queueing more tables for redistribution.

Add certificates & keys and test cases for contrib/sslinfo

Use `echo` + `sed` to add/remove options in postgresql.conf of
master node and standby node. This method could completely restore
the SSL related options in postgresql.conf. The imperfect point is
this way may overwrite the existing certificates and keys under data directory.

We use newly created certificates and keys, instead of certificates in src/test/ssl/ssl. Because there some fields in those certificates are missing in the test.
`sslinfo` will only be compiled and packaged when `--with-openssl` option is enabled in configuration, otherwise `sslinfo` is omitted.

After introducing sysctl settings in commit 42930ed1, gpexpand fails to
shutdown a segment with:
`'Shutdown failed! [Errno12] Cannot allocate memory'`

This is most likely due to the n1-standard-1 instance_type
having only 3.75 GB of memory, which is too small causing our python
utilities to fail when allocating multiple threads. This can be worked
around by setting the `-B` parallel_process option to a low value, or
setting the kernel parameter vm.overcommit_memory to 0. However, we
would like to test the standard sysctl settings as recommended to
customers, rather than supporting unrealistic edge cases such as
machines with very low memory.
Co-authored-by: NNikolaos Kalampalikis <nkalampalikis@pivotal.io>

Release candidates for server and clients are stored in GCS and the S3
resources were deprecated four months ago. The S3 resources were left in
the pipeline to give downstream consumers an opportunity to update their
CI pipelines to use the new GCS resources. This commit finalizes the
deprecation of the S3 resources.
Authored-by: NBradford D. Boyle <bboyle@pivotal.io>

In case of extended queries, snapshot information is shared between
reader and writer QEs using files.  The writer obtains a snapshot and
writes it to file.  This patch fixes a bug that passed incorrect address
of the array of in-progress transactions for writing to file.  The bug
caused hard to reproduce errors in production workloads that involved
extended queries (bind/execute libpq messages, declare cursor
statements, certain PL/* statements such as RETURN and EXECUTE in
pl/pgsql).

Reviewed-by: Adam Berlin and Jesse Zhang

Functions under test are dumpSharedLocalSnapshot_forCursor() and
readSharedLocalSnapshot_forCursor().  Significant amount of global state needs
to be created for the test to be able to invoke the two functions.  What we are
testing here is, whether correct snapshot information is included in what is
written to file.  Validation is performed by reading the contents of the file
and compairing them with expected values.

An implicit rule to run a unittest is defined in src/Makefile.mock.  This patch
overrides it such that the required directory for the cursor snapshot file is
created before running the test.

Reviewed-by: Adam Berlin and Jesse Zhang

Currently shareinputscan doesn't handle rescan properly, and to fully
support it, there will be a lot of code changes. After some discussions,
we decide to disable shareinputscan with outer refs for now.

The recommended sysctl settings for user clusters are out-of-date, and
after some investigation we've discovered a good minimal set of
recommended defaults.  These will be updated in the documentation, but
we also want to set them in the VMs used for our CLI Behave tests so
that we use values similar to those that users will have in their
environments, so this commit adds a section to the Concourse cluster
task file to do so.
Co-authored-by: NJamie McAtamney <jmcatamney@pivotal.io>
Co-authored-by: NKalen Krempely <kkrempely@pivotal.io>
Co-authored-by: NNikolaos Kalampalikis <nkalampalikis@pivotal.io>

Commit d0bf8563 added citext to the
default make target of 5X_STABLE, but it was never added to master.
This brings master in syn with 5X_STABLE, and sets the init_file to
use the main test/regress file.

Reported-by: Lisa Owen

Before GP6, the stats sender process works for gpperfmon and metrics
collector. If any of them enabled, stats sender should start.
From GP6 metrics collector become a standalone bgworker.
So this commit fix startup condition for stats sender, now it only
starts when gpperfmon enabled.

Previously, we did not wait for the mirrors to apply the filesystem
changes involved in a CREATE DATABASE command before 'stat'ing
relfilenodes in all the database instances (including mirrors) in the
test setup. This flakiness was introduced in PR #7792.
Co-authored-by: NDavid Kimura <dkimura@pivotal.io>

Previously, it was possible for a utility to connect to a database of
the wrong version, if you for instance had a 5 instance running and
ran a 6 utility.  There was logic in gppylib to check GPDB versions,
but it was not being used anywhere and had not been updated since GPDB
5.

This commit updates the version logic to the current version and adds
a check in initFromCatalog that will error out if it tries to connect
to a database of a different version.
Co-authored-by: NJamie McAtamney <jmcatamney@pivotal.io>
Co-authored-by: NNikolaos Kalampalikis <nkalampalikis@pivotal.io>

FTS takes responsibility for updating gp_segment_configuration, in each
fts probe cycle, FTS firstly gets a copy of current configuration, then
probe the segments based on it and finally free the copy in the end. In
the probe stage, FTS might start/close transactions many times, so FTS
should not update current copy of gp_segment_configuration when a new
transaction is started.

Auxiliary bgworkers should skip resource group assignment

bgworker_should_start_mpp() is mainly used to check whether the DTX
recovery process has recovered the distributed transactions and set
the shmDtmStarted to true. We used to let postmaster keep trying if
a DTX recovery process didn't finish and it kept postmaster in a
very high CPU usage which is not cool. Because DTX recovery process
is also a bgworker and a bgworker quiting will also force a call to
maybe_start_bgworker(), it's unnecessary to set StartWorkerNeeded to
tell postmaster keep trying.

* docs - pxf jdbc connection pooling

* edits requested by david

* use present tense in note

* misc edits requested by francisco

* docs - misc fdw- and foreign table-related doc updates

* Postgres Planner

We've hard-coded values controlled by neither our test nor the code,
this breaks on any platform with a different OpenSSL configuration than
when the author of commit 85725149 ran the test.

This reverts commit 85725149.

Seeking motivation from upstream thread
https://postgr.es/m/CAD21AoDuAYsRb3Q9aobkFZ6DZMWxsyg4HOmgkwgeWNfSkTwGxw@mail.gmail.com
which has script to detect declared but not defined functions. Using
that script found following functions on GPDB code in that category.

The script is bare minimum and flags these but the output contains
other noise too and can be modified to enhance it into tool but that's
for some other day.

Authored-by: NTingfang Bao <bbao@pivotal.io>

Authored-by: NTingfang Bao <bbao@pivotal.io>