The test injects a fault in StartTransaction().  This function is so
generic that any concurrent test may trigger the fault, leading to
failed or misleading outcome.  The test was using old faultinjector,
which emitted status of a fault as NOTICE message.  NOTICE messages do
not appear in isolation2 answer files.  So this problem went
undetected.

(cherry picked from commit d160d5c1)

Remove NOTICE messages that follow a gp_inject_fault() select statement.
Replace the boolean value with text 'Success:' returned by the new
interfae.

For reference, the following sed script was used to identify 't'
following a ' gp_inject_fault ' line:

/^ gp_inject_fault $/{
   $!{ N
       :again1
       N
       s/ t$/ Success:/
       t again1
     }
   }

(cherry picked from commit 748f86fb)

This test has failed at least once due to the terminate query being executed
before the to be terminated 'create table' statement.  This was evident from
master logs.  The commit makes it more reliable by injecting a fault and
waiting for the fault to be trigggered before executing pg_terminate_backend().
As a side benefit, we no longer need to create any additional table.

(cherry picked from commit 63bed5eb)

These tests were invoking the new fault injector interface using its
temporary name - gp_inject_fault2().  Now the new interface is the
only one present in the repository, named as gp_inject_fault().

(cherry picked from commit 590ac108)

The new interface employs a special libpq connection parameter and a
libpq message to convey fault details to destination postmaster.  This
interface has been in use for some time now.  It was getting cumbersome
to maintain both the old as well as the new interface.

To be clear, we now have only one interface to inject fault and that is
"gp_inject_fault()".

Existing tests that were using gp_inject_fault2() will be updated in a
follow up commit.

(cherry picked from commit 83c336ad)

This makes the signature of the new interface match that of the old
gp_inject_fault interface, so as to make it easy to replace the old interface
with the new one.

(cherry picked from commit 41779acc)

If a test decides to fail because not all segments are running, it will
be nice to get the details of segments that are found to be not running.

The details printed by this patch will only be shown by behave upon a test
failure.  They will not be shown when a test passes.
Reviewed-by: NGeorgios Kokolatos <gkokolatos@pivotal.io>
Reviewed-by: NJacob Champion <pchampion@pivotal.io>
Reviewed-by: NDavid Krieger <dkrieger@pivotal.io>
(cherry picked from commit 2237ef21)

When reading the Behave test code for stopping segments, it is not obvious that
we are waiting for the segment to stop. Therefore we are explicitly adding the
`-w` flag to show the blocking behavior.
Co-authored-by: NNikolaos Kalampalikis <nkalampalikis@pivotal.io>
(cherry picked from commit 1e946005)

When there are no bats testcases present, the current coding returned an
error due to the glob not being able to expand. This was made clear by
the recent removal of the last test, and the discussion in #8270. Rather
than using a glob, use ls and if statement. This allows us to keep the
unittest framework in place even if there are currently no tests.
Co-authored-by: NAmil Khanzada <akhanzada@pivotal.io>
Co-authored-by: NDaniel Gustafsson <dgustafsson@pivotal.io>
(cherry picked from commit fabc6ca5)

Any temporary tables created in a session are cleaned up while exiting.
However, consider the below actions being performed:
1. A temporary table table is created. (Backend is used for this session)
2. Backend for temp table recieves SIGUSR1 signal -> it invokes procsignal_sigusr1_handler
3. Immediately, a SIGTERM is recieved by the backend before anything is
processed in procsignal_sigusr1_handler.

Here, SIGTERM will invoke die and it will attemp to remove the temp relations and commit
the transaction. While committing the transaction, SyncRepWaitForLSN will be invoked and if the
Commit LSN for the temp table session is greater than then LSN shipped to the Standby, the backend
will wait for a byte to be written on the fd in WaitLatch and keep on polling until it recieves
a byte on the pipe. However, Walsender will not be able to wake the process up, as it uses
SIGUSR1 signal to communiate with the backend process and since the signal handler are not
re-entrant the backend will not terminate.

Before this commit as well, if we are in SIGUSR1 handler, SyncRepWaitForLSN will exit early,
however, if the jump to die happened without setting the flag InSIGUSR1Handler, SyncRepWaitForLSN
will not realize that its already in SIGUSR1 handler and will hit the issue as described above.
In this commit, instead of using the flag we use pthread_sigmask to identify if we are already in
SIGUSR1 handler.
Co-authored-by: NSoumyadeep Chakraborty <sochakraborty@pivotal.io>
Co-Authored-by: NJesse Zhang <jzhang@pivotal.io>

Wait until we've inserted the fts_probe skip fault, AND we have
observed the fault being hit. This ensures that we've completed the
in-progress fts probe before continuing on with a test.

Previously, if we don't wait and our test continues forward to inject
a panic, then then in-progress fts probe would do its job and promote
the mirror.
Reported-by: NAsim R P <apraveen@pivotal.io>
(cherry picked from commit 38dd2da3)

This patch addresses two main scenarios:
1) Allowing multiple probes both internal and external to reuse the same
results when appropriate (ie: piggybacking on previous results). Multiple
requests should share the same results if they all request before the start of
a new fts loop, and after the results of the previous probe.

2) Ensuring fresh results from an external probe. When a request occurs during
a current probe in progress, this request should get fresh results rather
"piggybacking" or using the current results.

We use similar logic as the checkpointer code to detect whether a probe is in
progress with a probe start tick and probe end tick. To request a probe, we
send a signal requesting a fts results, then wait for a new loop to start,
then wait again for that current loop to finish. This implementation uses a
busy wait loop, which includes a short sleep. In the future, we can
leverage the upstream conditaion variable implementation which enables us
to signal multiple fts notify processes.

This was done via a manual cherry-pick from
a674b6b3025b9dc56c4cb34b3330f8b7bc1bf757.
Co-authored-by: NSoumyadeep Chakraborty <sochakraborty@pivotal.io>
Co-authored-by: NKalen Krempely <kkrempely@pivotal.io>
Co-authored-by: NDavid Krieger <dkrieger@pivotal.io>
Co-authored-by: NTaylor Vesely <tvesely@pivotal.io>
Co-Authored-by: NAlexandra Wang <lewang@pivotal.io>
Co-Authored-by: NJimmy Yih <jyih@pivotal.io>

* Add docs for sslinfo contrib module

* Postgres -> Greenplum

* Add statement of non-differentiation

* Another postgresql -> Greenplum

* fix copy/paste error

The calculation for minimum amount of memory to memory intensive
operators was using kB for both input and output, leading to an
assertion failure in case the minimum memory requirement was less
than a kB. Since we are using a full uint64 for calculating, we
have room for using just bytes instead leaving the formatting to
the caller (which is just EXPLAIN ANALYZE).
Reviewed-by: NWeinan Wang <wewang@pivotal.io>

Notable bug fixes since previous 9.4 STABLE merge (9.4.20):

        * Skip updating pg_class tuple for mapped indexes during
          reindex.

        * Sync timezones with IANA release tzcode2019a.

        * A bunch of fixes on updatable views.

        * Fix for a crash in synchronous replication that Paul Guo
          reported.

        * PANIC on fsync() failure.

        * Fix for various crashes due to incorrect initialization of
          transaction state, reported by Richard Guo.

        * The upstream commit "Postpone aggregate checks until after collation
          is assigned" has been reverted because it conflicts with Greenplum's
          window functions with group-by implementation.  See 253fb4d2f7be for
          more details.
Co-authored-by: NGeorgios Kokolatos <gkokolatos@pivotal.io>
Co-authored-by: NPengzhou Tang <ptang@pivotal.io>
Co-authored-by: NWeinan Wang <wewang@pivotal.io>

We have a kind of query that its InitPlan execute in ExecutorStart.
If Explain Analyze this kind of query, as well as, some memory or disk
information requires to collect in the main plan, the QD will crash.
Since queryDesc->showstatctx->stats_gathered is assigned to true
in ExecSetParamPlan function before ExecutorRun, So we only gather
Initplan metrics and left other slices information over. Segment fault when
some execution node print out metrics message.

To fix this problem, variable stats_gathered only be assigned after slice 0
metrics information collection.

fix issue:#6951

Use `echo` + `sed` to add/remove options in postgresql.conf of
master node and standby node. This method could completely restore
the SSL related options in postgresql.conf. The imperfect point is
this way may overwrite the existing certificates and keys under
data directory.

We use newly created certificates and keys, instead of certificates
in src/test/ssl/ssl. Because there some fields in those certificates
are missing in the test. `sslinfo` will only be compiled and packaged
when `--with-openssl` option is enabled in configuration, otherwise
`sslinfo` is omitted.

(cherry picked from commit 508bdb11)

* docs - updates and corrections to ALTER PARTITION clause of ALTER TABLE

-- Corrected ALTER PARTITION clause syntax
-- Clarified description for ALTER PARTITION clause.
-- Clarified information about altering partitions in Admin. Guide
-- Added example in Ref. Guide and Admin. Guide
-- Other minor edits.

* docs - edits based on review comments.

* docs - reorganize/reorder the reference guide topics

* remove Greenplum qualifier, edits requested by david

* remove Reference from sql commands, sys catalogs topic titles

ICG changes for commit "Add 4 xforms for LOJ index apply on dynamic table
scan"

These transforms existed for non-partition tables, but not for partition
tables.
Authored-by: NChris Hajas <chajas@pivotal.io>

The drop tablespace tests that introduce panics were not disabling
fts, which is a source of flakey tests.

Introduce a disable_fts() function that we can call before each
test. The fault reset at the end of each test will reenable fts for
the next test.

(cherry picked from commit fa64cc90)

This commit adds the GPDB side changes required to support GIN Indexes with
ORCA.  It also adds a new test file gp_gin_indexes to test plans produced for
ORCA/planner.

GIN indexes are not supported with index expression or predicate constraints.
ORCA does not support it currently for other types of indexes too.

ALTER TABLE DROP COLUMN followed by reorganize leads to loss of column
encoding settings of the dropped column. When the column's compresstype
encoding is incorrect, we can encounter block version mismatch error
later during block info validation check of the dropped column.

One idea was to skip dropped columns when constructing AOCSScanDesc.
However, dropping all columns is a special case that is not easily handled
because it is not equivalent to deleted rows. Instead, the fix is to preserve
column encoding settings even for dropped columns.
Co-authored-by: NSoumyadeep Chakraborty <sochakraborty@pivotal.io>
Co-authored-by: NIvan Leskin <leskin.in@arenadata.io>

GUC gp_enable_gpperfmon is defined to be set only at postmaster
restart.  Having a check hook that checks if the process setting it
has superuser privileges is meaningless.  The check hook is removed.

GUC gp_gpperfmon_send_interval is intended to be set only by
superuser.  Adjust its definition accordingly and leverage checks
built into GUC framework for superuser privileges.  The check hook for
this GUC tried to achieve the same but incorrectly.  If the check hook
was invoked at the beginning of main query processing loop by a
backend process, it would crash.  At the beginning of the main loop, a
transaction is not started yet.  The check hook invokes superuser()
interface, which performs catalog access.  Doing so without starting a
transaction is a recipe for crashing badly.  Such a crash was observed
in production at least once.

Thank you Jesse Zhang for suggesting to remove superuser check.

The patch doesn't add any tests because, after removing the check
hooks, the checks built into GUC framework are being used.  That code
path is well exercised by existing regression tests.

Reviewed-by: Daniel Gustafsson
(cherry picked from commit b5729a03)

Some callers of pg_mkdir_p() consider EEXIST as a non-error case, this
is incorrect.  When pg_mkdir_p() sets errno to EEXIST it means that the
target path already exists but is not a directory.  So as long as
pg_mkdir_p() returns -1 we should consider it as a failure.
Co-authored-by: NPaul Guo <pguo@pivotal.io>
Co-authored-by: NNing Yu <nyu@pivotal.io>
(cherry picked from commit dd6b310e)

(cherry picked from commit c8d4e32f)

pg_mkdir_p() is used by initdb and pg_basebackup to create a directory
as well as its parent directories.  The logic used to be like below:

    if (stat(path) < 0)
    {
        if (mkdir(path) < 0)
            retval = -1;
    }

It first checks for the existence of the path, if path does not
pre-exist then it creates the directory.  However if two processes try
to create path concurrently, then one possible execution order is as
below:

    A: stat(path) returns -1, so decide to create it;
    B: stat(path) returns -1, so decide to create it;
    B: mkdir(path) returns 0, successfully created path;
    A: mkdir(path) returns -1, fail to create path as it already exist;

This race condition could be fixed by swapping the order of stat() and
mkdir(), this is also what the "mkdir -p" command does.
Co-authored-by: NPaul Guo <pguo@pivotal.io>
Co-authored-by: NNing Yu <nyu@pivotal.io>
(cherry picked from commit bd8abd95)

package_tarball.bash is used in many places outside of the gpdb
repository so any modifications should be done very carefully.

This commit also introduces package_tarball.bats as an example file for
other concourse users wishing to add .bats files.
Authored-by: NAmil Khanzada <akhanzada@pivotal.io>
Reviewed-by: NMark Sliva <msliva@pivotal.io>
(cherry picked from commit 434e5e3d)

And reset it after to make sure no memory leaks there.

For instance, the deserialized transValues for new entries (not that
temporary) are not in group_buf, and not freed.
Co-authored-by: NNing Yu <nyu@pivotal.io>
(cherry picked from commit f780ca11)

So we could account its usage via GET_TOTAL_USED_SIZE(hashtable) and
decide if it's time to spill.

And now it's not needed to pfree() the transValue since it's in mpool
and will be reset after spill.
Co-authored-by: NGang Xiong <gxiong@pivotal.io>
Co-authored-by: NRichard Guo <riguo@pivotal.io>
(cherry picked from commit c9258ae3)