This commit adds the GPDB side changes required to support GIN Indexes with
ORCA.  It also adds a new test file gp_gin_indexes to test plans produced for
ORCA/planner.

GIN indexes are not supported with index expression or predicate constraints.
ORCA does not support it currently for other types of indexes too.

ALTER TABLE DROP COLUMN followed by reorganize leads to loss of column
encoding settings of the dropped column. When the column's compresstype
encoding is incorrect, we can encounter block version mismatch error
later during block info validation check of the dropped column.

One idea was to skip dropped columns when constructing AOCSScanDesc.
However, dropping all columns is a special case that is not easily handled
because it is not equivalent to deleted rows. Instead, the fix is to preserve
column encoding settings even for dropped columns.
Co-authored-by: NSoumyadeep Chakraborty <sochakraborty@pivotal.io>
Co-authored-by: NIvan Leskin <leskin.in@arenadata.io>

GUC gp_enable_gpperfmon is defined to be set only at postmaster
restart.  Having a check hook that checks if the process setting it
has superuser privileges is meaningless.  The check hook is removed.

GUC gp_gpperfmon_send_interval is intended to be set only by
superuser.  Adjust its definition accordingly and leverage checks
built into GUC framework for superuser privileges.  The check hook for
this GUC tried to achieve the same but incorrectly.  If the check hook
was invoked at the beginning of main query processing loop by a
backend process, it would crash.  At the beginning of the main loop, a
transaction is not started yet.  The check hook invokes superuser()
interface, which performs catalog access.  Doing so without starting a
transaction is a recipe for crashing badly.  Such a crash was observed
in production at least once.

Thank you Jesse Zhang for suggesting to remove superuser check.

The patch doesn't add any tests because, after removing the check
hooks, the checks built into GUC framework are being used.  That code
path is well exercised by existing regression tests.

Reviewed-by: Daniel Gustafsson
(cherry picked from commit b5729a03)

Some callers of pg_mkdir_p() consider EEXIST as a non-error case, this
is incorrect.  When pg_mkdir_p() sets errno to EEXIST it means that the
target path already exists but is not a directory.  So as long as
pg_mkdir_p() returns -1 we should consider it as a failure.
Co-authored-by: NPaul Guo <pguo@pivotal.io>
Co-authored-by: NNing Yu <nyu@pivotal.io>
(cherry picked from commit dd6b310e)

(cherry picked from commit c8d4e32f)

pg_mkdir_p() is used by initdb and pg_basebackup to create a directory
as well as its parent directories.  The logic used to be like below:

    if (stat(path) < 0)
    {
        if (mkdir(path) < 0)
            retval = -1;
    }

It first checks for the existence of the path, if path does not
pre-exist then it creates the directory.  However if two processes try
to create path concurrently, then one possible execution order is as
below:

    A: stat(path) returns -1, so decide to create it;
    B: stat(path) returns -1, so decide to create it;
    B: mkdir(path) returns 0, successfully created path;
    A: mkdir(path) returns -1, fail to create path as it already exist;

This race condition could be fixed by swapping the order of stat() and
mkdir(), this is also what the "mkdir -p" command does.
Co-authored-by: NPaul Guo <pguo@pivotal.io>
Co-authored-by: NNing Yu <nyu@pivotal.io>
(cherry picked from commit bd8abd95)

package_tarball.bash is used in many places outside of the gpdb
repository so any modifications should be done very carefully.

This commit also introduces package_tarball.bats as an example file for
other concourse users wishing to add .bats files.
Authored-by: NAmil Khanzada <akhanzada@pivotal.io>
Reviewed-by: NMark Sliva <msliva@pivotal.io>
(cherry picked from commit 434e5e3d)

And reset it after to make sure no memory leaks there.

For instance, the deserialized transValues for new entries (not that
temporary) are not in group_buf, and not freed.
Co-authored-by: NNing Yu <nyu@pivotal.io>
(cherry picked from commit f780ca11)

So we could account its usage via GET_TOTAL_USED_SIZE(hashtable) and
decide if it's time to spill.

And now it's not needed to pfree() the transValue since it's in mpool
and will be reset after spill.
Co-authored-by: NGang Xiong <gxiong@pivotal.io>
Co-authored-by: NRichard Guo <riguo@pivotal.io>
(cherry picked from commit c9258ae3)

AggStates are now pointers allocated in aggcontext with type INTERNAL,
just spilling the pointers don't decrease the memory usage and have
possible memory leak if combining states without free.

This commit serialize the aggstates, write the real data into file and
free the memory.

(cherry picked from commit 6dadce04)

* docs - can now reset pxf on host and cluster

* remove the (re)s

NON_PRODUCTION_FILES.txt is empty and now no longer used.
The external pipelines which use these scripts were also updated to
remove the files.
Co-authored-by: NTingfang Bao <bbao@pivotal.io>
Co-authored-by: NAmil Khanzada <akhanzada@pivotal.io>
(cherry picked from commit 0844231d)

In resource group mode we ever introduced an operator memory automatic
enlarging logic for hashagg, the point is to let hashagg fail on actual
OOM instead of a soft quote checking, this helps to let hashagg run
successfully with an initial low operator memory.

However a bug was introduced by the auto enlarging logic, hashagg
spilling can be disabled in resource group mode by accident.

On the other hand we introduced a memory_spill_ratio=0 mode in resource
group to use statement_mem for operators, which is the same behavior as
resource queue.  The statement_mem setting is usually large enough and
fine tuned by the users, in such a case we do not need the auto enlarge
for hashagg, and it is better to keep the old quote checking behavior.

In such a case we revert the hashagg related changes from below commits:

- 40d955d6 Rid resource group on hashagg spill evaluation (#8199)
- ede74cdc resgroup: reduce log level for operator memory overuse
- f053e6cd resgroup: allow memory overuse for hashagg spill meta data
- 90795402 resgroup: allow operators enlarge their memory quota
Reviewed-by: NAdam Lee <ali@pivotal.io>
Reviewed-by: NWeinan WANG <wewang@pivotal.io>

Discussion: https://groups.google.com/a/greenplum.org/d/msg/gpdb-dev/30hiTArxsgo/aydVMcrXBQAJ

(cherry picked from commit 3827546a)

When a failed raised by `CREATE DATABASE`, relevant files, directories
should not be leftover.

Add new DB information into `pendingDbDeletes` list, so that promise `CREATE DATABASE` as a 2pc safe command.
Co-authored-by: NAsim R P <apraveen@pivotal.io>
(cherry picked from commit 7c89cde7)

We accidentally skipped the fts probe on the primary in
f2ca17a0 when it should be skipped on the master. The fault point
'fts_probe' can only be encountered on the master.

(cherry picked from commit 3e1332ad)

postgresql_conf_gp_addition is just an example for what can be injected
into postgresql.conf via gpinitsystem, but the values in there make
little sense to ship for production.

It has not been changed in many years and is mistakenly copied into the
files we distributed with Greenplum (via gpMgmt/Makefile target that
copies bin/ files).
Authored-by: NAmil Khanzada <akhanzada@pivotal.io>
(cherry picked from commit feb3a399)

There is different build system code that defines which contrib
extensions are built and installed.
gpAux/Makefile should not be building any contrib extensions.
Authored-by: NAmil Khanzada <akhanzada@pivotal.io>
(cherry picked from commit 85e13f7f)

* docs - add examples for GDD and add locking information for SELECT...FOR <lock_strength>

* docs - remove Postgres Planner restriction for split update.

* docs - revert previous change for split update and Postrgres Planner

* docs - fix typos

They will no longer be packaged per below commit:
52a9618bAuthored-by: NAmil Khanzada <akhanzada@pivotal.io>
(cherry picked from commit 87eacc2a)

The installation target in gpMgmt/ blindly copied Perl programs and
modules in order for the gpload test to be able to run gpdiff from
PATH. This causes non-production code to be erroneously installed
and required an elaborate dance to remove before packaging.

This removes the installation and instead installs the required
Perl code from regress as symlinks just like how other test suites
are doing it.

A careful observer might note that TEST_REMOTE.py still call the
Perl programs from PATH, and that's Ok (for now) as it's only to
be invoked via the clients package CI and that CI performs a copy
step of gpdiff much like how this patch does a symlink. There are
probably parts of that which can be cleaned up and simplified but
for now we are focusing on the core server parts.

Removing the non-defunct NON_PRODUCTION_FILES step remains as a
task after this.
Reviewed-by: NHeikki Linnakangas <hlinnakangas@pivotal.io>
Reviewed-by: NJacob Champion <pchampion@pivotal.io>
Reviewed-by: NAmil Khanzada <akhanzada@pivotal.io>
(cherry picked from commit 52a9618b)

When given --print-failure-diffs, pg_regress prints the contents of
regression.diffs after a failure.

(cherry picked from commit b5365010)

Print pg_regress failure diffs using fgets.

(cherry picked from commit 99e54168)

The test cases of gpfdist_ssl sometimes
fail on pipeline. We don't know the root
cause yet. So just disable them. And we will
enable them after fixing.

ExecMaterializeSlot() transformed any tuple to a virtual tuple via
slot_getallattrs(), then formed a heaptuple from it, ctid was lost here
since virtual tuples have no system columns.

This commit copies the entire htup directly if we have a regular
physical tuple but not locally palloc'd.

(cherry picked from commit 1eeae2e9)

Also, bump ORCA version to 3.61.0

Also bump version to 3.60.0.

An elusive missing header file was detected when the Ubuntu job failed
in CI. Centos and Mac did not show any indication of a missing header
file and no side effects were detected. However, Ubuntu was returning an
int32 in makeDefElem instead of a 64-bit pointer.
Co-authored-by: NAshwin Agrawal <aagrawal@pivotal.io>
Co-authored-by: NJesse Zhang <jzhang@pivotal.io>
Co-authored-by: NFrancisco Guerrero <aguerrero@pivotal.io>

Currently, PXF is not propagating the format options in the external
table framework. In Foreign Data Wrappers, these options are defined at
the foreign table-level and are being propagated to PXF.

In order for the PXF Server to better support both FDW and external
tables we are consistently passing format information from both clients.

The function create_target_tablespace_layout() correctly decorates
link as a const and also follows the pattern of the rest of the
tablespace functions. Operate on a local pointer instead and don't
mutate on the referenced argument. Also, if the intension was to
mutate the referenced argument, then it should have first be pfreed.
Reviewed-by: NDaniel Gustafsson <dgustafsson@pivotal.io>
Reviewed-by: NSoumyadeep Chakraborty <sochakraborty@pivotal.io>
(cherry picked from commit 8e8c5b33)

* Docs: Using 'Postgres Planner' terminology consistently

* Postgres planner -> Postgres Planner

* remove another reference to 'legacy' planner

* Few more fixes

* Remove syntax options for creating OIDS; add pivotal-only statements of non-support

* Some edits from Lisa

* Clarify error with oids on partitioned/co table.

* Feedback from Mel

* Add deprecation/removal wording suggested by Rob

* Add centos7 command to start config.service

* Add some text around starting immediately

For partitioned tables, in case of DELETE/UPDATE operation on the root
table, we must acquire ExclusiveLock on the root and leaf partitions, so
that any other operation requesting lock higher than AccessShareLock and
ShareLock must wait on QD.

If we don't acquire lock for the leaf partitions, other concurrent DML
operation may be dispatched to the segments and a deadlock can occur.
Refer to the test case added for GDD.

resultRelations may not have all the entries including the root and leaf
partition tables, so in this fix we explicitly acquire locks on all the
tables.

While GDD is on, it will be able to detect the deadlock and cancel one
of the query, so we don't acquire locks on the leaf parts if the DML
operation is on the root.
Co-authored-by: NAlexandra Wang <lewang@pivotal.io>

The PostgreSQL 9.2 merge introduced a couple of new cursor options,
including CURSOR_OPT_GENERIC_PLAN, which clashed with the 0x0040 value that
we were already using in GPDB for the GPDB-specific CURSOR_OPT_UPDATABLE
option. Fix the conflict.

The CURSOR_OPT_UPDATABLE flag was used in cursors opened in PL/pgSQL, so
the mixup caused generic plans to be used for all PL/pgSQL cursors. Add
a test case for that.

The PostgreSQL 9.2 merge happened in GPDB 6, so backport to GPDB 6.
Reviewed-by: NAsim R P <apraveen@pivotal.io>
Reviewed-by: NPengzhou Tang <ptang@pivotal.io>

Ooops.

We only injected the skip fault for fts once, causing a race condition
in the test. Here's the problematic timeline:

1. Inject skip fault for fts
2. Fts runs, skip fault is gone
3. Panic primary
4. Inject Noop record into mirror
5. Promote mirror
6. Noop record never replayed and test fails.

(cherry picked from commit f2ca17a0)