The compiled python files ending with 'c' were only to some degree
removed on make clean, as the list in the Makefile hadn't been kept
in sync with the gitignore list.

Discussion: https://github.com/greenplum-db/gpdb/pull/8179
Reviewed by Bradford Boyle, Kalen Krempely, Jamie McAtamney and
many more

The .gitignore for gpMgmt had lots of entries for long since removed
applications. The apps and their respective delete commit were:

  * gp_df a7405a274c8ad392141ed4006adde8c506467af9
  * gpbitmapreindex e9b9081bf948f60aadf4f41a4e3f8ff43ce888ce
  * gpdbrestore & gpcrondump 8922315e
  * gpdebug 5ef6b86b
  * gpfilespace 5a3a39bc
  * gpkill 29f17dd8
  * gpmigrator 68178574

Discussion: https://github.com/greenplum-db/gpdb/pull/8179
Reviewed by Bradford Boyle, Kalen Krempely, Jamie McAtamney and
many more

And reset it after to make sure no memory leaks there.

For instance, the deserialized transValues for new entries (not that
temporary) are not in group_buf, and not freed.
Co-authored-by: NNing Yu <nyu@pivotal.io>

So we could account its usage via GET_TOTAL_USED_SIZE(hashtable) and
decide if it's time to spill.

And now it's not needed to pfree() the transValue since it's in mpool
and will be reset after spill.
Co-authored-by: NGang Xiong <gxiong@pivotal.io>
Co-authored-by: NRichard Guo <riguo@pivotal.io>

AggStates are now pointers allocated in aggcontext with type INTERNAL,
just spilling the pointers don't decrease the memory usage and have
possible memory leak if combining states without free.

This commit serialize the aggstates, write the real data into file and
free the memory.

* docs - can now reset pxf on host and cluster

* remove the (re)s

NON_PRODUCTION_FILES.txt is empty and now no longer used.
The external pipelines which use these scripts were also updated to
remove the files.
Co-authored-by: NTingfang Bao <bbao@pivotal.io>
Co-authored-by: NAmil Khanzada <akhanzada@pivotal.io>

In resource group mode we ever introduced an operator memory automatic
enlarging logic for hashagg, the point is to let hashagg fail on actual
OOM instead of a soft quote checking, this helps to let hashagg run
successfully with an initial low operator memory.

However a bug was introduced by the auto enlarging logic, hashagg
spilling can be disabled in resource group mode by accident.

On the other hand we introduced a memory_spill_ratio=0 mode in resource
group to use statement_mem for operators, which is the same behavior as
resource queue.  The statement_mem setting is usually large enough and
fine tuned by the users, in such a case we do not need the auto enlarge
for hashagg, and it is better to keep the old quote checking behavior.

In such a case we revert the hashagg related changes from below commits:

- 40d955d6 Rid resource group on hashagg spill evaluation (#8199)
- ede74cdc resgroup: reduce log level for operator memory overuse
- f053e6cd resgroup: allow memory overuse for hashagg spill meta data
- 90795402 resgroup: allow operators enlarge their memory quota
Reviewed-by: NAdam Lee <ali@pivotal.io>
Reviewed-by: NWeinan WANG <wewang@pivotal.io>

Discussion: https://groups.google.com/a/greenplum.org/d/msg/gpdb-dev/30hiTArxsgo/aydVMcrXBQAJ

* docs - add examples for GDD and add locking information for SELECT...FOR <lock_strength>

* docs - remove Postgres Planner restriction for split update.

* docs - revert previous change for split update and Postrgres Planner

* docs - fix typos

They will no longer be packaged per below commit:
52a9618bAuthored-by: NAmil Khanzada <akhanzada@pivotal.io>

The installation target in gpMgmt/ blindly copied Perl programs and
modules in order for the gpload test to be able to run gpdiff from
PATH. This causes non-production code to be erroneously installed
and required an elaborate dance to remove before packaging.

This removes the installation and instead installs the required
Perl code from regress as symlinks just like how other test suites
are doing it.

A careful observer might note that TEST_REMOTE.py still call the
Perl programs from PATH, and that's Ok (for now) as it's only to
be invoked via the clients package CI and that CI performs a copy
step of gpdiff much like how this patch does a symlink. There are
probably parts of that which can be cleaned up and simplified but
for now we are focusing on the core server parts.

Removing the non-defunct NON_PRODUCTION_FILES step remains as a
task after this.
Reviewed-by: NHeikki Linnakangas <hlinnakangas@pivotal.io>
Reviewed-by: NJacob Champion <pchampion@pivotal.io>
Reviewed-by: NAmil Khanzada <akhanzada@pivotal.io>

Fixing a small bug in my regexp searching for duplicates yielded two
additional hits. Fix by adding to the mailmap file.

It seems that the duplicate prototype snuck in during the 8.4 merge,
which should've been mostly a noop as I had synced pgcrypto with 8.4
in 16d26ee9. I had however placed the prototype on the wrong line
and so git merge was able to cleanly place the protype a few lines
down, creating a duplicate.

The gpstringsubs program replaces tokens in input .source test files
when creating output .sql/.out files for pg_regress. There were no
more callsites for the gp_glob_connect token however so remove that,
and also remove tests for gpwhich_<binary> tokens which aren't in
use in tests and thus might start failing randomly, which is exactly
what happened in another patch where Perl programs were no longer
incorrectly installed in PATH. Without this, removing the Perl test
apps from PATH results in errors like this:

	which: no get_ereport.pl in ( .. )
Reviewed-by: NHeikki Linnakangas <hlinnakangas@pivotal.io>

The test cases of gpfdist_ssl sometimes
fail on pipeline. We don't know the root
cause yet. So just disable them. And we will
enable them after fixing.

* Fix quirk of isolation2 test

- Fix the line continuation behaviour and document
Instead of matching "; *$" to detect the end of a sql end, the original
code was using ";$". This creates a totally different output for tests
like:
a)
SELECT * FROM t1;
SELECT * FROM t2;
b)
SELECT * FROM t1;<space>
SELECT * FROM t2;
For a), two SELECT statements will be sent separately.
For b), they will be sent together as
        "SELECT * FROM t1; SELECT * FROM t2;"

Spaces are playing too much magic here, some of them could be a bug in
the test itself. For example, the oom_query test should NOT pass since
it misses error output in the expected.

This change unifies the behaviours by matching with "; *$". So if a line
is end with ';', no matter spaces follows it or not, the statement will
be sent at once.

- Document the known issue of multi SQL in one line.

This issue was found during debugging pipeline failure of #8211
Authored-by: NChen Mulong <muchen@pivotal.io>

We accidentally skipped the fts probe on the primary in
f2ca17a0 when it should be skipped on the master. The fault point
'fts_probe' can only be encountered on the master.

Also, bump ORCA version to 2.61.0

Also bump version to 3.60.0.

This function was modified in GPDB just for pg_regress. pg_regress
needs to locate gpdiff.pl and gpstringsubs.pl scripts where it uses
find_other_exec(). Since not correct version was passed to it, this
modification was required to find_other_exec(). These scripts have
code to correctly print the version information on "-V". So, regular
unmodified version of find_other_exec() works if the version is passed
correctly from pg_regress.
Reviewed-by: NAsim R P <apraveen@pivotal.io>
Reviewed-by: NDaniel Gustafsson <dgustafsson@pivotal.io>
Reviewed-by: NHeikki Linnakangas <hlinnakangas@pivotal.io>

- Adds hint to user on how to disable FTS.

- Registers warnings outside of the core injection framework

  This allows us to create warnings that are GPDB specific, for
  things like FTS without tainting the core framework which is
  theoretically Postgres-only dependent.

We don't build the HTML docs in GPDB like we do in PostgreSQL, but let's
be tidy anyway.

* We don't allow tabs in SGML files. There's a "make check-tabs" target
  to check for them.

* Fix reference to non-existing SQL-ALTERROLE-title sections. (Should be
  SQL-ALTERROLE)

* Fix copy-pasto in link to ALTER RESOURCE GROUP.

* In func.sgml, a tag was typoed.

* In pg_dumpall.sgml, the term-tag is not allowed in this part in DocBook
  format.
Reviewed-by: NLisa Owen <lowen@pivotal.io>

When a failed raised by `CREATE DATABASE`, relevant files, directories
should not be leftover.

Add new DB information into `pendingDbDeletes` list, so that promise `CREATE DATABASE` as a 2pc safe command.
Co-authored-by: NAsim R P <apraveen@pivotal.io>

ExecMaterializeSlot() transformed any tuple to a virtual tuple via
slot_getallattrs(), then formed a heaptuple from it, ctid was lost here
since virtual tuples have no system columns.

This commit copies the entire htup directly if we have a regular
physical tuple but not locally palloc'd.

An elusive missing header file was detected when the Ubuntu job failed
in CI. Centos and Mac did not show any indication of a missing header
file and no side effects were detected. However, Ubuntu was returning an
int32 in makeDefElem instead of a 64-bit pointer.
Co-authored-by: NAshwin Agrawal <aagrawal@pivotal.io>
Co-authored-by: NJesse Zhang <jzhang@pivotal.io>
Co-authored-by: NFrancisco Guerrero <aguerrero@pivotal.io>

* Docs: Using 'Postgres Planner' terminology consistently

* Postgres planner -> Postgres Planner

* remove another reference to 'legacy' planner

* Few more fixes

* Remove syntax options for creating OIDS; add pivotal-only statements of non-support

* Some edits from Lisa

* Clarify error with oids on partitioned/co table.

* Feedback from Mel

* Add deprecation/removal wording suggested by Rob

* Add centos7 command to start config.service

* Add some text around starting immediately

postgresql_conf_gp_addition is just an example for what can be injected
into postgresql.conf via gpinitsystem, but the values in there make
little sense to ship for production.

It has not been changed in many years and is mistakenly copied into the
files we distributed with Greenplum (via gpMgmt/Makefile target that
copies bin/ files).
Authored-by: NAmil Khanzada <akhanzada@pivotal.io>

There is different build system code that defines which contrib
extensions are built and installed.
gpAux/Makefile should not be building any contrib extensions.
Authored-by: NAmil Khanzada <akhanzada@pivotal.io>

Currently, PXF is not propagating the format options in the external
table framework. In Foreign Data Wrappers, these options are defined at
the foreign table-level and are being propagated to PXF.

In order for the PXF Server to better support both FDW and external
tables we are consistently passing format information from both clients.

The PostgreSQL 9.2 merge introduced a couple of new cursor options,
including CURSOR_OPT_GENERIC_PLAN, which clashed with the 0x0040 value that
we were already using in GPDB for the GPDB-specific CURSOR_OPT_UPDATABLE
option. Fix the conflict.

The CURSOR_OPT_UPDATABLE flag was used in cursors opened in PL/pgSQL, so
the mixup caused generic plans to be used for all PL/pgSQL cursors. Add
a test case for that.

The PostgreSQL 9.2 merge happened in GPDB 6, so backport to GPDB 6.
Reviewed-by: NAsim R P <apraveen@pivotal.io>
Reviewed-by: NPengzhou Tang <ptang@pivotal.io>

Git stores the 'x' flag for files in the repository. For no good reason,
some files had it set differently from upstream. It's harmless, but let's
avoid spurious differences.

These were removed from upstream years ago already, but were left over in
GPDB for some reason. GPDB never supported these.