First, it support shell script and command:

The new syntax is:
```
! <some script>
```
This is required to run something like gpfaultinjector, gpstart, gpstop, etc.

Second, it support utility mode with blocking command and join:

The new syntax is:
```
2U&: <sql>
2U<:
```
The above example means:
- blocking in utility mode on dbid 2
- join back in previous session in utility mode on dbid 2

Fix the exception handling to allow the test to complete, and log the failed
commands instead of abort the test. This will make sure all the cleaning steps
are executed and not blocking following tests.

This also include init_file for diff comparison to ignore timestamp output from
gpfaultinjector.

Installing PL languages via gppkg without doing anything to check
the validity and functionality of the actual installation isn't
terribly exciting, and on top of that it's also already tested in
package/language.

The tests in package/plperl were all but two just copies of the
tests in pl/plperl/sql except each query were in a separate file.
The two original tests were testing for memory errors which were
fixed a very long time ago and for which the plperl code has been
rewritten since. Neither test causes issues when tested with large
inputs so remove as well.

Also remove tests in language/plperl which overlap in functionality
with what we already have.

I had to jump through some hoops to run the same test queries with
optimizer_cte_inlining both on and off, like the original tests did. The
actual tests queries are in qp_with_functional.sql, which is included by
two launcher scripts that set the options. The launcher tests are put in
the test schedule, instead of qp_with_functional.sql.

When ORCA is not enabled, the tests are run with gp_cte_sharing on and off.
That's not quite the same as inlining, but it's similar in nature in that
it makes sense to run test queries with it enabled and disabled. There were
some tests for that in with_clause and qp_with_clause tests already, but I
don't think some extra coverage will hurt us.

This is just a straightforward conversion, there might be overlapping tests
between these new tests and existing 'with_clause', 'qp_with_clause', and
upstream 'with' tests. We can clean them up later; these new tests run in
a few seconds on my laptop, so that's not urgent.

A few tests were tagged with "@skip" in the original tests. Test queries
58, 59, and 60 produced different results on different invocations,
apparently depending on the order that a subquery returned rows (OPT-2497).
I left them in TINC, as skipped tests, pending a decision on what to do
about them. Queries 28 and 29 worked fine, so I'm not sure why they were
tagged as "@skip OPT-3035" in the first place. I converted them over and
enabled like the other tests.

These error cases are covered by the 'with_clause' test in the main suite.

Commit ac2fd680 introduced ON MASTER for all external tables, a
clause until then only available on EXECUTE. The support in pg_dump
did however assume there was an ON clause, which isn't neccesarily
the case for external tables in 4.3 (and below), causing "illegal
ON clause" logging without setting an appropriate ON clause. Fix
by adding support for non-5.0 servers and rearrange logic to make
it clearer.

Also; error out on incorrect ON clause, as that could indicate
catalog corruption and include the relation name in the error
logging to aid debugging; and use static string/char PQExpBuffer
functions where possible to avoid overhead.

Apply to pg_dump and cdb_dump_agent.

The cdb_dump/restore tools are linking some of the pg_dump objects
for common code. When building, and linking, the shared objects into
pg_dump folder however there is a race condition in parallel builds
where an object compiled in pg_dump can be overwritten by the same
object compiled from pg_dump/cdb while being read by the linker in
pg_dump. While rare, an error in the CI pipeline indicates that this
might have caused a compilation failure. Fix by symlinking the shared
code into pg_dump/cdb to create a local copy.

While there, did some general tidying up: break a few long lines into
more readable chunks; make use of the $^ make variable to reference
objects from the target definition; set the $(X) extension properly;
ensure that all generated files are removed on make clean; use the
variables rather than hardcoding paths; remove references to objects
that no longer exists.

The UPDATE cases are sufficiently covered by the 'update' test in the main
test suite. The DELETE cases are covered by the 'delete' test, and many
others.

We have plenty of INSERTs with constants all over the normal test suite.
For example, in the 'insert' test. For the AO and AOCS tables, in
'appendonly' and 'aocs' tests, and the 'uao_dml' tests.

For the partitioned cases, we have inserts in the 'partition' and
'partition1' tests, for example.

These tests use column aliases that "shadow" real column names. Not very
interesting, IMHO, but keep one copy, moved to the 'as_alias' test in
main test suite. Remove the other copies, with different datatypes, they
seem totally redundant.

There's really no need to test this with all different datatypes.

(These tests also had nothing to do with MPP-21090, like the description
tags claimed.)

These expressions are as simple as it gets. If any of these queries would
fail, the system would be so broken that you would notice it very quickly.

(These tests had nothing to do with MPP-21090, like the description tags
claimed. That bug happened when you dropped a column and then added a
partition, and there are no DROP COLUMNS or ADD PARTITIONS in these tests.)

The macOS ld64 linker has an assertion on empty DATA segments
within linker Atoms. This assertion trips on the resource_manager
since it only contains uninitialized variables placed for the BSS
segment. This fails linking the backend on the resource_manager
SUBSYS object. Without anything initialized, an no exported
function symbols, the sections are:

$ nm -mgU src/backend/utils/resource_manager/SUBSYS.o
0000000000000004 (common) (alignment 2^2) external _Gp_resource_manager_policy
0000000000000001 (common) external _ResourceScheduler

With the initialization of the ResourceScheduler GUC variable:

$ nm -mgU src/backend/utils/resource_manager/SUBSYS.o
0000000000000004 (common) (alignment 2^2) external _Gp_resource_manager_policy
0000000000000004 (__DATA,__common) external _ResourceScheduler

Since the resource_manager in its current state is off anyways it
seems harmless to initialize to the correct value.

The old test script had some special logic in it. It ran the test scripts
as usual, but it also did EXPLAIN on each query, and checked that the
EXPLAIN output contained "Dynamic Index Scan" (except for a few queries
that were marked with the "@negtest True" tag). I didn't copy over that
approach; instead, I added added explicit EXPLAINs to each query, and let
gpdiff.pl to check that the plan matches that in the expected output.
That's slightly less fragile, the tests will now fail if there are some
other changes to the plans, even if they still use a Dynamic Index Scans.
I think that's quite reasonable, and shouldn't cause too many false
positives. These queries are fairly simple, so there aren't very many
possible plans for them anyway. Worst case is that we'll have to update
the expected output every now and then, when there are changes to the
planners.

The original tests were run with:
  SELECT disable_xform('CXformDynamicGet2DynamicTableScan')

but I left that out of the new tests. With Orca and
CXformDynamicGet2DynamicTableScan, some test queries fell back the Postgres
planner. That didn't seem very interesting. I'm guessing that the point
of CXformDynamicGet2DynamicTableScan was to force the queries to use an
(Dynamic) Index scan, rather than a Table Scan, but Orca seems to choose
Index Scans for all the quries even without it.

Some test queries were tagged with "@negtest True", but nevertheless used
Dynamic Index Scans, so they would've passed the original Dynamic Index
Scan checks too, if they hadn't been tagged. Perhaps Orca has improved
since the tests were written?

Float8, time, timestamp, and timestamptz have the same typlen, typyval,
typtype, typalign and typstorage attributes as int8. It is therefore
sufficient to run these tests with int8.

Likewise, date has the same type attributes as int4, so the tests on int4
cover the same ground.

In commit 989c502f, I already removed a bunch of these, but didn't catch
them all then. This removes the remaining tests on "int", where an
identical test on "int4" also exists. "int" is just an alias for "int4", so
there is no difference.

I just realized the reason these tests existed in the first place: the
"int" tests were supposed to actually test the "int2" datatype, not "int4".
The test values in the "int" test have been chosen so that they would fit
in a 16-bit integer. So if we wanted to restore the original purpose of
these tests, we should search & replace "int" with "int2". But testing all
the different datatypes isn't too interesting anyway, and there's no reason
to believe that int2 would be a special, and at the same time not test many
datatypes with different lengths, alignments and pass-by-value flags, so
let's just remove these.

Commit 5a2d9a5a cherry-picked some code to plug a memory leak from
test_postmaster_connection(), but the surrounding code looks quite
different in the upstream version where it was cherry-picked from. In our
8.3-based version of the function, 'optlines' is modified in the loop, and
doesn't point to the original allocated array anymore, so we passed a bogus
pointer to free().

This made e.g. "pg_ctl restart -w" throw GLIBC warnings.

Remove the offending call to free_readfile(), reverting that back to the
way it was before commit 5a2d9a5a. Yes, that re-introduces the memory
leak, but that's not a big deal. pg_ctl is not a server program, it will
exit as soon as it gets its job done, so the leak is very short-lived.
It was plugged only to silence Coverity warnings in the first place.

The cdb_bsa agents were all following a similar pattern, strduping
the optarg and then strncpy()'ing the already allocated string into
a second allocated string which was used for processing. Remove the
extra malloc() + strncpy() since we already have perfectly good
strings to operate on from the optarg processing. Also remove calls
to free() just before exit since thats unnecessary.

There was a mechanism in the DMLSQLTestCase class, in __init__.py, to
enumerate all the different plans that ORCA could generate for them, and
execute them all. The test cases marked with the "@execute_all_plans True"
tag were supposed to be executed that way. However, it was broken: the
DMLSQLTestCase class was not used, the test_dml class defined in
test_dml.py inherited directly from SQLTestCase, not DMLSQLTestCase, so all
the tests were executed the usual way.

I assume that was an oversight, although it's also possible that these
tests were copy-pasted from somewhere else, and the @execute_all_plans were
just left over and not meant to do anything. In any case, I don't think
it's worthwhile to enumerate and execute all the plans. These test queries
are fairly simple, so there probably aren't very many plans for each query,
and they are not that special, anyway.

To avoid confusion, remove the broken mechanism, and all the
@execute_all_plans tags.

Previously, backup files were keyed only to the dbid of the segment from
which they were backed up, which caused problems when trying to restore
files backed up on a mirror segment, restore files to a new cluster with
a different dbid mapping, and so forth.

This commit fixes the above issues by changing the backup file naming
convention to incorporate the content id of the backed-up segment and
enables restoring files based on the content id.  Also, gpdbrestore
seamlessly handles mixing the old and new filename formats, so users can
still restore a backup made with a GPDB 4.3 version of gpcrondump that
used the old format.

heterogeneous_singlekey_nonptcol_df_4.sql is identical to
heterogeneous_singlekey_nonptcol_df_3.sql

and

heterogeneous_singlekey_ptcol_df_4.sql is identical to
heterogeneous_singlekey_ptcol_df_3.sql

The "notsupportedsql" directory doesn't exist.

* GPDB DOCS - Remove error table logging information.

* resolve conflict updated feature summary

[ci skip]

The test `dml_oids_constraint` had two problems:

0. It assumes that CTAS has implicit hash distribution, which is not true for ORCA
0. Given the random distribution policy, it is sensitive to the number of segments

Making the CTAS explicitly hash-distributed resolves both issues

Signed-off-by: NDavid Sharp <dsharp@pivotal.io>

In Concourse: use `regexp:` s3 files for versioning with git describe.
For packaging scripts, pass in the filename with @GP_VERSION@ and
substitute the version string in the filename in the script.
Signed-off-by: NDavid Sharp <dsharp@pivotal.io>

Not necessary since the TINC code got merged into the GPDB code.
Signed-off-by: NJingyi Mei <jmei@pivotal.io>

* doc mods for per-function guc setting support

* incorporate david s comments

[ci skip]

removing template relnotes from oss build. Pivotal-specific relnotes now hosted in private repo (#1948)

[ci skip]

[ci skip]

This is a fix-up for 4ca52894 to add missing index on oid
for pg_resgroup and pg_resgroupcapablity. The index is declared to be
unique so dispatching of the oids will success.

* Fix gpload reuse table issues

This commit fixes issues below:

1, reports unexpected error when reuse_table is enabled
2, doesn't recognize GPDB delimiter syntax like E'\t'
3, unnecessary case sensible of column names
Signed-off-by: NAdam Lee <ali@pivotal.io>
Signed-off-by: NNing Wu <nwu@pivotal.io>

* delimiter '\t' handled

* Fixed report warnings wrongly when reusable table exists

* remove the tailing space and simplify sql

This reverts commit 629b0e2a.

It made gpcloud tests to crash, for an unknown reason. I could not figure
out how to run the tests successfully to debug that, so I'm reverting the
patch, to unblock others.

Now that external tables use resource owners to clean up on abort, we
no longer need to ensure that ExecutorEnd gets called on failed portals.

Remove PG_TRY-CATCH block in nodeAgg.c that's no longer needed. This
demonstrates well why we should not call ExecutorEnd on abort; there
might well have been more places that previously needed a PG_TRY-CATCH
block like this, but didn't have one.

See also discussion on this on the gpdb-dev mailing list:
https://groups.google.com/a/greenplum.org/forum/#!topic/gpdb-dev/-BTKTbLuP1k

There is no limited_schedule anymore, kill the commented out line
invoking it.

The generated bash scripts for running the regression tests are
executing with "set -exo pipefail" where "-o pipefail" makes the
script exit on the first error. Since pg_regress will exit with
returncode 1 on a run with a failed suite, the script will exit
before reaching the check for regression.diffs. Fix by adding a
trap function instead like the other CI scripts.

This use the same construction as other CI scripts of using find
rather than hardcoding the path, which can be seen as overkill
since there's currently only a single file. This is however a
cheap operation and will ensure that we don't miss any future
regression.diffs added somewhere, or a chang to  pg_regress
outputdir parameter.

In the passing, also move existing tests GPDB-specific tests out of the
'alter_table' test. The 'alter_table' test is inherited from the upstream,
so let's try to keep it unmodified.

gp_resource_manager is reserved for later use to switch resource control stratagy from resource queue to an under-developing resource statagy named resource group. Eg:

gpconfig -c gp_resource_manager -v 'group'
gpconfig -c  gp_resource_manager -v 'queue'

To make it work, a restart of cluster is needed.