Server release candidate artifacts are not published until after an
extensive set of tests have passed in the CI pipeline. These tests
include ICW and all the CLI test suites. It is not unusual for time
between a commit being pushed to a release candidate being published to
be several hours, potentially slowing down the feedback cycle for
component teams.

This commit adds a "published" output to the compliation tasks. The new
build artifact is stored in an immutable GCS bucket with the version in
the filename. This makes it trivial for other pipelines to safely
consume the latest build.

This artifact **has not** passed any sort of testing (e.g., ICW) and
should only be used in development pipelines that need
near-instantaneous feedback on commits going into GPDB.

For the server-build artifact, `((rc-build-type-gcs))` will resolve to
`.debug` for the with asserts pipeline and `''` (i.e., empty string) for the
without asserts pipeline. The two types of server artifacts that are
"published" are:

1. server-build
2. server-rc

server-build is the output of the compilation task and has had no
testing; server-rc is a release candidate of the server component.
Authored-by: NBradford D. Boyle <bboyle@pivotal.io>
(cherry picked from commit 94a8ffc9)

* docs - add guc optimizer_use_gpdb_allocators

* docs - add information about improved GPDB query mem. mgmt.

Tests that a use can upgrade a partitioned heap table from 5X to 6X and
validates that the data exists on the upgraded cluster.
Co-authored-by: NAdam Berlin <aberlin@pivotal.io>

Issue is when new cluster has higher next xid than old cluster. In that
case, new cluster moves next xid backwards in `copy_xact_xlog_xid()`.
This is problematic as subsequent tuple inserts are not frozen, but will
have a lower xmin then the relfrozenxid defined on the relation. That
violates the definition of relfrozenxid and VACUUM FREEZE will choke.

This was identified by the upgrade integration tests which create a
fresh GPDB5 and GPDB6 cluster. In this scenario immediately after
initdb, GPDB6 has higher next xid than GPDB5.

Given that template0 is not connectable, it should not need vacuum.
Solution is to remove VACUUM FREEZE on template0 since it does not work
in all scenarios. Additionally this removes a difference with upstream
postgres.

I found the logic to decide the target locus hard to understand, so I
rewrote it in a table-driven approach. I hope it's not just me.

Fixes github issue https://github.com/greenplum-db/gpdb/issues/8711Reviewed-by: NZhenghua Lyu <zlv@pivotal.io>

* Can use gpcopy to migrate data

* Conditionalize gpcopy references

* Edits for review comments

We introduce a new type of memory pool and memory pool manager:
CMemoryPoolPalloc and CMemoryPoolPallocManager

The motivation for this PR is to improve memory allocation/deallocation
performance when using GPDB allocators. Additionally, we would like to
use the GPDB memory allocators by default (change the default for
optimizer_use_gpdb_allocators to on), to prevent ORCA from crashing when
we run out of memory (OOM). However, with the current way of doing
things, doing so would add around 10 % performance overhead to ORCA.

CMemoryPoolPallocManager overrides the default CMemoryPoolManager in
ORCA, and instead creates a CMemoryPoolPalloc memory pool instead of a
CMemoryPoolTracker. In CMemoryPoolPalloc, we now call MemoryContextAlloc
and pfree instead of gp_malloc and gp_free, and we don’t do any memory
accounting.

So where does the performance improvement come from? Previously, we
would (essentially) pass in gp_malloc and gp_free to an underlying
allocation structure (which has been removed on the ORCA side). However,
we would add additional headers and overhead to maintain a list of all
of these allocations. When tearing down the memory pool, we would
iterate through the list of allocations and explicitly free each one. So
we would end up doing overhead on the ORCA side, AND the GPDB side.
However, the overhead on both sides was quite expensive!

If you want to compare against the previous implementation, see the
Allocate and Teardown functions in CMemoryPoolTracker.

With this PR, we improve optimization time by ~15% on average and up to
30-40% on some queries which are memory intensive.

This PR does remove memory accounting in ORCA. This was only enabled
when the optimizer_use_gpdb_allocators GUC was set. By setting
`optimizer_use_gpdb_allocators`, we still capture the memory used when
optimizing a query in ORCA, without the overhead of the memory
accounting framework.

Additionally, Add a top level ORCA context where new contexts are created

The OptimizerMemoryContext is initialized in InitPostgres(). For each
memory pool in ORCA, a new memory context is created in
OptimizerMemoryContext.

Bumps ORCA version to 3.74.0

This is a re-commit of 1db9b27a, which didn't properly catch/rethrow
exceptions in gpos_init.
Co-authored-by: NShreedhar Hardikar <shardikar@pivotal.io>
Co-authored-by: NChris Hajas <chajas@pivotal.io>
(cherry picked from commit 99dfccc2)

Spotted when analyzing a CI failure pertaining to another test.

Reviewed by Heikki and Georgios.

(cherry picked from commit 6958b7e8)

The privious pattern used by the test was not strong enough.  It could
accidentally matched names of partitioned tables.  Name of a partition
being added is generated by suffixing a random number, if no name is
specified by the user.  E.g. "sales_1_prt_r1171829080_2_prt_usa".  At
least one time, the test failed CI due to this weakness.

The new pattern is strong enough to match only the auxiliary table
names that end with "_<oid>".

Reviewed by Heikki and Georgios.

(cherry picked from commit eca68bbc)

* docs - mention resource groups in spill file topic, misc edits

* edits requested by david

* docs - move gpmapreduce yaml info to own utility page; misc topic edits

* relocate topic and graphic, add shortdesc, fix linking

Earlier, COPY <Catalogtable> from <file> was allowed irrespective of the
value of allow_system_table_mods. This commit restricts such statements
only when allow_system_table_mods is set to ON.

Co-Authored-By: Ashwin Agrawal<aagrawal@pivotal.io>

This reverts commit 1db9b27a.

This is breaking a memory accounting test.

We introduce a new type of memory pool and memory pool manager:
CMemoryPoolPalloc and CMemoryPoolPallocManager

The motivation for this PR is to improve memory allocation/deallocation
performance when using GPDB allocators. Additionally, we would like to
use the GPDB memory allocators by default (change the default for
optimizer_use_gpdb_allocators to on), to prevent ORCA from crashing when
we run out of memory (OOM). However, with the current way of doing
things, doing so would add around 10 % performance overhead to ORCA.

CMemoryPoolPallocManager overrides the default CMemoryPoolManager in
ORCA, and instead creates a CMemoryPoolPalloc memory pool instead of a
CMemoryPoolTracker. In CMemoryPoolPalloc, we now call MemoryContextAlloc
and pfree instead of gp_malloc and gp_free, and we don’t do any memory
accounting.

So where does the performance improvement come from? Previously, we
would (essentially) pass in gp_malloc and gp_free to an underlying
allocation structure (which has been removed on the ORCA side). However,
we would add additional headers and overhead to maintain a list of all
of these allocations. When tearing down the memory pool, we would
iterate through the list of allocations and explicitly free each one. So
we would end up doing overhead on the ORCA side, AND the GPDB side.
However, the overhead on both sides was quite expensive!

If you want to compare against the previous implementation, see the
Allocate and Teardown functions in CMemoryPoolTracker.

With this PR, we improve optimization time by ~15% on average and up to
30-40% on some queries which are memory intensive.

This PR does remove memory accounting in ORCA. This was only enabled
when the optimizer_use_gpdb_allocators GUC was set. By setting
`optimizer_use_gpdb_allocators`, we still capture the memory used when
optimizing a query in ORCA, without the overhead of the memory
accounting framework.

Additionally, Add a top level ORCA context where new contexts are created

The OptimizerMemoryContext is initialized in InitPostgres(). For each
memory pool in ORCA, a new memory context is created in
OptimizerMemoryContext.

Bumps ORCA version to 3.74.0
Co-authored-by: NShreedhar Hardikar <shardikar@pivotal.io>
Co-authored-by: NChris Hajas <chajas@pivotal.io>

- as an experiment. We want to see if it becomes less flakey.

(cherry picked from commit b99920ff)

Otherwise it is necessary to always pass the flag set to 'no' for the package
builds. This commit completes <3827c3a6> regarding tab completion.
Reported-by: NBradford D. Boyle <bboyle@pivotal.io>
Reviewed-by: NBradford D. Boyle <bboyle@pivotal.io>
Reviewed-by: NXin Zhang <xzhang@pivotal.io>
(cherry picked from commit 15f80832)

When building gpdb with a non-empty DESTDIR, the build would fail
because certain Makefiles were not correctly account for it.
Additionally, when we make a symlink for `gpcheckcat` we *should not*
include DESTDIR in the target. A common usage for DESTDIR is to allow
moving the build artifacts after the build is completed. If the target
includes DESTDIR, than the link could point to a non-existing path.
Authored-by: NBradford D. Boyle <bboyle@pivotal.io>
(cherry picked from commit 7de35b7e)

* docs - add GUC optimizer_enable_dml

* docs - review comment updates. Add guc to table.

* docs - add optimizer_enable_dml GUC to ditamap

* docs - promote pxf protocol over s3 protocol

--Reordered listing of pxf and s3.
--Added link to pxf protocol from s3 protocol
--Updated some pxf/s3 information in g-external-tables.xml
 pxf uses CREATE EXTENSION, s3 uses CREATE PROTOCOL

* docs - updated ditamap - to put PXF first under "Working with External Data"

Corresponding ORCA commits:
* Refactor: Simplify property derivation in CExpression
* Support on-demand property derivation in CDrvdPropRelational
* Rename DrvdPropArray to DrvdProp and DrvdProp2dArray to DrvdPropArray
* Bump ORCA version to 3.73.0
Authored-by: NChris Hajas <chajas@pivotal.io>
(cherry picked from commit 043365cc)

* docs - Add note for gpfdist/gpload wrong line number in error messages

This will be backporrted to 6X_STABLE, 5X_STABLE, and 4.3.x

* docs - fixed typo

- otherwise the cluster remains running if a test fails, which is
  problematic for the next test run.

gp_tablespace_with_faults test writes no-op record and waits for
mirror to replay the same before deleting the tablespace
directories. This step fails sometime in CI and causes flaky
behavior. The is due to existing code behavior in startup and
walreceiver process. If primary writes big (means spanning across
multiple pages) xlog record, flushes only partial xlog record due to
XLogBackgroundFlush() but restarts before commiting the transaction,
mirror only receives partial record and waits to get complete
record. Meanwhile after recover, no-op record gets written in place of
that big record, startup process on mirror continues to wait to
receive xlog beyond previously received point to proceed further.

Hence, as temperory workaround till the actual code problem is not
resolved and to avoid failures for this test, switch xlog before
emitting no-op xlog record, to have no-op record at far distance from
previously emitted xlog record.

(cherry picked from commit efd76c4c)

- introduces test helpers to remove knowledge from the suite file.

If the number of rows is always passed around together with the actual
rows, there's a missing abstraction.