The commit enabled materialized view on both swap file and
concurrently mode. There are four main change on the code.
1. Add a new type 'PARENTSTMTTYPE_REFRESH_MATVIEW' on ParentStmtType in
Query struct. So that we can make the right query plan without gather
motion.
2. add a new 'transientrel_init' function which was called on
'initplan' to create temp table on segment.
3. Call the swap function on both master and segment.
4. Change the spi query to make the right diff on parallel system.
Co-authored-by: NZhenghua Lyu <kainwen@gmail.com>

Session order on segments is important in gdd tests, it determines the
cancellation order on deadlocks.  We used to create a full gang on the
BEGIN command, this was used in gdd tests to control the session orders
on segments.  However this behavior was changed during the optimization
of readonly queries, BEGIN command no longer creates a full gang.

So now we use the `RESET optimizer` command to trigger the gang
creation, we do not really care about the optimizer status although.
Reviewed-by: NZhenghua Lyu <zlv@pivotal.io>

Send btree indexes on AO tables as type btree

Previously, GPDB sent btree indexes on AO tables to ORCA as type bitmap
as btree indexes were not allowed on AO tables. This was a bit hacky, so
instead, send it over as type btree and let ORCA handle it properly.
Co-Authored-By: NAshuka Xue <axue@pivotal.io>
Co-Authored-By: NHans Zeller <hzeller@pivotal.io>

Bumps ORCA version to 3.75.0 and ICG test.

* docs - combine mgmt and client utilities into single section

* address some of the requested edits

* misc edits, conditionalizing

* minor edits requested by david

* correct the superscript referenced in the note

* use numbers for superscripts

* docs - provide more detail about pxf s3 select FILE_HEADER option

* address review comments from david and francisco

* add error message returned when both order/names differ

segspace test passes without need to set the GUC
gp_workfile_limit_per_segment. Just to set and reset this GUC setup
and cleanup test used to restart the cluster. Since it's not needed
delete these tests, which helps to save time. I don't have context on
why setup was needed earlier but definitely doesn't seem to be
required now.

Discussion:
https://groups.google.com/a/greenplum.org/d/msg/gpdb-dev/9lB2ovwKP3I/W6iQoN4GCAAJ

This guc makes ORCA use gpdb allocators. This allows for faster
optimization due to less overhead, reduced memory during optimization
due to smaller/fewer headers, and makes ORCA observe vmem limits instead
of crashing.

With setting optimizer_use_gpdb_allocators, we now track orca's memory
usage within resource groups/vmem. As a result, we need to update the test as
we go OOM slightly earlier when ORCA is enabled.
Authored-by: NChris Hajas <chajas@pivotal.io>

The old way was to construct a plan for a correlated subquery was to plan
the subquery as usual, except that Index Scans were not allowed. Then,
after constructing a Plan tree, the post-processing step apply_motion()
added Redistribute motion nodes so that the plan tree was executable on
any node. That was pretty simplistic; disabling Index Scans completely
obviously hurts performance. Also, because the planner didn't take into
account that all base relations are actually redistributed everywhere,
it was not reflected in the cost estimates, and the planner might choose
a sub-optimal plan.

To improve that, change the way that works, so that the Motions are added
earlier in the planning. The planner needs to know which restrictions
(WHERE clauses) refer to the outer query, i.e. which quals are correlated,
and make sure that those quals are always evaluated in the same slice as
the parent query. The outer query cannot pass parameters down through a
Motion node, so the subquery plan must not contain any Motion nodes
between the evaluation of the correlated variable, and the outer plan.
This is enforced by having a new CdbPathLocus type, "OuterQuery". A node
with OuterQuery locus must be evaluated in the outer query. It is mostly
the same as "general", which means that it can be evaluated anywhere, but
with the restriction that it is not OK to redistribute an input that has
OuterQuery locus. Whenever the planner node evaluates a correlated var,
that node must have Upper locus, by adding Motions below that node. This
has similar effect as the old approach, but gives the planner a bit more
flexibility, and the motions are taken into account in cost estimates.

This allows using Index Scans in subplans, but only if the Index Quals
don't contain correlated vars. This still isn't perfect, it would
sometimes be good to for example delay the evaluation of a correlated var
later, above a join node, because that might avoid expensive Redistribute
Motions. Even though this patch doesn't allow such plans yet, it's a step
in the right direction.

This moves the cdbllize() step to run *before* set_plan_references().
Now that we no longer add Motion nodes to an already-constructed plan
tree, we don't need the 'useExecutorVarFormat' stuff in many functions
anymore.

Fixes https://github.com/greenplum-db/gpdb/issues/8648Reviewed-by: NMelanie Plageman <mplageman@pivotal.io>
Reviewed-by: NSoumyadeep Chakraborty <sochakraborty@pivotal.io>

It was initially backported to greenplum before the src/common directory was
introduced. It is not needed anymore.

Addresses Github issue #8753
Reported-by: NYandong Yao <yyao@pivotal.io>
Reviewed-by: NHeikki Linnakangas <hlinnakangas@pivotal.io>

* Can use gpcopy to migrate data

* Conditionalize gpcopy references

* Edits for review comments

Other changes
--updated some queries and results to use state column from pg_stat_activity.
--update gp_toolkit.gp_workfile_* tables -based on replacing workfiles with buffiles
dev PR https://github.com/greenplum-db/gpdb/pull/6508

Updates in HTML format

Changed references to current_query to query, change some queries
http://docs-msk-gpdb6-dev.cfapps.io/6-0/admin_guide/perf_troubleshoot.html#topic5
http://docs-msk-gpdb6-dev.cfapps.io/6-0/admin_guide/workload_mgmt.html#topic27
http://docs-msk-gpdb6-dev.cfapps.io/6-0/admin_guide/workload_mgmt_resgroups.html
http://docs-msk-gpdb6-dev.cfapps.io/6-0/admin_guide/managing/startstop.html
http://docs-msk-gpdb6-dev.cfapps.io/6-0/ref_guide/config_params/guc-list.html#track_activity_query_size
http://docs-msk-gpdb6-dev.cfapps.io/6-0/ref_guide/sql_commands/DROP_RESOURCE_GROUP.html

Minor edit to column name
http://docs-msk-gpdb6-dev.cfapps.io/6-0/ref_guide/system_catalogs/pg_stat_activity.html

Reword change of column name from current_query to query.
http://docs-msk-gpdb6-dev.cfapps.io/6-0/install_guide/migrate.html

update gp_toolkit.gp_workfile_* tables -based on replacing workfiles with buffiles
http://docs-msk-gpdb6-dev.cfapps.io/6-0/ref_guide/gp_toolkit.html#topic32

Add schema to tablename
http://docs-msk-gpdb6-dev.cfapps.io/6-0/admin_guide/managing/monitor.html

Add new table_insert/update/delete() functions to stand in place of
heap_insert/update_delete(), and move the logic to dispatch the call to
AO, AOCO or external table function, depending on what kind of a table it
is. This moves the GPDB specific code out of the upstream functions,
making the diff vs upstream nicer to read.

PostgreSQL v12 introduced a new Table AM API. These new functions sit in
the same spot as the calls to the new API are made in v12. The signatures
don't quite match, but this is similar interface in spirit.
Reviewed-by: NGeorgios Kokolatos <gkokolatos@pivotal.io>

The reason to do this now is that the dumputils mock test was failing to
compile, on the 9.6 merge branch, because of the Assert that was added to
dumputils.c. I'm not sure why, but separating the GPDB functions to a
separate file seem like a good idea anyway, so let's just do that.

In master, we already got this fix as part of the 9.5 merge. Change it
slightly, so that we don't use the subplan's tuple descriptor as is, but
only copy the "tdhasoid" flag from it. It's in principle possible that
some unimportant information, like attribute names or typmods, are not
set up in the subplan's result type, and should be computed from the
Motion's target list. We're not seeing problems like that on master, but
on the 9.6 merge branch we are.

More importantly, this adds bespoken tests for this scenario. It arose
in the 'rowsecurity' test, but this doesn't have anything to do with
row-level security, so it was accidental.

We didn't backport the fix to 6X_STABLE before, so do that now.

Fixes https://github.com/greenplum-db/gpdb/issues/8765.
Reviewed-by: NGeorgios Kokolatos <gkokolatos@pivotal.io>

Authored-by: NBradford D. Boyle <bboyle@pivotal.io>

Server release candidate artifacts are not published until after an
extensive set of tests have passed in the CI pipeline. These tests
include ICW and all the CLI test suites. It is not unusual for time
between a commit being pushed to a release candidate being published to
be several hours, potentially slowing down the feedback cycle for
component teams.

This commit adds a "published" output to the compliation tasks. The new
build artifact is stored in an immutable GCS bucket with the version in
the filename. This makes it trivial for other pipelines to safely
consume the latest build.

This artifact **has not** passed any sort of testing (e.g., ICW) and
should only be used in development pipelines that need
near-instantaneous feedback on commits going into GPDB.

For the server-build artifact, `((rc-build-type-gcs))` will resolve to
`.debug` for the with asserts pipeline and `''` (i.e., empty string) for the
without asserts pipeline. The two types of server artifacts that are
"published" are:

1. server-build
2. server-rc

server-build is the output of the compilation task and has had no
testing; server-rc is a release candidate of the server component.
Authored-by: NBradford D. Boyle <bboyle@pivotal.io>

Previously, fsyc, heap_checksum and walrep tests used separate
databases. But seems with merge when src/makefiles/pgxs.mk added
--dbname to REGRESS_OPTS, all of these tests started using
`contrib_regression` database even if each of these tests defined
--dbname=<>, it was overridden from src/makefiles/pgxs.mk.

Hence, set USE_MODULE_DB=1 which makes these tests to use
--dbname=$(CONTRIB_TESTDB_MODULE) instead of
`contrib_regression`. This way these will again start having separate
database.

Command in next session gets executed in parallel to current session
in case "&" is used. Hence, if session 1 command is running very slow,
but session 0 commands gets executed fast, can cause flaky test. To
avoid the same, add retries to check for all processes for session are
blocked or not.

* docs - add guc optimizer_use_gpdb_allocators

* docs - add information about improved GPDB query mem. mgmt.

This reverts commit b81d4501.

The resource group tests use ORCA and need to be modified for this
change.

This guc makes ORCA use gpdb allocators. This allows for faster
optimization due to less overhead, reduced memory during optimization
due to smaller/fewer headers, and makes ORCA observe vmem limits instead
of crashing.
Authored-by: NChris Hajas <chajas@pivotal.io>

I found the logic to decide the target locus hard to understand, so I
rewrote it in a table-driven approach. I hope it's not just me.

Fixes github issue https://github.com/greenplum-db/gpdb/issues/8711Reviewed-by: NZhenghua Lyu <zlv@pivotal.io>

We introduce a new type of memory pool and memory pool manager:
CMemoryPoolPalloc and CMemoryPoolPallocManager

The motivation for this PR is to improve memory allocation/deallocation
performance when using GPDB allocators. Additionally, we would like to
use the GPDB memory allocators by default (change the default for
optimizer_use_gpdb_allocators to on), to prevent ORCA from crashing when
we run out of memory (OOM). However, with the current way of doing
things, doing so would add around 10 % performance overhead to ORCA.

CMemoryPoolPallocManager overrides the default CMemoryPoolManager in
ORCA, and instead creates a CMemoryPoolPalloc memory pool instead of a
CMemoryPoolTracker. In CMemoryPoolPalloc, we now call MemoryContextAlloc
and pfree instead of gp_malloc and gp_free, and we don’t do any memory
accounting.

So where does the performance improvement come from? Previously, we
would (essentially) pass in gp_malloc and gp_free to an underlying
allocation structure (which has been removed on the ORCA side). However,
we would add additional headers and overhead to maintain a list of all
of these allocations. When tearing down the memory pool, we would
iterate through the list of allocations and explicitly free each one. So
we would end up doing overhead on the ORCA side, AND the GPDB side.
However, the overhead on both sides was quite expensive!

If you want to compare against the previous implementation, see the
Allocate and Teardown functions in CMemoryPoolTracker.

With this PR, we improve optimization time by ~15% on average and up to
30-40% on some queries which are memory intensive.

This PR does remove memory accounting in ORCA. This was only enabled
when the optimizer_use_gpdb_allocators GUC was set. By setting
`optimizer_use_gpdb_allocators`, we still capture the memory used when
optimizing a query in ORCA, without the overhead of the memory
accounting framework.

Additionally, Add a top level ORCA context where new contexts are created

The OptimizerMemoryContext is initialized in InitPostgres(). For each
memory pool in ORCA, a new memory context is created in
OptimizerMemoryContext.

Bumps ORCA version to 3.74.0

This is a re-commit of 339dedf0d2, which didn't properly catch/rethrow
exceptions in gpos_init.
Co-authored-by: NShreedhar Hardikar <shardikar@pivotal.io>
Co-authored-by: NChris Hajas <chajas@pivotal.io>

Spotted when analyzing a CI failure pertaining to another test.

Reviewed by Heikki and Georgios.

The privious pattern used by the test was not strong enough.  It could
accidentally matched names of partitioned tables.  Name of a partition
being added is generated by suffixing a random number, if no name is
specified by the user.  E.g. "sales_1_prt_r1171829080_2_prt_usa".  At
least one time, the test failed CI due to this weakness.

The new pattern is strong enough to match only the auxiliary table
names that end with "_<oid>".

Reviewed by Heikki and Georgios.

* docs - mention resource groups in spill file topic, misc edits

* edits requested by david

This test is about MergeAppends, there's even a comment saying "we want a
plan with two MergeAppends". enable_mergejoin defaults to off in GPDB, so
we have to enable it to get the same plan as in upstream.

* docs - move gpmapreduce yaml info to own utility page; misc topic edits

* relocate topic and graphic, add shortdesc, fix linking

Reviewed by: David Krieger and Soumyadeep Chakraborty

This reverts commit 0b00feb6.

This is breaking a memory accounting test.

and follow the convention of explicitly using these keywords.
Any failed tests, including not reported, will add the 'FAILED' keyword in the
summary.
Reviewed-by: NAdam Berlin <aberlin@pivotal.io>

We introduce a new type of memory pool and memory pool manager:
CMemoryPoolPalloc and CMemoryPoolPallocManager

The motivation for this PR is to improve memory allocation/deallocation
performance when using GPDB allocators. Additionally, we would like to
use the GPDB memory allocators by default (change the default for
optimizer_use_gpdb_allocators to on), to prevent ORCA from crashing when
we run out of memory (OOM). However, with the current way of doing
things, doing so would add around 10 % performance overhead to ORCA.

CMemoryPoolPallocManager overrides the default CMemoryPoolManager in
ORCA, and instead creates a CMemoryPoolPalloc memory pool instead of a
CMemoryPoolTracker. In CMemoryPoolPalloc, we now call MemoryContextAlloc
and pfree instead of gp_malloc and gp_free, and we don’t do any memory
accounting.

So where does the performance improvement come from? Previously, we
would (essentially) pass in gp_malloc and gp_free to an underlying
allocation structure (which has been removed on the ORCA side). However,
we would add additional headers and overhead to maintain a list of all
of these allocations. When tearing down the memory pool, we would
iterate through the list of allocations and explicitly free each one. So
we would end up doing overhead on the ORCA side, AND the GPDB side.
However, the overhead on both sides was quite expensive!

If you want to compare against the previous implementation, see the
Allocate and Teardown functions in CMemoryPoolTracker.

With this PR, we improve optimization time by ~15% on average and up to
30-40% on some queries which are memory intensive.

This PR does remove memory accounting in ORCA. This was only enabled
when the optimizer_use_gpdb_allocators GUC was set. By setting
`optimizer_use_gpdb_allocators`, we still capture the memory used when
optimizing a query in ORCA, without the overhead of the memory
accounting framework.

Additionally, Add a top level ORCA context where new contexts are created

The OptimizerMemoryContext is initialized in InitPostgres(). For each
memory pool in ORCA, a new memory context is created in
OptimizerMemoryContext.

Bumps ORCA version to 3.74.0
Co-authored-by: NShreedhar Hardikar <shardikar@pivotal.io>
Co-authored-by: NChris Hajas <chajas@pivotal.io>

Earlier, COPY <Catalogtable> from <file> was allowed irrespective of the
value of allow_system_table_mods. This commit restricts such statements
only when allow_system_table_mods is set to ON.

Co-Authored-By: Ashwin Agrawal<aagrawal@pivotal.io>

Due to a option index error, exclude-tests, ignore-plans, prehook and
print-failure-diffs are not parsed from command line correctly.