The cause of the PANIC was an incorrectly populated list containing the
namespace information for the affected the relation. A GrantStmt contains the
necessary objects in a list named objects. This gets initially populated during
parsing (via the privilege_target rule) and processed during parse analysis based
on the target type and object type to RangeVar nodes, FuncWithArgs nodes or
plain names.

In Greenplum, the catalog information about the partition hierarchies is not
propagated to all segments. This information needs to be processed in the
dispatcher and to be added backed in the parsed statement for the segments to
consume.

In this commit, the partition hierarchy information is expanded only for the
target and object type required. The parsed statement gets updated
uncoditionally of partitioned references before dispatching for required types.

The privileges tests have been updated to get check for privileges in the
segments also.

Problem identified and initial patch by Fenggang <ginobiliwang@gmail.com>,
reviewed and refactored by me.

(cherry picked from commit 7ba2af39)

Current code for COPY FROM picks mode as COPY_DISPATCH for
non-distributed/non-replicated table as well. This causes crash. It
should be using COPY_DIRECT, which is normal/direct mode to be used
for such tables.

The crash was exposed by following SQL commands:

    CREATE TABLE public.heap01 (a int, b int) distributed by (a);
    INSERT INTO public.heap01 VALUES (generate_series(0,99), generate_series(0,98));
    ANALYZE public.heap01;

    COPY (select * from pg_statistic where starelid = 'public.heap01'::regclass) TO '/tmp/heap01.stat';
    DELETE FROM pg_statistic where starelid = 'public.heap01'::regclass;
    COPY pg_statistic from '/tmp/heap01.stat';

Important note: Yes, it's known and strongly recommended to not touch
the `pg_statistics` or any other catalog table this way. But it's no
good to panic either. The copy to `pg_statictics` is going to ERROR
out "correctly" and not crash after this change with `cannot accept a
value of type anyarray`, as there just isn't any way at the SQL level
to insert data into pg_statistic's anyarray columns. Refer:
https://www.postgresql.org/message-id/12138.1277130186%40sss.pgh.pa.us

(cherry picked from commit 6793882b)

* docs - move install guide to gpdb repo

--move Install Guide source files back to gpdb repo.
--update config.yml and gpdb-landing-subnav.erb files for OSS doc builds.
--removed refs directory - unused utility reference pages.
--Also added more info to creating a gpadmin user.

These files have conditionalized text (pivotal and oss-only).

./supported-platforms.xml
./install_gpdb.xml
./apx_mgmt_utils.xml
./install_guide.ditamap
./preinstall_concepts.xml
./migrate.xml
./install_modules.xml
./prep_os.xml
./upgrading.xml

* docs - updated supported platforms with PXF information.

* docs - install guide review comment update

-- renamed one file from supported-platforms.xml to platform-requirements.xml

* docs - reworded requirement/warning based on review comments.

* add ad steps in pxf krb doc

* From Lisa Owen

* distributing keytab using gpscp and gpssh

* Update gpdb-doc/markdown/pxf/pxf_kerbhdfs.html.md.erb
Co-Authored-By: NAlexander Denissov <denalex@users.noreply.github.com>

* Update gpdb-doc/markdown/pxf/pxf_kerbhdfs.html.md.erb
Co-Authored-By: NAlexander Denissov <denalex@users.noreply.github.com>

* misc formatting edits

* a few more formatting edits

Tab completion does not work for centos 7 as reported in #8575
[https://github.com/greenplum-db/gpdb/issues/8575] and installing from the rpms.
The root cause is that the binary is linked against libedit (v 0x402) which is
providing the necessary `rl_line_buffer` yet with different contents from
readline. In libedit, the variable contains the last flushed entry in the
history file whereas in readline it contains the current line in the interactive
terminal.

It is not necessary to link with libedit on the build packages so the flag is
removed.
Reviewed-by: NBradford D. Boyle <bboyle@pivotal.io>
(cherry picked from commit 3827c3a6)

* edits to AGGREGATE variables

* removing unnessesary words

* removed STATEFUNC from example

This issue was causing the build pipeline to go red. Reverting for now.

This reverts commit 2154dfae.

It has been discovered in GPDB v.6 and above that a 'GRAND ALL ON ALL TABLES IN
SCHEMA XXX TO YYY;' statement will lead to PANIC.

From the resulted coredumps, a now obsolete code in QD that tried to encode
objects in a partition reference into RangeVars was identified as the culprit.
The list that the resulting vars were ancored, was expecting and treating only
StrVars. The original code was added following the premise that catalog
informations were not available in Segments. Also it tried to optimise caching,
yet the code was not fully writen.

Instead, the offending block is removed which solves the issue and allows for
greater alignment with upstream.
Reviewed-by: NGeorgios Kokolatos <gkokolatos@pivotal.io>
(cherry picked from commit ba6148c6)

In commit "Check parallel plans correctly" we introduced a regression
which was only triggered by the out-tree diskquota tests, now we add a
simplified version of the tests to ICW to prevent future regressions.

The issue was fixed by "Replace planIsParallel by checking
Plan->dispatch flag".

Commit 7d74aa55 introduced a new function, planIsParallel() to check
whether the main plan tree needs the interconnect, by checking whether
it contains any Motion nodes. However, we already determine that, in
cdbparallelize(), by setting the Plan->dispatch flag. We were just not
checking it when deciding whether the interconnect needs to be set up.
Let's just check the 'dispatch' flag, like we did earlier in the
function, instead of introducing another way of determining whether
dispatching is needed.

I'm about to get rid of the Plan->nMotionNodes field soon, which is why
I don't want any new code to rely on it.

(cherry picked from commit c1851b62)

* Ship modified python module subprocess32 again

subprocess32 is preferred over subprocess according to python documentation.
In addition we long ago modified the code to use vfork() against fork() to
avoid some "Cannot allocate memory" kind of error (false alarm though - memory
is actually sufficient) on gpdb product environment that is usually with memory
overcommit disabled.  And we compiled and shipped it also but later it was just
compiled but not shipped somehow due to makefile change (maybe a regression).
Let's ship it again.

* Replace subprocess with our own subprocess32 in python code.

Cherry-picked 9c4a885b and
              da724e8d and
              a8090c13 and
              4354f28c

Authored-by: NTingfang Bao <bbao@pivotal.io>

Authored-by: NTingfang Bao <bbao@pivotal.io>

In order to maintain the gpdb build process better.
  gp-releng re-organize the build artifacts storage.

  Only the artifacts path changed, the content is still
  the same as before.
Authored-by: NTingfang Bao <bbao@pivotal.io>

When gp_use_legacy_hashops GUC was set, CTAS would not assign the
legacy hash class operator to the new table. This is because CTAS goes
through a different code path and uses the first operator class of the
SELECT's result when no distribution key is provided.

Backported from GPDB master 9040f296. There was one conflict:
the cdbhash_int4_ops operator class oid is different between 6X and
master (10196 in master vs. 10166 in 6X).

Remove unused var acquireResult in function AOCSDrop to
get rid of compiling warning.

The toolsmiths DockerHub repository is deprecated and replaced by pivotaldata.
Co-authored-by: NAmil Khanzada <akhanzada@pivotal.io>
Co-authored-by: NJose Munoz <jmunoz@pivotal.io>
(cherry picked from commit a69ed682)

- make modifications to the test scripts to prepare for use on
  Ubuntu. Previously they had only been used on Macosx.

- auto-formatted C and bash files in pg_upgrade tests.

- fixed bug in test script where we had been running the upgrade in
  dispatcher mode when attempting to upgrade the segment primaries.

Building stream.c has exposed various
> ignoring #pragma omp parallel [-Wunknown-pragmas]
when not linked with openmp.

Be explicit and ask openmp for parallelization only when the binary has been
linked with the library.

strncat unfortunately takes a misleading size argument which means
at most size from src. It is a bit of an antipattern in the string
family of functions and for that compilers will emit a warning if
it happens that the size argument matches the size of src, since that
is not what usually users of strncat want to do.

The usage of the function in the code was correct. However instead
of silencing the compiler, strncat was replaced with a front end
dynamic buffer and family of functions that postgres is providing.

Also protect against an invalid read in case that the size of the
result is zero.

- The corresponding ORCA PR is : https://github.com/greenplum-db/gporca/pull/533

- Change GUC value OPTIMIZER_UNEXPECTED_FAIL so that we log only unexpected failures.
Co-authored-by: NAbhijit Subramanya <asubramanya@pivotal.io>
Co-authored-by: NSambitesh Dash <sdash@pivotal.io>

- Fix "unelaborated friend declaration" warnings
- Fix "missing prototype" warnings
- Fix "generalized initializer lists are a C++ extension" warning

CTranslatorQueryToDXL.h:63:10: warning: unelaborated friend declaration is a C++11 extension; specify 'class' to befriend 'gpdxl::CTranslatorScalarToDXL' [-Wc++11-extensions]
                friend CTranslatorScalarToDXL;
                       ^
                       class

funcs.cpp:43:1: warning: no previous prototype for function 'DisableXform' [-Wmissing-prototypes]
DisableXform(PG_FUNCTION_ARGS)
^
funcs.cpp:76:1: warning: no previous prototype for function 'EnableXform' [-Wmissing-prototypes]
EnableXform(PG_FUNCTION_ARGS)
^
funcs.cpp:109:1: warning: no previous prototype for function 'LibraryVersion' [-Wmissing-prototypes]
LibraryVersion()
^
funcs.cpp:123:1: warning: no previous prototype for function 'OptVersion' [-Wmissing-prototypes]
OptVersion()
^
4 warnings generated.

CTranslatorDXLToScalar.cpp:730:9: warning: generalized initializer lists are a C++11 extension [-Wc++11-extensions]
        return { .oid_type = inner_type_oid, .type_modifier = type_modifier};

These tests use fault injection for testing, each of them must be put in
a separate testing group otherwise they will be flaky.

(cherry picked from commit 558cc68f)

Some tests use fault injection for testing, they could be flaky if they
are put into parallel testing groups as the injected faults can be
triggered by the tests in the same testing groups.  To make the testing
results deterministic each of these tests must be put in a separate
group.

Such kind of flaky tests are very confusing as a test fails randomly due
to fault injected in other tests, so it can be hard to debug.

So we add a checking at beginning of ICW and isolation2 tests, it raises
and error when fault injectors are put in parallel testing groups.

(cherry picked from commit d9c0d3f5)

For GPDB, to use the general plan tree walker/mutator, structure
plan_tree_base_prefix needs to be prefixed with in the context structure
and initialized appropriately. It is needed by function plan_tree_walker
to recur into the subplan in case of visiting a SubPlan node.

However, context structure SubqueryScanWalkerContext fails to do that.
So when we are trying to recurse through a plan tree to find a subquery
scan node, if there is a subplan there, we will crash.

This patch fixes that. Also it fixes github issue #8342 as well as the
panic in github issue #7279.
Reviewed-by: NMelanie Plageman <mplageman@pivotal.io>

When a mirror is moved from one host to another, the pg_hba.conf of the
corresponding primary should be updated with the entry for the new
mirror host to allow connections from mirror to primary segment
database. This is required by pg_rewind during gprecoverseg operations.

Also, an additional minor change is the condition for removing the old
directories, i think it was a typo error before, as the condition should have
checked for numOfWorkers>0 instead of >1, because if there is one host
only, we will not be removing the old directories otherwise.

* docs - add information about timestamp format YYYYMMDDHH24MISS

GPDB does not automatically convert string to timestamp

Will be backported to 6X_STABLE

* docs - fixed formatting in XML source

removed unneeded section element.

* docs - add gpfdist support for writing gzip compressed files.

Also added links to the gpfdist utility in other topics

Will be backported to 5X_STABLE

* docs - Update based on review comments.
--removed path requirement for compression.
--clarified note compression is not supported when gpfidist runs on Windows.

* Typo fix:  gpfidist -> gpfdist

* Typo fix:  gpfidist -> gpfdist

Append into pg_hba configs of expanded segments the addresses of the
primary and mirror segments. The hba config is copied from segment 0.
We must update it with the new primary/mirror pair information otherwise
connections between the systems will fail. This connection is necessary
during pg_rewind.
Co-authored-by: NBhuvnesh Chaudhary <bhuvnesh2703@gmail.com>
Co-authored-by: NJesse Zhang <sbjesse@gmail.com>