Shared buffer access rules mandate that a pin as well as content lock
in exclusive mode is needed to update a shared buffer.

When RETURNING is specified, ExecDelete would return a virtual-tuple slot
that could contain pointers into an already-unpinned disk buffer.  Another
process could change the buffer contents before we get around to using the
data, resulting in garbage results or even a crash.  This seems of fairly
low probability, which may explain why there are no known field reports of
the problem, but it's definitely possible.  Fix by forcing the result slot
to be "materialized" before we release pin on the disk buffer.

Back-patch to 9.0; in earlier branches there is no bug because
ExecProcessReturning sent the tuple to the destination immediately.  Also,
this is already fixed in HEAD as part of the writable-foreign-tables patch
(where the fix is necessary for DELETE RETURNING to work at all with
postgres_fdw).

We seemed to be doing that in this case.  This was caught by enabling
memory_protect_buffer_pool GUC.

Prior to this commit, the minirepro and gpsd util, captured the hll
counter from the `pg_statistic` table as `int[]` instead of a `bytea[]`,
which caused errors while trying to load it. This commit fixes this
issue.

Fix invalid reference to relcache entries.

After a relation is closed, the relcache entry
might be freed if its refcount goes to zero and we
should avoid further reference to it.

This is a fix to already existing bug and current
tests can cover the code changes here. So there is
no need to add new test case.

fuzzystrmatch is now an extension and can be loaded through CREATE EXTENSION. This
suite was probably more relevant when GPDB didn't support extensions.
Authored-by: NChris Hajas <chajas@pivotal.io>

misc.feature contained "Miscellaneous tests which do not belong to mgmt
utilities" and are not being run/maintained.

gplog.feature was an attempt to test the logging feature in gppylib, but
likewise hasn't been run/maintained.
Authored-by: NChris Hajas <chajas@pivotal.io>

The maintained tests for gpfdist are located in src/bin/gpfdist/regress.
Authored-by: NChris Hajas <chajas@pivotal.io>

The maintained tests for gpload are located in gpMgmt/bin/gpload_test.
Authored-by: NChris Hajas <chajas@pivotal.io>

This has been annoying from long time, as always fails with planner. It was
fixed earlier in CI by setting the statement_mem but ideally it to have it in
tests to pass anywhere and not just in CI.

Fixes #4668.

It was disabled in the 8.4 merge
Signed-off-by: NJoao Pereira <jdealmeidapereira@pivotal.io>

the iterator result is required between calls
Signed-off-by: NJoao Pereira <jdealmeidapereira@pivotal.io>

* docs - gpcopy new options --dry-run, --no-distribution-check

--Add limitation and warning about copying some types of partitioned tables with --no-distribution-check.
--Also added limitation and warning to Admin Guide.

* docs - gpcopy fix typo for new option --no-distribution-check

* docs - gpcopy new options. Removed unneeded xref.

* docs - add GUC gp_resource_group_bypass

Link to HTML on GPDB review doc site.
http://docs-gpdb-review-staging.cfapps.io/review/ref_guide/config_params/guc-list.html#gp_resource_group_bypass

GUC list by category
http://docs-gpdb-review-staging.cfapps.io/review/ref_guide/config_params/guc_category-list.html#topic444

* docs - new GUC gp_resource_group_bypass

--Edits based on review comments.
--Add link to GUC from Admin Guide.
--Update TOC for this guc and other gucs not in TOC.

* docs - review updates for GUC gp_resource_group_bypass

* docs - fix typos in definition of GUC gp_resource_group_bypass

* docs - update to GUC gp_resource_group_bypass based on dev changes.

* docs - update gphdfs parquet support information.

--Update parequet support to 1.7.0 and later.
--Change location of parquet bundle jar files to
   https://mvnrepository.com/artifact/org.apache.parquet/parquet-hadoop-bundle

previous location was
  http://parquet.apache.org/downloads/

* docs - review updates of gphdfs parquet support information.

Pointers from Relation object needs to be handled with special care. As having
refcount on the object doesn't mean the object is not modified. Incase of cache
invalidation message handling Relation object gets *rebuild*. As part of rebuild
only guarantee maintained is that Relation object address will not change. But
the memory addresses inside the Relation object gets freed and freshly allocated
and populated with latest data from catalog.

For example below code sequence is dangerous

    rel->rd_cdbpolicy = original_policy;
    GpPolicyReplace(RelationGetRelid(rel), original_policy);

If relcache invalidation message is served after assigning value to
rd_cdbpolicy, the rebuild will free the memory for rd_cdbpolicy (which means
original_policy) and replaced with current contents of
gp_distribution_policy. So, when GpPolicyReplace() called with original_policy
is going to access freed memory. Plus, rd_cdbpolicy will have stale value in
cache and not intended refreshed value. This issue was hit in CI few times and
reproduces with higher frequency with `-DRELCACHE_FORCE_RELEASE`.

Hence this patch fixes all uses to rd_cdbpolicy to make use of rd_cdbpolicy
pointer directly from Relation object and also to update the catalog first
before assigning the value to rd_cdbpolicy.

To keep it consistent with the "Create table" syntax, CTAS should also
disallow duplicate distributed keys, otherwise backup and restore will
mess up.

In ALTER TABLE SET DISTRIBUTED BY, the "WITH(reorganize=...)" option
must be specified before the "DISTRIBUTED ..." clause.

Parse 1.8.2 is consistent with the other python libraries
defined in the python-dependencies.txt file
Co-authored-by: NTodd Sedano <tsedano@pivotal.io>
Co-authored-by: NChris Hajas <chajas@pivotal.io>

* docs - PL/Container - added note - domain object not supported.

* docs - PL/Container - updated note for non-support of domain object

Require the use of cc on OpenBSD as the gcc compiler isn't
supporting the compiler flags that setup.py invokes. Also fix
library paths.

OpenBSD require that linking with a library using sigwait(3) be
compiled with -pthread.  This adds a kludge in the relevant
makefiles which is less than elegant, but it seemed the least
intrusive change to make.

In order to use backtrace() in error reporting on OpenBSD we need
to link with libexecinfo from ports as backtrace() is a glibc only
addition.

There is no need to build gpmapreduce separately as it's automatically
built on "make install", and with recent changes to detect libyaml it's
not even supported. Fix by removing separate step and instead use the
--enable-mapreduce switch to autoconf.

Greenplum Mapreduce requires libyaml, but was lacking a specific
test for it in autoconf. This worked since gpfdist has the same
check but when building with --disable-gpfdist we need to ensure
we have libyaml to avoid late compilation failures.

Rather than hardcoding to require /bin/bash, move to using a lookup
via "/usr/bin/env bash" to allow for greater portability of the code.
This also changes the Bash test to checking if the current shell is
actually Bash, rather than looking if bash is available on the file
system (since we by the above mentioned changes no longer need that).

The NetCheckNIC() functionality was only used by FileRep, and is
not even compiling properly under all possible preprocessor flags
it once supported. Remove.

The -not syntax is not portable across all platforms, most notably
OpenBSD, so use the more portable ! operator instead.

Make sure to include all required header files to silence compilers
that are picky about that.

According to recent tests, this case now works fine, so there's no reason
to reject it anymore.  (Even if there are still some OpenBSD platforms
in the wild where it doesn't work, removing the check won't break any case
that worked before.)

We can actually remove the entire test that discovers whether libpython
is threaded, since without the OpenBSD case there's no need to know that
at all.

Per report from Davin Potts.  Back-patch to all active branches.
Co-authored-by: NDaniel Gustafsson <dgustafsson@pivotal.io>

Following the change in 8fcd3fdd
to cost-based enable GUCs, failing to find a way to construct an
N-way join should be an error rather than debug (as in upstream).
Reported-by: NHeikki Linnakangas <hlinnakangas@pivotal.io>

Discussion: https://github.com/greenplum-db/gpdb/pull/5155Co-authored-by: NDaniel Gustafsson <dgustafsson@pivotal.io>

Instead of completely disabling the generation of Paths with disabled
plan types, add a high penalty to their cost estimates, like in the
upstream. This reduces our diff vs. upstream, making future merges more
straightforward.

Discussion: https://groups.google.com/a/greenplum.org/d/msg/gpdb-dev/Az2cDcqf73g/_tY6Yv1kBgAJCo-authored-by: NDaniel Gustafsson <dgustafsson@pivotal.io>
Reviewed-by: NVenkatesh Raghavan <vraghavan@pivotal.io>
Reviewed-by: NRichard Guo <riguo@pivotal.io>

As part of the Postgres 8.3 merge, all heap tables now automatically
create an array type. The array type will usually be created with
typname '_<heap_name>' since the automatically created composite type
already takes the typname '<heap_name>' first. If typname
'_<heap_name>' is taken, the logic will continue to prepend
underscores until no collision (truncating the end if typname gets
past NAMEDATALEN of 64). This might be an oversight in upstream
Postgres since certain scenarios involving creating a large number of
heap tables with similar names could result in a lot of typname
collisions until no heap tables with similar names can be
created. This is very noticable in Greenplum heap partition tables
because Greenplum has logic to automatically name child partitions
with similar names instead of having the user name each child
partition.

To prevent typname collision failures when creating a heap partition
table with a large number of child partitions, we will now stop
automatically creating the array type for child partitions.

References:
https://www.postgresql.org/message-id/flat/20070302234016.GF3665%40fetter.org
https://github.com/postgres/postgres/commit/bc8036fc666a8f846b1d4b2f935af7edd90eb5aa

The pg_get_partition_template_def and pg_get_partition_def functions
take accesss share locks, but do not release them until the end of the
transaction. If a transaction is long-running, this can conflict with
other user operations. Is it not necessary to hold the lock indefinitely
as the lock is only needed for the duration of the function call.
Co-authored-by: NChris Hajas <chajas@pivotal.io>
Co-authored-by: NKaren Huddleston <khuddleston@pivotal.io>

If a segment exists in gp_segment_configuration but its ip address can
not be resolved we will run into a runtime error on gang creation:

    ERROR:  could not translate host name "segment-0a", port "40000" to
    address: Name or service not known (cdbutil.c:675)

This happens even if segment-0a is a mirror and is marked as down.  With
this error queries can not be executed, gpstart and gpstop will also
fail.

One way to trigger the issue:

- create a multiple segments cluster;
- remove sdw1's dns entry from /etc/hosts on mdw;
- kill postgres primary process on sdw1;

FTS can detect this error and automatically switch to mirror, but
queries can not be executed.