An elusive missing header file was detected when the Ubuntu job failed
in CI. Centos and Mac did not show any indication of a missing header
file and no side effects were detected. However, Ubuntu was returning an
int32 in makeDefElem instead of a 64-bit pointer.
Co-authored-by: NAshwin Agrawal <aagrawal@pivotal.io>
Co-authored-by: NJesse Zhang <jzhang@pivotal.io>
Co-authored-by: NFrancisco Guerrero <aguerrero@pivotal.io>

* Docs: Using 'Postgres Planner' terminology consistently

* Postgres planner -> Postgres Planner

* remove another reference to 'legacy' planner

* Few more fixes

* Remove syntax options for creating OIDS; add pivotal-only statements of non-support

* Some edits from Lisa

* Clarify error with oids on partitioned/co table.

* Feedback from Mel

* Add deprecation/removal wording suggested by Rob

* Add centos7 command to start config.service

* Add some text around starting immediately

postgresql_conf_gp_addition is just an example for what can be injected
into postgresql.conf via gpinitsystem, but the values in there make
little sense to ship for production.

It has not been changed in many years and is mistakenly copied into the
files we distributed with Greenplum (via gpMgmt/Makefile target that
copies bin/ files).
Authored-by: NAmil Khanzada <akhanzada@pivotal.io>

There is different build system code that defines which contrib
extensions are built and installed.
gpAux/Makefile should not be building any contrib extensions.
Authored-by: NAmil Khanzada <akhanzada@pivotal.io>

Currently, PXF is not propagating the format options in the external
table framework. In Foreign Data Wrappers, these options are defined at
the foreign table-level and are being propagated to PXF.

In order for the PXF Server to better support both FDW and external
tables we are consistently passing format information from both clients.

The PostgreSQL 9.2 merge introduced a couple of new cursor options,
including CURSOR_OPT_GENERIC_PLAN, which clashed with the 0x0040 value that
we were already using in GPDB for the GPDB-specific CURSOR_OPT_UPDATABLE
option. Fix the conflict.

The CURSOR_OPT_UPDATABLE flag was used in cursors opened in PL/pgSQL, so
the mixup caused generic plans to be used for all PL/pgSQL cursors. Add
a test case for that.

The PostgreSQL 9.2 merge happened in GPDB 6, so backport to GPDB 6.
Reviewed-by: NAsim R P <apraveen@pivotal.io>
Reviewed-by: NPengzhou Tang <ptang@pivotal.io>

Git stores the 'x' flag for files in the repository. For no good reason,
some files had it set differently from upstream. It's harmless, but let's
avoid spurious differences.

These were removed from upstream years ago already, but were left over in
GPDB for some reason. GPDB never supported these.

Happened to spot two variations from pairing stations.

Ooops.

We only injected the skip fault for fts once, causing a race condition
in the test. Here's the problematic timeline:

1. Inject skip fault for fts
2. Fts runs, skip fault is gone
3. Panic primary
4. Inject Noop record into mirror
5. Promote mirror
6. Noop record never replayed and test fails.

I don't know what purpose this might have served originally, but it's
unused now.
Reviewed-by: NAsim R P <apraveen@pivotal.io>

This guc allows the optimizer to estimate the cardinality in case of DPE much
more accurately. On the TPC-DS test suite, it improves the execution time for
query 13 from 114s to 28s and for query 48 from 84s to 34s.

This is initial version of README to document FTS details. Lot more
details need to be captured to completely cover FTS working, but this
acts as good starting point.

I (Ashwin) performed minor touch-ups to the README.
Reviewed-by: NAsim R P <apraveen@pivotal.io>
Reviewed-by: NPengzhou Tang <ptang@pivotal.io>

For partitioned tables, in case of DELETE/UPDATE operation on the root
table, we must acquire ExclusiveLock on the root and leaf partitions, so
that any other operation requesting lock higher than AccessShareLock and
ShareLock must wait on QD.

If we don't acquire lock for the leaf partitions, other concurrent DML
operation may be dispatched to the segments and a deadlock can occur.
Refer to the test case added for GDD.

resultRelations may not have all the entries including the root and leaf
partition tables, so in this fix we explicitly acquire locks on all the
tables.

While GDD is on, it will be able to detect the deadlock and cancel one
of the query, so we don't acquire locks on the leaf parts if the DML
operation is on the root.
Co-authored-by: NAlexandra Wang <lewang@pivotal.io>
Signed-off-by: NAlexandra Wang <lewang@pivotal.io>
Co-authored-by: NAlexandra Wang <lewang@pivotal.io>

gpnetbench is an always installed binary which is shipped for every
production release of Greenplum, so it may as well live in src/bin
which is intended for just those binaries. The gp* namespace ensure
that it's presence won't collide with upstream additions, and since
we want to get rid of gpAux/ we must start moving these tools from
there anyways.

No functional changes are introduced, apart from the Makefile now
supporting "make uninstall" as well.
Reviewed-by: NJimmy Yih <jyih@pivotal.io>

The whole target is quite dubious and should be removed in favor of
just using 'make -C gpMgmt install', but until I know how this file
is at all used then let's at least clean out the worst offenders.
Reviewed-by: NBradford D. Boyle <bboyle@pivotal.io>

The greenplum_path target was referencing a program which no longer
exists and is thus dead code, and the devel_failtinj is no longer
defined at all.
Reviewed-by: NBradford D. Boyle <bboyle@pivotal.io>

The gpAux Makefile was checking for GCC version to ensure that GPDB
is compiled with the minimum version required. This is a layering
violation since we should be doing these checks in autoconf. Should
the compiler be let through autoconf then we are good to go, further
checking should not (need to) be performed.

The version checked for is also quite ancient by now, the odds of
it being the compiler on a system which can compile the rest of the
codebase is slim to none.
Reviewed-by: NBradford D. Boyle <bboyle@pivotal.io>

It was made unused by commit 8eed4217Co-authored-by: NPengzhou Tang <ptang@pivotal.io>

For commands called directly by the user, we provide the fix.

Since Behave and unit tests are supposed to behave as a normal user,
we do not provide the fix.  The fix is supposed to be done by the
commands themselves, and we want to test with an unmodified search_path
in the actual tests.
Co-authored-by: NJamie McAtamney <jmcatamney@pivotal.io>
Co-authored-by: NKalen Krempely <kkrempely@pivotal.io>
Co-authored-by: NNikolaos Kalampalikis <nkalampalikis@pivotal.io>
Co-authored-by: NShoaib Lari <slari@pivotal.io>
Co-authored-by: NDavid Krieger <dkrieger@pivotal.io>

This commit corresponds with ORCA commit "Implement Full Merge Join in ORCA".
It also bumps ORCA version to v3.59.0.

This commit includes the following changes to support merge join in
ORCA:
1. Update optimizer_expand_fulljoin guc to use traceflags instead of
disabling the transform.
2. Translator changes for Merge Join.
3. Add IsOpMergeJoinable() and GetMergeJoinOpFamilies() wrappers.
4. Introduces the guc optimizer_enable_mergejoin.

This allows a user to specify regress options to consistently make
isolation2 tests for pg_basebackup_with_tablespaces pass.

Currently, the tests fail if the user's source directory creates
a tablespace location directory path that is longer than the 100
character limit for pg_basebackup to add a tablespace location
directory to the backup.

This commit refactor the function `cdbpath_motion_for_join` to make
it clear and generate better plan for some cases.

In distributed computing system, to gather distributed data into a
singleQE should always be the last choice. Previous code for general
and segmentgeneral, when they are not ok_to_replicate, will try to
gather other locus to singleQD. This commit improves this by firstly
trying to add redistributed motion.

The logic for the join result's locus (outer's locus is general):
  1. if outer is ok to replicated, then result's locus is the same
     as inner's locus
  2. if outer is not ok to replicated (like left join or wts cases)
     2.1 if inner's locus is hashed or hashOJ, we try to redistribute
         outer as the inner, if fails, make inner singleQE
     2.2 if inner's locus is strewn, we try to redistribute
         outer and inner, if fails, make inner singleQE
     2.3 just return the inner's locus, no motion is needed

The logic for the join results' locus (outer's locus is segmentgenral):
- if both are SegmentGeneral:
     1. if both locus are equal, no motion needed, simply return
     2. For update cases. If resultrelation
        is SegmentGeneral, the update must execute
        on each segment of the resultrelation, if resultrelation's
        numsegments is larger, the only solution is to broadcast
        other
     3. no motion is needed, change both numsegments to common
       - if only one of them is SegmentGeneral:
         1. consider update case, if resultrelation is SegmentGeneral,
            the only solution is to broadcast the other
         2. if other's locus is singleQE or entry, make SegmentGeneral
            to other's locus
         3. the remaining possibility of other's locus is partitioned
            3.1 if SegmentGeneral is not ok_to_replicate, try to
                add redistribute motion, if fails gather each to
                singleQE
            3.2 if SegmentGeneral's numsegments is larger, just return
                other's locus
            3.3 try to add redistribute motion, if fails, gather each
                to singleQE

Locus type Replicated can only be generated by join operation.
And in the function cdbpathlocus_join there is a rule:
    `<any locus type> join <Replicated> => any locus type`

Proof by contradiction, it shows that when code arrives here,
it is impossible that any of the two input paths' locus
is Replicated. So we add two asserts here.

It was disabled by accident several months ago while implementing
`COPY (query) TO ON SEGMENT`, re-enable it.

```
commit bad6cebc
Author: Jinbao Chen <jinchen@pivotal.io>
Date:   Tue Nov 13 12:37:13 2018 +0800

    Support 'copy (select statement) to file on segment' (#6077)
```

WARNING: there are no safety protections on utility mode, it's not
recommended except disaster recovery situation.
Co-authored-by: NWeinan WANG <wewang@pivotal.io>

The recent sysctl changes(42930ed1) modified the ccp nodes.
Somehow, this causes memory issues on our ccp nodes for
Behave.  There was a recent, similar modification for
gpexpand(6f494638).

Resource group believe memory access speed always faster than disk,
and it adds hashagg executor node spill mechanism into its memory management.
If the hash table size overwhelms `max_mem`, in resource group model, the hash
table does not spill and fan out data. Resource group wants to grant more memory
for the hash table. However, this strategy impact hash collision rate, so that
some performance regression in some OLAP query.

We rid resource group guc when hashagg evaluate if it needs to spill.
Co-authored-by: NAdam Li <ali@pivotal.io>

MemoryAccounting_RequestQuotaIncrease() returns a number in bytes, but
here expects kB.

When building file `_pg.so` on MacOS platform, distutil will invoke
clang compiler with arguments `-arch x86_64 -arch i386`. But type
`int128` is not available for i386 architecture, thus following error occurs:

```
In file included from pgmodule.c:32:
In file included from include/postgres.h:47:
include/c.h:427:9: error: __int128 is not supported on this target
typedef PG_INT128_TYPE int128
        ^
include/pg_config.h:838:24: note: expanded from macro 'PG_INT128_TYPE'
                       ^
In file included from pgmodule.c:32:
In file included from include/postgres.h:47:
include/c.h:433:18: error: __int128 is not supported on this target
```

By adding `['-arch', 'x86_64']` into `extra_compile_args`, distutil will
remove `-arch i386` from compiler arguments, thus fixes compile error.

scan->raw_buf_done was used for custom external table only, refactor
to remove the MERGE_FIXME.

cstate->raw_buf_len is safe to use since we operate pstate->raw_buf
directly in this case.

About the `isjoininner`, I searched the history commit in merge branch,
it was removed by "e2fa76d8 - Use parameterized paths to generate
inner indexscans more flexibly" on upstream from 9.2, that MERGE_FIXME
was there because at that time functions which rely on `isjoininner`
refused to compile.