Greenplum diverges from upstream PostgreSQL in abort transaction workflow.
Greenplum must dispatch ABORT command to QEs.  Consequently, transaction ID in
top transaction state is reset before broadcasting the ABORT.

Exported snapshots are closely tied to the transaction that exported them.
They must be cleared before the transaction that exported them ceases to exist.
In order to abide by this sequence of operations, Greenplum must also clear
exported snapshots before marking the transaction as invalid.  This patch moves
the call to clear exported snapshots at a location that is right for Greenplum.

Fixes Github issue #8020

Reviewed-by: Georgios Kokolatos and Jimmy Yih

FDW or External table's INSERT actions are actually just some defined
callbacks but not real heap table INSERT actions.

Don't mark transactions doing write just because they insert into
foreign or external tables, some FDW extensions report an error because
they don't see XACT_EVENT_PRE_PREPARE coming.

```
SET "request.header.user-agent" = 'curl/7.29.0';
```

The double quote characters are needed in the above command, but
Greenplum lost them while dispatching, which reports syntax error.

We need to bump the catalog because of the change to the XLOG
introduced by the create tablespace pr.

(122c79f2)

The checkinc.py script was performing a check for the presence of
header files we're not shipping.  This was essentially guaranteed
to always pass as the code was able to compile, and has generally
provided little value.  Recent changes in macOS installations has
also made it fail with false positives, so finally remove it to
minimize developer time spent fighting quite useless tests.

Any checks for headers should be performed in autoconf before the
code is compiled, not during testing.

Discussion: https://github.com/greenplum-db/gpdb/issues/7961
https://groups.google.com/a/greenplum.org/d/msg/gpdb-dev/cCXsHwPr7GM/XbExPStmAgAJ
Reviewed-by: Ashwin Agrawal, Jimmy Yih

To match how bin_gpdb tar.gz is created.
Co-authored-by: NMark Sliva <msliva@pivotal.io>
Co-authored-by: NAmil Khanzada <akhanzada@pivotal.io>

For commit e995ae86Co-authored-by: NMark Sliva <msliva@pivotal.io>
Co-authored-by: NAmil Khanzada <akhanzada@pivotal.io>

ICG changes for Create LAS Correlated Apply on in filter context

Prior to this change it would only run on centos7.
Also pipelines that did not target centos7 would
fail to run.
Co-authored-by: NMark Sliva <msliva@pivotal.io>
Co-authored-by: NAmil Khanzada <akhanzada@pivotal.io>

This reverts commit c46bf52f.

* Adding reference topic and additional information around temp_tablespaces

* Clarify comma-separated list

Fix gen_pipeline.py OS issue

Modify TwoPhaseFileHeader to include a tablespace OID that needs to be
removed if an abort occurs.

We add a 'pending tablespace for deletion' that is scheduled during the
'create tablespace' command. If a node encounters an error, it can abort
using the in-memory value. The master then can send an ABORT PREPARED for all
segments that have prepared the create tablespace.

A successful 'create tablespace' needs to unschedule the deletion, or following
aborted transactions will accidentally delete the newly created tablespace.

XLOG records are written for the ABORT and ABORT_PREPARED of a transaction, and these records also
now contain a tablespace oid for deletion.

Notes:

During CREATE TABLESPACE, if an error occurs after the XLOG_XACT_PREPARE
record has been written, there should be no filesystem changes left
on disk for any of the segments. (master, standby, primary, or mirror)

If an error occurs after the XLOG_XACT_DISTRIBUTED_COMMIT, then the create
tablespace should succeed, and all tablespace filesystem changes should exist.
Co-authored-by: NTaylor Vesely <tvesely@pivotal.io>
Co-authored-by: NAlexandra Wang <lewang@pivotal.io>
Co-authored-by: NSoumyadeep Chakraborty <sochakraborty@pivotal.io>

* Add remote_regress folder in src/bin/gpfdist/
to test gpfdist which runs on a remote host, such
as a windows server. make installcheck_win can
run them.

* The scripts of start_gpfdist_remote_win
are used to start and stop gpfdist runing on windows.
If you want to test gpfdist on another os, such
as aix, you can create a forlder named
start_gpfdist_remote_aix and add some scripts
to start and stop gpfdist.

With GDD enabled, and under some simple cases (refer
the commit 6ebce733 and the function checkCanOptSelectLockingClause
for details), we might also do some optimizations for the
select statement with locking clause and limit clause.

Greenplum generates two-stage-merge sort or limit plan to implement
sort or limit and we can only lock tuples on segments. We prefer
locking more tuples on segments rather than locking the whole table.
Without the whole table lock, performance for OLTP should be improved.

Also, after lockrows data cannot be assumed in order, but we do merge
gather after lockrows. It is reasonable because even for postgres:
`select * from t order by c for update` cannot guarantee result's order.

The task that had been using it was removed
in commit: b9636096

* docs - GPCC no longer depends on gpperfmon_install

* Review comments from David

Issue is that in Postgres 9.5 there is a fixed amount of xl_info flags
that can be used. GPDB commit b5871009 added another flag which in
GPDB 9.5 merge branch exceeds the amount available.

Ashwin noticed the difference between XLOG_XACT_ONE_PHASE_COMMIT and
XLOG_XACT_COMMIT records is if distributed info is set. Instead we can
use a same record type in both cases if we setup distributed info in
RecordTransactionCommit() and then pass the distributed info through
xact_redo().

Now that the root cause for a single failing test on Ubuntu18.04 has
been merged(dc0a3cec), enable that test in the pipeline.

The link to the upstream PostgreSQL documentation had a typo and
was pointing to postgreql.org.

Two empty test files introduced in commit 364cbce7. Must have been a
think-O

This reverts commit 559d9d9f.

The recommended sysctl settings for user clusters are out-of-date, and
after some investigation we've discovered a good minimal set of
recommended defaults.  These will be updated in the documentation, but
we also want to set them in the VMs used for our CLI Behave tests so
that we use values similar to those that users will have in their
environments, so this commit adds a section to the Concourse cluster
task file to do so.
Co-authored-by: NJamie McAtamney <jmcatamney@pivotal.io>
Co-authored-by: NKalen Krempely <kkrempely@pivotal.io>

A Motion node may has a subplan whose targetlist need to be expanded
to make sure all entries of the subplan's hashExpr are in its
targetlist, in this case, the motion node should also update its
targetlist, otherwise, the difference of targetlist will cause
Motion node parse the tuple (especially for MemTuple) incorrectly
with a mismatched tuple description, make wrong result or even crash.

For a subplan, if we can get a Expr for distkey column from the
targetlist, the Expr must be in the targetlist or reference the
Vars in the targetlist, The old logic is, if the Expr is valid,
the Expr that reference Vars is also added to the targetlist.
For example, a subplan has targetlist (c1) and the locus whose
distkey is (c1::float8), so the Expr returned will be one
referenced Vars in the targetlist and then the old targetlist
will be extended to (c1, c1::float8). Obviously, the extension
of targetlist here is unnecessary, the Expr can evaluate from
Vars, meanwhile, motion need to transfer more columns in a tuple.
Co-authored-by: NAdam Lee <ali@pivotal.io>
Reviewed-by: NHeikki Linnakangas <hlinnakangas@pivotal.io>
Reviewed-by: NRichard Guo <rguo@pivotal.io>
Reviewed-by: NDaniel Gustafsson <dgustafsson@pivotal.io>

This commit handles a missed case in the previous commit: "Fix
algebrization of subqueries in queries with complex GROUP BYs".

The logic inside RunExtractAggregatesMutator's Var case was intended to
fix top-level Vars inside subqueries in the targetlist, but also
incorrectly fixed top-level Vars in subqueries inside of aggregates.

The gpconfig test was failing for UTF-8 characters when run on Ubuntu because
our test containers use the POSIX locale. In this commit, we have set the
`LC_TYPE` to `en_US.UTF-8` for the gpconfig test so that the test has the same
behavior on all platforms.
Co-authored-by: NJacob Champion <pchampion@pivotal.io>

The `gen_pipeline()` function called the `suggested_git_remote()` and the
`suggested_git_branch()` functions as default values for the `git_remote` and
`git_branch` parameters.  For `prod` pipeline, the `gen_pipeline()` function is
called with GPDB repo and `BASE_BRANCH`.  However, the `suggested_*()` functions
are called in the `gen_pipeline()` function definition resulting in error as
they are not applicable for the production branches.

Therefore, in this commit we have used `None` as the default and call the
`suggested_*()` functions only if the corresponding parameters are not provided
by the caller.
Co-authored-by: NJacob Champion <pchampion@pivotal.io>

* docs - pxf now bundled with postgresql 42.2.5 jar

* remove postgresql jar registration from the example

Authored-by: NTingfang Bao <bbao@pivotal.io>

greenplum-db/greenplum-database-release

Previously, releng team merged PR
https://github.com/greenplum-db/gpdb/pull/7906
to release open source Greenplum-db in gpdb repo, and later on, we
decided to move these work to new repo
greenplum-db/greenplum-database-release, this commit delete the
previouse work
we have done
Reviewed-by: NMark Sliva <msliva@pivotal.io>
Reviewed-by: NAmil Khanzada <akhanzada@pivotal.io>

ORCA commit: Skip group expression containing circular references
Authored-by: NChris Hajas <chajas@pivotal.io>

* Docs: add note that gpcopy inherits encryption from client connection

* Change note to clarify that SSL encryption is not supported

* typo fix

Authored-by: NTingfang Bao <bbao@pivotal.io>

The resource rename is needed in order to fetch the correct version.
Co-authored-by: NBradford D. Boyle <bboyle@pivotal.io>
Co-authored-by: NMark Sliva <msliva@pivotal.io>

Co-authored-by: NBradford D. Boyle <bboyle@pivotal.io>
Co-authored-by: NMark Sliva <msliva@pivotal.io>