As part of 8.3 merge, upstream commit 295e6398
"Implement lazy XID allocation" was merged. But transactionIds were still
allocated in StartTransaction as code changes required to make it work for GPDB
with distrbuted transaction was pending, thereby feature remained as
disabled. Some progress was made by commit
a54d84a3 "Avoid assigning an XID to
DTX_CONTEXT_QE_AUTO_COMMIT_IMPLICIT queries." Now this commit addresses the
pending work needed for handling deferred xid allocation correctly with
distributed transactions and fully enables the feature.

Important highlights of changes:

1] Modify xlog write and xlog replay record for DISTRIBUTED_COMMIT. Even if
transacion is read-only for master and no xid is allocated to it, it can still
be distributed transaction and hence needs to persist itself in such a case. So,
write xlog record even if no local xid is assigned but transaction is
prepared. Similarly during xlog replay of the XLOG_XACT_DISTRIBUTED_COMMIT type,
perform distributed commit recovery ignoring local commit. Which also means for
this case don't commit to distrbuted log, as its only used to perform reverse
map of localxid to distributed xid.

2] Remove localXID from gxact, as its no more needed to be maintained and used.

3] Refactor code for QE Reader StartTransaction. There used to be wait-loop with
sleep checking to see if SharedLocalSnapshotSlot has distributed XID same as
that of READER to assign reader some xid as that of writer, for SET type
commands till READER actually performs GetSnapShotData(). Since now a) writer is
not going to have valid xid till it performs some write, writers transactionId
turns out InvalidTransaction always here and b) read operations like SET doesn't
need xid, any more hence need for this wait is gone.

4] Thow error if using distributed transaction without distributed xid. Earlier
AssignTransactionId() was called for this case in StartTransaction() but such
scenario doesn't exist hence convert it to ERROR.

5] QD earlier during snapshot creation in createDtxSnapshot() was able to assign
localXid in inProgressEntryArray corresponding to distribXid, as localXid was
known by that time. That's no more the case and localXid mostly will get
assigned after snapshot is taken. Hence now even for QD similar to QE's snapshot
creation time localXid is not populated but later found in
DistributedSnapshotWithLocalMapping_CommittedTest(). There is chance to optimize
and try to match earlier behavior somewhat by populating gxact in
AssignTransactionId() once locakXid is known but currently seems not so much
worth it as QE's anyways have to perform the lookups.

Leverage the fact that inProgressEntryArray is sorted based on distribXid while
creating the snapshot in createDtxSnapshot. So, can break out fast in function
DistributedSnapshotWithLocalMapping_CommittedTest().

GetTopTransactionId() bumps the xid counter, doesn't seem required for this
function. For GPDB this function may be called by QE READER, causing XID
assignment in reader which is a voilation. Hence instead use
ReadNewTransactionId() to just read current value instead of bumping the value.

The old comment was unsure if toast tables and indexes have their Oids
in sync across the cluster. They do, so let's rely on that, which is
simpler.

Rearranges ereport() calls and variable declarations to match the
upstream style a little closer, and reformats some comments. The
printing of the distribution policy was using strcat() safely, but
it takes some thinking to be sure so add a comment explaining the
code.

When creating a table using the CTAS construction with an explicit
distribution clause, the duplicate check for distribution columns
wasn't applied:

  CREATE TABLE tt AS SELECT * FROM t DISTRIBUTED BY (c,c);

Move the duplicate distribution check to the parsing stage since
it is a syntax error to supply duplicate column references. This
alters the behavior such that supplying a duplicate non-existing
column will error out on the duplication rather than the column
being non-existing. Since syntax errors should have precedence
over semantic errors I however find that this is the correct order.

Remove the existing duplicate checks which only applied to CREATE
TABLE and ALTER TABLE, and add test cases to cover CTAS. The ticket
reference was removed since it's not public and also clearly didn't
cover all the cases. The DROP TABLE is also moved to not using IF
EXISTS since it would be an error if the table didn't exist at that
point.

Digging into the git history, it was removed sometime during the 8.2
(sic) merge, back in 2006. I don't think it was intentional.

It was added back in 2009. Seems quite clear at this point that we're not
going to change how this works.

It was added back in 2010, as part of a patch to:

commit d0ca3d8a4333db510ac68145c30ff917626d2037
Author: kentj <a@b>
Date:   Mon Jan 25 13:50:14 2010 -0800

    MPP-7734: initialize executor nodes for the active slice instead of the
    whole tree. Postponding the initialization of the subplans of an Append
    node to the time when it is processed.
    Send init gpmon packages for the subnodes of the Append node even
    though we don't initialize them.

    [git-p4: depot-paths = "//cdb2/main/": change = 43428]

However, it was reverted only a few weeks later:

commit 5cb7e64a093dfcc1bcbdca5ed74c261e9c56d3a3
Author: kentj <a@b>
Date:   Tue Feb 16 16:40:49 2010 -0800

    MPP-8031, MPP-7734: revert back the Append changes, since it breaks the
    explain analyze. It is hard to fix the explain analyze with the existing
    Append changes. It needs some more thinking.

    [git-p4: depot-paths = "//cdb2/main/": change = 45078]

The revert removed all use of the flag, but left the flag behind. Remove
it.

A HashJoin would always have an outer_plan, hence the if-statement was
pointless. It was added a long time ago in GPDB, due to a bug that was
related to Adaptive Joins. We haven't had Adaptive Joins for ages, so
it was dead code.

This reverts commit b20a3f92 and 37cdd250
since the gptransfer suite is failing.

S3QueryAbort's message was hardcoded to "Query is aborted". This commit
initializes it via constructor.

This was missed in b20a3f92

Signed-off-by: NChumki Roy <croy@pivotal.io>

Instead of simply printing which files are different, print the contents
of the diff files.

The gp_create_table suite tests that the maximum number of columns
can be specified as a distribution clause (well, it really tests
that CREATE TABLE allows the maximum number of columns since that
check applies first), and then tests to CTAS from the resulting
table. There is no reason to believe that there is a difference
between CTAS with 1600 columns and CTAS with fewer columns. Remove
this case to speed up the test significantly and also adjust the
DROP TABLE clauses to match reality.

Rather than relying on always having a full cloned repository when
building, add fallbacks which work for trees with constrained (such
as shallow-cloned trees), or no, history. When the tagged commit is
reachable, use git describe, else rely on the version string in
configure since that's required to always be there regardless of
distribution mode. If we have git tree then include the HEAD sha1
as well.

While there, clean up versioning to more clearly separate us from
upstream.

Enable existing GPPC tests and move more from TINC to ICW to get higher
coverage.

For now we keep gppc_test, gppc_demo and tabfunc_gppc_demo test suites
separate, may investigate more and merge them.
Signed-off-by: NAdam Lee <ali@pivotal.io>
Signed-off-by: NYuan Zhao <yuzhao@pivotal.io>

[ci skip]

Relocating memory usage dump from ExecutorEnd to MemoryAccounting_Reset() as we no longer call ExecutorEnd upon abort. This ensures that we dump memory usage (if configured) once per command before resetting memory accounting. (#2113)

Not to be confused with a TINC test with the same name.
This test was brought into the main suite in
b82c1e60 as an effort to increase
visibility into all the tests that we cared about. We never had the
bandwidth to look at the intent of this test though.

There are a plethora of problem of the bugbuster version of
`schema_topology`:

0. It has very unclear and mixed intent. For example, it depends on
gphdfs (which nobody outside Pivotal can build) but it just tests that
we are able to revoke privilege of that protocol.
0. It creates and switches databases
0. The vast number of cases in this test is duplicating coverage for
tests elsewhere in `installcheck-good` and TINC.

Burning this with fire.

This commit splits up the single backup.feature file (where each
scenario both dumps and restores data) into two files, 
dumps_only.feature which performs all necessary dumps and is 
run against a 4.3 database and restores_only.feature which 
performs the corresponding restores and is run against a 5 
database. It also adds and/or modifies Behave steps to account
for differences in functionality between 4.3 and 5. It uses the 
installed GPDB python version when running behave tests. This is
necessary when running different versions of Greenplum with the 
same behave feature file.

Authors: Karen Huddleston, Chris Hajas, Todd Sedano

gpdbrestore improperly filters out blocks that contain drop statements.
BEGIN..END blocks from dumps should always be included in the restore.

* Remove a previously deprecated utility
* Remove mentions of gpsnap
Signed-off-by: NMarbin Tan <mtan@pivotal.io>

Commit fae97ae7 accidentally removed the prototype for matrix_add(), even
though it didn't remove the function itself. Put back the prototype, to
silence compiler warning about missing prototype.

These datafiles were no longer used by any test, remove since they
will otherwise just linger in there.

[ci skip]

Error tables were removed in 5.0 but the syntax for creating them
on external tables in pg_dump was left in. Remove the offending
INTO <relname> syntax to ensure restoring works in 5.0+.

Apply to pg_dump and cdb_dump_agent.

appendPQExpBuffer() is a variadic function which interprets the
second parameter as a printf format string. Where we just append
a static string, use appendPQExpBufferStr() instead which is much
faster. Where a single char is added, use appendPQExpBufferChar().