This reverts commit b20a3f92 and 37cdd250
since the gptransfer suite is failing.

S3QueryAbort's message was hardcoded to "Query is aborted". This commit
initializes it via constructor.

This was missed in b20a3f92

Signed-off-by: NChumki Roy <croy@pivotal.io>

Instead of simply printing which files are different, print the contents
of the diff files.

The gp_create_table suite tests that the maximum number of columns
can be specified as a distribution clause (well, it really tests
that CREATE TABLE allows the maximum number of columns since that
check applies first), and then tests to CTAS from the resulting
table. There is no reason to believe that there is a difference
between CTAS with 1600 columns and CTAS with fewer columns. Remove
this case to speed up the test significantly and also adjust the
DROP TABLE clauses to match reality.

Rather than relying on always having a full cloned repository when
building, add fallbacks which work for trees with constrained (such
as shallow-cloned trees), or no, history. When the tagged commit is
reachable, use git describe, else rely on the version string in
configure since that's required to always be there regardless of
distribution mode. If we have git tree then include the HEAD sha1
as well.

While there, clean up versioning to more clearly separate us from
upstream.

Enable existing GPPC tests and move more from TINC to ICW to get higher
coverage.

For now we keep gppc_test, gppc_demo and tabfunc_gppc_demo test suites
separate, may investigate more and merge them.
Signed-off-by: NAdam Lee <ali@pivotal.io>
Signed-off-by: NYuan Zhao <yuzhao@pivotal.io>

[ci skip]

Relocating memory usage dump from ExecutorEnd to MemoryAccounting_Reset() as we no longer call ExecutorEnd upon abort. This ensures that we dump memory usage (if configured) once per command before resetting memory accounting. (#2113)

Not to be confused with a TINC test with the same name.
This test was brought into the main suite in
b82c1e60 as an effort to increase
visibility into all the tests that we cared about. We never had the
bandwidth to look at the intent of this test though.

There are a plethora of problem of the bugbuster version of
`schema_topology`:

0. It has very unclear and mixed intent. For example, it depends on
gphdfs (which nobody outside Pivotal can build) but it just tests that
we are able to revoke privilege of that protocol.
0. It creates and switches databases
0. The vast number of cases in this test is duplicating coverage for
tests elsewhere in `installcheck-good` and TINC.

Burning this with fire.

This commit splits up the single backup.feature file (where each
scenario both dumps and restores data) into two files, 
dumps_only.feature which performs all necessary dumps and is 
run against a 4.3 database and restores_only.feature which 
performs the corresponding restores and is run against a 5 
database. It also adds and/or modifies Behave steps to account
for differences in functionality between 4.3 and 5. It uses the 
installed GPDB python version when running behave tests. This is
necessary when running different versions of Greenplum with the 
same behave feature file.

Authors: Karen Huddleston, Chris Hajas, Todd Sedano

gpdbrestore improperly filters out blocks that contain drop statements.
BEGIN..END blocks from dumps should always be included in the restore.

* Remove a previously deprecated utility
* Remove mentions of gpsnap
Signed-off-by: NMarbin Tan <mtan@pivotal.io>

Commit fae97ae7 accidentally removed the prototype for matrix_add(), even
though it didn't remove the function itself. Put back the prototype, to
silence compiler warning about missing prototype.

These datafiles were no longer used by any test, remove since they
will otherwise just linger in there.

[ci skip]

Error tables were removed in 5.0 but the syntax for creating them
on external tables in pg_dump was left in. Remove the offending
INTO <relname> syntax to ensure restoring works in 5.0+.

Apply to pg_dump and cdb_dump_agent.

appendPQExpBuffer() is a variadic function which interprets the
second parameter as a printf format string. Where we just append
a static string, use appendPQExpBufferStr() instead which is much
faster. Where a single char is added, use appendPQExpBufferChar().

When merging upstream releases, or inventing new GPDB specific code,
new object types may be required to be added to pg_upgrade. This
documents the process on a broad level.

-- it is red in pipeline with a report of OoM, but that could actually
be a symptom of a missing library.

This reverts commit e4976920.

* Rewrite Kerberos test suite

* Remove obsolete Kerberos test stuff from pipeline and TINC

We now have a rewritten Kerberos test script in installcheck-world.

* Update ICW kerberos test to run on concourse container

This adds a whole new test script in src/test/regress, implemented in plain bash. It sets up temporary a KDC as part of the script, and doesn't therefore rely on a pre-existing Kerberos server, like the old MU_kerberos-smoke test job did. It does require MIT Kerberos server-side utilities to be installed, instead, but no server needs to be running, and no superuser privileges are required.

This supersedes the MU_kerberos-smoke behave tests. The new rewritten bash script tests the same things:
  1. You cannot connect to the server before running 'kinit' (to verify that the server doesn't just let anyone in, which could happen if the pg_hba.conf is misconfigured for the test, for example)
  2. You can connect, after running 'kinit'
  3. You can no longer connect, if the user account is expired

The new test script is hooked up to the top-level installcheck-world target.

There were also some Kerberos-related tests in TINC. Remove all that, too. They didn't seem interesting in the first place, looks like they were just copies of a few random other tests, intended to be run as a smoke test, after a connection had been authenticated with Kerberos, but there was nothing in there to actually set up the Kerberos environment in TINC.

Other minor patches added:

* Remove absolute path when calling kerberos utilities
-- assume they are on path, so that they can be accessed from various installs
-- add clarification message if sample kerberos utility is not found with 'which'

* Specify empty load library for kerberos tools

* Move kerberos test to its own script file
-- this allows a failure to be recorded without exiting Make, and
therefore the server can be turned off always

* Add trap for stopping kerberos server in all cases
* Use localhost for kerberos connection
Signed-off-by: NMarbin Tan <mtan@pivotal.io>
Signed-off-by: NChumki Roy <croy@pivotal.io>
Signed-off-by: NLarry Hamel <lhamel@pivotal.io>

The tiny2 suite was a cut-down version of qp_derived_table with
no new tests added. The stransaction suite was already covered
by the ICW suites gpdtm_plpgsql and trasaction. Neither of these
two suites were no longer connected to any test schedule.

Signed-off-by: NYuan Zhao <yuzhao@pivotal.io>

Now that external tables use resource owners to clean up on abort, we
no longer need to ensure that ExecutorEnd gets called on failed portals.

Remove PG_TRY-CATCH block in nodeAgg.c that's no longer needed. This
demonstrates well why we should not call ExecutorEnd on abort; there
might well have been more places that previously needed a PG_TRY-CATCH
block like this, but didn't have one.

See also discussion on this on the gpdb-dev mailing list:
https://groups.google.com/a/greenplum.org/forum/#!topic/gpdb-dev/-BTKTbLuP1k

Re-applying this since the blocker was removed by ca3f5520.
Signed-off-by: NAdam Lee <ali@pivotal.io>

The old cdbfast test suite had these, move to ICW.
Signed-off-by: NAdam Lee <ali@pivotal.io>
Signed-off-by: NYuan Zhao <yuzhao@pivotal.io>

Generate certificates for gpfdists then place them into master and
segments' data directories.

After this commit, specifying gpfdist's `--ssl` option with
`$MASTER_DATA_DIRECTORY/gpfdists` could enable gpfdists protocol for
gpdemo.

Also update `lalshell` to remove unnecessary lines.
Signed-off-by: NAdam Lee <ali@pivotal.io>
Signed-off-by: NYuan Zhao <yuzhao@pivotal.io>

[ci skip]

In order for higher-level wrappers around pg_upgrade to know what
pg_upgrade does, this adds a trivial file based queue which reports
information on the operations performed. When started with -X, or
--progress, pg_upgrade will write progress as a set of operations
to a logfile. At intervals of either X number of operations, or Y
number of seconds, the file will be released and a new created such
that the released file can be consumed by a wrapper program. The
files will be named <sequence>.inprogress until released when they
are renamed to <sequence>.done. The sequence is gapless. Once the
file is renamed to <sequence>.done, pg_upgrade will never touch it
again and it can at that point be removed.

The format of the progress file is a simple semicolon delimited
line-based protocol:

  <timestamp>;<cluster>;<operation>;<message>;

timestamp is the number of microseconds since epoch; cluster defines
either "new", "old" or "none"; operation defines what actually did
happen and message contains the user defined message per report. The
timestamp is monotonically increasing for each operation such that
reading files in order is less important, operations can still be
individually sorted.

This commit includes a small set of progress report operations, while
not likely to be the final set it's enough to show how more can be
added.

Discussion:
  https://groups.google.com/a/greenplum.org/forum/#!topic/gpdb-dev/CVgK36oU5Oo

Partitioned tables with external tables in the partition hierarchy
cannot be upgraded since external partition management is prohibited
in utility mode (the partitioning system catalogs are not replicated
to the segments). Add a check for external partitions and abort the
upgrade if found.

This test is a bug reproducer of gpfdist, move it from cdbfast to keep
the coverage.
Signed-off-by: NYuan Zhao <yuzhao@pivotal.io>

* Add unit test for gpstart for better code coverage.

* Initial stab into creating unit tests for gpstart
Signed-off-by: NMarbin Tan <mtan@pivotal.io>

* Initial stab into creating unit tests for gpstop
Signed-off-by: NMarbin Tan <mtan@pivotal.io>

Signed-off-by: NDavid Sharp <dsharp@pivotal.io>

Signed-off-by: NTom Meyer <tmeyer@pivotal.io>