When merging upstream releases, or inventing new GPDB specific code,
new object types may be required to be added to pg_upgrade. This
documents the process on a broad level.

-- it is red in pipeline with a report of OoM, but that could actually
be a symptom of a missing library.

This reverts commit e4976920.

* Rewrite Kerberos test suite

* Remove obsolete Kerberos test stuff from pipeline and TINC

We now have a rewritten Kerberos test script in installcheck-world.

* Update ICW kerberos test to run on concourse container

This adds a whole new test script in src/test/regress, implemented in plain bash. It sets up temporary a KDC as part of the script, and doesn't therefore rely on a pre-existing Kerberos server, like the old MU_kerberos-smoke test job did. It does require MIT Kerberos server-side utilities to be installed, instead, but no server needs to be running, and no superuser privileges are required.

This supersedes the MU_kerberos-smoke behave tests. The new rewritten bash script tests the same things:
  1. You cannot connect to the server before running 'kinit' (to verify that the server doesn't just let anyone in, which could happen if the pg_hba.conf is misconfigured for the test, for example)
  2. You can connect, after running 'kinit'
  3. You can no longer connect, if the user account is expired

The new test script is hooked up to the top-level installcheck-world target.

There were also some Kerberos-related tests in TINC. Remove all that, too. They didn't seem interesting in the first place, looks like they were just copies of a few random other tests, intended to be run as a smoke test, after a connection had been authenticated with Kerberos, but there was nothing in there to actually set up the Kerberos environment in TINC.

Other minor patches added:

* Remove absolute path when calling kerberos utilities
-- assume they are on path, so that they can be accessed from various installs
-- add clarification message if sample kerberos utility is not found with 'which'

* Specify empty load library for kerberos tools

* Move kerberos test to its own script file
-- this allows a failure to be recorded without exiting Make, and
therefore the server can be turned off always

* Add trap for stopping kerberos server in all cases
* Use localhost for kerberos connection
Signed-off-by: NMarbin Tan <mtan@pivotal.io>
Signed-off-by: NChumki Roy <croy@pivotal.io>
Signed-off-by: NLarry Hamel <lhamel@pivotal.io>

The tiny2 suite was a cut-down version of qp_derived_table with
no new tests added. The stransaction suite was already covered
by the ICW suites gpdtm_plpgsql and trasaction. Neither of these
two suites were no longer connected to any test schedule.

Signed-off-by: NYuan Zhao <yuzhao@pivotal.io>

Now that external tables use resource owners to clean up on abort, we
no longer need to ensure that ExecutorEnd gets called on failed portals.

Remove PG_TRY-CATCH block in nodeAgg.c that's no longer needed. This
demonstrates well why we should not call ExecutorEnd on abort; there
might well have been more places that previously needed a PG_TRY-CATCH
block like this, but didn't have one.

See also discussion on this on the gpdb-dev mailing list:
https://groups.google.com/a/greenplum.org/forum/#!topic/gpdb-dev/-BTKTbLuP1k

Re-applying this since the blocker was removed by ca3f5520.
Signed-off-by: NAdam Lee <ali@pivotal.io>

The old cdbfast test suite had these, move to ICW.
Signed-off-by: NAdam Lee <ali@pivotal.io>
Signed-off-by: NYuan Zhao <yuzhao@pivotal.io>

Generate certificates for gpfdists then place them into master and
segments' data directories.

After this commit, specifying gpfdist's `--ssl` option with
`$MASTER_DATA_DIRECTORY/gpfdists` could enable gpfdists protocol for
gpdemo.

Also update `lalshell` to remove unnecessary lines.
Signed-off-by: NAdam Lee <ali@pivotal.io>
Signed-off-by: NYuan Zhao <yuzhao@pivotal.io>

[ci skip]

In order for higher-level wrappers around pg_upgrade to know what
pg_upgrade does, this adds a trivial file based queue which reports
information on the operations performed. When started with -X, or
--progress, pg_upgrade will write progress as a set of operations
to a logfile. At intervals of either X number of operations, or Y
number of seconds, the file will be released and a new created such
that the released file can be consumed by a wrapper program. The
files will be named <sequence>.inprogress until released when they
are renamed to <sequence>.done. The sequence is gapless. Once the
file is renamed to <sequence>.done, pg_upgrade will never touch it
again and it can at that point be removed.

The format of the progress file is a simple semicolon delimited
line-based protocol:

  <timestamp>;<cluster>;<operation>;<message>;

timestamp is the number of microseconds since epoch; cluster defines
either "new", "old" or "none"; operation defines what actually did
happen and message contains the user defined message per report. The
timestamp is monotonically increasing for each operation such that
reading files in order is less important, operations can still be
individually sorted.

This commit includes a small set of progress report operations, while
not likely to be the final set it's enough to show how more can be
added.

Discussion:
  https://groups.google.com/a/greenplum.org/forum/#!topic/gpdb-dev/CVgK36oU5Oo

Partitioned tables with external tables in the partition hierarchy
cannot be upgraded since external partition management is prohibited
in utility mode (the partitioning system catalogs are not replicated
to the segments). Add a check for external partitions and abort the
upgrade if found.

This test is a bug reproducer of gpfdist, move it from cdbfast to keep
the coverage.
Signed-off-by: NYuan Zhao <yuzhao@pivotal.io>

* Add unit test for gpstart for better code coverage.

* Initial stab into creating unit tests for gpstart
Signed-off-by: NMarbin Tan <mtan@pivotal.io>

* Initial stab into creating unit tests for gpstop
Signed-off-by: NMarbin Tan <mtan@pivotal.io>

Signed-off-by: NDavid Sharp <dsharp@pivotal.io>

Signed-off-by: NTom Meyer <tmeyer@pivotal.io>

usermod -a to append supplementary groups doesn't work on SLES.
Signed-off-by: NDavid Sharp <dsharp@pivotal.io>

Signed-off-by: NDavid Sharp <dsharp@pivotal.io>

Signed-off-by: NJingyi Mei <jmei@pivotal.io>

Signed-off-by: NJingyi Mei <jmei@pivotal.io>
Signed-off-by: NDavid Sharp <dsharp@pivotal.io>

* In production pljava_classpath can only be set by super user or
in postgresql.conf
In order to allow developers to set it for testing we add a
GUC called pljava_classpath_insecure.
This can only be set in postgresql.conf or a super user
When it is true, pljava_classpath can be set by a user.

* make pljava_classpath_insecure visible in show all

* GPDB DOCS - Add BINARY keyword to COPY command

* Very minor edits

[ci skip]

Since we have the documentation in the repo now, avoid linking to
the web version since the link is bound to do stale quickly.

gpmigrator and gpmigrator_mirror are older utilities
to help the upgrade from 4.2 -> 4.3.
This is no longer necessary; we are removing all
related gpmigrator calls and mentions.

* Remove gpmigrator from travis
* Remove upgrade test
Signed-off-by: NChumki Roy <croy@pivotal.io>

Previously, -a -m from gpstart still resulted into a prompt due to a
bug.
This actually made it such that the changed line did not run
properly -- probably hung on the prompt, since we don't validate if
gpstart actually runs properly.
6edeefec fixed the gpstart prompt issue and
caused this issue (actually running with master only) to finally show
up, which made the later tests to fail for unabling queries to the
database.. Ensure that we start fresh by doing
a gpstart -a instead of just doing master only.

--only show error for lacking MASTER_DATA_DIRECTORY for --file, not
other options

And older version of GPDB generates a bad plan. Changing the query to
generate the right plan for testing workfile which has Hash join instead
of nested loop.
Signed-off-by: NShreedhar Hardikar <shardikar@gmail.com>

- Compile python 2.7.12 on centos6
- Update ext tarball on ivy to remove python 2.6.2
- Update ivy dependencies for python to point to 2.7.12
Signed-off-by: NTushar Dadlani <tdadlani@pivotal.io>
Signed-off-by: NTom Meyer <tmeyer@pivotal.io>

* In some test cases, we would like to have the main process shutdown
  without waiting for the pool thread. This is possible with a pool
  thread being a daemon.

- Modified patch from Jimmy
Signed-off-by: NLarry Hamel <lhamel@pivotal.io>

In commit e8dd6b61 we
changed the way gprestore_filter.py filters languages
and functions.  The new functionality is tested in the
base backup.feature suite and does not need to be
separately tested in netbackup.feature.

* adding supported 9.1 xml functions to built-in list

* updates to SQL2008 support table for new XML functions

* adding expressions to function table

The old cdbfast test suite had these, move to ICW.
Signed-off-by: NAdam Lee <ali@pivotal.io>
Signed-off-by: NYuan Zhao <yuzhao@pivotal.io>

- gpdb supports anyelement, anyarray, anynonarray, and any enum types
- add polymorphic types section to data types page
- add polymorphic function example to create function page
- add polymorphic info and example to pl/pgsql page

removing duplicated hadoop compatibility table; clarifying that some hadoop versions no longer supported (#2075)