contrib/bufferedtest depends on LZJB code which was removed long
before the open sourcing of Greenplum, and contrib/varblocktest
hasn't built in a long time due to the Makefile being wonky. Since
noone has missed these tools, remove them rather than keeping dead
code around.

They were not testing anything interesting anymore. Per discussion on
gpdb-dev mailing list.

Discussion: https://groups.google.com/a/greenplum.org/d/msg/gpdb-dev/gq07ixYkTqk/ehLfByeIAAAJ

In the first cut of the walreplication work, I left out the part that
drops a segfile during AO compaction, because there was no
straightforward native PostgreSQL function to drop just a single segfile.
However, that leads to a massive disk space leak, as the space freed by
VACUUM on an AO table is not reclaimed, until the same segment number is
used again for further insertions, so we clearly must do something about
that.

Instead of dropping the segfile altogether, like we used to, truncate the
file to 0 bytes. Mostly because that's more expedient to do, as we have
the functions to do that and to WAL-log that, whereas code to fully remove
just a single segfile doesn't exist. But if you squint a bit, you could
even call that an optimization: it saves the overhead of recreating the
file again, if you load more data to it later.

Pointed out by Coverity.

As Coverity pointed out, 'pelem' has already been dereferenced in the
function, and hence it can never be NULL here. And indeed it should never
be NULL.

With 'prule', we explicitly checked for NULL earlier in the function.

This test keeps failing in CI intermittently, needs to be looked into as seems
hitting the assertion currently
"Unexpected internal error (postmaster.c:5328)","FailedAssertion(""!(CountChildren(0x0001|0x0002) == 0)"", File: ""postmaster.c"", Line: 5328)"

Once postmaster cleanup PR is done will revisit the test.

The IDENTIFICATION stanza in the source files is of questionable
usefulness, but unexpanded CVS keywords that never existed in the
codebase where the expansion happened are even less useful.
Replace with the filepath to make these headers consistent with
upstream.

A PL/JAVA NULL point reference has been reported by coverity scan.
The issue is located in guc.c for a pljava guc option
"pljava_classpath_insecure" and fixed in this commit.

tuple store which just advances current position by N instead of getting
tuple N times.

Author: Max Yang <myang@pivotal.io>

Since ICW is a very longrunning process, attempt to reduce time
consumption by reducing overhead during testing while keeping the
test constant (coverage not reduced).

Avoid dropping/recreating underlying test tables when not required,
reduce the number of partitions in some cases and skip pointless
drops for objects we know doesn't exist. In total this shaves about
30-40 seconds off an ICW run on my local machine, mileage may vary.

Make sure that we are able to exchnage in an external partition, and
also ensure that truncation doesn't recurse into the ext partition.

This backports the below commit which moved from the source timezone
file to the newly introduced compact format.

  commit 097b24cea68ac167a82bb617eb1844c8be4eaf24
  Author: Tom Lane <tgl@sss.pgh.pa.us>
  Date:   Sat Nov 25 15:30:11 2017 -0500

    Replace raw timezone source data with IANA's new compact format.

    Traditionally IANA has distributed their timezone data in pure source
    form, replete with extensive historical comments.  As of release 2017c,
    they've added a compact single-file format that omits comments and
    abbreviates command keywords.  This form is way shorter than the pure
    source, even before considering its allegedly better compressibility.
    Hence, let's distribute the data in that form rather than pure source.

    I'm pushing this now, rather than at the next timezone database update,
    so that it's easy to confirm that this data file produces compiled zic
    output that's identical to what we were getting before.

    Discussion: https://postgr.es/m/1915.1511210334@sss.pgh.pa.us

The timezone data in Greenplum are from the base version of
PostgreSQL that the current version of Greenplum is based on.
This cause issues since it means we are years behind on tz
changes that have happened. This pulls in the timezone data
and code from PostgreSQL 10.1 with as few changes to Greenplum
as possible to minimize merge conflicts. The goal is to gain
data rather than features, and for Greenplum for each release
to be able to stay current with the iana tz database as it is
imported into upstream PostgreSQL.

This removes a Greenplum specific test for the Yakutsk timezone
as it was made obsolete by upstream tz commit 1ac038c2c3f25f72.

pg_ctl and gpstart appear to be running simultaneously, because we're
not waiting for pg_ctl to finish before moving to the next call.
(subprocess.Popen() doesn't return the process's return code -- it
returns a Popen instance.)

cs_walrep_2 in the master pipeline is currently failing roughly 50% of
the time due to a crash during test_negative. Let's see if this helps at
all.

Author: Xin Zhang <xzhang@pivotal.io>
Author: Jimmy Yih <jyih@pivotal.io>
Author: Ashwin Agrawal <aagrawal@pivotal.io>

dbid is not consistent (like content 0 mirror will always have dbid 5), as
segments get created in parallel and get assigned those based on order of
execution. Hence better convention to use is content-id, to specify which
segment to connect. Framework currently connects to primary of specified
content-id.

If later needed, maybe when we enable hot-standby feature can extend further to
start supporting connecting to specified content-ids mirror as well. Not
something needed as of now.

Author: Xin Zhang <xzhang@pivotal.io>
Author: Ashwin Agrawal <aagrawal@pivotal.io>

This is mostly cherry-pick of upstream commit
commit 970a1868
Author: Robert Haas <rhaas@postgresql.org>
Date:   Fri Dec 3 08:44:15 2010 -0500

    Use GUC lexer for recovery.conf parsing.

    This eliminates some crufty, special-purpose code and, as a non-trivial
    side benefit, allows recovery.conf parameters to be unquoted.

    Dimitri Fontaine, with review and cleanup by Alvaro Herrera, Itagaki
    Takahiro, and me.

The test uses \retcode to ignore the output of the external script, but
remember the return code in the expected result.

It's very useful for this scenario to ensure, we don't care for the
details of the recoveryfull operation, but we do care this operation
complete successfully.

The test stop the primary and trigger failover to its mirror, then fail
back to the original segment configuration. At the end of the test, the
gp_segment_configuration is the same as the beginning of the test.

Author: Xin Zhang <xzhang@pivotal.io>
Author: Jacob Champion <pchampion@pivotal.io>
Author: Ashwin Agrawal <aagrawal@pivotal.io>

Originally, we skip probing primary without mirrors, however, we still
use the num_primary_segments to indicate number of requests. This caused
inconsistency when processing the response, since, not all the primary
segments got probe response.

The fix is to only track the actual number of probe requests sent.

Author: Xin Zhang <xzhang@pivotal.io>
Author: Ashwin Agrawal <aagrawal@pivotal.io>

- detect primary goes down
- flip the role to m/p/d/n and p/m/u/n (role/prefer/status/mode) in
  gp_segment_configuration
- send promotion message to mirror to promote it

Author: Xin Zhang <xzhang@pivotal.io>
Author: Jacob Champion <pchampion@pivotal.io>
Author: Asim R P <apraveen@pivotal.io>

Author: Xin Zhang <xzhang@pivotal.io>
Author: Ashwin Agrawal <aagrawal@pivotal.io>

In the past, the utility mode will not be retried, and caused wrong
behavior when trying to connect to given segment under utility mode.

In this refactoring, both utility mode and normal mode will go through
connectdb() that has the retry logic.

Author: Xin Zhang <xzhang@pivotal.io>
Author: Ashwin Agrawal <aagrawal@pivotal.io>

We don't want to diff the output from `segwalrep.py recoverfull`, but we
do want to know if the command has succeeded or not. Add a new execution
mode, invoked via

    !\retcode EXECUTABLE [OPTIONS ...]

which will bracket the program's stdout/stderr with begin/end_ignore,
and also print out an exit code (which is not ignored).

Author: Jacob Champion <pchampion@pivotal.io>
Author: Xin Zhang <xzhang@pivotal.io>
Author: Ashwin Agrawal <aagrawal@pivotal.io>

There is no reason to believe that these bugs could reappear, in the same
form.

It set the master_mirroring_administrator_disable variable, but it's not
used for anything. Remove the variable, and all the code to pass that flag
through.

It was now unused.

There is a race condition in this test, where our disabling of the FTS
probe process doesn't actually have to happen immediately (it takes a
little while for the GUC to propagate). This may be the cause of some
test failures we see where the primary is falling out of sync with the
mirror before we want it to.

Try a basic hack to see if it helps anything; the real fix will be to
move to faultinjection, but that's another story.

Author: Ashwin Agrawal <aagrawal@pivotal.io>
Author: Jacob Champion <pchampion@pivotal.io>

SetSyncStandbysDefined and friends are now much simpler; there isn't
really anything interesting to test anymore. We already have an
integration test that covers the operation for the whole cluster.