This is mostly cherry-pick of upstream commit
commit 970a1868
Author: Robert Haas <rhaas@postgresql.org>
Date:   Fri Dec 3 08:44:15 2010 -0500

    Use GUC lexer for recovery.conf parsing.

    This eliminates some crufty, special-purpose code and, as a non-trivial
    side benefit, allows recovery.conf parameters to be unquoted.

    Dimitri Fontaine, with review and cleanup by Alvaro Herrera, Itagaki
    Takahiro, and me.

The test uses \retcode to ignore the output of the external script, but
remember the return code in the expected result.

It's very useful for this scenario to ensure, we don't care for the
details of the recoveryfull operation, but we do care this operation
complete successfully.

The test stop the primary and trigger failover to its mirror, then fail
back to the original segment configuration. At the end of the test, the
gp_segment_configuration is the same as the beginning of the test.

Author: Xin Zhang <xzhang@pivotal.io>
Author: Jacob Champion <pchampion@pivotal.io>
Author: Ashwin Agrawal <aagrawal@pivotal.io>

Originally, we don't wait for mirror in sync after recoverfull, and the
final state of the cluster would be non-deterministic. Since, we are
doing the full recovery here, hence the final state the mirrors should
always be in-sync with their primaries.

Author: Xn Zhang <xzhang@pivotal.io>
Author: Ashwin Agrawal <aagrawal@pivotal.io>

`rebuild` is different from `recoverfull`. It will create new directory
for the new mirror with new port for given content id.

`recoverfull` on the other hand will just recover all the failed mirrors
using existing information in gp_segment_configuration.

The both use basebackup to build the mirror.

We reduce port increment to avoid over the port limit after multiple
rebuild.

Author: Xin Zhang <xzhang@pivotal.io>
Author: Asim R P <apraveen@pivotal.io>

Ignore segments not marked as up for clusterstart operation.

Author: Xin Zhang <xzhang@pivotal.io>
Author: Asim R P <apraveen@pivotal.io>

Author: Xin Zhang <xzhang@pivotal.io>
Author: Asim R P <apraveen@pivotal.io>

Author: Jacob Champion <pchampion@pivotal.io>
Author: Asim R P <apraveen@pivotal.io>

Originally, we skip probing primary without mirrors, however, we still
use the num_primary_segments to indicate number of requests. This caused
inconsistency when processing the response, since, not all the primary
segments got probe response.

The fix is to only track the actual number of probe requests sent.

Author: Xin Zhang <xzhang@pivotal.io>
Author: Ashwin Agrawal <aagrawal@pivotal.io>

- detect primary goes down
- flip the role to m/p/d/n and p/m/u/n (role/prefer/status/mode) in
  gp_segment_configuration
- send promotion message to mirror to promote it

Author: Xin Zhang <xzhang@pivotal.io>
Author: Jacob Champion <pchampion@pivotal.io>
Author: Asim R P <apraveen@pivotal.io>

Author: Xin Zhang <xzhang@pivotal.io>
Author: Ashwin Agrawal <aagrawal@pivotal.io>

In the past, the utility mode will not be retried, and caused wrong
behavior when trying to connect to given segment under utility mode.

In this refactoring, both utility mode and normal mode will go through
connectdb() that has the retry logic.

Author: Xin Zhang <xzhang@pivotal.io>
Author: Ashwin Agrawal <aagrawal@pivotal.io>

We don't want to diff the output from `segwalrep.py recoverfull`, but we
do want to know if the command has succeeded or not. Add a new execution
mode, invoked via

    !\retcode EXECUTABLE [OPTIONS ...]

which will bracket the program's stdout/stderr with begin/end_ignore,
and also print out an exit code (which is not ignored).

Author: Jacob Champion <pchampion@pivotal.io>
Author: Xin Zhang <xzhang@pivotal.io>
Author: Ashwin Agrawal <aagrawal@pivotal.io>

If number of segments is not set to a value > 0, the assertion in
getgpsegmentCount() trips.  Executor calls this function from many
places.  Without this change, executing "select * from
gp_stat_replication;" trips the assertion on a promoted mirror because
number of contents is set to 0.

Author: Asim R P <apraveen@pivotal.io>
Author: Ashwin Agrawal <aagrawal@pivotal.io>

There is no reason to believe that these bugs could reappear, in the same
form.

It set the master_mirroring_administrator_disable variable, but it's not
used for anything. Remove the variable, and all the code to pass that flag
through.

It was now unused.

There is a race condition in this test, where our disabling of the FTS
probe process doesn't actually have to happen immediately (it takes a
little while for the GUC to propagate). This may be the cause of some
test failures we see where the primary is falling out of sync with the
mirror before we want it to.

Try a basic hack to see if it helps anything; the real fix will be to
move to faultinjection, but that's another story.

Author: Ashwin Agrawal <aagrawal@pivotal.io>
Author: Jacob Champion <pchampion@pivotal.io>

SetSyncStandbysDefined and friends are now much simpler; there isn't
really anything interesting to test anymore. We already have an
integration test that covers the operation for the whole cluster.

Revert SyncRepUpdateSyncStandbysDefined to its upstream contract: only
checkpointer may call it, and it acquires the SyncRepLock on its own.
If we don't need to ensure that the FTS responder sends back
syncrep=false immediately, the logic is vastly simplified, and we can
remove the assertion on the receiver end.

Also note that we don't need to set the GUC in memory directly; the
SIGHUP will load the new GUC from disk, further simplifying things here.

Author: Jacob Champion <pchampion@pivotal.io>
Author: Ashwin Agrawal <aagrawal@pivotal.io>

synchronous_standby_names is set on the first FTS probe after starting
the cluster, and it's written to disk, but it doesn't actually get
persisted in memory on all cluster processes. The backend that "sets"
the value exits without SIGHUP'ing the cluster, and it therefore
"resets" to the empty string. If you're lucky enough to restart the
cluster after this point, everything works as expected, but until then
the cluster is in a strange state.

This test (which currently fails) verifies that the
synchronous_standby_names GUC is actually set appropriately on the
segment.

The syncing should be done automatically by the mirror, when the mirror
connects to the primary.

Author: Marbin Tan <mtan@pivotal.io>
Author: Shoaib Lari <slari@pivotal.io>

gp_segment_configuration should be updated through
gp_add_segment_primary and gp_add_segment_mirror

- Add a fix for our status validation check.
We were checking an incorrect of elements in a tuple, it was always
return false if you only had primaries.
Make it such that the mirrors will always be assumed down when
validating when you only have primaries for the cluster.

Author: Marbin Tan <mtan@pivotal.io>

The column is called 'datadir', not 'directory'.

The vacuum was causing trouble, see github issue #4298. And it seems pretty
pointless or unrelated to gpexpand, anyway.

This reverts commit 4efea6f73ef1b828e7ca435a9a62790f787ed861. I found
the culprit, and don't need that debugging information anymore.

The "filespaces" argument was removed earlier, but I failed to remove the
corresponding %s from the string.

gp_remove_segment_persistent_entries() function doesn't exist any more.

To hunt down "not enough arguments for format string" failure we're seeing
MM_gpexpand_1 test in the pipeline

As a result of removing filespace, we need to pass the datadir instead.
That way, gp_add_segment_primary now matches gp_add_segment_mirror.

Author: Marbin Tan <mtan@pivotal.io>
Author: Jacob Champion <pchampion@pivotal.io>

Author: Marbin Tan <mtan@pivotal.io>

On further inspection, the code seems perfectly fine.

This reverts commit 0277b584 (which was already partially reverted in
0039844) and commit 4b4643f8. Commit 2b939748 already applied the
logical intent of these commits, as well as some improvements which were
accidentally reverted.

Author: Xin Zhang <xzhang@pivotal.io>
Author: Ashwin Agrawal <aagrawal@pivotal.io>