Rather than hardcoding to require /bin/bash, move to using a lookup
via "/usr/bin/env bash" to allow for greater portability of the code.
This also changes the Bash test to checking if the current shell is
actually Bash, rather than looking if bash is available on the file
system (since we by the above mentioned changes no longer need that).

The -not syntax is not portable across all platforms, most notably
OpenBSD, so use the more portable ! operator instead.

Discussion: https://github.com/greenplum-db/gpdb/pull/5155Co-authored-by: NDaniel Gustafsson <dgustafsson@pivotal.io>

Starting with this 8.4 commit 1d577f5e, backend
checks for the existence of the directory and if not present creates the
same. So, we can avoid creating the same in utilities.

This is needed during gprecoverseg full to preserve important files
such as pg_log files. We pass this flag down the call stack to prevent
other utilities such as gpinitstandby or gpaddmirror from using the
new flag. The new flag can be dangerous if not used properly and
should only be used when data directory file preservation is
necessary.

Co-authored-by: NJamie McAtamney <jmcatamney@pivotal.io>
Co-authored-by: NNadeem Ghani <nghani@pivotal.io>

We have a step to run gpinitstandby in mgmt_utils.py. Removing this code
to make it more likely that we standardize on using the step in
mgmt_utils.py.
Co-authored-by: NNadeem Ghani <nghani@pivotal.io>
Co-authored-by: NKevin Yeap <kyeap@pivotal.io>

- Add mirrors with and without standby, and ensure that the host
  assignment is identical between the two.
- Add mirrors, then kill one, and ensure that gprecoverseg operates
  correctly on the newly added mirror.
Co-authored-by: NNadeem Ghani <nghani@pivotal.io>
Co-authored-by: NJacob Champion <pchampion@pivotal.io>

Fix a bug where gpexpand would fail to run on a cluster that had a
standby master but no mirrors.
Co-authored-by: NNadeem Ghani <nghani@pivotal.io>
Co-authored-by: NKevin Yeap <kyeap@pivotal.io>

The gparray object was taking the existence of a standby as evidence
that the cluster had mirrors.
Co-authored-by: NNadeem Ghani <nghani@pivotal.io>
Co-authored-by: NKevin Yeap <kyeap@pivotal.io>

The pg_log directory has always been excluded using the pg_basebackup
exclude option (-E ./pg_log). With this change, we add it to the
static list inside of basebackup. Because of this change, we are able
to remove all instances of mkdir pg_log in our management
utilities. Previously, the utilities would always have to create the
pg_log directory after running pg_basebackup because the postmaster
does a validation check on the pg_log path existing.

This also helps us align better with upstream Postgres since the
pg_basebackup exclude option is Greenplum-specific and really not
needed at all. Our dynamic exclusion list hasn't changed for a very
long time (so it's pretty much static anyways) and is not maintained
in the utilities very well. We may actually remove the pg_basebackup
exclude option in the near future.

This commit updates the GPSD utility to capture the value of the column
`stainherit` and also the HLL counters stored in column `stavalues4`
generated for sample/full table scan based HLL analyze in `pg_statistic`
table.

This commit also updates minirepro utility to capture hyperloglog
counter
Signed-off-by: NEkta Khanna <ekhanna@pivotal.io>

* Rewrite circular buffer as a Python list

Since we end up returning a List object, we may as well keep is as
a List object from the start.
Co-authored-by: NDaniel Gustafsson <dgustafsson@pivotal.io>
Co-authored-by: NDivya Bhargov <dbhargov@pivotal.io>

- Add mirrors with and without standby, and ensure that the host
  assignment is identical between the two.
- Add mirrors, then kill one, and ensure that gprecoverseg operates
  correctly on the newly added mirror.
Co-authored-by: NNadeem Ghani <nghani@pivotal.io>
Co-authored-by: NJacob Champion <pchampion@pivotal.io>

For long running commands such as gpinitstandby with a large master data
directory, the server takes a long time. Therefore, there is no acitivity from
the client to the server. If the ClientAliveInterval is set, then the server
reports a timeout after ClientAliveInterval seconds.

Setting a ServerAliveInterval value less than the ClientAliveInterval interval
forces the client to send a Null message to the server.  Hence, avoiding the
timeout.
Co-authored-by: NJamie McAtamney <jmcatamney@pivotal.io>
Co-authored-by: NShoaib Lari <slari@pivotal.io>

We have added a test case to verify that the mirror configuration generated by
gpaddmirrors with the `-s` option is indeed spread over different hosts for each
of the primaries.
Co-authored-by: NJim Doty <jdoty@pivotal.io>
Co-authored-by: NJamie McAtamney <jmcatamney@pivotal.io>
Co-authored-by: NNadeem Ghani <nghani@pivotal.io>
Co-authored-by: NKevin Yeap <kyeap@pivotal.io>
Co-authored-by: NShoaib Lari <slari@pivotal.io>

Follow src/pl/plpython/README.md to see how to build
 and use plpython3u on GPDB.
Co-authored-by: NYandong Yao <yyao@pivotal.io>

We have added a unit test for gpaddmirrors when the mirror data directories are
provided interactively.
Co-authored-by: NNadeem Ghani <nghani@pivotal.io>
Co-authored-by: NShoaib Lari <slari@pivotal.io>

This reverts commit 15493596.

For long running commands such as gpinitstandby with a large master data
directory, the server takes a long time. Therefore, there is no acitivity from
the client to the server. If the ClientAliveInterval is set, then the server
reports a timeout after ClientAliveInterval seconds.

Setting a ServerAliveInterval value less than the ClientAliveInterval interval
forces the client to send a Null message to the server.  Hence, avoiding the
timeout.
Co-authored-by: NJamie McAtamney <jmcatamney@pivotal.io>
Co-authored-by: NShoaib Lari <slari@pivotal.io>
(cherry picked from commit 675aa8e3bd1d5bb187dc93d7ba494819cadb120e)

gpfdist was mistankenly misspelled as gpfidst, this fixes all found
occurrences in code and documentation.

Authored-by: NJim Doty <jdoty@pivotal.io>

Authored-by: NJim Doty <jdoty@pivotal.io>

This test explicitly creates some tables and verifies that they can be
read from after gpaddmirrors has been run on the cluster. The existing
TINC test that does this will be deleted after all the new gpaddmirrors
tests have been added.

We've also added some code to allow cluster creation on more than one
segment host.
Co-authored-by: NNadeem Ghani <nghani@pivotal.io>
Co-authored-by: NJamie McAtamney <jmcatamney@pivotal.io>

As part of the merge with PostgreSQL 9.1, two gpdb/master commits were
incorrectly reverted. This change replays the following commits:

commit dd42f3d4
Author: Todd Sedano <tsedano@pivotal.io>
Date:   Thu May 17 10:47:30 2018 -0700

    Syncing python-dependencies with pythonsrc-ext

    In comparing https://github.com/greenplum-db/pythonsrc-ext
    to python-dependencies.txt, there are several differences.

    pythonsrc-ext is the vendored python repo, so it should be
    in sync with pythonsrc-ext

    argparse is in pythonsrc-ext, adding it to python-dependencies.txt

    enum34, parse-type, ptyprocess, six are not in pythonsrc-ext,
    so moving them to python-developer-dependencies.txt
Authored-by: NTodd Sedano <tsedano@pivotal.io>

commit 7f9f11d3
Author: Daniel Gustafsson <dgustafsson@pivotal.io>
Date:   Fri May 18 09:43:48 2018 +0200

    Fix typo in comment

These scenarios come from the walrep_1 TINC Makefile target.  They have
been designed to run on both singlenode and multinode environments. The
tests have not been removed from TINC yet because some walrep_2 TINC
tests have dependencies.
Co-authored-by: NJimmy Yih <jyih@pivotal.io>
Co-authored-by: NNadeem Ghani <nghani@pivotal.io>

These scenarios come from the walrep_1 TINC Makefile target. They have
been designed to run on both singlenode and multinode environments.
The tests have not been removed from TINC yet because some walrep_2 TINC
tests have dependencies.
Co-authored-by: NJimmy Yih <jyih@pivotal.io>
Co-authored-by: NNadeem Ghani <nghani@pivotal.io>

These are ported from the master/standby TINC tests for checking the
following:

1. gpstart -a -y does nothing when there is no standby master configured
2. gpstop -a -y skips standby master when there is one configured
Co-authored-by: NNadeem Ghani <nghani@pivotal.io>
Co-authored-by: NJimmy Yih <jyih@pivotal.io>

The gpactivatestandby utility always used $MASTER_DATA_DIRECTORY
environment variable, even when the -d flag was populated by the user.
Fix the issue by actually using the directory path supplied by the user.
Co-authored-by: NKevin Yeap <kyeap@pivotal.io>
Co-authored-by: NJimmy Yih <jyih@pivotal.io>

Quick indent fix, the tables string was appended wrongly.
Signed-off-by: NMing LI <mli@apache.org>

We find out all the sequences in the table, and verify if the
   sequence should be created. later, we get the complete dump sql
   that included the sequnce DDL and table DDL, It can create
   the sequence before creating the table

   This implementation is to avoid locking table during drop table with sequence in transaction, meantime execute command `psql -f ...` to create this sequence.
Signed-off-by: NMing LI <mli@apache.org>

To "rebalance" a primary-mirror pair, gprecoverseg -r performs the following steps:

1.  bring down the acting primary
2.  issue a query that triggers the failover
3.  bring up the mirror (gprecoverseg -F)

Currently these 3 steps are happening in close succession. However, there is a chance that between step 2 and step 3, the mirror promotion happens slower than we expect. The implicit assumption here is that the acting mirror has finished transitioning to the primary role before step 3 is performed.

This patch adds a retry in "sort of step 2, definitely before step 3", to ensure a good state before we can bring up the mirror.

Co-authored-by: Jesse Zhang sbjesse@gmail.com
Co-authored-by: David Kimura dkimura@pivotal.io

Because the pg_dump handle the implicit sequence (serial column type)
and explicit sequence differently in different gpdb version, so we
need to detect the sequences which not included in the table dump sql,
then we should create them firstly. And also for all dependent sequence,
setval() to source value after data transferred.
Signed-off-by: NMing Li <mli@pivotal.io>

Prior to this commit, gpaddmirrors was missing two bits of work which
were previously done by gpinitsystem/gpcreateseg. When adding mirrors to
a cluster, the pg_hba.conf on primary segments has to be modified to
allow replication connections, e.g. for pgbasebackup. And after the
mirrors are built, the catalog has to be updated to reflect the new
cluster config.
Co-authored-by: NNadeem Ghani <nghani@pivotal.io>
Co-authored-by: NJim Doty <jdoty@pivotal.io>

This patch helps cut-down 8 secs out of 30 secs (for single node gpinitsystem as
part of gpdemo), mostly more for multinode setups.

- Removes explicit 2 sec and 1 sleep.
- Removes explicit call to CHECKPOINT, not required.
- Avoid starting primaries after initdb. As they were started and then shutdown
  pretty soon to start back cluster with required cmdline arguments.
- Add -w to mirror start.

Still more room for improvement and speedup, this just gets us started.

pycrypto version is now identical with the version in
python-dependencies.txt
Authored-by: NTodd Sedano <tsedano@pivotal.io>

Parse version is now identical with the version in
python-dependencies.txt
Authored-by: NTodd Sedano <tsedano@pivotal.io>

"False is not true" is not particularly helpful when looking at the CI.
Dump the diff file as part of the assertion message.