I don't know what purpose this might have served originally, but it's
unused now.
Reviewed-by: NAsim R P <apraveen@pivotal.io>

For commands called directly by the user, we provide the fix.

Since Behave and unit tests are supposed to behave as a normal user,
we do not provide the fix.  The fix is supposed to be done by the
commands themselves, and we want to test with an unmodified search_path
in the actual tests.
Co-authored-by: NJamie McAtamney <jmcatamney@pivotal.io>
Co-authored-by: NKalen Krempely <kkrempely@pivotal.io>
Co-authored-by: NNikolaos Kalampalikis <nkalampalikis@pivotal.io>
Co-authored-by: NShoaib Lari <slari@pivotal.io>
Co-authored-by: NDavid Krieger <dkrieger@pivotal.io>

When building file `_pg.so` on MacOS platform, distutil will invoke
clang compiler with arguments `-arch x86_64 -arch i386`. But type
`int128` is not available for i386 architecture, thus following error occurs:

```
In file included from pgmodule.c:32:
In file included from include/postgres.h:47:
include/c.h:427:9: error: __int128 is not supported on this target
typedef PG_INT128_TYPE int128
        ^
include/pg_config.h:838:24: note: expanded from macro 'PG_INT128_TYPE'
                       ^
In file included from pgmodule.c:32:
In file included from include/postgres.h:47:
include/c.h:433:18: error: __int128 is not supported on this target
```

By adding `['-arch', 'x86_64']` into `extra_compile_args`, distutil will
remove `-arch i386` from compiler arguments, thus fixes compile error.

Some gpexpand behave tests verify the behaviors of the `--duration` and
`--end` arguments, they expect the gpexpand data redistribution phase to
exist after 2 seconds before all the redistribution are done.  However
internally gpexpand check for the timeout at a 5-second interval, if the
data redistribution completes within 5 seconds then the tests will fail.
This is just what is happening recently.

Improve these tests by queueing more tables for redistribution.

Previously, it was possible for a utility to connect to a database of
the wrong version, if you for instance had a 5 instance running and
ran a 6 utility.  There was logic in gppylib to check GPDB versions,
but it was not being used anywhere and had not been updated since GPDB
5.

This commit updates the version logic to the current version and adds
a check in initFromCatalog that will error out if it tries to connect
to a database of a different version.
Co-authored-by: NJamie McAtamney <jmcatamney@pivotal.io>
Co-authored-by: NNikolaos Kalampalikis <nkalampalikis@pivotal.io>

Use pg_ctl to allow background worker connections during smart shutdown
and remove application_name whitelist, which was checked to allow
smart shutdown to proceed. When pg_ctl timeouts or the user decides
to Control-C , prompt user to continue shutting down in smart mode
while waiting forever, or shutdown in fast or immediate mode.
We allow the user to send a SIGINT to interrupt a long-running smart
mode shutdown and resend either a fast or immediate shutdown.
Co-authored-by: NJamie McAtamney <jmcatamney@pivotal.io>
Co-authored-by: NKalen Krempely <kkrempely@pivotal.io>
Co-authored-by: NShoai Lari <slari@pivotal.io>
Co-authored-by: NJacob Champion <pchampion@pivotal.io>
Co-authored-by: NNikolaos Kalampalikis <nkalampalikis@pivotal.io>

Setting a variable to itself is a no-op which can be removed. This
may have been introduced in error and instead masking a real bug,
but if it so then we have lived with it for two years so I'm opting
for removing.

Reviewed-by: Asim R P and Bhuvnesh Chaudhary

* Revert "CI: skip CLI behave tests that currently fail on ubuntu18.04"

This reverts commit 6a0cb2c6.
This commit is to re-enable behave test for centos/ubuntu

* Ouput the same error message format

gppkg for rpm and deb outputs error messages when consistency is broken, but
the message is not the same, which make a pipeline fail.

* Update gppkg for behave test and fix some minor error in gppkg

Now, the binary sample.gppkg is removed, instead we add sample-*.rpm and sample-*.deb.
Because sample.gppkg is platform specific and GP major version sensitive. The uploaded
rpm/deb files are available unless the rpm/deb file is incompatible on the specific
platform. GP_MAJORVERSION is dynamically retrieved from a makefile installed in the gpdb
folder, so the gp major version in the gppkg will be always correct.
sample.gppkg will only be generated when gppkg tag is provided

Now that the root cause for a single failing test on Ubuntu18.04 has
been merged(dc0a3cec), enable that test in the pipeline.

The gpconfig test was failing for UTF-8 characters when run on Ubuntu because
our test containers use the POSIX locale. In this commit, we have set the
`LC_TYPE` to `en_US.UTF-8` for the gpconfig test so that the test has the same
behavior on all platforms.
Co-authored-by: NJacob Champion <pchampion@pivotal.io>

In the Behave test for gpinitsystem, testing for database creation in default
timezone was done by setting the TZ variable to an empty string. On Ubuntu this
resulted in the `date +"%Z"` command giving `Universal` as the timezone rather
than `UTC`.

In this commit, the `TZ` variable is unset rather than set to an empty
string. Thus giving uniform behavior on Ubuntu and other platforms. Since we
already have a Behave step to unset a variable correctly, so we are removing
this incorrect step.
Co-authored-by: NJacob Champion <pchampion@pivotal.io>

We were not killing the postmaster processes because of the following reasons.

1. The `$` symbols passed to `Command` was expanded on the local machine due to
the `RemoteExecutionContext` implementation.  This made the awk filter a no-op.

2. We did not catch this on Centos because the Centos implementation of
/bin/kill will kill as many of the processes as are valid and ignoring the
invalid ones.  However, Ubuntu fails fast if any of the arguments are invalid.

3. The test step did not validate the result of `Command.run()`.  Thus ignoring
the failure.

We have replaced `Command` by `subprocess.check_call()`.
Co-authored-by: NJacob Champion <pchampion@pivotal.io>
Co-authored-by: NShoaib Lari <slari@pivotal.io>

1) Run TEST_REMOTE.py of gpload
2) Run psql SSL, gpfdist Windows pipe testing
Co-authored-by: NPeifeng Qiu <pqiu@pivotal.io>
Co-authored-by: NTingfang Bao <bbao@pivotal.io>
Co-authored-by: NXiaoran Wang <xiwang@pivotal.io>
Co-authored-by: NXiaodong Huo <xhuo@pivotal.io>

Prior to this commit, standard connections made to a primary segment
would hang forever if that segment was not started in utility mode,
because the call to cdb_setup() waits for a state that will never
arrive. The hang exists in 6X as well, for slightly different reasons
(an infinite wait on a semaphore).

Some utilities make use of pg_isready to verify segment status, and a
hung backend will report an incorrect status for the segment, which is
how this bug was discovered. The hung backend sticks around forever in
startup state as well.

Add an explicit check on QE segments that the incoming session does not
expect a dispatcher role, and bail out at the same place that we would
normally complain about a mismatched utility-mode connection. (If the
incoming session is set to the executor role, we'll raise a FATAL for
other reasons, but at least we won't hang.)

Add a regress test for the new behavior. We coopt the new
internal_connection test file for this, and rename it gp_connections.
We also add a more end-to-end regression test using the same
reproduction case as the original bug.
Co-authored-by: NJacob Champion <pchampion@pivotal.io>

Commit a993ef03 inadvertently introduced gp_role=utility in the
postmaster arguments when starting segments via gpsegstart. This is
because the sixth argument to SegmentStart is a boolean (utilityMode,
which defaults to False) but that commit incorrectly passes the
master_checksum_version instead. If checksums are enabled on master
(which sets the checksum version to 1, a truthy value), this code will
enable utility mode for the started segments.

SegmentStart does not take a checksum version argument, so remove it
entirely.

Ashwin noticed this in #6334, but we hadn't put two and two together
until now. An ideal followup would be to remove the huge number of
layers that make mistakes like this so easy to miss.
Co-authored-by: NJamie McAtamney <jmcatamney@pivotal.io>

We skip the following tests for the following reasons:
   - gpinitsystem: TZ interpretation difference for TZ '' vs unset
   - gppkg: underlying package zip file contains wrong OS version
   - gpactivatestandby: cause unknown
   - gprecoverseg: bash pipe interpretation difference
   - gpconfig: demo cluster has LOCALE set to POSIX, not UTF-8

We plan to fix these in a subsequent commit.
Co-authored-by: NDavid Krieger <dkrieger@pivotal.io>
Co-authored-by: NNikolaos Kalampalikis <nkalampalikis@pivotal.io>

We set up coverage collection on demo and ccp behave tests. We then define
new concourse steps that upload coverage files to the coverage bucket. This uses
a generic script we added that uploads a directory to a gcs bucket uri.
Co-authored-by: NNikolaos Kalampalikis <nkalampalikis@pivotal.io>
Co-authored-by: NKalen Krempely <kkrempely@pivotal.io>
Co-authored-by: NMark Sliva <msliva@pivotal.io>

The PostgreSQL semver regex did not account for developer versions
(e.g. 9.4.21 vs 9.5beta1). Relax the regex to make the test pass since
this was an oversight when the test was written.

Now that gpcheck has been removed update the documentation as well.

Due to the number of platforms we now support and variety of customer
specific environments, it is not practical to update gpcheck to cover
all these cases.
Co-authored-by: NJamie McAtamney <jmcatamney@pivotal.io>

Add instructions and a coverage config file for running local code coverage on
our Python test code.
Co-authored-by: NDavid Krieger <dkrieger@pivotal.io>
Co-authored-by: NJacob Champion <pchampion@pivotal.io>

Follow previous commits, make gpexpand tablespace input file's
separator consistent with gpexpand input file.

Co-authored-by: NWenlin Zhang <wzhang@pivotal.io>
Co-authored-by: NRenyuan Wang <rewang@pivotal.io>

This reverts commit 9fa19b76.

Analyze should update the statistics on the root partition when all the
leaf partitions have been analyzed. However, analyzedb runs analyze on
the leaf partitions in parallel and does not update the stats on the
root partition. Without the correct stats on the root partition, ORCA
ends up picking a bad plan.

Additionally, add tests to analyzedb to ensure root stats are updated.
In master/6X, the relpages/reltuples of a root partition table is not
updated in pg_class, rather this is calculated during planning time.
Therefore, the tests check that columns statistics in pg_statistic are
populated.

This will be backported to 6X_STABLE.

We simplify and refactor the parsing for gpaddmirrors, gpmovemirrors,
gprecoverseg and gpexpand.  This eliminates a few hundred lines of code.
In addition, this commit changes the format of the input lines for the
input files to these routines.  The separator is now '|' instead of ':'.

<Co-Authored-By> Mark Sliva <msliva@pivotal.io>
<Co-Authored-By> Jamie McCatamney <jmcatamney@pivotal.io>

Previously, PING_HOST swallowed any log messages from ping, and now we
display those messages in the error message.
Co-authored-by: NShoaib Lari <slari@pivotal.io>

Also add new tests for the integer with units GUC type.
Co-authored-by: NKalen Krempely <kkrempely@pivotal.io>

Escaping double quotes is no longer necessary since we now wrap the entire
value in single quotes and correctly escape it before writing it to the
postgresql.conf file.
Co-authored-by: NJacob Champion <pchampion@pivotal.io>

Before we were comparing the live value with the quoted value in the file,
so it would incorrectly print a GUCS ARE OUT OF SYNC warning. To fix this
we had to re-implement the GUC_scanstr function in Python.
Co-authored-by: NMark Sliva <msliva@pivotal.io>

If we don't do this, the postgresql.conf file can easily be corrupted.
Co-authored-by: NJacob Champion <pchampion@pivotal.io>

application_name explicitly filters non-ASCII bytes, so we went with
search_path.

We had to work around some ASCII/Unicode mixing in the tests, which will
probably come back to haunt us during the Python 3 move.
Co-authored-by: NMark Sliva <msliva@pivotal.io>

Currently,

    gpconfig -c guc -v "''"

and

    gpconfig -c guc -v "''''"

result in the same value being written to postgresql.conf:

    guc=''''''

To avoid this confusing behavior, assume that the string passed to
gpconfig is the exact value that the user wants to be reflected in the
configuration, even if that value already contains single quotes.
Co-authored-by: NMark Sliva <msliva@pivotal.io>

Change to `gpstop -u` for a quick GUC reload as opposed to a full
restart during tests. This requires us to choose GUCs that can be
reloaded during a SIGHUP (e.g. change gp_resource_group_cpu_limit to
checkpoint_completion_target).
Co-authored-by: NMark Sliva <msliva@pivotal.io>

There are no "edge cases". This change requires us to correctly escape
the search pattern for all cases, which is desirable anyway.
Co-authored-by: NMark Sliva <msliva@pivotal.io>

The gp_bash_functions Bash library had accumulated some dead code
variables, left over from earlier refactorings and removal of utils.
Remove and perform a little bit of cleaning.

Reviewed-by: Jacob Champion

Currently, explicit 'BEGIN' creates a full-size writer gang and starts a transaction
on it, the following 'END' will commit the transaction in a two-phase way. It can be
optimized for some cases:
case 1:
BEGIN;
SELECT * FROM pg_class;
END;

case 2:
BEGIN;
SELECT * FROM foo;
SELECT * FROM bar;
END;

case 3:
BEGIN;
INSERT INTO foo VALUES(1);
INSERT INTO bar VALUES(2);
END;

For case 1, it's unnecessary to create a gang and no need to have two-phase commit.
For case 2, it's unnecessary to have two-phase commit as the executors don't write
any XLOG.
For case 3, don't have to create a full-size writer gang and do two-phase commit on
a full-size gang.
Co-authored-by: NJialun Du <jdu@pivotal.io>

This commit removes Paramiko and refactors gpssh-exkeys to work
without it.  Specifically, gpssh-exkeys now assumes that SSH keys
between the master and each segment host are already set up before it
is run, and now only exchanges keys among the segment hosts.

We also take the opportunity to remove some dead code and improve some
file handling.
Co-authored-by: NDavid Krieger <dkrieger@pivotal.io>
Co-authored-by: NJacob Champion <pchampion@pivotal.io>
Co-authored-by: NShoaib Lari <slari@pivotal.io>

This commit adds some tests for gpssh-exkeys, as we want to be confident
in our test coverage before refactoring it to remove Paramiko.
Co-authored-by: NDavid Krieger <dkrieger@pivotal.io>
Co-authored-by: NJacob Champion <pchampion@pivotal.io>
Co-authored-by: NShoaib Lari <slari@pivotal.io>

GPexpand should take tablespaces into consideration. This means:

1. during the interview step stage, if there are user-created
     tablespaces in GreeplumDB cluster, we should generate a
     tablespace input file for GPexpand. User can modify it to
     use customized tablespace location on each new segment (Since
     we support create tablespace ... with (contentid=loc) syntax).

  2. during the template generatition stage, we should pack tablespace
     files generated by pg_basebackup into the template, and saves the
     restore paths into a json config file which is also a part of the
     template.

  3. During the stage of config a new segment, we should restore the
     tablespace files for primary. For mirrors, we just invoke pg_basebackup
     which has already handled tablespace cases.

An example of the whole process is shown below:

  * suppose there are tablespaces in the GreenplumDB cluster
  * tbs1 (oid1) -> location1; tbs2 (oid2) -> location2
  * and new segments are dbid1(primary), dbid2(mirror)

The tablespace input file will be:
```
  line1: tableSpaceNameOrders=tbs1:tbs2
  line2: tableSpaceOidOrders=oid1:oid2
  dbid1:loc1:loc2
```

The template's pg_tblspc dir will be:
  oid1
  oid2
  dumps
     |_ oid1
     |     |__ xxx_dbDummyID
     |
     |_ oid2
           |__ xxx_dbDummyID
  newTableSpaceInfo.json