I don't know what purpose this might have served originally, but it's
unused now.
Reviewed-by: NAsim R P <apraveen@pivotal.io>

For commands called directly by the user, we provide the fix.

Since Behave and unit tests are supposed to behave as a normal user,
we do not provide the fix.  The fix is supposed to be done by the
commands themselves, and we want to test with an unmodified search_path
in the actual tests.
Co-authored-by: NJamie McAtamney <jmcatamney@pivotal.io>
Co-authored-by: NKalen Krempely <kkrempely@pivotal.io>
Co-authored-by: NNikolaos Kalampalikis <nkalampalikis@pivotal.io>
Co-authored-by: NShoaib Lari <slari@pivotal.io>
Co-authored-by: NDavid Krieger <dkrieger@pivotal.io>

When building file `_pg.so` on MacOS platform, distutil will invoke
clang compiler with arguments `-arch x86_64 -arch i386`. But type
`int128` is not available for i386 architecture, thus following error occurs:

```
In file included from pgmodule.c:32:
In file included from include/postgres.h:47:
include/c.h:427:9: error: __int128 is not supported on this target
typedef PG_INT128_TYPE int128
        ^
include/pg_config.h:838:24: note: expanded from macro 'PG_INT128_TYPE'
                       ^
In file included from pgmodule.c:32:
In file included from include/postgres.h:47:
include/c.h:433:18: error: __int128 is not supported on this target
```

By adding `['-arch', 'x86_64']` into `extra_compile_args`, distutil will
remove `-arch i386` from compiler arguments, thus fixes compile error.

Previously, it was possible for a utility to connect to a database of
the wrong version, if you for instance had a 5 instance running and
ran a 6 utility.  There was logic in gppylib to check GPDB versions,
but it was not being used anywhere and had not been updated since GPDB
5.

This commit updates the version logic to the current version and adds
a check in initFromCatalog that will error out if it tries to connect
to a database of a different version.
Co-authored-by: NJamie McAtamney <jmcatamney@pivotal.io>
Co-authored-by: NNikolaos Kalampalikis <nkalampalikis@pivotal.io>

Use pg_ctl to allow background worker connections during smart shutdown
and remove application_name whitelist, which was checked to allow
smart shutdown to proceed. When pg_ctl timeouts or the user decides
to Control-C , prompt user to continue shutting down in smart mode
while waiting forever, or shutdown in fast or immediate mode.
We allow the user to send a SIGINT to interrupt a long-running smart
mode shutdown and resend either a fast or immediate shutdown.
Co-authored-by: NJamie McAtamney <jmcatamney@pivotal.io>
Co-authored-by: NKalen Krempely <kkrempely@pivotal.io>
Co-authored-by: NShoai Lari <slari@pivotal.io>
Co-authored-by: NJacob Champion <pchampion@pivotal.io>
Co-authored-by: NNikolaos Kalampalikis <nkalampalikis@pivotal.io>

Setting a variable to itself is a no-op which can be removed. This
may have been introduced in error and instead masking a real bug,
but if it so then we have lived with it for two years so I'm opting
for removing.

Reviewed-by: Asim R P and Bhuvnesh Chaudhary

* Revert "CI: skip CLI behave tests that currently fail on ubuntu18.04"

This reverts commit 6a0cb2c6.
This commit is to re-enable behave test for centos/ubuntu

* Ouput the same error message format

gppkg for rpm and deb outputs error messages when consistency is broken, but
the message is not the same, which make a pipeline fail.

* Update gppkg for behave test and fix some minor error in gppkg

Now, the binary sample.gppkg is removed, instead we add sample-*.rpm and sample-*.deb.
Because sample.gppkg is platform specific and GP major version sensitive. The uploaded
rpm/deb files are available unless the rpm/deb file is incompatible on the specific
platform. GP_MAJORVERSION is dynamically retrieved from a makefile installed in the gpdb
folder, so the gp major version in the gppkg will be always correct.
sample.gppkg will only be generated when gppkg tag is provided

1) Run TEST_REMOTE.py of gpload
2) Run psql SSL, gpfdist Windows pipe testing
Co-authored-by: NPeifeng Qiu <pqiu@pivotal.io>
Co-authored-by: NTingfang Bao <bbao@pivotal.io>
Co-authored-by: NXiaoran Wang <xiwang@pivotal.io>
Co-authored-by: NXiaodong Huo <xhuo@pivotal.io>

Due to the number of platforms we now support and variety of customer
specific environments, it is not practical to update gpcheck to cover
all these cases.
Co-authored-by: NJamie McAtamney <jmcatamney@pivotal.io>

Follow previous commits, make gpexpand tablespace input file's
separator consistent with gpexpand input file.

This reverts commit 9fa19b76.

Analyze should update the statistics on the root partition when all the
leaf partitions have been analyzed. However, analyzedb runs analyze on
the leaf partitions in parallel and does not update the stats on the
root partition. Without the correct stats on the root partition, ORCA
ends up picking a bad plan.

Additionally, add tests to analyzedb to ensure root stats are updated.
In master/6X, the relpages/reltuples of a root partition table is not
updated in pg_class, rather this is calculated during planning time.
Therefore, the tests check that columns statistics in pg_statistic are
populated.

This will be backported to 6X_STABLE.

We simplify and refactor the parsing for gpaddmirrors, gpmovemirrors,
gprecoverseg and gpexpand.  This eliminates a few hundred lines of code.
In addition, this commit changes the format of the input lines for the
input files to these routines.  The separator is now '|' instead of ':'.

<Co-Authored-By> Mark Sliva <msliva@pivotal.io>
<Co-Authored-By> Jamie McCatamney <jmcatamney@pivotal.io>

Previously, PING_HOST swallowed any log messages from ping, and now we
display those messages in the error message.
Co-authored-by: NShoaib Lari <slari@pivotal.io>

Before we were comparing the live value with the quoted value in the file,
so it would incorrectly print a GUCS ARE OUT OF SYNC warning. To fix this
we had to re-implement the GUC_scanstr function in Python.
Co-authored-by: NMark Sliva <msliva@pivotal.io>

If we don't do this, the postgresql.conf file can easily be corrupted.
Co-authored-by: NJacob Champion <pchampion@pivotal.io>

Currently,

    gpconfig -c guc -v "''"

and

    gpconfig -c guc -v "''''"

result in the same value being written to postgresql.conf:

    guc=''''''

To avoid this confusing behavior, assume that the string passed to
gpconfig is the exact value that the user wants to be reflected in the
configuration, even if that value already contains single quotes.
Co-authored-by: NMark Sliva <msliva@pivotal.io>

The gp_bash_functions Bash library had accumulated some dead code
variables, left over from earlier refactorings and removal of utils.
Remove and perform a little bit of cleaning.

Reviewed-by: Jacob Champion

Currently, explicit 'BEGIN' creates a full-size writer gang and starts a transaction
on it, the following 'END' will commit the transaction in a two-phase way. It can be
optimized for some cases:
case 1:
BEGIN;
SELECT * FROM pg_class;
END;

case 2:
BEGIN;
SELECT * FROM foo;
SELECT * FROM bar;
END;

case 3:
BEGIN;
INSERT INTO foo VALUES(1);
INSERT INTO bar VALUES(2);
END;

For case 1, it's unnecessary to create a gang and no need to have two-phase commit.
For case 2, it's unnecessary to have two-phase commit as the executors don't write
any XLOG.
For case 3, don't have to create a full-size writer gang and do two-phase commit on
a full-size gang.
Co-authored-by: NJialun Du <jdu@pivotal.io>

This commit removes Paramiko and refactors gpssh-exkeys to work
without it.  Specifically, gpssh-exkeys now assumes that SSH keys
between the master and each segment host are already set up before it
is run, and now only exchanges keys among the segment hosts.

We also take the opportunity to remove some dead code and improve some
file handling.
Co-authored-by: NDavid Krieger <dkrieger@pivotal.io>
Co-authored-by: NJacob Champion <pchampion@pivotal.io>
Co-authored-by: NShoaib Lari <slari@pivotal.io>

GPexpand should take tablespaces into consideration. This means:

1. during the interview step stage, if there are user-created
     tablespaces in GreeplumDB cluster, we should generate a
     tablespace input file for GPexpand. User can modify it to
     use customized tablespace location on each new segment (Since
     we support create tablespace ... with (contentid=loc) syntax).

  2. during the template generatition stage, we should pack tablespace
     files generated by pg_basebackup into the template, and saves the
     restore paths into a json config file which is also a part of the
     template.

  3. During the stage of config a new segment, we should restore the
     tablespace files for primary. For mirrors, we just invoke pg_basebackup
     which has already handled tablespace cases.

An example of the whole process is shown below:

  * suppose there are tablespaces in the GreenplumDB cluster
  * tbs1 (oid1) -> location1; tbs2 (oid2) -> location2
  * and new segments are dbid1(primary), dbid2(mirror)

The tablespace input file will be:
```
  line1: tableSpaceNameOrders=tbs1:tbs2
  line2: tableSpaceOidOrders=oid1:oid2
  dbid1:loc1:loc2
```

The template's pg_tblspc dir will be:
  oid1
  oid2
  dumps
     |_ oid1
     |     |__ xxx_dbDummyID
     |
     |_ oid2
           |__ xxx_dbDummyID
  newTableSpaceInfo.json

Previously, gpconfig used bash and Perl to manipulate the master and
segment postgresql.conf files.  This made it hard to understand the
transformation on string valued gucs.
Co-authored-by: NJamie McAtamney <jmcatamney@pivotal.io>
Co-authored-by: NMark Sliva <msliva@pivotal.io>
Co-authored-by: NKalen Krempely <kkrempely@pivotal.io>

At present, `gpstop` check `pg_stat_activity` table before stop cluster in smart model. However, we need to ignore those background workers that are part of the database implementation itself. For this purpose, we maintain a whitelist using its `application_name` field.

- Copy and package dependency DLLs
- Add gpload.bat to make gpload runnable in cmd
- Fix pygresql build script
- Replace gpload version variable

Now that the enterprise version of GPDB is only provided via RPM, including
gpseginstall in the distribution would cause conflicts if users try to install
GPDB with RPMs and with gpseginstall on the same cluster.  While it could be
preserved for use by the OSS community, there are several standard tools for
copying GPDB files to segment hosts in a cluster, and recommendations for using
one or more of those tools will be included in the GPDB documentation.

In addition to removing gpseginstall itself, this commit removes references to
it in other utilities' documentation and removes code in gppylib that was only
called by gpseginstall.
Co-authored-by: NJamie McAtamney <jmcatamney@pivotal.io>
Co-authored-by: NKalen Krempely <kkrempely@pivotal.io>

Because users may find it confusing when gpconfig prints '-' or 'None'
when no GUC value is found in the file, this commit updates gpconfig to
output a clearer message.

Apply the out-of-band messaging to failure reports as well, and fix up
the output of --file-compare. Add tests for each modified
implementation.
Co-Authored-By: NJamie McAtamney <jmcatamney@pivotal.io>
Co-Authored-By: NMark Sliva <msliva@pivotal.io>
Co-Authored-By: NShoaib Lari <slari@pivotal.io>

Requires a working libpq.dll. Set CMAKE_PREFIX_PATH to a successful
client build.

To run behave tests, behave now needs to be installed in the PATH. The
correct version of behave can be installed with pip using

    pip install -r gpMgmt/requirements-dev.txt

For the CI tests, use virtualenv, which allows behave to be installed
while still using greenplum PYTHONPATH for our vendored python modules.
This is more of a problem for platforms like centos6 where the system
python is a different version than the vendored version.
Co-authored-by: NKalen Krempely <kkrempely@pivotal.io>
Co-authored-by: NMark Sliva <msliva@pivotal.io>
Co-authored-by: NDavid Krieger <dkrieger@pivotal.io>
Co-authored-by: NJacob Champion <pchampion@pivotal.io>

In the case where the user wants to change the port of a mirror, but
leave it in the same directory on the same host, gpmovemirrors was
previously deleting the directory as part of the cleanup because it
did not recognize that the old and new directories were the same.
This commit adds a check to prevent that.

This commit also logs a warning if the move_config_file contains
identical attributes(host,port,data directory) for the old and new
mirror.
Co-authored-by: NJamie McAtamney <jmcatamney@pivotal.io>
Co-authored-by: NShoaib Lari <slari@pivotal.io>
Co-authored-by: NDavid Krieger <dkrieger@pivotal.io>

While actual logic relating to filespaces was already removed from gpmovemirrors
in a previous commit, there were still some comments and dead code referencing
filespaces, which this commit removes.
Co-authored-by: NJamie McAtamney <jmcatamney@pivotal.io>
Co-authored-by: NShoaib Lari <slari@pivotal.io>

Mostly in code comments, but some in user facing messaging.
Reviewed-by: NHeikki Linnakangas <hlinnakangas@pivotal.io>
Reviewed-by: NJimmy Yih <jyih@pivotal.io>

Removes imports of modules once required, but for which usage has
been removed over the years in various refactorings.
Reviewed-by: NHeikki Linnakangas <hlinnakangas@pivotal.io>
Reviewed-by: NJimmy Yih <jyih@pivotal.io>

We don't support Greenplum on these platforms.

Some files (e.g. Makefile.{hpux,solaris}) have been left in place
because they are upstream postgres files. Removing them isn't
worth the headache it would cause when merging commits from
postgres.
Authored-by: NBen Christel <bchristel@pivotal.io>

Previously, gpconfig would not allow setting a GUC to a value of empty string
(that is, "gpconfig -c guc_name -v ''" or the like) because it conflated passing
an empty string with not setting the flag at all, so this commit fixes that and
ensures that empty-string values print nicely.
Authored-by: NJamie McAtamney <jmcatamney@pivotal.io>

replication_slots tests are hitting frequent intermittent failures
during gprecoverseg operation. Some failures are because mirrors aren't
being marked 'up' after gprecoverseg exits cleanly, and we already know
that FTS probes have a known race that occasionally causes them to do
nothing on the first call.

As a stop-gap before the probe race is fixed, try a double-call to
gp_request_fts_probe_scan() to see if that helps the situation any.

Originally introduced in commit 02cccd85,
which was reverted due to unexpected unit test failures. Those are fixed
in this patch.
Co-authored-by: NJamie McAtamney <jmcatamney@pivotal.io>

Upgrade Behave to 1.2.6

Unexpected unit test failures; reverting for now.

This reverts commit 02cccd85.

replication_slots tests are hitting frequent intermittent failures
during gprecoverseg operation. Some failures are because mirrors aren't
being marked 'up' after gprecoverseg exits cleanly, and we already know
that FTS probes have a known race that occasionally causes them to do
nothing on the first call.

As a stop-gap before the probe race is fixed, try a double-call to
gp_request_fts_probe_scan() to see if that helps the situation any.
Co-authored-by: NJamie McAtamney <jmcatamney@pivotal.io>

test_RemoteOperation_logger_debug() has been flaking out on the CI
pipeline, and there's no indication of what is going wrong. Replace the
assertTrue() call, which gives no indication of the difference between
actual and expected, with mock.assert_has_calls(), which will tell us
exactly what the calls were in case of failure.

It's possible that this will fix the flake entirely. The previous test
implementation depended on logger.debug() to be called *first* with our
expected output, but given the poor isolation of our global logger
system, it's entirely possible that some other code occasionally calls
debug(). (That this is an issue at all indicates that this isn't really
a unit test, but that's not something to tackle here.) assert_has_calls()
doesn't mind how many other calls happen as long as the one we're
looking for is eventually made, and I think that matches the intent of
the test better anyway.

Backport to 6X_STABLE.