Currently, explicit 'BEGIN' creates a full-size writer gang and starts a transaction
on it, the following 'END' will commit the transaction in a two-phase way. It can be
optimized for some cases:
case 1:
BEGIN;
SELECT * FROM pg_class;
END;

case 2:
BEGIN;
SELECT * FROM foo;
SELECT * FROM bar;
END;

case 3:
BEGIN;
INSERT INTO foo VALUES(1);
INSERT INTO bar VALUES(2);
END;

For case 1, it's unnecessary to create a gang and no need to have two-phase commit.
For case 2, it's unnecessary to have two-phase commit as the executors don't write
any XLOG.
For case 3, don't have to create a full-size writer gang and do two-phase commit on
a full-size gang.
Co-authored-by: NJialun Du <jdu@pivotal.io>

This commit removes Paramiko and refactors gpssh-exkeys to work
without it.  Specifically, gpssh-exkeys now assumes that SSH keys
between the master and each segment host are already set up before it
is run, and now only exchanges keys among the segment hosts.

We also take the opportunity to remove some dead code and improve some
file handling.
Co-authored-by: NDavid Krieger <dkrieger@pivotal.io>
Co-authored-by: NJacob Champion <pchampion@pivotal.io>
Co-authored-by: NShoaib Lari <slari@pivotal.io>

GPexpand should take tablespaces into consideration. This means:

1. during the interview step stage, if there are user-created
     tablespaces in GreeplumDB cluster, we should generate a
     tablespace input file for GPexpand. User can modify it to
     use customized tablespace location on each new segment (Since
     we support create tablespace ... with (contentid=loc) syntax).

  2. during the template generatition stage, we should pack tablespace
     files generated by pg_basebackup into the template, and saves the
     restore paths into a json config file which is also a part of the
     template.

  3. During the stage of config a new segment, we should restore the
     tablespace files for primary. For mirrors, we just invoke pg_basebackup
     which has already handled tablespace cases.

An example of the whole process is shown below:

  * suppose there are tablespaces in the GreenplumDB cluster
  * tbs1 (oid1) -> location1; tbs2 (oid2) -> location2
  * and new segments are dbid1(primary), dbid2(mirror)

The tablespace input file will be:
```
  line1: tableSpaceNameOrders=tbs1:tbs2
  line2: tableSpaceOidOrders=oid1:oid2
  dbid1:loc1:loc2
```

The template's pg_tblspc dir will be:
  oid1
  oid2
  dumps
     |_ oid1
     |     |__ xxx_dbDummyID
     |
     |_ oid2
           |__ xxx_dbDummyID
  newTableSpaceInfo.json

Previously, gpconfig used bash and Perl to manipulate the master and
segment postgresql.conf files.  This made it hard to understand the
transformation on string valued gucs.
Co-authored-by: NJamie McAtamney <jmcatamney@pivotal.io>
Co-authored-by: NMark Sliva <msliva@pivotal.io>
Co-authored-by: NKalen Krempely <kkrempely@pivotal.io>

At present, `gpstop` check `pg_stat_activity` table before stop cluster in smart model. However, we need to ignore those background workers that are part of the database implementation itself. For this purpose, we maintain a whitelist using its `application_name` field.

- Copy and package dependency DLLs
- Add gpload.bat to make gpload runnable in cmd
- Fix pygresql build script
- Replace gpload version variable

Now that the enterprise version of GPDB is only provided via RPM, including
gpseginstall in the distribution would cause conflicts if users try to install
GPDB with RPMs and with gpseginstall on the same cluster.  While it could be
preserved for use by the OSS community, there are several standard tools for
copying GPDB files to segment hosts in a cluster, and recommendations for using
one or more of those tools will be included in the GPDB documentation.

In addition to removing gpseginstall itself, this commit removes references to
it in other utilities' documentation and removes code in gppylib that was only
called by gpseginstall.
Co-authored-by: NJamie McAtamney <jmcatamney@pivotal.io>
Co-authored-by: NKalen Krempely <kkrempely@pivotal.io>

Because users may find it confusing when gpconfig prints '-' or 'None'
when no GUC value is found in the file, this commit updates gpconfig to
output a clearer message.

Apply the out-of-band messaging to failure reports as well, and fix up
the output of --file-compare. Add tests for each modified
implementation.
Co-Authored-By: NJamie McAtamney <jmcatamney@pivotal.io>
Co-Authored-By: NMark Sliva <msliva@pivotal.io>
Co-Authored-By: NShoaib Lari <slari@pivotal.io>

Requires a working libpq.dll. Set CMAKE_PREFIX_PATH to a successful
client build.

To run behave tests, behave now needs to be installed in the PATH. The
correct version of behave can be installed with pip using

    pip install -r gpMgmt/requirements-dev.txt

For the CI tests, use virtualenv, which allows behave to be installed
while still using greenplum PYTHONPATH for our vendored python modules.
This is more of a problem for platforms like centos6 where the system
python is a different version than the vendored version.
Co-authored-by: NKalen Krempely <kkrempely@pivotal.io>
Co-authored-by: NMark Sliva <msliva@pivotal.io>
Co-authored-by: NDavid Krieger <dkrieger@pivotal.io>
Co-authored-by: NJacob Champion <pchampion@pivotal.io>

In the case where the user wants to change the port of a mirror, but
leave it in the same directory on the same host, gpmovemirrors was
previously deleting the directory as part of the cleanup because it
did not recognize that the old and new directories were the same.
This commit adds a check to prevent that.

This commit also logs a warning if the move_config_file contains
identical attributes(host,port,data directory) for the old and new
mirror.
Co-authored-by: NJamie McAtamney <jmcatamney@pivotal.io>
Co-authored-by: NShoaib Lari <slari@pivotal.io>
Co-authored-by: NDavid Krieger <dkrieger@pivotal.io>

While actual logic relating to filespaces was already removed from gpmovemirrors
in a previous commit, there were still some comments and dead code referencing
filespaces, which this commit removes.
Co-authored-by: NJamie McAtamney <jmcatamney@pivotal.io>
Co-authored-by: NShoaib Lari <slari@pivotal.io>

Mostly in code comments, but some in user facing messaging.
Reviewed-by: NHeikki Linnakangas <hlinnakangas@pivotal.io>
Reviewed-by: NJimmy Yih <jyih@pivotal.io>

Removes imports of modules once required, but for which usage has
been removed over the years in various refactorings.
Reviewed-by: NHeikki Linnakangas <hlinnakangas@pivotal.io>
Reviewed-by: NJimmy Yih <jyih@pivotal.io>

We don't support Greenplum on these platforms.

Some files (e.g. Makefile.{hpux,solaris}) have been left in place
because they are upstream postgres files. Removing them isn't
worth the headache it would cause when merging commits from
postgres.
Authored-by: NBen Christel <bchristel@pivotal.io>

Previously, gpconfig would not allow setting a GUC to a value of empty string
(that is, "gpconfig -c guc_name -v ''" or the like) because it conflated passing
an empty string with not setting the flag at all, so this commit fixes that and
ensures that empty-string values print nicely.
Authored-by: NJamie McAtamney <jmcatamney@pivotal.io>

replication_slots tests are hitting frequent intermittent failures
during gprecoverseg operation. Some failures are because mirrors aren't
being marked 'up' after gprecoverseg exits cleanly, and we already know
that FTS probes have a known race that occasionally causes them to do
nothing on the first call.

As a stop-gap before the probe race is fixed, try a double-call to
gp_request_fts_probe_scan() to see if that helps the situation any.

Originally introduced in commit 02cccd85,
which was reverted due to unexpected unit test failures. Those are fixed
in this patch.
Co-authored-by: NJamie McAtamney <jmcatamney@pivotal.io>

Upgrade Behave to 1.2.6

Unexpected unit test failures; reverting for now.

This reverts commit 02cccd85.

replication_slots tests are hitting frequent intermittent failures
during gprecoverseg operation. Some failures are because mirrors aren't
being marked 'up' after gprecoverseg exits cleanly, and we already know
that FTS probes have a known race that occasionally causes them to do
nothing on the first call.

As a stop-gap before the probe race is fixed, try a double-call to
gp_request_fts_probe_scan() to see if that helps the situation any.
Co-authored-by: NJamie McAtamney <jmcatamney@pivotal.io>

test_RemoteOperation_logger_debug() has been flaking out on the CI
pipeline, and there's no indication of what is going wrong. Replace the
assertTrue() call, which gives no indication of the difference between
actual and expected, with mock.assert_has_calls(), which will tell us
exactly what the calls were in case of failure.

It's possible that this will fix the flake entirely. The previous test
implementation depended on logger.debug() to be called *first* with our
expected output, but given the poor isolation of our global logger
system, it's entirely possible that some other code occasionally calls
debug(). (That this is an issue at all indicates that this isn't really
a unit test, but that's not something to tackle here.) assert_has_calls()
doesn't mind how many other calls happen as long as the one we're
looking for is eventually made, and I think that matches the intent of
the test better anyway.

Backport to 6X_STABLE.

Because the gpinitstandby flag for passing in a standby master data directory
was changed from -F to -S, this commit changes -F to -S in gpinitsystem to
match.

-S was already in use for specifying that the cluster should have its mirrors
set up in spread configuration, so that flag is changed to --mirror-mode, which
accepts the mirror mode ('group' or 'spread') as an argument.
Co-authored-by: NJamie McAtamney <jmcatamney@pivotal.io>
Co-authored-by: NMark Sliva <msliva@pivotal.io>

Despite my best efforts to avoid races on overloaded test containers,
test_join_and_indicate_progress_prints_dots_until_pool_is_done() has
been failing fairly often. Replace the simple-but-flaky time-based test
with an implementation that serializes the components of the race.

dbconn.connect() was using an odd pattern to set connection parameters
like client_encoding -- it would SET these parameters after the
connection was established and then issue a COMMIT. Even if your purpose
for connecting was read-only, you'd need to wait for a commit (which
also meant waiting for FTS, if a mirror was down). This is wasteful and
slow. Let's set those parameters during connection instead.

Now that we no longer wait for a transaction on connection, behave steps
that relied on those transactions need to be modified. Add unit tests to
cover the changed implementation.
Co-authored-by: NDavid Kimura <dkimura@pivotal.io>

Testing for equality in Python with 'is' tests for the variables
point to the same object, '==' tests for them containing the same
value. In this case we want '==' as we are comparing values.
Reviewed-by: NZhenghua Lyu <kainwen@gmail.com>

In the past several weeks, gpload test case 22 failed two times.
It may report: Fatal Python error: GC object already tracked.
We have reproduced this issue locally and in dev pipeline for 3
days, but it can't be reproduced.
So we decide to disable it now, in order to not blocking other PRs.

Any code dealing with replication ports was removed in previous commits; this
commit removes lingering references in comments and help text.
Co-authored-by: NJamie McAtamney <jmcatamney@pivotal.io>
Co-authored-by: NMark Sliva <msliva@pivotal.io>

For some areas of the ICW test framework -- isolation2 in particular --
the additional data written to stdout by gprecoverseg's progress
increased the load on the system significantly. (Some tests are
buffering stdout without bound, for instance.)  Additionally, the
updates were coming at ten times a second, which is an order of
magnitude more than the update interval we get from pg_basebackup
itself.

To help with this, we have have added a --no-progress flag that
suppresses the output of pg_basebackup.  We have also changed the
pg_basebackup progress update rate to once per second to minimize I/O.

The impacted regression/isolation2 tests utilizing gprecoverseg have
also been modified to use the --no-progress flag.
Co-authored-by: NJamie McAtamney <jmcatamney@pivotal.io>
Co-authored-by: NJacob Champion <pchampion@pivotal.io>

When -s is present, show pg_basebackup progress sequentially instead
of inplace. Useful when writing to a file, or if a tty does not support
escape sequences. Defaults to showing the progress inplace.

The gprecoverseg utility runs pg_basebackup in parallel on all segments that are
being recovered.  In this commit, we are logging the progress of each
pg_basebackup on its host and displaying them to the user of gprecoverseg.  The
progress files are deleted upons successful completion of gprecoverseg.

Unit tests have also been added.
Authored-by: NShoaib Lari <slari@pivotal.io>
Co-authored-by: NMark Sliva <msliva@pivotal.io>
Co-authored-by: NJacob Champion <pchampion@pivotal.io>
Co-authored-by: NEd Espino <edespino@pivotal.io>
Co-authored-by: NKalen Krempely <kkrempely@pivotal.io>

We add gpmovemirrors tests and add them to the pipeline as well.
Using behave tags, some of the tests can run locally on a demo
cluster.  We also introduce Behave tag logic to choose which
tests to run.

<Co-Authored-By> Kalen Krempely <kkrempely@pivotal.io>
<Co-Authored-By> Mark Sliva <msliva@pivotal.io>

This seems to have broken when we updated gprecoverseg to work with WAL
rep.

We removed the registerMirrorsInCatalog method from the
cluster-wide utilities.
Co-authored-by: NShoaib Lari <slari@pivotal.io>

Remove period from assertRaisesRegexp expectation to not be conflated
with the regex metacharacter.
Authored-by: NKalen Krempely <kkrempely@pivotal.io>

- get port from MASTER_DATA_DIRECTORY, so there is no confusion if
  PGPORT and MASTER_DATA_DIRECTORY are set to different clusters
- delete tmp status file 'gpexpand.standby.status' and copy the
  status file to standby directly
- get standby data directory from catalog instead of assuming its
  same with master
- copy gp_segment_configuration backup file to standby also, so
  standby can restore this catalog if master is down

This just removes unused imports, variables and functions as well
as cleans up a little bit of whitespace. No functional changes.
Reviewed-by: NAdam Berlin <aberlin@pivotal.io>
Reviewed-by: NShoaib Lari <slari@pivotal.io>
Reviewed-by: NKalen Krempely <kkrempely@pivotal.io>

Reword exception message to be more clear and concise.
Authored-by: NKalen Krempely <kkrempely@pivotal.io>