gpstart did a cluster-wide check of heap_checksum settings and refused
to start the cluster if this setting was inconsistent. This meant a
round of ssh'ing across the cluster which was causing OOM errors with
large clusters.

This commit moves the heap_checksum validation to gpsegstart.py, and
changes the logic so that only those segments which have the same
heap_checksum setting as master are started.

Author: Jim Doty <jdoty@pivotal.io>
Author: Nadeem Ghani <nghani@pivotal.io>
Author: Shoaib Lari <slari@pivotal.io>

- Change gpdeletsystem to delete tablespaces before datadir
- Refactor SegmentStart.noWait to pg_ctrl_wait
- Create PgBaseBackup class
- Revert the change to default Mirror status
- Assorted typos and bugfixes

Author: Shoaib Lari <slari@pivotal.io>
Author: Jim Doty <jdoty@pivotal.io>
Author: Nadeem Ghani <nghani@pivotal.io>

The gpexpand utility and associated code are modified to work with the WALREP
code.  Previously, gpexpand only worked with the primaries and relied on Filerep
to populate the mirrors. We are changing gpexpand such that it initializes the
mirrors using pg_basebackup and set them up for WAL replication.

Also, since the WALREP branch removed Filespaces, so we have also done the same
and replaced references to Filespaces by the Data Directory of the segments.

Author: Marbin Tan <mtan@pivotal.io>
Author: Shoaib Lari <slari@pivotal.io>
Author: Nadeem Ghani <nghani@pivotal.io>

The syncing should be done automatically by the mirror, when the mirror
connects to the primary.

Author: Marbin Tan <mtan@pivotal.io>
Author: Shoaib Lari <slari@pivotal.io>

The vacuum was causing trouble, see github issue #4298. And it seems pretty
pointless or unrelated to gpexpand, anyway.

This reverts commit 4efea6f73ef1b828e7ca435a9a62790f787ed861. I found
the culprit, and don't need that debugging information anymore.

To hunt down "not enough arguments for format string" failure we're seeing
MM_gpexpand_1 test in the pipeline

- Continuation of previous general filespace removal commits

Author: C.J. Jameson <cjameson@pivotal.io>

Persistent table are no more.

* Revert almost all the changes in smgr.c / md.c, to not go through
  the Mirrored* APIs.

* Remove mmxlog stuff. Use upstream "pending relation deletion" code
  instead.

* Get rid of multiple startup passes. Now it's just a single pass like
  in the upstream.

* Revert the way database drop/create are handled to the way it is in
  upstream. Doesn't use PT anymore, but accesses file system directly,
  and WAL-logs a single CREATE/DROP DATABASE WAL record.

* Get rid of MirroredLock

* Remove a few tests that were specific to persistent tables.

* Plus a lot of little removals and reverts to upstream code.

Long ago, we thought we might need to support multiple mirrors. But we
don't, and don't forsee it coming soon. Simplify the code to only ever
have one mirror, but still allow for the possibility of no mirrors

Author: Shoaib Lari <slari@pivotal.io>
Author: C.J. Jameson <cjameson@pivotal.io>

The gpArray use of GpDB and Segment classes was confusing. This change renames
GpDB to Segment and Segment to SegmentPair to clarify usage. Its a big diff, but
a simple, repeating change.

Author: Shoaib Lari <slari@pivotal.io>
Author: Marbin Tan <mtan@pivotal.io>
Author: C.J. Jameson <cjameson@pivotal.io>

Previously, RemoveDirectories and RemoveFiles used the unix command
"rm -rf", but this is inefficient for huge numbers of files.
Also, these functions accepted any globbed path.

Instead, use "rsync" to optimize deletion of files in a directory.
On a DCA using 1 million files, this increased speed by about 3x.

Also, this commit breaks up the different use-cases of deletion into
separate methods, adding methods RemoveDirectoryContents() and RemoveFile()
and RemoveGlob() to help isolate the assumptions of each case and
optimize for them.
Signed-off-by: NLarry Hamel <lhamel@pivotal.io>
Signed-off-by: NShoaib Lari <slari@pivotal.io>

Remove global variable table_expand_error by checking the pool of done
ExpandCommand(s).
Signed-off-by: NMarbin Tan <mtan@pivotal.io>

This commit adds a check for cluster state, heap_checksum setting on all
primary segments match heap_checksum setting on master, before doing the
expansion.

If all primaries match the master, gpexpand continues with setting up expansion
segments. Otherwise, it logs the inconsistent primaries and exits.
Signed-off-by: NMarbin Tan <mtan@pivotal.io>

gpexpand had a lot of code in the __main__ module method, along with
global vars used by other methods and classes in the module.

This commit introduces a main() method, which can be called from unit tests,
and converts global vars to params and fields.
Signed-off-by: NShoaib Lari <slari@pivotal.io>

Signed-off-by: NNadeem Ghani <nghani@pivotal.io>

When gpexpand is run with a specific duration or end-time using '-d' or
'-e' flag, there might be an off-chance that gpexpand reports that the
re-distribute was successful, however, that's not the case.
gpexpand status details reports as COMPLETED, but the data has not been
redistributed and is still distributed by RANDOM.
Signed-off-by: NNadeem Ghani <nghani@pivotal.io>
Signed-off-by: NTushar Dadlani <tdadlani@pivotal.io>

Signed-off-by: NLarry Hamel <lhamel@pivotal.io>

Signed-off-by: NMarbin Tan <mtan@pivotal.io>

* Address pylint warnings and errors

- Fix whitespace and indentation
- Remove unused variables
- Fix syntax errors
Signed-off-by: NMarbin Tan <mtan@pivotal.io>

Make consistent with:
https://github.com/greenplum-db/gpdb/commit/d7e6e0ecac5c49d16c68fcb827a23b0116d07b5aSigned-off-by: NChumki Roy <croy@pivotal.io>
Signed-off-by: NC.J. Jameson <cjameson@pivotal.io>

Signed-off-by: NC.J. Jameson <cjameson@pivotal.io>

This removes all utilities which are known to be deprecated from
gpMgmt and it also clears out any command line switches marked as
deprecated along with code marked deprecated. It also removes the
leftovers from previously removed utilities such as gpsuspend and
gpcheck_hostdump etc and fixes mentions of renamed utilities.

There should be no functional change from this commit as the only
removed utils have been marked deprecated for some time.

* fix gpexpand Worker-thread hang problem
When using python 2.7, gpexpand will hang when the command "gpexpand -D gpadmin" is done. sys.exit calls thread.join() which waits for each thread to exit; The sub-thread Worker will still try getting from the WorkerPool Queue. 

however, when using python 2.6.2, it's OK. Maybe the Python threading module changed somehow.

* Changes in gpexpand:
- Explicitly close the connection to the database before trying to stop
  it. Otherwise the database wouldn't stop due to active connection and
  the script will fail.
- Close the connection to the database before leaving the script.
  Otherwise it will not return to the command line