Originally, gp_add_segment() is only used to add a primary segment. We
created a new gp_add_segment_primary() to keep the original
functionality, which automatically generate dbid and contentid.

We generalized the gp_add_segment() to be able to directly update
gp_segment_configuration and pg_filespace_entry for adding any type of
segments with full specification of segment and filespace mappings. In
this case, the new gp_add_segment() doesn't generate dbid and contentid
automatically, and rely on input parameters.

Originally there is separate code path in gp_add_segment_mirror() to
figure out primary dbid, such logic is actually common even with
gp_add_segment(), which can add both primary and mirror.

In that case, we refactor the primary dbid detection logic in common
function add_segment(), and refactor the gp_add_segment_mirror() to use
the add_segment() instead of add_segment_config_entry().

We update gpinitsystem to use the function gp_add_segment() instead of
update the gp_segment_configuration and pg_filespace_entry tables
directly via SQL.
Signed-off-by: NXin Zhang <xzhang@pivotal.io>

Concourse doesn't support AIX natively, we need to clone the repo
with the correspond commit on remote machine, compile the packages,
and download them back to concourse container as output.

Testing client and loader for platform without gpdb server is
another challenge. We setup GPDB server on concourse container just
like most installcheck tests, and use SSH tunnel to forward ports
from and to the remote host. This way both CL tools and GPDB server
feel they are on the same machine, and the test can run normally.

Previous code us python package psutil to get the mount
information of the system which will read the content
of /etc/mtab. In some environments, /etc/mtab does not
contain the mount point information of cgroups. In this
commit, we scan /proc/self/mounts to find out cgroup
mount point.

To update 5.json, we ran:

cat src/include/catalog/*.h | perl src/backend/catalog/process_foreign_keys.pl > gpMgmt/bin/gppylib/data/5.json
Signed-off-by: NJacob Champion <pchampion@pivotal.io>

    $ make -j -s install
    ...
    --- subprocess32, Linux only
    /bin/sh: line 3: [: =: unary operator expected
    --- stream
    ...
    Greenplum Database installation complete.

When `$(BLD_ARCH)` is empty, the check becomes `[ = 'aix7_ppc_64' ]`, and gets
the unary operator expected error.

Using gp_segment_configuration catalog table easily can find if mirrors exist or
not, do not need special table to communicate the same. Earlier
gp_fault_strategy used to convey 'n' for mirrorless system, 'f' for replication
and 's' for san mirrors. Since support for 's' was removed in 5.0 only purpose
gp_fault_strategy served was mirrored or not mirrored system. Hence deleting the
gp_fault_strategy table and at required places using gp_segment_configuration to
find the required info.

1. copy *test/regress/*.pm file to install locaiton to support
regression test diff
2. set LIBPATH and GP_LIBPATH_FOR_PYTHON env for AIX
Signed-off-by: NPeifeng Qiu <pqiu@pivotal.io>

This utility used to confirm data transfered by doing a md5 digest. This
commit changes behavior to use sha256 instead.
Signed-off-by: NShoaib Lari <slari@pivotal.io>

Signed-off-by: NNadeem Ghani <nghani@pivotal.io>

Some backups with Data Domain contained an incorrect path in their
report file. When checking whether the backup timestamp was in a pre or
post content-id format, restore would fail since the path in the report
file didn't match the expected pattern. Instead, we now check for the
timestamp format in a more generalized way to account for this discrepancy.
Signed-off-by: NChris Hajas <chajas@pivotal.io>

The MoveTransFilespaceLocally class in filespace.py used to verify that two directories
are equivalent by doing a md5 on their contents. This commit changes behavior to use
sha256 instead.
Signed-off-by: NNadeem Ghani <nghani@pivotal.io>

Signed-off-by: NShoaib Lari <slari@pivotal.io>

Previously, gpdbrestore would analyze all schemas in the database during
a schema-only restore. Now, only those schemas that were restored will
be analyzed.
Signed-off-by: NChris Hajas <chajas@pivotal.io>

There are times where the flags/tags are not passed in the pipeline
configuration but the jobs shows as green anwyays (false positive).
This is due to the return code being 0 event though we don't do
anything.

Force an error when we don't specify flags/tags when running behave
with the makefile.

Signed-off-by: NLarry Hamel <lhamel@pivotal.io>
Signed-off-by: NShoaib Lari <slari@pivotal.io>

In GpMirrorListToBuild.buildMirrors() if there are no mirrors to
build, then return True, so that gprecoverseg returns a return code
of 0.
Signed-off-by: NShoaib Lari <slari@pivotal.io>
Signed-off-by: NNadeem Ghani <nghani@pivotal.io>

* whitespace reformat
Signed-off-by: NShoaib Lari <slari@pivotal.io>

* gprecverseg unit test: remove redundancy
Signed-off-by: NShoaib Lari <slari@pivotal.io>

* Fix gprecoverseg recursive behavior

gprecoverseg called itself during a rebalance, through a Command object.
But this command didn't signal failures through a non-zero ret-code. So
the top-level gprecoverseg didn't check its stdout/stderr for error
messages and didn't echo them to its own stdout, though they were being
logged.

This commit changes behavior so gprecoverseg make another object and
invokes its run() method. Any errors are now shown to the user and
logged.
Signed-off-by: NMarbin Tan <mtan@pivotal.io>
Signed-off-by: NNadeem Ghani <nghani@pivotal.io>

This column has a different name between master and previous GPDB
versions.
Signed-off-by: NKaren Huddleston <khuddleston@pivotal.io>

There is a bug with python 2.7 where you can't use hashlib.md5() with a
system that has fips mode on. python 2.7 will segfault if you run the
following
`python -c "import ssl; import hashlib; m = hashlib.md5(); m.update('abc');"`

Use sha256 instead as a workaround of the python 2.7 md5 issue.

gp_era saves the hashed value into a file which gets read when creating
a new mirror. It's mainly used to see if any segments gets out of
synced with the new era file.

As AssignResGroupOnMaster() is called before the transaction is
actually started, so the failure won't cause transaction abort,
we need handle the error to prevent slot leaking.
Signed-off-by: NZhenghua Lyu <zlv@pivotal.io>

This bumps the copyright years to the appropriate years after not
having been updated for some time.  Also reformats existing code
headers to match the upstream style to ensure consistency.

The original gpload test suites is intended to be run locally,
so gpdb cluster and test must be on the same machine.

For platform that doesn't have GPDB server available but have
loader package, we need to run the test remotely.

As part of a previous commit, the WorkerPool of threads will raise if
the number of workers is set to 0. This prevents a coding error from
resulting in no work, where work was expected.

This commit reaches through code to protect against the numWorkers
parameter being 0.

In several cases, the number of workers is set to the number of segments
or the number of databases. We do not protect those with the expectation
that something is more significantly wrong if those sums are 0, and that
an exception would be fitting in those cases.
Signed-off-by: NMarbin Tan <mtan@pivotal.io>

Signed-off-by: NNadeem Ghani <nghani@pivotal.io>

Rework how environment variables transmitted in local bash and remote
ssh commands. If you have a bash command that starts with a conditional,
you'll get a syntax error.

To fix this, we added ampersands to join the setting of the environment
variables and the command itself.

NOTE: We removed an ExecutionContext static variable that recorded
environment variables. This feature was not used anywhere.
Signed-off-by: NMarbin Tan <mtan@pivotal.io>

Previously, RemoveDirectories and RemoveFiles used the unix command
"rm -rf", but this is inefficient for huge numbers of files.
Also, these functions accepted any globbed path.

Instead, use "rsync" to optimize deletion of files in a directory.
On a DCA using 1 million files, this increased speed by about 3x.

Also, this commit breaks up the different use-cases of deletion into
separate methods, adding methods RemoveDirectoryContents() and RemoveFile()
and RemoveGlob() to help isolate the assumptions of each case and
optimize for them.
Signed-off-by: NLarry Hamel <lhamel@pivotal.io>
Signed-off-by: NShoaib Lari <slari@pivotal.io>

This utility is slated for removal in v6. It is unused in general.
Fix obvious compile errors and basic logic for logging.
Signed-off-by: NShoaib Lari <slari@pivotal.io>

Signed-off-by: NLarry Hamel <lhamel@pivotal.io>

Signed-off-by: NLarry Hamel <lhamel@pivotal.io>

Signed-off-by: NMarbin Tan <mtan@pivotal.io>
Signed-off-by: NLarry Hamel <lhamel@pivotal.io>

Previously, during gpinitsystem, the standby was instantiated in the middle of
setting up the master. This ordering caused problems because
initializing the standby could cause an exit when an error
occurred. As a result of this early exit, the gp_toolkit and DCA gucs
were not set properly.

Instead, initialize the standby after the master is finished.
------------------------------------------
Previously the exit return code for gpinitsystem was always non-zero.
Now, it is non-zero only in an error or warning case.

The issue was due to SCAN_LOG interpretation of an empty string as a line count of one.
Fixed by changing to word count.
------------------------------------------
Initializing a standby can no longer cause gpinitsystem to exit early.
Added extra logging/output about standby master status.
Tell user at the end of gpinitsystem if gpinitstandby failed.
------------------------------------------
Signed-off-by: NMarbin Tan <mtan@pivotal.io>

This change adds a check to gpaddmirrors: First check if heap_checksum
setting is consistent across cluster. If not, fail immediately, else
continue with the normal workflow.
Signed-off-by: NShoaib Lari <slari@pivotal.io>
Signed-off-by: NMarbin Tan <mtan@pivotal.io>

Signed-off-by: NNadeem Ghani <nghani@pivotal.io>

1. Add dependency packages to Ivy
2. Modify set_bld_arch.sh to correctly recongize aix7
3. Disable unsupported python libraries on aix7.
4. Disable gpmapreduce for aix7
5. Set ADDON_DIR for aix7
   Signed-off-by: Peifeng Qiu pqiu@pivotal.io

This commit fixed the regresstion introduced by commit
WorkerPool: Error out if numWorkers is 0 or less.
Details in https://github.com/greenplum-db/gpdb/pull/3036Signed-off-by: NXiang Sheng <stanly.sxiang@gmail.com>

Remove global variable table_expand_error by checking the pool of done
ExpandCommand(s).
Signed-off-by: NMarbin Tan <mtan@pivotal.io>

This commit adds a check for cluster state, heap_checksum setting on all
primary segments match heap_checksum setting on master, before doing the
expansion.

If all primaries match the master, gpexpand continues with setting up expansion
segments. Otherwise, it logs the inconsistent primaries and exits.
Signed-off-by: NMarbin Tan <mtan@pivotal.io>

gpexpand had a lot of code in the __main__ module method, along with
global vars used by other methods and classes in the module.

This commit introduces a main() method, which can be called from unit tests,
and converts global vars to params and fields.
Signed-off-by: NShoaib Lari <slari@pivotal.io>

Signed-off-by: NNadeem Ghani <nghani@pivotal.io>