gpmigrator and gpmigrator_mirror are older utilities
to help the upgrade from 4.2 -> 4.3.
This is no longer necessary; we are removing all
related gpmigrator calls and mentions.

* Remove gpmigrator from travis
* Remove upgrade test
Signed-off-by: NChumki Roy <croy@pivotal.io>

--only show error for lacking MASTER_DATA_DIRECTORY for --file, not
other options

- Compile python 2.7.12 on centos6
- Update ext tarball on ivy to remove python 2.6.2
- Update ivy dependencies for python to point to 2.7.12
Signed-off-by: NTushar Dadlani <tdadlani@pivotal.io>
Signed-off-by: NTom Meyer <tmeyer@pivotal.io>

* In some test cases, we would like to have the main process shutdown
  without waiting for the pool thread. This is possible with a pool
  thread being a daemon.

- Modified patch from Jimmy
Signed-off-by: NLarry Hamel <lhamel@pivotal.io>

In commit e8dd6b61 we
changed the way gprestore_filter.py filters languages
and functions.  The new functionality is tested in the
base backup.feature suite and does not need to be
separately tested in netbackup.feature.

Previously, any restore that filtered based on table or
schema name did not restore user-created CASTs because
they were not included in the filter. This commit adds
support for restoring casts with table and schema filters
and the --change-schema flag.

Gptransfer will hang at the end of the program if there are existing
threads when the --validate=md5 argument is passed on Python 2.7. This adds explicit exit calls.

Authors: Chris Hajas and Jamie McAtamney

pg_amop, pg_amproc oids are not synchronized from master and segments, hence
adding them to known differences. There are more tables which needs to be
cleaned-up here based on function RelationNeedsSynchronizedOIDs() in catalog.c
based leaving that for separate commit.

Also, since there exist simpler way to skip performing check for column use the
same for `indcheckxmin` column of pg_index instead of what was added as part of
commit 79caf1c0

Also, cleanup some checks existing for versions prior to 4.1.

-- for guc's that are read-only when not accompanied by
--skipvalidation,
-- instead of saying a hidden guc doesn't exist, say that it is not
changeable and refer user to documentation
Signed-off-by: NChumki Roy <croy@pivotal.io>

The fault `twophase_transaction_commit_prepared` would cause the segment to
PANIC while writing a `COMMIT PREPARED` record on the segment. The master would
then retry the `COMMIT PREPARED` while the postmaster was still resetting on the
segment causing the master itself to `PANIC`. This patch introduces a new fault
injector `finish_prepared_before_commit`  which will not cause a `PANIC`, but
just error out since the intent of the test is to ensure that the retry works
correctly.

A collection typo fixes that were lying around.

[ci skip]

- Update sql/answer files to set timezone to PST.
- This prevents test failures if the tests are run on a
  machine that has a different timezone.

Authors: Chris Hajas and Karen Huddleston

The gpcheckcat 'inconsistent' check should skip the pg_index.indcheckxmin column
for now as due to the HOT feature, the value of this column can be different
between the master and the segments.

A more long term fix will resolve the underlying issue that makes the
indcheckxmin column value to be different between the master and the segments.

- Also remove unwanted test: the abort test should be skipped,
  based on feedback from Storage team
Signed-off-by: NC.J. Jameson <cjameson@pivotal.io>

`gpinitsystem`, `gpstart` and `gpstop` had different prompting before:

$ gpinitsystem -c gpinitsystem_config 
...
Continue with Greenplum creation **Yy/Nn>**
n
...

$ gpstart
...

Continue with Greenplum instance startup **Yy|Nn (default=N):**
**>** n
...

$ gpstop
...

Continue with Greenplum instance shutdown **Yy|Nn (default=N):**
**>** n
...

This commit makes them consistent:

$ gpinitsystem -c gpinitsystem_config 
...

Continue with Greenplum creation Yy|Nn (default=N):
> n

The commit includes a UDF to walk dirty shared buffers and a new fault
`fault_counter` to count the number of files fsync'ed by checkpointer process.

Also another new fault `bg_buffer_sync_default_logic` to flush all buffers for
BgBufferSync() for the background writer process.

Previously, backup files were keyed only to the dbid of the segment from
which they were backed up, which caused problems when trying to restore
files backed up on a mirror segment, restore files to a new cluster with
a different dbid mapping, and so forth.

This commit fixes the above issues by changing the backup file naming
convention to incorporate the content id of the backed-up segment and
enables restoring files based on the content id.  Also, gpdbrestore
seamlessly handles mixing the old and new filename formats, so users can
still restore a backup made with a GPDB 4.3 version of gpcrondump that
used the old format.

* Fix gpload reuse table issues

This commit fixes issues below:

1, reports unexpected error when reuse_table is enabled
2, doesn't recognize GPDB delimiter syntax like E'\t'
3, unnecessary case sensible of column names
Signed-off-by: NAdam Lee <ali@pivotal.io>
Signed-off-by: NNing Wu <nwu@pivotal.io>

* delimiter '\t' handled

* Fixed report warnings wrongly when reusable table exists

* remove the tailing space and simplify sql

* Run behave tests in concourse

* Initial steps to enable us to run behave tests in concourse.
  This will only work for single node behave tests.
Signed-off-by: NChumki Roy <croy@pivotal.io>

The gp_external_grant_privileges GUC was needed before 4.0 to let
non superusers create external tables for gphdfs and http protocols.
This GUC was however deprecated during the 4.3 cycle so remove all
traces of it. The utility of the GUC was replaced in 4.0 when rights
management for external tables was implemented with the normal GRANT
REVOKE framework so this has been dead code for quite some time.

Remove GUC, code which handles it, all references to it from the
documentation and a release notes entry.

Added a test for following scenario:
When analyzedb is run without --skip_root_partition flag like below,
it should also analyze the root partitions.

`analyzedb -d <dbname> -s <schema> -a`

The root parition table gets listed only in the report file (and not in
any state files), hence added appropriate routines for checking the root
partition in the report file.
Signed-off-by: NJemish Patel <jpatel@pivotal.io>

When analyzedb is run without --skip_root_partition flag like below,
it should also analyze the root partitions.

`analyzedb -d <dbname> -s <schema> --full -a`

The expected key format for `leaf_root_dict` used for the check
is a tuple form and the result returned from the SQL was a string
concatenated with ".". Because of this, candidates could not be matched
in the dictionary and we never got the root partition details.

Made the appropriate changes in analyzedb so that it will
identify and analyze on the root partitions.
Signed-off-by: NJemish Patel <jpatel-pivotal@pivotal.io>

Signed-off-by: NMarbin Tan <mtan@pivotal.io>

Signed-off-by: NLarry Hamel <lhamel@pivotal.io>

The "Execute on" syntax was added when performing a "\d" on external tables in gpdb5, but is not present in previous gpdb versions. When transferring from clusters on previous versions, we will ignore this syntax.

* We were accessing a non-existent variable as it's being generated by
  the later function. Swapping the two methods.

* Fix on top of a86e4901

Signed-off-by: NMarbin Tan <mtan@pivotal.io>

The gppersistentrebuild tool was trying to use string_agg without
casting the oid to text. This used to work before but not anymore. The
behave tests only have two scenarios but the second scenario would
always fail because it would run when a primary/mirror segment pair
were still in resync mode.

* When mocking, we may setup patches that are not cleaned up properly
  and causes test pollutions. Ensure that we do tearDown properly and
  raise a condition if we are not.
Signed-off-by: NMarbin Tan <mtan@pivotal.io>

When running gpconfig -s with --file option, we can now view the
postgresql.conf guc settings instead of just viewing the user guc
settings in the database. This option also reports if there are any
inconsistencies with the postgresql.conf between the segments.

reformat gpconfig:
* enable unit testing

remove gpconfig white-space

Add gpconfig behave tests:
* When you run gpconfig with -s and --file, you can sometimes run into
a nonetype string format error. This is due to us not joining all the
threads properly and closing them out properly.

* Added gpconfig behave tests to ensure that we are not missing anything
 in the unittest tests. As we are mocking the pool, we need to make sure
 that we are not doing anything silly, so adding minimal behave tests.
 Single node is probably enough.
Signed-off-by: NChumki Roy <croy@pivotal.io>
Signed-off-by: NMarbin Tan <mtan@pivotal.io>

Signed-off-by: NMarbin Tan <mtan@pivotal.io>