gpmigrator and gpmigrator_mirror are older utilities
to help the upgrade from 4.2 -> 4.3.
This is no longer necessary; we are removing all
related gpmigrator calls and mentions.

* Remove gpmigrator from travis
* Remove upgrade test
Signed-off-by: NChumki Roy <croy@pivotal.io>

--only show error for lacking MASTER_DATA_DIRECTORY for --file, not
other options

* In some test cases, we would like to have the main process shutdown
  without waiting for the pool thread. This is possible with a pool
  thread being a daemon.

- Modified patch from Jimmy
Signed-off-by: NLarry Hamel <lhamel@pivotal.io>

Previously, any restore that filtered based on table or
schema name did not restore user-created CASTs because
they were not included in the filter. This commit adds
support for restoring casts with table and schema filters
and the --change-schema flag.

Gptransfer will hang at the end of the program if there are existing
threads when the --validate=md5 argument is passed on Python 2.7. This adds explicit exit calls.

Authors: Chris Hajas and Jamie McAtamney

pg_amop, pg_amproc oids are not synchronized from master and segments, hence
adding them to known differences. There are more tables which needs to be
cleaned-up here based on function RelationNeedsSynchronizedOIDs() in catalog.c
based leaving that for separate commit.

Also, since there exist simpler way to skip performing check for column use the
same for `indcheckxmin` column of pg_index instead of what was added as part of
commit 79caf1c0

Also, cleanup some checks existing for versions prior to 4.1.

-- for guc's that are read-only when not accompanied by
--skipvalidation,
-- instead of saying a hidden guc doesn't exist, say that it is not
changeable and refer user to documentation
Signed-off-by: NChumki Roy <croy@pivotal.io>

The fault `twophase_transaction_commit_prepared` would cause the segment to
PANIC while writing a `COMMIT PREPARED` record on the segment. The master would
then retry the `COMMIT PREPARED` while the postmaster was still resetting on the
segment causing the master itself to `PANIC`. This patch introduces a new fault
injector `finish_prepared_before_commit`  which will not cause a `PANIC`, but
just error out since the intent of the test is to ensure that the retry works
correctly.

A collection typo fixes that were lying around.

[ci skip]

The gpcheckcat 'inconsistent' check should skip the pg_index.indcheckxmin column
for now as due to the HOT feature, the value of this column can be different
between the master and the segments.

A more long term fix will resolve the underlying issue that makes the
indcheckxmin column value to be different between the master and the segments.

`gpinitsystem`, `gpstart` and `gpstop` had different prompting before:

$ gpinitsystem -c gpinitsystem_config 
...
Continue with Greenplum creation **Yy/Nn>**
n
...

$ gpstart
...

Continue with Greenplum instance startup **Yy|Nn (default=N):**
**>** n
...

$ gpstop
...

Continue with Greenplum instance shutdown **Yy|Nn (default=N):**
**>** n
...

This commit makes them consistent:

$ gpinitsystem -c gpinitsystem_config 
...

Continue with Greenplum creation Yy|Nn (default=N):
> n

The commit includes a UDF to walk dirty shared buffers and a new fault
`fault_counter` to count the number of files fsync'ed by checkpointer process.

Also another new fault `bg_buffer_sync_default_logic` to flush all buffers for
BgBufferSync() for the background writer process.

Previously, backup files were keyed only to the dbid of the segment from
which they were backed up, which caused problems when trying to restore
files backed up on a mirror segment, restore files to a new cluster with
a different dbid mapping, and so forth.

This commit fixes the above issues by changing the backup file naming
convention to incorporate the content id of the backed-up segment and
enables restoring files based on the content id.  Also, gpdbrestore
seamlessly handles mixing the old and new filename formats, so users can
still restore a backup made with a GPDB 4.3 version of gpcrondump that
used the old format.

* Fix gpload reuse table issues

This commit fixes issues below:

1, reports unexpected error when reuse_table is enabled
2, doesn't recognize GPDB delimiter syntax like E'\t'
3, unnecessary case sensible of column names
Signed-off-by: NAdam Lee <ali@pivotal.io>
Signed-off-by: NNing Wu <nwu@pivotal.io>

* delimiter '\t' handled

* Fixed report warnings wrongly when reusable table exists

* remove the tailing space and simplify sql

The gp_external_grant_privileges GUC was needed before 4.0 to let
non superusers create external tables for gphdfs and http protocols.
This GUC was however deprecated during the 4.3 cycle so remove all
traces of it. The utility of the GUC was replaced in 4.0 when rights
management for external tables was implemented with the normal GRANT
REVOKE framework so this has been dead code for quite some time.

Remove GUC, code which handles it, all references to it from the
documentation and a release notes entry.

When analyzedb is run without --skip_root_partition flag like below,
it should also analyze the root partitions.

`analyzedb -d <dbname> -s <schema> --full -a`

The expected key format for `leaf_root_dict` used for the check
is a tuple form and the result returned from the SQL was a string
concatenated with ".". Because of this, candidates could not be matched
in the dictionary and we never got the root partition details.

Made the appropriate changes in analyzedb so that it will
identify and analyze on the root partitions.
Signed-off-by: NJemish Patel <jpatel-pivotal@pivotal.io>

Signed-off-by: NMarbin Tan <mtan@pivotal.io>

Signed-off-by: NLarry Hamel <lhamel@pivotal.io>

* We were accessing a non-existent variable as it's being generated by
  the later function. Swapping the two methods.

* Fix on top of a86e4901

Signed-off-by: NMarbin Tan <mtan@pivotal.io>

The gppersistentrebuild tool was trying to use string_agg without
casting the oid to text. This used to work before but not anymore. The
behave tests only have two scenarios but the second scenario would
always fail because it would run when a primary/mirror segment pair
were still in resync mode.

* When mocking, we may setup patches that are not cleaned up properly
  and causes test pollutions. Ensure that we do tearDown properly and
  raise a condition if we are not.
Signed-off-by: NMarbin Tan <mtan@pivotal.io>

When running gpconfig -s with --file option, we can now view the
postgresql.conf guc settings instead of just viewing the user guc
settings in the database. This option also reports if there are any
inconsistencies with the postgresql.conf between the segments.

reformat gpconfig:
* enable unit testing

remove gpconfig white-space

Add gpconfig behave tests:
* When you run gpconfig with -s and --file, you can sometimes run into
a nonetype string format error. This is due to us not joining all the
threads properly and closing them out properly.

* Added gpconfig behave tests to ensure that we are not missing anything
 in the unittest tests. As we are mocking the pool, we need to make sure
 that we are not doing anything silly, so adding minimal behave tests.
 Single node is probably enough.
Signed-off-by: NChumki Roy <croy@pivotal.io>
Signed-off-by: NMarbin Tan <mtan@pivotal.io>

Signed-off-by: NMarbin Tan <mtan@pivotal.io>

* Fixing a hardcoded python library path that may not be valid for other
  python version.
Signed-off-by: NMarbin Tan <mtan@pivotal.io>

Due to changes of external table "on master" feature
Signed-off-by: NHaozhou Wang <hawang@pivotal.io>

  - Add 2 catalog tables for resource group
  - add one column in pg_authid to record the resource group of a role
  - bump catalog version
  - hard code default resource group of roles temporarily
Signed-off-by: NKenan Yao <kyao@pivotal.io>

We will first support external table query execution on master segment
before we fully support task dispatch on master segment.

1. Modify grammar to enable ON clause for external table.
2. Modify ExtTableEntry catalog to record on clause.
3. Update legacy planner to support on clause for external table.
4. Update pg_dump.
5. Update regress tests.
Signed-off-by: NHaozhou Wang <hawang@pivotal.io>
Signed-off-by: NAdam Lee <ali@pivotal.io>
Signed-off-by: NYuan Zhao <yuzhao@pivotal.io>

* Update catalog version

For "Support ON MASTER clause for external table"

* Add on master regression tests for gpcloud

Signed-off-by: NChumki Roy <croy@pivotal.io>

Replaces long running behave tests in gptransfer suite with unit tests
since they're only testing argument parsing. This shortens the gptransfer
suite by about 15 minutes.

Authors: Chris Hajas and Jamie McAtamney

* rename 5.0.json to 5.json
* the code expects the file to be named based on the major version
  number, which now only has a single digit
Signed-off-by: NMarbin Tan <mtan@pivotal.io>