INSERT needs to Lock the child table on QD, otherwise the following
dead lock scenario may happen if INSERT runs concurrently with VACUUM
drop phase on an AppendOnly partitioned table.

1. VACUUM on QD: acquired AccessExclusiveLock on the child table
2. INSERT on QE: acquired RowExclusiveLock on the child table
3. VACUUM on QE: waiting for AccessExclusiveLock on the child table
4. INSERT on QD: waiting for AccessShareLock at ExecutorEnd() on the
  child table, this is after QE sends back which child it inserted

Note that in step 2, INSERT only locks the child table on QE, it does
not lock the child table on QD in the previous code. This patch adds the
lock on QD as well to prevent the above dead lock.

Added `insert_while_vacuum_drop` and updated `partition_locking` to
reflect the changes

A warning message is displayed to a user that attempts to perform
`pg_create_physical_replication_slot()` on any individual segment,
including the master segment so that they are aware that this command
is not made to be distributed to the entire cluster.

* Remove unused IVYREPO_ variables for CentOS 6/7

Ivy is no longer used for CentOS 6/7.
Authored-by: NAmil Khanzada <akhanzada@pivotal.io>

* Remove unused IVYREPO_ variables for Coverity

Ivy is no longer used for CentOS 6/7 and will go away.
The coverity pipeline is also puased at this point in time, but we'll
clean this up now.
Co-authored-by: NBradford D. Boyle <bboyle@pivotal.io>
Co-authored-by: NAmil Khanzada <akhanzada@pivotal.io>

* docs - gpexpand.status_detail external_writable field

* gpexpand.status tbl - updated col now timestamp without tz

* docs - organize contrib modules in ref guide

* provides

* insignificant edit

* docs - new pxf parquet write options

* add bytes

* line edit: the PXF -> PXF

* Line edit: which -> that

* docs - describe pxf jdbc profile QUOTE_COLUMNS option

* clarify per comment from ivan

* ALTER TABLE. Edits and fix example.

* ALTER TYPE. Add IF NOT EXISTS and edits

* ALTER VIEW. edits and add examples

* clusterdb. small edit

* CREATE EXTENSION. Edits mostly applied earlier.

* CREATE FUNCTION. Compatibility edit.

* CREATE RULE. Edits

* EXPLAIN. small edits.

* DROP RULE. minor edits

* GRANT. revise server USAGE privilege description.

* pg_restore. lots of edits. doesn't read PGDATABASE envvar.

* REASSIGN OWNED. edits. Remove qualification that it doesn't change the database ownership

* REASSIGN OWNED. edits. Remove qualification that it doesn't change the database ownership

* Remove extraneous char

* Changes from review

Co-authored-by: NJamie McAtamney <jmcatamney@pivotal.io>

This reverts commit d0839b23.

It's a Bourne Again script, Not a Bourne script.
Co-authored-by: NSoumyadeep Chakraborty <sochakraborty@pivotal.io>
Co-authored-by: NOliver Albertini <oalbertini@pivotal.io>
Co-authored-by: NAmil Khanzada <akhanzada@pivotal.io>
Co-authored-by: NKris Macoskey <kmacoskey@pivotal.io>

The logic hasn't actually changed much (we are still validating the the
postgres version and gpdb_src SHA match), but has become clearer and
easier to follow:
* avoids use of globals
* separates assert function into two to have clearer scope separation
Co-authored-by: NSoumyadeep Chakraborty <sochakraborty@pivotal.io>
Co-authored-by: NAmil Khanzada <akhanzada@pivotal.io>
Co-authored-by: NOliver Albertini <oalbertini@pivotal.io>
Co-authored-by: NKris Macoskey <kmacoskey@pivotal.io>
Co-authored-by: NBen Christel <bchristel@pivotal.io>

As a kill may be not taken effect at "the database is killed on hosts "
immediately, so we double check the cluster and stop database if it is
still running. But if kill takes effect after checking, stop_database
will raise an exception for gpstop will fail to stop an unstarted cluster.
To fix this flaky case, we stop database first and check the database
status later. It will raise an exception only when gpstop failed and
the cluster is still running.

For some areas of the ICW test framework -- isolation2 in particular --
the additional data written to stdout by gprecoverseg's progress
increased the load on the system significantly. (Some tests are
buffering stdout without bound, for instance.)  Additionally, the
updates were coming at ten times a second, which is an order of
magnitude more than the update interval we get from pg_basebackup
itself.

To help with this, we have have added a --no-progress flag that
suppresses the output of pg_basebackup.  We have also changed the
pg_basebackup progress update rate to once per second to minimize I/O.

The impacted regression/isolation2 tests utilizing gprecoverseg have
also been modified to use the --no-progress flag.
Co-authored-by: NJamie McAtamney <jmcatamney@pivotal.io>
Co-authored-by: NJacob Champion <pchampion@pivotal.io>

When -s is present, show pg_basebackup progress sequentially instead
of inplace. Useful when writing to a file, or if a tty does not support
escape sequences. Defaults to showing the progress inplace.

The gprecoverseg utility runs pg_basebackup in parallel on all segments that are
being recovered.  In this commit, we are logging the progress of each
pg_basebackup on its host and displaying them to the user of gprecoverseg.  The
progress files are deleted upons successful completion of gprecoverseg.

Unit tests have also been added.
Authored-by: NShoaib Lari <slari@pivotal.io>
Co-authored-by: NMark Sliva <msliva@pivotal.io>
Co-authored-by: NJacob Champion <pchampion@pivotal.io>
Co-authored-by: NEd Espino <edespino@pivotal.io>
Co-authored-by: NKalen Krempely <kkrempely@pivotal.io>

These files used to be installed to ${prefix}/bin/ directory, but as
they are not user documents we should not install them.
Reviewed-by: NBradford D. Boyle <bboyle@pivotal.io>
Reviewed-by: NBob Bao <bbao@pivotal.io>

The fly commands outputted by gen_pipeline needed to be updated for a
chance in the naming convention of one of the secrets files used.
Co-authored-by: NBradford Boyle <bboyle@pivotal.io>
Co-authored-by: NKris Macoskey <kmacoskey@pivotal.io>

Include gpaddmirrors in the distribution.
Co-authored-by: NKalen Krempely <kkrempely@pivotal.io>
Co-authored-by: NMark Sliva <msliva@pivotal.io>

Corresponding ORCA commit: "Fix crash during expression to DXL
translation in correlated NLJ"

We add gpmovemirrors tests and add them to the pipeline as well.
Using behave tags, some of the tests can run locally on a demo
cluster.  We also introduce Behave tag logic to choose which
tests to run.

<Co-Authored-By> Kalen Krempely <kkrempely@pivotal.io>
<Co-Authored-By> Mark Sliva <msliva@pivotal.io>

This seems to have broken when we updated gprecoverseg to work with WAL
rep.

We removed the registerMirrorsInCatalog method from the
cluster-wide utilities.
Co-authored-by: NShoaib Lari <slari@pivotal.io>

Because set_next_rotation_time() was being called once per call to
logfile_rotate(), the timestamp used to create the log file name was
being advanced multiple times per rotation. This led to the creation of
empty log files on disk that looked like they were from "the future".

This code badly needs to be refactored -- the initial intent, for the
text and csv implementations to be independent, is no longer the case --
but for now, we can hack around this problem by pulling the call to
set_next_rotation_time() up one level. This will only be called if *all*
logfiles are correctly rotated, to match the upstream logic (on failure,
we loop around to the beginning and try to open all log files again).
Co-authored-by: NJamie McAtamney <jmcatamney@pivotal.io>

Log rotations were happening well after the timestamp on the log file
(e.g. a per-minute rotation might actually take place thirty seconds
after it was supposed to).

After the 9.2 merge, we were incorrectly shadowing the `now` variable,
so the rotation delay only got bigger and bigger. Periodic logging (such
as from FTS) caused us to break out of the wait by luck, but otherwise
the syslogger would take longer and longer to wake up to perform a
rotation.
Co-authored-by: NJamie McAtamney <jmcatamney@pivotal.io>

tracker story here:

  https://www.pivotaltracker.com/story/show/162139588

  When we compile the gpdb, we observer a error in the stdout:

  Error Copying Documentation: /root/gpdb/gpAux/docs/release is not
  present

  We sent a email to confirm with gpdb dev about this issue. and got
  the reply is to remove it completely.
Authored-by: NBob Bao <bbao@pivotal.io>

For GP 6 beta, the release engineering team is removing the apr-util
package from the list of bundled dependencies. Users will be asked
to provide their own apr-util package, which can differ in the version
on each platform. So it is necessary to verify gpperfmon is workable
with the platform provided apr-util on each supported platform.

Originally, gpperfmon test is done in CLI test suite, only covers centos6.
This commit changed it to a dedicated suite in order to test multiple
platforms.
Note: on SLES12 does not need to install libapr-util1 to run gpmmon

With the creation of 6X_STABLE, the pipelines creating artifacts from
the master branch need to separate from 6X_STABLE in blob store
Authored-by: NKris Macoskey <kmacoskey@pivotal.io>

Gp_role is asserted to GP_ROLE_DISPATCH

In some scenarios ERROR "append-only table version -1 is invalid" is
being hit. Code inspection doens't reveal any clues why it can
happen. So, for now we thought of adding more info to figure out the
cause. Backtrace would definitely be helpful when the error hits.

Also, check the format version as soon as entry is read from
catalog. Plus also would be greatly helpful if can have corefile when
the problem happens so under the controlled environment with guc
evelate ERROR to PANIC. GUC
`debug_appendonly_print_verify_write_block` usage for same is not
ideal but wish to avoid adding special guc just for this case, hence
piggyback on this existing uinteresting debug guc for now.
Co-authored-by: NDaniel Gustafsson <dgustafsson@pivotal.io>

* dblink - xref to postgres 9.4 docs

* docs - add some using and writing your own FDW info

* add quote missing from mpp_execute master option

* foreign dash data, misc edits

ci: remove `-oss` from pattern used to parse version output

Remove period from assertRaisesRegexp expectation to not be conflated
with the regex metacharacter.
Authored-by: NKalen Krempely <kkrempely@pivotal.io>

* gpexpand
- remove `-D` flag and specify expansion schema is created in postgres database
- Add gpstate -x flag

* From review: consistent formatting for gpexpand and postgres

* Edit from review

* README mentions how gpupgrade and gpbackup both use the database "regression" that remains after an ICW run in the directory "regress".
Authored-by: NLarry Hamel <lhamel@pivotal.io>