Reviewed-by: NDavid Yozie <dyozie@pivotal.io>

Since Greenplum is now based on PostgreSQL 9.4, update links to the
upstream documentation to refer to the 9.4 docset rather than older
versions. Also fix a few mentions of 8.3(.23) in the text along with
some spacing.
Reviewed-by: NDavid Yozie <dyozie@pivotal.io>

The search.cpan.org site has been discontinued and replaced with
metacpan.org, fix by replacing all links in the documentation.
There is an additional link in the testcode which will be fixed
as we merge with upstream.
Reviewed-by: NDavid Yozie <dyozie@pivotal.io>

In all but a few places we referred to the upstream documentation as
"PostgreSQL documentation", this fixes the few places where we had a
different spelling.
Reviewed-by: NDavid Yozie <dyozie@pivotal.io>

* docs - discuss pxf column projection

* add note about filter serialization

* column projection is always enabled

* edits requested by david

Related PR discussion:

kmacoskey:

Another piece of this can be removed with the same intention as the
other removal.

This line in the Makefile is related to a kfw snowflake dependency that
was being vendored into the Clients/Loaders package as brought in by
Ivy. Note the path, ext. It was dropped from Ivy by releng awhile ago,
2c6f4c9d#diff-271f398f0c07147024e7980737cddb7d . The cp here probably
should have been removed along with the change to Ivy, but it was
missed. It can be removed - but lets make sure the Clients team is
aware.

pf-qiu:

I think it's safe to do this now.
Co-authored-by: NBradford D. Boyle <bboyle@pivotal.io>
Co-authored-by: NBen Christel <bchristel@pivotal.io>

Co-authored-by: NSoumyadeep Chakraborty <sochakraborty@pivotal.io>
Co-authored-by: NBen Christel <bchristel@pivotal.io>

Since we have removed the Ivy dependency manager entirely from GPDB 6+,
we do not need this code anymore. It used to copy Ivy-managed libraries
into the Greenplum installation for packaging.
Co-authored-by: NSoumyadeep Chakraborty <sochakraborty@pivotal.io>
Co-authored-by: NBen Christel <bchristel@pivotal.io>

INSERT needs to Lock the child table on QD, otherwise the following
dead lock scenario may happen if INSERT runs concurrently with VACUUM
drop phase on an AppendOnly partitioned table.

1. VACUUM on QD: acquired AccessExclusiveLock on the child table
2. INSERT on QE: acquired RowExclusiveLock on the child table
3. VACUUM on QE: waiting for AccessExclusiveLock on the child table
4. INSERT on QD: waiting for AccessShareLock at ExecutorEnd() on the
  child table, this is after QE sends back which child it inserted

Note that in step 2, INSERT only locks the child table on QE, it does
not lock the child table on QD in the previous code. This patch adds the
lock on QD as well to prevent the above dead lock.

Added `insert_while_vacuum_drop` and updated `partition_locking` to
reflect the changes

A warning message is displayed to a user that attempts to perform
`pg_create_physical_replication_slot()` on any individual segment,
including the master segment so that they are aware that this command
is not made to be distributed to the entire cluster.

* Remove unused IVYREPO_ variables for CentOS 6/7

Ivy is no longer used for CentOS 6/7.
Authored-by: NAmil Khanzada <akhanzada@pivotal.io>

* Remove unused IVYREPO_ variables for Coverity

Ivy is no longer used for CentOS 6/7 and will go away.
The coverity pipeline is also puased at this point in time, but we'll
clean this up now.
Co-authored-by: NBradford D. Boyle <bboyle@pivotal.io>
Co-authored-by: NAmil Khanzada <akhanzada@pivotal.io>

* docs - gpexpand.status_detail external_writable field

* gpexpand.status tbl - updated col now timestamp without tz

* docs - organize contrib modules in ref guide

* provides

* insignificant edit

* docs - new pxf parquet write options

* add bytes

* line edit: the PXF -> PXF

* Line edit: which -> that

* docs - describe pxf jdbc profile QUOTE_COLUMNS option

* clarify per comment from ivan

* ALTER TABLE. Edits and fix example.

* ALTER TYPE. Add IF NOT EXISTS and edits

* ALTER VIEW. edits and add examples

* clusterdb. small edit

* CREATE EXTENSION. Edits mostly applied earlier.

* CREATE FUNCTION. Compatibility edit.

* CREATE RULE. Edits

* EXPLAIN. small edits.

* DROP RULE. minor edits

* GRANT. revise server USAGE privilege description.

* pg_restore. lots of edits. doesn't read PGDATABASE envvar.

* REASSIGN OWNED. edits. Remove qualification that it doesn't change the database ownership

* REASSIGN OWNED. edits. Remove qualification that it doesn't change the database ownership

* Remove extraneous char

* Changes from review

Co-authored-by: NJamie McAtamney <jmcatamney@pivotal.io>

This reverts commit d0839b23.

It's a Bourne Again script, Not a Bourne script.
Co-authored-by: NSoumyadeep Chakraborty <sochakraborty@pivotal.io>
Co-authored-by: NOliver Albertini <oalbertini@pivotal.io>
Co-authored-by: NAmil Khanzada <akhanzada@pivotal.io>
Co-authored-by: NKris Macoskey <kmacoskey@pivotal.io>

The logic hasn't actually changed much (we are still validating the the
postgres version and gpdb_src SHA match), but has become clearer and
easier to follow:
* avoids use of globals
* separates assert function into two to have clearer scope separation
Co-authored-by: NSoumyadeep Chakraborty <sochakraborty@pivotal.io>
Co-authored-by: NAmil Khanzada <akhanzada@pivotal.io>
Co-authored-by: NOliver Albertini <oalbertini@pivotal.io>
Co-authored-by: NKris Macoskey <kmacoskey@pivotal.io>
Co-authored-by: NBen Christel <bchristel@pivotal.io>

As a kill may be not taken effect at "the database is killed on hosts "
immediately, so we double check the cluster and stop database if it is
still running. But if kill takes effect after checking, stop_database
will raise an exception for gpstop will fail to stop an unstarted cluster.
To fix this flaky case, we stop database first and check the database
status later. It will raise an exception only when gpstop failed and
the cluster is still running.

For some areas of the ICW test framework -- isolation2 in particular --
the additional data written to stdout by gprecoverseg's progress
increased the load on the system significantly. (Some tests are
buffering stdout without bound, for instance.)  Additionally, the
updates were coming at ten times a second, which is an order of
magnitude more than the update interval we get from pg_basebackup
itself.

To help with this, we have have added a --no-progress flag that
suppresses the output of pg_basebackup.  We have also changed the
pg_basebackup progress update rate to once per second to minimize I/O.

The impacted regression/isolation2 tests utilizing gprecoverseg have
also been modified to use the --no-progress flag.
Co-authored-by: NJamie McAtamney <jmcatamney@pivotal.io>
Co-authored-by: NJacob Champion <pchampion@pivotal.io>

When -s is present, show pg_basebackup progress sequentially instead
of inplace. Useful when writing to a file, or if a tty does not support
escape sequences. Defaults to showing the progress inplace.

The gprecoverseg utility runs pg_basebackup in parallel on all segments that are
being recovered.  In this commit, we are logging the progress of each
pg_basebackup on its host and displaying them to the user of gprecoverseg.  The
progress files are deleted upons successful completion of gprecoverseg.

Unit tests have also been added.
Authored-by: NShoaib Lari <slari@pivotal.io>
Co-authored-by: NMark Sliva <msliva@pivotal.io>
Co-authored-by: NJacob Champion <pchampion@pivotal.io>
Co-authored-by: NEd Espino <edespino@pivotal.io>
Co-authored-by: NKalen Krempely <kkrempely@pivotal.io>

These files used to be installed to ${prefix}/bin/ directory, but as
they are not user documents we should not install them.
Reviewed-by: NBradford D. Boyle <bboyle@pivotal.io>
Reviewed-by: NBob Bao <bbao@pivotal.io>

The fly commands outputted by gen_pipeline needed to be updated for a
chance in the naming convention of one of the secrets files used.
Co-authored-by: NBradford Boyle <bboyle@pivotal.io>
Co-authored-by: NKris Macoskey <kmacoskey@pivotal.io>

Include gpaddmirrors in the distribution.
Co-authored-by: NKalen Krempely <kkrempely@pivotal.io>
Co-authored-by: NMark Sliva <msliva@pivotal.io>

Corresponding ORCA commit: "Fix crash during expression to DXL
translation in correlated NLJ"

We add gpmovemirrors tests and add them to the pipeline as well.
Using behave tags, some of the tests can run locally on a demo
cluster.  We also introduce Behave tag logic to choose which
tests to run.

<Co-Authored-By> Kalen Krempely <kkrempely@pivotal.io>
<Co-Authored-By> Mark Sliva <msliva@pivotal.io>

This seems to have broken when we updated gprecoverseg to work with WAL
rep.

We removed the registerMirrorsInCatalog method from the
cluster-wide utilities.
Co-authored-by: NShoaib Lari <slari@pivotal.io>

Because set_next_rotation_time() was being called once per call to
logfile_rotate(), the timestamp used to create the log file name was
being advanced multiple times per rotation. This led to the creation of
empty log files on disk that looked like they were from "the future".

This code badly needs to be refactored -- the initial intent, for the
text and csv implementations to be independent, is no longer the case --
but for now, we can hack around this problem by pulling the call to
set_next_rotation_time() up one level. This will only be called if *all*
logfiles are correctly rotated, to match the upstream logic (on failure,
we loop around to the beginning and try to open all log files again).
Co-authored-by: NJamie McAtamney <jmcatamney@pivotal.io>

Log rotations were happening well after the timestamp on the log file
(e.g. a per-minute rotation might actually take place thirty seconds
after it was supposed to).

After the 9.2 merge, we were incorrectly shadowing the `now` variable,
so the rotation delay only got bigger and bigger. Periodic logging (such
as from FTS) caused us to break out of the wait by luck, but otherwise
the syslogger would take longer and longer to wake up to perform a
rotation.
Co-authored-by: NJamie McAtamney <jmcatamney@pivotal.io>

tracker story here:

  https://www.pivotaltracker.com/story/show/162139588

  When we compile the gpdb, we observer a error in the stdout:

  Error Copying Documentation: /root/gpdb/gpAux/docs/release is not
  present

  We sent a email to confirm with gpdb dev about this issue. and got
  the reply is to remove it completely.
Authored-by: NBob Bao <bbao@pivotal.io>

For GP 6 beta, the release engineering team is removing the apr-util
package from the list of bundled dependencies. Users will be asked
to provide their own apr-util package, which can differ in the version
on each platform. So it is necessary to verify gpperfmon is workable
with the platform provided apr-util on each supported platform.

Originally, gpperfmon test is done in CLI test suite, only covers centos6.
This commit changed it to a dedicated suite in order to test multiple
platforms.
Note: on SLES12 does not need to install libapr-util1 to run gpmmon