Tag existing tests with @demo_cluster and @concourse_cluster
annotations. The @demo_cluster tag denotes the scenario can run
locally. The @concourse_cluster tag denotes the scenario that requires
a remote cluster.
Co-authored-by: NJamie McAtamney <jmcatamney@pivotal.io>
Authored-by: NKalen Krempely <kkrempely@pivotal.io>
(cherry picked from commit 53aad9c3)

Templatize gpmovemirrors into a single concourse job to run both
concourse_cluster and demo_cluster tags separate tasks.
Co-Authored-By: NKalen Krempely <kkrempely@pivotal.io>
(cherry picked from commit 0df65394)

Each gpexpand test case need a new environment.
Previous code uses pkill to make sure no postgres
process remaining and then uses gpstop to double
confirm.

However, pkill to terminate cluster may over-kill
some other postgres process on the hosts. What's
more, the interface `Given a working directory of
the test as '/data/gpdata/gpexpand'` will remove
the directory which also remove the $MASTER_DATA_DIRECTORY,
so the following `gpstop` is useless because
$MASTER_DATA_DIRECTORY is gone. Also, this introduce
race conditon.

This commit removes pkill and use gpstop interface
to build a clean environment for each test case. Also
it makes sure that `gpstop` is invoked at the very
beginning of each test case.

It tried to make sure gpfdist is ready by sleep 3 seconds before
queries, but that could not guarantee. This commit changes the sleep to
a retry until gpfdist is ready.

(cherry picked from commit f616837a)

We've seen frequent pg_rewind test failures due to promotion timeout. We
should increase the timeout. Also, add code to error out on timeout.
Co-authored-by: NNing Yu <nyu@pivotal.io>

Reviewed-by: Ashwin Agrawal

Previously the default value is 0, so normally log rotation happens each
day by default according to the other log related guc configurations.
That often makes a single log file rather large in real users'
environment.  Limiting the log file size so that people could easiler
open and search the log files.

1) pm_launch_walreceiver was introduced in b824fe8f
   to avoid a potential deadlock when cluster is down after
   updating the new role in gp_segment_configuration but before
   sending promote signal. It is buggy now since some changes.
   We need to put pm_launch_walreceiver in shared memory since
   it's used in different processes.

2) The following assertion is not always true in a real scenario if
   the segment is being promoted. We could simply stop the connection
   for the !am_mirror case.

   ProcessStartupPacket()
     if (am_ftshandler)
     {
        Assert(am_mirror);

Ning Yu and we had some discussions for these bugs. He contributed for
issue 1 also.
Co-authored-by: NHaozhou Wang <hawang@pivotal.io>

Related PR discussion:

kmacoskey:

Another piece of this can be removed with the same intention as the
other removal.

This line in the Makefile is related to a kfw snowflake dependency that
was being vendored into the Clients/Loaders package as brought in by
Ivy. Note the path, ext. It was dropped from Ivy by releng awhile ago,
2c6f4c9d#diff-271f398f0c07147024e7980737cddb7d . The cp here probably
should have been removed along with the change to Ivy, but it was
missed. It can be removed - but lets make sure the Clients team is
aware.

pf-qiu:

I think it's safe to do this now.

(cherry-picked from 22646c10)
Co-authored-by: NBradford D. Boyle <bboyle@pivotal.io>
Co-authored-by: NBen Christel <bchristel@pivotal.io>

(cherry-picked from c0f49617)
Co-authored-by: NSoumyadeep Chakraborty <sochakraborty@pivotal.io>
Co-authored-by: NBen Christel <bchristel@pivotal.io>

Since we have removed the Ivy dependency manager entirely from GPDB 6+,
we do not need this code anymore. It used to copy Ivy-managed libraries
into the Greenplum installation for packaging.

(cherry-picked from 1cc3dfc8)
Co-authored-by: NSoumyadeep Chakraborty <sochakraborty@pivotal.io>
Co-authored-by: NBen Christel <bchristel@pivotal.io>

* Remove unused IVYREPO_ variables for CentOS 6/7

Ivy is no longer used for CentOS 6/7.
Authored-by: NAmil Khanzada <akhanzada@pivotal.io>

* Remove unused IVYREPO_ variables for Coverity

Ivy is no longer used for CentOS 6/7 and will go away.
The coverity pipeline is also puased at this point in time, but we'll
clean this up now.

(cherry-picked from c4c99bb0)
Co-authored-by: NBradford D. Boyle <bboyle@pivotal.io>
Co-authored-by: NAmil Khanzada <akhanzada@pivotal.io>

INSERT needs to Lock the child table on QD, otherwise the following
dead lock scenario may happen if INSERT runs concurrently with VACUUM
drop phase on an AppendOnly partitioned table.

1. VACUUM on QD: acquired AccessExclusiveLock on the child table
2. INSERT on QE: acquired RowExclusiveLock on the child table
3. VACUUM on QE: waiting for AccessExclusiveLock on the child table
4. INSERT on QD: waiting for AccessShareLock at ExecutorEnd() on the
  child table, this is after QE sends back which child it inserted

Note that in step 2, INSERT only locks the child table on QE, it does
not lock the child table on QD in the previous code. This patch adds the
lock on QD as well to prevent the above dead lock.

Added `insert_while_vacuum_drop` and updated `partition_locking` to
reflect the changes

(cherry-picked from c7fce844)

For ORCA commit "Fix crash when merging group containing CTE consumer"
Authored-by: NChris Hajas <chajas@pivotal.io>

BuildPartitionNodeFromRoot will always use root relid no matter
what parameter is passed to it.

There is an optimization we might do: for some cases, like if user directly insert into
a child partition, then set estate->es_result_partitions to NULL to avoid dispatch it
since a huge partition table's partition info may cost much memory. We can spike this
later.

* Fix bin_gpdb_centos6_icw_green versioned_file to store artifacts in gpdb_branch_6X_STABLE
* Generate pipeline

Greenplum does not support this. These were originally removed in the
postgres 8.3 merge iteration but the 8.4 through 9.4 merge iterations
brought them back accidentally.

This commit is going directly to 6X_STABLE because the postgres 9.5
merge iteration will remove the *.po translator files for master.

For GP 6 beta, the release engineering team is removing the apr-util
package from the list of bundled dependencies. Users will be asked
to provide their own apr-util package, which can differ in the version
on each platform. So it is necessary to verify gpperfmon is workable
with the platform provided apr-util on each supported platform.

Originally, gpperfmon test is done in CLI test suite, only covers centos6.
This commit changed it to a dedicated suite in order to test multiple
platforms.
Note: on SLES12 does not need to install libapr-util1 to run gpmmon

* docs - gpexpand.status_detail external_writable field

* gpexpand.status tbl - updated col now timestamp without tz

* docs - organize contrib modules in ref guide

* provides

* insignificant edit

* ALTER TABLE. Edits and fix example.

* ALTER TYPE. Add IF NOT EXISTS and edits

* ALTER VIEW. edits and add examples

* clusterdb. small edit

* CREATE EXTENSION. Edits mostly applied earlier.

* CREATE FUNCTION. Compatibility edit.

* CREATE RULE. Edits

* EXPLAIN. small edits.

* DROP RULE. minor edits

* GRANT. revise server USAGE privilege description.

* pg_restore. lots of edits. doesn't read PGDATABASE envvar.

* REASSIGN OWNED. edits. Remove qualification that it doesn't change the database ownership

* REASSIGN OWNED. edits. Remove qualification that it doesn't change the database ownership

* Remove extraneous char

* Changes from review

We add gpmovemirrors tests and add them to the pipeline as well.
Using behave tags, some of the tests can run locally on a demo
cluster.  We also introduce Behave tag logic to choose which
tests to run.

<Co-Authored-By> Kalen Krempely <kkrempely@pivotal.io>
<Co-Authored-By> Mark Sliva <msliva@pivotal.io>
(cherry-picked from fdf8b611)

This seems to have broken when we updated gprecoverseg to work with WAL
rep.

(cherry-picked from e9388cdb)

We removed the registerMirrorsInCatalog method from the
cluster-wide utilities.
Co-authored-by: NShoaib Lari <slari@pivotal.io>
(cherry-picked from db9f0b14)

Because set_next_rotation_time() was being called once per call to
logfile_rotate(), the timestamp used to create the log file name was
being advanced multiple times per rotation. This led to the creation of
empty log files on disk that looked like they were from "the future".

This code badly needs to be refactored -- the initial intent, for the
text and csv implementations to be independent, is no longer the case --
but for now, we can hack around this problem by pulling the call to
set_next_rotation_time() up one level. This will only be called if *all*
logfiles are correctly rotated, to match the upstream logic (on failure,
we loop around to the beginning and try to open all log files again).
Co-authored-by: NJamie McAtamney <jmcatamney@pivotal.io>
(cherry picked from commit fc53bbf8)

Log rotations were happening well after the timestamp on the log file
(e.g. a per-minute rotation might actually take place thirty seconds
after it was supposed to).

After the 9.2 merge, we were incorrectly shadowing the `now` variable,
so the rotation delay only got bigger and bigger. Periodic logging (such
as from FTS) caused us to break out of the wait by luck, but otherwise
the syslogger would take longer and longer to wake up to perform a
rotation.
Co-authored-by: NJamie McAtamney <jmcatamney@pivotal.io>
(cherry picked from commit 14a794e8)

These files used to be installed to ${prefix}/bin/ directory, but as
they are not user documents we should not install them.
Reviewed-by: NBradford D. Boyle <bboyle@pivotal.io>
Reviewed-by: NBob Bao <bbao@pivotal.io>

(cherry picked from commit b176c4e6)

Include gpaddmirrors in the distribution.
Co-authored-by: NKalen Krempely <kkrempely@pivotal.io>

Corresponding ORCA commit: "Fix crash during expression to DXL
translation in correlated NLJ"

In some scenarios ERROR "append-only table version -1 is invalid" is
being hit. Code inspection doens't reveal any clues why it can
happen. So, for now we thought of adding more info to figure out the
cause. Backtrace would definitely be helpful when the error hits.

Also, check the format version as soon as entry is read from
catalog. Plus also would be greatly helpful if can have corefile when
the problem happens so under the controlled environment with guc
evelate ERROR to PANIC. GUC
`debug_appendonly_print_verify_write_block` usage for same is not
ideal but wish to avoid adding special guc just for this case, hence
piggyback on this existing uinteresting debug guc for now.
Co-authored-by: NDaniel Gustafsson <dgustafsson@pivotal.io>

It can handle defaults more correctly.
Co-authored-by: NOliver Albertini <oalbertini@pivotal.io>
Co-authored-by: NKris Macoskey <kmacoskey@pivotal.io>
Co-authored-by: NAmil Khanzada <akhanzada@pivotal.io>

Co-authored-by: NOliver Albertini <oalbertini@pivotal.io>
Co-authored-by: NKris Macoskey <kmacoskey@pivotal.io>
Co-authored-by: NAmil Khanzada <akhanzada@pivotal.io>

* dblink - xref to postgres 9.4 docs

* docs - add some using and writing your own FDW info

* add quote missing from mpp_execute master option

* foreign dash data, misc edits

ci: remove `-oss` from pattern used to parse version output

Remove period from assertRaisesRegexp expectation to not be conflated
with the regex metacharacter.
Authored-by: NKalen Krempely <kkrempely@pivotal.io>

* gpexpand
- remove `-D` flag and specify expansion schema is created in postgres database
- Add gpstate -x flag

* From review: consistent formatting for gpexpand and postgres

* Edit from review