Previously was using master.
Co-authored-by: NOliver Albertini <oalbertini@pivotal.io>
Co-authored-by: NBradford D. Boyle <bboyle@pivotal.io>

Postgres optimizes the storage of the ItemPointer in the Gin Posting
List. It only stores 11 bits for the offset number because heap tables
only have enough tuples per block to fit in 11 bits. However,
Greenplum append-only tables store 16 bits worth of offset numbers.

Initially we thought we'd need to modify decode_varbyte() but it turns
out that it is OK.  It handles 48 bits already.
Co-authored-by: NAlexandra Wang <lewang@pivotal.com>
Co-authored-by: NAshwin Agrawal <aagrawal@pivotal.io>

(cherry picked from commit 5e918bf6)

* docs - remove gpsyncmaster and add pg_basebackup to backend program list.

* Updates from review

Make "demo_cluster" the default tag (ie: no tag). That is, when a
scenario does not specify a behave tag it uses the demo cluster
environment to run. Use the concourse_cluster tag if a scenario requires
a remote cluster. Concourse runs the scenarios with no tag that use a
demo cluster by filtering for --tags=~concourse_cluster.
Authored-by: NKalen Krempely <kkrempely@pivotal.io>
(cherry picked from commit 40028cf4)

Regenerate pipeline from CLI_BEHAVE_TESTS refactor.

(cherry picked from commit 5dc3bfaa)

Renamed to CLI_BEHAVE_TESTS for clarity

(cherry picked from commit 87c4a52c)

Tag existing tests with @demo_cluster and @concourse_cluster
annotations. The @demo_cluster tag denotes the scenario can run
locally. The @concourse_cluster tag denotes the scenario that requires
a remote cluster.
Co-authored-by: NJamie McAtamney <jmcatamney@pivotal.io>
Authored-by: NKalen Krempely <kkrempely@pivotal.io>
(cherry picked from commit 735da14f)

Now that the previous commits have placed these tests in CLI_TESTS,
we can remove references to this symbol.

(cherry picked from commit 55add2c5)

Tag existing tests with @demo_cluster and @concourse_cluster
annotations. The @demo_cluster tag denotes the scenario can run
locally. The @concourse_cluster tag denotes the scenario that requires
a remote cluster.
Co-authored-by: NJamie McAtamney <jmcatamney@pivotal.io>
Authored-by: NKalen Krempely <kkrempely@pivotal.io>
(cherry picked from commit ffce7734)

Tag existing tests with @demo_cluster and @concourse_cluster
annotations. The @demo_cluster tag denotes the scenario can run
locally. The @concourse_cluster tag denotes the scenario that requires
a remote cluster.
Co-authored-by: NJamie McAtamney <jmcatamney@pivotal.io>
Authored-by: NKalen Krempely <kkrempely@pivotal.io>
(cherry picked from commit d67ecc3f)

Tag existing tests with @demo_cluster and @concourse_cluster
annotations. The @demo_cluster tag denotes the scenario can run
locally. The @concourse_cluster tag denotes the scenario that requires
a remote cluster.
Authored-by: NKalen Krempely <kkrempely@pivotal.io>
(cherry picked from commit e3457667)

Tag existing tests with @demo_cluster and @concourse_cluster
annotations. The @demo_cluster tag denotes the scenario can run
locally. The @concourse_cluster tag denotes the scenario that requires
a remote cluster.
Authored-by: NKalen Krempely <kkrempely@pivotal.io>
(cherry picked from commit a5130acb)

Now that the previous commits have placed these tests in CLI_TESTS,
we can remove references to this symbol.

(cherry picked from commit 8988618e)

Tag existing tests with @demo_cluster and @concourse_cluster
annotations. The @demo_cluster tag denotes the scenario can run
locally. The @concourse_cluster tag denotes the scenario that requires
a remote cluster.
Co-authored-by: NJamie McAtamney <jmcatamney@pivotal.io>
Authored-by: NKalen Krempely <kkrempely@pivotal.io>
(cherry picked from commit f676d2cc)

Tag existing tests with @demo_cluster and @concourse_cluster
annotations. The @demo_cluster tag denotes the scenario can run
locally. The @concourse_cluster tag denotes the scenario that requires
a remote cluster.
Co-authored-by: NJamie McAtamney <jmcatamney@pivotal.io>
Authored-by: NKalen Krempely <kkrempely@pivotal.io>
(cherry picked from commit 37a09d64)

Tag existing tests with @demo_cluster and @concourse_cluster
annotations. The @demo_cluster tag denotes the scenario can run
locally. The @concourse_cluster tag denotes the scenario that requires
a remote cluster.
Co-authored-by: NJamie McAtamney <jmcatamney@pivotal.io>
Authored-by: NKalen Krempely <kkrempely@pivotal.io>
(cherry picked from commit 8bb12f08)

Tag existing tests with @demo_cluster and @concourse_cluster
annotations. The @demo_cluster tag denotes the scenario can run
locally. The @concourse_cluster tag denotes the scenario that requires
a remote cluster.

(cherry picked from commit e7b5f462)

Tag existing tests with @demo_cluster and @concourse_cluster
annotations. The @demo_cluster tag denotes the scenario can run
locally. The @concourse_cluster tag denotes the scenario that requires
a remote cluster.
Authored-by: NKalen Krempely <kkrempely@pivotal.io>
(cherry picked from commit 642c4c8f)

Tag existing tests with @demo_cluster and @concourse_cluster
annotations. The @demo_cluster tag denotes the scenario can run
locally. The @concourse_cluster tag denotes the scenario that requires
a remote cluster.
Co-authored-by: NJamie McAtamney <jmcatamney@pivotal.io>
Authored-by: NKalen Krempely <kkrempely@pivotal.io>
(cherry picked from commit 53aad9c3)

Templatize gpmovemirrors into a single concourse job to run both
concourse_cluster and demo_cluster tags separate tasks.
Co-Authored-By: NKalen Krempely <kkrempely@pivotal.io>
(cherry picked from commit 0df65394)

Each gpexpand test case need a new environment.
Previous code uses pkill to make sure no postgres
process remaining and then uses gpstop to double
confirm.

However, pkill to terminate cluster may over-kill
some other postgres process on the hosts. What's
more, the interface `Given a working directory of
the test as '/data/gpdata/gpexpand'` will remove
the directory which also remove the $MASTER_DATA_DIRECTORY,
so the following `gpstop` is useless because
$MASTER_DATA_DIRECTORY is gone. Also, this introduce
race conditon.

This commit removes pkill and use gpstop interface
to build a clean environment for each test case. Also
it makes sure that `gpstop` is invoked at the very
beginning of each test case.

It tried to make sure gpfdist is ready by sleep 3 seconds before
queries, but that could not guarantee. This commit changes the sleep to
a retry until gpfdist is ready.

(cherry picked from commit f616837a)

We've seen frequent pg_rewind test failures due to promotion timeout. We
should increase the timeout. Also, add code to error out on timeout.
Co-authored-by: NNing Yu <nyu@pivotal.io>

Reviewed-by: Ashwin Agrawal

Previously the default value is 0, so normally log rotation happens each
day by default according to the other log related guc configurations.
That often makes a single log file rather large in real users'
environment.  Limiting the log file size so that people could easiler
open and search the log files.

1) pm_launch_walreceiver was introduced in b824fe8f
   to avoid a potential deadlock when cluster is down after
   updating the new role in gp_segment_configuration but before
   sending promote signal. It is buggy now since some changes.
   We need to put pm_launch_walreceiver in shared memory since
   it's used in different processes.

2) The following assertion is not always true in a real scenario if
   the segment is being promoted. We could simply stop the connection
   for the !am_mirror case.

   ProcessStartupPacket()
     if (am_ftshandler)
     {
        Assert(am_mirror);

Ning Yu and we had some discussions for these bugs. He contributed for
issue 1 also.
Co-authored-by: NHaozhou Wang <hawang@pivotal.io>

Related PR discussion:

kmacoskey:

Another piece of this can be removed with the same intention as the
other removal.

This line in the Makefile is related to a kfw snowflake dependency that
was being vendored into the Clients/Loaders package as brought in by
Ivy. Note the path, ext. It was dropped from Ivy by releng awhile ago,
2c6f4c9d#diff-271f398f0c07147024e7980737cddb7d . The cp here probably
should have been removed along with the change to Ivy, but it was
missed. It can be removed - but lets make sure the Clients team is
aware.

pf-qiu:

I think it's safe to do this now.

(cherry-picked from 22646c10)
Co-authored-by: NBradford D. Boyle <bboyle@pivotal.io>
Co-authored-by: NBen Christel <bchristel@pivotal.io>

(cherry-picked from c0f49617)
Co-authored-by: NSoumyadeep Chakraborty <sochakraborty@pivotal.io>
Co-authored-by: NBen Christel <bchristel@pivotal.io>

Since we have removed the Ivy dependency manager entirely from GPDB 6+,
we do not need this code anymore. It used to copy Ivy-managed libraries
into the Greenplum installation for packaging.

(cherry-picked from 1cc3dfc8)
Co-authored-by: NSoumyadeep Chakraborty <sochakraborty@pivotal.io>
Co-authored-by: NBen Christel <bchristel@pivotal.io>

* Remove unused IVYREPO_ variables for CentOS 6/7

Ivy is no longer used for CentOS 6/7.
Authored-by: NAmil Khanzada <akhanzada@pivotal.io>

* Remove unused IVYREPO_ variables for Coverity

Ivy is no longer used for CentOS 6/7 and will go away.
The coverity pipeline is also puased at this point in time, but we'll
clean this up now.

(cherry-picked from c4c99bb0)
Co-authored-by: NBradford D. Boyle <bboyle@pivotal.io>
Co-authored-by: NAmil Khanzada <akhanzada@pivotal.io>

INSERT needs to Lock the child table on QD, otherwise the following
dead lock scenario may happen if INSERT runs concurrently with VACUUM
drop phase on an AppendOnly partitioned table.

1. VACUUM on QD: acquired AccessExclusiveLock on the child table
2. INSERT on QE: acquired RowExclusiveLock on the child table
3. VACUUM on QE: waiting for AccessExclusiveLock on the child table
4. INSERT on QD: waiting for AccessShareLock at ExecutorEnd() on the
  child table, this is after QE sends back which child it inserted

Note that in step 2, INSERT only locks the child table on QE, it does
not lock the child table on QD in the previous code. This patch adds the
lock on QD as well to prevent the above dead lock.

Added `insert_while_vacuum_drop` and updated `partition_locking` to
reflect the changes

(cherry-picked from c7fce844)

For ORCA commit "Fix crash when merging group containing CTE consumer"
Authored-by: NChris Hajas <chajas@pivotal.io>

BuildPartitionNodeFromRoot will always use root relid no matter
what parameter is passed to it.

There is an optimization we might do: for some cases, like if user directly insert into
a child partition, then set estate->es_result_partitions to NULL to avoid dispatch it
since a huge partition table's partition info may cost much memory. We can spike this
later.

* Fix bin_gpdb_centos6_icw_green versioned_file to store artifacts in gpdb_branch_6X_STABLE
* Generate pipeline

Greenplum does not support this. These were originally removed in the
postgres 8.3 merge iteration but the 8.4 through 9.4 merge iterations
brought them back accidentally.

This commit is going directly to 6X_STABLE because the postgres 9.5
merge iteration will remove the *.po translator files for master.