Co-authored-by: NDavid Sharp <dsharp@pivotal.io>
Co-authored-by: NGoutam Tadi <gtadi@pivotal.io>
(cherry picked from commit 8af57a90)

This will move us towards dropping Ubuntu 16.04 in preparation for
adding an Ubuntu 18.04 build, as well as removing remaining uses of
Conan.

5X_STABLE and master have diverged enough that this pipeline can't build
5X_STABLE PRs. Add "base_branch" filter (newly added to the
github-pr-resource) to build only PRs against master.
Authored-by: NDavid Sharp <dsharp@pivotal.io>
(cherry picked from commit 4f3d158e)
(cherry picked from commit 76e7b822)

There was a concern that an exception during GetNonHistoricCatalogSnapshot would be problematic after setting the global variable and not resetting it back to its original value. This patch threads the desired distributed transaction context into GetNonHistoricCatalogSnapshot without modifying global state.

No longer rely on a global variable to determine the distributed snapshot context.

* docs - address lock exhaustion shared mem error msg o

* capitalize Out in title

Recursively create tablespace directories if they do not exist but we need them when re-redoing some tablespace related xlogs (e.g. database create with a tablespace) on mirror.

It is observed many time that gp_replica_check test fails because some mirror nodes
can not be brought up before testing recently. The related log looks like this:

2019-04-17 14:52:14.951 CST [23030] FATAL:  could not create directory "pg_tblspc/65546/PG_12_201904072/65547": No such file or directory
2019-04-17 14:52:14.951 CST [23030] CONTEXT:  WAL redo at 0/3011650 for Database/CREATE: copy dir 1663/1 to 65546/65547

That is because some mirror nodes can not be recovered after previous testing,
not due to gp_replica_check itself. The root cause is that tablespace recovery
related. Pengzhou Tang and Hao Wu digged that intially and kindly found a mini
repro as below.

run on shell:
  rm -rf /tmp/some_isolation2_pg_basebackup_tablespace
  mkdir -p /tmp/some_isolation2_pg_basebackup_tablespace

copy and run the below sql on psql client:
  drop tablespace if exists some_isolation2_pg_basebackup_tablespace;
  create tablespace some_isolation2_pg_basebackup_tablespace location '/tmp/some_isolation2_pg_basebackup_tablespace';
  \!gpstop -ra -M fast;
  drop database if exists some_database_with_tablespace;
  create database some_database_with_tablespace tablespace some_isolation2_pg_basebackup_tablespace;
  drop database some_database_with_tablespace;
  drop tablespace some_isolation2_pg_basebackup_tablespace;
  \!gpstop -ra -M immediate;

The root cause is on mirror after drop database & drop tablespace, 'immediate'
stop causes the pg_control file not up-to-date with latest redo start lsn (this
is allowed), when the node restarts, it re-redoes 'create database
some_database_with_tablespace tablespace
some_isolation2_pg_basebackup_tablespace' but the tablespace directories have
been deleted in previous redoing.

The 'could not create directory' error could happen on re-redoing create table
in a tablespace also. We've seen this case on the ci environment, but that is
because missing of a get_parent_directory() call in the 'create two parents'
code block in TablespaceCreateDbspace(). Changing it to a simpler call
pg_mkdir_p() instead.

Also it seems that the src_path could be missing also in dbase_redo() for the
example below. For example re-redoing at the alter step since tbs1 directory is
deleted in later 'drop tablespace tbs1'.
  alter database db1 set tablespace tbs2;
  drop tablespace tbs1;

There is discussion on upstream about this,
  https://www.postgresql.org/message-id/flat/CAEET0ZGx9AvioViLf7nbR_8tH9-%3D27DN5xWJ2P9-ROH16e4JUA%40mail.gmail.com

In this patch I recreate those directories to avoid this error. Other solutions
include ignoring the directory-not-existing error or forcing a flush when
redoing those kind of checkpoint xlogs which are added normally in drop
database, etc.

Let's revert or update the code change after the solution is finalized on
upstream.

Now that the enterprise version of GPDB is only provided via RPM, including
gpseginstall in the distribution would cause conflicts if users try to install
GPDB with RPMs and with gpseginstall on the same cluster.  While it could be
preserved for use by the OSS community, there are several standard tools for
copying GPDB files to segment hosts in a cluster, and recommendations for using
one or more of those tools will be included in the GPDB documentation.

In addition to removing gpseginstall itself, this commit removes references to
it in other utilities' documentation and removes code in gppylib that was only
called by gpseginstall.
Co-authored-by: NJamie McAtamney <jmcatamney@pivotal.io>
Co-authored-by: NKalen Krempely <kkrempely@pivotal.io>
(cherry picked from commit 64014685)

CI reported some read failures just after the write action, it's
probably because the write action is not flushed yet by AWS.

Sleep and retry if AWS returns "NoSuchKey" error. Update the workflow
but not the test cases because users might have the same issue.
Co-authored-by: NPeifeng Qiu <pqiu@pivotal.io>

Includes ICG changes for ORCA commit:
'Convert FULL OUTER JOIN to LEFT JOIN when possible'

Authored-by: NChris Hajas <chajas@pivotal.io>

Also add a few more tests for unions to ICG.

(cherry picked from commit 96f34a8c)

Because users may find it confusing when gpconfig prints '-' or 'None'
when no GUC value is found in the file, this commit updates gpconfig to
output a clearer message.

Apply the out-of-band messaging to failure reports as well, and fix up
the output of --file-compare. Add tests for each modified
implementation.
Co-Authored-By: NJamie McAtamney <jmcatamney@pivotal.io>
Co-Authored-By: NMark Sliva <msliva@pivotal.io>
Co-Authored-By: NShoaib Lari <slari@pivotal.io>
(cherry picked from commit b8bd3577)

The test should wait for the transactions to be in the right state
before promoting standby.  This commit adds a wait step to ensure just
that.  One of the ICW jobs in CI failed because the test promoted the
standby before the transactions were preprared on master.  This should
no longer happen now.

Transactions that are in the middle of two phase commit are suspended on
master.  Standby is promoted while they are suspended.  Based on what XLOG
records are emitted by master, the standby is expected to perform DTM recovery
and complete the transactions upon promotion.

This feature enables tests to run SQL on standby after it is promoted.  Use
"-1S: <sql>" to run <sql> statement on standby.  It is assumed that the standby
is already promoted.

Skip gprecoverseg tablespace tests until tablespaces successfully work
with pg_rewind.
Co-authored-by: NMark Sliva <msliva@pivotal.io>
(cherry picked from commit aae20c86)

(cherry picked from commit bb532dec)

Co-authored-by: NJacob Champion <pchampion@pivotal.io>
(cherry picked from commit b81ee6cd)

Create the tablespace directory on all segments to support
running multi-host tablespace tests.
Co-authored-by: NJacob Champion <pchampion@pivotal.io>
(cherry picked from commit 83f48aac)

Remove unused behave step:
"the user waits for "{process_name}" to finish running"

(cherry picked from commit 3a3a4b70036925fb0ef11354b76fb20461afe931)

Remove behave step "the database is killed on hosts mdw,sdw1,sdw2" in
favor of "the database is not running. This shuts down the databse more
cleanly, and avoids any potential race conditons.

(cherry picked from commit 8079bd2be75f0624212d7c4907ea16e8c5a5d199)

Remove "user kills a primary postmaster process" in favor of the more
generic equivalent step "user stops all {segment_type} processes" which
can take either a primary or mirror.

For clarity and accuracy rename "user kills all {segment_type}
processes" to "user stops all {segment_type} processes".

(cherry picked from commit 8eb9dffb726a9218166af9ba06a91efb7c492826)

Cleanly shutdown segments using pg_ctl to avoid potential race
conditions by grep'ing for the pid via ps. This is a much simplier
approach to help with maintainability and extensibility.

(cherry picked from commit bd9bbaf837be3d4f7334822d4d14469702725c55)

Previously we error out in gp_replica_check test if the cluster is not in
synced, which is usually due to bugs revealed by previous tests, instead of
gp_replica_check itself.

Print more detailed message so that people are not confused with the status and
this failure reason of the gp_replica_check test.

* Updates to GUCs from Cyrille's reviews

* Revise description of temp_buffers to match 6.0 behavior

* Fix a couple typos

* For effective_cache_size GUC, show default in blocks and size.

DTX_STATE_ACTIVE_NOT_DISTRIBUTED indicates the transaction is started on QD and
two-phase commit is not required yet, in this state, distributed transaction ID
is also generated. If this transaction is not a two-phase transaction, e.g. implicit
select-only transaction, it will also need to acquire the ProcArrayLock to clear
the distributed transaction ID, and that leads to unnecessary lock contention.

We found that if gp_segment_configuration is locked, then it will
fail by triggering FTS. We got the stack below
	#2  0x0000000000a6bb29 in ExceptionalCondition at assert.c:66
	#3  0x0000000000aac19a in enable_timeout timeout.c:143
	#4  0x0000000000aacb6c in enable_timeout_after timeout.c:473
	#5  0x00000000008e86ef in ProcSleep at proc.c:1300
	#6  0x00000000008deb70 in WaitOnLock at lock.c:1894
	#7  0x00000000008e019e in LockAcquireExtended at lock.c:1205
	#8  0x00000000008dd2d8 in LockRelationOid at lmgr.c:102
	#9  0x000000000051c928 in heap_open at heapam.c:1083
	#10 0x0000000000b7feaf in getCdbComponentInfo at cdbutil.c:173
	#11 0x0000000000b81365 in cdbcomponent_getCdbComponents at cdbutil.c:606
	#12 0x00000000007603e1 in ftsMain at fts.c:351
	#13 0x0000000000760715 in ftsprobe_start at fts.c:121
	#14 0x00000000004cc7b0 in ServerLoop ()
	#15 0x00000000008769bf in PostmasterMain at postmaster.c:1531
	#16 0x000000000079098b in main ()
So it is that FTS hasn't initialized timeout. Any process that
wants to use timeout must call initilization first. This is the
root cause gpexpand job fails on master pipeline in build 71 and 79.
We added this initialization in FTS and GDD.

* Docs - update pg_class and pg_index catalog table references

* dyozie review comments

concourse/scripts/gppkg_behave.sh, which installs an identical copy of
GPHOME into a new location, existed for exactly one test. Modify that
test to perform its own copy-install.

(cherry picked from commit e6b40a71)

gpexpand requires centos6, ICW, and CLI jobs to be enabled, but it was
being included in the groups when only a subset of those requirements
was met, so fly refused the pipeline. With this change, centos6/CLI
pipelines should work again.

(cherry picked from commit 52a21767)

  This is different from the gpdb-master,
  The gpdb-master pipeline save the RC artifacts
  to the server/published/master/
Authored-by: NBob Bao <bbao@pivotal.io>