This commit includes changes to the server to ensure that the utilities:
pg_rewind and pg_basebackup can be changed to support recovery in a
multi-segment-singular-host setting. We link pg_tblspc to a <dbid>
subdirectory of the tablespace, rather than to the path of the
tablespace directly, and we remove the <dbid> from the tablespace
version directory. At the same time, we have designed towards preserving
the response to pg_tablespace_location(<tablespace_oid>) such that it
does not return the dbid suffix. The design is such that it is the
responsibility of the utilities to append the dbid as and when required.

Before this commit:
  * the symlink to the tablespace directory looks like:
      pg_tblspc/spcoid/ -> /<tablespace_location>/
  * Under the symlink target, we would have the following:
    GPDB_MAJORVER_CATVER_db<dbid>/dboid/relfilenode
  * pg_tablespace_location(tsoid) returns: <tablespace_location>
e.g.
  * pg_tblspc/20981/ -> /data1/tsp1
  * Under /data1/tsp1: GPDB_6_201902061_db1/19849/192814
  * pg_tablespace_location(20981) returns: /data1/tsp1

After this commit:
  * the symlink to the tablespace directory looks like:
      pg_tblspc/spcoid/ -> /<tablespace_location>/<dbid>
  * Under the symlink target, we would have the following:
      GPDB_MAJORVER_CATVER/dboid/relfilenode
  * pg_tablespace_location(tsoid) returns: <tablespace_location>

e.g.
  * pg_tblspc/20981/ -> /data1/tsp1/1
  * Under /data1/tsp1/1: GPDB_6_201902061/19849/192814
  * pg_tablespace_location(20981) returns: /data1/tsp1

Motivation:

When tablespaces were aligned to upstream postgres, while removing
filespaces, we added the `tablespace_version_directory()` function to
supply each segment with a unique tablespace directory name. This was
accomplished by appending the 'magic' `GpIdentity.dbid` global variable
to the `GP_TABLESPACE_VERSION_DIRECTORY` in `tablespace_version_directory()`.

This is problematic for several reasons- but perhaps most severely is
the fact that in order to use any code in libpgcommon.so that references
this value, you need to first set the `GpIdentity.dbid` global,
otherwise any functions that deal with tablespaces will be broken in
unpredictable ways.

An example is pg_rewind- where `GetRelationPath()` will not return a valid
relation unless you repeatedly toggle the `GpIdentity.dbid` between the
value of the source or target segment dependant on the context of which
relfiles are being examined.

This commit bumps the catalog version here we have made breaking changes
in the tablespace filesystem layout.
Co-authored-by: NAdam Berlin <aberlin@pivotal.io>
Co-authored-by: NTaylor Vesely <tvesely@pivotal.io>
Co-authored-by: NSoumyadeep Chakraborty <sochakraborty@pivotal.io>

Co-authored-by: NDavid Sharp <dsharp@pivotal.io>
Co-authored-by: NGoutam Tadi <gtadi@pivotal.io>

* docs - address lock exhaustion shared mem error msg o

* capitalize Out in title

* docs - enhance pxf jdbc partitioning content

* add missing comma

* simplify some content

There was a concern that an exception during GetNonHistoricCatalogSnapshot would be problematic after setting the global variable and not resetting it back to its original value. This patch threads the desired distributed transaction context into GetNonHistoricCatalogSnapshot without modifying global state.

No longer rely on a global variable to determine the distributed snapshot context.

A motion hazard is a deadlock between motions, a classic motion hazard
in a join executor is formed by its inner and outer motions, it can be
prevented by prefetching the inner plan, refer to motion_sanity_check()
for details.

A similar motion hazard can be formed by the outer motion and the join
qual motion.  A join executor fetches a outer tuple, filters it with the
join qual, then repeat the process on all the outer tuples.  When there
are motions in both outer plan and the join qual then below state is
possible:

0. processes A and B belong to the join slice, process C belongs to the
   outer slice, process D belongs to the JoinQual slice;
1. A has read the first outer tuple and is fetching tuples from D;
2. D is waiting for ACK from B;
3. B is fetching the first outer tuple from C;
4. C is waiting for ACK from A;

So a deadlock is formed A->D->B->C->A.  We can prevent it also by
prefetching the join qual.
Reviewed-by: NJesse Zhang <jzhang@pivotal.io>
Reviewed-by: NGang Xiong <gxiong@pivotal.io>
Reviewed-by: NZhenghua Lyu <zlv@pivotal.io>

Had set to 1 for a test pipeline and forgot to put it back.
Co-authored-by: NDavid Sharp <dsharp@pivotal.io>
Co-authored-by: NJason Vigil <jvigil@pivotal.io>

This will move us towards dropping Ubuntu 16.04 in preparation for
adding an Ubuntu 18.04 build, as well as removing remaining uses of
Conan.

5X_STABLE and master have diverged enough that this pipeline can't build
5X_STABLE PRs. Add "base_branch" filter (newly added to the
github-pr-resource) to build only PRs against master.
Authored-by: NDavid Sharp <dsharp@pivotal.io>

Fix auto_explain init file for #7195

This patch includes

1. fix init_file
2. use float number for memory usage
3. Update the last SQL whose running time may be less than 1ms
4. Update test case: use newly created table other than pg_class
5. Add answer file for orca & enable nestloop

1. Add test cases for auto_explain
	* Add init_file to filter out random numbers
	* set CLIENT_MIN_MESSAGES to show `LOG` message to result file
2. Add auto_explain item in gpdb/GNUmakefile.in

Now that the enterprise version of GPDB is only provided via RPM, including
gpseginstall in the distribution would cause conflicts if users try to install
GPDB with RPMs and with gpseginstall on the same cluster.  While it could be
preserved for use by the OSS community, there are several standard tools for
copying GPDB files to segment hosts in a cluster, and recommendations for using
one or more of those tools will be included in the GPDB documentation.

In addition to removing gpseginstall itself, this commit removes references to
it in other utilities' documentation and removes code in gppylib that was only
called by gpseginstall.
Co-authored-by: NJamie McAtamney <jmcatamney@pivotal.io>
Co-authored-by: NKalen Krempely <kkrempely@pivotal.io>

heaptuple_form_to() zero's out the buffer, so this commit does same
for memtuple_form_to(). This is done to have zeros in padding areas
and hence help get good compression for AO tables.

This also helps to eliminate the flakiness seen in appendonly test for
compression ratio. Now given same input the compression ratio will be
same.

Discussion:
https://groups.google.com/a/greenplum.org/d/msg/gpdb-dev/1N5Qi-4WPis/EAIdwvy9CAAJ

* docs - clarify pxf filter partitioning support for hive

* clarify the hadoop cfg update content

* remove cluster start

* edits requested by david

* remove disable statement per shivram

Includes ICG changes for ORCA commit:
'Convert FULL OUTER JOIN to LEFT JOIN when possible'

Authored-by: NChris Hajas <chajas@pivotal.io>

Also add a few more tests for unions to ICG.

The functionality to send alerts via email (and snmp was removed in
commit 65822b80 but I missed removing
the autoconf check for the libcurl feature required. Since there are
no more consumers, remove the check and feature macro as well.
Reviewed-by: NJimmy Yih <jyih@pivotal.io>

The test should wait for the transactions to be in the right state
before promoting standby.  This commit adds a wait step to ensure just
that.  One of the ICW jobs in CI failed because the test promoted the
standby before the transactions were preprared on master.  This should
no longer happen now.

* docs - add pxf jdbc.connection.transactionIsolation server cfg property

* use read uncommitted in example; add external

Because users may find it confusing when gpconfig prints '-' or 'None'
when no GUC value is found in the file, this commit updates gpconfig to
output a clearer message.

Apply the out-of-band messaging to failure reports as well, and fix up
the output of --file-compare. Add tests for each modified
implementation.
Co-Authored-By: NJamie McAtamney <jmcatamney@pivotal.io>
Co-Authored-By: NMark Sliva <msliva@pivotal.io>
Co-Authored-By: NShoaib Lari <slari@pivotal.io>

Transactions that are in the middle of two phase commit are suspended on
master.  Standby is promoted while they are suspended.  Based on what XLOG
records are emitted by master, the standby is expected to perform DTM recovery
and complete the transactions upon promotion.

This feature enables tests to run SQL on standby after it is promoted.  Use
"-1S: <sql>" to run <sql> statement on standby.  It is assumed that the standby
is already promoted.

* docs - jdbc cfg supports connection- and session-level properties

* some edits requested by david

* reword jdbc server cfg opening paragraph

* clarify rejected session prop/value chars as requested by ivan

* docs - discuss pxf fragment metadata caching

* a large number of

Remove unused behave step:
"the user waits for "{process_name}" to finish running"

Remove behave step "the database is killed on hosts mdw,sdw1,sdw2" in
favor of "the database is not running. This shuts down the databse more
cleanly, and avoids any potential race conditons.

Remove "user kills a primary postmaster process" in favor of the more
generic equivalent step "user stops all {segment_type} processes" which
can take either a primary or mirror.

For clarity and accuracy rename "user kills all {segment_type}
processes" to "user stops all {segment_type} processes".

Cleanly shutdown segments using pg_ctl to avoid potential race
conditions by grep'ing for the pid via ps. This is a much simplier
approach to help with maintainability and extensibility.

S3KeyReaderTest.MTReadWithUnexpectedFetchDataAtSecondRound is a
flaky case, related to multithread timing.

The case setup S3KeyReader and try to download in parallel with 2
chunks(threads). When any of them encounters an error, all thread
will abort with the shared error.

The case assumed that the first created thread will call fetchData()
twice before another thread fetch with error. But if the first
thread is never scheduled to run, the second thread will call
fetchData() first and sets the shared error. Then the first thread
continues and will exit at the first call to fetchData(), reporting
shared error.

Modify the second call to fetchData() to be at most once.

Recursively create tablespace directories if they do not exist but we need them when re-redoing some tablespace related xlogs (e.g. database create with a tablespace) on mirror.

It is observed many time that gp_replica_check test fails because some mirror nodes
can not be brought up before testing recently. The related log looks like this:

2019-04-17 14:52:14.951 CST [23030] FATAL:  could not create directory "pg_tblspc/65546/PG_12_201904072/65547": No such file or directory
2019-04-17 14:52:14.951 CST [23030] CONTEXT:  WAL redo at 0/3011650 for Database/CREATE: copy dir 1663/1 to 65546/65547

That is because some mirror nodes can not be recovered after previous testing,
not due to gp_replica_check itself. The root cause is that tablespace recovery
related. Pengzhou Tang and Hao Wu digged that intially and kindly found a mini
repro as below.

run on shell:
  rm -rf /tmp/some_isolation2_pg_basebackup_tablespace
  mkdir -p /tmp/some_isolation2_pg_basebackup_tablespace

copy and run the below sql on psql client:
  drop tablespace if exists some_isolation2_pg_basebackup_tablespace;
  create tablespace some_isolation2_pg_basebackup_tablespace location '/tmp/some_isolation2_pg_basebackup_tablespace';
  \!gpstop -ra -M fast;
  drop database if exists some_database_with_tablespace;
  create database some_database_with_tablespace tablespace some_isolation2_pg_basebackup_tablespace;
  drop database some_database_with_tablespace;
  drop tablespace some_isolation2_pg_basebackup_tablespace;
  \!gpstop -ra -M immediate;

The root cause is on mirror after drop database & drop tablespace, 'immediate'
stop causes the pg_control file not up-to-date with latest redo start lsn (this
is allowed), when the node restarts, it re-redoes 'create database
some_database_with_tablespace tablespace
some_isolation2_pg_basebackup_tablespace' but the tablespace directories have
been deleted in previous redoing.

The 'could not create directory' error could happen on re-redoing create table
in a tablespace also. We've seen this case on the ci environment, but that is
because missing of a get_parent_directory() call in the 'create two parents'
code block in TablespaceCreateDbspace(). Changing it to a simpler call
pg_mkdir_p() instead.

Also it seems that the src_path could be missing also in dbase_redo() for the
example below. For example re-redoing at the alter step since tbs1 directory is
deleted in later 'drop tablespace tbs1'.
  alter database db1 set tablespace tbs2;
  drop tablespace tbs1;

There is discussion on upstream about this,
  https://www.postgresql.org/message-id/flat/CAEET0ZGx9AvioViLf7nbR_8tH9-%3D27DN5xWJ2P9-ROH16e4JUA%40mail.gmail.com

In this patch I recreate those directories to avoid this error. Other solutions
include ignoring the directory-not-existing error or forcing a flush when
redoing those kind of checkpoint xlogs which are added normally in drop
database, etc.

Let's revert or update the code change after the solution is finalized on
upstream.

* docs - pxf init/sync support to master standby

* edits requested by david

* edits requested by francisco and oliver

* pxf sync from master TO standby or seg host

* identify sync run on master in pxf sync option description

strncat unfortunately takes a misleading size argument which means
at most size from src. It is a bit of an antipattern in the string
family of functions and for that compilers will emit a warning if
it happens that the size argument matches the size of src, since that
is not what usually users of strncat want to do.

The usage of the function in the code was correct. However instead
of silencing the compiler, strncat was replaced with the dynamic buffer
family of operations that postgres provides for the frontend.

Also protect against an invalid read in case that the size of the
result is zero.
Reviewed-by: NAsim R P <apraveen@pivotal.io>
Reviewed-by: NDaniel Gustafsson <dgustafsson@pivotal.io>

The curl slist API properly handle NULLs so we can be less verbose
and skip the check before passing to the slist cleanup function.
Reviewed-by: NFrancisco Guerrero <aguerrero@pivotal.io>