The --load-extension option was misspelt as load_extension.  Resource group
tests are difficult to run locally, neither are they run in PR pipeline.  That
makes it difficult to catch such errors.

This test has failed at least once due to the terminate query being executed
before the to be terminated 'create table' statement.  This was evident from
master logs.  The commit makes it more reliable by injecting a fault and
waiting for the fault to be trigggered before executing pg_terminate_backend().
As a side benefit, we no longer need to create any additional table.

The extension is created by the test harness, after creating the regression /
isolation2test databases.  Tests should directly start using the extension and
not attempt to create it.  In addition to simplifying tests a little bit, this
change avoids an error (duplicate key value violates unique constraint on
pg_extension) when two or more tests execute create extension command
concurrently.  The error is not a problem in practice, it expected because of
the way create extension DDL works.  It is, however, unacceptable for
regress-style tests that expect a deterministic response to each SQL command.

In pg_get_expr(), after getting the relname, if the table that the relid tells
is dropped, an error will raise later when opening the relation to get column
names.

pg_get_expr() is used by GPDB add-on view 'pg_partitions' which is widely used
by regression tests for partition tables. Lots of parallel test cases issue view
pg_partitions and drop partition tables concurrently, so those cases are very
flaky. Serialize test cases will cost more testing time and be fragile, so GPDB
holds a AccessShareLock here to make tests stable.

Assume user1 has the privilege to database db1 and user2 has not, when
user1 try to create a schema in db1 and authorize it to user2, a
permission denied error is reported in QE. The RCA is QD set current
user to user2 before dispatching the query to QEs, so QE will also
set current user to user2, however, user2 has no provilege to create
schema in database db1.

To fix this, we delay setting the current user to user2 until the query
is dispatched to QEs.

GPDB used to allow command like "START_REPLICATION %X/%X [SYNC]" to
start a replication, user can specify SYNC option to skip waiting
synchronous in replications. Now start replication command is made
similar to upstream, the SYNC option is not supported, however, the
internal flag "synchronous" is still used and always be false which
make master and standby never synchronized.

Some of the task files were used by jobs in the orca pipeline, but those
jobs have been removed so the files are not being used anymore.
Co-authored-by: NKaren Huddleston <khuddleston@pivotal.io>
Co-authored-by: NDavid Sharp <dsharp@pivotal.io>

Ensure that error messages in src/backend/parser follow the upstream
guidelines for formatting, styling and content.

 * Start hints and details with uppercase and end with period
 * Start messages with lowercase with no ending period
 * Avoid breaking messages in code to make grepping easier

This also cleans up the worst whitespace offences around error
messages.
Reviewed-by: NTang Pengzhou <ptang@pivotal.io>

Initializing a struct with zeroes using {0} is legal in C99, but cause
a warning in Clang when the first member of the struct isn't a scalar
variable:

gpfdist.c:1299:44: warning: suggest braces around initialization of subobject
                   [-Wmissing-braces]
        struct fstream_filename_and_offset fos = {0};
                                                  ^
                                                  {}
This is a bug in Clang, but we also don't want to turn off this class
of warnings due to a this trivial false positive, as that might hide
actual warnings somewhere. Since the struct in question is guaranteed
to be set before being read, we can avoid the explicit initialization.

The value set in default_pos was never used, so remove the variable
entirely. The consumers were most likely removed in a refactoring at
some point.
Reviewed-by: NJimmy Yih <jyih@pivotal.io>

Allocating memory manually with malloc() requires checking that the
allocation was granted before dereferencing the pointer. To fix use
pg_malloc() which is guaranteed to return a valid pointer or error
out. Also update the reclaim to use a matching pg_free() call.
Reviewed-by: NJimmy Yih <jyih@pivotal.io>

- Copy and package dependency DLLs
- Add gpload.bat to make gpload runnable in cmd
- Fix pygresql build script
- Replace gpload version variable

Lockmode of Update|Delete or Select-for-update Statement
is controlled by: whether the table is AO or heap and the
GUC gp_enable_global_deadlock_detector.

The logic for lockmode is:
  1. Select-for-update always hold ExclusiveLock
  2. UPDATE|DELETE on AO tables always hold ExclusiveLock
  3. UPDATE|DELETE on heap tables hold ExclusiveLock when
     gp_enable_global_deadlock_detector is off, otherwise
     hold RowExclusiveLock

We hold locks in parser stage and Initplan before executing,
the lockmode should be the same at the two stages.

This commit fixes lockmode issues to make things correct.
Co-authored-by: NShujie Zhang <shzhang@pivotal.io>

* Docs: fixing several broken links

* Docs: fixing several broken links

* Adding info about GPORCA support notice

* add missing comma

Force the mirror to create a restartpoint, and as a side-effect replay
the DROP TABLESPACE DDL before removing the tablespace directory.
Co-authored-by: NSoumyadeep Chakraborty <sochakraborty@pivotal.io>

The test was intended to confirm that pg_basebackup wouldn't overwrite
and existing tablespace directory (without --force-overwrite)
Before: The command was failing because -1 is an invalid dbid
Now: The command is failing as originally intended.
Co-authored-by: NTaylor Vesely <tvesely@pivotal.io>

The server now removes the source dbid from the end of any tablespace symlink
target found in its response to BASE_BACKUP: This means that it removes it from
the tablespace header as well as any tar directory entries for a given
tablespace.  The pg_basebackup client now adds the target dbid at the end of
the symlink target returned from the server response in order to create the
correct symlink:
<target_datadir>/pg_tblspc/<tablespace_oid> -> <tablespace_location>/<target_db_id>

Note: All of the above is applicable to user-defined tablespaces.

Also, we renamed some of the ddl objects in the isolation2 tests for
tablespaces as they were too long. Also, reducing the length of the LOCATION
clause of the tablespace object helped us avoid the tar header limit of 100
characters for symlinks.
Co-authored-by: NSoumyadeep Chakraborty <sochakraborty@pivotal.io>
Co-authored-by: NTaylor Vesely <tvesely@pivotal.io>
Co-authored-by: NAdam Berlin <aberlin@pivotal.io>

This commit includes changes to the server to ensure that the utilities:
pg_rewind and pg_basebackup can be changed to support recovery in a
multi-segment-singular-host setting. We link pg_tblspc to a <dbid>
subdirectory of the tablespace, rather than to the path of the
tablespace directly, and we remove the <dbid> from the tablespace
version directory. At the same time, we have designed towards preserving
the response to pg_tablespace_location(<tablespace_oid>) such that it
does not return the dbid suffix. The design is such that it is the
responsibility of the utilities to append the dbid as and when required.

Before this commit:
  * the symlink to the tablespace directory looks like:
      pg_tblspc/spcoid/ -> /<tablespace_location>/
  * Under the symlink target, we would have the following:
    GPDB_MAJORVER_CATVER_db<dbid>/dboid/relfilenode
  * pg_tablespace_location(tsoid) returns: <tablespace_location>
e.g.
  * pg_tblspc/20981/ -> /data1/tsp1
  * Under /data1/tsp1: GPDB_6_201902061_db1/19849/192814
  * pg_tablespace_location(20981) returns: /data1/tsp1

After this commit:
  * the symlink to the tablespace directory looks like:
      pg_tblspc/spcoid/ -> /<tablespace_location>/<dbid>
  * Under the symlink target, we would have the following:
      GPDB_MAJORVER_CATVER/dboid/relfilenode
  * pg_tablespace_location(tsoid) returns: <tablespace_location>

e.g.
  * pg_tblspc/20981/ -> /data1/tsp1/1
  * Under /data1/tsp1/1: GPDB_6_201902061/19849/192814
  * pg_tablespace_location(20981) returns: /data1/tsp1

Motivation:

When tablespaces were aligned to upstream postgres, while removing
filespaces, we added the `tablespace_version_directory()` function to
supply each segment with a unique tablespace directory name. This was
accomplished by appending the 'magic' `GpIdentity.dbid` global variable
to the `GP_TABLESPACE_VERSION_DIRECTORY` in `tablespace_version_directory()`.

This is problematic for several reasons- but perhaps most severely is
the fact that in order to use any code in libpgcommon.so that references
this value, you need to first set the `GpIdentity.dbid` global,
otherwise any functions that deal with tablespaces will be broken in
unpredictable ways.

An example is pg_rewind- where `GetRelationPath()` will not return a valid
relation unless you repeatedly toggle the `GpIdentity.dbid` between the
value of the source or target segment dependant on the context of which
relfiles are being examined.

This commit bumps the catalog version here we have made breaking changes
in the tablespace filesystem layout.
Co-authored-by: NAdam Berlin <aberlin@pivotal.io>
Co-authored-by: NTaylor Vesely <tvesely@pivotal.io>
Co-authored-by: NSoumyadeep Chakraborty <sochakraborty@pivotal.io>

Co-authored-by: NDavid Sharp <dsharp@pivotal.io>
Co-authored-by: NGoutam Tadi <gtadi@pivotal.io>

* docs - address lock exhaustion shared mem error msg o

* capitalize Out in title

* docs - enhance pxf jdbc partitioning content

* add missing comma

* simplify some content

There was a concern that an exception during GetNonHistoricCatalogSnapshot would be problematic after setting the global variable and not resetting it back to its original value. This patch threads the desired distributed transaction context into GetNonHistoricCatalogSnapshot without modifying global state.

No longer rely on a global variable to determine the distributed snapshot context.

A motion hazard is a deadlock between motions, a classic motion hazard
in a join executor is formed by its inner and outer motions, it can be
prevented by prefetching the inner plan, refer to motion_sanity_check()
for details.

A similar motion hazard can be formed by the outer motion and the join
qual motion.  A join executor fetches a outer tuple, filters it with the
join qual, then repeat the process on all the outer tuples.  When there
are motions in both outer plan and the join qual then below state is
possible:

0. processes A and B belong to the join slice, process C belongs to the
   outer slice, process D belongs to the JoinQual slice;
1. A has read the first outer tuple and is fetching tuples from D;
2. D is waiting for ACK from B;
3. B is fetching the first outer tuple from C;
4. C is waiting for ACK from A;

So a deadlock is formed A->D->B->C->A.  We can prevent it also by
prefetching the join qual.
Reviewed-by: NJesse Zhang <jzhang@pivotal.io>
Reviewed-by: NGang Xiong <gxiong@pivotal.io>
Reviewed-by: NZhenghua Lyu <zlv@pivotal.io>

Had set to 1 for a test pipeline and forgot to put it back.
Co-authored-by: NDavid Sharp <dsharp@pivotal.io>
Co-authored-by: NJason Vigil <jvigil@pivotal.io>

This will move us towards dropping Ubuntu 16.04 in preparation for
adding an Ubuntu 18.04 build, as well as removing remaining uses of
Conan.

5X_STABLE and master have diverged enough that this pipeline can't build
5X_STABLE PRs. Add "base_branch" filter (newly added to the
github-pr-resource) to build only PRs against master.
Authored-by: NDavid Sharp <dsharp@pivotal.io>

Fix auto_explain init file for #7195

This patch includes

1. fix init_file
2. use float number for memory usage
3. Update the last SQL whose running time may be less than 1ms
4. Update test case: use newly created table other than pg_class
5. Add answer file for orca & enable nestloop

1. Add test cases for auto_explain
	* Add init_file to filter out random numbers
	* set CLIENT_MIN_MESSAGES to show `LOG` message to result file
2. Add auto_explain item in gpdb/GNUmakefile.in

Now that the enterprise version of GPDB is only provided via RPM, including
gpseginstall in the distribution would cause conflicts if users try to install
GPDB with RPMs and with gpseginstall on the same cluster.  While it could be
preserved for use by the OSS community, there are several standard tools for
copying GPDB files to segment hosts in a cluster, and recommendations for using
one or more of those tools will be included in the GPDB documentation.

In addition to removing gpseginstall itself, this commit removes references to
it in other utilities' documentation and removes code in gppylib that was only
called by gpseginstall.
Co-authored-by: NJamie McAtamney <jmcatamney@pivotal.io>
Co-authored-by: NKalen Krempely <kkrempely@pivotal.io>

heaptuple_form_to() zero's out the buffer, so this commit does same
for memtuple_form_to(). This is done to have zeros in padding areas
and hence help get good compression for AO tables.

This also helps to eliminate the flakiness seen in appendonly test for
compression ratio. Now given same input the compression ratio will be
same.

Discussion:
https://groups.google.com/a/greenplum.org/d/msg/gpdb-dev/1N5Qi-4WPis/EAIdwvy9CAAJ

* docs - clarify pxf filter partitioning support for hive

* clarify the hadoop cfg update content

* remove cluster start

* edits requested by david

* remove disable statement per shivram

Includes ICG changes for ORCA commit:
'Convert FULL OUTER JOIN to LEFT JOIN when possible'

Authored-by: NChris Hajas <chajas@pivotal.io>

Also add a few more tests for unions to ICG.