* Source for the segment recovery flowchart

* Rename gprecovery-flow-chart.graffle to recovermatrix.graffle to match png filename

* docs - add info about pxf jdbc statement properties

* misc edits requested by david

Co-authored-by: NGoutam Tadi <gtadi@pivotal.io>
Co-authored-by: NAmil Khanzada <akhanzada@pivotal.io>

The relallvisible field in pg_class contains the number of heap pages
that are marked as all visible for any given heap relation. This value
is used by the planner to determine the cost of running an
IndexOnlyScan.

In theory, it is possible to use the visibility map of an AO table to
enable a index-only scan for AO tables. If there is no entry in the
pg_aovisimap table, then all tuples in the aoseg file up to the EOF will
be visible, and and index-only scan should not need to visit the AO
relation. It's not clear that enabling IndexOnlyScans will buy us
anything with AO relations, so it makes sense to just disable it for
now.

Until and unless we decide to enable index only scans of AO tables,
always set relallvisible to 0 for AO relations. This should prevent the
planner from choosing an index only scan for AO relations.

Like heap tables, AO tables have a visibility map, but the AO
visibilitymap serves a different purpose than the visibilitymap in heap
tables. In Heap tables, the visibility map is an optimization used to
determine which tuples can be returned directly from an index without
needing to consult with the heap table. Heap visibility maps do not
affect the visibility of tuples, but instead are used as a hint for
whether an IndexOnlyScan needs to visit the underlying relation. The AO
visibility map is used to track which tuples have been deleted from an
AO table, and thus their actual visibility to a table scan. AO relations
do not use a relation fork for a visibility map, and instead track the
visibility of tuples in an auxiliary table.
Co-authored-by: NAdam Berlin <aberlin@pivotal.io>

When determine waiting relation in global dead lock detector,
we should also check whether the waiter's lockmode conflict with
holder's holdmask.
Co-authored-by: NNing Yu <nyu@pivotal.io>
Reviewed-by: NZhenghua Lyu <zlv@pivotal.io>

The utilities reference page contained (unlinked) mentions of long
since deprecated utilities, and since they aren't shipping it's time
to remove them:

	gpsizecalc - removed in June 2010
	gpskew - removed in October 2009
	gprebuildsystem - removed in January 2010
	gpchecknet - removed in April 2010
	gpcheckos - removed in July 2016

Reviewed-by: Lisa Owen
Reviewed-by: David Yozie

gpdetective was removed in 4.3.5, so it seems about time to also
remove the documentation for it (which was unreachable due to the
app being marked deprecated).

Reviewed-by: Lisa Owen
Reviewed-by: David Yozie

There was a race condition where replay of a DROP TABLESPACE command on
a replication mirror could occur after the tests had already physically
removed the directories. It should be safe if the tablespace directories
continue to exist after the test is complete, so don't clean them up.

* docs - add pageinspect module landing page

* remove peer scope

* cannot inspect ao or external relations

Commit 0b6f15b8 removed the only
callsite of memtuple_get_nullp_len(), so remove the function too
to avoid a compiler warning (and reduce cruft).
Reviewed-by: NTang Pengzhou <ptang@pivotal.io>
Reviewed-by: NAshwin Agrawal <aagrawal@pivotal.io>

This moves the errorpath for EXTERNAL TABLE in LIKE INCLUDING in
under the context aware error handling. This makes the error when
creating an external table the same as when creating a foreign
table. As a bonus it avoids a few checks in case of error.

Reviewed-by: Jimmy Yih
Reviewed-by: Asim R P

INCLUDING STORAGE was introduced in PostgreSQL 9.0, but support for
setting the AO/AOCS/ENCODING storage options on the created table
was never supported as we merged upstream. This extends support for
copying over the table storage type as well as attribute encodings
when creating a table with INCLUDING STORAGE or ALL.

Reported-by: Cyrille Lintz
Reviewed-by: Jimmy Yih
Reviewed-by: Asim R P

The pageinspect code only knows how to handle heap relations, so make
sure to error out gracefully on AO/AOCS and EXTERNAL relations.
Reported-by: NLisa Owen <lowen@pivotal.io>
Reviewed-by: NAsim R P <apraveen@pivotal.io>

s/specifed/specified/

At present, `gpstop` check `pg_stat_activity` table before stop cluster in smart model. However, we need to ignore those background workers that are part of the database implementation itself. For this purpose, we maintain a whitelist using its `application_name` field.

The "combine" function for the int4 sum/avg aggregate functions is
backported to Greenplum 6.0 in commit 313cef6e (from
postgres/postgres@11c8669c0cc) but we made the inevitable omission of
setting the "strictness" of int4_avg_combine to false. This by itself is
harmless as the actual function body *does* guard against NULL input,
but it also prohibits a whole host of optimizations when the executor
and planner can detect NULL input early on and short-circuit the
execution. Oops.

This patch flips the `proisstrict` flag back to true for
int4_avg_combine.

Backpatch to 6X_STABLE.

optimizer_enable_dml is set to true by default. When set to false, ORCA will
fall back to planner for all DML queries.

* docs - add to pxf upgrade procedure (v5.0.1 to v5.3.2)

* remove extraneous to

When referring to the product name and not a database instance running
Greenplum, the capitalization should be "Greenplum Database".

The --load-extension option was misspelt as load_extension.  Resource group
tests are difficult to run locally, neither are they run in PR pipeline.  That
makes it difficult to catch such errors.

This test has failed at least once due to the terminate query being executed
before the to be terminated 'create table' statement.  This was evident from
master logs.  The commit makes it more reliable by injecting a fault and
waiting for the fault to be trigggered before executing pg_terminate_backend().
As a side benefit, we no longer need to create any additional table.

The extension is created by the test harness, after creating the regression /
isolation2test databases.  Tests should directly start using the extension and
not attempt to create it.  In addition to simplifying tests a little bit, this
change avoids an error (duplicate key value violates unique constraint on
pg_extension) when two or more tests execute create extension command
concurrently.  The error is not a problem in practice, it expected because of
the way create extension DDL works.  It is, however, unacceptable for
regress-style tests that expect a deterministic response to each SQL command.

In pg_get_expr(), after getting the relname, if the table that the relid tells
is dropped, an error will raise later when opening the relation to get column
names.

pg_get_expr() is used by GPDB add-on view 'pg_partitions' which is widely used
by regression tests for partition tables. Lots of parallel test cases issue view
pg_partitions and drop partition tables concurrently, so those cases are very
flaky. Serialize test cases will cost more testing time and be fragile, so GPDB
holds a AccessShareLock here to make tests stable.

Assume user1 has the privilege to database db1 and user2 has not, when
user1 try to create a schema in db1 and authorize it to user2, a
permission denied error is reported in QE. The RCA is QD set current
user to user2 before dispatching the query to QEs, so QE will also
set current user to user2, however, user2 has no provilege to create
schema in database db1.

To fix this, we delay setting the current user to user2 until the query
is dispatched to QEs.

GPDB used to allow command like "START_REPLICATION %X/%X [SYNC]" to
start a replication, user can specify SYNC option to skip waiting
synchronous in replications. Now start replication command is made
similar to upstream, the SYNC option is not supported, however, the
internal flag "synchronous" is still used and always be false which
make master and standby never synchronized.

Some of the task files were used by jobs in the orca pipeline, but those
jobs have been removed so the files are not being used anymore.
Co-authored-by: NKaren Huddleston <khuddleston@pivotal.io>
Co-authored-by: NDavid Sharp <dsharp@pivotal.io>

Ensure that error messages in src/backend/parser follow the upstream
guidelines for formatting, styling and content.

 * Start hints and details with uppercase and end with period
 * Start messages with lowercase with no ending period
 * Avoid breaking messages in code to make grepping easier

This also cleans up the worst whitespace offences around error
messages.
Reviewed-by: NTang Pengzhou <ptang@pivotal.io>

Initializing a struct with zeroes using {0} is legal in C99, but cause
a warning in Clang when the first member of the struct isn't a scalar
variable:

gpfdist.c:1299:44: warning: suggest braces around initialization of subobject
                   [-Wmissing-braces]
        struct fstream_filename_and_offset fos = {0};
                                                  ^
                                                  {}
This is a bug in Clang, but we also don't want to turn off this class
of warnings due to a this trivial false positive, as that might hide
actual warnings somewhere. Since the struct in question is guaranteed
to be set before being read, we can avoid the explicit initialization.

The value set in default_pos was never used, so remove the variable
entirely. The consumers were most likely removed in a refactoring at
some point.
Reviewed-by: NJimmy Yih <jyih@pivotal.io>

Allocating memory manually with malloc() requires checking that the
allocation was granted before dereferencing the pointer. To fix use
pg_malloc() which is guaranteed to return a valid pointer or error
out. Also update the reclaim to use a matching pg_free() call.
Reviewed-by: NJimmy Yih <jyih@pivotal.io>