Previously, gpconfig used bash and Perl to manipulate the master and
segment postgresql.conf files.  This made it hard to understand the
transformation on string valued gucs.
Co-authored-by: NJamie McAtamney <jmcatamney@pivotal.io>
Co-authored-by: NMark Sliva <msliva@pivotal.io>
Co-authored-by: NKalen Krempely <kkrempely@pivotal.io>

* docs - add module landing page for gp_sparse_vector

* edits requested by david

* docs - revisit hstore module doc contents

* edits requested by david

TAR has a limitation that symlink targets must be shorter than 100
characters. Base backup will truncate the symlink target path beyond
100 characters to match the TAR spec.

This change introduces a warning to the user while creating a
tablespace with some hints for how to avoid the problem.

* Add tests around pg_rewind's behavior with tablespaces.
* pg_rewind now lops off the source dbid from the symlink target before
populating the filemap.
* pg_rewind now appends the target dbid to the symlink target when
creating it and also creates the symlink target directory. Both of these
actions are performed as a separate action: `create_tablespace_layout`.
We distinguish tablespace symlinks with the `is_gp_tablespace` flag.
* pg_rewind now fetches the target dbid by invoking the postgres binary.
* Add test for pg_rewind with tablespaces which tests for objects
created after promotion of the replica. We introduced a new hook in the
pg_rewind suite: `before_master_restart_after_rewind`, which executes
right after pg_rewind completes and the master is restarted.
* Add test for pg_rewind with tablepace objects created before standby
promotion.
* Add pg_rewind test for objects removed after standby promotion
* Enable gprecoverseg tests around tablespaces that were waiting on
these changes.
Co-authored-by: NTaylor Vesely <tvesely@pivotal.io>
Co-authored-by: NAdam Berlin <aberlin@pivotal.io>
Co-authored-by: NTaylor Vesely <tvesely@pivotal.io>

Align contrib/Makefile with upstream by adding the missing modules
in the SUBDIRS targetlist. The set of modules built when issueing
a root level "make install" is still based on GNUMakefile in the
root, but this allows for "make -C contrib/" and have it do what
one could reasonably expect - all modules that compiles are also
compiled.

This removes a FIXME added when merging 9.2

It's recommended to compare identity against None with 'is' or 'is not'
rather than by equality with the '==' or '!=' operators, even if custom
comparison operators aren't implemented. 'is' conveys meaning, and is a
hair faster (which in this case is irrelevant).

Reviewed-by: Abhijit Subramanya

* docs - add pxf server cfg topic, rework other cfg topics, misc edits

* some of the edits requested by francisco

* add Hive back in to list of JDBC SQL dbs

* sneaking in a misc unrelated formatting fix

* edits requested by david

In gpexpand behave test there is a flaky failed test in "verify that the
cluster has 8 new segments".

We suspect some segments may be down after expansion.  So we dump the
gp_segment_configuration on that exception to confirm.
Co-authored-by: NNing Yu <nyu@pivotal.io>
Reviewed-by: NZhenghua Lyu <zlv@pivotal.io>

The default libcurl on centos7 uses NSS SSL library instead of OPENSSL.
GPDB with that libcurl will report "Unknown cipher in list: AES128-SHA"
on connecting gpfdist SSL server.
Actually, we don't have to set cipher list in GPDB. It may choose stronger
cipher on negotiation. So remove setting cipher list.

* Source for the segment recovery flowchart

* Rename gprecovery-flow-chart.graffle to recovermatrix.graffle to match png filename

* docs - add info about pxf jdbc statement properties

* misc edits requested by david

Co-authored-by: NGoutam Tadi <gtadi@pivotal.io>
Co-authored-by: NAmil Khanzada <akhanzada@pivotal.io>

The relallvisible field in pg_class contains the number of heap pages
that are marked as all visible for any given heap relation. This value
is used by the planner to determine the cost of running an
IndexOnlyScan.

In theory, it is possible to use the visibility map of an AO table to
enable a index-only scan for AO tables. If there is no entry in the
pg_aovisimap table, then all tuples in the aoseg file up to the EOF will
be visible, and and index-only scan should not need to visit the AO
relation. It's not clear that enabling IndexOnlyScans will buy us
anything with AO relations, so it makes sense to just disable it for
now.

Until and unless we decide to enable index only scans of AO tables,
always set relallvisible to 0 for AO relations. This should prevent the
planner from choosing an index only scan for AO relations.

Like heap tables, AO tables have a visibility map, but the AO
visibilitymap serves a different purpose than the visibilitymap in heap
tables. In Heap tables, the visibility map is an optimization used to
determine which tuples can be returned directly from an index without
needing to consult with the heap table. Heap visibility maps do not
affect the visibility of tuples, but instead are used as a hint for
whether an IndexOnlyScan needs to visit the underlying relation. The AO
visibility map is used to track which tuples have been deleted from an
AO table, and thus their actual visibility to a table scan. AO relations
do not use a relation fork for a visibility map, and instead track the
visibility of tuples in an auxiliary table.
Co-authored-by: NAdam Berlin <aberlin@pivotal.io>

When determine waiting relation in global dead lock detector,
we should also check whether the waiter's lockmode conflict with
holder's holdmask.
Co-authored-by: NNing Yu <nyu@pivotal.io>
Reviewed-by: NZhenghua Lyu <zlv@pivotal.io>

The utilities reference page contained (unlinked) mentions of long
since deprecated utilities, and since they aren't shipping it's time
to remove them:

	gpsizecalc - removed in June 2010
	gpskew - removed in October 2009
	gprebuildsystem - removed in January 2010
	gpchecknet - removed in April 2010
	gpcheckos - removed in July 2016

Reviewed-by: Lisa Owen
Reviewed-by: David Yozie

gpdetective was removed in 4.3.5, so it seems about time to also
remove the documentation for it (which was unreachable due to the
app being marked deprecated).

Reviewed-by: Lisa Owen
Reviewed-by: David Yozie

There was a race condition where replay of a DROP TABLESPACE command on
a replication mirror could occur after the tests had already physically
removed the directories. It should be safe if the tablespace directories
continue to exist after the test is complete, so don't clean them up.

* docs - add pageinspect module landing page

* remove peer scope

* cannot inspect ao or external relations

Commit 0b6f15b8 removed the only
callsite of memtuple_get_nullp_len(), so remove the function too
to avoid a compiler warning (and reduce cruft).
Reviewed-by: NTang Pengzhou <ptang@pivotal.io>
Reviewed-by: NAshwin Agrawal <aagrawal@pivotal.io>

This moves the errorpath for EXTERNAL TABLE in LIKE INCLUDING in
under the context aware error handling. This makes the error when
creating an external table the same as when creating a foreign
table. As a bonus it avoids a few checks in case of error.

Reviewed-by: Jimmy Yih
Reviewed-by: Asim R P

INCLUDING STORAGE was introduced in PostgreSQL 9.0, but support for
setting the AO/AOCS/ENCODING storage options on the created table
was never supported as we merged upstream. This extends support for
copying over the table storage type as well as attribute encodings
when creating a table with INCLUDING STORAGE or ALL.

Reported-by: Cyrille Lintz
Reviewed-by: Jimmy Yih
Reviewed-by: Asim R P

The pageinspect code only knows how to handle heap relations, so make
sure to error out gracefully on AO/AOCS and EXTERNAL relations.
Reported-by: NLisa Owen <lowen@pivotal.io>
Reviewed-by: NAsim R P <apraveen@pivotal.io>

s/specifed/specified/

At present, `gpstop` check `pg_stat_activity` table before stop cluster in smart model. However, we need to ignore those background workers that are part of the database implementation itself. For this purpose, we maintain a whitelist using its `application_name` field.

The "combine" function for the int4 sum/avg aggregate functions is
backported to Greenplum 6.0 in commit 313cef6e (from
postgres/postgres@11c8669c0cc) but we made the inevitable omission of
setting the "strictness" of int4_avg_combine to false. This by itself is
harmless as the actual function body *does* guard against NULL input,
but it also prohibits a whole host of optimizations when the executor
and planner can detect NULL input early on and short-circuit the
execution. Oops.

This patch flips the `proisstrict` flag back to true for
int4_avg_combine.

Backpatch to 6X_STABLE.

optimizer_enable_dml is set to true by default. When set to false, ORCA will
fall back to planner for all DML queries.

* docs - add to pxf upgrade procedure (v5.0.1 to v5.3.2)

* remove extraneous to

When referring to the product name and not a database instance running
Greenplum, the capitalization should be "Greenplum Database".

The --load-extension option was misspelt as load_extension.  Resource group
tests are difficult to run locally, neither are they run in PR pipeline.  That
makes it difficult to catch such errors.