* docs - reorganize/reorder the reference guide topics

* remove Greenplum qualifier, edits requested by david

* remove Reference from sql commands, sys catalogs topic titles

The drop tablespace tests that introduce panics were not disabling
fts, which is a source of flakey tests.

Introduce a disable_fts() function that we can call before each
test. The fault reset at the end of each test will reenable fts for
the next test.

The tree I was working off clearly had stale files, which led me to
include two utils which were removed some time ago: gpcheckutil.py
and gpcheck.py. Remove these two from their respective Makefiles.

Also fix a Bash error in the Stream symlink test, the logoical AND
requires [[ .. ]]; rather than [ .. ];.

Both of these spotted while repeatedly running make install with
trees in various states.

We have a kind of query that its InitPlan execute in ExecutorStart.
If Explain Analyze this kind of query, as well as, some memory or disk
information requires to collect in the main plan, the QD will crash.
Since queryDesc->showstatctx->stats_gathered is assigned to true
in ExecSetParamPlan function before ExecutorRun, So we only gather
Initplan metrics and left other slices information over. Segment fault when
some execution node print out metrics message.

To fix this problem, variable stats_gathered only be assigned after slice 0
metrics information collection.

reproduce process:

create table t(a int);
explain analyze select a
from (
	select a
	from t
	where a > (
		select avg(a)
		from t
	)
) as foo
order by a
limit 1
fix issue:#6951

Reviewed-by: NDaniel Gustafsson <dgustafsson@pivotal.io>

- before this change, the guc would setup a pPostCreate hook to be run
  after the creation of a partition table, but we never actually read
  from that hook setting, so it is dead code.

ICG changes for commit "Add 4 xforms for LOJ index apply on dynamic table
scan"

These transforms existed for non-partition tables, but not for partition
tables.
Authored-by: NChris Hajas <chajas@pivotal.io>

Co-authored-by: NMelanie Plageman <mplageman@pivotal.io>

Static function and variables are unused for mock files and hence
generate warnings. Hence, ignore only those warnings.
Co-authored-by: NMelanie Plageman <mplageman@pivotal.io>

"-w" flag inhibits all warning messages. That's not ideal. Hence,
remove the flag.

Commit feb3a399 removed the example
postgresql_conf_gp_additions file for gpinitsystem, but the refactor
of the gpMgmt/bin Makefiles didn't get the memo as the file existed
when that work was done. Fix by removing the file to avoid warning
during make install.

The calculation for minimum amount of memory to memory intensive
operators was using kB for both input and output, leading to an
assertion failure in case the minimum memory requirement was less
than a kB. Since we are using a full uint64 for calculating, we
have room for using just bytes instead leaving the formatting to
the caller (which is just EXPLAIN ANALYZE).
Reviewed-by: NWeinan Wang <wewang@pivotal.io>

While hacking on another thing, I spotted that we had numerous lines
with tab + 1 space indentation, which is clearly wrong. Fix by using
just tab indentation as per the style guide.

* docs - GDD SELECT...FOR locking

Add information about GDD managing SELECT...FOR <locking strength> queries

Currently this feature is in MASTER only.

* docs - updates from review comments

* docs - fix typo

Adds tablespace OID to pg_xlogdump output for xl_xact_commit and xl_xact_abort records for debugging purposes. The OID information is actually redundant and can be backtraced through the commit record, however it could still be useful in case the first message is not available.
Co-authored-by: NIvan Leskin <leskin.in@arenadata.io>

This commit adds the GPDB side changes required to support GIN Indexes with
ORCA.  It also adds a new test file gp_gin_indexes to test plans produced for
ORCA/planner.

GIN indexes are not supported with index expression or predicate constraints.
ORCA does not support it currently for other types of indexes too.

There is a pipeline for unit tests in GPDB used in most cases.

However, unit tests of src/backend/access/gin introduced by 99360f54
used a custom implementation of unit test build script. This led to
errors e.g. when a compiler different than GCC was used to build GPDB.

Rewrite Makefile in order to unify test infrastructure with common
pattern used in the backend while retaining test isolation from the
backend objects.

See also similar Makefile: src/backend/catalog/test/Makefile
at 122c79f2

Note the Makefile in src/backend/access/gin/test is different from
currently most used version of a backend unit test Makefile. These
differences and motivation for them is described in the README.

Run pgindent on ginpostlist_fakes.c
Reviewed-by: NAdam Berlin <aberlin@pivotal.io>
Reviewed-by: NDaniel Gustafsson <dgustafsson@pivotal.io>
Reviewed-by: NHeikki Linnakangas <hlinnakangas@pivotal.io>
Co-authored-by: NIvan Leskin <leskin.in@arenadata.io>

The following SQL snippet demonstrates dependencies on array types derived from partitions of partitioned tables:

```
CREATE TABLE table_part(a INT, b INT) PARTITION BY RANGE(b) (PARTITION part1 START(0) END(42));
CREATE TABLE replacement(LIKE table_part);
ALTER TABLE table_part EXCHANGE PARTITION part1 WITH TABLE replacement;
CREATE TABLE dependant(d table_part_1_prt_part1[]);
```

Such dependencies cannot be carried over to the upgraded cluster, and pg_upgrade should fail when they are present. The user should take extra steps to resolve the situation.

When check fails, the output is similar to the following:

```
Checking array types derived from partitions                fatal

Array types derived from partitions of a partitioned table must not have dependants.
OIDs of such types found and their original partitions:
16393 public.table_part_1_prt_part1
16471 public.table_part2_1_prt_part1
Failure, exiting
```
Reviewed-by: NSoumyadeep Chakraborty <sochakraborty@pivotal.io>
Co-authored-by: NIvan Leskin <leskin.in@arenadata.io>

ALTER TABLE DROP COLUMN followed by reorganize leads to loss of column
encoding settings of the dropped column. When the column's compresstype
encoding is incorrect, we can encounter block version mismatch error
later during block info validation check of the dropped column.

One idea was to skip dropped columns when constructing AOCSScanDesc.
However, dropping all columns is a special case that is not easily handled
because it is not equivalent to deleted rows. Instead, the fix is to preserve
column encoding settings even for dropped columns.
Co-authored-by: NSoumyadeep Chakraborty <sochakraborty@pivotal.io>
Co-authored-by: NIvan Leskin <leskin.in@arenadata.io>

The big gpMgmt Makefile rewrite in b5aba18b
was a paddle short for make distclean in gpMgmt/bin/lib as it was
missing the 'rm' invocation.

Some callers of pg_mkdir_p() consider EEXIST as a non-error case, this
is incorrect.  When pg_mkdir_p() sets errno to EEXIST it means that the
target path already exists but is not a directory.  So as long as
pg_mkdir_p() returns -1 we should consider it as a failure.
Co-authored-by: NPaul Guo <pguo@pivotal.io>
Co-authored-by: NNing Yu <nyu@pivotal.io>

pg_mkdir_p() is used by initdb and pg_basebackup to create a directory
as well as its parent directories.  The logic used to be like below:

    if (stat(path) < 0)
    {
        if (mkdir(path) < 0)
            retval = -1;
    }

It first checks for the existence of the path, if path does not
pre-exist then it creates the directory.  However if two processes try
to create path concurrently, then one possible execution order is as
below:

    A: stat(path) returns -1, so decide to create it;
    B: stat(path) returns -1, so decide to create it;
    B: mkdir(path) returns 0, successfully created path;
    A: mkdir(path) returns -1, fail to create path as it already exist;

This race condition could be fixed by swapping the order of stat() and
mkdir(), this is also what the "mkdir -p" command does.
Co-authored-by: NPaul Guo <pguo@pivotal.io>
Co-authored-by: NNing Yu <nyu@pivotal.io>

package_tarball.bash is used in many places outside of the gpdb
repository so any modifications should be done very carefully.

This commit also introduces package_tarball.bats as an example file for
other concourse users wishing to add .bats files.
Authored-by: NAmil Khanzada <akhanzada@pivotal.io>
Reviewed-by: NMark Sliva <msliva@pivotal.io>

We have added tests to ensure that the search_path setting is taking effect. We
also are using a test database for the tests.
Co-authored-by: NJamie McAtamney <jmcatamney@pivotal.io>
Co-authored-by: NJacob Champion <pchampion@pivotal.io>
Co-authored-by: NNikolaos Kalampalikis <nkalampalikis@pivotal.io>

The unsetting of search_path started an implicit transaction that can
affect the behavior of subsequent SQL statements. A simpler solution
(thanks Heikki!) is to just set the search_path for the entire
connection using the connection string options.
Co-authored-by: NJamie McAtamney <jmcatamney@pivotal.io>
Co-authored-by: NJacob Champion <pchampion@pivotal.io>
Co-authored-by: NNikolaos Kalampalikis <nkalampalikis@pivotal.io>

GUC gp_enable_gpperfmon is defined to be set only at postmaster
restart.  Having a check hook that checks if the process setting it
has superuser privileges is meaningless.  The check hook is removed.

GUC gp_gpperfmon_send_interval is intended to be set only by
superuser.  Adjust its definition accordingly and leverage checks
built into GUC framework for superuser privileges.  The check hook for
this GUC tried to achieve the same but incorrectly.  If the check hook
was invoked at the beginning of main query processing loop by a
backend process, it would crash.  At the beginning of the main loop, a
transaction is not started yet.  The check hook invokes superuser()
interface, which performs catalog access.  Doing so without starting a
transaction is a recipe for crashing badly.  Such a crash was observed
in production at least once.

Thank you Jesse Zhang for suggesting to remove superuser check.

The patch doesn't add any tests because, after removing the check
hooks, the checks built into GUC framework are being used.  That code
path is well exercised by existing regression tests.

Reviewed-by: Daniel Gustafsson

Happened to spot these due to recent correspondence.

With the revamp of the gpMgmt Makefiles, the catalog json and stream
binary are now compiled on make all, so we should perform them there
rather than opt for building on make install. This still leaves some
modules which are compiled on make install, such as pygresql, but a
small start is still a start.

Discussion: https://github.com/greenplum-db/gpdb/pull/8179
Reviewed by Bradford Boyle, Kalen Krempely, Jamie McAtamney and
many more

The psi package is no longer installed anywhere, so remove the target
from the Makefile.

Reported-by: Shaoqi Bai
Discussion: https://github.com/greenplum-db/gpdb/pull/8179
Reviewed by Bradford Boyle, Kalen Krempely, Jamie McAtamney and
many more

Discussion: https://github.com/greenplum-db/gpdb/pull/8179
Reviewed by Bradford Boyle, Kalen Krempely, Jamie McAtamney and
many more