Unique indexes are now enforced throughout gpexpand, so there's no need to
warn or treat them specially. (gpexpand used to first make all tables
randomly distributed, and while in that state, unique indexes could not be
enforced. But it doesn't do that anymore.)

This also removes the gp_status_detail.rank column. There's no particular
reason to process tables with indexes before others anymore, and that was
the only criteria used for ranking.
Reviewed-by: NMelanie Plageman <mplageman@pivotal.io>

Commit 46b21685 added a similar check for Zstandard. This makes QuickLZ
behave the same.

Fixes https://github.com/greenplum-db/gpdb/issues/5107Reviewed-by: NGeorgios Kokolatos <gkokolatos@pivotal.io>
Reviewed-by: NAdam Berlin <aberlin@pivotal.io>
Reviewed-by: NDaniel Gustafsson <dgustafsson@pivotal.io>

This commit fixes a subtle bug that was papered over by clever
optimizations: the compiler *often* takes the clue that a zero `nattrs`
(implied by a NULL `policy`) leads to an empty loop, and skips straight
through to closing the relation, avoiding the dereference of
`policy->opclasses`. That doesn't always happen though, if you're lucky,
the generated code *does* dereference `policy->opclasses` even when we
have a NULL policy. It seems like we had run into this before, as
evidenced by the (removed in this patch) ternary on `policy->nattrs`.

Co-authored-by: NSambitesh Dash <sdash@pivotal.io>

* docs - update template database information.

Updated ddl-database.xml topic - section About Template Databases
Also removed draft text from expand topics, gpexpand and pg_tablespace

* docs - update template database information based on review comments.
-Changed text - postgres is not a template database
-changed title to About Template and Default Databases

* Small line edit

One of the Behave tests was calling gpcheckcat -R with an invalid
check name, causing the test to silently fail.  This commit corrected
that call.

Additionally, the SQL script used to generate the error for that test
was making a catalog change that would be nearly impossible to hit "in
the wild," and was causing the repair script to encounter an assertion
failure.  The script has been changed to generate a more reasonable
error, one that is more relevant to what we want to test.

Finally, the Behave step for running repair scripts has been modified
to actually error out if a repair script encounters a non-zero return
code (when it was silently failing before) and give the user a
descriptive error message.
Co-authored-by: NJamie McAtamney <jmcatamney@pivotal.io>

gcc's -Wempty-body warns about the current usage when compiling
postgres without --enable-cassert.

When the planner creates a subquery as part of grouping planning, fix
the target list of the outer query to correctly refer to the outputs of
the subquery. Previously, the parse tree's target list was sometimes left
unchanged, so that it still contained Vars referring to the original
relations, which have been pushed down to the subquery. After adding
the subquery, the outer query only contains a single RTE for the subquery.

We usually got away with a bogus target list in the outer query, because
we had already decided how to construct the plan, and usually didn't look
at the original parse tree anymore. In some cases, though, when the caller
asked for it by passing 'use_root=true', we were already doing the right
thing, but otherwise we didn't bother. There's little downside to fixing
the target list, though, so let's do it unconditionally.

Fixes https://github.com/greenplum-db/gpdb/issues/6754Reviewed-by: NMelanie Plageman <mplageman@pivotal.io>

GPDB's AllocSizeIsValid() was off by one byte, compared to upstream's.
If the argument was exactly MaxAllocSize, the GPDB version would return
false, whereas upstream's would return true.

We had compensated for this in enlargeStringInfo(). But it wasn't quite
right: if the caller needed exactly MaxAllocSize bytes (including the null
terminator), the code would truncate it down to MaxAllocSize - 1, and
therefore allocate one byte too little. That's unlikely to cause any real
trouble on production systems, because in practice malloc() will round up
the allocation to some alignment boundary. But with assertions enabled, it
is easy to demonstrate e.g. with this:

postgres=# select repeat('x', 1073741816);
WARNING:  detected write past chunk end in printtup 0x7f8eabfff040 (aset.c:1505)
Reviewed-by: NGeorgios Kokolatos <gkokolatos@pivotal.io>
Reviewed-by: NMelanie Plageman <mplageman@pivotal.io>

apu-1-config is still being pulled from Ivy for sles, so we need to keep
the --with-apu-config flag for sles.
Co-authored-by: NAmil Khanzada <akhanzada@pivotal.io>
Co-authored-by: NSambitesh Dash <sdash@pivotal.io>

* docs - update util ref pages for 9.0beta3

* edits requested by mel

* address comments from chuck

* ALTER DEFAULT PRIVILEGES. Add new command to sql ref

* ALTER ROLE. Substantive changes are already in the dita.

* ALTER TABLE. add [IF EXISTS], STATISTICS DISTINCT

* CREATE CAST. Add elipses to argtype

* CREATE DATABASE. Change dbowner to user_name

* CREATE FUNCTION. Remove SETOF and LIKE from RETURNS clause, notes on argument names

* CREATE ROLE. add USER, SYSID options

* CREATE TABLE.  LIKE clause syntax reorg and behavior. CHECK clause.

* psql. add \ddp and \drds meta commands

* REVOKE. add optional GROUP keyword

* SELECT. Many changes. Lock behavior changes. Compatibility notes.

* VACUUM. new parenthesized form of command

* psql. Add missing square bracket

* ALTER ROLE. Edits only.

* ALTER TABLE. Change STATISTICS DISTINCT to (attribute_option = n)

* ALTER TABLESPACE. Add tablespace_parameter

* ANALYZE. Change STATISTICS DISTNCT to SET (n_distinct = ...)

* CHECKPOINT. forces a restartpoint if run during recovery.

* CLUSTER. Mention FILLFACTOR and ALTER TABLE cluster options

* CREATE DOMAIN. domain can become null in spite of NOT NULL

* CREATE CONVERSION. Note command is similar to standard create translation.

* CREATE INDEX. Index name is optional, GiST example, edits.

* CREATE LANGUAGE. Add OR REPLACE clause, edits.

* CREATE OPERATOR. Deprecate => operator, note about lexical precedence

* REASSIGN OWNED. Add required privileges and other notes

* REINDEX. Clarify which indexes are reindexed for each option

* DROP CONVERSION. Note command is similar to standard DROP TRANSLATION

* SELECT. RANGE/ROWS edits, mostly made in earlier commits.

* SHOW. Add examples with output

* VACUUM. VACUUM FULL caveats.

* Formatting fixes

* Updates from reviews

Co-authored-by: NAmil Khanzada <akhanzada@pivotal.io>
Co-authored-by: NDavid Sharp <dsharp@pivotal.io>

In gpAux/Makefile, stop passing --with-apr-config for centos{6,7}. It
was pointing at the version of apr-1-config which was pulled down by ivy
and not the one on the PATH (which holds the version of apr-1-config
bundled with the OS distro).
Co-authored-by: NSambitesh Dash <sdash@pivotal.io>
Co-authored-by: NDavid Sharp <dsharp@pivotal.io>
Co-authored-by: NAmil Khanzada <akhanzada@pivotal.io>

* analyze - add paragraph about distinct values per column

* copy - new options and syntax

* COPY - add legacy syntax

* COPY - changes from 9.0Beta3 merge

* CREATE VIEW - add note about creating existing view

* EXPLAIN - new options; examples not updated.

* FETCH - small edits

* GRANT - small consistency edits; syntax consistency for optional GROUP keyword

* MOVE - optional FROM, IN keywords

* MOVE - optional GROUP keyword, consistency edits

* ROLLBACK TO SAVEPOINT - clarification/edits around cursor rollback

* bump postgresql url reference to 9.4

* bump postgresql url reference to 9.4

* bump postgresql url reference to 9.4

* delimiter -> delimiter_character

* Add a few more words around defaults for omitted options

* bump postgresql url reference to 9.4

* bump postgresql url reference to 9.4

* analyze - add paragraph about distinct values per column

* copy - new options and syntax

* COPY - add legacy syntax

* COPY - changes from 9.0Beta3 merge

* CREATE VIEW - add note about creating existing view

* EXPLAIN - new options; examples not updated.

* FETCH - small edits

* GRANT - small consistency edits; syntax consistency for optional GROUP keyword

* MOVE - optional FROM, IN keywords

* MOVE - optional GROUP keyword, consistency edits

* ROLLBACK TO SAVEPOINT - clarification/edits around cursor rollback

* delimiter -> delimiter_character

* Add a few more words around defaults for omitted options

We added to the FIXME comment a link to an outstanding discussion on
gpdb-dev mailing list about whether or not it is correct for semi-join
to have quals deferred like this. If it is not, then the code should be
changed and the assert can go back to the same as upstream. If it is,
then the assert is right.
Co-authored-by: NMelanie Plageman <melanieplageman@gmail.com>

* ALTER DEFAULT PRIVILEGES. Add new command to sql ref

* ALTER ROLE. Substantive changes are already in the dita.

* ALTER TABLE. add [IF EXISTS], STATISTICS DISTINCT

* CREATE CAST. Add elipses to argtype

* CREATE DATABASE. Change dbowner to user_name

* CREATE FUNCTION. Remove SETOF and LIKE from RETURNS clause, notes on argument names

* CREATE ROLE. add USER, SYSID options

* CREATE TABLE.  LIKE clause syntax reorg and behavior. CHECK clause.

* psql. add \ddp and \drds meta commands

* REVOKE. add optional GROUP keyword

* SELECT. Many changes. Lock behavior changes. Compatibility notes.

* VACUUM. new parenthesized form of command

* psql. Add missing square bracket

* Review comments

In Greenplum, during vacuuming of an append optimized relation, the
query dispatcher needs to update its tupcount information from the segments.
Vacuum calls UpdateMasterAosegTotals with the appropriate snapshot as an
argument.

UpdateMasterAosegTotals was introduced in <d9415f4cc5db95a0> (closed source)
and it was called with the special static `SnapshotNow`. Possibly due to
oversight, while gathering the information of columnar append optimized
relations, it reused the static instead of the arg, which happened to match.

In 9.4 SnapshotNow was removed in favor of MVCC catalog scans.

The comment wondered if the use of apparently different snapshots was
intentional. The conclusion is that it was not. Vacuum is holding the
appropriate snapshot which is now used in both cases.
Reviewed-by: NHeikki Linnakangas <hlinnakangas@pivotal.io>
Reviewed-by: NAshwin Agrawal <aagrawal@pivotal.io>

This re-implements compression that was lost in the WorkFile/BufFile
refactoring. This new implementation uses Zstandard rather than zlib.

Compression can be enabled with "gp_workfile_compression=on". It is
disabled by default.
Reviewed-by: NMel Kiyama <mkiyama@pivotal.io>
Reviewed-by: NYandong Yao <yyao@pivotal.io>

Temporary files have been somewhat inconsistent across different
operations. Some operations used the Greenplum-specific "workfile" API,
while others used the upstream BufFile API directly.

The workfile API provides some extra features: workfiles are visible in
the gp_toolkit views, and you can limit their size with the
gp_workfile_limit_* GUCs. The temporary files that didn't go through the
workfile API were exempt, which is not cool.

To make things consistent, remove the workfile APIs. Use BufFiles
directly everywhere. Re-implement the user-facing view and tracking the
limits, on top of the BufFile API, so that those features are not lost.

The workfile API also supported compressing the temporary files using
zlib. That feature is lost with this commit, but will be re-introduced by
the next commit.

Another feature that this removes, is checksumming temporary files.
That doesn't seem very useful, so we can probably live without it. But
if it's still needed, then that should also be re-implemented on top
of the BufFile API later.

Discussion: https://groups.google.com/a/greenplum.org/d/msg/gpdb-dev/8Xe9MGor0pM/SjqiOo83BAAJReviewed-by: NMel Kiyama <mkiyama@pivotal.io>
Reviewed-by: NYandong Yao <yyao@pivotal.io>

The way dropped columns are handled in ALTER TABLE was changed back in
2017, commit 62d66c06, but gpexpand didn't get the memo. Dropped columns
of custom datatypes no longer pose any problems.
Reviewed-by: NVenkatesh Raghavan <vraghavan@pivotal.io>

The code to look up the hash functions for a Reshuffle plan used
get_opfamily_proc() instead of the more versatile
cdb_hashproc_in_family() function, which is used in other similar places
where we need to look up the hash functions for a distribution key. Like
in makeCdbHashForRelation(). That lead to errors if the datatype didn't
have a hash function defined directly for the datatype, but only via a
binary-coercible cast. Domain and enum types are such cases, for example.

Fixes https://github.com/greenplum-db/gpdb/issues/6901Reviewed-by: NDavid Kimura <dkimura@pivotal.io>
Reviewed-by: NVenkatesh Raghavan <vraghavan@pivotal.io>

Co-authored-by: NBen Christel <bchristel@pivotal.io>
Co-authored-by: NSambitesh Dash <sdash@pivotal.io>
Co-authored-by: NAmil Khanzada <akhanzada@pivotal.io>
Co-authored-by: NDavid Sharp <dsharp@pivotal.io>

GPDB 5 and 6 only require xerces-c for gporca. We were originally
pulling it from Ivy during the build process, but it is now bundled with
gporca.
Co-authored-by: NBen Christel <bchristel@pivotal.io>
Co-authored-by: NSambitesh Dash <sdash@pivotal.io>
Co-authored-by: NAmil Khanzada <akhanzada@pivotal.io>
Co-authored-by: NDavid Sharp <dsharp@pivotal.io>

This reverts commit 1cf5e595.

This reverts commit 07d9a5f8.

* docs - add GUC gp_enable_global_deadlock_detector

Add Note text to DELETE, LOCK, UPDATE about GDD.
Also, update lock descriptions in dml.xml to point to GDD section.

* docs - review updates for gp_enable_global_deadlock_detector GUC

When a primary segment crashes, there is a possibility of the
postmaster.pid being left behind. If another process starts actively
using that pid, pg_rewind will failed when it tries to start the
failed segment in single-user mode (crash recovery to ensure clean
shutdown). It should be safe to remove the file before pg_rewind
during gprecoverseg since gprecoverseg ensures all failed segments are
shut down (pg_ctl stop or via forceful termination).

Also, revert some gprecoverseg full calls in gprecoverseg.feature
Behave test to be back to gprecoverseg incremental. They were marked
with WALREP_FIXME last year waiting for pg_rewind which we now have.
Co-authored-by: NJimmy Yih <jyih@pivotal.io>

Issue is a hardcoded port in the default `net.ipv4.ip_local_port_range`
of 32768 - 60999. It is possible that another processes is using the
port, in which case the test will fail.  This commit changes to a port
outside that range.  Alternatively, We could dynamically fetch free
ports, but that adds extra unnecessary complexity.
Co-authored-by: NAlexandra Wang <lewang@pivotal.io>

This reverts commit 4f686127 as it
has been causing strange failures in the main pipeline.

Commits 41c919ce, 0ad14f38 and 08d7ca18 removed these columns, but
forgot to update the docs accordingly.

Commit ad6b920f accidentally removed
the headerguards due to them being namedd GPHDFS, even though they
aren't GPHDFS related. This caused a lot of redefenition warnings at
build time, which are fixed by this.

In Greenplum 5.X the recursive CTE feature was hidden behind a GUC as
it wasn't deemed of production quality just yet. Commit 20152cbf
removed that GUC in order to make stabilization work easier, there are
still enough rough edges to not consider recursive CTE a feature which
is on by default. This brings back the GUC using the same name in order
to be backwards compatible even though "prototype" is a bit misleading
as this point.

Also adds a test and tidies up a few comments in surrounding code.

Discussion: https://groups.google.com/a/greenplum.org/forum/#!topic/gpdb-dev/Lt6l3wDyxTYCo-authored-by: NGeorgios Kokolatos <gkokolatos@pivotal.io>