As we merge with upstream and by that keep refining the Postgres
planner, legacy planner is no longer a suitable name. This changes
all variations of the spelling (legacy planner, legacy optimizer,
legacy query optimizer etc) to say "Postgres" rather than "legacy".
Reviewed-by: NVenkatesh Raghavan <vraghavan@pivotal.io>
Reviewed-by: NDavid Yozie <dyozie@pivotal.io>
Reviewed-by: NGeorgios Kokolatos <gkokolatos@pivotal.io>

* docs - GDD - add information about specific type of deadlocks.

* docs - GDD - update deadlock information based on review comments.

* typo fix

* Small edit for clarity

* Removing duplicate info

Changing version in path from /600 to /6-0 for consistency with other docs and to recognize that only the minor version digit is significant for the doc build (#7141)

The GUC which enables recursive CTEs is in the currently released
version called gp_recursive_cte_prototype, but in order to reflect
the current state of the code it's now renamed to gp_recursive_cte.
By default the GUC is still off, but that might change before we
ship the next release.

The previous GUC name is still supported, but marked as deprecated,
in order to make upgrades easier.
Reviewed-by: NIvan Novick <inovick@pivotal.io>
Reviewed-by: NGeorgios Kokolatos <gkokolatos@pivotal.io>

This method was introduced to improve the data redistribution
performance during gpexpand phase2, however per benchmark results the
effect does not reach our expectation.  For example when expanding a
table from 7 segments to 8 segments the reshuffle method is only 30%
faster than the traditional CTAS method, when expanding from 4 to 8
segments reshuffle is even 10% slower than CTAS.  When there are indexes
on the table the reshuffle performance can be worse, and extra VACUUM is
needed to actually free the disk space.  According to our experiments
the bottleneck of reshuffle method is on the tuple deletion operation,
it is much slower than the insertion operation used by CTAS.

The reshuffle method does have some benefits, it requires less extra
disk space, it also requires less network bandwidth (similar to CTAS
method with the new JCH reduce method, but less than CTAS + MOD).  And
it can be faster in some cases, however as we can not automatically
determine when it is faster it is not easy to get benefit from it in
practice.

On the other side the reshuffle method is less tested, it is possible to
have bugs in corner cases, so it is not production ready yet.

In such a case we decided to retire it entirely for now, we might add it
back in the future if we can get rid of the slow deletion or find out
reliable ways to automatically choose between reshuffle and ctas
methods.

Discussion: https://groups.google.com/a/greenplum.org/d/msg/gpdb-dev/8xknWag-SkI/5OsIhZWdDgAJReviewed-by: NHeikki Linnakangas <hlinnakangas@pivotal.io>
Reviewed-by: NAshwin Agrawal <aagrawal@pivotal.io>

Make sure there is whitespace before xref tags when used in normal
text to avoid links squashed together with text.
Reviewed-by: NLisa Owen <lowen@pivotal.io>

* docs - CTE available with INSERT, UPDATE, DELETE

-updated GUC
-updated Admin Guide topic WITH Queries (Common Table Expressions)

updates to SELECT, INSERT, UPDATE DELETE will be part of postgres 9.2 merge.

* docs - CTE updates from review comments

* docs - CTE more updates from review comments

* docs - CTE - updates from review comments

* Experimental -> Beta wording

* reword from 'experimental' to 'beta'

* Experimental -> Beta in markdown source

* typo fix

* Removing SuSE 11 details in Beta notes

* Warn not to run gpbackup/gprestore during expand.

* Updates from review

* docs - add new host/seg resource group usage views

* clarifications requested by ning

docs - exchange partition, add limitation exchanging with a partitioned table is not supported. (#6961)

* docs - appendoptimized is an alias for appendonly storage option

* option names lower case, add legacy, misc edits

* docs - update compression information
-Add a topic that lists features that support configuring compression.
-update gp_workfile_compression GUC. Remove requirement to install zstd.

* docs - update compression information based on review comments.
-changed description to point to feature/utility for specific compression information.
-added general, pivotal, and OSS specific information about compression requirements.

* docs - compression information - reivew edit.

Unique indexes are now enforced throughout gpexpand, so there's no need to
warn or treat them specially. (gpexpand used to first make all tables
randomly distributed, and while in that state, unique indexes could not be
enforced. But it doesn't do that anymore.)

This also removes the gp_status_detail.rank column. There's no particular
reason to process tables with indexes before others anymore, and that was
the only criteria used for ranking.
Reviewed-by: NMelanie Plageman <mplageman@pivotal.io>

* docs - update template database information.

Updated ddl-database.xml topic - section About Template Databases
Also removed draft text from expand topics, gpexpand and pg_tablespace

* docs - update template database information based on review comments.
-Changed text - postgres is not a template database
-changed title to About Template and Default Databases

* Small line edit

This re-implements compression that was lost in the WorkFile/BufFile
refactoring. This new implementation uses Zstandard rather than zlib.

Compression can be enabled with "gp_workfile_compression=on". It is
disabled by default.
Reviewed-by: NMel Kiyama <mkiyama@pivotal.io>
Reviewed-by: NYandong Yao <yyao@pivotal.io>

Temporary files have been somewhat inconsistent across different
operations. Some operations used the Greenplum-specific "workfile" API,
while others used the upstream BufFile API directly.

The workfile API provides some extra features: workfiles are visible in
the gp_toolkit views, and you can limit their size with the
gp_workfile_limit_* GUCs. The temporary files that didn't go through the
workfile API were exempt, which is not cool.

To make things consistent, remove the workfile APIs. Use BufFiles
directly everywhere. Re-implement the user-facing view and tracking the
limits, on top of the BufFile API, so that those features are not lost.

The workfile API also supported compressing the temporary files using
zlib. That feature is lost with this commit, but will be re-introduced by
the next commit.

Another feature that this removes, is checksumming temporary files.
That doesn't seem very useful, so we can probably live without it. But
if it's still needed, then that should also be re-implemented on top
of the BufFile API later.

Discussion: https://groups.google.com/a/greenplum.org/d/msg/gpdb-dev/8Xe9MGor0pM/SjqiOo83BAAJReviewed-by: NMel Kiyama <mkiyama@pivotal.io>
Reviewed-by: NYandong Yao <yyao@pivotal.io>

The way dropped columns are handled in ALTER TABLE was changed back in
2017, commit 62d66c06, but gpexpand didn't get the memo. Dropped columns
of custom datatypes no longer pose any problems.
Reviewed-by: NVenkatesh Raghavan <vraghavan@pivotal.io>

* docs - add GUC gp_enable_global_deadlock_detector

Add Note text to DELETE, LOCK, UPDATE about GDD.
Also, update lock descriptions in dml.xml to point to GDD section.

* docs - review updates for gp_enable_global_deadlock_detector GUC

* docs - remove/replace references to gphdfs

* misc edits

* stronger statement (note) about pxf replacing gphdfs

* stronger statement

* edit to note requested by david

* haved -> have... doh

* docs - misc pxf edits

* add pxf to loading/unloading page and subnav

* edit requested by lav

Replace the use of the built-in hashing support for built-in datatypes, in
cdbhash.c, with the normal PostgreSQL hash functions. Now is a good time
to do this, since we've already made the change to use jump consistent
hashing in GPDB 6, so we'll need to deal with the upgrade problems
associated with changing the hash functions, anyway.

It is no longer enough to track which columns/expressions are used to
distribute data. You also need to know the hash function used. For that,
a new field is added to gp_distribution_policy, to record the hash
operator class used for each distribution key column. In the planner,
a new opfamily field is added to DistributionKey, to track that throughout
the planning.

Normally, if you do "CREATE TABLE ... DISTRIBUTED BY (column)", the
default hash operator class for the datatype is used. But this patch
extends the syntax so that you can specify the operator class explicitly,
like "... DISTRIBUTED BY (column opclass)". This is similar to how an
operator class can be specified for each column in CREATE INDEX.

To support upgrade, the old hash functions have been converted to special
(non-default) operator classes, named cdbhash_*_ops. For example, if you
want to use the old hash function for an integer column, you could do
"DISTRIBUTED BY (intcol cdbhash_int4_ops)". The old hard-coded whitelist
of operators that have "compatible" cdbhash functions has been replaced
by putting the compatible hash opclasses in the same operator family. For
example, all legacy integer operator classes, cdbhash_int2_ops,
cdbhash_int4_ops and cdbhash_int8_ops, are all part of the
cdbhash_integer_ops operator family).

This removes the pg_database.hashmethod field. The hash method is now
tracked on a per-table and per-column basis, using the opclasses, so it's
not needed anymore.

To help with upgrade from GPDB 5, this introduces a new GUC called
'gp_use_legacy_hashops'. If it's set, CREATE TABLE uses the legacy hash
opclasses, instead of the default hash opclasses, if the opclass is not
specified explicitly. pg_upgrade will set the new GUC, to force the use of
legacy hashops, when restoring the schema dump. It will also set the GUC
on all upgraded databases, as a per-database option, so any new tables
created after upgrade will also use the legacy opclasses. It seems better
to be consistent after upgrade, so that collocation between old and new
tables work for example. The idea is that some time after the upgrade, the
admin can reorganize all tables to use the default opclasses instead. At
that point, he should also clear the GUC on the converted databases. (Or
rather, the automated tool that hasn't been written yet, should do that.)

ORCA doesn't know about hash operator classes, or the possibility that we
might need to use a different hash function for two columns with the same
datatype. Therefore, it cannot produce correct plans for queries that mix
different distribution hash opclasses for the same datatype, in the same
query. There are checks in the Query->DXL translation, to detect that
case, and fall back to planner. As long as you stick to the default
opclasses in all tables, we let ORCA to create the plan without any regard
to them, and use the default opclasses when translating the DXL plan to a
Plan tree. We also allow the case that all tables in the query use the
"legacy" opclasses, so that ORCA works after pg_upgrade. But a mix of the
two, or using any non-default opclasses, forces ORCA to fall back.

One curiosity with this is the "int2vector" and "aclitem" datatypes. They
have a hash opclass, but no b-tree operators. GPDB 4 used to allow them
as DISTRIBUTED BY columns, but we forbid that in GPDB 5, in commit
56e7c16b. Now they are allowed again, so you can specify an int2vector
or aclitem column in DISTRIBUTED BY, but it's still pretty useless,
because the planner still can't form EquivalenceClasses on it, and will
treat it as "strewn" distribution, and won't co-locate joins.

Abstime, reltime, tinterval datatypes don't have default hash opclasses.
They are being removed completely on PostgreSQL v12, and users shouldn't
be using them in the first place, so instead of adding hash opclasses for
them now, we accept that they can't be used as distribution key columns
anymore. Add a check to pg_upgrade, to refuse upgrade if they are used
as distribution keys in the old cluster. Do the same for 'money' datatype
as well, although that's not being removed in upstream.

The legacy hashing code for anyarray in GPDB 5 was actually broken. It
could produce a different hash value for two arrays that are considered
equal, according to the = operator, if there were differences in e.g.
whether the null bitmap was stored or not. Add a check to pg_upgrade, to
reject the upgrade if array types were used as distribution keys. The
upstream hash opclass for anyarray works, though, so it is OK to use
arrays as distribution keys in new tables. We just don't support binary
upgrading them from GPDB 5. (See github issue
https://github.com/greenplum-db/gpdb/issues/5467). The legacy hashing of
'anyrange' had the same problem, but that was new in GPDB 6, so we don't
need a pg_upgrade check for that.

This also tightens the checks ALTER TABLE ALTER COLUMN and CREATE UNIQUE
INDEX, so that you can no longer create a situation where a non-hashable
column becomes the distribution key. (Fixes github issue
https://github.com/greenplum-db/gpdb/issues/6317)

Discussion: https://groups.google.com/a/greenplum.org/forum/#!topic/gpdb-dev/4fZVeOpXllQCo-authored-by: NMel Kiyama <mkiyama@pivotal.io>
Co-authored-by: NAbhijit Subramanya <asubramanya@pivotal.io>
Co-authored-by: NPengzhou Tang <ptang@pivotal.io>
Co-authored-by: NChris Hajas <chajas@pivotal.io>
Reviewed-by: NBhuvnesh Chaudhary <bchaudhary@pivotal.io>
Reviewed-by: NNing Yu <nyu@pivotal.io>
Reviewed-by: NSimon Gao <sgao@pivotal.io>
Reviewed-by: NJesse Zhang <jzhang@pivotal.io>
Reviewed-by: NZhenghua Lyu <zlv@pivotal.io>
Reviewed-by: NMelanie Plageman <mplageman@pivotal.io>
Reviewed-by: NYandong Yao <yyao@pivotal.io>

* docs - remove gptransfer from docs
--removed gptransfer topics, references to gptransfer, and images.
--also updated text in gpcopy-migrate as rough update for 6.0

* docs - remove gptransfer from docs - review updates

* docs - REPEATABLE READ xact mode is supported. SERIALIZABLE falls back to REPEATABLE READ.

* Note that GPDB doesn't implement PGSQL SSI transactions

* Review comments

* docs - updates for online expand

* docs - online expand - edits based on review comments.
updated catalog table information.
removed draft comments.

* docs - reorg pxf content, add multi-server, objstore content

* misc edits, SERVER not optional

* add server, remove creds from examples

* address comments from alexd

* most edits requested by david

* add Minio to table column name

* edits from review with pxf team (start)

* clear text credentials, reorg objstore cfg page

* remove steps with XXX placeholder

* add MapR to supported hadoop distro list

* more objstore config updates

* address objstore comments from alex

* one parquet data type mapping table, misc edits

* misc edits from david

* add mapr hadoop config step, misc edits

* fix formatting

* clarify copying libs for MapR

* fix pxf links on CREATE EXTERNAL TABLE page

* misc edits

* mapr paths may differ based on version in use

* misc edits, use full topic name

* update OSS book for pxf subnav restructure

The INTO ERROR TABLE syntax has been deprecated since Greenplum 5
shipped.

Discussion: https://groups.google.com/a/greenplum.org/forum/#!topic/gpdb-dev/mzYcVk_G5UwReviewed-by: NDavid Yozie <dyozie@pivotal.io>

* Docs - update docs to note that system columns are unavailable in
queries on replicated tables.

* Edits from reviewers

* docs - migrate using gpcopy- remove same number of source, dest. hosts restriction.

* docs - migrate using gpcopy- updates based on review comments.

* docs - first HA updates that use WAL replication.

--Removed references to filerep
--Updated segment instance states to use WAL rep states
--Other misc. updates.

* docs -  HA updates for WAL replication - review comment upates

* Adds window examples to query topic in admin guide

* Fix some clear errors in CREATE TYPE and CREATE FUNCTION SQL

* Changes from review comments

* Add missing period

* docs - discuss the global deadlock detector

* some of the edits requested by david

* move opening paragraph to release note

* reorg content, add a bit about local deadlock

* guc can be reloaded

* concurrent update AND DELETE