This commit is part of Add Partitioned Indexes #7047.

Constraint names must now match their index. Fix ICW tests where this
assumption no longer holds.
Co-authored-by: NKalen Krempely <kkrempely@pivotal.io>

This commit is part of Add Partitioned Indexes #7047.

Don't allow check constraints to be dropped on partition leaves. Only
allow them to be dropped from the root.

This restriction had been relaxed on PRIMARY KEY and UNIQUE constraints
after adding INTERNAL_AUTO dependencies to constraints. However, we need
to maintain the restiction on check constrants because they are
currently not being protected by pg_depend.
Co-authored-by: NKalen Krempely <kkrempely@pivotal.io>

This commit is part of Add Partitioned Indexes #7047.

Until indexes can be upgraded drop all indexes. Drop from the root
partition instead of on intermediate/leaf partitions, since they now
cascade due to their INTERNAL_AUTO dependencies.
Co-authored-by: NKalen Krempely <kkrempely@pivotal.io>

This commit is part of Add Partitioned Indexes #7047.

During exchange and split partition update the pg_depend relationships and
exchange constraint names.

AttachPartitionEnsureIndexes was cherry-picked from:
commit 8b08f7d4
Author: Alvaro Herrera <alvherre@alvh.no-ip.org>
Date:   Fri Jan 19 11:49:22 2018 -0300

    Local partitioned indexes

    When CREATE INDEX is run on a partitioned table, create catalog entries
    for an index on the partitioned table (which is just a placeholder since
    the table proper has no data of its own), and recurse to create actual
    indexes on the existing partitions; create them in future partitions
    also.

    As a convenience gadget, if the new index definition matches some
    existing index in partitions, these are picked up and used instead of
    creating new ones.  Whichever way these indexes come about, they become
    attached to the index on the parent table and are dropped alongside it,
    and cannot be dropped on isolation unless they are detached first.

    To support pg_dump'ing these indexes, add commands
        CREATE INDEX ON ONLY <table>
    (which creates the index on the parent partitioned table, without
    recursing) and
        ALTER INDEX ATTACH PARTITION
    (which is used after the indexes have been created individually on each
    partition, to attach them to the parent index).  These reconstruct prior
    database state exactly.

    Reviewed-by: (in alphabetical order) Peter Eisentraut, Robert Haas, Amit
            Langote, Jesper Pedersen, Simon Riggs, David Rowley
    Discussion: https://postgr.es/m/20171113170646.gzweigyrgg6pwsg4@alvherre.pgsqlCo-authored-by: NKalen Krempely <kkrempely@pivotal.io>

This commit adds partitioned indexes from upstream Postgres. This commit is
mostly cherry-picked from Postgres 11 and bug fixes from Postgres 12.

Differences from upstream:
 - Postgres has two additional relkind's - RELKIND_PARTITIONED_TABLE and
   RELKIND_PARTITIONED_INDEX, which have no on disk storage. Greenplum does not
   have these additional relkinds. Thus, partitioned indexes have physical
   storage.
 - CREATE INDEX ON ONLY <table> DDL has not yet been implemented
 - ALTER INDEX ATTACH PARTITION DDL has not yet been implemented

Constraint changes:
 - Constraints and their backing index have the same names. Thus, partitions of
   a table no longer share the same constraint name, and are instead related to
   their parent via INTERNAL_AUTO dependencies.

Index changes:
 - Child partition indexes can no longer be directly dropped, and must be
   dropped from their root. This includes mid-level and leaf indexes.
 - Adding indexes to mid-level partitions cascade to their children.

These changes are mostly cherry-picked from:
commit 8b08f7d4
Author: Alvaro Herrera <alvherre@alvh.no-ip.org>
Date:   Fri Jan 19 11:49:22 2018 -0300

    Local partitioned indexes

    When CREATE INDEX is run on a partitioned table, create catalog entries
    for an index on the partitioned table (which is just a placeholder since
    the table proper has no data of its own), and recurse to create actual
    indexes on the existing partitions; create them in future partitions
    also.

    As a convenience gadget, if the new index definition matches some
    existing index in partitions, these are picked up and used instead of
    creating new ones.  Whichever way these indexes come about, they become
    attached to the index on the parent table and are dropped alongside it,
    and cannot be dropped on isolation unless they are detached first.

    To support pg_dump'ing these indexes, add commands
        CREATE INDEX ON ONLY <table>
    (which creates the index on the parent partitioned table, without
    recursing) and
        ALTER INDEX ATTACH PARTITION
    (which is used after the indexes have been created individually on each
    partition, to attach them to the parent index).  These reconstruct prior
    database state exactly.

    Reviewed-by: (in alphabetical order) Peter Eisentraut, Robert Haas, Amit
            Langote, Jesper Pedersen, Simon Riggs, David Rowley
    Discussion: https://postgr.es/m/20171113170646.gzweigyrgg6pwsg4@alvherre.pgsql

Changes were also cherry-picked from the following Postgres commits:
  eb7ed3f3 - Allow UNIQUE indexes on partitioned tables
  ae366aa5 - Detach constraints when partitions are detached
  19184fcc - Simplify coding to detach constraints when detaching partition
  c7d43c4d - Correct attach/detach logic for FKs in partitions
  17f206fb - Set pg_class.relhassubclass for partitioned indexes
Co-authored-by: NKalen Krempely <kkrempely@pivotal.io>

Changing version in path from /600 to /6-0 for consistency with other docs and to recognize that only the minor version digit is significant for the doc build (#7141)

* Bump ORCA version to 3.29.0

* ORCA: Updating subquery plans in ICG expected files

This change is needed for ORCA PR https://github.com/greenplum-db/gporca/pull/449.
Some subquery plans changed in minor ways in the ICG test.
Co-authored-by: NChris Hajas <chajas@pivotal.io>

Currently, a randomly distributed table cannot be created with a
primary key or unique index. We should put this restriction for ALTER
TABLE SET DISTRIBUTED RANDOMLY as well. This was caught by gpcheckcat
distribution_policy check.
Co-authored-by: NAlexandra Wang <lewang@pivotal.io>

The Greenplum-specific relationHasUniqueIndex() function is similar to
the already existing relationHasPrimaryKey() function. We should make
it available for extern use, and put them closer since they could be
used together sometimes.
Co-authored-by: NAlexandra Wang <lewang@pivotal.io>

The gp-addon repo was used to provide the quick-lz artifacts but this
has been moved in to the Docker images used for building GPDB6.
Co-authored-by: NSambitesh Dash <sdash@pivotal.io>
Co-authored-by: NBradford D. Boyle <bboyle@pivotal.io>

Not user-facing documentation changed, only code comments. Spotted
while reading code.

The range partition handling in get_part_rule() seemed to have some
parts copypasted from the list handling, and subsequently ended up
looking at and setting variables never to be read. Remove the unused
code and also ensure to exit early in case we weren't called with
the correct data.
Reviewed-by: NGeorgios Kokolatos <gkokolatos@pivotal.io>

Saving the partitioning type into a string was done with an elaborate
switch() only for the string to be consumed in a single errorpath.
Rather than spending cycles and memory for a rare error, inline the
partition type string handling into the errorpath.
Reviewed-by: NGeorgios Kokolatos <gkokolatos@pivotal.io>

get_part_rule() must only return NULL for this addition to make any
sense to continue with, but we silently ignored when it didn't with
a rather odd codepath. Since get_part_rule() is defined to always
error out on the conditions we don't want, add an assertion for this
and reindent the code to rely on it doing the right thing.

Also discard unused returnvalues.
Reviewed-by: NGeorgios Kokolatos <gkokolatos@pivotal.io>

When altering a partitioned table and adding an incorrectly specified
partition, an assertion was hit rather than gracefully erroring out.
Make sure that the requested partition matches the underlying table
definition before continuing down into the altering code. This also
adds a testcase for this.

Reported-by: Kalen Krempely in #6967
Reviewed-by: NPaul Guo <pguo@pivotal.io>
Reviewed-by: NGeorgios Kokolatos <gkokolatos@pivotal.io>

The GUC which enables recursive CTEs is in the currently released
version called gp_recursive_cte_prototype, but in order to reflect
the current state of the code it's now renamed to gp_recursive_cte.
By default the GUC is still off, but that might change before we
ship the next release.

The previous GUC name is still supported, but marked as deprecated,
in order to make upgrades easier.
Reviewed-by: NIvan Novick <inovick@pivotal.io>
Reviewed-by: NGeorgios Kokolatos <gkokolatos@pivotal.io>

This method was introduced to improve the data redistribution
performance during gpexpand phase2, however per benchmark results the
effect does not reach our expectation.  For example when expanding a
table from 7 segments to 8 segments the reshuffle method is only 30%
faster than the traditional CTAS method, when expanding from 4 to 8
segments reshuffle is even 10% slower than CTAS.  When there are indexes
on the table the reshuffle performance can be worse, and extra VACUUM is
needed to actually free the disk space.  According to our experiments
the bottleneck of reshuffle method is on the tuple deletion operation,
it is much slower than the insertion operation used by CTAS.

The reshuffle method does have some benefits, it requires less extra
disk space, it also requires less network bandwidth (similar to CTAS
method with the new JCH reduce method, but less than CTAS + MOD).  And
it can be faster in some cases, however as we can not automatically
determine when it is faster it is not easy to get benefit from it in
practice.

On the other side the reshuffle method is less tested, it is possible to
have bugs in corner cases, so it is not production ready yet.

In such a case we decided to retire it entirely for now, we might add it
back in the future if we can get rid of the slow deletion or find out
reliable ways to automatically choose between reshuffle and ctas
methods.

Discussion: https://groups.google.com/a/greenplum.org/d/msg/gpdb-dev/8xknWag-SkI/5OsIhZWdDgAJReviewed-by: NHeikki Linnakangas <hlinnakangas@pivotal.io>
Reviewed-by: NAshwin Agrawal <aagrawal@pivotal.io>

Previously, during initializing ResultRelations in InitPlan on
QD, it always builds the relids as all the relation oids in a
partition table (including root and all its inheritors).
Sometimes we does not need all the relids.

A typical case is for ao partition table. When we directly
insert into a specific child partition, the plan's ResultRelation
only contains the child partition. And if we still make relids
as root and all its inheritors, during `assignPerRelSegno`,
it might lock each aoseg file on AccessShare mode on QEs. It
causes confusion that the insert statement is only for a child
partition but holding other partition's lock.

This commit changes the relids building logic as:
  - if the ResultRelation contains the root partition, then
    relids is root and all its inheritors
  - otherwise, relids is a map of ResultRelations to get the
    element's relation oid

Make sure all ereports() starts with a lowercase letter, and move
longer explanations to errdetail/errhints. Also fix expected error
output to match.
Reviewed-by: NAdam Berlin <aberlin@pivotal.io>
Reviewed-by: NJacob Champion <pchampion@pivotal.io>

While harmless, knowing it's there is enough and I can't unsee it.
Also reduce the scope of the variable as it has no outside users.
Co-authored-by: NJacob Champion <pchampion@pivotal.io>
Reviewed-by: NJacob Champion <pchampion@pivotal.io>

Make sure there is whitespace before xref tags when used in normal
text to avoid links squashed together with text.
Reviewed-by: NLisa Owen <lowen@pivotal.io>

One of these changes appears to have possibly introduced a serious
performance regression in the master pipeline. To avoid destabilizing
work over the weekend, I'm reverting for now and we can investigate more
fully next week.

This reverts the following commits:
"gprecoverseg: Show progress of pg_basebackup on each segment"
    1b38c6e8
"Add gprecoverseg -s to show progress sequentially"
    9e89b5ad
"gpinitstandby: guide the user on single-host systems"
    c9c3c351
"gpinitstandby: rename -F to -S and document it"
    ba3eb5b4

* docs - add pxf config info for s3 server-side encryption

* add xref to aws docs for setting bucket encryption

* create bucket and key in same AZ, section title formatting

* remove number from bucket name variable

* s3 config - xref to hadoop s3a prop section

* copy edits requested by david

* add the bucket variable numbers back in

* docs - CTE available with INSERT, UPDATE, DELETE

-updated GUC
-updated Admin Guide topic WITH Queries (Common Table Expressions)

updates to SELECT, INSERT, UPDATE DELETE will be part of postgres 9.2 merge.

* docs - CTE updates from review comments

* docs - CTE more updates from review comments

* docs - CTE - updates from review comments

* Experimental -> Beta wording

* vacuumdb - add -t option support for multiple tables

* reindexdb - add -t option support for multiple tables; misc edits

* psql - misc additions, edits, and reformats

* pg_dump - misc edits, additions, formatting

* add * syntax for descendant tables

* pg_dumpall - add -d, --dbname option

* PREPARE - add note about search_path

* TRUNCATE - add * option

* SELECT - add LATERAL, NO KEY UPDATE, KEY SHARE, and related edits to locking clause

* ALTER ROLE. Re-order syntax and descriptions.

* ALTER RULE. Add new command and fix some conref issues

* COPY. Add FREEZE option and edits.

* CREATE INDEX. Minor synopsis update.

* CREATE SCHEMA. Add IF NOT EXISTS forms of command

* CREATE TABLE. Syntax changes. Add SET CONSTRAINTS command.

* bgworker. Add bgworker topic to reference guide

* CREATE TABLE AS. Add WITH/WITHOUT OIDS

* Changes from review

* remove some parens, add trigger note

* remove extra whitespace in CREATE TABLE

* CREATE TABLE AS - correct DISTRIBUTED BY syntax

* Review comments on CREATE TABLE.

* pg_dump - add parallelization note

* CREATE TABLE - fix whitespace again?

* Remove docs for LATERAL

* Add LATERAL to unsupported feature list

After the WALrep changes, the previous -F (filespace) option was
co-opted to be the new standby data directory option. This isn't a
particularly obvious association.

Change the option to -S. (-D would have been better, but that's already
in use as a short alias for --debug.) Also document this option in the
official gpinitstandby help.

When a standby is initialized on the same host as the original master,
remind the user that the data directory and port need to be explicitly
set.

When -s is present, show pg_basebackup progress sequentially instead
of inplace. Useful when writing to a file, or if a tty does not support
escape sequences. Defaults to showing the progress inplace.

The gprecoverseg utility runs pg_basebackup in parallel on all segments that are
being recovered.  In this commit, we are logging the progress of each
pg_basebackup on its host and displaying them to the user of gprecoverseg.  The
progress files are deleted upons successful completion of gprecoverseg.

Unit tests have also been added.
Authored-by: NShoaib Lari <slari@pivotal.io>
Co-authored-by: NMark Sliva <msliva@pivotal.io>
Co-authored-by: NJacob Champion <pchampion@pivotal.io>
Co-authored-by: NEd Espino <edespino@pivotal.io>
Co-authored-by: NKalen Krempely <kkrempely@pivotal.io>

* reword from 'experimental' to 'beta'

* Experimental -> Beta in markdown source

* typo fix

* Removing SuSE 11 details in Beta notes

Removes a merge fixme
Reviewed-by: NHeikki Linnakangas <hlinnakangas@pivotal.io>

We had three test modules to test the gppc partner connector library.
All three were the same in the way the test module was built and called, so
we might as well merge them into a single module to simplify things. This
also makes the tests run faster, as we no longer create a separate database
to test each module, but just one. Leaving behind fewer databases should
also speed up the later stages in installcheck-world, like gpcheckcat,
gp_replica_check, and pg_upgrade, run faster, because we leave behind fewer
databases for them to process.

There was a lot of identical test code and tests among the three modules.
Remove the duplication while we're at it.

The byteafunc2 test function was intentionally changing the pass-by-ref
argument, which is a big no-no. It doesn't make much sense to test that,
because that's a programmer error, and all bets are off. Change the test to
do something more interesting.

Normal client connection to GPDB server will be represented as
MyProcPort struct in QD. This is also true for connection from QD to QE.
But for background worker and auxiliary process such as global deadlock
detector, there is no client concept, and thus MyProcPort is set to NULL.

For example, dispatcherAYT() will be called to check QD-Client connection.
But there is no client in bgworker case. adjustMasterRouting() should be called 
on QE, so assert check MyProcPort is not NULL.

Build mode does not use any information from the cluster and does not
affect its running status, in fact it does not require a cluster exists
at all.
Co-authored-by: NHubert Zhang <hzhang@pivotal.io>
Co-authored-by: NNing Yu <nyu@pivotal.io>

After the changes to gp_distribution_policy to add distribution opclass,
the distribution policy checks in gpcheckcat need to be updated to
consider the distclass column as well.

Additionally, the old repair action that was taken as part of this check
is no longer valid, since we can't redistribute a child partition
directly. Instead, leave the correct action up to the DB admin.
Co-authored-by: NJimmy Yih <jyih@pivotal.io>
Co-authored-by: NTaylor Vesely <tvesely@pivotal.io>

The citext test was altered in the past to add a distribution column
other than citext. Since we now allow for extension datatypes to be
distribution columns with the new hashing work, remove the workaround
and re-align the test with upstream.

This also updates the testdata from upstream to work in Greenplum.
Reviewed-by: NHeikki Linnakangas <hlinnakangas@pivotal.io>

Previously, analyzedb would run `analyze rootpartition <tablename>` on
root partitions. However, with the introduction of merging leaf
statistics into the root in 9c1b1ae3, this command is no longer
necessary. The root stats are automatically updated when analyzing the leaf
partitions, and thus this command is unnecessary.

As a result, the --skip_root_stats flag has been removed since this flag
is now the default behavior.

The deleted tests in this commit were redundant, they simply added the
`skip-root-stats` flag to copies of other scenarios. This coverage was added in
existing scenarios.
Authored-by: NChris Hajas <chajas@pivotal.io>

We are in the process of renaming the secrets files in
gp-continuous-integration to use the .prod suffix consistently.
Co-authored-by: NBen Christel <bchristel@pivotal.io>
Co-authored-by: NSambitesh Dash <sdash@pivotal.io>