Spell out VACUUM ANALYZE rather than using a pointless, and silly,
shortening. No functional change.
Reviewed-by: NGeorgios Kokolatos <gkokolatos@pivotal.io>
Reviewed-by: NJimmy Yih <jyih@pivotal.io>

This reintroduces commit ba3eb5b4, which was reverted in 659f0ee5.

After the WALrep changes, the previous -F (filespace) option was
co-opted to be the new standby data directory option. This isn't a
particularly obvious association.

Change the option to -S. (-D would have been better, but that's already
in use as a short alias for --debug.) Also document this option in the
official gpinitstandby help.

This reintroduces commit c9c3c351, which was reverted in 659f0ee5.

When a standby is initialized on the same host as the original master,
remind the user that the data directory and port need to be explicitly
set.

Commit 6610b941 removed the use of createTemplate() from gpexpand.
There are no more callers, and as that commit pointed out, the
implementation is unsafe. We can also get rid of DiskUsage and
LocalDirCopy.
Co-authored-by: NJamie McAtamney <jmcatamney@pivotal.io>

The function was using an anti-pattern where an argument is a static
length char array. Array arguments in C don't really exist but compilers
accept them and without any 'const' or 'static' decorators they don't
always emit a warning.

The proposed patch only fixes the unsafeness of the function
declaration but it does not address the usefulness of it.
Reviewed-by: NDaniel Gustafsson <dgustafsson@pivotal.io>

Make sure ereport() calls start with a lower case letter, and don't
end with a period. Also remove superfluous mentions of Greenplum
Database from messages.

Reviewed-by: Heikki Linnakangas <hlinnakangas@pivotal.io>
Reviewed-by: NAdam Berlin <aberlin@pivotal.io>

Commit dfee2ff7 exposed workfile_mgr_cache_entries_get_copy()
which returns a structure containing WorkFileUsagePerQuery entries,
but the definition of WorkFileUsagePerQuery was not make public. A
forward declaration was placed in the workfile_mgr.h file but it
suffered from a redefinition warning (in clang at least):

workfile_mgr.c:93:3: warning: redefinition of typedef
					 'WorkFileUsagePerQuery' is a C11 feature
					 [-Wtypedef-redefinition]
} WorkFileUsagePerQuery;
  ^
../../../../src/include/utils/workfile_mgr.h:25:38:
					note: previous definition is here
typedef struct WorkFileUsagePerQuery WorkFileUsagePerQuery;
                                     ^
Fix by moving the WorkFileUsagePerQuery definition to the header
from which is moved in f1ef3668. If we expose an API which
returns data in the struct we need to also make the definition
available to callers.
Reviewed-by: NGeorgios Kokolatos <gkokolatos@pivotal.io>
Reviewed-by: NTeng Zhang <tezhang@pivotal.io>

If ResQueueHashNew() returns NULL it means that the processing ran
out of memory and we need to error out. This is a situation which
clearly can happen in production, so we need to upgrade the assert
to a runtime check which properly errors out instead of fail on a
subsequent NULL pointer dereference.
Reviewed-by: NGeorgios Kokolatos <gkokolatos@pivotal.io>

* docs - GDD - add information about specific type of deadlocks.

* docs - GDD - update deadlock information based on review comments.

* typo fix

* Small edit for clarity

* Removing duplicate info

This commit is part of Add Partitioned Indexes #7047.

For the partitioned index PR#7047, we add tests for the use of internal
auto dependencies in both the existing index-backed constraints and in
standalone partitioned indexes.
Co-authored-by: NMark Sliva <msliva@pivotal.io>

This commit is part of Add Partitioned Indexes #7047.

AttachPartitionEnsureIndexes iterates through all the indexes on an
incoming partition and adds INTERNAL_AUTO dependencies to the ones match
the definition of the parent partition's partitioned indexes.

These are all the tests that broke when we started exchanging/testing
for regular indexes on partitioned tables.
Co-authored-by: NKalen Krempely <kkrempely@pivotal.io>

This commit is part of Add Partitioned Indexes #7047.

Remove 'fix_ill_named_constraint' query from 'part_constraint' test.
This query expects all UNIQUE and PRIMARY KEY constraints on partitioned
tables to have identical names. This is no longer true after enforcing
UNIQUE and PRIMARY KEY constraints to have the same name as the
underlying index. These tests will need to be replaced with ones that
validate the integrity of the pg_depend entries for partitioned indexes
and constraints.
Co-authored-by: NKalen Krempely <kkrempely@pivotal.io>

This commit is part of Add Partitioned Indexes #7047.

After adding INTERNAL_AUTO, and dependencies between partitioned indexs,
many tests that assumed that we need to manually delete indexes added to
leaf partitions need updating.
Co-authored-by: NKalen Krempely <kkrempely@pivotal.io>

This commit is part of Add Partitioned Indexes #7047.

Tests were added to verify:
 - Index backed constraint names have matching index name
 - Constraints on partition tables including ADD PARTITION and EXCHANGE
   PARTITION.
 - Constraints and indexes can be upgraded. This includes testing directly in
   pg_regress, or creating tables to be used by pg_upgrade.
Co-authored-by: NTaylor Vesely <tvesely@pivotal.io>
Co-authored-by: NKalen Krempely <kkrempely@pivotal.io>
Co-authored-by: NJesse Zhang <sbjesse@gmail.com>

This commit is part of Add Partitioned Indexes #7047.

Constraint names must now match their index. Fix ICW tests where this
assumption no longer holds.
Co-authored-by: NKalen Krempely <kkrempely@pivotal.io>

This commit is part of Add Partitioned Indexes #7047.

Don't allow check constraints to be dropped on partition leaves. Only
allow them to be dropped from the root.

This restriction had been relaxed on PRIMARY KEY and UNIQUE constraints
after adding INTERNAL_AUTO dependencies to constraints. However, we need
to maintain the restiction on check constrants because they are
currently not being protected by pg_depend.
Co-authored-by: NKalen Krempely <kkrempely@pivotal.io>

This commit is part of Add Partitioned Indexes #7047.

Until indexes can be upgraded drop all indexes. Drop from the root
partition instead of on intermediate/leaf partitions, since they now
cascade due to their INTERNAL_AUTO dependencies.
Co-authored-by: NKalen Krempely <kkrempely@pivotal.io>

This commit is part of Add Partitioned Indexes #7047.

During exchange and split partition update the pg_depend relationships and
exchange constraint names.

AttachPartitionEnsureIndexes was cherry-picked from:
commit 8b08f7d4
Author: Alvaro Herrera <alvherre@alvh.no-ip.org>
Date:   Fri Jan 19 11:49:22 2018 -0300

    Local partitioned indexes

    When CREATE INDEX is run on a partitioned table, create catalog entries
    for an index on the partitioned table (which is just a placeholder since
    the table proper has no data of its own), and recurse to create actual
    indexes on the existing partitions; create them in future partitions
    also.

    As a convenience gadget, if the new index definition matches some
    existing index in partitions, these are picked up and used instead of
    creating new ones.  Whichever way these indexes come about, they become
    attached to the index on the parent table and are dropped alongside it,
    and cannot be dropped on isolation unless they are detached first.

    To support pg_dump'ing these indexes, add commands
        CREATE INDEX ON ONLY <table>
    (which creates the index on the parent partitioned table, without
    recursing) and
        ALTER INDEX ATTACH PARTITION
    (which is used after the indexes have been created individually on each
    partition, to attach them to the parent index).  These reconstruct prior
    database state exactly.

    Reviewed-by: (in alphabetical order) Peter Eisentraut, Robert Haas, Amit
            Langote, Jesper Pedersen, Simon Riggs, David Rowley
    Discussion: https://postgr.es/m/20171113170646.gzweigyrgg6pwsg4@alvherre.pgsqlCo-authored-by: NKalen Krempely <kkrempely@pivotal.io>

This commit adds partitioned indexes from upstream Postgres. This commit is
mostly cherry-picked from Postgres 11 and bug fixes from Postgres 12.

Differences from upstream:
 - Postgres has two additional relkind's - RELKIND_PARTITIONED_TABLE and
   RELKIND_PARTITIONED_INDEX, which have no on disk storage. Greenplum does not
   have these additional relkinds. Thus, partitioned indexes have physical
   storage.
 - CREATE INDEX ON ONLY <table> DDL has not yet been implemented
 - ALTER INDEX ATTACH PARTITION DDL has not yet been implemented

Constraint changes:
 - Constraints and their backing index have the same names. Thus, partitions of
   a table no longer share the same constraint name, and are instead related to
   their parent via INTERNAL_AUTO dependencies.

Index changes:
 - Child partition indexes can no longer be directly dropped, and must be
   dropped from their root. This includes mid-level and leaf indexes.
 - Adding indexes to mid-level partitions cascade to their children.

These changes are mostly cherry-picked from:
commit 8b08f7d4
Author: Alvaro Herrera <alvherre@alvh.no-ip.org>
Date:   Fri Jan 19 11:49:22 2018 -0300

    Local partitioned indexes

    When CREATE INDEX is run on a partitioned table, create catalog entries
    for an index on the partitioned table (which is just a placeholder since
    the table proper has no data of its own), and recurse to create actual
    indexes on the existing partitions; create them in future partitions
    also.

    As a convenience gadget, if the new index definition matches some
    existing index in partitions, these are picked up and used instead of
    creating new ones.  Whichever way these indexes come about, they become
    attached to the index on the parent table and are dropped alongside it,
    and cannot be dropped on isolation unless they are detached first.

    To support pg_dump'ing these indexes, add commands
        CREATE INDEX ON ONLY <table>
    (which creates the index on the parent partitioned table, without
    recursing) and
        ALTER INDEX ATTACH PARTITION
    (which is used after the indexes have been created individually on each
    partition, to attach them to the parent index).  These reconstruct prior
    database state exactly.

    Reviewed-by: (in alphabetical order) Peter Eisentraut, Robert Haas, Amit
            Langote, Jesper Pedersen, Simon Riggs, David Rowley
    Discussion: https://postgr.es/m/20171113170646.gzweigyrgg6pwsg4@alvherre.pgsql

Changes were also cherry-picked from the following Postgres commits:
  eb7ed3f3 - Allow UNIQUE indexes on partitioned tables
  ae366aa5 - Detach constraints when partitions are detached
  19184fcc - Simplify coding to detach constraints when detaching partition
  c7d43c4d - Correct attach/detach logic for FKs in partitions
  17f206fb - Set pg_class.relhassubclass for partitioned indexes
Co-authored-by: NKalen Krempely <kkrempely@pivotal.io>

Changing version in path from /600 to /6-0 for consistency with other docs and to recognize that only the minor version digit is significant for the doc build (#7141)

* Bump ORCA version to 3.29.0

* ORCA: Updating subquery plans in ICG expected files

This change is needed for ORCA PR https://github.com/greenplum-db/gporca/pull/449.
Some subquery plans changed in minor ways in the ICG test.
Co-authored-by: NChris Hajas <chajas@pivotal.io>

Currently, a randomly distributed table cannot be created with a
primary key or unique index. We should put this restriction for ALTER
TABLE SET DISTRIBUTED RANDOMLY as well. This was caught by gpcheckcat
distribution_policy check.
Co-authored-by: NAlexandra Wang <lewang@pivotal.io>

The Greenplum-specific relationHasUniqueIndex() function is similar to
the already existing relationHasPrimaryKey() function. We should make
it available for extern use, and put them closer since they could be
used together sometimes.
Co-authored-by: NAlexandra Wang <lewang@pivotal.io>

The gp-addon repo was used to provide the quick-lz artifacts but this
has been moved in to the Docker images used for building GPDB6.
Co-authored-by: NSambitesh Dash <sdash@pivotal.io>
Co-authored-by: NBradford D. Boyle <bboyle@pivotal.io>

Not user-facing documentation changed, only code comments. Spotted
while reading code.

The range partition handling in get_part_rule() seemed to have some
parts copypasted from the list handling, and subsequently ended up
looking at and setting variables never to be read. Remove the unused
code and also ensure to exit early in case we weren't called with
the correct data.
Reviewed-by: NGeorgios Kokolatos <gkokolatos@pivotal.io>

Saving the partitioning type into a string was done with an elaborate
switch() only for the string to be consumed in a single errorpath.
Rather than spending cycles and memory for a rare error, inline the
partition type string handling into the errorpath.
Reviewed-by: NGeorgios Kokolatos <gkokolatos@pivotal.io>

get_part_rule() must only return NULL for this addition to make any
sense to continue with, but we silently ignored when it didn't with
a rather odd codepath. Since get_part_rule() is defined to always
error out on the conditions we don't want, add an assertion for this
and reindent the code to rely on it doing the right thing.

Also discard unused returnvalues.
Reviewed-by: NGeorgios Kokolatos <gkokolatos@pivotal.io>

When altering a partitioned table and adding an incorrectly specified
partition, an assertion was hit rather than gracefully erroring out.
Make sure that the requested partition matches the underlying table
definition before continuing down into the altering code. This also
adds a testcase for this.

Reported-by: Kalen Krempely in #6967
Reviewed-by: NPaul Guo <pguo@pivotal.io>
Reviewed-by: NGeorgios Kokolatos <gkokolatos@pivotal.io>

The GUC which enables recursive CTEs is in the currently released
version called gp_recursive_cte_prototype, but in order to reflect
the current state of the code it's now renamed to gp_recursive_cte.
By default the GUC is still off, but that might change before we
ship the next release.

The previous GUC name is still supported, but marked as deprecated,
in order to make upgrades easier.
Reviewed-by: NIvan Novick <inovick@pivotal.io>
Reviewed-by: NGeorgios Kokolatos <gkokolatos@pivotal.io>

This method was introduced to improve the data redistribution
performance during gpexpand phase2, however per benchmark results the
effect does not reach our expectation.  For example when expanding a
table from 7 segments to 8 segments the reshuffle method is only 30%
faster than the traditional CTAS method, when expanding from 4 to 8
segments reshuffle is even 10% slower than CTAS.  When there are indexes
on the table the reshuffle performance can be worse, and extra VACUUM is
needed to actually free the disk space.  According to our experiments
the bottleneck of reshuffle method is on the tuple deletion operation,
it is much slower than the insertion operation used by CTAS.

The reshuffle method does have some benefits, it requires less extra
disk space, it also requires less network bandwidth (similar to CTAS
method with the new JCH reduce method, but less than CTAS + MOD).  And
it can be faster in some cases, however as we can not automatically
determine when it is faster it is not easy to get benefit from it in
practice.

On the other side the reshuffle method is less tested, it is possible to
have bugs in corner cases, so it is not production ready yet.

In such a case we decided to retire it entirely for now, we might add it
back in the future if we can get rid of the slow deletion or find out
reliable ways to automatically choose between reshuffle and ctas
methods.

Discussion: https://groups.google.com/a/greenplum.org/d/msg/gpdb-dev/8xknWag-SkI/5OsIhZWdDgAJReviewed-by: NHeikki Linnakangas <hlinnakangas@pivotal.io>
Reviewed-by: NAshwin Agrawal <aagrawal@pivotal.io>

Previously, during initializing ResultRelations in InitPlan on
QD, it always builds the relids as all the relation oids in a
partition table (including root and all its inheritors).
Sometimes we does not need all the relids.

A typical case is for ao partition table. When we directly
insert into a specific child partition, the plan's ResultRelation
only contains the child partition. And if we still make relids
as root and all its inheritors, during `assignPerRelSegno`,
it might lock each aoseg file on AccessShare mode on QEs. It
causes confusion that the insert statement is only for a child
partition but holding other partition's lock.

This commit changes the relids building logic as:
  - if the ResultRelation contains the root partition, then
    relids is root and all its inheritors
  - otherwise, relids is a map of ResultRelations to get the
    element's relation oid

Make sure all ereports() starts with a lowercase letter, and move
longer explanations to errdetail/errhints. Also fix expected error
output to match.
Reviewed-by: NAdam Berlin <aberlin@pivotal.io>
Reviewed-by: NJacob Champion <pchampion@pivotal.io>

While harmless, knowing it's there is enough and I can't unsee it.
Also reduce the scope of the variable as it has no outside users.
Co-authored-by: NJacob Champion <pchampion@pivotal.io>
Reviewed-by: NJacob Champion <pchampion@pivotal.io>

Make sure there is whitespace before xref tags when used in normal
text to avoid links squashed together with text.
Reviewed-by: NLisa Owen <lowen@pivotal.io>

One of these changes appears to have possibly introduced a serious
performance regression in the master pipeline. To avoid destabilizing
work over the weekend, I'm reverting for now and we can investigate more
fully next week.

This reverts the following commits:
"gprecoverseg: Show progress of pg_basebackup on each segment"
    1b38c6e8
"Add gprecoverseg -s to show progress sequentially"
    9e89b5ad
"gpinitstandby: guide the user on single-host systems"
    c9c3c351
"gpinitstandby: rename -F to -S and document it"
    ba3eb5b4

* docs - add pxf config info for s3 server-side encryption

* add xref to aws docs for setting bucket encryption

* create bucket and key in same AZ, section title formatting

* remove number from bucket name variable

* s3 config - xref to hadoop s3a prop section

* copy edits requested by david

* add the bucket variable numbers back in