* Bump ORCA version to 3.29.0

* ORCA: Updating subquery plans in ICG expected files

This change is needed for ORCA PR https://github.com/greenplum-db/gporca/pull/449.
Some subquery plans changed in minor ways in the ICG test.
Co-authored-by: NChris Hajas <chajas@pivotal.io>

Currently, a randomly distributed table cannot be created with a
primary key or unique index. We should put this restriction for ALTER
TABLE SET DISTRIBUTED RANDOMLY as well. This was caught by gpcheckcat
distribution_policy check.
Co-authored-by: NAlexandra Wang <lewang@pivotal.io>

The Greenplum-specific relationHasUniqueIndex() function is similar to
the already existing relationHasPrimaryKey() function. We should make
it available for extern use, and put them closer since they could be
used together sometimes.
Co-authored-by: NAlexandra Wang <lewang@pivotal.io>

The gp-addon repo was used to provide the quick-lz artifacts but this
has been moved in to the Docker images used for building GPDB6.
Co-authored-by: NSambitesh Dash <sdash@pivotal.io>
Co-authored-by: NBradford D. Boyle <bboyle@pivotal.io>

Not user-facing documentation changed, only code comments. Spotted
while reading code.

The range partition handling in get_part_rule() seemed to have some
parts copypasted from the list handling, and subsequently ended up
looking at and setting variables never to be read. Remove the unused
code and also ensure to exit early in case we weren't called with
the correct data.
Reviewed-by: NGeorgios Kokolatos <gkokolatos@pivotal.io>

Saving the partitioning type into a string was done with an elaborate
switch() only for the string to be consumed in a single errorpath.
Rather than spending cycles and memory for a rare error, inline the
partition type string handling into the errorpath.
Reviewed-by: NGeorgios Kokolatos <gkokolatos@pivotal.io>

get_part_rule() must only return NULL for this addition to make any
sense to continue with, but we silently ignored when it didn't with
a rather odd codepath. Since get_part_rule() is defined to always
error out on the conditions we don't want, add an assertion for this
and reindent the code to rely on it doing the right thing.

Also discard unused returnvalues.
Reviewed-by: NGeorgios Kokolatos <gkokolatos@pivotal.io>

When altering a partitioned table and adding an incorrectly specified
partition, an assertion was hit rather than gracefully erroring out.
Make sure that the requested partition matches the underlying table
definition before continuing down into the altering code. This also
adds a testcase for this.

Reported-by: Kalen Krempely in #6967
Reviewed-by: NPaul Guo <pguo@pivotal.io>
Reviewed-by: NGeorgios Kokolatos <gkokolatos@pivotal.io>

The GUC which enables recursive CTEs is in the currently released
version called gp_recursive_cte_prototype, but in order to reflect
the current state of the code it's now renamed to gp_recursive_cte.
By default the GUC is still off, but that might change before we
ship the next release.

The previous GUC name is still supported, but marked as deprecated,
in order to make upgrades easier.
Reviewed-by: NIvan Novick <inovick@pivotal.io>
Reviewed-by: NGeorgios Kokolatos <gkokolatos@pivotal.io>

This method was introduced to improve the data redistribution
performance during gpexpand phase2, however per benchmark results the
effect does not reach our expectation.  For example when expanding a
table from 7 segments to 8 segments the reshuffle method is only 30%
faster than the traditional CTAS method, when expanding from 4 to 8
segments reshuffle is even 10% slower than CTAS.  When there are indexes
on the table the reshuffle performance can be worse, and extra VACUUM is
needed to actually free the disk space.  According to our experiments
the bottleneck of reshuffle method is on the tuple deletion operation,
it is much slower than the insertion operation used by CTAS.

The reshuffle method does have some benefits, it requires less extra
disk space, it also requires less network bandwidth (similar to CTAS
method with the new JCH reduce method, but less than CTAS + MOD).  And
it can be faster in some cases, however as we can not automatically
determine when it is faster it is not easy to get benefit from it in
practice.

On the other side the reshuffle method is less tested, it is possible to
have bugs in corner cases, so it is not production ready yet.

In such a case we decided to retire it entirely for now, we might add it
back in the future if we can get rid of the slow deletion or find out
reliable ways to automatically choose between reshuffle and ctas
methods.

Discussion: https://groups.google.com/a/greenplum.org/d/msg/gpdb-dev/8xknWag-SkI/5OsIhZWdDgAJReviewed-by: NHeikki Linnakangas <hlinnakangas@pivotal.io>
Reviewed-by: NAshwin Agrawal <aagrawal@pivotal.io>

Previously, during initializing ResultRelations in InitPlan on
QD, it always builds the relids as all the relation oids in a
partition table (including root and all its inheritors).
Sometimes we does not need all the relids.

A typical case is for ao partition table. When we directly
insert into a specific child partition, the plan's ResultRelation
only contains the child partition. And if we still make relids
as root and all its inheritors, during `assignPerRelSegno`,
it might lock each aoseg file on AccessShare mode on QEs. It
causes confusion that the insert statement is only for a child
partition but holding other partition's lock.

This commit changes the relids building logic as:
  - if the ResultRelation contains the root partition, then
    relids is root and all its inheritors
  - otherwise, relids is a map of ResultRelations to get the
    element's relation oid

Make sure all ereports() starts with a lowercase letter, and move
longer explanations to errdetail/errhints. Also fix expected error
output to match.
Reviewed-by: NAdam Berlin <aberlin@pivotal.io>
Reviewed-by: NJacob Champion <pchampion@pivotal.io>

While harmless, knowing it's there is enough and I can't unsee it.
Also reduce the scope of the variable as it has no outside users.
Co-authored-by: NJacob Champion <pchampion@pivotal.io>
Reviewed-by: NJacob Champion <pchampion@pivotal.io>

Make sure there is whitespace before xref tags when used in normal
text to avoid links squashed together with text.
Reviewed-by: NLisa Owen <lowen@pivotal.io>

One of these changes appears to have possibly introduced a serious
performance regression in the master pipeline. To avoid destabilizing
work over the weekend, I'm reverting for now and we can investigate more
fully next week.

This reverts the following commits:
"gprecoverseg: Show progress of pg_basebackup on each segment"
    1b38c6e8
"Add gprecoverseg -s to show progress sequentially"
    9e89b5ad
"gpinitstandby: guide the user on single-host systems"
    c9c3c351
"gpinitstandby: rename -F to -S and document it"
    ba3eb5b4

* docs - add pxf config info for s3 server-side encryption

* add xref to aws docs for setting bucket encryption

* create bucket and key in same AZ, section title formatting

* remove number from bucket name variable

* s3 config - xref to hadoop s3a prop section

* copy edits requested by david

* add the bucket variable numbers back in

* docs - CTE available with INSERT, UPDATE, DELETE

-updated GUC
-updated Admin Guide topic WITH Queries (Common Table Expressions)

updates to SELECT, INSERT, UPDATE DELETE will be part of postgres 9.2 merge.

* docs - CTE updates from review comments

* docs - CTE more updates from review comments

* docs - CTE - updates from review comments

* Experimental -> Beta wording

* vacuumdb - add -t option support for multiple tables

* reindexdb - add -t option support for multiple tables; misc edits

* psql - misc additions, edits, and reformats

* pg_dump - misc edits, additions, formatting

* add * syntax for descendant tables

* pg_dumpall - add -d, --dbname option

* PREPARE - add note about search_path

* TRUNCATE - add * option

* SELECT - add LATERAL, NO KEY UPDATE, KEY SHARE, and related edits to locking clause

* ALTER ROLE. Re-order syntax and descriptions.

* ALTER RULE. Add new command and fix some conref issues

* COPY. Add FREEZE option and edits.

* CREATE INDEX. Minor synopsis update.

* CREATE SCHEMA. Add IF NOT EXISTS forms of command

* CREATE TABLE. Syntax changes. Add SET CONSTRAINTS command.

* bgworker. Add bgworker topic to reference guide

* CREATE TABLE AS. Add WITH/WITHOUT OIDS

* Changes from review

* remove some parens, add trigger note

* remove extra whitespace in CREATE TABLE

* CREATE TABLE AS - correct DISTRIBUTED BY syntax

* Review comments on CREATE TABLE.

* pg_dump - add parallelization note

* CREATE TABLE - fix whitespace again?

* Remove docs for LATERAL

* Add LATERAL to unsupported feature list

After the WALrep changes, the previous -F (filespace) option was
co-opted to be the new standby data directory option. This isn't a
particularly obvious association.

Change the option to -S. (-D would have been better, but that's already
in use as a short alias for --debug.) Also document this option in the
official gpinitstandby help.

When a standby is initialized on the same host as the original master,
remind the user that the data directory and port need to be explicitly
set.

When -s is present, show pg_basebackup progress sequentially instead
of inplace. Useful when writing to a file, or if a tty does not support
escape sequences. Defaults to showing the progress inplace.

The gprecoverseg utility runs pg_basebackup in parallel on all segments that are
being recovered.  In this commit, we are logging the progress of each
pg_basebackup on its host and displaying them to the user of gprecoverseg.  The
progress files are deleted upons successful completion of gprecoverseg.

Unit tests have also been added.
Authored-by: NShoaib Lari <slari@pivotal.io>
Co-authored-by: NMark Sliva <msliva@pivotal.io>
Co-authored-by: NJacob Champion <pchampion@pivotal.io>
Co-authored-by: NEd Espino <edespino@pivotal.io>
Co-authored-by: NKalen Krempely <kkrempely@pivotal.io>

* reword from 'experimental' to 'beta'

* Experimental -> Beta in markdown source

* typo fix

* Removing SuSE 11 details in Beta notes

Removes a merge fixme
Reviewed-by: NHeikki Linnakangas <hlinnakangas@pivotal.io>

We had three test modules to test the gppc partner connector library.
All three were the same in the way the test module was built and called, so
we might as well merge them into a single module to simplify things. This
also makes the tests run faster, as we no longer create a separate database
to test each module, but just one. Leaving behind fewer databases should
also speed up the later stages in installcheck-world, like gpcheckcat,
gp_replica_check, and pg_upgrade, run faster, because we leave behind fewer
databases for them to process.

There was a lot of identical test code and tests among the three modules.
Remove the duplication while we're at it.

The byteafunc2 test function was intentionally changing the pass-by-ref
argument, which is a big no-no. It doesn't make much sense to test that,
because that's a programmer error, and all bets are off. Change the test to
do something more interesting.

Normal client connection to GPDB server will be represented as
MyProcPort struct in QD. This is also true for connection from QD to QE.
But for background worker and auxiliary process such as global deadlock
detector, there is no client concept, and thus MyProcPort is set to NULL.

For example, dispatcherAYT() will be called to check QD-Client connection.
But there is no client in bgworker case. adjustMasterRouting() should be called 
on QE, so assert check MyProcPort is not NULL.

Build mode does not use any information from the cluster and does not
affect its running status, in fact it does not require a cluster exists
at all.
Co-authored-by: NHubert Zhang <hzhang@pivotal.io>
Co-authored-by: NNing Yu <nyu@pivotal.io>

After the changes to gp_distribution_policy to add distribution opclass,
the distribution policy checks in gpcheckcat need to be updated to
consider the distclass column as well.

Additionally, the old repair action that was taken as part of this check
is no longer valid, since we can't redistribute a child partition
directly. Instead, leave the correct action up to the DB admin.
Co-authored-by: NJimmy Yih <jyih@pivotal.io>
Co-authored-by: NTaylor Vesely <tvesely@pivotal.io>

The citext test was altered in the past to add a distribution column
other than citext. Since we now allow for extension datatypes to be
distribution columns with the new hashing work, remove the workaround
and re-align the test with upstream.

This also updates the testdata from upstream to work in Greenplum.
Reviewed-by: NHeikki Linnakangas <hlinnakangas@pivotal.io>

Previously, analyzedb would run `analyze rootpartition <tablename>` on
root partitions. However, with the introduction of merging leaf
statistics into the root in 9c1b1ae3, this command is no longer
necessary. The root stats are automatically updated when analyzing the leaf
partitions, and thus this command is unnecessary.

As a result, the --skip_root_stats flag has been removed since this flag
is now the default behavior.

The deleted tests in this commit were redundant, they simply added the
`skip-root-stats` flag to copies of other scenarios. This coverage was added in
existing scenarios.
Authored-by: NChris Hajas <chajas@pivotal.io>

We are in the process of renaming the secrets files in
gp-continuous-integration to use the .prod suffix consistently.
Co-authored-by: NBen Christel <bchristel@pivotal.io>
Co-authored-by: NSambitesh Dash <sdash@pivotal.io>

* Support non simple vars from ProjectionInfo
* Add code to comments and add add_projection_index_header function

A large set of tests were wrapped in an ignore block in the with
suite due to them not working properly in the past. Since most of
these have been addressed, it's time to break up the block and
ensure testing coverage.

This removes as much of the ignore as possible, and updates the
underlying returned data to match. This will create merge conflicts
with upstream but since we wont merge more code before cutting the
next release it's better to have sane tests for the lifecycle of
the next release, and we can always revert this on master as we
statrt merging again.

The trigger tests are left under ignore, even though they seem to
work quite well, since atmsort cannot handle that output yet.
Reviewed-by: NHeikki Linnakangas <hlinnakangas@pivotal.io>

An inner ModifyTable node must run to completion even if the outer
node can be satisfied with a squelched inner. Ensure to run the node
to completion when asked to Squelch to not risk losing modifications.

This adds a testcase from the upstream with test suite to the GPDB
with_clause suite. The original test is under an ignore block, but
even with lifting that the output is different due to state being
set up by prior tests which happen to fail in GPDB.
Reviewed-by: NHeikki Linnakangas <hlinnakangas@pivotal.io>

To test that fts probe can handle dns error, a fault named
get_dns_cached_address is added, however, this fault might
effect any connections that call getCdbComponentInfo() and
make the fts_errors flaky. To fix this, mark the dns fault
to fts probe only.

1.  Use fakeroot to install/uninstall deb package in gppkg
2. Throw an exception when installing an existing gppkg/deb package
3. Throw an exception when updating a non-existing gppkg/deb package
Co-authored-by: NHaozhou Wang <hawang@pivotal.io>

Existing code was globbing for libstdc++.so.* and copying it to gpdb
lib, we are now ignoring the .py since that should not be vendored.

* also cleaned up bash, no need for trailing '/.' on copy. Thanks d#!

[#164347190]
Co-authored-by: NOliver Albertini <oalbertini@pivotal.io>
Co-authored-by: NNandish Jayaram <njayaram@pivotal.io>