* docs - CTE available with INSERT, UPDATE, DELETE

-updated GUC
-updated Admin Guide topic WITH Queries (Common Table Expressions)

updates to SELECT, INSERT, UPDATE DELETE will be part of postgres 9.2 merge.

* docs - CTE updates from review comments

* docs - CTE more updates from review comments

* docs - CTE - updates from review comments

* Experimental -> Beta wording

* vacuumdb - add -t option support for multiple tables

* reindexdb - add -t option support for multiple tables; misc edits

* psql - misc additions, edits, and reformats

* pg_dump - misc edits, additions, formatting

* add * syntax for descendant tables

* pg_dumpall - add -d, --dbname option

* PREPARE - add note about search_path

* TRUNCATE - add * option

* SELECT - add LATERAL, NO KEY UPDATE, KEY SHARE, and related edits to locking clause

* ALTER ROLE. Re-order syntax and descriptions.

* ALTER RULE. Add new command and fix some conref issues

* COPY. Add FREEZE option and edits.

* CREATE INDEX. Minor synopsis update.

* CREATE SCHEMA. Add IF NOT EXISTS forms of command

* CREATE TABLE. Syntax changes. Add SET CONSTRAINTS command.

* bgworker. Add bgworker topic to reference guide

* CREATE TABLE AS. Add WITH/WITHOUT OIDS

* Changes from review

* remove some parens, add trigger note

* remove extra whitespace in CREATE TABLE

* CREATE TABLE AS - correct DISTRIBUTED BY syntax

* Review comments on CREATE TABLE.

* pg_dump - add parallelization note

* CREATE TABLE - fix whitespace again?

* Remove docs for LATERAL

* Add LATERAL to unsupported feature list

After the WALrep changes, the previous -F (filespace) option was
co-opted to be the new standby data directory option. This isn't a
particularly obvious association.

Change the option to -S. (-D would have been better, but that's already
in use as a short alias for --debug.) Also document this option in the
official gpinitstandby help.

When a standby is initialized on the same host as the original master,
remind the user that the data directory and port need to be explicitly
set.

When -s is present, show pg_basebackup progress sequentially instead
of inplace. Useful when writing to a file, or if a tty does not support
escape sequences. Defaults to showing the progress inplace.

The gprecoverseg utility runs pg_basebackup in parallel on all segments that are
being recovered.  In this commit, we are logging the progress of each
pg_basebackup on its host and displaying them to the user of gprecoverseg.  The
progress files are deleted upons successful completion of gprecoverseg.

Unit tests have also been added.
Authored-by: NShoaib Lari <slari@pivotal.io>
Co-authored-by: NMark Sliva <msliva@pivotal.io>
Co-authored-by: NJacob Champion <pchampion@pivotal.io>
Co-authored-by: NEd Espino <edespino@pivotal.io>
Co-authored-by: NKalen Krempely <kkrempely@pivotal.io>

* reword from 'experimental' to 'beta'

* Experimental -> Beta in markdown source

* typo fix

* Removing SuSE 11 details in Beta notes

Removes a merge fixme
Reviewed-by: NHeikki Linnakangas <hlinnakangas@pivotal.io>

We had three test modules to test the gppc partner connector library.
All three were the same in the way the test module was built and called, so
we might as well merge them into a single module to simplify things. This
also makes the tests run faster, as we no longer create a separate database
to test each module, but just one. Leaving behind fewer databases should
also speed up the later stages in installcheck-world, like gpcheckcat,
gp_replica_check, and pg_upgrade, run faster, because we leave behind fewer
databases for them to process.

There was a lot of identical test code and tests among the three modules.
Remove the duplication while we're at it.

The byteafunc2 test function was intentionally changing the pass-by-ref
argument, which is a big no-no. It doesn't make much sense to test that,
because that's a programmer error, and all bets are off. Change the test to
do something more interesting.

Normal client connection to GPDB server will be represented as
MyProcPort struct in QD. This is also true for connection from QD to QE.
But for background worker and auxiliary process such as global deadlock
detector, there is no client concept, and thus MyProcPort is set to NULL.

For example, dispatcherAYT() will be called to check QD-Client connection.
But there is no client in bgworker case. adjustMasterRouting() should be called 
on QE, so assert check MyProcPort is not NULL.

Build mode does not use any information from the cluster and does not
affect its running status, in fact it does not require a cluster exists
at all.
Co-authored-by: NHubert Zhang <hzhang@pivotal.io>
Co-authored-by: NNing Yu <nyu@pivotal.io>

After the changes to gp_distribution_policy to add distribution opclass,
the distribution policy checks in gpcheckcat need to be updated to
consider the distclass column as well.

Additionally, the old repair action that was taken as part of this check
is no longer valid, since we can't redistribute a child partition
directly. Instead, leave the correct action up to the DB admin.
Co-authored-by: NJimmy Yih <jyih@pivotal.io>
Co-authored-by: NTaylor Vesely <tvesely@pivotal.io>

The citext test was altered in the past to add a distribution column
other than citext. Since we now allow for extension datatypes to be
distribution columns with the new hashing work, remove the workaround
and re-align the test with upstream.

This also updates the testdata from upstream to work in Greenplum.
Reviewed-by: NHeikki Linnakangas <hlinnakangas@pivotal.io>

Previously, analyzedb would run `analyze rootpartition <tablename>` on
root partitions. However, with the introduction of merging leaf
statistics into the root in 9c1b1ae3, this command is no longer
necessary. The root stats are automatically updated when analyzing the leaf
partitions, and thus this command is unnecessary.

As a result, the --skip_root_stats flag has been removed since this flag
is now the default behavior.

The deleted tests in this commit were redundant, they simply added the
`skip-root-stats` flag to copies of other scenarios. This coverage was added in
existing scenarios.
Authored-by: NChris Hajas <chajas@pivotal.io>

We are in the process of renaming the secrets files in
gp-continuous-integration to use the .prod suffix consistently.
Co-authored-by: NBen Christel <bchristel@pivotal.io>
Co-authored-by: NSambitesh Dash <sdash@pivotal.io>

* Support non simple vars from ProjectionInfo
* Add code to comments and add add_projection_index_header function

A large set of tests were wrapped in an ignore block in the with
suite due to them not working properly in the past. Since most of
these have been addressed, it's time to break up the block and
ensure testing coverage.

This removes as much of the ignore as possible, and updates the
underlying returned data to match. This will create merge conflicts
with upstream but since we wont merge more code before cutting the
next release it's better to have sane tests for the lifecycle of
the next release, and we can always revert this on master as we
statrt merging again.

The trigger tests are left under ignore, even though they seem to
work quite well, since atmsort cannot handle that output yet.
Reviewed-by: NHeikki Linnakangas <hlinnakangas@pivotal.io>

An inner ModifyTable node must run to completion even if the outer
node can be satisfied with a squelched inner. Ensure to run the node
to completion when asked to Squelch to not risk losing modifications.

This adds a testcase from the upstream with test suite to the GPDB
with_clause suite. The original test is under an ignore block, but
even with lifting that the output is different due to state being
set up by prior tests which happen to fail in GPDB.
Reviewed-by: NHeikki Linnakangas <hlinnakangas@pivotal.io>

To test that fts probe can handle dns error, a fault named
get_dns_cached_address is added, however, this fault might
effect any connections that call getCdbComponentInfo() and
make the fts_errors flaky. To fix this, mark the dns fault
to fts probe only.

1.  Use fakeroot to install/uninstall deb package in gppkg
2. Throw an exception when installing an existing gppkg/deb package
3. Throw an exception when updating a non-existing gppkg/deb package
Co-authored-by: NHaozhou Wang <hawang@pivotal.io>

Existing code was globbing for libstdc++.so.* and copying it to gpdb
lib, we are now ignoring the .py since that should not be vendored.

* also cleaned up bash, no need for trailing '/.' on copy. Thanks d#!

[#164347190]
Co-authored-by: NOliver Albertini <oalbertini@pivotal.io>
Co-authored-by: NNandish Jayaram <njayaram@pivotal.io>

These hooks are used by diskquota extension to detect Heap/AO/CO table change.
For Heap table, smgr related hooks will be utilized the same way as Postgresql.
For AO/CO table, we add some new hook position, TruncateAOSegmentFile(),
BufferedAppendWrite() and copy_append_only_data() due to the lack of
abstract storage layer for AO/CO tables.
Co-authored-by: NHaozhou Wang <hawang@pivotal.io>
Co-authored-by: NHao Wu <gfphoenix78@gmail.com>

Commit 4eb65a53 bring GPDB the ability to allow tables to be distributed on
a subset of segments, that commit taken care of the replicated table well for
SELECT/UPDATE/INSERT on a subset, however, COPY FROM was not.

For COPY FROM replicated table, only one replica should be picked to provide
data and the QE whose segid match gp_session_id % segment_size will be chosen,
obviously, for table on a subset of segments, a valid QE might be chosen.

To fix it, the real numsegments of replicated table should be used instead of
current segment size, what's more, dispatcher can allocate gangs on a subset
of segments now, QD can directly allocate picked gang directly, QE doesn't need
to care about whether it should provide data anymore.

For COPY TO replicated table, we also should allocated correct QEs matching the
numsegments of replicated table.
Co-authored-by: NGang Xiong <gxiong@pivotal.io>

checkPolicyForUniqueIndex() checks if distribution key conflict with unique/primary
key, for example, unique index is not allowed for random-distributed table and
allowed for replicated-distributed table, for normal distributed, the set of
columns being indexed should be a superset of the table.

What about entry-distributed table? (eg, table created in utility mode, it has
no records in gp_distribution_policy and GpPolicyFetch translated it to
entry-distributed)? Such tables are localized in a single db, so adding a unique
index should also be allowed.

This is spotted by the assertion in checkPolicyForUniqueIndex() when checking
the conflict for normal distributed tables.

This fixes #5880

Now each QE in a session is assigned with a unique identifier, meanwhile, QD
dispatches a slice table to all QEs and each slice in the slice table has a
bitmapset of QE identifiers. QE go through all slices and decide the slice it
belongs to by checking its identifier in the bitmapset.

A problem is, the QE identifier was never resued when a QE was destroyed, so the
identifier was incrementally increased until the bitmapset is inefficient and
insufficient to hold it.

This commit fixes it by reusing the QE identifiers to limit it in a reasonable
range.
Co-authored-by: NAshwin Agrawal <aagrawal@pivotal.io>

- add a rename task to make the artifacts use the new naming convention
- keep pushing to the s3 resources so that other pipelines using those
  published artifacts don't break (although those s3 resources are now
  deprecated).
Co-authored-by: NSambitesh Dash <sdash@pivotal.io>
Co-authored-by: NAmil Khanzada <akhanzada@pivotal.io>
Co-authored-by: NBen Christel <bchristel@pivotal.io>

Issue is that we point the first snapshot's in-progress transaction
array at the second block that we've allocated for xips. This results in
every snapshot N pointing to snapshot N+1's block of allocated xips.
This meant that the memory allocated which was intended for snapshot 0
was never used. Worse though is that the last snapshot's xip points to
an address that was not memory allocated for xips.

This bug manifests itself in interesting ways because it essentially
corrupts a local snapshot passed from a writer to a reader if we are
unfortunate enough to use the last indexed shared local snapshot
in-progress transaction array. When the snapshot is corrupt then we
cannot guarantee the correct visibility of tuples in a table.
Co-authored-by: NAshwin Agrawal <aagrawal@pivotal.io>

* Warn not to run gpbackup/gprestore during expand.

* Updates from review

Co-authored-by: NDavid Sharp <dsharp@pivotal.io>
Co-authored-by: NBradford D. Boyle <bboyle@pivotal.io>
Co-authored-by: NSoumyadeep Chakraborty <sochakraborty@pivotal.io>
Co-authored-by: NSambitesh Dash <sdash@pivotal.io>

We don't have Subject Alternative Name support, and won't until 9.5.
Make sure we uncomment these tests when we get there.
Co-authored-by: NDavid Krieger <dkrieger@pivotal.io>

This has been broken since 94fcdf3f6, which disallowed setting a
distribution policy for child partitions. Now that the behave tests
correctly fail this test, we're disabling it for now while we decide
what the correct repair action should be.
Co-authored-by: NJimmy Yih <jyih@pivotal.io>

The two separate queries made as part of the distribution_policy checks
overlapped when the table in question was randomly distributed. This led
to two copies of the same DROP CONSTRAINT statement in the repair
scripts, which failed.

To fix, we can remove the overlapping WHERE clause from the second
query, and only focus on column-distributed tables there.
Co-authored-by: NJimmy Yih <jyih@pivotal.io>

Several layers were discarding failures. The bash script and the psql
call itself need to stop on the first failure.
Co-authored-by: NJimmy Yih <jyih@pivotal.io>

Commit 352eb65c had already fixed this bug, in a way that did not
ignore failed return codes from the repair scripts that are running.

This reverts commit 061da8fc.
Co-authored-by: NJimmy Yih <jyih@pivotal.io>

We generate the fake ItemPointer for AO table's tuple, but it is
a little different from the HEAP table's ItemPointer, the offset
of the pointer can be 0 when it is a fake ItemPointer, but we
treat 0 to be invalid in the previous codes, In order to run
correctly, we set the 16th bit of offset to be 1 as a flag, then
it can pass some check, but the value is not the real value, so when
using the fake ItemPointer, we have to note to process the flag.

The bitmap index build code expects to see the tuple IDs in order. 
It gets confused when it sees offset number 32768(0x8000) 
before offset number 1, this commit convert 32768(0x8000) to
be 0.

* this way when you ssh onto the VM, you have e.g. psql available to you
Authored-by: NOliver Albertini <oalbertini@pivotal.io>

- This will allow existing PXF Release (java) continue working with the
  gpdb 6

To query the gpexpand schema we must connect to "postgres" database now,
update a forgot-to-changed dbname in gpexpand behave tests

The -D option was retired in commit
ec708f34, but we did not update the
tests accordingly.

Now we retire the -D option from gpexpand tests, too.