* Squashed commit of the following:

commit c5f4e38fa669b3daf4c5812b924ad8704e0bdcdc
Author: dyozie <dyozie@pivotal.io>
Date:   Fri Mar 1 14:40:19 2019 -0800

    SELECT - add LATERAL, NO KEY UPDATE, KEY SHARE, and related edits to locking clause

commit b6bb9e9002491bff4ced8c619ee3f4f9fb8699f3
Author: dyozie <dyozie@pivotal.io>
Date:   Fri Mar 1 09:10:50 2019 -0800

    TRUNCATE - add * option

commit 363921783d794bb18b188e585163cd998b7a2f97
Author: David Yozie <dyozie@pivotal.io>
Date:   Thu Feb 28 16:56:50 2019 -0800

    PREPARE - add note about search_path

commit 4ec9f8879f90212c34602b7ccaf9ba93c0cfbe8f
Author: David Yozie <dyozie@pivotal.io>
Date:   Thu Feb 28 16:50:32 2019 -0800

    pg_dumpall - add -d, --dbname option

commit 00c97fca7a4a8420e4a148249af2cafa937c21f8
Author: David Yozie <dyozie@pivotal.io>
Date:   Thu Feb 28 16:41:56 2019 -0800

    add * syntax for descendant tables

commit ab9bd0f96ca38639b4b1ebc97cc8f4009dc7cb91
Author: Lisa Owen <lowen@pivotal.io>
Date:   Thu Feb 28 09:45:22 2019 -0800

    pg_dump - misc edits, additions, formatting

commit 2de77b61903dbbd56f5011545bf23fa794e0a5b5
Author: Lisa Owen <lowen@pivotal.io>
Date:   Wed Feb 27 15:04:03 2019 -0800

    psql - misc additions, edits, and reformats

commit 43bbeed814c56037bbe9bed2bd0b778475264e1c
Author: Lisa Owen <lowen@pivotal.io>
Date:   Wed Feb 27 10:49:00 2019 -0800

    reindexdb - add -t option support for multiple tables; misc edits

commit b9128c528c258d5c84457db12a2a43439a9bacef
Author: Lisa Owen <lowen@pivotal.io>
Date:   Wed Feb 27 10:47:24 2019 -0800

    vacuumdb - add -t option support for multiple tables

* Squashed commit of the following:

commit c8d0bd57acc1a038b2bbd0d9a6900762bf19498e
Author: Chuck Litzell <litzellc@pivotal.io>
Date:   Fri Mar 1 15:33:35 2019 -0600

    CREATE TABLE. Syntax changes. Add SET CONSTRAINTS command.

commit a34ddfac6a5c6aea43b71f7445d6e8d82ab43592
Author: Chuck Litzell <litzellc@pivotal.io>
Date:   Fri Mar 1 16:00:51 2019 -0600

    CREATE TABLE AS. Add WITH/WITHOUT OIDS

commit dde4d614cf10a178b6e1b1dbc8d64608b171924f
Author: Chuck Litzell <litzellc@pivotal.io>
Date:   Fri Mar 1 15:58:54 2019 -0600

    bgworker. Add bgworker topic to reference guide

commit 2749f2e735cc0b59e1153f04fe693e83a812bfc5
Author: Chuck Litzell <litzellc@pivotal.io>
Date:   Wed Feb 27 17:27:57 2019 -0600

    CREATE SCHEMA. Add IF NOT EXISTS forms of command

commit 581274c1d5b218443a9516c160c290aeb73e0999
Author: Chuck Litzell <litzellc@pivotal.io>
Date:   Wed Feb 27 16:43:40 2019 -0600

    CREATE INDEX. Minor synopsis update.

commit b83065abce53d0021eee01c1e8cdf0c7a53d60d4
Author: Chuck Litzell <litzellc@pivotal.io>
Date:   Wed Feb 27 16:13:19 2019 -0600

    COPY. Add FREEZE option and edits.

commit a0800bb10b581cccff97ef5193ef430b53061327
Author: Chuck Litzell <litzellc@pivotal.io>
Date:   Wed Feb 27 15:37:40 2019 -0600

    ALTER RULE. Add new command and fix some conref issues

commit dcf917189d601c1ef9d7c894ea016bd6139e29e8
Author: Chuck Litzell <litzellc@pivotal.io>
Date:   Wed Feb 27 14:58:58 2019 -0600

    ALTER ROLE. Re-order syntax and descriptions.

commit c5f4e38fa669b3daf4c5812b924ad8704e0bdcdc
Author: dyozie <dyozie@pivotal.io>
Date:   Fri Mar 1 14:40:19 2019 -0800

    SELECT - add LATERAL, NO KEY UPDATE, KEY SHARE, and related edits to locking clause

commit b6bb9e9002491bff4ced8c619ee3f4f9fb8699f3
Author: dyozie <dyozie@pivotal.io>
Date:   Fri Mar 1 09:10:50 2019 -0800

    TRUNCATE - add * option

commit 363921783d794bb18b188e585163cd998b7a2f97
Author: David Yozie <dyozie@pivotal.io>
Date:   Thu Feb 28 16:56:50 2019 -0800

    PREPARE - add note about search_path

commit 4ec9f8879f90212c34602b7ccaf9ba93c0cfbe8f
Author: David Yozie <dyozie@pivotal.io>
Date:   Thu Feb 28 16:50:32 2019 -0800

    pg_dumpall - add -d, --dbname option

commit 00c97fca7a4a8420e4a148249af2cafa937c21f8
Author: David Yozie <dyozie@pivotal.io>
Date:   Thu Feb 28 16:41:56 2019 -0800

    add * syntax for descendant tables

commit ab9bd0f96ca38639b4b1ebc97cc8f4009dc7cb91
Author: Lisa Owen <lowen@pivotal.io>
Date:   Thu Feb 28 09:45:22 2019 -0800

    pg_dump - misc edits, additions, formatting

commit 2de77b61903dbbd56f5011545bf23fa794e0a5b5
Author: Lisa Owen <lowen@pivotal.io>
Date:   Wed Feb 27 15:04:03 2019 -0800

    psql - misc additions, edits, and reformats

commit 43bbeed814c56037bbe9bed2bd0b778475264e1c
Author: Lisa Owen <lowen@pivotal.io>
Date:   Wed Feb 27 10:49:00 2019 -0800

    reindexdb - add -t option support for multiple tables; misc edits

commit b9128c528c258d5c84457db12a2a43439a9bacef
Author: Lisa Owen <lowen@pivotal.io>
Date:   Wed Feb 27 10:47:24 2019 -0800

    vacuumdb - add -t option support for multiple tables

* Changes from review

* json-data. Edits, mostly already present in GPDB doc.

* ALTER FOREIGN TABLE. ENABLE/DISABLE trigger forms.

* ALTER INDEX. Add ALL IN TABLESPACE.

* ALTER SYSTEM. new command

* ALTER TABLE. ALL IN TABLESPACE, edits, MVCC warning.

* ALTER TABLESPACE. minor changes only.

* CREATE ROLE. Take out warning about older clients/MD5

* CREATE TABLE. multiple edits.

* CREATE VIEW. Edits. Added RECURSIVE, but also added a Beta warning.

* DISCARD. Add SEQUENCES; edits.

* DROP OWNED. Minor edits.

* GRANT. Add default privileges for object types.

* psql. Many edits.

* SELECT. Many edits.

* gpbackup. typo fix.

* TRUNCATE. Add note truncate is not mvcc-safe.

* VACUUM. Restore description of example

* Corrections from reviews

* Add missing period.

It's kilobytes, not kilo bytes.

This removes a whole set of GUCs which are no longer used in the
codebase. Also remove relevant documentation.
Reviewed-by: NAshwin Agrawal <aagrawal@pivotal.io>
Reviewed-by: NHeikki Linnakangas <hlinnakangas@pivotal.io>
Reviewed-by: NGeorgios Kokolatos <gkokolatos@pivotal.io>

As we merge with upstream and by that keep refining the Postgres
planner, legacy planner is no longer a suitable name. This changes
all variations of the spelling (legacy planner, legacy optimizer,
legacy query optimizer etc) to say "Postgres" rather than "legacy".
Reviewed-by: NVenkatesh Raghavan <vraghavan@pivotal.io>
Reviewed-by: NDavid Yozie <dyozie@pivotal.io>
Reviewed-by: NGeorgios Kokolatos <gkokolatos@pivotal.io>

This just removes unused imports, variables and functions as well
as cleans up a little bit of whitespace. No functional changes.
Reviewed-by: NAdam Berlin <aberlin@pivotal.io>
Reviewed-by: NShoaib Lari <slari@pivotal.io>
Reviewed-by: NKalen Krempely <kkrempely@pivotal.io>

* Avoid unnecessary start/end_ignore blocks

* The test tables were supposed to be created in a dedicated schema, but
  because of the RESET ALL commands, they were created in 'public' schema
  instead. Fix by replacing "RESET ALL" with "RESET
  test_print_direct_dispatch_info"

* Don't try to DROP tables that should not exist yet.

* Don't bother DROPping test tables as we go. They will be dropped at the
  end, when we drop the whole schema.

* Use fewer partitions in a test on partitioned table.

* Use fewer rows in 'tblexecutions' test table. This reduces the execution
  time by about 20 s.

We had essentially the same test in 'partition' already.

* Avoid using start/end_ignore blocks where not necessary.
* Remove unnecessary DROP commands.
* Use begin/commit when building test tables, to reduce 2PC overhead
* Reuse test tables, rather than drop and recreate them

Shaves a few seconds from the total execution time.

Commit 90a957eb changed the test to create fewer partitions, but didn't
update the NOTICEs in expected output accordingly. NOTICEs are ignored
when comparing the expected output with actual output, but it's still nice
to keep the expected output in sync with reality.

* Update relation's stats in pg_class during vacuum full.

Hash index depends on estimation of numbers of tuples and pages of relations,
incorrect value could be a reason of significantly growing of index. Vacuum
full recreates heap and reindex all indexes before renewal stats. The patch
fixes that, so indexes will see correct values.

Backpatch to v10 only because earlier versions haven't usable hash index and
growing of hash index is a single user-visible symptom.

Author: Amit Kapila
Reviewed-by: Ashutosh Sharma, me
Discussion: https://www.postgresql.org/message-id/flat/20171115232922.5tomkxnw3iq6jsg7@inml.weebeastie.net

* Collect QE's relpages and reltuples to QD

And logic in swap_relation_files() to collect QE's relpages and reltuples to QD when doing vacuum full
Co-authored-by: NJimmy Yih <jyih@pivotal.io>
Co-authored-by: NAshwin Agrawal <aagrawal@pivotal.io>

* Add test

Add test to verify that relpages and reltuples has become proper numbers when vacuum full
Co-authored-by: NTaylor Vesely <tvesely@pivotal.io>

* Update PR pipeline failures
Reviewed-by: NAdam Berlin <aberlin@pivotal.io>
Reviewed-by: NAlexandra Wang <lewang@pivotal.io>
Reviewed-by: NJimmy Yih <jyih@pivotal.io>
Reviewed-by: NAshwin Agrawal <aagrawal@pivotal.io>
Reviewed-by: NTaylor Vesely <tvesely@pivotal.io>

this commit re-enable commit 1ec65820

Reviewed-by Adam Berlin

Now gpexpand uses database `postgres` to store
expand information instead of a user-specific
database.
Co-authored-by: NJialun Du <jdu@pivotal.io>

Make sure array type is not missed after partition exchange.

This commit is to ensure that we have basic coverage to create and
insert data into an appendonly row or column oriented tables with
quicklz, zstd, zlib, or rle. It uses get_ao_compression_ratio() function
to check insert success by comparing the uncompressed/compressed sizes.

It includes API testing of invalid scenarios, such as invalid compress
level or unsupported compresstype in a table format (e.g. rle is not
supported with appendonly row-oriented tables).
Co-authored-by: NMelanie Plageman <mplageman@pivotal.io>
Co-authored-by: NDavid Kimura <dkimura@pivotal.io>
Co-authored-by: NJesse Zhang <sbjesse@gmail.com>

Reword exception message to be more clear and concise.
Authored-by: NKalen Krempely <kkrempely@pivotal.io>

Authored-by: NKalen Krempely <kkrempely@pivotal.io>

We have an assert failure for copy to partition table if root and child relation
have different attno. The root cause is that 'reconstructTupleValues'
has a wrong value of newNumAttrs lenth.

  * Add configure_flags_with_extensions to the pipeline template file
  * regenerate the gpdb_master pipeline and use fly to set it up.
Co-authored-by: NBob Bao <bbao@pivotal.io>
Co-authored-by: NNing Fu <nfu@pivotal.io>

Writable external table has an entry in gp_distribution_policy
so it has numsegments field. Previous code skips any external
tables so that their numsegments fields are not updated. This
commit fixes this by:
  1. add a column in status_detail table to record whether the
      table is writable external and invokes correct SQL to expand
      such tables.
  2. Support `Alter external table <tab> expand table` for writable
      external tables.

Co-authored-by: Zhenghua Lyu zlv@pivotal.io

Upstream commit <568d4138> introduced a proper MVCC model for catalog
lookups. This change means that catalog lookups must be avoided if not in a
proper transaction state. In PortalSetBackoffWeight(), a check was validating if
the call was in a transaction and in case of failure no catalog look ups were
perfomed nor a backend entry was initialized.

This commit initializes a backend entry in all cases with a proper weight. No
catalogue lookups are performed outside of a transaction state.

Also it tidies up a bit the interface for the initialisation of backoff entries
as it removes the responsibility for calculating the backoff weight from the
caller.

Removes 94_MERGE_FIXME.
Co-authored-by: NJimmy Yih <jyih@pivotal.io>
Reviewed-by: NAshwin Agrawal <aagrawal@pivotal.io>
Reviewed-by: NDaniel Gustafsson <dgustafsson@pivotal.io>
Reviewed-by: NGang Xiong <gxiong@pivotal.io>

On sql "alter table parttab exchange partition for (1) with table y",
we checked if the table has same columns with the partion table. But
on sql "alter table parttab alter partition for (1) exchange partition
for (1) with table x", we forgot the check. Add the check back.

The database argument of the Gpexpand() constructor of the gpexpand
behave test was retired in 1c262c6e, but
a recently merged PR still contains use of this argument.

* Build necessary extensions for release

* Add new  '--enable-debug-extensions' configure option, if provide the option,
  the extensions include:

      gp_distribution_policy
      gp_internal_tools
      gp_sparse_vector
      gp_replica_check
      gp_inject_fault
      gp_debug_numsegments

* Rename configure_flags to configure_flags_with_extensions

  The configure_flags_with_extensions SHOULD BE set to

      configure_flags_with_extensions: "--enable-cassert --enable-debug-extensions"

    in these secrets files

       secrets/gpdb_master-ci-secrets.dev.yml
       secrets/gpdb_master-ci-secrets.prod.yml
Co-authored-by: NBob Bao <bbao@pivotal.io>
Co-authored-by: NNing Fu <nfu@pivotal.io>

If no python in $GPHOME/ext/python/bin/python, $PYTHONHOME will be set
to empty string, which make python env wrong.
Signed-off-by: NTingfang Bao <bbao@pivotal.io>

We need to do a cluster expansion which will check if there are partial
tables, we need to drop the partial tables to keep the cluster expansion
run correctly.

- Change rollback complete message for online expand needn't restart
- fsync the status file write operation to make sure that the data
  has been sync to disk

Previously, test cases `partial_table` and `subselect_gp2` are
in the same test group so that they might be running concurrently.

`partital_table` contains a statement: `update gp_distribution_policy`,
`subselect_gp2` contains a statement: `VACUUM FULL pg_authid`. These
two statements may lead to local deadlock on QD when running concurrently
if GDD is disabled.

If GDD is disabled,

  `update gp_distribution_policy`'s lock-acquire:
    1. at parsing stage, lock `gp_distribution_policy` in Exclusive Mode
      2. later when it needs to check authentication, lock `pg_authid` in
           AccessShare Mode

  `VACUUM FULL pg_authid`'s lock-acquire:
    1. lock pg_authid in Access Exclusive Mode
      2. later when rebuilding heap, it might delete some dependencies,
           this will do GpPolicyRemove, which locks `gp_distribution_policy`
	        in RowExclusive Mode

So there is a potential local deadlock.

If the target is not a table, it must error out. So it's better
to do permission check first, or the logic may access some fields
which is null-able for non-table object and cause crash.

Add tests to ensure that a table can be expanded correctly even if it
contains misplaced tuples.

Modify gpexpand phase 1, so it can continue to retry the failed
work by re-run gpexpand, if it fails after releasing the catalog
lock.

The current progress of gpexpand segment preparation is
1. create template base on master
2. lock catalog
3. build and start new segments
4. update gp_segment_configuration (then new transaction will see new nodes)
5. unlock catalog
6. create schema and table for phase2(data redistribution)

If it fails before step 5, it can roll back to original state by
running gpexpand with -r. But if it fails after step5, it can not
roll back. Because new database/table/schema may be created after
unlocking the catalog. And new data may be inserted into new
segments.

If it fails in step 6 now, DBA can do nothing. They can not roll back
and also can not conitnue to retry the failing work in step 6 without
complex manual intervention. So we change the behaviour here, if
gpexpand finds that the last expansion didn't complete successfully
and can not roll back, it will cleanup the schemas and tables built
in step 6 last time and retry the step 6.

I noticed that we haven't a sanity check for the correctness of
"numsegments" of a table, the numsegments might be larger than
the size of cluster, eg, table is expanded after a global
transaction is started and then the table is accessed in that
transaction or gp_distribution_policy is corrupt in some way,
Dispatcher or interconnect cannot handle such case, so add a
sanity check to error it out.

This sanity check is skipped in UTILITY mode.  getgpsegementcount()
returns -1 in UTILITY mode which will always make numsegments sanity
check failed, so skip it for UTILITY mode.

Spell out VACUUM ANALYZE rather than using a pointless, and silly,
shortening. No functional change.
Reviewed-by: NGeorgios Kokolatos <gkokolatos@pivotal.io>
Reviewed-by: NJimmy Yih <jyih@pivotal.io>

This reintroduces commit ba3eb5b4, which was reverted in 659f0ee5.

After the WALrep changes, the previous -F (filespace) option was
co-opted to be the new standby data directory option. This isn't a
particularly obvious association.

Change the option to -S. (-D would have been better, but that's already
in use as a short alias for --debug.) Also document this option in the
official gpinitstandby help.

This reintroduces commit c9c3c351, which was reverted in 659f0ee5.

When a standby is initialized on the same host as the original master,
remind the user that the data directory and port need to be explicitly
set.