* Add configure_flags_with_extensions to the pipeline template file
  * regenerate the gpdb_master pipeline and use fly to set it up.
Co-authored-by: NBob Bao <bbao@pivotal.io>
Co-authored-by: NNing Fu <nfu@pivotal.io>

Writable external table has an entry in gp_distribution_policy
so it has numsegments field. Previous code skips any external
tables so that their numsegments fields are not updated. This
commit fixes this by:
  1. add a column in status_detail table to record whether the
      table is writable external and invokes correct SQL to expand
      such tables.
  2. Support `Alter external table <tab> expand table` for writable
      external tables.

Co-authored-by: Zhenghua Lyu zlv@pivotal.io

Upstream commit <568d4138> introduced a proper MVCC model for catalog
lookups. This change means that catalog lookups must be avoided if not in a
proper transaction state. In PortalSetBackoffWeight(), a check was validating if
the call was in a transaction and in case of failure no catalog look ups were
perfomed nor a backend entry was initialized.

This commit initializes a backend entry in all cases with a proper weight. No
catalogue lookups are performed outside of a transaction state.

Also it tidies up a bit the interface for the initialisation of backoff entries
as it removes the responsibility for calculating the backoff weight from the
caller.

Removes 94_MERGE_FIXME.
Co-authored-by: NJimmy Yih <jyih@pivotal.io>
Reviewed-by: NAshwin Agrawal <aagrawal@pivotal.io>
Reviewed-by: NDaniel Gustafsson <dgustafsson@pivotal.io>
Reviewed-by: NGang Xiong <gxiong@pivotal.io>

On sql "alter table parttab exchange partition for (1) with table y",
we checked if the table has same columns with the partion table. But
on sql "alter table parttab alter partition for (1) exchange partition
for (1) with table x", we forgot the check. Add the check back.

The database argument of the Gpexpand() constructor of the gpexpand
behave test was retired in 1c262c6e, but
a recently merged PR still contains use of this argument.

* Build necessary extensions for release

* Add new  '--enable-debug-extensions' configure option, if provide the option,
  the extensions include:

      gp_distribution_policy
      gp_internal_tools
      gp_sparse_vector
      gp_replica_check
      gp_inject_fault
      gp_debug_numsegments

* Rename configure_flags to configure_flags_with_extensions

  The configure_flags_with_extensions SHOULD BE set to

      configure_flags_with_extensions: "--enable-cassert --enable-debug-extensions"

    in these secrets files

       secrets/gpdb_master-ci-secrets.dev.yml
       secrets/gpdb_master-ci-secrets.prod.yml
Co-authored-by: NBob Bao <bbao@pivotal.io>
Co-authored-by: NNing Fu <nfu@pivotal.io>

If no python in $GPHOME/ext/python/bin/python, $PYTHONHOME will be set
to empty string, which make python env wrong.
Signed-off-by: NTingfang Bao <bbao@pivotal.io>

We need to do a cluster expansion which will check if there are partial
tables, we need to drop the partial tables to keep the cluster expansion
run correctly.

- Change rollback complete message for online expand needn't restart
- fsync the status file write operation to make sure that the data
  has been sync to disk

Previously, test cases `partial_table` and `subselect_gp2` are
in the same test group so that they might be running concurrently.

`partital_table` contains a statement: `update gp_distribution_policy`,
`subselect_gp2` contains a statement: `VACUUM FULL pg_authid`. These
two statements may lead to local deadlock on QD when running concurrently
if GDD is disabled.

If GDD is disabled,

  `update gp_distribution_policy`'s lock-acquire:
    1. at parsing stage, lock `gp_distribution_policy` in Exclusive Mode
      2. later when it needs to check authentication, lock `pg_authid` in
           AccessShare Mode

  `VACUUM FULL pg_authid`'s lock-acquire:
    1. lock pg_authid in Access Exclusive Mode
      2. later when rebuilding heap, it might delete some dependencies,
           this will do GpPolicyRemove, which locks `gp_distribution_policy`
	        in RowExclusive Mode

So there is a potential local deadlock.

If the target is not a table, it must error out. So it's better
to do permission check first, or the logic may access some fields
which is null-able for non-table object and cause crash.

Add tests to ensure that a table can be expanded correctly even if it
contains misplaced tuples.

Modify gpexpand phase 1, so it can continue to retry the failed
work by re-run gpexpand, if it fails after releasing the catalog
lock.

The current progress of gpexpand segment preparation is
1. create template base on master
2. lock catalog
3. build and start new segments
4. update gp_segment_configuration (then new transaction will see new nodes)
5. unlock catalog
6. create schema and table for phase2(data redistribution)

If it fails before step 5, it can roll back to original state by
running gpexpand with -r. But if it fails after step5, it can not
roll back. Because new database/table/schema may be created after
unlocking the catalog. And new data may be inserted into new
segments.

If it fails in step 6 now, DBA can do nothing. They can not roll back
and also can not conitnue to retry the failing work in step 6 without
complex manual intervention. So we change the behaviour here, if
gpexpand finds that the last expansion didn't complete successfully
and can not roll back, it will cleanup the schemas and tables built
in step 6 last time and retry the step 6.

I noticed that we haven't a sanity check for the correctness of
"numsegments" of a table, the numsegments might be larger than
the size of cluster, eg, table is expanded after a global
transaction is started and then the table is accessed in that
transaction or gp_distribution_policy is corrupt in some way,
Dispatcher or interconnect cannot handle such case, so add a
sanity check to error it out.

This sanity check is skipped in UTILITY mode.  getgpsegementcount()
returns -1 in UTILITY mode which will always make numsegments sanity
check failed, so skip it for UTILITY mode.

Spell out VACUUM ANALYZE rather than using a pointless, and silly,
shortening. No functional change.
Reviewed-by: NGeorgios Kokolatos <gkokolatos@pivotal.io>
Reviewed-by: NJimmy Yih <jyih@pivotal.io>

This reintroduces commit ba3eb5b4, which was reverted in 659f0ee5.

After the WALrep changes, the previous -F (filespace) option was
co-opted to be the new standby data directory option. This isn't a
particularly obvious association.

Change the option to -S. (-D would have been better, but that's already
in use as a short alias for --debug.) Also document this option in the
official gpinitstandby help.

This reintroduces commit c9c3c351, which was reverted in 659f0ee5.

When a standby is initialized on the same host as the original master,
remind the user that the data directory and port need to be explicitly
set.

Commit 6610b941 removed the use of createTemplate() from gpexpand.
There are no more callers, and as that commit pointed out, the
implementation is unsafe. We can also get rid of DiskUsage and
LocalDirCopy.
Co-authored-by: NJamie McAtamney <jmcatamney@pivotal.io>

The function was using an anti-pattern where an argument is a static
length char array. Array arguments in C don't really exist but compilers
accept them and without any 'const' or 'static' decorators they don't
always emit a warning.

The proposed patch only fixes the unsafeness of the function
declaration but it does not address the usefulness of it.
Reviewed-by: NDaniel Gustafsson <dgustafsson@pivotal.io>

Make sure ereport() calls start with a lower case letter, and don't
end with a period. Also remove superfluous mentions of Greenplum
Database from messages.

Reviewed-by: Heikki Linnakangas <hlinnakangas@pivotal.io>
Reviewed-by: NAdam Berlin <aberlin@pivotal.io>

Commit dfee2ff7 exposed workfile_mgr_cache_entries_get_copy()
which returns a structure containing WorkFileUsagePerQuery entries,
but the definition of WorkFileUsagePerQuery was not make public. A
forward declaration was placed in the workfile_mgr.h file but it
suffered from a redefinition warning (in clang at least):

workfile_mgr.c:93:3: warning: redefinition of typedef
					 'WorkFileUsagePerQuery' is a C11 feature
					 [-Wtypedef-redefinition]
} WorkFileUsagePerQuery;
  ^
../../../../src/include/utils/workfile_mgr.h:25:38:
					note: previous definition is here
typedef struct WorkFileUsagePerQuery WorkFileUsagePerQuery;
                                     ^
Fix by moving the WorkFileUsagePerQuery definition to the header
from which is moved in f1ef3668. If we expose an API which
returns data in the struct we need to also make the definition
available to callers.
Reviewed-by: NGeorgios Kokolatos <gkokolatos@pivotal.io>
Reviewed-by: NTeng Zhang <tezhang@pivotal.io>

If ResQueueHashNew() returns NULL it means that the processing ran
out of memory and we need to error out. This is a situation which
clearly can happen in production, so we need to upgrade the assert
to a runtime check which properly errors out instead of fail on a
subsequent NULL pointer dereference.
Reviewed-by: NGeorgios Kokolatos <gkokolatos@pivotal.io>

* docs - GDD - add information about specific type of deadlocks.

* docs - GDD - update deadlock information based on review comments.

* typo fix

* Small edit for clarity

* Removing duplicate info

This commit is part of Add Partitioned Indexes #7047.

For the partitioned index PR#7047, we add tests for the use of internal
auto dependencies in both the existing index-backed constraints and in
standalone partitioned indexes.
Co-authored-by: NMark Sliva <msliva@pivotal.io>

This commit is part of Add Partitioned Indexes #7047.

AttachPartitionEnsureIndexes iterates through all the indexes on an
incoming partition and adds INTERNAL_AUTO dependencies to the ones match
the definition of the parent partition's partitioned indexes.

These are all the tests that broke when we started exchanging/testing
for regular indexes on partitioned tables.
Co-authored-by: NKalen Krempely <kkrempely@pivotal.io>

This commit is part of Add Partitioned Indexes #7047.

Remove 'fix_ill_named_constraint' query from 'part_constraint' test.
This query expects all UNIQUE and PRIMARY KEY constraints on partitioned
tables to have identical names. This is no longer true after enforcing
UNIQUE and PRIMARY KEY constraints to have the same name as the
underlying index. These tests will need to be replaced with ones that
validate the integrity of the pg_depend entries for partitioned indexes
and constraints.
Co-authored-by: NKalen Krempely <kkrempely@pivotal.io>

This commit is part of Add Partitioned Indexes #7047.

After adding INTERNAL_AUTO, and dependencies between partitioned indexs,
many tests that assumed that we need to manually delete indexes added to
leaf partitions need updating.
Co-authored-by: NKalen Krempely <kkrempely@pivotal.io>

This commit is part of Add Partitioned Indexes #7047.

Tests were added to verify:
 - Index backed constraint names have matching index name
 - Constraints on partition tables including ADD PARTITION and EXCHANGE
   PARTITION.
 - Constraints and indexes can be upgraded. This includes testing directly in
   pg_regress, or creating tables to be used by pg_upgrade.
Co-authored-by: NTaylor Vesely <tvesely@pivotal.io>
Co-authored-by: NKalen Krempely <kkrempely@pivotal.io>
Co-authored-by: NJesse Zhang <sbjesse@gmail.com>

This commit is part of Add Partitioned Indexes #7047.

Constraint names must now match their index. Fix ICW tests where this
assumption no longer holds.
Co-authored-by: NKalen Krempely <kkrempely@pivotal.io>

This commit is part of Add Partitioned Indexes #7047.

Don't allow check constraints to be dropped on partition leaves. Only
allow them to be dropped from the root.

This restriction had been relaxed on PRIMARY KEY and UNIQUE constraints
after adding INTERNAL_AUTO dependencies to constraints. However, we need
to maintain the restiction on check constrants because they are
currently not being protected by pg_depend.
Co-authored-by: NKalen Krempely <kkrempely@pivotal.io>

This commit is part of Add Partitioned Indexes #7047.

Until indexes can be upgraded drop all indexes. Drop from the root
partition instead of on intermediate/leaf partitions, since they now
cascade due to their INTERNAL_AUTO dependencies.
Co-authored-by: NKalen Krempely <kkrempely@pivotal.io>

This commit is part of Add Partitioned Indexes #7047.

During exchange and split partition update the pg_depend relationships and
exchange constraint names.

AttachPartitionEnsureIndexes was cherry-picked from:
commit 8b08f7d4
Author: Alvaro Herrera <alvherre@alvh.no-ip.org>
Date:   Fri Jan 19 11:49:22 2018 -0300

    Local partitioned indexes

    When CREATE INDEX is run on a partitioned table, create catalog entries
    for an index on the partitioned table (which is just a placeholder since
    the table proper has no data of its own), and recurse to create actual
    indexes on the existing partitions; create them in future partitions
    also.

    As a convenience gadget, if the new index definition matches some
    existing index in partitions, these are picked up and used instead of
    creating new ones.  Whichever way these indexes come about, they become
    attached to the index on the parent table and are dropped alongside it,
    and cannot be dropped on isolation unless they are detached first.

    To support pg_dump'ing these indexes, add commands
        CREATE INDEX ON ONLY <table>
    (which creates the index on the parent partitioned table, without
    recursing) and
        ALTER INDEX ATTACH PARTITION
    (which is used after the indexes have been created individually on each
    partition, to attach them to the parent index).  These reconstruct prior
    database state exactly.

    Reviewed-by: (in alphabetical order) Peter Eisentraut, Robert Haas, Amit
            Langote, Jesper Pedersen, Simon Riggs, David Rowley
    Discussion: https://postgr.es/m/20171113170646.gzweigyrgg6pwsg4@alvherre.pgsqlCo-authored-by: NKalen Krempely <kkrempely@pivotal.io>

This commit adds partitioned indexes from upstream Postgres. This commit is
mostly cherry-picked from Postgres 11 and bug fixes from Postgres 12.

Differences from upstream:
 - Postgres has two additional relkind's - RELKIND_PARTITIONED_TABLE and
   RELKIND_PARTITIONED_INDEX, which have no on disk storage. Greenplum does not
   have these additional relkinds. Thus, partitioned indexes have physical
   storage.
 - CREATE INDEX ON ONLY <table> DDL has not yet been implemented
 - ALTER INDEX ATTACH PARTITION DDL has not yet been implemented

Constraint changes:
 - Constraints and their backing index have the same names. Thus, partitions of
   a table no longer share the same constraint name, and are instead related to
   their parent via INTERNAL_AUTO dependencies.

Index changes:
 - Child partition indexes can no longer be directly dropped, and must be
   dropped from their root. This includes mid-level and leaf indexes.
 - Adding indexes to mid-level partitions cascade to their children.

These changes are mostly cherry-picked from:
commit 8b08f7d4
Author: Alvaro Herrera <alvherre@alvh.no-ip.org>
Date:   Fri Jan 19 11:49:22 2018 -0300

    Local partitioned indexes

    When CREATE INDEX is run on a partitioned table, create catalog entries
    for an index on the partitioned table (which is just a placeholder since
    the table proper has no data of its own), and recurse to create actual
    indexes on the existing partitions; create them in future partitions
    also.

    As a convenience gadget, if the new index definition matches some
    existing index in partitions, these are picked up and used instead of
    creating new ones.  Whichever way these indexes come about, they become
    attached to the index on the parent table and are dropped alongside it,
    and cannot be dropped on isolation unless they are detached first.

    To support pg_dump'ing these indexes, add commands
        CREATE INDEX ON ONLY <table>
    (which creates the index on the parent partitioned table, without
    recursing) and
        ALTER INDEX ATTACH PARTITION
    (which is used after the indexes have been created individually on each
    partition, to attach them to the parent index).  These reconstruct prior
    database state exactly.

    Reviewed-by: (in alphabetical order) Peter Eisentraut, Robert Haas, Amit
            Langote, Jesper Pedersen, Simon Riggs, David Rowley
    Discussion: https://postgr.es/m/20171113170646.gzweigyrgg6pwsg4@alvherre.pgsql

Changes were also cherry-picked from the following Postgres commits:
  eb7ed3f3 - Allow UNIQUE indexes on partitioned tables
  ae366aa5 - Detach constraints when partitions are detached
  19184fcc - Simplify coding to detach constraints when detaching partition
  c7d43c4d - Correct attach/detach logic for FKs in partitions
  17f206fb - Set pg_class.relhassubclass for partitioned indexes
Co-authored-by: NKalen Krempely <kkrempely@pivotal.io>

Changing version in path from /600 to /6-0 for consistency with other docs and to recognize that only the minor version digit is significant for the doc build (#7141)

* Bump ORCA version to 3.29.0

* ORCA: Updating subquery plans in ICG expected files

This change is needed for ORCA PR https://github.com/greenplum-db/gporca/pull/449.
Some subquery plans changed in minor ways in the ICG test.
Co-authored-by: NChris Hajas <chajas@pivotal.io>

Currently, a randomly distributed table cannot be created with a
primary key or unique index. We should put this restriction for ALTER
TABLE SET DISTRIBUTED RANDOMLY as well. This was caught by gpcheckcat
distribution_policy check.
Co-authored-by: NAlexandra Wang <lewang@pivotal.io>