This reverts commit c3726b09.

The original commit, which we're re-introducing with this Revert, was failing
because some isolation2 tests had this situation:

Session one successfully reindexes with a blocking call without committing,
continuing to hold the lock. Session two attempts to create an index on the
same table, which is locked by session one.

We introduced backgrounding of session two, which allows session one to return
and commit, releasing the lock. Our change enables the test runner to behave in
a realistic concurrent way, after the introduction of the
ShareRowExclusiveLock.
Co-authored-by: NJesse Zhang <sbjesse@gmail.com>

After 8.4 merge, we have two restrictlist 'mergeclause_list'
and 'hashclause_list' in function 'add_paths_to_joinrel'. We
use mergeclause_list in cdb motion in hashjoin. But some of
keys should not been used as distribution keys.

Add a whitelist that which operator is  distribution-compatible.

In b94afa13 we added a test
terminate_in_gang_creation, it creates a table `foo`; later in test
pg_terminate_backend it will fail due to it can not create a table with
the same name.

Fixed in terminate_in_gang_creation by dropping the table at the end.

(cherry picked from commit 53a0bc6e)

In 00da6bbf we added a test case in pg_terminate_backend, however as
it uses fault injection it will not work stably in a parallel test
greoup.

Moved this test case to a separate test group.

In upstream Postgres, pg_locks exposes part of the lock manager so that
DBA's can inspect locks taken by various backends. In Greenplum, we
modified pg_lock_status() -- the function that underlies pg_locks -- to

  a) provide additional Greenplum-specific information (e.g.
     mppsessionid); and

  b) aggregate the locks from master and all primary segments.

One consequence of the implementation we chose to achieve point b above,
is that queries that involve both pg_locks and a distributed table won't
work. If you're lucky (planner or ORCA places the function call in the
top slice), it "won't work" and throws an error like this at you:

ERROR:  query plan with multiple segworker groups is not supported

If you're not lucky (planner or ORCA schedules the function call on a
different slice running on master), it most likely secretly doesn't do
point b and only returns locks from the master.

Before we fix pg_locks, rewrite the isolation2 test case "starve_case"
to separate the repeated queries in pg_locks from the main query. This
does logically the same thing in PL/pgSQL but it's now safe.

This commit also adds starve_case back to the isolation2 schedule.

Now that the distribution hash algorithm is changed, the running time for the
test case deadlock is unbearable (e.g. 1745.14 sec on pipeline). This patch
fixes this and adds a simple sanity check.
Co-authored-by: NAlexandra Wang <lewang@pivotal.io>
Co-authored-by: NPaul Guo <paulguo@gmail.com>

As gpdb has been merged to postgres 9.2 there are some changes in the
logics, we have noticed that some test cases of GDD, the Global Deadlock
Detector, have been unstable.  Fixed / improved some GDD test cases.

* gdd: upgrade lock in LOCK TABLE command.

GDD (Global Deadlock Detector) need to know whether a relation lock is
releasable before end of transaction, this information stored in a
lock's holdTillEndXact flag.  This flag is set in relation_close() and
LOCK TABLE command used to acquire the lock by `heap_open(lockmode)` and
`heap_close(NoLock)`; but now it calls `LockRelationOid()` directly so
this flag is no longer set by LOCK TABLE command.

To fix the issue now we explicitly set that flag in LOCK TABLE command.

This issue is covered by existing gdd testcase dist-deadlock-07.sql

* gdd: add short delay before recheck gdd period GUC.

The GDD GUC gp_global_deadlock_detector_period can be reloaded via
`gpstop -u`, however this operation might not take effect immediately,
added a short delay before rechecking its value.

This is to improve the stability of the test case gdd/prepare.

* gdd: do not trigger gdd on local deadlock tests

GDD (Global Deadlock Detector) can also detect local deadlocks, however
it might cancel different node(s) with the LDD (Local Deadlock
Detector), this might cause local deadlock tests to hang.

To fix it we move the local deadlock tests into a separate test group,
the GDD checking period is set to a long time so in theory LDD should
always be triggered fist as LDD checking period is hard coded to 5
seconds.

This is to fix the flaky test case gdd/local-deadlock-03.

* gdd: fix typos in test cases.

s/idel/idle/g

* gdd: increase the sleep time in barrier()

In the gdd test cases we employee a helper function barrier() to ensure
the execution order of different transactions on all segments.  So far
the barrier() function is simply implemented via pg_sleep(2), however as
we run many test cases concurrently it's still observed on slow machines
that the execution order is wrong now and then.

As a workaround we increased the sleep time from 2 to 4, this is to
improve the stability of test cases gdd/non-lock-107, etc..

This reverts commit c27a4275.

Revert "Add new test to isolation2_schedule"

This reverts commit 0031400d.

Revert "Fix spelling of the newly added fault name"

This reverts commit 2a40de91.

Pipeline hangs due to concurrent index creation on AO table.

Reference PR #5531.
Co-authored-by: NAsim R P <apraveen@pivotal.io>

We really shouldav done it in c27a4275, oops.

Creating the block directory auxiliary table for an append-only table requires
an AccessExclusive lock (we're not sure why). We defer the creation of the
block directory until creation of the first index on an AO table. This leads to
a lock upgrade later, which causes a dead lock when two concurrent sessions
both believe they are creating the first index.

Acquire a ShareRowExclusiveLock on the target table while creating an index.
This allows only one of the concurrent users to have access to the table at a
time, and then allows the winner to aquire an AccessExclusive lock when
creating the AOBlkDir table.
Co-authored-by: NJimmy Yih <jyih@pivotal.io>
Co-authored-by: NJesse Zhang <sbjesse@gmail.com>

Fixes https://github.com/greenplum-db/gpdb/issues/5551

There was a double free issue which was fixed in 772bca3f, add test
case for it.

    commit 772bca3f
    Author: Pengzhou Tang <ptang@pivotal.io>
    Date:   Tue Mar 20 22:06:50 2018 -0400

        Fix SIGSEGV issue when freeing gangs

        Previously, to avoid the leak of the gang if someone terminates
        the query in the middle of gang creation, we added a global pointer
        named CurrentGangCreating so the partially created gang can also be
        destroyed at the end of the transaction. However, the memory context
        named GangContext where CurrentGangCreating was created may be reset
        before CurrentGangCreating is actually destroyed and a SIGSEGV may
        occur. So this commit makes sure that CurrentGangCreating is destroyed
        ahead of other created gangs and the reset of GangContext.
Co-Authored-by: NJialun Du <jdu@pivotal.io>
Co-Authored-by: NNing Yu <nyu@pivotal.io>

NULLs are not allowed in the PARTITION START or END clauses, but the checks
for NULLs were performed only after sorting the list of partitions. The
sort function did not consider the possibility of NULLs, and would therefore
pass NULLs to comparison functions, even if they're marked as strict. That
could cause a segfault.

Fixes https://github.com/greenplum-db/gpdb/issues/5503

Fixes https://github.com/greenplum-db/gpdb/issues/5511

When the query_planner determines that a relation does not to need
scanning due to constraint exclusion, it will create a 'dummy' plan for
that operation. When we plan a split update, it does not understand this
'dummy' plan shape, and will fail with an assertion.

Instead, because an excluded relation will never return tuples, do not
attempt to create a split update at all.

Co-authored-by: Alexandra Wang<lewang@pivotal.io>
Co-authored-by: Gang Xiong<gxiong@pivotal.io>

Previously, Greenplum uses a very simple algorithm to reduce hash
value to segment id: it just does a modulo computation. Simply
modulo is not suitable for cluster expanding.

In this commit, we change cdbhash to use jump consistent hash algorithm.
More details, please refer the proposal on gpdb-dev mail-list:
https://groups.google.com/a/greenplum.org/forum/#!topic/gpdb-dev/f5bqTzAZjAsCo-authored-by: NZhenghua Lyu <zlv@pivotal.io>
Co-authored-by: NShujie Zhang <shzhang@pivotal.io>
Co-authored-by: NJialun Du <jdu@pivotal.io>
Co-authored-by: NDavid Kimura <dkimura@pivotal.io>
Co-authored-by: NAsim R P <apraveen@pivotal.io>

This is the GPDB side commit for the ORCA commit.

A note on test case changes :

In `subselect_gp` the plans becomes similar to Planner generated plans.

In `qp_gist_indexes4`, we started picking Index Scan over Bitmap Scan.

Bumped the ORCA version to v2.70.0

* Slient DNS error completely when DNSLookupAsError is false

Previously, errors are not thrown up within getAddressesForDBid() if
DNSLookupAsError is false, however, caller still throw up similar
error outside which make FTS probe has no chance to mark segment to
down.

* Force a FTS probe when getCdbComponentInfo() report an error

To don't break following queries, request a FTS probe immediately
when segments info can't be fetched.

* Destroy segment ip cache HTAB within freeCdbComponentDatabases

FTS can't detect a DNS error if the cached segment ip of the hostname
is still accessible. FTS free the cdb_component_dbs in every loop, so
it's reasonable to destroy segment ip cache HTAB too and do a new DNS
resolving in a new loop.

* Add test case to test fts handle dns error

Non-leaf heap partition tables and their auxiliary tables
(aoseg/toast/...) are exempt from age computations.  The test validates
that auxiliary tables of partitioned table are created with relfrozenxid
= 0.
Co-authored-by: NDavid Kimura <dkimura@pivotal.io>

Surely the intention here was to create an index on 'foo' and 'bar' table,
not two indexes on 'foo'. Now the second CREATE UNIQUE INDEX fails, like
the comment says it should.

As part of moving away from Hungarian notation in the GPORCA codebase,
the integration points between GPORCA and GPDB in the translator have
been renamed to the new convention used in GPORCA. The libraries
currently updated to the new notation in GPORCA are Naucrates and GPOS.
The new naming convention is a custom version of common C++ naming
conventions. The style guide for this convention can be found in the
GPORCA repository.

Also bump ORCA version to 2.69.0
Co-authored-by: NShreedhar Hardikar <shardikar@pivotal.io>
Co-authored-by: NMelanie Plageman <mplageman@pivotal.io>
Co-authored-by: NEkta Khanna <ekhanna@pivotal.io>
Co-authored-by: NAbhijit Subramanya <asubramanya@pivotal.io>
Co-authored-by: NSambitesh Dash <sdash@pivotal.io&gt;&lt;Paste>
Co-authored-by: NDhanashree Kashid <dkashid@pivotal.io>
Co-authored-by: NOmer Arap <oarap@pivotal.io>

The purpose of this refactor is to more closely align the GUC with postgres. It
started as a suggestion in https://github.com/greenplum-db/gpdb/pull/4790.
There are still differences, particularly around when this GUC can be set. In
GPDB it can be set by anyone at any time (PGC_USERSET), however in postgres it
is limited to postmaster restart (PGC_POSTMASTER). This difference was kept on
purpose until we have more buy-in as it is a bigger change on the end-user.
Co-authored-by: NJoao Pereira <jdealmeidapereira@pivotal.io>

We don't support this at the moment. Would be nice to make it work, but
in the meanwhile, let's at least give a proper error message, instead of
attempting to read the AO table as if it was a heap table.

There was a FIXME comment about this, and it was also reported in github
issue #4735.

Don't require a USING clause, when there is no cast between the old and
new datatype are not compatible, when altering an external table. It'll
just be ignored, we're not going to rewrite any data in an external
source. And conversely, don't allow USING, because it'll just be ignored.

Fixes https://github.com/greenplum-db/gpdb/issues/3356

In setop plannning, we had assertions that checked that FLOW_SINGLETON
flows had segindex=0. I'm not sure what segindex 0 means; is it "any"?
In any case, it's possible to have an input that resides on a single QE,
different from 0, as evidenced by the new test query.

Fixes https://github.com/greenplum-db/gpdb/issues/3807

This reverts commit 2a38a9cd.

As part of moving away from Hungarian notation in the GPORCA codebase,
the integration points between GPORCA and GPDB in the translator have
been renamed to the new convention used in GPORCA. The libraries
currently updated to the new notation in GPORCA are Naucrates and GPOS.
The new naming convention is a custom version of common C++ naming
conventions. The style guide for this convention can be found in the
GPORCA repository.

Also bump ORCA version to 2.69.0
Co-authored-by: NShreedhar Hardikar <shardikar@pivotal.io>
Co-authored-by: NMelanie Plageman <mplageman@pivotal.io>
Co-authored-by: NEkta Khanna <ekhanna@pivotal.io>
Co-authored-by: NAbhijit Subramanya <asubramanya@pivotal.io>
Co-authored-by: NSambitesh Dash <sdash@pivotal.io&gt;&lt;Paste>
Co-authored-by: NDhanashree Kashid <dkashid@pivotal.io>
Co-authored-by: NOmer Arap <oarap@pivotal.io>

When SQL standard table inheritance was added in upstream (by commit
2fb6cc90 in Postgres 7.1), mentioning a table in the FROM clause of a
query would necessarily mean traversing through the inheritance
hierarchy. The need to distinguish between the (legacy, less common, but
legitimate nonetheless) intent of not recursing into child tables gave
rise to two things: the guc `sql_inheritance` which toggles the default
semantics of parent tables, and the `ONLY` keyword used in front of
parent table names to explicitly skip descendant tables.

ORCA doesn't like queries that skip descendant tables: it falls back to
the legacy planner as soon as it detects that intent.

Way way back in Greenplum-land, when external tables were given a
separate designation in relstorage (RELSTORAGE_EXTERNAL), we seemed to
have added code in parser (parse analysis) so that queries on external
tables *never* recurse into their child tables, regardless of what the
user specifies -- either via `ONLY` or `*` in the query, or via guc
`sql_inheritance`. Technically, that process scrubs the range table
entries to hard-code "do not recurse".

The combination of those two things -- hard coding "do not recurse" in
the RTE for the analyzed parse tree and ORCA detecting intent of `ONLY`
through RTE -- led ORCA to *always* fall back to planner when an
external table is mentioned in the FROM clause. commit 013a6e9d tried
fixing this by *detecting harder* whether there's an external table.

The behavior of the parse-analyzer hard coding a "do not recurse" in the
RTE for an external table seems wrong for several reasons:

  1. It seems unnecessarily defensive

  2. It doesn't seem to belong in the parser.

     a. While changing "recurse" back to "do not recurse" abounds, all
     other occurrences happen in the planner as an optimization for
     childless tables.

     b. It deprives an optimizer of the actual intent expressed by the
     user: because of this hardcoding, neither ORCA nor planner would
     have a way of knowing whether the user specified `ONLY` in the
     query.

     c. It deprives the user of the ability to use child tables with an
     external table, either deliberately or coincidentally.

     d. A corollary is that any old views created as `SELECT a,b FROM
     ext_table` will be perpetuated as `SELECT a,b FROM ONLY ext_table`.

This commit removes this defensive setting in the parse analyzer. As a
consequence, we're able to reinstate the simpler RTE check before commit
013a6e9d. Queries and new views will include child tables as expected.

Fixes https://github.com/greenplum-db/gpdb/issues/1015

The ResultRelInfos we build for the partitions, in slot_get_partition(),
don't contain the ProjectInfo needed to execute RETURNING. We need to
look that up in the parent ResultRelInfo, and when executing it, be
careful to use the "parent" version of the tuple, the one before
mapping the columns for the target partition.

Fixes github issue #4735.

Index only scan is a new feature in PG9.2 merge. We do not have the support in
eagerfree related functions.

Previously, cdbdisp_finishCommand did three things:
1. cdbdisp_checkDispatchResult
2. cdbdisp_getDispatchResult
3. cdbdisp_destroyDispatcherState

However, cdbdisp_finishCommand didn't make code cleaner or more
convenient to use, in contrast, it makes error handling more
difficult and makes code more complicated and inconsistent.

This commit also reset estate->dispatcherState to NULL to avoid
re-entry of cdbdisp_* functions.

Previously, CdbDispatchPlan might be called to dispatch nothing if
1. init plan is parallel but main plan is not. CdbDispatchPlan is
still called for main plan.
2. init plan is not parallel, CdbDispatchPlan is still called for
init plan.

The reason is DISPATCH_PARALLEL stands for the whole plan include
main plan and init plans, this commit add ways to seperately tell
which plan is parallel exactly to avoid unnecessary dispatching.

The distributed_transactions test contains a serializable
transaction. This serializable transaction may intermittently cause
the appendonly test to fail when run in the same test group. The
appendonly test runs VACUUM on some appendonly tables and checks that
last_sequence is nonzero in gp_fastsequence. Serializable transactions
make concurrent VACUUM operations on appendonly tables exit early.

To fix the contention, let's move the distributed_transactions test to
another test group.

appendonly test failure diff:
*** 632,640 ****
   NormalXid |      0 | t        |             0
   NormalXid |      0 | t        |             1
   NormalXid |      0 | t        |             2
!  NormalXid |      1 | t        |             0
!  NormalXid |      1 | t        |             1
!  NormalXid |      1 | t        |             2
  (6 rows)

--- 630,638 ----
   NormalXid |      0 | t        |             0
   NormalXid |      0 | t        |             1
   NormalXid |      0 | t        |             2
!  NormalXid |      1 | f        |             0
!  NormalXid |      1 | f        |             1
!  NormalXid |      1 | f        |             2
  (6 rows)

Repro:
1: CREATE TABLE heap_table (a int, b int);
1: INSERT INTO heap_table SELECT i, i FROM generate_series(1,100)i;
1: CREATE TABLE ao_table WITH (appendonly=true) AS SELECT * FROM heap_table;
1: SELECT gp_segment_id, * FROM gp_dist_random('gp_fastsequence') WHERE gp_segment_id = 0;
2: BEGIN ISOLATION LEVEL SERIALIZABLE;
2: SELECT 1;
1: VACUUM ao_table; -- VACUUM exits early
1: SELECT gp_segment_id, * FROM gp_dist_random('gp_fastsequence') WHERE gp_segment_id = 0;
2: END;
1: VACUUM ao_table; -- VACUUM completes
1: SELECT gp_segment_id, * FROM gp_dist_random('gp_fastsequence') WHERE gp_segment_id = 0;

- Created the needed control files
- Renamed and adapted the installation script
- Remove redundant tests in regress that were just checking installation
- Add tests to ensure installation is successfull
- Updated the Makefile to support the extension and add information
about regression tests
- Add contrib/gp_internal_tools tests to ICW
- Remove session_level_memory_consumption from regress test suite

Prior to this commit, there was no support for GiST indexes in GPORCA.
For queries involving GiST indexes, ORCA was selecting Table Scan paths
as the optimal plan. These plans could take up to 300+ times longer than
Planner, which generated a index scan plan using the GiST index.

Example:
```
CREATE TABLE gist_tbl (a int, p polygon);
CREATE TABLE gist_tbl2 (b int, p polygon);
CREATE INDEX poly_index ON gist_tbl USING gist(p);

INSERT INTO gist_tbl SELECT i, polygon(box(point(i, i+2),point(i+4,
i+6))) FROM generate_series(1,50000)i;
INSERT INTO gist_tbl2 SELECT i, polygon(box(point(i+1, i+3),point(i+5,
i+7))) FROM generate_series(1,50000)i;

ANALYZE;
```
With the query `SELECT count(*) FROM gist_tbl, gist_tbl2 WHERE
gist_tbl.p <@ gist_tbl2.p;`, we see a performance increase with the
support of GiST.

Before:
```
EXPLAIN SELECT count(*) FROM gist_tbl, gist_tbl2 WHERE gist_tbl.p <@ gist_tbl2.p;
                                                     QUERY PLAN
---------------------------------------------------------------------------------------------------------------------
 Aggregate  (cost=0.00..171401912.12 rows=1 width=8)
   ->  Gather Motion 3:1  (slice2; segments: 3)  (cost=0.00..171401912.12 rows=1 width=8)
         ->  Aggregate  (cost=0.00..171401912.12 rows=1 width=8)
               ->  Nested Loop  (cost=0.00..171401912.12 rows=335499869 width=1)
                     Join Filter: gist_tbl.p <@ gist_tbl2.p
                     ->  Table Scan on gist_tbl2  (cost=0.00..432.25 rows=16776 width=101)
                     ->  Materialize  (cost=0.00..530.81 rows=49997 width=101)
                           ->  Broadcast Motion 3:3  (slice1; segments: 3)  (cost=0.00..525.76 rows=49997 width=101)
                                 ->  Table Scan on gist_tbl  (cost=0.00..432.24 rows=16666 width=101)
 Optimizer status: PQO version 2.65.1
(10 rows)

Time: 170.172 ms
SELECT count(*) FROM gist_tbl, gist_tbl2 WHERE gist_tbl.p <@ gist_tbl2.p;
 count
-------
 49999
(1 row)

Time: 546028.227 ms
```

After:
```
EXPLAIN SELECT count(*) FROM gist_tbl, gist_tbl2 WHERE gist_tbl.p <@ gist_tbl2.p;
                                                  QUERY PLAN
---------------------------------------------------------------------------------------------------------------
 Aggregate  (cost=0.00..21749053.24 rows=1 width=8)
   ->  Gather Motion 3:1  (slice2; segments: 3)  (cost=0.00..21749053.24 rows=1 width=8)
         ->  Aggregate  (cost=0.00..21749053.24 rows=1 width=8)
               ->  Nested Loop  (cost=0.00..21749053.24 rows=335499869 width=1)
                     Join Filter: true
                     ->  Broadcast Motion 3:3  (slice1; segments: 3)  (cost=0.00..526.39 rows=50328 width=101)
                           ->  Table Scan on gist_tbl2  (cost=0.00..432.25 rows=16776 width=101)
                     ->  Bitmap Table Scan on gist_tbl  (cost=0.00..21746725.48 rows=6667 width=1)
                           Recheck Cond: gist_tbl.p <@ gist_tbl2.p
                           ->  Bitmap Index Scan on poly_index  (cost=0.00..0.00 rows=0 width=0)
                                 Index Cond: gist_tbl.p <@ gist_tbl2.p
 Optimizer status: PQO version 2.65.1
(12 rows)

Time: 617.489 ms

SELECT count(*) FROM gist_tbl, gist_tbl2 WHERE gist_tbl.p <@ gist_tbl2.p;
 count
-------
 49999
(1 row)

Time: 7779.198 ms
```

GiST support was implemented by sending over GiST index information to
GPORCA in the metadata using a new index enum specifically for GiST.
Signed-off-by: NBhuvnesh Chaudhary <bchaudhary@pivotal.io>
Signed-off-by: NEkta Khanna <ekhanna@pivotal.io>

I don't think anyone's tried to run this on Greenplum, until @imshubhangi
reported in github issue #5241, that it doesn't work. Most of the failures
are just a matter of updating the expected output for extra NOTICES, and
using gpdiff.pl to mask out differences in row order. That's what this
patch does.

However, there is one remaining failure, in the COPY TO STDOUT at the
end of the 'sjis' test, which looks like a genuine bug. That needs to be
investigated and fixed, but let's do that as a separate commit.

Changed only the files regress_gp