1. Add custom external table dependency with protocol to keep the dumped
   SQL ordered correctly. (create protocol must execute before creating custom
   external table)

2. Remove \" from dumped location URI

3. Support dump multiple locations
Signed-off-by: NPeifeng Qiu <pqiu@pivotal.io>
Signed-off-by: NYuan Zhao <yuzhao@pivotal.io>

Commit 2a3df5d0 introduced broken changes to the ereport() calls in the
external table command. Fix by including the proper format specifier.
Also concatenate the error message to a single row while at it to
improve readability.

Spotted by Heikki Linnakangas

Update the .ans file to conform to the test output for 5.0.

Seen failures multiple times in pipeline at point for checking if database is in
changetracking or not for failover_to_mirror scenarios. For this case primary is
paniced and mirror takes over but due to work-load running, master may see
panics due to 2PC erroring out after retries. The test should be able to
continue working in such a condition, hence adding wait_for_database to be up
after master panic.

* Timeout was removed from the WorkerPool on commit
  c7dfcc45

When dropping an external table, the pg_exttable catalog entry must
be removed as well to avoid dangling entries building up. Reverse
the logic in deleting the catalog entry to make sure we delete it
before scanning for a potential next one. Also fix up the tests for
this as they were testing the removal of the pg_class catalog entry
rather than the pg_exttable entry.

Reported by Ashwin Agrawal in Github issue #1498

Signed-off-by: NYuan Zhao <yuzhao@gmail.com>

External tables with protocols file or http which specify more URIs
than segments won't be allowed to query until the cluster has been
extended. Rather than silently allow creation, issue a warning and
hint to the user to assist troubleshooting. Writable tables error
out for the same problem but avoiding to change behavior this late
in the release cycle seems like a good idea.

Creating an external table with duplicate location URIs is a syntax
error so move the duplicate check to the parsing step. This simplifies
the external table command code as well as provides an error message
with context marker to the user.

Also fix up test sources to match the new error messages and remove
the duplicate test from gpfdist since we already test that in ICW.

This was fixed in GPDB 4.3 immediately after it was brought up but was
never ported to 5.0. Issue was introduced in
https://github.com/greenplum-db/gpdb/commit/232eb64ad9f93dce8941f7b124a98f0c21c3350b
which has the initial discussion.

In rollup() processing, when the grouping cols are processed, the target list is
traversed for each grouping col to find a match. Each node in the target list is
mutated in `replace_grouping_columns_mutator()` to find match for grouping cols.

In the failing query:

```
SELECT COUNT(DISTINCT c) AS r
FROM (
    SELECT c
    , CASE WHEN (e = 2) THEN 1 END AS f
    , 1 AS g FROM foo
    ) f_view
GROUP BY ROLLUP(f,g);
```

grouping cols ‘g’ is a constant 1. when target entry for ‘f’ is being mutated,
we mutate all the fields of caseexpr recursively. since the `result` field in
caseexpr is a const with same value as that of ‘g’, it gets considered as a
match and we replace the node by null which later results in crash.

Essentially, since we traverse all the fields of expr, a const
node (appearing anywhere in caseexpr) with same value as that of const grouping
col will be matched.

Hence the following query also fails where one of the arg of equality operator
is a const with same value as that of ‘g’.

```
SELECT COUNT(DISTINCT c) AS r
FROM (
    SELECT c
    , CASE WHEN (e = 1) THEN 2 END AS f
    , 1 AS g FROM foo
    ) f_view
GROUP BY ROLLUP(f,g);
```

Fix:
1. refactored the `replace_grouping_columns_mutator()` to have separate
routines for target list replacement and qual replacement.
2. for target list, just look at top level node. do not recurse.
3. for quals, the top level is always some kind of expression, so we
need to recurse to find the match. Skip replacing constants here.

[#136294589]
Signed-off-by: NXin Zhang <xzhang@pivotal.io>

This verifies PR #1514

Also fix case break due to options keyword.
Signed-off-by: NYuan Zhao <yuzhao@pivotal.io>

Signed-off-by: NAdam Lee <ali@pivotal.io>

Put it at the tail-end of the pipeline first for visibility, and then we
can iterate to get it quicker.
Signed-off-by: NYuan Zhao <yuzhao@pivotal.io>

These tests do not really give us any extra coverage different from
our other TINC catalog and storage tests. Historically, they haven't
flagged any persistent table issues we have seen in the field and we
were surprised it didn't flag any persistent table issues from the 8.3
merge but other TINC tests did. Therefore, it is safe to remove these
tests without fear of losing coverage.

==================================================
periodic_failover: 3 hours 8 minutes (2 tests)
 - skip checkpoint fault (yes/no)
 - kills segments periodically while running SQLLoadTest concurrency workload
   - ~80 minutes of sleep consisting of sleeping every 5-10 minutes before segment killing thread wakes up
 - recover full or recover incrementally (depends on skip checkpoint yes/no)
 - gpcheckcat, wait until database is in-sync, checkmirrorintegrity

==================================================
Drop_DB: 8 minutes (1 test)
 - move database base dir and try to dropdb
 - move database base dir back and try to dropdb
 - createdb back the database
 - gpcheckcat, wait until database is in-sync, checkmirrorintegrity

==================================================
crash_system: 1 hour 11 minutes (2 tests)
 - skip checkpoint fault (yes/no)
 - test crash after checkpoint (in-sync or changetracking) while running SQLLoadTest concurrency workload
   - sleeps for 10-15 minutes while SQLLoadTest concurrency workload runs
   - if changetracking needed, puts system in changetracking and sleeps for another 10-15 minutes
   - issues checkpoint
   - restart database (gpstop immediate and gpstart)
 - if changetracking, do incremental recovery
 - gpcheckcat, wait until database is in-sync, checkmirrorintegrity

==================================================
kill_primaries: 25 minutes (2 tests + partition_table_scenario)
 - skip checkpoint fault (yes/no)
 - failover to mirror while running SQLLoadTest concurrency workload
   - sleeps for 2 minutes
   - kills all primary segments
   - restart database (gpstop immediate and gpstart)
 - run incremental recovery
 - gpcheckcat, wait until database is in-sync, checkmirrorintegrity

partition_table_scenario: (1 test)
 - runs PartitionTableQueries workload
   - create tiny amount of different partition tables in a transaction but finish without commiting or aborting
 - issue checkpoint
 - restart database (gpstop immediate and gpstart)
 - run incremental recovery
 - gpcheckcat, wait until database is in-sync, checkmirrorintegrity

==================================================
postmaster_reset: 1 hour 15 minutes (4 tests)
 - put database in change tracking (yes/no)
 - kill all or half segment processes while running SQLLoadTest concurrency workload
   - sleeps for 2 minutes before killing segment processes
   - tries to restart database (gpstop immediate and gpstart) in an infinite loop after killing segment processes
 - run incremental recovery
 - gpcheckcat, wait until database is in-sync, checkmirrorintegrity

==================================================
OOM: 1 hour 41 minutes (2 tests)
 - sleep for 40 minutes while JoinQueryTest workload runs to create OOM
   - if with load needed, runs SQLLoadTest concurrency workload concurrently with the JoinQueryTest workload
   - assumes 20 concurrent threads in 2 iterations will produce OOM on VM (with comment saying need to increase for physical systems)
 - run incremental recovery and hits database with sql query every 10 seconds to see if database is up
 - gpcheckcat, wait until database is in-sync, checkmirrorintegrity

The test_PT_RebuildPT.py tests are spared as they test our PT rebuild
tool. These tests will be migrated to our Behave framework later in
the future.

Signed-off-by: NXin Zhang <xzhang@pivotal.io>

In a few cases planner makes a wrong decision about converting a
correlated subquery into a join, which results in:

0. Wrong results because the transformation was not equivalent; OR
0. Execution errors because the plans contained invalid params (across
  slices)

This commit adds tests for those cases.

Considering following repro with correlated subquery:

```
-- foo.c is a text
select * from foo where not exists (select * from bar where foo.c = foo.c || 'a');
```

This query will crash because referring param across slices. Here is the plan:

```
xzhang=#  explain select * from foo where not exists (select * from bar where foo.c = foo.c || 'a');
                                             QUERY PLAN
-----------------------------------------------------------------------------------------------------
 Gather Motion 3:1  (slice3; segments: 3)  (cost=1.06..2.13 rows=4 width=3)
   ->  Hash Left Anti Semi Join  (cost=1.06..2.13 rows=2 width=3)
         Hash Cond: foo.c::text = "Expr_SUBQUERY".csq_c0
         ->  Redistribute Motion 3:3  (slice1; segments: 3)  (cost=0.00..1.03 rows=1 width=3)
               Hash Key: foo.c::text
               ->  Seq Scan on foo  (cost=0.00..1.01 rows=1 width=3)
         ->  Hash  (cost=1.05..1.05 rows=1 width=32)
               ->  Redistribute Motion 3:3  (slice2; segments: 3)  (cost=0.01..1.05 rows=1 width=32)
                     Hash Key: "Expr_SUBQUERY".csq_c0
                     ->  Result  (cost=0.01..1.02 rows=1 width=0)
                           One-Time Filter: ($0::text || 'a'::text) IS NOT NULL
                           ->  Seq Scan on bar  (cost=0.01..1.01 rows=1 width=0)
 Optimizer status: legacy query optimizer
(13 rows)
```

IsCorrelatedOpExpr() is checking an opexpr in format of:

```
foo(outer.var) OP bar(inner.var)
```

Then, extract the bar(inner.var) as innerExpr.

It assumes one side of the operator must be an inner
variable, and one side of the operator must be an outer variable.

However, there are cases where both of `opexp->args` are referring
to outer variables. Then, it mistakenly pick one side as inner variable.

Add a check to fail to extract `innerExpr` if both sides of opexr are outer variables
and return false.

Then, the subquery cannot be decorrelated, and hence remain as subplan. The correct query
plan is:

```
gpadmin=# explain select * from foo where not exists (select * from bar where foo.c = foo.c || 'a');
                                                      QUERY PLAN
----------------------------------------------------------------------------------------------------------------------
 Gather Motion 3:1  (slice2; segments: 3)  (cost=0.00..1474.89 rows=26400 width=32)
   ->  Seq Scan on foo  (cost=0.00..1474.89 rows=8800 width=32)
         Filter: NOT ((subplan))
         SubPlan 1
           ->  Result  (cost=0.01..1062.01 rows=32067 width=4)
                 One-Time Filter: $0 = ($0 || 'a'::text)
                 ->  Result  (cost=1158.21..2120.20 rows=32067 width=4)
                       ->  Materialize  (cost=1158.21..2120.20 rows=32067 width=4)
                             ->  Broadcast Motion 3:3  (slice1; segments: 3)  (cost=0.01..1062.01 rows=32067 width=4)
                                   ->  Seq Scan on bar  (cost=0.01..1062.01 rows=32067 width=4)
 Optimizer status: legacy query optimizer
(11 rows)
```

[#135636211]
Signed-off-by: NHaiSheng Yuan <hyuan@pivotal.io>

Revert "Updating qp_dpe_optimizer.out to reflect the updated qp_dpe.out because of partition selector undo selection. (#1462)"

This reverts commit ce765c03.

This reverts commit 48c495a1.

Update the .ans file to conform to the test.

This reverts commit 7cfe487b.
Signed-off-by: NXin Zhang <xzhang@pivotal.io>

root causes:
1, cxt.rel was not initialized in transformCreateExternalStmt()
2, creating external table was considered as a simple single statement, but it should be a list of statements with extra actions.
Signed-off-by: NPeifeng Qiu <pqiu@pivotal.io>

Update catversion in 4.3.json for keyword 'OPTIONS'.

In case user wanna specify parameters with OPTIONS other than config
file in LOCATION, it provides flexible and friendly grammar.

Usage:
CREATE EXTERNAL TABLE xxx LOCATION xxx FORMAT xxx
[OPTIONS ( [option_key 'option_value' [, ...]] )]
xxx;
Signed-off-by: NKuien Liu <kliu@pivotal.io>
Signed-off-by: NPeifeng Qiu <pqiu@pivotal.io>
Signed-off-by: NAdam Lee <ali@pivotal.io>
Signed-off-by: NYuan Zhao <yuzhao@pivotal.io>
Signed-off-by: NJesse Zhang <jzhang@pivotal.io>

In rollup() processing, when the grouping cols are processed, the target list is
traversed for each grouping col to find a match. Each node in the target list is
mutated in `replace_grouping_columns_mutator()` to find match for grouping cols.

In the failing query:

```
SELECT COUNT(DISTINCT c) AS r
FROM (
    SELECT c
    , CASE WHEN (e = 2) THEN 1 END AS f
    , 1 AS g FROM foo
    ) f_view
GROUP BY ROLLUP(f,g);
```

grouping cols ‘g’ is a constant 1. when target entry for ‘f’ is being mutated,
we mutate all the fields of caseexpr recursively. since the `result` field in
caseexpr is a const with same value as that of ‘g’, it gets considered as a
match and we replace the node by null which later results in crash.

Essentially, since we traverse all the fields of expr, a const
node (appearing anywhere in caseexpr) with same value as that of const grouping
col will be matched.

Hence the following query also fails where one of the arg of equality operator
is a const with same value as that of ‘g’.

```
SELECT COUNT(DISTINCT c) AS r
FROM (
    SELECT c
    , CASE WHEN (e = 1) THEN 2 END AS f
    , 1 AS g FROM foo
    ) f_view
GROUP BY ROLLUP(f,g);
```

Fix:
1. refactored the `replace_grouping_columns_mutator()` to have separate
routines for target list replacement and qual replacement.
2. for target list, just look at top level node. do not recurse.
3. for quals, the top level is always some kind of expression, so we
need to recurse to find the match. Skip replacing constants here.
Signed-off-by: NXin Zhang <xzhang@pivotal.io>

* Backs up search path, optimizer settings, and gp_default_storage_options
* These settings will only be restored during database restores (`-e` option)

Updating qp_dpe_optimizer.out to reflect the updated qp_dpe.out because of partition selector undo selection. (#1462)

The test_skip_checkpoint_abort_transaction.py test is testing fix for
MPP-17297/MPP-17817 where a failure in commit processing the standby was not
handled properly by the master.  However, the code changes in the original bug
fix in 4.2.3.0 are no longer applicable to the 5.0 code line, and the replication
mechanism between standy and master has changed to walrep since then.

Therefore, I am removing this test altogether.

The test_mpp20964.py test checks the pg_locks table for any locks that have not
been released by the finished transactions.  However, the testcase is no longer
applicable to the 5.0 branch as even after backing out the fix for the original
issue in cdbdisp.c:

```
diff --git a/src/backend/cdb/dispatcher/cdbdisp.c b/src/backend/cdb/dispatcher/cdbdisp.c
index 39cdea7..a00d35f 100644
--- a/src/backend/cdb/dispatcher/cdbdisp.c
+++ b/src/backend/cdb/dispatcher/cdbdisp.c
@@ -215,8 +215,8 @@ cdbdisp_finishCommand(struct CdbDispatcherState *ds)
         * appear anymore and will finish soon. Also ERROR report doesn't
         * go to the client anyway since we are in proc_exit.
         */
-   if (proc_exit_inprogress)
-           return;
+ //if (proc_exit_inprogress)
+ //  return;

        /*
         * Wait for all QEs to finish. Don't cancel them.
```

the pg_locks table does not contain any locks held by finished transactions.

Therefore, I am removing this test.

add gpaddon_src to pr_pipeline.yml file.