These tests are very tightly coupled with filerep and persistent
tables implemenation.

We do need to have tests in replacement but these would be coded not
in TINC and mostly much less scenarios as in general walreplication
mirrors active for full ICW validates most of crashrecovery
logic. What's missing mostly is real crash scenarios in middle of
active workload like creating/dropping objects and all.

Unfortunately the cluster crashes anyway two tests later. Rather than
comment out half the tests to get a fake green, put this set of tests
back. We'll just have to solve this one problem at a time.

This reverts commit 5982a72614492916187ca27fc660d7cc7e3b69e1.

The promotion logic that gpactivatestandby relies on doesn't work yet,
and when these tests fail, they leave the cluster completely unusable.

This removes the make target storage_filerep.

test_switch_13_24.py was using fault injector `dtm_xlog_distributed_commit`,
while test_switch_01_12.py has tests for `dtm_broadcast_commit_prepared`. From
code `dtm_xlog_distributed_commit` is set after commit record is written in
`RecordTransactionCommit()` and `dtm_broadcast_commit_prepared` is set just
before commit prepared is broadcasted to segments in
`doNotifyingCommitPrepared()` which gets called right after
`RecordTransactionCommit()`. No 2PC state change between these two fault
injector points making the tests at these 2 points redundant.

Hence, reducing one more box on CI by moving some combinations to
test_switch.py and deleting the test_switch_13_24.py.

Flow for test permutations was:
0. Runs some pre-SQLs
1. Gets the cluster in desired state
2. pg_switch_xlog() on all the segments
3. Injects suspend fault for desired fault
4. Runs some triggers-SQLs
5. Crashes gpstop -i and start the db
6. 5 times calls pg_switch_xlog()
7. Runs some post-SQLs
8. gpcheckcat and gpcheckmirrorseg

- cluster_states: SYNC, CT, RESYNC
- fault_types: dtm_broadcast_prepare, dtm_broadcast_commit_prepared
- crash_types: gpstop_i

This seems all covered and redundant with cs_pg_twophase_switch_01_12 and
cs_pg_twophase_switch_13_24, as only diference in flow in step 6. But don't
think that makes any difference to what's being tested.

The storage/basic/partition tests were a collection of bugfixes from
primarily older version of Greenplum. This moves the valuable tests
over to ICW and removes the ones which are already covered in existing
ICW suites. The decision for each individual tests is elaborated on in
the list below:

* MPP-3379 was testing an ancient bug and MPP-3553 a hypothetical
  bug from before the current partition code was written. Combined
  the two since the added clause will still cover the ancient 3379
  issue and remove the other from Tinc.

* MPP-3625 was mostly already covered by existing tests in partition
  and bfv_partition. Added a test for splitting a non-existing default
  as that was the only thing covered.

* MPP-6297 is testing the pg_get_partition_def() function after the
  partition hierarchy has been amended via ALTER TABLE, which is
  already covered by the partition suite. Additionally it tested
  running pg_dump, which use pg_get_partition_def() heavily, and
  this is covered by our pg_upgrade tests.

* MPP-6379 tested that partitions inherited unique indexes, the test
  was moved to the partition_indexing suite.

* MPP-6489 tested ALTER TABLE .. SET DISTRIBUTED BY for subpartitions
  which didn't seem covered in ICW so moved to alter_distribution_policy
  suite

* MPP-7661 tested for an old bug where pg_dump incorrectly handled
  partition hierarchies created with the EVERY syntax. pg_upgrade
  tests in ICW will run this test on hierarchies from the partition
  suites so remove.

* MPP-7734 tested for excessive memory consumption in the case of a
  very deep partitioning hierarchy. The issue in question was fixed
  in 4.0 and the test clocks in at ~ 1.5 minutes, so remove the test
  to save time in the test pipeline. The test was more of a stress
  test than a regression test at this point, and while thats not of
  no importance, we should run stresstests deliberately and not
  accidentally.

* MPP-6537/6538 tested that new partitions introduced in the hierarchy
  correctly inherited the original owner. Test moved to the partition
  suite.

* MPP-7002 tested that splitting partitions worked after a column
  had been dropped. Since we had a similar test already in the
  partition suite, extend that test to also cover splitting.

* MPP-7164 is testing for partition rank after DDL operations on the
  partition hierarchy. I can see that we are testing something along
  the lines of this in ICW so pulled it across. In the process, fixed
  the test to actually run the SQL properly and not error out on a
  syntax error. Also removed a duplicated file for setting up the views.

* MPP-9548 tested for the case of insert data into a column which
  was dropped and then re-added. Since we already had that covered
  in the partition suite, simply extended the comment on that case
  and added another projection to really cover it completely.

* MPP-7232 tests that pg_get_partition_def() supports renamed partitions
  correctly. Added to ICW but the pg_dump test removed.

* MPP-17740 tested the .. WITH (tablename = <t> ..) syntax which we
  to some extent already covered, but added a separate case to test
  permutations of it.

* MPP-18673 covers an old intermittent bug where concurrent sessions
  splitting partitions would corrupt the relcache. The test in Tinc
  didn't however run concurrently so the underlying bug wasn't being
  tested, so remove the test. If we want a similar test at some point
  it should be an isolationtester suite.

* MPP-17761 tests an old bug where splitting an added CO partition
  failed. Since we didn't have much in terms of testing splitting
  CO, extended the block doing split testing with that.

* MPP-17707-* were essentially the same test but with varying storage
  options. While a lot of this is covered elsehwere, these tests were
  really easy to read so rolled them all into a new suite called
  partition_storage.

* MPP-12775 covered yet another split/exchange scenario. Added a
  short variant to the partition suite.

* MPP-17110 tests for an old regression in attribute encoding for
  added columns in partitioning hierarchies. Removed the part of
  the test that checked compression ratio as AO compression should
  be tested elsewhere.

* The partition_ddl2 test was moved over as partition_ddl more or
  less unchanged.

This also removes unused answer files mpp8031.ans.orca and query99
for which there were no corresponding tests, as well as the data
file used for copying data into the tests (a copy of which already
exists in src/test/regress/data).

Since GPDB has a lot of downstream code in the COPY codepath, testing
that we are still able to interrupt a COPY in progress is of interest.
Move the Tinc test to ICW and in the process make it actually test the
interruption by adding enough data into the testpath that it will be
able to hit before the copy is done. The original coding only added
100 rows to be copied which always finishes well before the interrupt
fault injection manages to run.

The indexes on these tests were quite useless. In 'alter_tab_drop_col_2',
the DROP COLUMN also dropped the index, so we got no chance to use it in
any queries. In 'alter_tab_drop_col_3', ORCA doesn't know how to use the
index for the IS FALSE query. In the translation from TINC, I removed the
former case, but kept the latter one, and also added another query where
ORCA also know how to use the index.

This removes the last tests from the QP_optimizer-functional test suite,
so remove all traces of that from the Makefiles and the Concourse pipeline.

These so-called boundary tests didn't actually go very close to the max size
of a text/bytea/varchar column. The maximum of a single datum is 1 GB, and
the value used here was only 10 MB.

I was tempted to just remove the tests, but since someone felt that we need
to test largish values, let's try to keep the coverage. So to compensate
for the removal of the TINC tests, add similar test queries to the "toast"
pg_regress test. Also make one of the large columns the distribution key,
so that we test detoasting of the distribution key.

I don't think using the subquery in the INSERT and UPDATE was the point of
the test, so I replaced that with a much faster function to generate test
data.

We had tests of a varchar's length limit in the 'varchar' test already, so
that aspect of the dml_boundary_varchar test also didn't seem interesting.

This removes the last tests from the "optimizer_functional_part2" set of
tests. Remove that from the Makefile and concourse pipeline file.

* abandon pulse and use CCP framework
* add new tinc target for mpp_interconnect
* put source code of ickm along with interconnect test scripts.

Signed-off-by: NJimmy Yih <jyih@pivotal.io>

Migrating gpexpand test from using pulse to terraform. This uses a new
feature added in CCP "NUMBER_OF_NODES" which deploys a specific amount
of VMs for the cluster.

Add CUSTOM_ENV and PRE_TEST_SCRIPT properties in order to run tests
with specific expectations.

There was probably coverage for some of these in the existing tests
already, but I couldn't quickly pinpoint the exact tests, so let's at least
move these out of TINC.

These tests executed various built-in functions, in different contexts
like in a FROM clause, or in a subquery. We have similar tests for
functions with different attributes in the main test suite's
'qp_functions_in_*' tests. The fact that these are built-in functions
doesn't make any difference to planning, so remove these tests as
redundant.

Also remove some useless Makefile targets.

Signed-off-by: NChris Hajas <chajas@pivotal.io>

Now that subtransaction information is no longer maintained in shared
snapshot between readers and a writer, the tests are obsolete.  Faults
used by the tests are removed as well.

This commit contains a gpdb_master pipeline change - to remove
"subtransaction" job that used to run the tests being removed by this
commit.
Signed-off-by: NAsim R P <apraveen@pivotal.io>

auto-vacuum limit would be reached first and then warn limit, followed by other
limits. So, there is no reason to rollback after bumping the xid to auto-vacuum
limit. Can land in all kinds of wierd issues by doing the same. Practically,
these tests need to be fully re-written maybe by modifying the GUCs and then
actually generating XID to reach the same instead of similating by bumping the
counter, but will attempt that in another commit.

I modified the test slightly, to eliminate the sleep, making the test run
faster.

These targets are not invoked directly anywhere so remove them.

The fault `twophase_transaction_commit_prepared` would cause the segment to
PANIC while writing a `COMMIT PREPARED` record on the segment. The master would
then retry the `COMMIT PREPARED` while the postmaster was still resetting on the
segment causing the master itself to `PANIC`. This patch introduces a new fault
injector `finish_prepared_before_commit`  which will not cause a `PANIC`, but
just error out since the intent of the test is to ensure that the retry works
correctly.

This test stage ran filerep_end_to_end test suite work-load and peformed crash
testing with the same. Takes around 2 hours for completion, It runs with
extremely heavy lifting things like gpfdist, resource queues, etc... which don't
seem of too much interest to crash testing. In long run really need much
simplfied crash recovery testing by creating wrk load to create all kinds of
xlog records in system and recover with it instead of piggy-backing for
work-loads on these already heavy test jobs.

- Also remove unwanted test: the abort test should be skipped,
  based on feedback from Storage team
Signed-off-by: NC.J. Jameson <cjameson@pivotal.io>

The storage_persistent_filerep_accessmethods_and_vacuum task contained
two filerep tests:

FilerepResync.test_filerep_resysnc ... 585576.51 ms ... ok
FilerepProcArrayRemoveTestCase.test_verify_ct_after_procarray_removal ... 136553.63 ms ... ok

Move them together with the filerep_end_to_end_xlog_ctlog_cons task, so that
all the filerep tests are in the same task. To reflect that, rename the
'filerep_end_to_end_xlog_ctlog_cons task' to just 'filerep'.

The 'storage_persistent_filerep_accessmethods_and_vacuum' task was the
slowest task among the parallel tasks in the job, so moving the two filerep
tests elsewhere should reduce overall runtime of the storage job.

The test launched one backend, and ran ALTER TABLE ADD COLUMN in it. Before
committing the ALTER TABLE transaction, it launched another backend, and
ran a query against the table. That will block, because the ALTER TABLE
is holding an AccessExclusiveLock on the table. It then committed the
first transaction, letting the second transaction to continue.

That's not a very interesting test case. ALTER TABLE grabs an
AccessExclusiveLock very early, so it's hard to imagine it would fail to
hold it. Also hard to imagine what could go wrong in having the other query
block on the lock. Backends frequently block on all kinds of locks, for
short periods of time, so just blocking on a lock isn't very interesting.
Furthermore, AFAICS, if the ALTER TABLE somehow failed to acquire the lock,
so that the query didn't block, the test would return the same output, and
therefore wouldn't catch that case.

There is no tincrepo/mpp/gpdb/tests/utilities/upgrade directory.

In addition to pipeline changes, Some tests are modified slightly so
that they need less data overall to go OOM.

There are only a few tests left in "uao", most have now been moved to
installcheck-world. The "uao" job now runs in about 4 minutes, of which
2 minutes is spent on setting up the test cluster. That seems like
overkill, so simplify things by merging the remaining uao fault injection
tests into the larger "storage" job.

I removed these tests in commit 1398c14e, but forgot to remove it from
this Makefile.

These tests merely test that "gpinitactivestandby --help" and
"gpinitstandby --help" produce a particular output. That's not very
interesting, --help seems highly unlikely to suddenly stop working. Also,
memorizing the exact help output in a test adds a hurdle to keeping the
--help text up-to-date - you will need to remember to update the
test too, whenever you update the help text.

What would be useful, is a test to check that --help covers all the options
and provides useful advice to a DBA. But that cannot be automated.

I removed the last tests from that directory in commit 4d23ab19.

This commit cleans up some tinc.py calls, updates project names, and
splits the cs-walrep test into two targets for CI test
parallelization.