The test disconnects, and immediately reconnects and checks that temp
table from the previous session has been removed. However, the old backend
removes the tables only after the disconnection, so there's a race
condition here: the new session might query pg_tables and still see the
table, if the old session was not fast enough to remove it first.

Add a two-second sleep, to make the race condition less likely to happen.
This is not guaranteed to avoid the race altogether, but I don't see an
easy way to wait for the specific event that the old backend has exited.

A sleep in a test that doesn't run in parallel with other tests is a bit
annoying, as it directly makes the whole regression suite slower. To
compensate for that, move the test to run in parallel with the other
nearby bfv_* tests. Those other tests don't use temp tables, so it still
works.

Also, modify the validation query in the test that checks that there are
no temp tables, to return whole rows, not just COUNT(*). That'll make
analyzing easier, if this test fails.

The uao_dml and uaocs_dml tests were identical, except that there were
a few additional queries in the uao_dml suite that were missing from the
uaocs_dml suite, and various cosmetic differences in table names and such.
Use the new "uao templating" mechanism in pg_regress to generate both
cases from a single set of files.

I didn't look for redundant or useless tests in these suites. At a quick
glance, at least many of the select tests seem quite pointless, so it would
be good to cull these in the future.

This is a straightforward copy-paste, except for thing: I changed all the
quicklz-compressed tables to use zlib instead, so that these tests work
without support for legacy quicklz compression. AFAICT, the compression
isn't a very interesting aspect for these tests (I even considered making
them uncompressed.)

These tests test for an old bug, where vacuuming an empty AO relation would
not advance its relfrozenxid. Doesn't seem very likely for that particular
bug to reappear, but seems like a good idea to have some coverage for
freezing and relfrozenxid advancement. In this rewritten form, these tests
are fairly quick, too (1-2 seconds on my laptop).

We have some VACUUM FULL commands in other tests already, but these tests
seem more comprehensive than the other tests.

Change the queries that check tuple counts on a particular segment,
in utility mode, to not print out the exact tuple counts, but a crude
classification of 0, 1, <5 or more tuples. That's less sensitive to how
the tuples are distributed across segments.

The locks_reindex test is moved to the regular regression suite. I rewrote
it to use a more advanced "locktest" view, copied from the
partition_locking test, which doesn't rely on utility mode.

These changes should make the tests work with even larger clusters,
but I've only tested with 1, 2, and 3 nodes.

These tests need the usual "lineitem", "orders" and "suppliers" test
tables, but use only a subset of the data that we have in the the
data/*.csv files. Therefore, split the csv files into two parts, the
"small" set, used by these new tests, and the rest.

The main ExplainAnalyzeTestCase tests in here were running standard
TPC-H-like test queries, and checking that the EXPLAIN ANALYZE output
contains a "Memory: " line for every plan node. That seems a bit excessive,
because looking at how those lines are printed, there is no reason to
believe that you might have a Memory line for some nodes but not others.
I don't think we need to test for that.

In order to still have minimal coverage for the explain_memory_verbosity
GUC, add a small test case to the main regression suite for that.

The mpp20785 test was testing for an old bug where you got an out of
memory error with this query. However, the test was broken, because the
test schema was nowhere to be found, so it simply resulted in a "schema not
found" error. Perhaps we could put it back if we could find the original
schema somewhere, but it's useless as it is.

This commit also resolves #1646.
Porting only the function `tuplesort_get_stats` in tuplesort.c from the
Postgres commit
https://github.com/postgres/postgres/blob/9bd27b7c9e998087f390774bd0f43916813a2847/src/backend/utils/sort/tuplesort.c
Link to gpdb-dev discussion - https://groups.google.com/a/greenplum.org/forum/#!topic/gpdb-dev/V-zIshnNzyE

I'm not sure what exactly the purpose of each of these tests are, and they
probably need more culling and cleanup, but for now, just move them out of
bugbuster, so that we can get rid of bugbuster as a separate suite.

I don't see any reason to expect that this would behave differently with
an external table than a normal one.

Move the test from bugbuster to the normal regression suite.

This moves 'spi', 'spilltodisk', and 'wrkloadadmin' tests. The goal is to
get rid of bugbuster as a separate suite altogether, and this commit is one
step towards that goal.

- The dpe.sql file was duplicated in bugbuster - so we remove that one.
- Result files were old, and hence updated.
Signed-off-by: NShreedhar Hardikar <shardikar@pivotal.io>

No reason was specified for running those tests serially, tried and seem to run
fine in parallel. Also, dropped few drop tables from gpdtm_plpgsql test.

There's no need to allocate ExecWorkFile struct in TopMemoryContext. We
don't rely on a callback to clean them up anymore, they will be closed
by the normal ResourceOwner and MemoryContext mechanisms. Otherwise,
the memory allocated for the struct is leaked on abort.

In the passing, make the error handling more robust.

Also modify tuplestorenew.c, to allocate everything in a MemoryContext,
rather than using gp_malloc() directly. ResourceOwner and MemoryContext
mechanisms will now take care of cleanup on abort, instead of the custom
end-of-xact callback.

Remove execWorkFile mock test. The only thing it tested was that
ExecWorkFile was allocated in TopMemoryContext. That didn't seem very
interesting to begin with, and the assumption is now false, anyway. We
could've changed the test to test that it's allocated in
CurrentMemoryContext instead, but I don't see the point.

Per report in private from Karthikeyan Rajaraman. Test cases also by him,
with some cleanup by me.

This patch ports the tests for exception blocks in udfs from tinc.

There is considerable overlap between the aoco_alter/aoco_alter_sql_test tests
and the test/regress/*/uao_dll tests. This is the first commit to move several
of the aoco_alter/aoco_alter_sql_test tests to ICG.

This commit moves alter_ao_table_oid_column.sql and alter_ao_table_oid_row.sql
from sql/uao_ddl directories to sql/uao_col and sql/uao_row directories
respectively.

After this change, the sql/uao_ddl and expected/uao_ddl directories are only
used for generation of intermediate .sql and .out files during the processing of
the --ao-dir option of regress.  Therefore, I added .gitignore files in these
directories.

The bulk_dense_content_rle_compress.sql and small_content_rle_compress_hdr.sql
tests are moved to rle.sql.

Teh table name used in the alter_drop_allcol.source is renamed to avoid conflict
with the same table name used in other tests.

Remove migrated aoco_alter tests from tinc.

Since these tests are part of ICG, so we don't need them in tinc anymore.

For some reason the function_extensions test was removed from the
regress schedule in the past but it seems quite handy since it does
excercise extensions to upstream. Add test for the recently caught
PlannedStmt issue, fix the existing tests and add to greenplum
schedule.

- `optimizer_analyze_root_partition` GUC functionality is introduced.
When optimizer_analyze_root_partition is enabled, it will enable stats
collection on root partitions when a plain analyze is run on a root
partition table
- `optimizer_analyze_midlevel_partition` GUC functionality is introduced
When optimizer_analyze_midlevel_partition is enabled, it will enable
stats collection on midlevel partitions.

Expectation:
Case 1:
```sql
set optimizer_analyze_root_partition=off;
analyze tablename;
-- Stats should only be collected for the leaf tables
analyze rootpartition tablename;
-- Stats should be only collected for the root table
```

Case 2:
```sql
set optimizer_analyze_root_partition=on;
analyze tablename;
-- Stats should be collected for the root and the leaf table;
analyze rootpartition tablename;
-- Stats should be only collected for the root table
```
Signed-off-by: NOmer Arap <oarap@pivotal.io>

The code in rewriteDefine.c didn't know about AO tables, and tried to use
heap_beginscan() on an AO table. It resulted in an ERROR, because even for
an empty table, RelationGetNumberOfBlocks() returned 1, but we tried to
scan it like a heap table, heap_getnext() couldn't find the block. To fix,
add an explicit check that you can only turn heap tables into views this
way. This also forbids turning an external table into a view. (We could
possible relax that limitation, but it's a pointless feature anyway we only
support for backwards-compatibility reasons in PostgreSQL.)

Also add test case for this.

Constraints are not uniquely identified by namespace and name alone.
It's possible to have multiple constraints with the same name, but attached
to different tables or domains. So include parent relation and domain OIDs
in the key.

This was uncovered by a regression test in the TINC test suite. Move the
OID consistency tests from TINC into the normal regression test suite, to
have test coverage for this.

QE reader leverages SharedLocalSnapshot to perform visibility checks. QE writer
is responsible to keep the SharedLocalSnapshot up to date. Before this fix,
SharedLocalSnapshot was only updated by writer while acquiring the snapshot. But
if transaction id is assigned to subtransaction after it has taken the snapshot,
it was not reflected. Due to this when QE reader called
TransactionIdIsCurrentTransactionId, it may get sometimes false based on timings
for subtransaction ids used by QE writer to insert/update tuples. Hence to fix
the situation, SharedLocalSnapshot is now updated when assigning transaction id
and deregistered if subtransaction aborts.

Also, adding faultinjector to suspend cursor QE reader instead of guc/sleep used
in past. Moving cursor tests from bugbuster to ICG and adding deterministic test
to exercise the behavior.

Fixes #1276, reported by @pengzhout

* Merge 4 alter_ao_part_exch* tests into one to avoid redundant
   partitioned table creation.
 * Avoid creating bitmap index when not required.
 * Merge analyze* tests into analyze_ao_table_every_dml.sql to reduce
   number of tables that need to be created.
 * Merge cluster.sql into alter_ao_table_index.sql.
 * Merge auxiliary_tables.sql into create_ao_tables.sql.
 * Add update and delete statements where they were missing.
 * Remove DROP TABLE statements.
 * Remove unnecessary "select * from ..." to reduce diff time.
 * Merge truncate_ao_table.sql into create_ao_tables.sql.
 * Merge truncate_ao_table_part.sql into alter_ao_part_tables.sql.
 * Remove redundant COPY tests.  COPY command is already tested by
   analyze_ao_table_every_dml.sql.
 * Remove "ALTER TABLE ... CLUSTER" tests, these commands should
   report error if run against append-optimized tables.
 * Remove fillfactor tests, fillfactor is not applicable to
   append-optimized tables.
 * Merge blocksize tests into one to minimize creating tables.
 * Merge *ao_table_compresstype.sql into one compresstype.sql.

This reverts commit 506032ac.  VACUUM FULL on
AO/CO is fixed to not give up during drop phase and the tests need not worry
about segfiles in AWAITING_DROP state.

Note that the tests should NOT be run concurrently with any other tests that
use serializable transactions.  VACUUM FULL on AO/CO continues to avoid
deleting segfiles in AWAITING_DROP state in presence of concurrent serializable
transactions that may need the segfiles.

Some of the TINC tests under "limit_removal" heading were testing
subtransaction functionality but not limit remotal in particular.  Move them to
distributed_transactions.sql.  The limit in "limit removal" refers to the
in-progress subtransactions array SharedSnapshotSlot->subxids.  If the limit of
MaxGpSavePoints subtransactions is reached, the array is spilled over to disk.
More elaborate tests for this funcionality exist in TINC
(mpp/gpdb/tests/storage/sub_transaction_limit_removal).

Signed-off-by: NPengzhou Tang <ptang@pivotal.io>

Signed-off-by: NNikos Armenatzoglou <nikos.armenatzoglou@gmail.com>

The uao/uao_catalog_tabs tests were originally created in TINC because
the uao/uaocs auxiliary tables needed the uao/uaocs table's oid. Some
gp_toolkit functions also needed the oid and were thought to require
to be called on segments using utility mode too. TINC made this easy
since it's doing simple Python scripting. These requirements have been
dismissed by using PL/pgSQL functions in order to get these tests
running in installcheck.

Author: Jimmy Yih

It was disabled back in 2011, with a commit that merely said:

> Temporarily disable leastsquares -- it's causing grief on 64 bit platforms.

I have no idea what the grief was, but presumably it's been fixed since.
And if it hasn't, we want to find out.

I made a half-baked attempt at silencing differences that were due to
source filenames and line number, in commit 0bf31cd6, but didn't notice
then that the whole test case was in fact ignored. Fix that correctly.

pg_get_constraintdef() looks up information on the constraint
from the syscache, which uses SnapshotNow to fetch the information. But the
query itself uses an MVCC snapshot. If the query's MVCC snapshot sees a
constraint that is concurrently dropped so that it's not visible to SnapshotNow,
pg_get_constraintdef() will report error.

Thanks Heikki for above description.

No need to create a whole new library for this. It's included in every
installation anyway, as part of gp_toolkit. Might as well just include
it in the backend, and make the build slightly simpler.

Some of the functions in funcs.cpp were only used by debugging/testing
code. They were included in the libgpoptudfs.so library, but it was built
in a strange way, from gpAux/Makefile. Move it to a contrib module, along
with the regression tests using it.

This commit leaves only the functions actually used by backend code in
funcs.cpp, and moves the debugging functions only used by the optional
UDFs into the contrib module.

NOTE: The regression tests don't actually work at the moment. But that's not
new, they didn't work before either. I don't have the patience to start
fixing them, so I'm just moving them to the new location in their current
form.

NOTE2: The new module is called orca_debug, but I think some test code
in GPORCA repository expects to find this in libgoptudfs.so. Will need
to fix the scripts there.

The 'ao_create_alter_valid_table' test in bugbuster was completely defunct,
as the expected output just contained an error about connecting to a
non-existent database. However, the exact same test queries were included
in the schema_topology test. Schema_toplogy is huge, and some of the tests
in it require a separate output file for ORCA, which these test queries do
not, so splitting it up into smaller parts seems like a good idea.

Environment variables of QD should not be included in the startup packet when making connections to segments.
Add ICG test case to guarantee the QD env variable should not effect GUCs on segments
Signed-off-by: NPengzhou Tang <ptang@pivotal.io>

GPDB currently has 2 APIs of creating work files
  1. Using workfile_mgr :

    1. This uses a workfile set to contain all the related workfiles.
    2. All workfiles are created under the same directory. As a result,
       when it is time to delete the workfiles, we can simply delete the directory.
    3. There is no mechanism to control the exact name of the workfile created.
       All workfiles are referenced using an ID. `"%s/spillfile\_f%u", work\_set-\>path, file\_no);`
    4. Deletion of workiles is automatically handled in case of an exception,
       as all the files are contained in one directory, thus no counter/files are leaked.

  2. Using ExecWorkfile :

    1. There is a Create/Open/Read/Write/Flush/Close API.
    2. This is the lower-level API also used by workfile_mgr to manage the workfiles directly.
    3. Each operator is resposible to correctly close the workfile it creates.

  When we write to a workfile, we reserve and add the number of bytes to used_segspace,
  and add that amount to `workfile->size`. Then, we call `workfile_update_in_progress_size`.
  If the workfile has an owning workset, we update the workset's `in_progress_size` counter.
  However, if it has no such workset, we completely ignore the count -
  and this is what ultimately leaks the `used_segspace` counter.

  This is a fix to account for bytes written in workfiles that are not part of any workfile set -
  that is those created using the lower level API. Hence when we abort or error out,
  we subtract this amount from the `used_segspace` counter.

Authors:
Shreedhar Hardikar <shardikar@pivotal.io>
Xin Zhang <xzhang@pivotal.io>
Karthikeyan Jambu Rajaraman <karthi.jrk@gmail.com>
Nikos Armenatzoglou <nikos.armenatzoglou@gmail.com>

[#127140493] This checkin will make ICG green in gpdb_pr pipeline.

- Separate the alter_table_ao as its own test group

- Separate bitmap_index as its own test group

- Make the utable and uview independent across tests

Both gp_numeric_agg and qp_misc_jiras are using uviews, which caused side-effect
across tests. This fix totally separate those two tests.

- Separate `partition_indexing` as its own group

Sometime, this test failed with under low resource:
ERROR:  Inserting with invalid TID (0,0) into relation id 220616 'mpp3033b_stringu1', relfilenode 221541, segment file #0, serial number 0