All of these tests were already in the 'partition' test of the main suite,
in more or less identical form.

I did add one little query to 'partition', to test that START of a partition
range cannot be NULL. The existing test only covered "END (NULL)".

Also, stop testing for long-gone scripts gpaddpart, gpcreatepart and
gpdeletepart utilities.

Signed-off-by: NYuan Zhao <yuzhao@pivotal.io>

The "Execute on" syntax was added when performing a "\d" on external tables in gpdb5, but is not present in previous gpdb versions. When transferring from clusters on previous versions, we will ignore this syntax.

There are plenty of VACUUMs on append-only tables elsewhere in the
regular regression suite, e.g. in 'vacuum_gp', 'appendonly', 'aocs',
uao_compaction tests, and many more.

Subtransactions with append-only tables are covered at least by the
uao_dml/uao_dml_[row|column] tests in the main regression suite.

These ALTER TABLE commands are covered by the 'alter_table_ao' and
'alter_table_aocs' tests in the main regression suite.

Currently we see FTS failure in the CI pipeline due to the filerep resync
manager and workers are still in the recovery mode while relcache got
initialized.

This reverts commit f8f6587e and
0e9d10d2.

We have sufficient coverage for these debugging functions in
src/test/regress/sql/sql/gp_toolkit_ao_funcs.sql already.

Bad things happen otherwise.  One case in point is create database
followed by a crash.  Create database requests a checkpoint after
inserting new tuple into pg_database.  Crash happens right after
create database commits and before clog update is not flushed to disk.
Relcache initialization before xlog replay will set HEAP_XMIN_INVALID
hint bit in the newly created database's tuple because clog did not
report the xmin as committed.

The need for heap access methods before xlog replay is removed by commit
e2d6aa1481f6cdbd846d4b17b68eb4387dae9211.  This commit simply moves the
relcache initialization to pass4, where it is really needed.

Do not bother to remove relcache init file at the end of crash recovery pass2.

Error out if relation cache initialized at wrong time.

There are only a few tests left in "uao", most have now been moved to
installcheck-world. The "uao" job now runs in about 4 minutes, of which
2 minutes is spent on setting up the test cluster. That seems like
overkill, so simplify things by merging the remaining uao fault injection
tests into the larger "storage" job.

These are the same tests queries for column-oriented append-only tables,
as those moved by commit 11a5a807, for row-oriented append-only tables.
There were two additional tests that were never executed for row-oriented
tables though: phantom_reads_update_serializable and
phantom_reads_delete_serializable. I believe that was an oversight in the
original test suite; they are now also executed for row-oriented tables.

We use the UAO templating mechanism, to run the same test files against
row- and column-oriented tables. To make that work, fix a bug in the
templating mechanism pg_regress.c: if the --ao-dir argument was shorter
than 7 characters, the uao directory was not detected correctly.

We have sufficient test coverage for gpload in the Behave tests in
gpMgmt/test/behave/mgmt_utils. And we have plenty of tests for external
tables elsewhere.

The combination of using an external table or gpload as the source, and
append-only table as the target, is not particularly interesting. So remove
the tests.

Update pg_dump, cdb_dump_agent, psql to fix MU tests after ON MASTER patch

cURL automatically adds the CONTENT-LENGTH header when performs "PUT"
request, we also did it without this commit, AWS is OK with that, but
others would report duplicate CONTENT-LENGTH error.
Signed-off-by: NYuan Zhao <yuzhao@pivotal.io>

palloc and repalloc will check request size is under MaxAllocSize,
but the implementation(AllocSetAllocImpl, AllocSetReallocImpl) allocs
ALLOC_BLOCKHDRSZ + ALLOC_CHUNKHDRSZ more bytes to store block
information. Later MemoryContextNoteAlloc is called with block size.
If palloc request size is between MaxAllocSize - (ALLOC_BLOCKHDRSZ +
ALLOC_CHUNKHDRSZ) and MaxAllocSize, an internal error occurs.

Fix: simply remove those assertions, the calculations in
MemoryContextNoteAlloc and MemoryContextNoteFree should work with larger
values.

This resolves https://github.com/greenplum-db/gpdb/issues/1090.
Signed-off-by: NPeifeng Qiu <pqiu@pivotal.io>
Signed-off-by: NYuan Zhao <yuzhao@pivotal.io>

AWS S3 returns internal errors (HTTP response code 500) serveral times,
unusual but better not to ignore, this commit re-sends requests when it
happens.
Signed-off-by: NHaozhou Wang <hawang@pivotal.io>

* We were accessing a non-existent variable as it's being generated by
  the later function. Swapping the two methods.

* Fix on top of a86e4901

Signed-off-by: NMarbin Tan <mtan@pivotal.io>

Signed-off-by Gang Xiong <gxiong@pivotal.io>

Signed-off-by Gang Xiong <gxiong@pivotal.io>

The gppersistentrebuild tool was trying to use string_agg without
casting the oid to text. This used to work before but not anymore. The
behave tests only have two scenarios but the second scenario would
always fail because it would run when a primary/mirror segment pair
were still in resync mode.

* When mocking, we may setup patches that are not cleaned up properly
  and causes test pollutions. Ensure that we do tearDown properly and
  raise a condition if we are not.
Signed-off-by: NMarbin Tan <mtan@pivotal.io>

When running gpconfig -s with --file option, we can now view the
postgresql.conf guc settings instead of just viewing the user guc
settings in the database. This option also reports if there are any
inconsistencies with the postgresql.conf between the segments.

reformat gpconfig:
* enable unit testing

remove gpconfig white-space

Add gpconfig behave tests:
* When you run gpconfig with -s and --file, you can sometimes run into
a nonetype string format error. This is due to us not joining all the
threads properly and closing them out properly.

* Added gpconfig behave tests to ensure that we are not missing anything
 in the unittest tests. As we are mocking the pool, we need to make sure
 that we are not doing anything silly, so adding minimal behave tests.
 Single node is probably enough.
Signed-off-by: NChumki Roy <croy@pivotal.io>
Signed-off-by: NMarbin Tan <mtan@pivotal.io>

Signed-off-by: NMarbin Tan <mtan@pivotal.io>

Signed-off-by: NMarbin Tan <mtan@pivotal.io>

* Fixing a hardcoded python library path that may not be valid for other
  python version.
Signed-off-by: NMarbin Tan <mtan@pivotal.io>

* Using uninitialized value kstr.

Field kstr.isPrefixOnly is uninitialized when calling memcpy on line
3536. Setting kstr is OK since the memcpy will copy the value to ret.

* Buffer not null terminated

Calling strncpy with a maximum size argument of 1024 bytes on
destination array work_set_path of size 1024 bytes might leave the
destination string unterminated.

Upstream precedent : 586dd5d6

Commit 821b8e10 changed pg_dump's shouldPrintColumn() function to return
true for all columns in an external table, and changed dumpExternal() to use
the shouldPrintColumn() function. However, that change was not made to the
copy of shouldPrintColumn() in cdb_dump_include.c.

That oversight didn't cause any ill effects, until commit 58b86683 came
along and refreshed cdb_dump_agent.c's copy of the dumpExternal() function,
to match that in pg_dump.c. With that change cdb_dump_agent's dumpExternal()
started to call shouldPrintColumn() for external tables, but its version of
shouldPrintColumn() didn't have the change to always return true for
external tables yet. As a result, external tables didn't have any of their
columns included.

To fix, update cdb_dump's copy of shouldPrintColumn() to match that in
pg_dump.c.

Many callers of get_funcs_for_compression(), like the one from
appendonly_insert_init() pass a reference to the relcache as argument,
like this:

  get_funcs_for_compression(NameStr(rel->rd_appendonly->compresstype));

If a relcache invalidation happens while looking up compression type,
that reference to rel->rd_appendonly is free'd and becomes invalid.
get_funcs_for_compression() performs a catalog lookup, and can therefore
invalidate the reference itself, before using it.

Make get_funcs_for_compression (GetCompressionImplementation, actually)
to tolerate that, by making a copy of the argument before calling
heap_open, which can cause the cache invalidation.

RelationData->rd_index potentially has the same problem, but I don't see
any problematic call sites that would pass a rel->rd_index reference
to a function like this. In rd_appendonly, all the other fields but
compresstype are pass-by-value types (Oid, bool, etc.), that don't have
the same hazard.

The crash can be reproduced easily by compiling with --enable-testutils,
setting gp_test_system_cache_flush_force=plain, in a psql session, and
performing any select or update on a compressed AO table. I am not adding
a regression test, as it would only work with --enable-testutils, but it
would be a good idea to run all the regression tests with that GUC enabled,
every once in a while.