This was mostly already in bfv_partition. For completeness, move the code
from the TINC test to bfv_partition to also execute the query, not just
EXPLAIN it.

There's a test in the main suite's 'partition' test for this already,
with the 'deep_part' test table.

There is already a similar test in the 'partition' test in the main test
suite. Search for "-- Sub-partition insertion" in partition.sql to find
it.

It's not allowed. We already have tests for both CREATE TABLE and ALTER
TABLE ADD PARTITION, with "WITH OIDS=false", in the 'partition' test of
the main test suite.

Rename the test tables in 'partition' test, so that they don't clash with
the test tables in 'partition1' test. Change a few validation queries to
not get confused if there are unrelated tables with partitions in the
database. With these changes, we can run 'partition' and 'partition1' in
the same parallel group, which is a more logical grouping.

All of these tests used the same test table, but it was dropped and
re-created for each test. To speed things up, create it once, and wrap each
test in a begin-rollback block.

The access plan of one of the tests varied depending on optimizer_segments,
and it caused a difference in the ERROR message. The TINC tests were always
run with 2 segments, but you got a different plan and message with 3 or
more segments. Added a "SET optimizer_segments=2" to stabilize that, and a
comment explaining the situation.

The transformation of "NOT IN" or "= ALL" incorrectly added a condition
that the outer side of the join must not be NULL. But that's not true,
when the other side is an empty set.

Discovered while moving old TINC tests to the main test suite. The
'lasj_hash_all_3' test exposed this issue, when run with ORCA disabled.
This has apparently gone unnoticed, because no-one has run those tests
without ORCA. There were also incorrect query results memorized, for tests
'lasj_notin_multiple_3' and 'lasj_hash_notin_multiple_3'. They are similar
queries, but because ORCA falls back to the planner for them, we had
memorized the Planner results for them, and hadn't noticed that the
results were in fact incorrect.

Fixes github issue #1907.

We have a bunch of tests on subpartitioned tables in the main suite, e.g.
in the 'partition' and 'bfv_partition' tests. And these tests were marked
with "@skip", anyway, so they were not run by TINC either.

I moved them to the 'partition_pruning_with_fn' test, but because there is
no functions involved in these tests, I renamed the test to just
'partition_pruning'.

There is possibly some overlap with these new tests and the existing ones
in the same file, but will let future test archeologists to decide that.
These are cheap tests, in any case.

________________________________________________________________________________________________________
*** CID 163038:  Uninitialized members  (UNINIT_CTOR)
/extensions/gpcloud/src/s3interface.cpp: 556 in S3MessageParser::S3MessageParser(const Response &)()
550     S3MessageParser::S3MessageParser(const Response &resp) {
551         // Compatible S3 services don't always return XML
552         if (resp.getRawData().data() == NULL) {
553             message = "Unknown error";
554             code = "Unknown error code";
555
>>>     CID 163038:  Uninitialized members  (UNINIT_CTOR)
>>>     Non-static class member "xmlptr" is not initialized in this constructor nor in any functions that it calls.
556             return;
557         }

________________________________________________________________________________________________________
*** CID 163037:  Uninitialized members  (UNINIT_CTOR)
/extensions/gpcloud/include/restful_service.h: 37 in Response::Response(ResponseStatus, PGAllocator<unsigned char> &)()
31     class Response {
32        public:
33         explicit Response(ResponseStatus status) : status(status) {
34         }
35         explicit Response(ResponseStatus status, S3MemoryContext& context)
36             : status(status), dataBuffer(context) {
>>>     CID 163037:  Uninitialized members  (UNINIT_CTOR)
>>>     Non-static class member "responseCode" is not initialized in this constructor nor in any functions that it calls.
37         }

________________________________________________________________________________________________________
*** CID 163033:  Uninitialized members  (UNINIT_CTOR)
/extensions/gpcloud/include/s3params.h: 13 in S3Params::S3Params(const std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>> &, bool, const std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>> &, const
+std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>> &)()
7
8     class S3Params {
9        public:
10         S3Params(const string& sourceUrl = "", bool useHttps = true, const string& version = "",
11                  const string& region = "")
12             : s3Url(sourceUrl, useHttps, version, region), keySize(0), chunkSize(0), numOfChunks(0) {
>>>     CID 163033:  Uninitialized members  (UNINIT_CTOR)
>>>     Non-static class member "verifyCert" is not initialized in this constructor nor in any functions that it calls.
13         }

________________________________________________________________________________________________________
*** CID 163032:  Error handling issues  (UNCAUGHT_EXCEPT)
/extensions/gpcloud/src/compress_writer.cpp: 9 in CompressWriter::~CompressWriter()()
3     uint64_t S3_ZIP_COMPRESS_CHUNKSIZE = S3_ZIP_DEFAULT_CHUNKSIZE;
4
5     CompressWriter::CompressWriter() : writer(NULL), isClosed(true) {
6         this->out = new char[S3_ZIP_COMPRESS_CHUNKSIZE];
7     }
8
>>>     CID 163032:  Error handling issues  (UNCAUGHT_EXCEPT)
>>>     An exception of type "S3RuntimeError" is thrown but the throw list "throw()" doesn't allow it to be thrown. This will cause a call to unexpected() which usually calls terminate().
9     CompressWriter::~CompressWriter() {
10         this->close();
11         delete this->out;
12     }

________________________________________________________________________________________________________
*** CID 163029:  Memory - illegal accesses  (STRING_NULL)
/extensions/gpcloud/src/gpwriter.cpp: 49 in _ZN8GPWriter18constructRandomStrB5cxx11Ev()
43     string GPWriter::constructRandomStr() {
44         int randomDevice = ::open("/dev/urandom", O_RDONLY);
45         char randomData[32];
46         size_t randomDataLen = 0;
47
48         while (randomDataLen < sizeof(randomData)) {
>>>     CID 163029:  Memory - illegal accesses  (STRING_NULL)
>>>     Function "read" does not terminate string "randomData[randomDataLen]".

________________________________________________________________________________________________________
*** CID 163028:  Performance inefficiencies  (PASS_BY_VALUE)
/extensions/gpcloud/src/s3interface.cpp: 265 in S3InterfaceService::listBucket(S3Url)()
259     // ListBucket lists all keys in given bucket with given prefix.
260     //
261     // Return NULL when there is failure due to network instability or
262     // service unstable, so that caller could retry.
263     //
264     // Caller should delete returned object.
>>>     CID 163028:  Performance inefficiencies  (PASS_BY_VALUE)
>>>     Passing parameter s3Url of type "S3Url" (size 256 bytes) by value.
Signed-off-by: NHaozhou Wang <hawang@pivotal.io>
Signed-off-by: NYuan Zhao <yuzhao@pivotal.io>
Signed-off-by: NAdam Lee <ali@pivotal.io>

Without this commit, test scripts will re-compile gpcloud anyway, better
to use the one from package instead in the main pipeline.

[ci skip]

The storage_persistent_filerep_accessmethods_and_vacuum task contained
two filerep tests:

FilerepResync.test_filerep_resysnc ... 585576.51 ms ... ok
FilerepProcArrayRemoveTestCase.test_verify_ct_after_procarray_removal ... 136553.63 ms ... ok

Move them together with the filerep_end_to_end_xlog_ctlog_cons task, so that
all the filerep tests are in the same task. To reflect that, rename the
'filerep_end_to_end_xlog_ctlog_cons task' to just 'filerep'.

The 'storage_persistent_filerep_accessmethods_and_vacuum' task was the
slowest task among the parallel tasks in the job, so moving the two filerep
tests elsewhere should reduce overall runtime of the storage job.

If the segrelid, visimaprelid, or the visimapidxid fields are 0 for an AO
table, then error out.

These are already covered by the 'appendonly' and 'uao_dml/uao_dml' tests
in the main test suite.

"int" and "int4" are the same datatype. They are mapped to the same early
on, in the parsing stage. For those tests that have been duplicated for
many different datatypes, remove the _int_ variant, leaving just the
_int4_ variant.

We have sufficient coverage for these in the 'geometry' test in the main
test suite. There are no UPDATEs in 'geometry', but that seems
uninteresting; there is no reason to believe that UPDATE on these types
would behave differently from an UPDATE on any other type.

The test descriptions were misleading, BTW. These are not "boundary"
tests, the values used are not close to the min or max values the data
types can represent.

PR pipeline: use 'max_in_flight: 1' and 'version: every' to ensure all PRs flow through the pipeline
Signed-off-by: NJingyi Mei <jmei@pivotal.io>

Signed-off-by: NTom Meyer <tmeyer@pivotal.io>

Signed-off-by: NTom Meyer <tmeyer@pivotal.io>

We have a bunch of existing tests that test whether ORCA falls back. They
set optimizer_log_failure='all' and client_min_messages='log', and grep
the output for "Planner" or "Planner produced plan". That's error-prone.

This new GUC makes that kind of tests easier and more robust. You can
simply set the GUC, and if there are no extra INFO messages in the output,
ORCA didn't fall back.

We had already disabled ORCA for some queries, but the query to estimate
reltuples and relpages still used ORCA. This makes ANALYZE slightly faster
(because ORCA is slow to start up), and avoids spurious messages about
GPORCA falling back when the new optimizer_trace_failure option is used.

Currently, we have a job compile_gpdb_custom_config_centos6 in
pr_pipeline and master that compiles gpdb with a custom config, but the
purpose of this job is unclear. It turns out that this job is for
compiling gpdb the open-source way, so we renamed the job as
compile_gpdb_open_source and changed the name of task yml and scripts.
Signed-off-by: NDavid Sharp <dsharp@pivotal.io>

* Move all the udf_insert_* tests to the main suite.

* Move 'volatilefn_dml_int8' test to the main suite.

* Remove all the other volatilefn_dml_* tests. There's no reason to believe
  that the data types used in the table would make a difference, so keeping
  one variant of this is enough.

* Run behave tests in concourse

* Initial steps to enable us to run behave tests in concourse.
  This will only work for single node behave tests.
Signed-off-by: NChumki Roy <croy@pivotal.io>

Below columns are added:

* rsgid: resgroup id;
* rsgname: resgroup name;
* rsgqueueduration: queued duration in the resgroup;

As resgroup is not fully implemented yet there is only dummy output for
these columns at the moment.

Add view gp_toolkit.gp_resgroup_status for resource group runtime
statistics information, such as cpu usage, memory usage, etc..

Example:

    -- query resource group runtime statistics information
    SELECT * FROM gp_toolkit.gp_resgroup_status;

A helper function pg_resgroup_get_status_kv() is also provided to
collect these information.

A new dir src/backend/utils/resgroup/ is created to hold the resgroup
source code.

As resgroup is not fully implemented yet there is only dummy output for
this view at the moment.

Add view gp_toolkit.gp_resgroup_config for resource group information,
such as concurrency, cpu rate limitation, etc..

Example:

    -- query resource group config information
    SELECT * FROM gp_toolkit.gp_resgroup_config;

docker image jtarchie/pr updated recently, which breaks our git
submodule pulling, fall back to the working tag.
Signed-off-by: NAdam Lee <ali@pivotal.io>
Signed-off-by: NYuan Zhao <yuzhao@pivotal.io>

I'm not how many of these tests are worth keeping at all, but as a first
step, let's get them into the main test suite. The runtime of the parallel
group with these tests is about 1 minute on my laptop.

This is mostly a straightforward move, with mechanical changes removing the
useless TINC tags. However, some tests were marked with the special
"@executemode ORCA_PLANNER_DIFF" tag. The code in __init__.py ran those
tests differently. Normally, you run a test, and comparet the output with
the expected output file. But those ORCA_PLANNER_DIFF tests were run twice,
with and without ORCA, and the results were compared against each other,
instead of the expected output stored in the repository. The expected
outputs of those tests were out of date, you got a different error message
now than what was remembered in the expected output.

Now those tests are run as normal pg_regress tests. I updated the expected
output of those tests to match what you get nowadays.

pg_size_pretty() is completely ordinary and boring function. It doesn't
do database access or anything like that. We can live without testing it
in every possible context.

* dml_boundary_bool seems redundant. We surely have a lot of other tests
  involving boolean columns.
* dml_boundary_char is redundant with the 'char' test in main suite.
* dml_boundary_money was disabled.
* dml_boundary_smallint was redundant with 'int2' test in main suite.
* dml_boundary_integer was redundant with 'int4' test in main suite.
* dml_boundary_bigint was redundant with 'int8' test in main suite.

There are a lot of similar and more interesting tests for direct dispatch
in the main test suite ('direct_dispatch' test, for example).

The negative test cases, like inserting into a randomly distributed table
so that direct dispatch is not possible, were not very interesting, so I
removed them. A mistake in the other direction, i.e. not performing direct
dispatch when we could have, does not lead to incorrect query results, just
slower performance, which is why we need bespoken tests for those. But
if we erroneously do direct dispatch for a query, where it is not legal,
that will lead to incorrect query results, so we should catch those cases
by all the other tests in the test suite.

There was one test case, involving DEFAULTs in the INSERT statement, that
I thought is marginally worth keeping, so added a case for that in
'direct_dispatch' before removing it.