The transformation of "NOT IN" or "= ALL" incorrectly added a condition
that the outer side of the join must not be NULL. But that's not true,
when the other side is an empty set.

Discovered while moving old TINC tests to the main test suite. The
'lasj_hash_all_3' test exposed this issue, when run with ORCA disabled.
This has apparently gone unnoticed, because no-one has run those tests
without ORCA. There were also incorrect query results memorized, for tests
'lasj_notin_multiple_3' and 'lasj_hash_notin_multiple_3'. They are similar
queries, but because ORCA falls back to the planner for them, we had
memorized the Planner results for them, and hadn't noticed that the
results were in fact incorrect.

Fixes github issue #1907.

We have a bunch of tests on subpartitioned tables in the main suite, e.g.
in the 'partition' and 'bfv_partition' tests. And these tests were marked
with "@skip", anyway, so they were not run by TINC either.

I moved them to the 'partition_pruning_with_fn' test, but because there is
no functions involved in these tests, I renamed the test to just
'partition_pruning'.

There is possibly some overlap with these new tests and the existing ones
in the same file, but will let future test archeologists to decide that.
These are cheap tests, in any case.

________________________________________________________________________________________________________
*** CID 163038:  Uninitialized members  (UNINIT_CTOR)
/extensions/gpcloud/src/s3interface.cpp: 556 in S3MessageParser::S3MessageParser(const Response &)()
550     S3MessageParser::S3MessageParser(const Response &resp) {
551         // Compatible S3 services don't always return XML
552         if (resp.getRawData().data() == NULL) {
553             message = "Unknown error";
554             code = "Unknown error code";
555
>>>     CID 163038:  Uninitialized members  (UNINIT_CTOR)
>>>     Non-static class member "xmlptr" is not initialized in this constructor nor in any functions that it calls.
556             return;
557         }

________________________________________________________________________________________________________
*** CID 163037:  Uninitialized members  (UNINIT_CTOR)
/extensions/gpcloud/include/restful_service.h: 37 in Response::Response(ResponseStatus, PGAllocator<unsigned char> &)()
31     class Response {
32        public:
33         explicit Response(ResponseStatus status) : status(status) {
34         }
35         explicit Response(ResponseStatus status, S3MemoryContext& context)
36             : status(status), dataBuffer(context) {
>>>     CID 163037:  Uninitialized members  (UNINIT_CTOR)
>>>     Non-static class member "responseCode" is not initialized in this constructor nor in any functions that it calls.
37         }

________________________________________________________________________________________________________
*** CID 163033:  Uninitialized members  (UNINIT_CTOR)
/extensions/gpcloud/include/s3params.h: 13 in S3Params::S3Params(const std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>> &, bool, const std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>> &, const
+std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>> &)()
7
8     class S3Params {
9        public:
10         S3Params(const string& sourceUrl = "", bool useHttps = true, const string& version = "",
11                  const string& region = "")
12             : s3Url(sourceUrl, useHttps, version, region), keySize(0), chunkSize(0), numOfChunks(0) {
>>>     CID 163033:  Uninitialized members  (UNINIT_CTOR)
>>>     Non-static class member "verifyCert" is not initialized in this constructor nor in any functions that it calls.
13         }

________________________________________________________________________________________________________
*** CID 163032:  Error handling issues  (UNCAUGHT_EXCEPT)
/extensions/gpcloud/src/compress_writer.cpp: 9 in CompressWriter::~CompressWriter()()
3     uint64_t S3_ZIP_COMPRESS_CHUNKSIZE = S3_ZIP_DEFAULT_CHUNKSIZE;
4
5     CompressWriter::CompressWriter() : writer(NULL), isClosed(true) {
6         this->out = new char[S3_ZIP_COMPRESS_CHUNKSIZE];
7     }
8
>>>     CID 163032:  Error handling issues  (UNCAUGHT_EXCEPT)
>>>     An exception of type "S3RuntimeError" is thrown but the throw list "throw()" doesn't allow it to be thrown. This will cause a call to unexpected() which usually calls terminate().
9     CompressWriter::~CompressWriter() {
10         this->close();
11         delete this->out;
12     }

________________________________________________________________________________________________________
*** CID 163029:  Memory - illegal accesses  (STRING_NULL)
/extensions/gpcloud/src/gpwriter.cpp: 49 in _ZN8GPWriter18constructRandomStrB5cxx11Ev()
43     string GPWriter::constructRandomStr() {
44         int randomDevice = ::open("/dev/urandom", O_RDONLY);
45         char randomData[32];
46         size_t randomDataLen = 0;
47
48         while (randomDataLen < sizeof(randomData)) {
>>>     CID 163029:  Memory - illegal accesses  (STRING_NULL)
>>>     Function "read" does not terminate string "randomData[randomDataLen]".

________________________________________________________________________________________________________
*** CID 163028:  Performance inefficiencies  (PASS_BY_VALUE)
/extensions/gpcloud/src/s3interface.cpp: 265 in S3InterfaceService::listBucket(S3Url)()
259     // ListBucket lists all keys in given bucket with given prefix.
260     //
261     // Return NULL when there is failure due to network instability or
262     // service unstable, so that caller could retry.
263     //
264     // Caller should delete returned object.
>>>     CID 163028:  Performance inefficiencies  (PASS_BY_VALUE)
>>>     Passing parameter s3Url of type "S3Url" (size 256 bytes) by value.
Signed-off-by: NHaozhou Wang <hawang@pivotal.io>
Signed-off-by: NYuan Zhao <yuzhao@pivotal.io>
Signed-off-by: NAdam Lee <ali@pivotal.io>

Without this commit, test scripts will re-compile gpcloud anyway, better
to use the one from package instead in the main pipeline.

[ci skip]

The storage_persistent_filerep_accessmethods_and_vacuum task contained
two filerep tests:

FilerepResync.test_filerep_resysnc ... 585576.51 ms ... ok
FilerepProcArrayRemoveTestCase.test_verify_ct_after_procarray_removal ... 136553.63 ms ... ok

Move them together with the filerep_end_to_end_xlog_ctlog_cons task, so that
all the filerep tests are in the same task. To reflect that, rename the
'filerep_end_to_end_xlog_ctlog_cons task' to just 'filerep'.

The 'storage_persistent_filerep_accessmethods_and_vacuum' task was the
slowest task among the parallel tasks in the job, so moving the two filerep
tests elsewhere should reduce overall runtime of the storage job.

If the segrelid, visimaprelid, or the visimapidxid fields are 0 for an AO
table, then error out.

These are already covered by the 'appendonly' and 'uao_dml/uao_dml' tests
in the main test suite.

"int" and "int4" are the same datatype. They are mapped to the same early
on, in the parsing stage. For those tests that have been duplicated for
many different datatypes, remove the _int_ variant, leaving just the
_int4_ variant.

We have sufficient coverage for these in the 'geometry' test in the main
test suite. There are no UPDATEs in 'geometry', but that seems
uninteresting; there is no reason to believe that UPDATE on these types
would behave differently from an UPDATE on any other type.

The test descriptions were misleading, BTW. These are not "boundary"
tests, the values used are not close to the min or max values the data
types can represent.

PR pipeline: use 'max_in_flight: 1' and 'version: every' to ensure all PRs flow through the pipeline
Signed-off-by: NJingyi Mei <jmei@pivotal.io>

Signed-off-by: NTom Meyer <tmeyer@pivotal.io>

Signed-off-by: NTom Meyer <tmeyer@pivotal.io>

We have a bunch of existing tests that test whether ORCA falls back. They
set optimizer_log_failure='all' and client_min_messages='log', and grep
the output for "Planner" or "Planner produced plan". That's error-prone.

This new GUC makes that kind of tests easier and more robust. You can
simply set the GUC, and if there are no extra INFO messages in the output,
ORCA didn't fall back.

We had already disabled ORCA for some queries, but the query to estimate
reltuples and relpages still used ORCA. This makes ANALYZE slightly faster
(because ORCA is slow to start up), and avoids spurious messages about
GPORCA falling back when the new optimizer_trace_failure option is used.

Currently, we have a job compile_gpdb_custom_config_centos6 in
pr_pipeline and master that compiles gpdb with a custom config, but the
purpose of this job is unclear. It turns out that this job is for
compiling gpdb the open-source way, so we renamed the job as
compile_gpdb_open_source and changed the name of task yml and scripts.
Signed-off-by: NDavid Sharp <dsharp@pivotal.io>

* Move all the udf_insert_* tests to the main suite.

* Move 'volatilefn_dml_int8' test to the main suite.

* Remove all the other volatilefn_dml_* tests. There's no reason to believe
  that the data types used in the table would make a difference, so keeping
  one variant of this is enough.

* Run behave tests in concourse

* Initial steps to enable us to run behave tests in concourse.
  This will only work for single node behave tests.
Signed-off-by: NChumki Roy <croy@pivotal.io>

Below columns are added:

* rsgid: resgroup id;
* rsgname: resgroup name;
* rsgqueueduration: queued duration in the resgroup;

As resgroup is not fully implemented yet there is only dummy output for
these columns at the moment.

Add view gp_toolkit.gp_resgroup_status for resource group runtime
statistics information, such as cpu usage, memory usage, etc..

Example:

    -- query resource group runtime statistics information
    SELECT * FROM gp_toolkit.gp_resgroup_status;

A helper function pg_resgroup_get_status_kv() is also provided to
collect these information.

A new dir src/backend/utils/resgroup/ is created to hold the resgroup
source code.

As resgroup is not fully implemented yet there is only dummy output for
this view at the moment.

Add view gp_toolkit.gp_resgroup_config for resource group information,
such as concurrency, cpu rate limitation, etc..

Example:

    -- query resource group config information
    SELECT * FROM gp_toolkit.gp_resgroup_config;

docker image jtarchie/pr updated recently, which breaks our git
submodule pulling, fall back to the working tag.
Signed-off-by: NAdam Lee <ali@pivotal.io>
Signed-off-by: NYuan Zhao <yuzhao@pivotal.io>

I'm not how many of these tests are worth keeping at all, but as a first
step, let's get them into the main test suite. The runtime of the parallel
group with these tests is about 1 minute on my laptop.

This is mostly a straightforward move, with mechanical changes removing the
useless TINC tags. However, some tests were marked with the special
"@executemode ORCA_PLANNER_DIFF" tag. The code in __init__.py ran those
tests differently. Normally, you run a test, and comparet the output with
the expected output file. But those ORCA_PLANNER_DIFF tests were run twice,
with and without ORCA, and the results were compared against each other,
instead of the expected output stored in the repository. The expected
outputs of those tests were out of date, you got a different error message
now than what was remembered in the expected output.

Now those tests are run as normal pg_regress tests. I updated the expected
output of those tests to match what you get nowadays.

pg_size_pretty() is completely ordinary and boring function. It doesn't
do database access or anything like that. We can live without testing it
in every possible context.

* dml_boundary_bool seems redundant. We surely have a lot of other tests
  involving boolean columns.
* dml_boundary_char is redundant with the 'char' test in main suite.
* dml_boundary_money was disabled.
* dml_boundary_smallint was redundant with 'int2' test in main suite.
* dml_boundary_integer was redundant with 'int4' test in main suite.
* dml_boundary_bigint was redundant with 'int8' test in main suite.

There are a lot of similar and more interesting tests for direct dispatch
in the main test suite ('direct_dispatch' test, for example).

The negative test cases, like inserting into a randomly distributed table
so that direct dispatch is not possible, were not very interesting, so I
removed them. A mistake in the other direction, i.e. not performing direct
dispatch when we could have, does not lead to incorrect query results, just
slower performance, which is why we need bespoken tests for those. But
if we erroneously do direct dispatch for a query, where it is not legal,
that will lead to incorrect query results, so we should catch those cases
by all the other tests in the test suite.

There was one test case, involving DEFAULTs in the INSERT statement, that
I thought is marginally worth keeping, so added a case for that in
'direct_dispatch' before removing it.

We have sufficient coverage for these in the main regression suite already.

* Partitions are covered by the 'partindex_test', 'partition', and
  'partition1' tests.

* ALTER TABLE on non-partitioned tests are covered in 'alter_table' test.

* SET DISTRIBUTION POLICY is covered by the 'alter_distribution_policy'
  test. It seems unnecessary test so many different data types,
  there is no datatype-specific handling in the set distribution policy
  code.

* Merge recent changes

* Remove examples with deprecated functions

* Move examples using deprecated analytic functions to MADlib doc, and redo with MADlib functions.

* Removes deprecated functions, relocates the unnest() function, and links to MADlib topic.

[ci skip]

This will make the test more reliable, because extra waiting in the relcache
initialization caused some case the segment took longer to recover.

Bad things happen otherwise. One case in point is create database
followed by a crash. Create database requests a checkpoint after
inserting new tuple into pg_database. Crash happens right after
create database commits and before clog update is not flushed to disk.
Relcache initialization before xlog replay will set HEAP_XMIN_INVALID
hint bit in the newly created database's tuple because clog did not
report the xmin as committed.

The FileRep processes including:
- recovery process
- resync manager process
- resync worker process

The need for heap access methods before xlog replay is removed by commit
e2d6aa1481f6cdbd846d4b17b68eb4387dae9211. This commit simply moves the
relcache initialization to pass4, where it is really needed.

Do not bother to remove relcache init file at the end of crash recovery pass2.

Error out if relation cache initialized at wrong time.

Handling of gp_reject_internal_tcp_conn was defined twice in the
GUC file, remove one.