Update gpcheckcloud configure template with autocompress,
and clean some codes and comments.
Signed-off-by: NPeifeng Qiu <pqiu@pivotal.io>

If 'autocompress' in s3 configure file is set to 'true', all data will be
compressed before uploaded to S3. In this way, we can reduce the network
traffic significantly, which means money-saving as well.

The data will be compressed in 'NO_FLUSH' way before injected into underlayer
S3KeyWriter's buffer, and the latter will invoke RESTFul layer to finish data
uploading. We don't buffer data issued from s3extprotocol in compression layer,
that is, all data blocks will be injected into ZStream to deflate immediately.
It is because experimental results show little performance improvement by doing
this while consuming more memory.

Signed-off-by: Peifeng Qiu, Adam Lee

1. Support "CANCEL" during uploading, the uploaded parts in S3 will be safely deleted.
2. The s3ext will not retry 3 times after the query is canceled by users.
3. Error messages are fixed.
Signed-off-by: NHaozhou Wang <hawang@pivotal.io>

Support downloading and uploading with special characters in URL, such
as "?&=:+".
Signed-off-by: NAdam Lee <ali@pivotal.io>
Signed-off-by: NHaozhou Wang <hawang@pivotal.io>

The extension of s3 files for writing is determined by
the format of data source (from FORMAT clause) to replace
the default 'data'. For example, file with '*.csv' is
more friendly for users.
Signed-off-by: NPeifeng Qiu <pqiu@pivotal.io>

1, fixed a bug in POST method
2, support uploading in gpcheckcloud
3, add regression cases for gpcheckcloud uploading
Signed-off-by: NAdam Lee <ali@pivotal.io>
Signed-off-by: NHaozhou Wang <hawang@pivotal.io>

add a set of url related utilites, such as getSchemaFromURL(), getRegionFromURL(),
getBucketFromURL(), getPrefixFromURL(), replaceSchemaFromURL(), which are used
in both s3writers and s3readers. merge URL parser and URL utilities into an unique
s3url files.
Signed-off-by: NKuien Liu <kliu@pivotal.io>

1. Refactor ListBucket function of S3Service, get rid of all pointers. Throw exception if listBucket() fails.
2. Refactor XML response message extraction code.
3. Fix unit tests.
Signed-off-by: NHaozhou Wang <hawang@pivotal.io>
Signed-off-by: NPeifeng Qiu <pqiu@pivotal.io>

Signed-off-by: NAdam Lee <ali@pivotal.io>
Signed-off-by: NKuien Liu <kliu@pivotal.io>

Replace path of WET tests as a date based random string, so we are able to run
WET regression cases on different pipelines simultaneously.

Also could configure s3 config file's path in Makefile now, which was
hard coded in sql files.
Signed-off-by: NAdam Lee <ali@pivotal.io>
Signed-off-by: NKuien Liu <kliu@pivotal.io>

Gcc 6.1 complains about "tautological compare". Per the comment, the
intention here is to unconditionally fail the assertion, so use a more
straightforward Assert(false) to do that.

While the compiler will do a perfectly good job of optimizing this
the function doesn't add anything to readability of the code given
how trivial the function is. Remove and inline the init code.

The UPD signalling code was added due to a bug in the pthreads code
in OS X 10.6 Snow Leopard which prevent pthread_cond_timedwait() to
work as intended. This version of the OS was discontinued in 2011
so it's time to retire this work around. OS X is not a supported
GPDB platform and as such we only make a best effort to run it on
the currently latest version.

In the process also make OS X use _timedwait() and not the relative
function for simplicity.

Commit 1eeea564 forgot to add the test queries to the ORCA-specific expected
output file, causing the test to fail. But on closer inspection, there
aren't any real differences between the ORCA and non-ORCA outputs, so let's
just remove the alternative file.

This seems to have been harmless, by pure chance. Passing the 0 (false)
instead of -1 as the location would only affect the context information in
error messages. Passing -1 as the boolean 'include_dropped' variable makes
expandRTE to include dropped columns in the returned list, but that seems
to be harmless too, given what the caller uses the list for. Nevertheless,
it's clearly a bug, so fix it.

Seems better to be precise with these. AOCS tables already used int8s for
these, so this makes things more consistent, too.

Whitespace and comment fixes, to follow the usual project style.

Remove duplicated function comments between the .h and .c files. Per the
usual project convention, all explanations of a function and its arguments
are in the .c file, and the .h file only contains the prototypes. There
were some additional comments about the "sections" of the files that
seemed useful, but were only in the .h files. I moved those to the .c
files instead.

Turns out that commit 6c025b52 subtly
changed the CRC calculation. The old crc32cFinish() inline function
returned the final checksum, while the new FIN_CRC32C() macro modifies
the variable in place. The old calls to crc32cFinish() discarded the
return value, and were therefore in fact no-ops. That was surely not
intentional, but it doesn't make any difference to the strength of
the checksum, so it doesn't seem worth changing from previous releases.

Looking at old git history, this was added back in 2009. The related
ticket on adding it said:

    Add GUC that make every buffer page from the buffer pool flushed on
    eviction.

    Note that this will NOT necessarily flush all buffer pages when the
    postmaster is shutdown. I think this is acceptable for our purposes.

    (Our purpose is to make sure that overwrites of the buffer pages are not
    lost and instead are always written to disk so we can catch errors)

I'm not sure what errors that was meant to catch, or how, but I don't
think we have any regression tests or anything that uses it anymore.
Let's remove it, to make merging with upstream easier.

These were only used by CaQL. I didn't realize that earlier..

This actually returns 6 rows and not 5, not sure why that hasn't
triggered before but the reorganisation of gpdiff uncovered it.

The variadic and default parameters tests are specific to GPDB so
move them to our own regress schedule file.

Not invoking string interpolation when it's not needed is quite a
micro optimization but over the course of many iterations it can
still be worthwhile. Only strings containing variables or escape
sequences need to be within quote marks.

This commit contains a collection of tricks to perform some level
of micro optimizations of mostly the regex code for matching output
lines in the test parsing but also some of the logic.

 * Avoid capturing parens grouped matches that we dont use: when
   matching with m/(foo|bar)/ where the match isn't used with $1
   the cost of allocating the match in memory can be avoided by
   instead matching for m/(?:foo|bar)/ which instructs Perl to
   throw away the exact match.
 * Break out early in logic: when possible reorder such that a
   static check might skip over the regex entirely. Where we
   match for line with a comment marker we can for example do
   a first check for the comment marker and only perform the
   subsequent checks if that was found. Also assist Perl with
   branch prediction in case that can be beneficial.
 * Remove case insensitive matching where possible: if we
   know that we can provide the right case in the regex instead
   there is a lot of save on processing.
 * Collapse regexes: where possible ensure that a single regex
   does more by collapsing regexes together with more logic in
   figuring out what matched.
 * Remove regexes: where possible, remove the regex entirely and
   instead use the index() function which for short strings is
   much faster (short is around 10k chars).
 * Rather than looping over arrays with foreach for trivial
   operations, use map() instead.

gpdiff had support for ignoring header lines in query output which
we aren't using in any testsuite at all. Since this required regex
checking for each line in the output (it could be turned on for a
section of the output) this feature has a non-trivial cost. Remove
as an optimization for the tests that we do have.

The previous commits have removed all usage of CaQL. It's no longer
needed.

Use the get_type_name() lsyscache function directly.

The IndexInfo structs are used by ORCA to decide which indexes match an
expression, and there may be some other similar use for them too. This patch
changes it to fetch the index expressions and predicates from the relcache,
rather than deparsing them directly from the pg_index catalog. The versions
stored in the relcache have been run through const-evaluation and
canonicalization passes, so they are slightly different. I believe the
versions stored in the relcache are what we actually want here, for the
reasons explained in RelationGetIndexPredicate() and
RelationGetIndexExpressions().

It was used for DELETEs, and SELECT COUNT(*). Replace with the usual
syscache lookups, systable scans, and simple_heap_update/delete calls,
like these things are done in the upstream.

There were only a few callers using it. Use the upstream
SearchSysCacheAttName() function instead.