Rely on the credentials being in the environment rather than explicitly
passing them. Add a comment to document the requirement.

See also: 0d840bb2

When we create Left Anti Semi Join relations in case the left
side is a dummy relation, it implies that the join relation
is also a dummy relation.
Signed-off-by: NOmer Arap <oarap@pivotal.io>

QE_READER and QE_ENTRY_DB_SINGLETON are not expected to allocate transaction
IDs. They are just helper sub-processes to WRITER and should be using its xid.

SET default_tablespace ideally should just be read-only transaction and
shouldn't allocate xid. Currently get_tablespace_oid() is acquiring lock for
other purposes to protect for concurrency, ends up assigning transaction IDs to
take the lock. Setting the GUC doesn't need to acquire the lock, plus it creates
the problem as multiple readers and a writer in a gang all end-up assigning
independent xid when dispatched this GUC. This happens because SET runs as
QE_AUTO_COMMIT_IMPLICIT for writer and hence has deferred allocation of
transaction ID. The filespace ICW tests detected this case since hits assert
FailedAssertion("!(!((proc->xid) != ((TransactionId) 0)))", File: "procarray.c"
without the fix.

Simplest scenario the problem can be demostrated is:
- DROP TABLE a;
- CREATE TABLE a(a INT, b INT);
- INSERT INTO a VALUES (generate_series(1,2), generate_series(3,4)); -- generates 1 reader and 1 writer gang
- SET default_tablespace='pg_default';

The commit xlog record carries the information for objects which serves us for
crash-recovery till post-commit persistent object work is done, hence cannot
allow checkpoint in between. So, use inCommit flag to achieve the same. This
makes it consistent with FinishPreparedTransaction() as well.

Without the fix leaves out dangling persistent table entries and files on-disk
for following sequence:

- start transaction to drop a table
- `RecordTransactionCommit()`
- Checkpoint before `AtEOXact_smgr()`
- Crash/Recover

* docs: add Enum datatype

* docs: add a release note for enum datatype

* remove optimizer support statement.

The cs-aoco-compression job was added with intention to run on a CRON
instead of on every commit batch. However, I didn't notice that the
aggregate it points to contains a commit trigger... so the project
needs its own aggregate. This aggregate can be turned into a reference
later if any other test jobs become CRON triggered.

ANALYZE should only produce leaf partitions stats when gucs
optimizer_analyze_midlevel_partition and optimizer_analyze_root_partition
are set to off

This fix makes the behaviour of ANALYZE consistent with the intent of
the GUCs.

This commit also resolves #1646.
Porting only the function `tuplesort_get_stats` in tuplesort.c from the
Postgres commit
https://github.com/postgres/postgres/blob/9bd27b7c9e998087f390774bd0f43916813a2847/src/backend/utils/sort/tuplesort.c
Link to gpdb-dev discussion - https://groups.google.com/a/greenplum.org/forum/#!topic/gpdb-dev/V-zIshnNzyE

This commit adds back `resultRelations` to the out func for
`PlannedStmt`. The omission affects `debug_print_plan` and `explain
verbose` (Until 8.4+ I guess)

This seems to be an unintended omission introduced during the
back-and-forth between 4e01c159 and 35f25e89.

This commit reverted the recreation logic for primary writer gang involved in commit e28c84b2 which make some
test cases got unexpected answer files, this commit also improved getAvailableGang() so it can go through the
whole freelist of reader gangs until it found a healthy gang.

This commit adds 2 C&S to start with running natively. Also, adds logic to
print out diffs for these in case of failures for easy troubleshotting.

```
When you send a request, you must tell Amazon S3 which of the preceding
options you have chosen in your signature calculation, by adding the
x-amz-content-sha256 header with one of the following values:

If you choose chunked upload options, set the header value to
STREAMING-AWS4-HMAC-SHA256-PAYLOAD.
If you choose to upload payload in a single chunk, set the header value
to the payload checksum (signed payload option), or set the value to the
literal string UNSIGNED-PAYLOAD (unsigned payload option).
```

ref:
http://docs.aws.amazon.com/AmazonS3/latest/API/sigv4-auth-using-authorization-header.html

We used to not hash or sign the payload, it's not quite secure with
HTTPS off, which is enabled by default although. And some compatible S3
services don't support "unsigned payload option", this commit fixes it.
Signed-off-by: NAdam Lee <ali@pivotal.io>
Signed-off-by: NYuan Zhao <yuzhao@pivotal.io>

Issue: #1759

Replaces long running behave tests in gptransfer suite with unit tests
since they're only testing argument parsing. This shortens the gptransfer
suite by about 15 minutes.

Authors: Chris Hajas and Jamie McAtamney

* docs: Updatable cursors support

* fix PL/gpSQL typos

* Move updatable cursors release note to changed features and note that it is a clarification and doc change.

The test output with ORDER BY was not fully defined, because there were
some rows with the same key column, but different second column. Remove
ORDER BY from those queries, and let gpdiff.pl mask the difference in row
order instead.

Commit 20332d98 disabled the ALTER TABLE
CLUSTER ON syntax. This removes those tests from the gptransfer test
suite.

We have most of these exact same queries in
src/bin/gpfist/regress/input/exttab1.source already, and the rest are not
very interesting. Some of the other queries not in exttab1.source already
queries regular tables, rather than external tables, or were so similar
to existing queries that they don't provide any meaningful extra coverage.

The reason to attack test this right now is that commit 743b886a broke
the test. It'd be trivial to just fix the expected output, but let's rather
remove it altogether. (Most of the other query* files in this directory
could probably be removed too, or moved to src/bin/gpfdist/regress, but
I'll leave that for tomorrow.)

Make the test deterministics. Similar to
https://github.com/greenplum-db/gpdb/pull/1655

partition.

Analyze root partition should not generate statistics for the leaf or
midlevel partitions, immaterial of the value of the GUC
optimizer_analyze_midlevel_partition.

This fix makes it consistent with our docs.

* updates to language create/alter/drop allowing db owners

* identify pl/r and pl/python as untrusted

In order to have the PR status properly reported to github, we extracted
out the action of reporting "success" into a separate job. Each task
before that reports "failure"s themselves.
Signed-off-by: NMarbin Tan <mtan@pivotal.io>

- The "custom" compile typically takes a few minutes less than the main
  compile, so this won't delay feedback
- If the custom compile goes red, it will issue a failure to the github
  hook. But without this change, the ICW would still run and possibly
  issue a later "success" to github.
Signed-off-by: NMarbin Tan <mtan@pivotal.io>

There is no legitimate reason for these to be called with an invalid type.
Turn the checks into ERRORs.

Tuplesorts take care of cleaning up any memory and temporary files on
abort, no need to have PG_TRY()-PG_CATCH() blocks to do that. These blocks
were introduced a long time ago, in a patch related to workfile caching,
but we don't do workfile caching anymore anyway.

Commit 74a1efd7 missed including the _optimizer expected output for
the test ported from Bugbuster to ICW. This fixes by including the
same hunks remembered in the planner output file.

Since there is no GPDB specific trigger suite in ICW, and this bug
was originally part of the Rio sprint back in the day, I added it
to qp_misc_jiras. The suite already contained a version of the test
so extended it to to include the segment redistribution check as
well rather than duplicating for essentially the same test. This
test relies on the rows being allocated to segment id 0, which
should hold true for test setups with 2/3 segments. Should this be
a problem going forward the test should be rewritten to check for
all rows being co-located on a single segment but for now the more
readable version is kept.

It's worth noting that the expected output in the Bugbuster suite
has the incorrect behaviour remembered, but inside ignore blocks
so this bug has never actually been tested for.

Use a macro to encapsulate the pattern of calling curl_easy_setopt(), and
ereporting if it fails.

There was a rather ugly mechanism in fileam.c, to ensure that the
utl_fclose() function was called for any still-open external table, on
abort. That relied on a global variable and assumed that there can be only
one external table open at a time, which seems like a shaky assumption.

With this patch, utl_file() is no longer guaranteed to be called on abort.
Instead, make the UTL_FILE implementations responsible for cleaning up any
low-level resources, like file descriptors or handles to external libraries
(like libcurl), themselves. Use the ResourceOwner mechanism for that.

For curl- and EXECUTE-style external tables, register a new ResourceOwner
hook to close with the libcurl handles, and launched subprocesses, on
abort. For file external tables, modify fstream.c to open the underlying
file with OpenTransientFile(), so that it gets closed automatically on
abort.

This changes the url_fopen() API, so that it never returns a NULL. Instead
of returning NULL, and reporting an error in the *response_code/string
arguments, make url_fopen() always report errors directly with ereport().
That makes the callers simpler, and allows the implementations to flexibly
report whatever details of the error they have available.

OpenTransientFile() opens an arbitrary file, like open(), but will track
the file descriptor by the resource owner mechanism, so that it is
automatically closed on abort. We already had a similar mechanism for
temporary files, and for files opened with fopen(), this fills the gap of
when you want a plain file descriptor rather than a FILE *.

I need this for the next commit, which will use this to automatically close
files opened by the fstream.c on abort.

This is a backport of upstream commit 1f67078e. I only backported the
addition of the new function, not the changes to the various callers to
use the new function, to minimize merge conflicts.

This moves all the implementation-specific parts of the big UTL_FILE struct
out of the common url.h file, into the implementation-specific .c files.
Makes the code more readable, as you have all the implementation-specific
stuff in a single .c file now.

To avoid memory leaks. The callers better be in a sufficiently long-lived
memory context.

It's not necessary to clear the error buffer on every call. Also,
GPOS_NEW_ARRAY() is a pretty slow way of clearing memory. This greatly
reduces the overhead of planning simple queries with ORCA.

On my laptop, this reduces the time for planning "SELECT 123;" from
about 60 ms to 5 ms.

I'm not sure what exactly the purpose of each of these tests are, and they
probably need more culling and cleanup, but for now, just move them out of
bugbuster, so that we can get rid of bugbuster as a separate suite.