Separate key downloading and decompressing functions, make it more
flexible and robust.
Signed-off-by: NAdam Lee <ali@pivotal.io>
Signed-off-by: NPeifeng Qiu <pqiu@pivotal.io>

* change code style to avoid AllowShortFunctionsOnASingleLine

* add a few cases for uncompression

* gzip reader phase 2: read() with the buffer <= size of chuck

* gzip reader phase 3: read() with the size of chuck < internal read buffer

* gzip reader phase 4: read() with the buffer >> size of chuck
Signed-off-by: NKuien Liu <kliu@pivotal.io>

s3interface is to handle all s3 restful api wrapper, like listBucket()
and fetchData(), s3restful is to handle HTTP GET, and PUT in future.

Also introducing google mock for unit tests.

refactor, optimize and rename them to gpreader and s3reader.

Currently, gpdemo does not build mirrors for Darwin and Sparc. This was a
restriction from the early 2010s and should not be an issue today with
modern systems.

Saving the error message into the static buffer can lead to overflow
since the string copy isn't bounded. Move to using a bounded copy and
also indicate truncated messages by ending them with ".."

Considering that this doesn't seem to have happened indicates that it's
not worth dynamically extending/shrinking the buffer to cope with the
input, simply ensuring to not overflow is probably enough.

Fixed the function return value declarations to be consistent while
in there fixing another thing. There are more style fixes that could
be applied but one thing at a time.

Spotted when working on related code.

If we are unable to unlink the temporary files in the shareinput
cleanup it seems quite aggressive to insist_log() and forcibly
exit. This cleanup can be called as part of transaction abort.

When bfz_close() is called in the codepath during the abortion of a
transaction we must avoid throwing even more errors unless the
situation calls for it. For bfz_close() it's fine to lower the
ereport level to WARNING in this case. Longer term we should move
this, and other, codepaths away from calling unlink() directly and
instead use the API provided but this closes a current issue in
ICG so better to close this immediately and refactor all callsites
when having a clean ICG.

The LOG_GANG_DEBUG statements that include members in the Gang struct
should only run after we've validated that the Gang struct pointers
aren't NULL to avoid segfaulting.

Found this README while reading some code and rather than fixing
the incorrect path specification it seemed equally useful to just
remove it since it carries very little information.

When create gang failed and all segment DB are alive,
we could retry when any of the following conditions is met:
1) This is the writer gang.
2) This is the first reader gang and the writer gang is healthy.
3) All failed connections are due to segment recovery.

Signed-off-by: NShreedhar Hardikar <shardikar@gmail.com>

This closes #836
Signed-off-by: NMarc Spehlmann <cramja@gmail.com>

reorder header file

This closes #858

Currently gp_connections_per_thread=0 means using main thread to
do dispatch instead of creating new threads. It's not working if
segments report error before sending back any data.
For example:
create table foo(id int);
insert into foo values(0);
set gp_connections_per_thread=0;
select 100/id from foo;

We disable it for now and will reenable it when we have the new
implementation of non-threaded dispatcher.

No functional changes are included in this diff, only trailing
whitespace cleanup.

The agebracket.yml mapred test case is testing the reduction with
a simple reducer summing the values. GPMapred has a builtin SUM
reducer however which will yield the same result, extend the test
with a copy of the agebracket.yml test using the builtin ensuring
that the output is identical.

I was alerted to this by a compiler warning, which this also fixes. Was
broken by the 8.3 merge.

We had backported similar changes to the other pg_*_is_visible()
functions earlier, so for consistency, do the same for the new functions
we got from the 8.3 merge.

The related upstream commit was:

commit 66bb74db
Author: Tom Lane <tgl@sss.pgh.pa.us>
Date:   Mon Dec 15 18:09:41 2008 +0000

    Arrange for the pg_foo_is_visible and has_foo_privilege families of functions
    to return NULL, instead of erroring out, if the target object is specified by
    OID and we can't find that OID in the catalogs.  Since these functions operate
    internally on SnapshotNow rules, there is a race condition when using them
    in user queries: the query's MVCC snapshot might "see" a catalog row that's
    already committed dead, leading to a failure when the inquiry function is
    applied.  Returning NULL should generally provide more convenient behavior.
    This issue has been complained of before, and in particular we are now seeing
    it in the regression tests due to another recent patch.

This test was introduced in commit c36ecd6f.
It does not actually test the changes of that commit, instead it was aiming to test some other scenarios.
It seems that the test does not have a deterministic behavior.
We remove it for now, and we will investigate if the scenarios that we were aiming to test are possible and write deterministic tests.

The recent hacks to make xlogdump work with the 8.3 merge had a
thinko where the initialization of the struct ended up inside the
guards for version checks effectively breaking it for < 9.0.

Spotted (and reviewed) by Heikki Linnakangas

It doesn't seem too useful, to hae a DEBUG1 of tuple counts. I doubt
anyone has taken advantage of that in years. Removing it allows us
to refactor the code inside GetPlannedStmtLogLevel() back into
GetCommandLogLevel(), like it is in the upstream.

There are no callers to it outside the file, so why not. Reduces the
diff vs. upstream.