latestCompletedXid in ShmemVariableCache is used during visibility checking to
exit early from TransactionIdIsInProgress, if transaction being checked is
higher than latestCompletedXid. During merge to 8.3 this value was set
incorrectly after pass1 of recovery, hence it reflected value what checkpoint
provides for nextXid. In GPDB pass3 is where xlog records are replayed which
would bump-up the nextXID to correctly reflect in-flight transactions. So,
latestCompletedXid needs to be set after pass3 has completed to record correct
value. Without the fix latestCompletedXid was pointing to stale transaction id
yeilding incorrect visibility of tuples post recovery.

Repro:
create table foo(id int);
-- manually kill -9 a segment QE process to simulate SIGSEGV, and force postmaster reset on that segment
select * from foo; -- fails currently, ideally shouldn't fail as well
select * from foo; -- expected to succeed but it fails with error message like "ERROR: relation with OID xxxxxx does not exist"

Thank You @eurekaka for reporting.
Fixes: #1094

Signed-off-by: NXin Zhang <xzhang@pivotal.io>

This function is used in codegen unittests.
In this commit, we also address a minor cpplint error in GenerateSlotGetAttr.

This commit
- backs out the test addition in 9c98a4f8
- adds the test to a more appropriate place: `bfv_subquery`

Generate code for advance_aggregates function that supports
SUM on int4 and float8 datatypes.
This implementation doesn't support:
i) percentile and ordered aggregates, and
ii) NULL and pass-by-ref arguments.

for gprestore -S option.

Add behave tests and unit tests.

Authors: Chumki Roy & Marbin Tan

Looks like this got applied to wrong side of the if(). It's a bit
disturbing that we haven't seen any failures because of this. Trying
to release the content lock, when we're not holding it, should trigger
an error.

Also, need to use ReleaseContentLock() rather than LWLockRelease
directly.

These functions are marked as deprecated and were removed in commit
f84888 but they were still in use by gppkg which I missed. Revert
and put them back until gppkg replaces the usage of these with non-
deprecated code.

Reported by Dave Cramer

This failure was introduced when we set the new GUC
(codegen_optimization_level) during init.
Fixes issue from this commit :
f160fa5a.

Refactor code to use common routine to fetch PT info for xlogging. Check can
be easliy added at this common place to validate persistent info is
available. Plus still add check during recovery for persistentTID zero. As
with postgres upstream merges possible the function to populate persistent info is
not called at all, so this check will not hit during xlog record construction
but atleast gives clear clue during recovery.

Vacuum lazy is harmless, but also no benefit to perform just extra work.
Vacuum full could turn out dangerous as it has potential to move tuples
around causing the TIDs for tuples to change, which violates its reference
from gp_relation_node. In general persistent table has all frozen tuples
so vacuum full is harmless too, but for example one scenario where it becomes
dangerous is zero-page case due to failure after page extension but before
page initialization. Also, since all the tuples in persistent table are
frozen inserts, skip it from database age calculation.

Close: #1057
Fix: #912

Closes: #1050
Closes: #1028

If a QE crashes for reasons such as SIGSEGV, SIGKILL or PANIC, segment
postmaster reset fails sometimes. The root cause is: primary segment
postmaster would first tell child processes to exit, then start a filerep
peer reset process to instruct mirror postmaster do reset; the filerep peer
reset process would only exit when mirror postmaster finishes or fails the
reset procedure; primary postmaster would wait for the termination
of important processes such as AutoVacuum, BgWriter, CheckPoint, filerep peer
reset process etc, before it resets share memory and restarts auxiliary
processes; however, in some cases, primary postmaster would be stuck in filerep
peer reset step, if mirror postmaster is hanging/waiting for some events;
if this happens, filerep peer reset process would wait there until timeout(1 hour),
and retry 10 times before reports failure to primary postmaster (so 10 hours in total);
so the final result is primary postmaster takes 10 hours to report reset failure.

This happens almost every time on mirror segment host machine with poor performance
for reasons that: mirror postmaster would do similar reset procedure with
primary postmaster, i.e, notify child processes to exit and wait their
terminations and then restart auxiliary processes; filerep peer reset process
would first connect to mirror postmaster to request a postmaster reset, then it would
check the reset status of mirror every 10ms by connecting to mirror postmaster;
so it can happen that filerep peer reset process keeps connecting mirror
postmaster, which would lead to continuous dead_end backend processes forked,
while at the same time mirror postmaster waits for the exit of all dead_end backend
processes, so it is possible that the speed of generating new dead_end processes
exceeds the exit speed, and hence mirror postmaster can never see the clearance of
child processes. All in all, this can lead to hang issue and failure of postmaster reset.

This issue exists for master postmaster reset as well on heavy workload circumstances.

If a QD crashes for reasons such as SIGSEGV, SIGKILL or PANIC,
postmaster reset fails sometimes. The root cause is: postmaster
would first tell child processes to exit, and then wait for the
termination of important processes such as AutoVacuum, BgWriter,
CheckPoint etc, before it resets share memory and restarts auxiliary
processes; however, WAL writer process is missed in the waiting list,
so it can happen that postmaster spawns StartupProcess and then notices
the exit of WAL writer, so it tells StartupProcess to exit; then
postmaster would notice the abnormal exit of StartupProcess in turn,
and treats it as recovery failure, then call exit() itself. Thus, we end
up with no postmaster process on master node at all.

This happens almost everytime when master host machine has poor performance.

Used when compiling generated code. EXPLAIN codegen also
runs optimize with this optimize level, making it easier to see
the features the compiler optimizes.

standardize compile flags rules of pg_basebackup with other utils like psql, pg_ctl etc.

isSimplyUpdatableRelation may get an invalid input from statements like
"DECLARE XX CURSOR FOR SELECT * FROM LOWER('HH')" where a function is selected.

1. Remove retry mechanism for reader gang and non "in recovery mode" error, gp_segment_connect_timeout is default
set to 10 mins, so it should be long enough to say we temporary lost the segments.

2. Fix "in recovery mode" retry mechanism, original codes can not recognize a in-recovery-mode error.

3. Add failure details. "failed to acquire resources on one or more segments" hide too many details.

1. Remove retry mechanism for reader gang and non "in recovery mode" error, gp_segment_connect_timeout is default
set to 10 mins, so it should be long enough to say we temporary lost the segments.

2. Fix "in recovery mode" retry mechanism, original codes can not recognize a in-recovery-mode error.

3. Add failure details. "failed to acquire resources on one or more segments" hide too many details.

4. Only destroy all gangs when create writer gang failed, otherwise it may clean cursor opened gangs and cause
unexpected error.

1.Fix primary writer gang leak: accidentally set PrimaryWriterGang to NULL which cause disconnectAndDestroyAllGangs()
  can not destroy primary writer gang.
2.Fix gang leak: when creating gang, if retry count exceed the limitation, forget to destroy the failed gang.
3.Remove duplicate sanity check before dispatchCommand().
4.Remove unnecessary error-out when a broken Gang is no longer needed.
5.Fix thread leak problem
6.Enhance error handling for cdbdisp_finishCommand

The regression was executed serially from the beginning
of construction, as time going, more tests are accumulated,
which consume much time on Concourse/Pipeline. So we modify
these tests and run them with parallel schedule.

Add regression tests on writing lots of files onto s3, join query between
local table and s3 external table, mixed data format query (with different
data format, CSV and TEXT, or TEXT with different delimiters).
Signed-off-by: NKuien Liu <kliu@pivotal.io>

1. change upload option -u -f to -u
2. update the usage help message
3. update gpcheckcloud regression tests

Signed-off-by: Kuien Liu, Peifeng Qiu

update the case which reads 5120 small files to read 2001 files.

Update gpcheckcloud configure template with autocompress,
and clean some codes and comments.
Signed-off-by: NPeifeng Qiu <pqiu@pivotal.io>

If 'autocompress' in s3 configure file is set to 'true', all data will be
compressed before uploaded to S3. In this way, we can reduce the network
traffic significantly, which means money-saving as well.

The data will be compressed in 'NO_FLUSH' way before injected into underlayer
S3KeyWriter's buffer, and the latter will invoke RESTFul layer to finish data
uploading. We don't buffer data issued from s3extprotocol in compression layer,
that is, all data blocks will be injected into ZStream to deflate immediately.
It is because experimental results show little performance improvement by doing
this while consuming more memory.

Signed-off-by: Peifeng Qiu, Adam Lee

1. Support "CANCEL" during uploading, the uploaded parts in S3 will be safely deleted.
2. The s3ext will not retry 3 times after the query is canceled by users.
3. Error messages are fixed.
Signed-off-by: NHaozhou Wang <hawang@pivotal.io>

Support downloading and uploading with special characters in URL, such
as "?&=:+".
Signed-off-by: NAdam Lee <ali@pivotal.io>
Signed-off-by: NHaozhou Wang <hawang@pivotal.io>

The extension of s3 files for writing is determined by
the format of data source (from FORMAT clause) to replace
the default 'data'. For example, file with '*.csv' is
more friendly for users.
Signed-off-by: NPeifeng Qiu <pqiu@pivotal.io>

1, fixed a bug in POST method
2, support uploading in gpcheckcloud
3, add regression cases for gpcheckcloud uploading
Signed-off-by: NAdam Lee <ali@pivotal.io>
Signed-off-by: NHaozhou Wang <hawang@pivotal.io>

add a set of url related utilites, such as getSchemaFromURL(), getRegionFromURL(),
getBucketFromURL(), getPrefixFromURL(), replaceSchemaFromURL(), which are used
in both s3writers and s3readers. merge URL parser and URL utilities into an unique
s3url files.
Signed-off-by: NKuien Liu <kliu@pivotal.io>