Previously accounting tree and reference to accounts were pointer based.
This prevents us from freeing accounts at the end of each statement as
later statements of that transaction can potentially access memory and
the corresponding accounts. This resulted in a perceived memory leak as
the accounts were not freed per-statement.

In this implementation we convert pointers to index that can refer to an
array of accounts. This allows us to partition indexes based on
"generation" to define which accounts are live and which are already
dead. This is also a soft-pointer so we don't crash accessing it. This
also allows us to optimize the previous heavyweight "rollover" at the
time of generation overflow as we no longer need to detect the validity
of accounts based on generation, and rather we can just refer to logical
range of indexes to identify the liveness.

[#120028219]

Two sub query cases and a stress case which could trigger segment fault with former version.
Signed-off-by: NPeifeng Qiu <pqiu@pivotal.io>

1. Add PGAllocator class to delegate palloc to std::vector.
2. Replace all significant buffer with S3Vector, which uses
PGAllocator to allocate memory.
3. Fix header problems, retire s3common.h and s3common.cpp
4. Fix unit tests and regression tests.
Signed-off-by: NKuien Liu <kliu@pivotal.io>
Signed-off-by: NAdam Lee <ali@pivotal.io>
Signed-off-by: NHaozhou Wang <hawang@pivotal.io>

We assume that only the main thread should handle signals, so
we need to make child threads be quiet with a set of signals.
Signed-off-by: NKuien Liu <kliu@pivotal.io>

The implementation of multi-thread in S3 writers is different from
that of S3 reader because it creates threads dynamically (with benefits
when data size to upload is small). But when exception is catched in
thread, it needs to notify the main thread again, otherwise it may
hang for waiting to create a new thread.
Signed-off-by: NKuien Liu <kliu@pivotal.io>

Some production systems, especially docker images, have no system-wide
third-party header files. This commit could also reads header files from
GPDB directory to compile gpcheckcloud.
Signed-off-by: NHaozhou Wang <hawang@pivotal.io>

If user specifies bad config value, for example, chunksize > INT_MAX,
or threadnum < 0 etc, we add range check to prevent undefined value.
Signed-off-by: NKuien Liu <kliu@pivotal.io>

When config file does not exist or is failed to parse, we write
log to pg_log by default. In previous, related logs are lost.
Besides, we initialize segid and segnum before parsing config
file to print correct message to users.
Signed-off-by: NHaozhou Wang <hawang@pivotal.io>

strcmpci() is a function, served in built-in lib/ini, which
compares strings while ignoring case. With it we can set the
log_level as 'DEBUG', 'debug', or 'Debug', or log_type as
'Internal' or 'internal'. It affects only the log related
configures in gps3ext extension.
Signed-off-by: NHaozhou Wang <hawang@pivotal.io>

To ignore line numbers and so on in expected and results outputs while
pg_regress detecting failures.

When SELECT FROM an external table with constraint 'LIMIT 0',
the transaction will abort. We add this regression test to
verify gps3ext can handle this case.
Signed-off-by: NPeifeng Qiu <pqiu@pivotal.io>

In some cases QEs may be reset, then S3 extension stays on undefined
behaviors. It may or may not catch the transaction abort signal.
For example, when GPDB cannot parse imported data from S3, query
and transaction are aborted and reset as new QE context starts.

To solve it, we keep the signal of transaction cancel once it is
captured until we stop all running threads and exit gracefully.
Signed-off-by: NHaozhou Wang <hawang@pivotal.io>

1. add s3ext specific exception types and handlers
2. enchance the error messages
3. refactor CHECK_OR_DIE marcos
4. refactor S3RestfulService and S3Interface classes
5. refactor S3BucketReader's function getNextKey()
6. retire two kinds of "ResponseStatus": RESPONSE_FAIL, RESPONSE_ABORT
7. update corresponding unit tests
Signed-off-by: NHaozhou Wang <hawang@pivotal.io>

1. Handle multiple config sections.
2. Able to check if the specified section exists.
3. Update regression test cases to verify different config sections in
one query.
Signed-off-by: NAdam Lee <ali@pivotal.io>

Also update some methods and variables' names.
Signed-off-by: NKuien Liu <kliu@pivotal.io>

Avoid cycle including, keep codes simple and clear.
Signed-off-by: NHaozhou Wang <hawang@pivotal.io>

Support different configurations, except logging related ones, for
different s3 external tables.

Also retired most of the global variables.

Signed-off-by: Haozhou Wang, Kuien Liu

Refactor compress_writer and s3key_writer to support
row larger than chunk buffer, fix a corner case when
the size of compression output is bigger than input.

Add two regression tests to generate rows with mixed length,
some of rows are larger than the size of chunkbuffer.
Signed-off-by: NPeifeng Qiu <pqiu@pivotal.io>

Support to interrupt gpcheckcloud by signals, including SIGTERM, SIGINT
SIGABRT, SIGTSTP. For example, user can press CTRL-C to stop the program.
And, we fix a deadlock in multi-thread upload during canceling. Besides,
we further improve the message of gpreader for ABORT event.

Signed-off-by: Haozhou Wang, Peifeng Qiu

Pivotal tracker ID: [#129425589]

Signed-off-by: NAdam Lee <ali@pivotal.io>

GCC 4.4.2 doesn't fully implement move semantics, use swap() instead of
std::move() to move vectors as a workaround.
Signed-off-by: NAdam Lee <ali@pivotal.io>

Signed-off-by: NPeifeng Qiu <pqiu@pivotal.io>

To detect the query interruption event (e.g., triggered by user),
we should consider both QueryCancelPending variable and transaction
status. Here QueryCancelPending is not sufficient because it will be
reset before the extprotocol last call, then we cannot distinguish
normal exit/finish from abnormal transaction abort.

Signed-off-by: Peifeng Qiu, Haozhou Wang

1. modify s3key_writer, support concurrent multipart uploading.
2. update upload cancel logic.
3. add unit tests.
Signed-off-by: NKuien Liu <kliu@pivotal.io>

1. remove several tests which are duplicate to s3 regression.
2. add timer to record elapsed time of each test.
3. fix an issue that upload failure will not be captured.

Signed-off-by: Haozhou Wang, Peifeng Qiu, Adam Lee

Signed-off-by: NHaozhou Wang <hawang@pivotal.io>

Update copyright statement

Adresses #1056

Four characters "'", "\", "<", ">" could cause shell injection in
gphdfs. This quick fix filters out them to avoid the security issue.
Signed-off-by: NHaozhou Wang <hawang@pivotal.io>
Signed-off-by: NPeifeng Qiu <pqiu@pivotal.io>

gpcheck was deprecated in 4.3.6.0 but the configuration file was
never removed. Clean out together with install rule.

Reported by zhangyuvip

CI for some reason is getting hung, waiting for someone to hit
enter in probe_config. So, disabling pagination for psql for the same.

If there is an unknown configuration parameter in postgresql.conf,
currently we silently print a log message and skip reading any
configuration parameter from postgresql.conf. This does not prevent
database startup. As some of our gucs depend on explicitly mentioned
values on certain platforms (e.g., gp_vmem_protect_limit is set to 8GB
explicitly in postgresql.conf, and without explicit mention, it defaults
to 0 on darwin platforms), this can result in assertion failure and
confusing dtm errors as QEs fail to start with 0 vmem protect limit (and
potentially other invalid guc values).

This change prevent database startup if there is any unknown
configuration parameter.

This is relevant to Github issue #1201. However, for issue #1201 we will
need another fix to correct bugbuster test so that it doesn't put
unrecognized parameter in the postgresql.conf.

* Revert "Preventing gpstart if there is an unknown configuration parameter in postgresql.conf."

This reverts commit 6fa636cd924aea83e22964e0d88d0e9736369cc2.

* Error out GPDB start if guc context is PGC_POSTMASTER.

designed to verify no QE left alive when QD has exited, but QE exit in this
situation is an asynchronous procedure, so we have to wait some time before we
do verification.

In certain rare cases, crash recovery may prevent start up due to stale
relfilenodes in relcache init file.  E.g. database crashes right after reindex
operation on critical system tables in "template1" database is committed.

This can happen only in case of "template1" database because MyDatabaseId is
set to "template1" during crash recovery.  Relcache needs to be initialized in
pass 2 of recovery so that heap access methods can be used to update persistent
tables.  Relcache init file written in pass 2 contains critical relations and
indexes from "template1" database.  Pass 3 applies redo records from xlog to
all relations other than persistent tables.  Relfilenodes in the relcache init
file may become out of date during pass 3.  Note that pass 3 does not need
relcache and therefore does not bother keeping it up to date.  Pass 4 should
initialize relcache from scratch rather than using a stale init file created by
pass 2.
Signed-off-by: NAshwin Agrawal <aagrawal@pivotal.io>