* DOCS: SELECT - Add RECURSIVE keyword to WITH clause

* docs: recursive CTE - add experimental designation. Add GUC gp_recursive_cte_prototype

* docs: updated recursive CTE docs based on review comments.

Signed-off-by: NNadeem Ghani <nghani@pivotal.io>

-1 is a valid value for errordata_stack_depth until errstart() has
been called which inits the stack. In correct usage it's thus fine
to use it as an array subscript without inspecting it, since demote
and dismiss are useless outside an error context. Incorrect usage
would however cause a memory error so guard against that since it's
the right thing to do.

The memory allocated is immediately orhpaned since the pointer is
overwritten. Remove and avoid leaking.

strncpy() can render the buffer non-NULL terminated on input which
is MAXPGPATH characters long. This is not only incorrect code, it's
a mismerge since upstream has never used strncpy here, but strlcpy.
Fix by replacing with strlcpy calls, thus aligning us with upstream
better as well.

The num_cwords member was left blank which cause issues for the
freeing function which operates based on its value.

{f}stat() can fail and reading the stat buffer without checking for
status is bad hygiene. Ensure to always test return value and take
the appropriate error path in case of stat error.

Iff the value isn't a varlena, then dataLen and dataStart won't be
set but we may still come to inspect them later on. Avoid reading
uninitialized variables by zeroing them in that case and check for
that before reading.

The strncpy() will make path2 non-NULL terminated in the (rare?)
cases where filespaceLocation2 is MAXPGPATH chars long. Use strlcpy
instead like other places in the codepath does and ensure we always
have a NULL terminated string.

As part of a previous commit, the WorkerPool of threads will raise if
the number of workers is set to 0. This prevents a coding error from
resulting in no work, where work was expected.

This commit reaches through code to protect against the numWorkers
parameter being 0.

In several cases, the number of workers is set to the number of segments
or the number of databases. We do not protect those with the expectation
that something is more significantly wrong if those sums are 0, and that
an exception would be fitting in those cases.
Signed-off-by: NMarbin Tan <mtan@pivotal.io>

Signed-off-by: NNadeem Ghani <nghani@pivotal.io>

Rework how environment variables transmitted in local bash and remote
ssh commands. If you have a bash command that starts with a conditional,
you'll get a syntax error.

To fix this, we added ampersands to join the setting of the environment
variables and the command itself.

NOTE: We removed an ExecutionContext static variable that recorded
environment variables. This feature was not used anywhere.
Signed-off-by: NMarbin Tan <mtan@pivotal.io>

Previously, RemoveDirectories and RemoveFiles used the unix command
"rm -rf", but this is inefficient for huge numbers of files.
Also, these functions accepted any globbed path.

Instead, use "rsync" to optimize deletion of files in a directory.
On a DCA using 1 million files, this increased speed by about 3x.

Also, this commit breaks up the different use-cases of deletion into
separate methods, adding methods RemoveDirectoryContents() and RemoveFile()
and RemoveGlob() to help isolate the assumptions of each case and
optimize for them.
Signed-off-by: NLarry Hamel <lhamel@pivotal.io>
Signed-off-by: NShoaib Lari <slari@pivotal.io>

This utility is slated for removal in v6. It is unused in general.
Fix obvious compile errors and basic logic for logging.
Signed-off-by: NShoaib Lari <slari@pivotal.io>

Signed-off-by: NLarry Hamel <lhamel@pivotal.io>

Signed-off-by: NLarry Hamel <lhamel@pivotal.io>

Signed-off-by: NMarbin Tan <mtan@pivotal.io>
Signed-off-by: NLarry Hamel <lhamel@pivotal.io>

Fix Travis by adding --disable-pxf to configure command. We could
always just add json-c-devel package and see if it works... but we
need to determine if we want Travis to compile pxf or not first. We
currently have Travis set up to not compile gpAux/extensions so this
PR just adds to the current setup.

* docs: Add checksums overview to Admin Guide

* Add sentence and xref about gpstart verifying checksums.

* Fixing broken href to gpstart.xml

* Edits from review comments

Current FTS probe relies on file replication data structures mostly
found in pmModuleState.  This commit establishes an empty FTS probe
response packet that primaries using segment WAL replication can fill
out later.

We also introduce the 'w' fault_strategy (WAL replication) to replace
the 'f' fault_strategy (file replication).

This commit is carved and modified from a previously closed PR:
https://github.com/greenplum-db/gpdb/pull/2936

Authors: Abhijit Subramanya and Jimmy Yih

* Refactor GPHDFS regression to run for pxf

* remove customized Hadoop home location

* PXF tarball creation inside GPDB pipeline

* Remove legacy directory

* Use enable_pxf instead of with_pxf

* Change the directory location and update readme

When conan is used to build the dependencies (orca & xerces) of gpdb, it
copies the headers and libraries to the path specified in
the imports section of the conanfile.txt.
Changing the target copy location to /usr/local/include and
/usr/local/lib as its the default for gpdb. In case the user
prefers to have a different directory, they can change the location
accordingly.

Rather than appending to a StringInfo, return a string. The caller can
append that to a StringInfo if he wants to. And instead of passing a
prefix as argument, the caller can prepend that too.

Both callers passed the same format string, so just embed that in the
function itself.

Don't append a trailing "; ". It's easier for the caller to append it,
if it's preferred, than to remove it afterwards.

Also add a regression test for the 'gp_enable_fallback_plan' GUC. There
were none before. The error message you get with that GUC disabled uses the
gp_guc_list_show function.

Plus other minor cleanup.

Previously, during gpinitsystem, the standby was instantiated in the middle of
setting up the master. This ordering caused problems because
initializing the standby could cause an exit when an error
occurred. As a result of this early exit, the gp_toolkit and DCA gucs
were not set properly.

Instead, initialize the standby after the master is finished.
------------------------------------------
Previously the exit return code for gpinitsystem was always non-zero.
Now, it is non-zero only in an error or warning case.

The issue was due to SCAN_LOG interpretation of an empty string as a line count of one.
Fixed by changing to word count.
------------------------------------------
Initializing a standby can no longer cause gpinitsystem to exit early.
Added extra logging/output about standby master status.
Tell user at the end of gpinitsystem if gpinitstandby failed.
------------------------------------------
Signed-off-by: NMarbin Tan <mtan@pivotal.io>

This change adds a check to gpaddmirrors: First check if heap_checksum
setting is consistent across cluster. If not, fail immediately, else
continue with the normal workflow.
Signed-off-by: NShoaib Lari <slari@pivotal.io>
Signed-off-by: NMarbin Tan <mtan@pivotal.io>

Signed-off-by: NNadeem Ghani <nghani@pivotal.io>

The locking contract to access LSN of a page is:
1. Content lock must be held in exclusive mode,
OR
2. Content lock must be held in shared mode and buffer header spinlock
must be held.

PageGetLSN() and BufferGetLSNAtomic() now assert that this contract is
maintained for shared buffers. To make the implementation for
PageGetLSN() a little easier, move to a static inline function instead
of a macro. Callers passing a PageHeader must now explicitly cast to
Page.
Signed-off-by: NAsim R P <apraveen@pivotal.io>
Signed-off-by: NAshwin Agrawal <aagrawal@pivotal.io>

Certain callers of PageGetLSN weren't correctly holding the buffer
spinlock; it is needed whenever the buffer content lock is not held in
exclusive mode.

For heapam.c, also ensure that we don't access the LSN after releasing
the lock.
Signed-off-by: NAsim R P <apraveen@pivotal.io>

Most, if not all, of the queries in the qp_olap_windowerr test, contained
gpdiff "mvd" directives, to tell gpdiff what the expected order of output
rows is. However, all of the queries in that test fail on purpose, because
of varios errors. That means that the "mvd" directives didn't do anything,
because there were not result sets in the output.

However, commit de548159, added a few tests that return a result set,
to the end of the test script. That caused the preceding "mvd" directives
to be applied, incorrectly, to those new result sets. That produced a lot
of messages like "specified MVD column out of range: 3 vs 1" in the
console. While harmless, they didn't cause the test to fail, let's be tidy.

'nuff said.

It might cause conflicts, safe to remove once put `PG_MODULE_MAGIC` into
`extern "C"` block.

1. Add dependency packages to Ivy
2. Modify set_bld_arch.sh to correctly recongize aix7
3. Disable unsupported python libraries on aix7.
4. Disable gpmapreduce for aix7
5. Set ADDON_DIR for aix7
   Signed-off-by: Peifeng Qiu pqiu@pivotal.io

This commit fixed the regresstion introduced by commit
WorkerPool: Error out if numWorkers is 0 or less.
Details in https://github.com/greenplum-db/gpdb/pull/3036Signed-off-by: NXiang Sheng <stanly.sxiang@gmail.com>