The libchurl abstraction layer has many internal helper functions
which weren't marked static and thus exported. Fix by marking all
as static.
Reviewed-by: NFrancisco Guerrero <aguerrero@pivotal.io>

Spotted while reading code.

CI reported some read failures just after the write action, it's
probably because the write action is not flushed yet by AWS.

Sleep and retry if AWS returns "NoSuchKey" error. Update the workflow
but not the test cases because users might have the same issue.
Co-authored-by: NPeifeng Qiu <pqiu@pivotal.io>

This reverts commit 668558dd.

CI reported some read failures just after the write action, it's
probably because the write action is not flushed yet by AWS.

Sleep and retry if AWS returns "NoSuchKey" error. Update the workflow
but not the test cases because users might have the same issue.

Previously we error out in gp_replica_check test if the cluster is not in
synced, which is usually due to bugs revealed by previous tests, instead of
gp_replica_check itself.

Print more detailed message so that people are not confused with the status and
this failure reason of the gp_replica_check test.

* Updates to GUCs from Cyrille's reviews

* Revise description of temp_buffers to match 6.0 behavior

* Fix a couple typos

* For effective_cache_size GUC, show default in blocks and size.

* Docs - update pg_class and pg_index catalog table references

* dyozie review comments

The script used in pipelines that search for explain plan changes
lists those queries that have cost, row or plan changes. In many
cases the user will want to investigate those changes further.

A new set of options generates two directories that are easy to
compare, one contains the baseline plans, one file per plan, and
the other contains the changed plans of the test.

$ ~/workspace/gpdb/concourse/scripts/perfsummary.py --help
usage: perfsummary.py [-h] [--baseLog BASELOG] [--diffDir DIFFDIR]
                      [--diffThreshold DIFFTHRESHOLD] [--diffLevel DIFFLEVEL]
                      [log_file]

Summarize the test suite execute and explain log

positional arguments:
  log_file              log file with explain/execute output

optional arguments:
  -h, --help            show this help message and exit
  --baseLog BASELOG     specify a log file from a base version to compare to
  --diffDir DIFFDIR     request diff files to be created and specify a
                        directory to place diffs into
  --diffThreshold DIFFTHRESHOLD
                        specify a numerical threshold to record plan diffs
                        with a performance regression of more than n percent
  --diffLevel DIFFLEVEL
                        specify which diff files to generate: 1 = all diffs, 2
                        = ignore cost diffs, 3 = plan diffs only

Connectivity and loaders are no longer vendored. Loaders will be
merged into clients. Add script to create msi package.

Requires a working libpq.dll. Set CMAKE_PREFIX_PATH to a successful
client build.

Behavior of stat() is not stable across C Runtime versions.
Implementation of msvcrt.dll calls FindFirstFile(), while in normal
CRT that omes with Visual Studio or Redistribute packages, it calls
CreateFile().
CreateFile() is problematic here, if path is a pipe, it will open
the pipe once then close, causing the other side to connect to the
wrong pipe. Skip stat() if the name is pipe and pretend there is one.

- Don't include strings.h when build with MSVC
- C99 Syntax fix for struct initializer.
- Call event_set to initialize event struct. Later libevent version
expect this behavior.
- Use recv instead of read. Latest CRT implementation of read only
works on files, not sockets.

- Fix upstream build system for MSVC, add option to build.pl to
only build client tools.
- Add detection for MSVC SDK version. If this field is missing,
8.1 is assumed. The latest compiler will complain about this.
- Various small fixes

* Add a list of SQL keywords to the ref guide

* Fix error from review

* Update from review comment

Story: https://www.pivotaltracker.com/story/show/164917628

   The gp-integration-testing pipeline needs the gp-clients RC
   package as a input, and then create the clients RPM base on it.
   The custom installer patch also need it to build client bin installer.
Co-authored-by: NXiaoran Wang <xiwang@pivotal.io>
Co-authored-by: NShaoqi Bai <sbai@pivotal.io>
Co-authored-by: NBob Bao <bbao@pivotal.io>

ORCA Explain plans will now contain:
`Optimizer: Pivotal Optimizer (GPORCA) version 3.35.0`

instead of:
`Optimizer: PQO version 3.35.0`

Authored-by: Chris Hajas chajas@pivotal.io

Follow-up to the previous commit, which worked fine for CentOS6 but fell
apart with CentOS7.

Because our vendored Python doesn't contain an RPATH/RUNPATH pointer to
the location of its libpython, trying to execute it directly will result
in failures at link time. The previous commit took the approach that
greenplum_path.sh takes, which is to hardcode an LD_LIBRARY_PATH that
makes up for this bug. This approach works for CentOS6, which is running
Python 2.6 as its system version. On CentOS7, which has Python 2.7, the
LD_LIBRARY_PATH causes the system Python to use the vendored
libpython.so.2.7, and virtualenv fails.

Instead of forcing a cross-linking situation with LD_LIBRARY_PATH, fix
the problem in the vendored Python binary, by using patchelf to set up a
proper RUNPATH. (We originally tried to build our vendored Python with
an RPATH set at compile time, but the only way to do that without
knowing the eventual installation prefix is by setting a relative RPATH
using the `$ORIGIN` construct, and virtualenv is unfortunately
incompatible with that.) We do this on any platforms that provide a
patchelf binary, and do our best to limp along on all others.

Along the way, get rid of the run_behave.yml task, which has been
confusing us for the entirety of this work. CCP jobs now use
run_behave_on_ccp_cluster.yml consistently.
Co-authored-by: NKalen Krempely <kkrempely@pivotal.io>

To run behave tests, behave now needs to be installed in the PATH. The
correct version of behave can be installed with pip using

    pip install -r gpMgmt/requirements-dev.txt

For the CI tests, use virtualenv, which allows behave to be installed
while still using greenplum PYTHONPATH for our vendored python modules.
This is more of a problem for platforms like centos6 where the system
python is a different version than the vendored version.
Co-authored-by: NKalen Krempely <kkrempely@pivotal.io>
Co-authored-by: NMark Sliva <msliva@pivotal.io>
Co-authored-by: NDavid Krieger <dkrieger@pivotal.io>
Co-authored-by: NJacob Champion <pchampion@pivotal.io>

This variable name is not renamed in commit cb447dc4
And will lead to compile error when cassert enabled.

AppendOnlyStorageWrite and BufferedAppend are two helper struct to do insert
for AO tables. They share many variables, but some of them with same name but
different semantic. This PR corrects their names to be consistent.

Skip gprecoverseg tablespace tests until tablespaces successfully work
with pg_rewind.
Co-authored-by: NMark Sliva <msliva@pivotal.io>

Co-authored-by: NJacob Champion <pchampion@pivotal.io>

Create the tablespace directory on all segments to support
running multi-host tablespace tests.
Co-authored-by: NJacob Champion <pchampion@pivotal.io>

Shutting down the primaries using SIGKILL keeps the PID files around,
whereas SIGTERM more gracefully stops the primaries and removes the PID
files. When PID files hang around, it keeps future tests from being able
to create a cluster.

* update pl/r docs to use CREATE EXTENSION

* update pl/perl, /python, update deprecation notices, add CASCADE info

* Feedback from Chuck

* Update postgresql.org hrefs to 9.4

* Feedback from Lisa

Since Greenplum is now based on PostgreSQL 9.4, update links to the
upstream documentation to refer to the 9.4 docset rather than older
versions. Also fix a few mentions of 8.3(.23) in the text along with
some spacing.
Reviewed-by: NDavid Yozie <dyozie@pivotal.io>

Commit 4e337a33 enabled RECURSIVE
CTEs by default, update the docs to reflect the current state of
the GUC.
Reviewed-by: NLisa Owen <lowen@pivotal.io>
Reviewed-by: NDavid Yozie <dyozie@pivotal.io>

Commit 4e337a33 enabled RECURSIVE
CTEs by default, so let's remove the final instances where we did
override the previous default value.
Reviewed-by: NLisa Owen <lowen@pivotal.io>
Reviewed-by: NDavid Yozie <dyozie@pivotal.io>

The contrib/ directory in Greenplum had over the years had multiple
files removed for various reasons. This tends to create confusion
and pointless merge conflicts as we merge with upstream. This brings
back a lot of files we generally don't really care about, to align
us with upstream. See below for discussions on the various files:

* adminpack is already imported into Greenplum core in order to
  better support pgAdmin3
* earthdistance isn't supported as it relies on upstream features
  not present in Greenplum
* lo is not supported in Greenplum as large objects are disabled
* chkpass and intagg are uninteresting and obsolete modules which
  are being removed from upstream

We found that if gp_segment_configuration is locked, then it will
fail by triggering FTS. We got the stack below
	#2  0x0000000000a6bb29 in ExceptionalCondition at assert.c:66
	#3  0x0000000000aac19a in enable_timeout timeout.c:143
	#4  0x0000000000aacb6c in enable_timeout_after timeout.c:473
	#5  0x00000000008e86ef in ProcSleep at proc.c:1300
	#6  0x00000000008deb70 in WaitOnLock at lock.c:1894
	#7  0x00000000008e019e in LockAcquireExtended at lock.c:1205
	#8  0x00000000008dd2d8 in LockRelationOid at lmgr.c:102
	#9  0x000000000051c928 in heap_open at heapam.c:1083
	#10 0x0000000000b7feaf in getCdbComponentInfo at cdbutil.c:173
	#11 0x0000000000b81365 in cdbcomponent_getCdbComponents at cdbutil.c:606
	#12 0x00000000007603e1 in ftsMain at fts.c:351
	#13 0x0000000000760715 in ftsprobe_start at fts.c:121
	#14 0x00000000004cc7b0 in ServerLoop ()
	#15 0x00000000008769bf in PostmasterMain at postmaster.c:1531
	#16 0x000000000079098b in main ()
So it is that FTS hasn't initialized timeout. Any process that
wants to use timeout must call initilization first. This is the
root cause gpexpand job fails on master pipeline in build 71 and 79.
We added this initialization in FTS and GDD.