The `gen_pipeline()` function called the `suggested_git_remote()` and the
`suggested_git_branch()` functions as default values for the `git_remote` and
`git_branch` parameters.  For `prod` pipeline, the `gen_pipeline()` function is
called with GPDB repo and `BASE_BRANCH`.  However, the `suggested_*()` functions
are called in the `gen_pipeline()` function definition resulting in error as
they are not applicable for the production branches.

Therefore, in this commit we have used `None` as the default and call the
`suggested_*()` functions only if the corresponding parameters are not provided
by the caller.
Co-authored-by: NJacob Champion <pchampion@pivotal.io>

* docs - pxf now bundled with postgresql 42.2.5 jar

* remove postgresql jar registration from the example

Authored-by: NTingfang Bao <bbao@pivotal.io>

greenplum-db/greenplum-database-release

Previously, releng team merged PR
https://github.com/greenplum-db/gpdb/pull/7906
to release open source Greenplum-db in gpdb repo, and later on, we
decided to move these work to new repo
greenplum-db/greenplum-database-release, this commit delete the
previouse work
we have done
Reviewed-by: NMark Sliva <msliva@pivotal.io>
Reviewed-by: NAmil Khanzada <akhanzada@pivotal.io>

ORCA commit: Skip group expression containing circular references
Authored-by: NChris Hajas <chajas@pivotal.io>

* Docs: add note that gpcopy inherits encryption from client connection

* Change note to clarify that SSL encryption is not supported

* typo fix

Authored-by: NTingfang Bao <bbao@pivotal.io>

The resource rename is needed in order to fetch the correct version.
Co-authored-by: NBradford D. Boyle <bboyle@pivotal.io>
Co-authored-by: NMark Sliva <msliva@pivotal.io>

Co-authored-by: NBradford D. Boyle <bboyle@pivotal.io>
Co-authored-by: NMark Sliva <msliva@pivotal.io>

Co-authored-by: NBradford D. Boyle <bboyle@pivotal.io>
Co-authored-by: NMark Sliva <msliva@pivotal.io>

We are introducing a new pattern of using BATS and python unit testing
for files inside the concourse/scripts folder.
Currently, there are no unit tests, but this pattern allows them to
easily be added.
Co-authored-by: NMark Sliva <msliva@pivotal.io>
Co-authored-by: NAmil Khanzada <akhanzada@pivotal.io>

The task was originally separating NON_PRODUCTION_FILES.txt out of the
bin_gpdb tarball and putting them into the QAUTILS_TARBALL, and leaving
OUTPUT_TARBALL with the rest of the files.
However, QAUTILS_TARBALL is never used, the only important function is
to remove the NON_PRODUCTION_FILES.txt.

With this commit, we refactor the separate_qautils_files to only remove
files. Also, we introduce a new pattern for TDD'ing bash scripts by
using the MIT-licensed BATS framework.
Co-authored-by: NMark Sliva <msliva@pivotal.io>
Co-authored-by: NAmil Khanzada <akhanzada@pivotal.io>
(cherry-picked from 5458b94b)

OpenSSL will load config file under the path it's being configured.
If the path is writable by non-admin user, a malicious user can
potentially inject code on openssl invocation. Change dependency
path to c:\windows\system32.

Reference: https://curl.haxx.se/docs/CVE-2019-5443.html

In the Behave test for gpinitsystem, testing for database creation in default
timezone was done by setting the TZ variable to an empty string. On Ubuntu this
resulted in the `date +"%Z"` command giving `Universal` as the timezone rather
than `UTC`.

In this commit, the `TZ` variable is unset rather than set to an empty
string. Thus giving uniform behavior on Ubuntu and other platforms. Since we
already have a Behave step to unset a variable correctly, so we are removing
this incorrect step.
Co-authored-by: NJacob Champion <pchampion@pivotal.io>

We were not killing the postmaster processes because of the following reasons.

1. The `$` symbols passed to `Command` was expanded on the local machine due to
the `RemoteExecutionContext` implementation.  This made the awk filter a no-op.

2. We did not catch this on Centos because the Centos implementation of
/bin/kill will kill as many of the processes as are valid and ignoring the
invalid ones.  However, Ubuntu fails fast if any of the arguments are invalid.

3. The test step did not validate the result of `Command.run()`.  Thus ignoring
the failure.

We have replaced `Command` by `subprocess.check_call()`.
Co-authored-by: NJacob Champion <pchampion@pivotal.io>
Co-authored-by: NShoaib Lari <slari@pivotal.io>

- Ensure that indexes are placed into the default tablespace on the master and segments.
- Default tablespace GUC should not affect where a temporary table or its indexes are placed.
- Default tablespace should override explict database tablespace configuration
- ALTER DATABASE SET TABLESPACE should not impact SET default_tablespace to 'some_tablespace'
- Objects created when default_tablespace is set should end up 'some_tablespace'.

   Add the windows clinets testing jobs to pipelines
Authored-by: NTingfang Bao <bbao@pivotal.io>

1) Run TEST_REMOTE.py of gpload
2) Run psql SSL, gpfdist Windows pipe testing
Co-authored-by: NPeifeng Qiu <pqiu@pivotal.io>
Co-authored-by: NTingfang Bao <bbao@pivotal.io>
Co-authored-by: NXiaoran Wang <xiwang@pivotal.io>
Co-authored-by: NXiaodong Huo <xhuo@pivotal.io>

It's introduced in 'e00098', it happens in such case:

Tx 1: generate xid 1000 on segment 0 (xmin=1000, globalxmin=1000)
Tx 2: generate xid 2000 on segment 0 (xmin=1000, globalxmin=1000)
Tx 1: finish
Tx 3: generate xid 3000 on segment 0 (xmin=2000, globalxmin=1000)
Tx 2: finish
Tx 4: generate xid 4000 on segment 0 (xmin=3000, globalxmin=2000)

In this scenario, tx4 can advance the DistributedLogShared->oldestXmin to 2000,
tx3 should use its globalxmin 1000 instead of the value of
DistributedLogShared->oldestXmin.

Improve PciIntervalFromScalarBoolOr() to use faster algorithm
Only allow const expr evaluation for (const cmp const) ignoring casts
Remove FDisjointLeft check in CRange::PrngExtend()
Co-authored-by: NChris Hajas <chajas@pivotal.io>
Co-authored-by: NShreedhar Hardikar <shardikar@pivotal.io>

It is superseded by NON_PRODUCTION_FILES.txt
Co-authored-by: NMark Sliva <msliva@pivotal.io>
Co-authored-by: NAmil Khanzada <akhanzada@pivotal.io>

Prior to this commit, standard connections made to a primary segment
would hang forever if that segment was not started in utility mode,
because the call to cdb_setup() waits for a state that will never
arrive. The hang exists in 6X as well, for slightly different reasons
(an infinite wait on a semaphore).

Some utilities make use of pg_isready to verify segment status, and a
hung backend will report an incorrect status for the segment, which is
how this bug was discovered. The hung backend sticks around forever in
startup state as well.

Add an explicit check on QE segments that the incoming session does not
expect a dispatcher role, and bail out at the same place that we would
normally complain about a mismatched utility-mode connection. (If the
incoming session is set to the executor role, we'll raise a FATAL for
other reasons, but at least we won't hang.)

Add a regress test for the new behavior. We coopt the new
internal_connection test file for this, and rename it gp_connections.
We also add a more end-to-end regression test using the same
reproduction case as the original bug.
Co-authored-by: NJacob Champion <pchampion@pivotal.io>

Buffer corruption was caused by not correctly terminating the
readlink(), as of commit ccfa3ab7. We restored a previous line of code
that allows correct termination.
Co-authored-by: NJacob Champion <pchampion@pivotal.io>

So that machines with the same user do not accidentally clobber
existing pipeline.
Also remove the -b option because it is now default and rework some
logic.
Co-authored-by: NMark Sliva <msliva@pivotal.io>
Co-authored-by: NAmil Khanzada <akhanzada@pivotal.io>

* gpmovemirrors change input separator from colon to |

* gpaddmirrors change input separator from colon to |, remove mirror string

* gprecoverseg change input separator from colon to |

* gpexpand change input separator

* fix from Chuck

* docs - add diskquota module to reference guide

* Add missing quote; minor edits

* Make module doc more consistent with other module docs.

* Updates from reviews

* Add section about shared memory and diskquota.max_active_tables

* docs - add info about configing hive access via jdbc

* add About to title

* remove some nesting, misc edits from review

* better table column names? clearer Authenticated User values

* edits requested by alex, adjust heading levels

If a transaction has only updated one QE, we can do one-phase commit
there.

If one-phase commit transactions don't write pg_distributedlog, the tuples'
visibility will be checked only with the local snapshot. This will result in an 
incorrect result in repeatable read isolation level.

For example:
create table t(a int);
tx 1: BEGIN ISOLATION LEVEL REPEATABLE READ;
tx 2: insert into t values(1);
tx 1: select * from t where a = 1;
tx 2: insert into t values(1);
tx 2: insert into t values(2);
tx 1: select * from t;

The first SELECT of tx1 will create a distributed snapshot on QD and a local
snapshot on segment 1, and the later SELECT of tx1 will create a local snapshot
on segment 2. In this way, the later SELECT sees the first and the third tuple
but not the second one.
Co-authored-by: NHeikki Linnakangas <hlinnakangas@pivotal.io>
Co-authored-by: NHubert Zhang <hzhang@pivotal.io>
Co-authored-by: NGang Xiong <gxiong@pivotal.io>

The code doesn't call ExecCopySlotMemTupleTo() as the comments was
speaking, and the two times of memtuple_form_to() are confusing, at
least to me, update it.

This backports upstream commit bd1693e8,
slightly modified to resolve the comments conflict.

	Author: Tom Lane <tgl@sss.pgh.pa.us>
	Date:   Thu Jun 23 10:55:59 2016 -0400

	    Fix small memory leak in partial-aggregate deserialization functions.

	    A deserialize function's result is short-lived data during partial
	    aggregation, since we're just going to pass it to the combine function
	    and then it's of no use anymore.  However, the built-in deserialize
	    functions allocated their results in the aggregate state context,
	    resulting in a query-lifespan memory leak.  It's probably not possible for
	    this to amount to anything much at present, since the number of leaked
	    results would only be the number of worker processes.  But it might become
	    a problem in future.  To fix, don't use the same convenience subroutine for
	    setting up results that the aggregate transition functions use.

	    David Rowley

	    Report: <10050.1466637736@sss.pgh.pa.us>

Test does not include tablespace drop.

Commit a993ef03 inadvertently introduced gp_role=utility in the
postmaster arguments when starting segments via gpsegstart. This is
because the sixth argument to SegmentStart is a boolean (utilityMode,
which defaults to False) but that commit incorrectly passes the
master_checksum_version instead. If checksums are enabled on master
(which sets the checksum version to 1, a truthy value), this code will
enable utility mode for the started segments.

SegmentStart does not take a checksum version argument, so remove it
entirely.

Ashwin noticed this in #6334, but we hadn't put two and two together
until now. An ideal followup would be to remove the huge number of
layers that make mistakes like this so easy to miss.
Co-authored-by: NJamie McAtamney <jmcatamney@pivotal.io>

We skip the following tests for the following reasons:
   - gpinitsystem: TZ interpretation difference for TZ '' vs unset
   - gppkg: underlying package zip file contains wrong OS version
   - gpactivatestandby: cause unknown
   - gprecoverseg: bash pipe interpretation difference
   - gpconfig: demo cluster has LOCALE set to POSIX, not UTF-8

We plan to fix these in a subsequent commit.
Co-authored-by: NDavid Krieger <dkrieger@pivotal.io>
Co-authored-by: NNikolaos Kalampalikis <nkalampalikis@pivotal.io>

We have added Ubuntu 18.04 support for the behave CLI tests.
Co-authored-by: NJacob Champion <pchampion@pivotal.io>
Co-authored-by: NDavid Krieger <dkrieger@pivotal.io>
Co-authored-by: NNikolaos Kalampalikis <nkalampalikis@pivotal.io>

This is a followup to commit 89f01461 - "CLI code coverage: add coverage
to the production pipeline". This updates the master-generated pipeline
now that gpperfmon has been updated to collect and upload coverage.

* docs - add resgroup best prac section addressing low swap

* edits requested by david