Prior to this change it would only run on centos7.
Also pipelines that did not target centos7 would
fail to run.
Co-authored-by: NMark Sliva <msliva@pivotal.io>
Co-authored-by: NAmil Khanzada <akhanzada@pivotal.io>

This reverts commit c46bf52f.

* Adding reference topic and additional information around temp_tablespaces

* Clarify comma-separated list

Fix gen_pipeline.py OS issue

Modify TwoPhaseFileHeader to include a tablespace OID that needs to be
removed if an abort occurs.

We add a 'pending tablespace for deletion' that is scheduled during the
'create tablespace' command. If a node encounters an error, it can abort
using the in-memory value. The master then can send an ABORT PREPARED for all
segments that have prepared the create tablespace.

A successful 'create tablespace' needs to unschedule the deletion, or following
aborted transactions will accidentally delete the newly created tablespace.

XLOG records are written for the ABORT and ABORT_PREPARED of a transaction, and these records also
now contain a tablespace oid for deletion.

Notes:

During CREATE TABLESPACE, if an error occurs after the XLOG_XACT_PREPARE
record has been written, there should be no filesystem changes left
on disk for any of the segments. (master, standby, primary, or mirror)

If an error occurs after the XLOG_XACT_DISTRIBUTED_COMMIT, then the create
tablespace should succeed, and all tablespace filesystem changes should exist.
Co-authored-by: NTaylor Vesely <tvesely@pivotal.io>
Co-authored-by: NAlexandra Wang <lewang@pivotal.io>
Co-authored-by: NSoumyadeep Chakraborty <sochakraborty@pivotal.io>

* Add remote_regress folder in src/bin/gpfdist/
to test gpfdist which runs on a remote host, such
as a windows server. make installcheck_win can
run them.

* The scripts of start_gpfdist_remote_win
are used to start and stop gpfdist runing on windows.
If you want to test gpfdist on another os, such
as aix, you can create a forlder named
start_gpfdist_remote_aix and add some scripts
to start and stop gpfdist.

With GDD enabled, and under some simple cases (refer
the commit 6ebce733 and the function checkCanOptSelectLockingClause
for details), we might also do some optimizations for the
select statement with locking clause and limit clause.

Greenplum generates two-stage-merge sort or limit plan to implement
sort or limit and we can only lock tuples on segments. We prefer
locking more tuples on segments rather than locking the whole table.
Without the whole table lock, performance for OLTP should be improved.

Also, after lockrows data cannot be assumed in order, but we do merge
gather after lockrows. It is reasonable because even for postgres:
`select * from t order by c for update` cannot guarantee result's order.

The task that had been using it was removed
in commit: b9636096

* docs - GPCC no longer depends on gpperfmon_install

* Review comments from David

Issue is that in Postgres 9.5 there is a fixed amount of xl_info flags
that can be used. GPDB commit b5871009 added another flag which in
GPDB 9.5 merge branch exceeds the amount available.

Ashwin noticed the difference between XLOG_XACT_ONE_PHASE_COMMIT and
XLOG_XACT_COMMIT records is if distributed info is set. Instead we can
use a same record type in both cases if we setup distributed info in
RecordTransactionCommit() and then pass the distributed info through
xact_redo().

Now that the root cause for a single failing test on Ubuntu18.04 has
been merged(dc0a3cec), enable that test in the pipeline.

The link to the upstream PostgreSQL documentation had a typo and
was pointing to postgreql.org.

Two empty test files introduced in commit 364cbce7. Must have been a
think-O

This reverts commit 559d9d9f.

The recommended sysctl settings for user clusters are out-of-date, and
after some investigation we've discovered a good minimal set of
recommended defaults.  These will be updated in the documentation, but
we also want to set them in the VMs used for our CLI Behave tests so
that we use values similar to those that users will have in their
environments, so this commit adds a section to the Concourse cluster
task file to do so.
Co-authored-by: NJamie McAtamney <jmcatamney@pivotal.io>
Co-authored-by: NKalen Krempely <kkrempely@pivotal.io>

A Motion node may has a subplan whose targetlist need to be expanded
to make sure all entries of the subplan's hashExpr are in its
targetlist, in this case, the motion node should also update its
targetlist, otherwise, the difference of targetlist will cause
Motion node parse the tuple (especially for MemTuple) incorrectly
with a mismatched tuple description, make wrong result or even crash.

For a subplan, if we can get a Expr for distkey column from the
targetlist, the Expr must be in the targetlist or reference the
Vars in the targetlist, The old logic is, if the Expr is valid,
the Expr that reference Vars is also added to the targetlist.
For example, a subplan has targetlist (c1) and the locus whose
distkey is (c1::float8), so the Expr returned will be one
referenced Vars in the targetlist and then the old targetlist
will be extended to (c1, c1::float8). Obviously, the extension
of targetlist here is unnecessary, the Expr can evaluate from
Vars, meanwhile, motion need to transfer more columns in a tuple.
Co-authored-by: NAdam Lee <ali@pivotal.io>
Reviewed-by: NHeikki Linnakangas <hlinnakangas@pivotal.io>
Reviewed-by: NRichard Guo <rguo@pivotal.io>
Reviewed-by: NDaniel Gustafsson <dgustafsson@pivotal.io>

This commit handles a missed case in the previous commit: "Fix
algebrization of subqueries in queries with complex GROUP BYs".

The logic inside RunExtractAggregatesMutator's Var case was intended to
fix top-level Vars inside subqueries in the targetlist, but also
incorrectly fixed top-level Vars in subqueries inside of aggregates.

The gpconfig test was failing for UTF-8 characters when run on Ubuntu because
our test containers use the POSIX locale. In this commit, we have set the
`LC_TYPE` to `en_US.UTF-8` for the gpconfig test so that the test has the same
behavior on all platforms.
Co-authored-by: NJacob Champion <pchampion@pivotal.io>

The `gen_pipeline()` function called the `suggested_git_remote()` and the
`suggested_git_branch()` functions as default values for the `git_remote` and
`git_branch` parameters.  For `prod` pipeline, the `gen_pipeline()` function is
called with GPDB repo and `BASE_BRANCH`.  However, the `suggested_*()` functions
are called in the `gen_pipeline()` function definition resulting in error as
they are not applicable for the production branches.

Therefore, in this commit we have used `None` as the default and call the
`suggested_*()` functions only if the corresponding parameters are not provided
by the caller.
Co-authored-by: NJacob Champion <pchampion@pivotal.io>

* docs - pxf now bundled with postgresql 42.2.5 jar

* remove postgresql jar registration from the example

Authored-by: NTingfang Bao <bbao@pivotal.io>

greenplum-db/greenplum-database-release

Previously, releng team merged PR
https://github.com/greenplum-db/gpdb/pull/7906
to release open source Greenplum-db in gpdb repo, and later on, we
decided to move these work to new repo
greenplum-db/greenplum-database-release, this commit delete the
previouse work
we have done
Reviewed-by: NMark Sliva <msliva@pivotal.io>
Reviewed-by: NAmil Khanzada <akhanzada@pivotal.io>

ORCA commit: Skip group expression containing circular references
Authored-by: NChris Hajas <chajas@pivotal.io>

* Docs: add note that gpcopy inherits encryption from client connection

* Change note to clarify that SSL encryption is not supported

* typo fix

Authored-by: NTingfang Bao <bbao@pivotal.io>

The resource rename is needed in order to fetch the correct version.
Co-authored-by: NBradford D. Boyle <bboyle@pivotal.io>
Co-authored-by: NMark Sliva <msliva@pivotal.io>

Co-authored-by: NBradford D. Boyle <bboyle@pivotal.io>
Co-authored-by: NMark Sliva <msliva@pivotal.io>

Co-authored-by: NBradford D. Boyle <bboyle@pivotal.io>
Co-authored-by: NMark Sliva <msliva@pivotal.io>

We are introducing a new pattern of using BATS and python unit testing
for files inside the concourse/scripts folder.
Currently, there are no unit tests, but this pattern allows them to
easily be added.
Co-authored-by: NMark Sliva <msliva@pivotal.io>
Co-authored-by: NAmil Khanzada <akhanzada@pivotal.io>

The task was originally separating NON_PRODUCTION_FILES.txt out of the
bin_gpdb tarball and putting them into the QAUTILS_TARBALL, and leaving
OUTPUT_TARBALL with the rest of the files.
However, QAUTILS_TARBALL is never used, the only important function is
to remove the NON_PRODUCTION_FILES.txt.

With this commit, we refactor the separate_qautils_files to only remove
files. Also, we introduce a new pattern for TDD'ing bash scripts by
using the MIT-licensed BATS framework.
Co-authored-by: NMark Sliva <msliva@pivotal.io>
Co-authored-by: NAmil Khanzada <akhanzada@pivotal.io>
(cherry-picked from 5458b94b)

OpenSSL will load config file under the path it's being configured.
If the path is writable by non-admin user, a malicious user can
potentially inject code on openssl invocation. Change dependency
path to c:\windows\system32.

Reference: https://curl.haxx.se/docs/CVE-2019-5443.html

In the Behave test for gpinitsystem, testing for database creation in default
timezone was done by setting the TZ variable to an empty string. On Ubuntu this
resulted in the `date +"%Z"` command giving `Universal` as the timezone rather
than `UTC`.

In this commit, the `TZ` variable is unset rather than set to an empty
string. Thus giving uniform behavior on Ubuntu and other platforms. Since we
already have a Behave step to unset a variable correctly, so we are removing
this incorrect step.
Co-authored-by: NJacob Champion <pchampion@pivotal.io>

We were not killing the postmaster processes because of the following reasons.

1. The `$` symbols passed to `Command` was expanded on the local machine due to
the `RemoteExecutionContext` implementation.  This made the awk filter a no-op.

2. We did not catch this on Centos because the Centos implementation of
/bin/kill will kill as many of the processes as are valid and ignoring the
invalid ones.  However, Ubuntu fails fast if any of the arguments are invalid.

3. The test step did not validate the result of `Command.run()`.  Thus ignoring
the failure.

We have replaced `Command` by `subprocess.check_call()`.
Co-authored-by: NJacob Champion <pchampion@pivotal.io>
Co-authored-by: NShoaib Lari <slari@pivotal.io>

- Ensure that indexes are placed into the default tablespace on the master and segments.
- Default tablespace GUC should not affect where a temporary table or its indexes are placed.
- Default tablespace should override explict database tablespace configuration
- ALTER DATABASE SET TABLESPACE should not impact SET default_tablespace to 'some_tablespace'
- Objects created when default_tablespace is set should end up 'some_tablespace'.

   Add the windows clinets testing jobs to pipelines
Authored-by: NTingfang Bao <bbao@pivotal.io>