Previously was using master.
Co-authored-by: NOliver Albertini <oalbertini@pivotal.io>
Co-authored-by: NBradford D. Boyle <bboyle@pivotal.io>

Make "demo_cluster" the default tag (ie: no tag). That is, when a
scenario does not specify a behave tag it uses the demo cluster
environment to run. Use the concourse_cluster tag if a scenario requires
a remote cluster. Concourse runs the scenarios with no tag that use a
demo cluster by filtering for --tags=~concourse_cluster.
Authored-by: NKalen Krempely <kkrempely@pivotal.io>
(cherry picked from commit 40028cf4)

Regenerate pipeline from CLI_BEHAVE_TESTS refactor.

(cherry picked from commit 5dc3bfaa)

* Remove unused IVYREPO_ variables for CentOS 6/7

Ivy is no longer used for CentOS 6/7.
Authored-by: NAmil Khanzada <akhanzada@pivotal.io>

* Remove unused IVYREPO_ variables for Coverity

Ivy is no longer used for CentOS 6/7 and will go away.
The coverity pipeline is also puased at this point in time, but we'll
clean this up now.

(cherry-picked from c4c99bb0)
Co-authored-by: NBradford D. Boyle <bboyle@pivotal.io>
Co-authored-by: NAmil Khanzada <akhanzada@pivotal.io>

* Fix bin_gpdb_centos6_icw_green versioned_file to store artifacts in gpdb_branch_6X_STABLE
* Generate pipeline

For GP 6 beta, the release engineering team is removing the apr-util
package from the list of bundled dependencies. Users will be asked
to provide their own apr-util package, which can differ in the version
on each platform. So it is necessary to verify gpperfmon is workable
with the platform provided apr-util on each supported platform.

Originally, gpperfmon test is done in CLI test suite, only covers centos6.
This commit changed it to a dedicated suite in order to test multiple
platforms.
Note: on SLES12 does not need to install libapr-util1 to run gpmmon

We add gpmovemirrors tests and add them to the pipeline as well.
Using behave tags, some of the tests can run locally on a demo
cluster.  We also introduce Behave tag logic to choose which
tests to run.

<Co-Authored-By> Kalen Krempely <kkrempely@pivotal.io>
<Co-Authored-By> Mark Sliva <msliva@pivotal.io>
(cherry-picked from fdf8b611)

This job is used to test online expand, it will create a cluster
with two segments, then expand to 3 and run all ICW to check
whether the cluster is OK after expansion.
As restart is forbidden in online expand test, so we exclude all
the case which contains restart opertaion. But if someone add a
new test with restart, the job may fail. Manual intervention to
exclude the test is needed.
So we move this job to our own dev pipeline to reduce the impact
on prod pipeline.

  * Add configure_flags_with_extensions to the pipeline template file
  * regenerate the gpdb_master pipeline and use fly to set it up.
Co-authored-by: NBob Bao <bbao@pivotal.io>
Co-authored-by: NNing Fu <nfu@pivotal.io>

The gp-addon repo was used to provide the quick-lz artifacts but this
has been moved in to the Docker images used for building GPDB6.
Co-authored-by: NSambitesh Dash <sdash@pivotal.io>
Co-authored-by: NBradford D. Boyle <bboyle@pivotal.io>

- add a rename task to make the artifacts use the new naming convention
- keep pushing to the s3 resources so that other pipelines using those
  published artifacts don't break (although those s3 resources are now
  deprecated).
Co-authored-by: NSambitesh Dash <sdash@pivotal.io>
Co-authored-by: NAmil Khanzada <akhanzada@pivotal.io>
Co-authored-by: NBen Christel <bchristel@pivotal.io>

This commit moves all of the intermediate concourse resources (which are
only used internally in the pipeline) to now use GCP buckets as opposed
to AWS S3 storage.

The ((pipeline-name)) variable added to gen_pipeline.py is needed for
supporting the naming conventions that are used for new GCS objects,
which namespace artifacts by pipeline.
Co-authored-by: NBen Christel <bchristel@pivotal.io>
Co-authored-by: NAmil Khanzada <akhanzada@pivotal.io>
Co-authored-by: NSambitesh Dash <sdash@pivotal.io>
Co-authored-by: NOliver Albertini <oalbertini@pivotal.io>

This reverts commit 8758e2bf.

This commit moves all of the intermediate concourse resources (which are
only used internally in the pipeline) to now use GCP buckets as opposed
to AWS S3 storage.
Co-authored-by: NBen Christel <bchristel@pivotal.io>
Co-authored-by: NAmil Khanzada <akhanzada@pivotal.io>
Co-authored-by: NSambitesh Dash <sdash@pivotal.io>
Co-authored-by: NOliver Albertini <oalbertini@pivotal.io>

Co-authored-by: NSambitesh Dash <sdash@pivotal.io>
Co-authored-by: NAmil Khanzada <akhanzada@pivotal.io>

We now pull down the libsigar artifacts (.targz) from GCS for Centos
instead of ivy.

After removing all the ivy dependencies for Centos{6,7}, `make
sync_tools` no longer creates {GPDB_SRC}/gpAux/ext/rhel{6,7}_x86_64
in the concourse instance. This commit creates the necessary directories
for Centos{6,7} platform.
Co-authored-by: NNandish Jayaram <njayaram@pivotal.io>
Co-authored-by: NSambitesh Dash <sdash@pivotal.io>
Signed-off-by: NSambitesh Dash <sdash@pivotal.io>

gphdfs is being deprecated for GPDB 6. This commit removes the
following:
- Remove gphdfs from pipelines
- Remove gphdfs from gpcontrib
- Remove gphdfs from Makefile
- Remove tests using gphdfs
- Remove GUCs, code, and comments used for gphdfs
- Remove gphdfs from GPDB docs
- Remove avro and parquet formats

We are renaming the docker images we use with concourse to have the gpdb
version in the actual image name rather than in the tag. The contents of
the images will be the same as before.
Authored-by: NKaren Huddleston <khuddleston@pivotal.io>

Co-authored-by: NKaren Huddleston <khuddleston@pivotal.io>
Co-authored-by: NNandish Jayaram <njayaram@pivotal.io>

This reverts commit 39dfead3
and reapplies commit 39dfead3.

The original commit was reverted due to a test failure in gpcloud, but
the failing test has been disabled.
Co-authored-by: NAmil Khanzada <akhanzada@pivotal.io>
Co-authored-by: NKaren Huddleston <khuddleston@pivotal.io>

- update pipeline to point to the new Ubuntu 16.04 image
- the Docker image tag in use was stale and no longer built. Update to the
new, supported, tag.
Co-authored-by: NJacob Champion <pchampion@pivotal.io>
Co-authored-by: NJamie McAtamney <jmcatamney@pivotal.io>
Co-authored-by: NJesse Zhang <jzhang@pivotal.io>
Co-authored-by: NJim Doty <jdoty@pivotal.io>

If Global Deadlock Detector is enabled, then the table lock may
downgrade to RowExclusiveLock, It may lead two problems:

1. When updating distributed keys concurrently, SplitUpdate node
   would generate more tuples in the table.
2. When updating concurrently, it may trigger the EvalPlanQual
   function, when the SubPlan has Motion node, it can not execute
   correctly.

Now we add a GUC for GDD, if it is disabled, we execute these
UPDATE statement serially, if it is enabled, we raise an error when
updating concurrently.

Co-authored-by: Zhenghua Lyu zlv@pivotal.io

This reverts commit 05611ba6.

We are no longer consuming python from Ivy. We are now building it
ourselves against the version of OpenSSL provided on the OS.
Co-authored-by: NKaren Huddleston <khuddleston@pivotal.io>
Co-authored-by: NBen Christel <bchristel@pivotal.io>
Co-authored-by: NDavid Sharp <dsharp@pivotal.io>

Acceptance tests for gpstate were lacking, making it difficult to prove
that the previous commit didn't break anything. Start remedying that
here -- these tests are not exhaustive by any means, but they're a good
start.

The new Concourse job is called "gpstate".

The new (or modified) behave steps are as follows:

  - "user kills all {type} processes [with {signal}]"

    This step previously worked only with primary processes, and would
    only send SIGKILL. It now allows either primary or mirror to be
    specified, along with the specific signal to be sent. If no signal
    is specified, SIGTERM will be sent.

  - "a standard local demo cluster is created"

    This step creates a standard 3 content demo cluster with mirrors and
    master standby.

  - "a standard local demo cluster is running"

    This step checks that a standard demo cluster as defined above
    is currently running. If it is not, it creates one. This leads to a
    speed improvement in the tests that use this step.
Co-authored-by: NMark Sliva <msliva@pivotal.io>
Co-authored-by: NJamie McAtamney <jmcatamney@pivotal.io>

Previously added these to the task, but missed adding them to the
pipeline.
Co-authored-by: NBradford D. Boyle <bboyle@pivotal.io>
Co-authored-by: NDavid Sharp <dsharp@pivotal.io>

Co-authored-by: NDavid Sharp <dsharp@pivotal.io>
Co-authored-by: NBradford D. Boyle <bboyle@pivotal.io>
Co-authored-by: NBen Christel <bchristel@pivotal.io>

gptransfer is replaced by gpcopy.

Note: gpdb-doc/ is not updated yet.

The CLI group had two `compile_gpdb_centos6` jobs.

icw_gporca_centos6 job generates the icw_gporca_centos6_dump. gpexpand
has icw_gporca_centos6_dump as input, hence make it just depend on
that particular job instead of all the ICW jobs. This makes the
gpexpand job same as pg_upgrade job. Also, importantly marks the
real dependency instead of perceived one.

This also removes the last remaining walrep job from pipeline file
using tinc. Those tests are anyways broken and can't be run. Plan is
to port some of the relevant to regress or behave.

For GPDB 6 Beta centos 6/7 need to be passing for the same commit
to be a valid release candidate.
Co-authored-by: NKris Macoskey <kmacoskey@pivotal.io>

- Assumes single node setup for CI. The test generates its own cluster.

The AIX server in not available, so we remove the test job to
release candidate. This is temp change until the server is available.

These tests were run in separate job before as needed cluster to be created for
them with no mirrors. These could run only without mirrors due to limitation of
allowing only one walsender-walreceiver connection (max_walsender=1). But now
that restriction no more exists, can run these tests similar to any other tests
as part of regular ICW even if cluster is created with mirror.

Co-authored-by: NKaren Huddleston <khuddleston@pivotal.io>
Co-authored-by: NBradford Boyle <bboyle@pivotal.io>

The concourse container for terraform tests is only used to ssh to the
terraform cluster so we can use a lighter weight image.

Note: regression_tests_gphdfs_mapr_centos does not use the ccp image
because it runs tests and GPDB from the Concourse container, and uses a
terraform host to run hadoop/mapr
Co-authored-by: NKaren Huddleston <khuddleston@pivotal.io>
Co-authored-by: NDavid Sharp <dsharp@pivotal.io>

Co-authored-by: NDavid Sharp <dsharp@pivotal.io>
Co-authored-by: NBen Christel <bchristel@pivotal.io>

and centos6-build for the OSS build job. The image for this job was left
unmodified when we previously introduced the centos6-build image for
enterprise compile jobs. We changed the OSS job to also use the
centos6-build image so we could remove the centos-gpdb-test-6 image.

We also renamed the image resources to
centos<version>-<test,build>-gpdb6, to clarify what they are
building/testing. Docker resource names should match the repository and
tag of the docker image.
Co-authored-by: NDavid Sharp <dsharp@pivotal.io>
Co-authored-by: NBen Christel <bchristel@pivotal.io>
Co-authored-by: NKaren Huddleston <khuddleston@pivotal.io>