There were basically two test scenarios in this TINC test:

1. ALTER TABLE SET DISTRIBUTED BY. We have plenty of coverage for that
   in the 'alter_distribution_policy' test already, so remove it.

2. Dropping a column that is part of the distribution key. We had some
   coverage for that too, e.g. n the 'alter_table' test. But that's an
   upstream test, so I'd rather not rely on that staying the same. There
   were a few other tests too, but droppign the distribution key column
   seemed accidental rather than deliberate on those, so add a bespoken
   test case for this in 'alter_distribution_policy'

We don't really need to test compression with every different compression
level. Don't need to test so many different block sizes either.

These tests executed various built-in functions, in different contexts
like in a FROM clause, or in a subquery. We have similar tests for
functions with different attributes in the main test suite's
'qp_functions_in_*' tests. The fact that these are built-in functions
doesn't make any difference to planning, so remove these tests as
redundant.

Signed-off-by: NPeifeng Qiu <pqiu@pivotal.io>

Storage tests are quite slow due to fsync on a busy concourse worker.
This will disable fsync, and should eliminate many timing issues we have
encountered due to the change in the characteristics of the concourse
instance since we migrated to GCP.

The runaway-query test is inconsistent and proper investigation into its
flakiness needs to be done. For the time being, since this feature is
stable and unchanging, we will remove this test job.

* Pipeline for GPORCA build, GPDB compile & ICG w/ ORCA on ubuntu
* gpadmin uses bash by default & configures GSS api
* Use a param `TEST_SUITE` to determine the test-suite being run
* Merge orca and gpdb compilation jobs & run ICW
  - Fix variable names for gpdb with orca jobs and artifacts
  - Adjust pipeline to horizontal layout
* Refactor and print compiler versions
  - Supply C/C++ Compiler settings as arguments to our build scripts
  - Instead of using `update-alternatives` we pass environment arguments to conan install
Signed-off-by: NC.J. Jameson <cjameson@pivotal.io>
Signed-off-by: NLarry Hamel <lhamel@pivotal.io>

Various people working on documentation, either user-facing or READMEs,
have concourse unnecessarily run against their PRs. With this change,
there will be less load on the PR pipeline and PRs with only docs
changes won't get a green or red mark from concourse.

This extends a configuration from the master pipeline now to the PR
pipeline, since it's now supported for the PR resource.

Signed-off-by: NShoaib Lari <slari@pivotal.io>
Signed-off-by: NC.J. Jameson <cjameson@pivotal.io>

Current code has a hard-coded dict of catalog tables and their
attributes. A typo in one of the names surfaced as a bug.

This PR fixes the typo, and this commit adds a test which will fail if
gpcheckcat's list of table names is not consistent with the database.
Signed-off-by: NMarbin Tan <mtan@pivotal.io>
Signed-off-by: NNadeem Ghani <nghani@pivotal.io>

Causes catalog tests to fail as we resort to oid, which does not exist.
Also causes exception in SUMMARY and terminates.

TINC isolation parser fails to deal with lines that have trailing whitespace
after ";".

To validate that a reindex operation indeed created a new index, the tests need
to inspect relfilenode values of the index's entry in pg_class.  Previously,
the GUC gp_select_invisible was turned on to get relfilenode prior to reindex.
Use of the GUC made the tests fickle.  E.g. running an unrelated test that
creates and drops a table with identical name as one of the index names in
reindex test is suffice to make the reindex tests fail.

Found several redundant tests, removed them on the way.

It's only used within vacuum.c, so let's have it there rather than pollute
the headers.

This allows overriding the heuristic on whether a query has an ORDER BY.

Use the directive in one of the queries in the 'gporca' test, which
contains a subquery with an ORDER BY that fools the atmsort's usual
heuristic. The overall order of the query is not well-defined, even though
there is an ORDER BY in the subquery. The current implementation of
DISTINCT in fact always also sorts the output, which is why this test
is passing, but that is about to be relaxed soon, when we merge upstream
commit 63247bec.

If a window function had a PARTITION BY clause, but the planner was able
to deduce that it's a constant at runtime, we would still try to distribute
the rows according to the non-existent hash expression. Creating a hash
locus with no hash expressions tripped an assertion.

Fixes github issues #3423 and #3446. Backpatch to 5X_STABLE.

The job needs to run all steps that communicate with the cluster
using tags that give network access to that account

The new CI environment seems to behave in a peculiar way.  There is a
delay of several seconds between a child process dies with abort() and
the parent process receives a signal notifying child's death.  The
storage tests frequently inject PANIC faults and run select queries in
a loop until success to determine if the postmaster has finished reset
(shutdown followed by startup).  Due to the above mentioned delay, the
tests incrrectly determine that postmaster is up when the postmaster
hasn't even initiated shutdown sequence.  This commit causes a test to
wait for 30 seconds before starting to bombard the postmater with
select queries to determine if it's up.  A proper fix is in the
making, until then, this fix should make the CI green.

Allow GUC optimizer_join_arity_for_associativity_commutativity
to be visible to regular database users.

* docs - resgroup gp_resgroup_memory_policy GUC, show/set limit bypass

* Update workload_mgmt_resgroups.xml

Using a symlink to make the file-renaming smoother on the PR pipeline
was not a good idea... Concourse maybe doesn't support following
symlinks in the `build_plan.task.file` part of a pipeline.yml
Signed-off-by: NC.J. Jameson <cjameson@pivotal.io>

* doc: Warning about add cgroup kernel issue to resource group docs.

* docs: fix typo

* docs: add orca guc optimizer_join_arity_for_associativity_commutativity

will be backported to 5X_STABLE

* fix typo

`make_pathkeys_for_sortclauses` with a `true` last argument promises to
canonicalize the returned path keys. We somehow cargo-culted a few
unnecessary `canonicalize_pathkeys` immediately after those calls.

This commit removes such superfluous calls to `canonicalize_pathkeys`.
Signed-off-by: NMax Yang <myang@pivotal.io>

It took me a while to notice that there are two copies of the "ema" stuff,
and then to notice that they were not, in fact, 100% identical.

Signed-off-by: NKris Macoskey <kmacoskey@pivotal.io>

Now that all "prod" pipeline are running in one GCP concourse
deployment, CCP jobs get run in different AWS accounts based on the
cluster's networking requirements.  For the time being, only DPM jobs
need access to the resources in the Pivotal network (ddboost server, and
the Netbackup server).  The gpdb5-pipeline account can accomidate more
concurrent clusters due to its more expanize IP address space, and so
networking needs aside it will be the AWS account where all clusters
should be launched.
Signed-off-by: NKris Macoskey <kmacoskey@pivotal.io>
Signed-off-by: NJim Doty <jdoty@pivotal.io>

Signed-off-by: NAlexandra Wang <lewang@pivotal.io>

It is necessary to use the image_resource in external task yamls for
pipeline jobs that use a sles based docker image. Currently unknown why
the sles image in particular causes image_resource issues.
Signed-off-by: NAlexandra Wang <lewang@pivotal.io>
Signed-off-by: NKris Macoskey <kmacoskey@pivotal.io>

This is compatible with concourse 3 and 2.7.3 but the pipeline at this
commit can only be run on the gpdb concourse deployment in GCP (it is
the deployment with the external workers).
Signed-off-by: NDivya Bhargov <dbhargov@pivotal.io>
Signed-off-by: NJim Doty <jdoty@pivotal.io>
Signed-off-by: NAlexandra Wang <lewang@pivotal.io>

Boolean variables in concourse 3 do not work with the older {{}}
interpolation syntax. They must use the newer syntax: (())

Therefore the pipeline at this commit is incompatible with concourse
2.7.3
Signed-off-by: NDivya Bhargov <dbhargov@pivotal.io>
Signed-off-by: NKris Macoskey <kmacoskey@pivotal.io>

This bug was introduced in commit 7e268107, which changed the way we
track the "current" ordering in the planner.

the kernel does not provide IO pipe API that force sends data completeness in once invocation.
We need to evaluate whether should go on do IO after once return