* docs: add orca guc optimizer_join_arity_for_associativity_commutativity

will be backported to 5X_STABLE

* fix typo

`make_pathkeys_for_sortclauses` with a `true` last argument promises to
canonicalize the returned path keys. We somehow cargo-culted a few
unnecessary `canonicalize_pathkeys` immediately after those calls.

This commit removes such superfluous calls to `canonicalize_pathkeys`.
Signed-off-by: NMax Yang <myang@pivotal.io>

It took me a while to notice that there are two copies of the "ema" stuff,
and then to notice that they were not, in fact, 100% identical.

Signed-off-by: NKris Macoskey <kmacoskey@pivotal.io>

Now that all "prod" pipeline are running in one GCP concourse
deployment, CCP jobs get run in different AWS accounts based on the
cluster's networking requirements.  For the time being, only DPM jobs
need access to the resources in the Pivotal network (ddboost server, and
the Netbackup server).  The gpdb5-pipeline account can accomidate more
concurrent clusters due to its more expanize IP address space, and so
networking needs aside it will be the AWS account where all clusters
should be launched.
Signed-off-by: NKris Macoskey <kmacoskey@pivotal.io>
Signed-off-by: NJim Doty <jdoty@pivotal.io>

Signed-off-by: NAlexandra Wang <lewang@pivotal.io>

It is necessary to use the image_resource in external task yamls for
pipeline jobs that use a sles based docker image. Currently unknown why
the sles image in particular causes image_resource issues.
Signed-off-by: NAlexandra Wang <lewang@pivotal.io>
Signed-off-by: NKris Macoskey <kmacoskey@pivotal.io>

This is compatible with concourse 3 and 2.7.3 but the pipeline at this
commit can only be run on the gpdb concourse deployment in GCP (it is
the deployment with the external workers).
Signed-off-by: NDivya Bhargov <dbhargov@pivotal.io>
Signed-off-by: NJim Doty <jdoty@pivotal.io>
Signed-off-by: NAlexandra Wang <lewang@pivotal.io>

Boolean variables in concourse 3 do not work with the older {{}}
interpolation syntax. They must use the newer syntax: (())

Therefore the pipeline at this commit is incompatible with concourse
2.7.3
Signed-off-by: NDivya Bhargov <dbhargov@pivotal.io>
Signed-off-by: NKris Macoskey <kmacoskey@pivotal.io>

This bug was introduced in commit 7e268107, which changed the way we
track the "current" ordering in the planner.

the kernel does not provide IO pipe API that force sends data completeness in once invocation.
We need to evaluate whether should go on do IO after once return

- Use gcc-6 by default for gpdb_ubuntu build
Signed-off-by: NGoutam Tadi <gtadi@pivotal.io>

On Ubuntu, env vars are not inherited during `su gpadmin -c "commands"`
in the same way as Centos. The PATH was missing the gpdb install dir,
and psql commands failed.
Signed-off-by: NLarry Hamel <lhamel@pivotal.io>

Signed-off-by: NC.J. Jameson <cjameson@pivotal.io>

Currently /usr/sbin/sshd gives out the following error:

```
Missing privilege separation directory: /var/run/sshd
```
Signed-off-by: NGoutam Tadi <gtadi@pivotal.io>

- We receive compiled GPDB tar output through a shared dir
- Separate Compile and ICW jobs for ubuntu
- Increase number of parallel client connections
Signed-off-by: NC.J. Jameson <cjameson@pivotal.io>

To support Ubuntu, we saw this error:
```
make -C contrib/pg_upgrade check

/bin/sh test_gpdb.sh -C -s -o ../../gpAux/gpdemo/datadirs/ -b /usr/local/gpdb/bin

test_gpdb.sh: 38: test_gpdb.sh: [[: not found
test_gpdb.sh: 68: test_gpdb.sh: pushd: not found
test_gpdb.sh: 69: test_gpdb.sh: time: not found
```

There is an important difference between /bin/sh and /bin/bash in
Ubuntu, so we force use of bash here. This commit assumes that /bin is
in path, such that /bin/bash is found.
Signed-off-by: NGoutam Tadi <gtadi@pivotal.io>
Signed-off-by: NLarry Hamel <lhamel@pivotal.io>

Signed-off-by: NC.J. Jameson <cjameson@pivotal.io>

Best to do this workaround with the gpadmin user setup because that's
what it's needed for
Signed-off-by: NGoutam Tadi <gtadi@pivotal.io>

The gp_persistent_relation_node_check() UDF implementation used pg_atoi() to
convert relefile nodes to OIDs. This overflows a signed integer value
when the relfile nodes exceed 2^31 (2 billion plus).

We have changed this to use strtoul() instead

We rely on oidin_subr() to cover null and string overflow from
strToRelfilenode()

Add unittest for gp_strtoul and gp_persistent_relation_node_check

Add test coverage for converting string to unsigned value.
Previously, we were converting with pg_atoi, signed int, which failed on
values greater than equal to 2^31.

Add cmockery test for gp_persistent_relation_node_check that will hit
the conversion path.
Add a functional test to call the conversion directly.
Signed-off-by: NLarry Hamel <lhamel@pivotal.io>
Signed-off-by: NNadeem Ghani <nghani@pivotal.io>
Signed-off-by: NC.J. Jameson <cjameson@pivotal.io>
Signed-off-by: NMarbin Tan <mtan@pivotal.io>

This was broken by commit 7e268107, which refactored the code that deals
with path keys and sorts in plangroupext.c. The new function
make_sort_from_pathkeys_and_groupingcol(), which replaced the old
make_sort_from_reordered_groupcols() function, didn't work quite the same
as the old function. I'm not sure what exactly went wrong there, but the
caller already has the column number and operator information at hand, so
we can use it to construct the Sort directly, without trying to re-find the
original target list entries of the sort columns.

Commit 7e268107 also neglected the comments in
make_sort_from_pathkeys_and_groupingcol(), but this commit removes the whole
function.

Fixes github issue #3447.

copy from on segment will not check distributing policy
when the table is distributed randomly.
Signed-off-by: NXiaoran Wang <xiwang@pivotal.io>

The GPDB5 deployment in the gpdb5-pipeline AWS account has a week of
instability, and to the best of our knowledge this appears to be due to
the "fan out" pattern that is manifested in our pipeline.  After the
compile steps are completed, many, many jobs are kicked off
simultaneaously.  This structure, combined with the many copies of the
master pipeline cause such a large simulaneous workload that concourse
resources like volume creation can not keep up with the demand.

We do not contend that this structure is the correct structure, and we
welcome collaboration on what jobs should live where.  Hopefully this
will keep the deployment humming for the near future.
Signed-off-by: NDivya Bhargov <dbhargov@pivotal.io>

* docs: fixes to ALTER EXTERNAL TABLE

see https://github.com/greenplum-db/gpdb/issues/3338

* updates based on review comments.
--updated command syntax
--reorganized information
--added parameter definition USING expression

It hadn't been kept up-to-date lately, and no-one has complained, so
apparently it's not needed.

Let's try to keep the upstream test files as unmodified as possible, to
make merging easier.

Both 'filter' and 'qp_olap_windowerr' used a test table named
'filter_test'. Rename the one in 'qp_olap_windowerr' to avoid confusion.

We have two implementations of tuplesort: the "regular" one inherited
from upstream, in tuplesort.c, and a GPDB-specific tuplesort_mk.c. We had
modified all the callers to check the gp_enable_mk_sort GUC, and deal with
both of them. However, that makes merging with upstream difficult, and
litters the code with the boilerplate to check the GUC and call one of
the two implementations.

Simplify the callers, by providing a single API that hides the two
implementations from the rest of the system. The API is the tuplesort_*
functions, as in upstream. This requires some preprocessor trickery,
so that tuplesort.c can use the tuplesort_* function names as is, but in
the rest of the codebase, calling tuplesort_*() will call a "switcheroo"
function that decides which implementation to actually call. While this
is more lines of code overall, it keeps all the ugliness confined in
tuplesort.h, not littered throughout the codebase.

Due to 39e726819f75e13, a resourc group is not assigned for
SET/SHOW command which means GUC memory_spill_ratio will
stay unchanged even you set current role to another role.

SET/SHOW commands are not meant to use memory_spill_ratio,
so it should be safe to keep memory_spill_ratio until a non
SET/SHOW command is executed.

SET or SHOW GUC won't take significant resource including cpu and
memory, we can safely let it run. It's useful that not be blocked
when changing from a busy group immediately to another group. For
later integration with WLM, it's also helpful.

We only check the type of current statement when there are no free
slots in the current group, so the overhead of parsing will not
affect normal procedure.

These tests were derived from the resqueue tests, however they did not
work stable enough with resgroup. We'll disable them for now to make the
pipeline green. Later we'll refactor and reenable them.