* docs - qualify some resource-queue specific content (part 1)

* explicitly state resource groups do not use gp_vmem_protect_limit

* RQ/RG qualify some gucs and system tables/views

* qualify gp_toolkit RQ content

* add RG segment memory calculation

* clarify resgroup perseg memory based on active primary segs on host

* remove max_resource_groups guc again

* Several small fixes of the tests

1, ignore two generated test files.
2, remove the string containing unpredictable segment numbers.
3, drop tables in external_table case, so we could run multiple times of it once.

* Fix cases which are unpredictable

> commit 3bbedbe9
> Author: Heikki Linnakangas <hlinnakangas@pivotal.io>
> Date:   Thu Nov 2 10:04:58 2017 +0200
>
>     Wake up faster, if a segment returns an error.
>     Previously, if a segment reported an error after starting up the
>     interconnect, it would take up to 250 ms for the main thread in the QD
>     process to wake up and poll the dispatcher connections, and to see that
>     there was an error. Shorten that time, by waking up immediately if the
>     QD->QE libpq socket becomes readable while we're waiting for data to
>     arrive in a Motion node.
>     This isn't a complete solution, because this will only wake up if one
>     arbitrarily chosen connection becomes readable, and we still rely on
>     polling for the others. But this greatly speeds up many common scenarios.
>     In particular, the "qp_functions_in_select" test now runs in under 5 s
>     on my laptop, when it took about 60 seconds before.

> Before this commit, the master would only check every 250 ms if one of the
> segments had reported an error. Now it wakes up and cancels the whole query as
> soon as it receives an error from the first segment. That makes it more likely
> that the other segments have not yet reached the same number of errors as what
> is memorized in the expected output.

These two cases check:

1, when selecting from a cte fails, one of the external table of the cte
reached the error limit, how many errors happened in the other external
table of the cte, which would not reached the limit.

2, when selecting from an external table with two locations mapped to
two segments each, one segment reached the reject limit, the other also
reached the same.

We could not predict these two results without special test files, even
without that commit actually. This commit removes the cte case and
checks at least one segment failed in case readable_query26.

This reverts commit 02301627.

resource group test resgroup/resgroup_alter_memory continues to fail.
Reverting this back to last known green.

This small batch of commits contains the changes to support hashing for
SELECT DISTINCT queries, and UNION/INTERSECT/EXCEPT.

This reverts commit c3d8a92e.

The mpp_resrouce_group tests on centos 6 & 7 were failing on the
isolation2/resgroup_alter_memory tests

add_slice_to_motion doesn't use the sortPathKeys argument for anything, so
let's remove it.

Previously, to speed up dispatching, cdbdisp_dispatchToGang_async
and cdbdisp_waitDispatchFinish_async are designed to use nonblock
flush to dispatch commands in bulk, however, risks exist that some
commands are not fully dispatched in corner error cases, so QD must
do a force flush before handling such connections, otherwise QD will
get stuck.

* Do UnassignResGroup within prepareTransaction too, prepareTransaction()
will put QE out of any transactions temporarily until the second commit
command arrives, so any failures in this gap will cause leaks of resource
group including slots etc.
* Clean code, move UnassignResGroup() into AtEOXact_ResGroup() so resource
group related codes will not spread across prepare/commit/abort functions.
* Do not call callback functions in PrepareTransaction because the transaction
is not trully commited.

This test was simply large, and also took quite a long time to run, so it's
nice to split it up. Furthermore, only some of the tests produce different
output with ORCA. Split the test so that the tests that use EXPLAIN, or
produce different output with ORCA for some other reason, go to a new
'bfv_partition_plans' test, and the rest remain in 'bfv_partition'.

In the resgroup concourse we use bash here doc to execute commands on
remote server, but once a ssh command is executed the preceding commands
are all ignored. This is because the here doc is hijacked or consumed by
the inner ssh.

Fixed by redirect inner ssh's stdin from /dev/null.

The installcheck-resgroup (ICR) error code was replaced with the diff
watcher's error code. Now pass it to the shell correctly.

Signed-off-by: NAshwin Agrawal <aagrawal@pivotal.io>

Moved lot of filerep code to ftsprobefilerep.c, so now ftsprobe.c becomes very
specific to walrep.

Also, along the way removed calling `probePollOut()` and `probePollIn()` from
`probeSegmentHelper()` as its not needed.

Unit test check_superuser_connection_limit()
Signed-off-by: NXin Zhang <xzhang@pivotal.io>

Case 1: FTS probe handler hang
Case 2: FTS probe handler FATAL and got killed
Case 3: FTS probe handler ERROR out
Signed-off-by: NXin Zhang <xzhang@pivotal.io>

Use special connection parameter "gpconntype" to distinguish a normal libpq
connection from a connection initiated by FTS prober process for probing a
primary segment. It's similar to the "replication=true" for the
WalSender process. Currently we support `gpconntype="fts"`, this can be
extended to support other backend connection types.

Currently, for the FTS handler backend, we support `FTS_MSG_TYPE_PROBE` message type.
The rest of the message handling and process creation is similar to
WalSender process.
Signed-off-by: NAshwin Agrawal <aagrawal@pivotal.io>
Signed-off-by: NAsim R P <apraveen@pivotal.io>

Previous ERROR was causing confusion in the logs and also resulted in a
user-facing error.

Also support start master in utility_mode properly to avoid FTS getting
started under master-only utility mode, which gave the following error
message:

```
"ERROR","22000","Greenplum Database number of segments inconsistency: count is 1 from pg_catalog.gp_id table, but 3 from getCdbComponentDatabases()",,,,,,,0,,"cdbutil.c",340,
```
Signed-off-by: NXin Zhang <xzhang@pivotal.io>
Signed-off-by: NAsim R P <apraveen@pivotal.io>

* Remove the double-link list for DDL callback function 
* Retire ResGroupOpts.
* Remove CommandCounterIncrement() in AlterResourceGroup as we no longer
query pg_resgroupcapability in the callback function.
* alter memory_spill_ratio don't wakeup slots
* acquire AccessExlusiveLock on pg_resgroupcapability in CreateResGroup
* Change interface of GetResGroupCapabilities
* update test cases
Signed-off-by: NNing Yu <nyu@pivotal.io>

- Use the latest gcc (from /opt/gcc_env.sh).
- Add commands to .bash_profile/.bashrc so that stopping and starting the
  container works as expected.
Signed-off-by: NBen Christel <bchristel@pivotal.io>

* docs: analyzedb add --gen_profile_only

Will be ported to master

* docs: fix typo.

Fail on error instead of continuing on. If the test fails, we will search for
diff files in the TINC directory and print them all out. This will allow us to
see the diff file contents from the Concourse job UI. The trap function is
copied from the run_tinc_test.sh native Concourse TINC test script.

The sceipt was setting the env `export PGDATABASE=gptest` but didn't `createdb
gptest`. So, gpconfig to set the GUC failed with `FATAL: database "gptest" does
not exist`. Flip the order and `createdb gptest` first.

Flow for test permutations was:
0. Runs some pre-SQLs
1. Gets the cluster in desired state
2. pg_switch_xlog() on all the segments
3. Injects suspend fault for desired fault
4. Runs some triggers-SQLs
5. Crashes gpstop -i and start the db
6. 5 times calls pg_switch_xlog()
7. Runs some post-SQLs
8. gpcheckcat and gpcheckmirrorseg

- cluster_states: SYNC, CT, RESYNC
- fault_types: dtm_broadcast_prepare, dtm_broadcast_commit_prepared
- crash_types: gpstop_i

This seems all covered and redundant with cs_pg_twophase_switch_01_12 and
cs_pg_twophase_switch_13_24, as only diference in flow in step 6. But don't
think that makes any difference to what's being tested.

Signed-off-by: NDivya Bhargov <dbhargov@pivotal.io>

Signed-off-by: NDivya Bhargov <dbhargov@pivotal.io>

The default collation order on Mac doesn't appear to use Unicode
collation (which ignores spaces on its first pass). For example:

Unicode Collation Order:
a
 a
  a
b
 b
  b

C Collation Order:
  a
  b
 a
 b
a
b

This causes failures for these tests; it's strange but not something we
want to debug right now. Since we're not testing collation in these
tests, remove the ORDER BY from these tests to let atmsort sort the
contents of the SELECTs. Note that this requires us to fix the expected
leading whitespace as well, since it affects atmsort order.
Signed-off-by: NAsim R P <apraveen@pivotal.io>

* adding files for new gpbackup utils

* Additional overview info, basic usage information.

* finished basic usage topics

* gpbackup reference page

* reference page work

* finishing gprestore ref page

* adding reference pages to overview, ditamap

* feedback from Oak

* docs: update split limitation for multi-level partition.

* docs: Edits based on review comments.

* docs: add PL/Container documentation

* docs - pl/container - add information about installing Docker images for OSS repo.

* docs: fix typo. update version of tar.gz Docker images to beta1

This commit added an ERR trap that exits the script immediately.
Unfortunately, the gpconfig line that follows doesn't appear to succeed
(and may never have?), which prevents the TINC tests from running
entirely. To get the CI moving again, revert for now.

This reverts commit c8350a80.