Resource group cpu rate limitation is implemented with cgroup on linux
system. When resource group is enabled via GUC we check whether cgroup
is available and properly configured on the system. A sub cgroup is
created for each resource group, cpu quota and share weight will be set
depends on the resource group configuration. The queries will run under
these cgroups, and the cpu usage will be restricted by cgroup.

The cgroups directory structures:
* /sys/fs/cgroup/{cpu,cpuacct}/gpdb: the toplevel gpdb cgroup
* /sys/fs/cgroup/{cpu,cpuacct}/gpdb/*/: cgroup for each resource group

The logic for cpu rate limitation:

* in toplevel gpdb cgroup we set the cpu quota and share weight as:

    cpu.cfs_quota_us := cpu.cfs_period_us * 256 * gp_resource_group_cpu_limit
    cpu.shares := 1024 * ncores

* for each sub group we set the cpu quota and share weight as:

    sub.cpu.cfs_quota_us := -1
    sub.cpu.shares := top.cpu.shares * sub.cpu_rate_limit

The minimum and maximum cpu percentage for a sub cgroup:

    sub.cpu.min_percentage := gp_resource_group_cpu_limit * sub.cpu_rate_limit
    sub.cpu.max_percentage := gp_resource_group_cpu_limit

The acutal percentage depends on how busy the system is.

gp_resource_group_cpu_limit is a GUC introduced to control the cpu
resgroups assigned on each host.

    gpconfig -c gp_resource_group_cpu_limit -v '0.9'

A new pipeline is created to perform the tests as we need privileged
permission to enable and setup cgroups on the system.
Signed-off-by: NNing Yu <nyu@pivotal.io>

In the updated tests, we used functions like disable_xform and
enable_xform to hint the optimizer to disallow/allow a particular
physical node. However, these functions are only available when GPDB
is built with GPORCA. Planner on the other hand accomplished this
via a GUC.

To avoid usage of these functions in tests, I have introduced couple
of GUCS that mimic the same planner behavior but now for GPORCA.
In this effort I needed to add an API inside GPORCA.

Please note: this test does not pass on macOS because
of an apparent failure of libsigar.

See comment in the test on how to omit a test on MacOS.
Signed-off-by: NMelanie Plageman <mplageman@pivotal.io>
Signed-off-by: NLarry Hamel <lhamel@pivotal.io>

* gpperfmon: update gpperfmon install user docs IPv6

Follow up for commit c52c768e

[ci-skip]

Signed-off-by: NMarbin Tan <mtan@pivotal.io>

When dumping a table with dropped columns, the columns are created
and dropped again during the restore to create binary-compatible
heap files. Using TEXT as the dummy attribute however cause toast
tables to be created, even for tables which have no other toastable
attributes. This cause catalog bloat and cause problems for upgrade
since the newly created toast table doesn't have a synchronized Oid
in the dumpfile. Fix by using INTEGER instead.

This is a partial backport of upstream commit 53fa850c
which fixes this case and more (where the more part relies on code
Greenplum has yet to merge, thus partial backport).

  commit 53fa850c
  Author: Tom Lane <tgl@sss.pgh.pa.us>
  Date:   Thu Jul 2 21:34:32 2009 +0000

    Fix up pg_dump's --binary-upgrade option so that it behaves properly with
	inherited columns and check constraints.  Per my recent trouble report.

Defer creation of the PQExpBuffer until we know that the object is
in fact dumpable to avoid leaking a few buffers.

The opfamily dump in pg_dump had an unfortunate typo that snuck in
with the initial pg_upgrade commit, and pg_dump had a broken check
query. Fix syntax.

Previously pg_upgrade was unable to upgrade from a 5.0 cluster due
to Text Search objects, enums, extensions and VIEW rules were left
out from the original import of pg_upgrade.  This adds support for
these objects.

Further, the binary-upgrade mode in pg_dump didn't synchronize the
Oids for the functions associated with the language.  Extract and
add the Oids into the dump to fix.

Also expand comments and fix a missing static declaration function
prototype.

Fixes github issue #1774.

Since python 2.7 has the unittest module included, we do not need to use
unittest2 anymore. This removes this dependency requirement. We
previously used the unittest2 module as it included features that were
not present in python 2.6. There are still unittest2 dependencies in the
TINC tests, but that may be a more intensive effort.

* Fortify SED_PG_CONF() shell function, as called by gpinitsystem

It's also called via gpcreateseg.sh

There is a chance where gpinitsystem may miss a GUC change in one of the
segments and left undetected. This would cause failures down the line.

More robust in three ways:
- Retry ssh calls in case they fail
- Actually error exit if it does fail, for quicker diagnosis if the
  value in postgres.conf did not get updated
- When keeping a version of the key-value pair in a comment, only
  archive the first-matched key-value (otherwise you get comments on
  your comments)

Use trap for retrying failed ssh
Use trap in a limited scope for now

If IPv6 is enabled, gpperfmon will complain that it can't connect to
the database because, gpmon does not have permission to access through
::1.

- Add behave test

- We only want to copy the file if it exists

Updates to configure to use `getversion --short` to generate a version
tag that can be used in `Makefile.global` for packaging etc...

* add info about pl/python multidim array support

* restructure array info so better matches upstream docs

* reorganize a bit, davids other requested edits

In the past there was functionality to decorate a stacktrace with
the developer in charge of the separate functions. This is not in
use anymore and it's not something we want either.

The following 2 functions are removed:
* ExecShareInputScanMarkPos
* ExecShareInputScanRestrPos

This reverts commit c9eafeb6.

Signed-off-by: NChris Hajas <chajas@pivotal.io>

GP_VERSION is used by gppkg and other bits to show what version of GPDB
it is build from.  This is also propogated via gp_config for all
packages and should have a valid value for GP_VERSION.

- We haven't changed cdb_bsa_util for quite some time and these tests
  are not useful to us. Additionally, changing the Netbackup version we
  compiled them with caused compilation to fail.

Signed-off-by: NKaren Huddleston <khuddleston@pivotal.io>

- Make sure to create the database first and then run the sql
Signed-off-by: NMarbin Tan <mtan@pivotal.io>

Across versions, separate files had been created. Going forward, there
will be just one gpperfmon.sql for setting up the gpperfmon database.

- Remove other iterations of gpperfmon sql in makefile
Signed-off-by: NC.J. Jameson <cjameson@pivotal.io>

* update docs for functions supporting COST and DEFAULT arg clauses

* updates per davids edits

* Enable analyzing root partitions
* Ensure that the name of the guc is clear
* Remove double negation (where possible)
* Update comments
* Co-locate gucs that have similar purpose
* Remove dead gucs
* Classify them correctly so that they are no longer hidden

We had cherry-picked some of the PostgreSQL 9.3 functions, but not all the
subsequent fixes to them. I believe we need at least upstream commit
66008564 to fix github issue #2430, and surely the other bug fixes are
needed too.

Rather than cherry-pick individual commits, this replaces the JSON related
source files with the upstream files in toto. This will make diffing and
maintenance of these backported JSON function easier in the future.

Formally, GPDB assumed Gp_interconnect_queue_depth was constant during the interconnect life-time which was incorrect for Cursors, if Gp_interconnect_queue_depth was changed after a Cursor was declared, a panic occurred. To avoid this, we make a copy of Gp_interconnect_queue_depth when interconnect is set up. Gp_interconnect_snd_queue_depth has no such problem because it is only used by senders and senders of Cursor will never receive the GUC change command.

cache.

* Relocates gpperfmon reference from GPCC

* Remove emcconnect history and itorators from PDF ditamap

* Remove iterator metrics subtopic from PDF ditamap

* Additional edits to acknowledge gpperfmon without GPCC

* Incorporate gpperfmon.conf reference into gpperfmon_install reference