Resource group comes with a view gp_toolkit.gp_resgroup_status to get
the running status of all the resource groups.  The cpu and memory
usages as displayed in json format, they are hard to parse and
understand by human.  To make it more user friendly we now provide two
new views on the status, the cpu and memory usages are flatten into
multiple columns.  They also group the status by segment or host so it
is easier to find out the usages at different levels.
Reviewed-by: NZhenghua Lyu <zlv@pivotal.io>

Daniel Gustafsson pointed offlist at the "diretly" typo in my last
libpqwalreceiver refactoring commit. I got a bit carried away and grepped
for other common misspellings.

Instead of providing two varians of each function, one with table OID, and
another with table name as argument, use a single function with 'regclass'
argument. The implicit text to regclass cast takes care of looking up the
relation by name. So from a user's point of view, you can pass an OID or
a table name as argument, but the C code only sees the OID.

This is how, for example, the pg_relation_size() function works these days,
too. (See upstream commit 706a3088.)

The functions allow obtaining or removing entries from the shared hash
table maintained on QD.  Default size of this hash table is 1000 and
entries are removed only after it is filled to capacity.  The two
functions should be helpful for testing as well as troubleshooting
issues with appendonly tables in production deployments.
Co-authored-by: NJimmy Yih <jyih@pivotal.io>

The gp_toolkit.gp_bloat_expected_pages view could mistakenly report a
partition table root as bloated even though partition table roots do
not contain data. This could deceive a user into running a costly
VACUUM FULL on the partition table when it was not needed.

This reverts commit 4750e1b6.

This is the final batch of commits from PostgreSQL 9.2 development,
up to the point where the REL9_2_STABLE branch was created, and 9.3
development started on the PostgreSQL master branch.

Notable upstream changes:

* Index-only scan was included in the batch of upstream commits. It
  allows queries to retrieve data only from indexes, avoiding heap access.

* Group commit was added to work effectively under heavy load. Previously,
  batching of commits became ineffective as the write workload increased,
  because of internal lock contention.

* A new fast-path lock mechanism was added to reduce the overhead of
  taking and releasing certain types of locks which are taken and released
  very frequently but rarely conflict.

* The new "parameterized path" mechanism was added. It allows inner index
  scans to use values from relations that are more than one join level up
  from the scan. This can greatly improve performance in situations where
  semantic restrictions (such as outer joins) limit the allowed join orderings.

* SP-GiST (Space-Partitioned GiST) index access method was added to support
  unbalanced partitioned search structures. For suitable problems, SP-GiST can
  be faster than GiST in both index build time and search time.

* Checkpoints now are performed by a dedicated background process. Formerly
  the background writer did both dirty-page writing and checkpointing. Separating
  this into two processes allows each goal to be accomplished more predictably.

* Custom plan was supported for specific parameter values even when using
  prepared statements.

* API for FDW was improved to provide multiple access "paths" for their tables,
  allowing more flexibility in join planning.

* Security_barrier option was added for views to prevents optimizations that
  might allow view-protected data to be exposed to users.

* Range data type was added to store a lower and upper bound belonging to its
  base data type.

* CTAS (CREATE TABLE AS/SELECT INTO) is now treated as utility statement. The
  SELECT query is planned during the execution of the utility. To conform to
  this change, GPDB executes the utility statement only on QD and dispatches
  the plan of the SELECT query to QEs.
Co-authored-by: NAdam Lee <ali@pivotal.io>
Co-authored-by: NAlexandra Wang <lewang@pivotal.io>
Co-authored-by: NAshwin Agrawal <aagrawal@pivotal.io>
Co-authored-by: NAsim R P <apraveen@pivotal.io>
Co-authored-by: NDaniel Gustafsson <dgustafsson@pivotal.io>
Co-authored-by: NGang Xiong <gxiong@pivotal.io>
Co-authored-by: NHaozhou Wang <hawang@pivotal.io>
Co-authored-by: NHeikki Linnakangas <hlinnakangas@pivotal.io>
Co-authored-by: NJesse Zhang <sbjesse@gmail.com>
Co-authored-by: NJinbao Chen <jinchen@pivotal.io>
Co-authored-by: NJoao Pereira <jdealmeidapereira@pivotal.io>
Co-authored-by: NMelanie Plageman <mplageman@pivotal.io>
Co-authored-by: NPaul Guo <paulguo@gmail.com>
Co-authored-by: NRichard Guo <guofenglinux@gmail.com>
Co-authored-by: NShujie Zhang <shzhang@pivotal.io>
Co-authored-by: NTaylor Vesely <tvesely@pivotal.io>
Co-authored-by: NZhenghua Lyu <zlv@pivotal.io>

* Implement CPUSET, a new management of cpu resource in resource
group which can reserve the specified cores for specified
resource group exclusively. This can ensure that there are always
available cpu resources for the group which has set CPUSET.
The most common scenario is allocating fixed cores for short
queries.

- One can use it by executing CREATE RESOURCE GROUP xxx WITH (
  cpuset='0-1', xxxx). 0-1 are the reserved cpu cores for
  this group. Or ALTER RESOURCE GROUP SET CPUSET '0,1' to modify
  the value.
- The syntax of CPUSET is a combination of the tuples, each
  tuple represents one core number or the core numbers interval,
  separated by comma. E.g. 0,1,2-3. All the core in CPUSET must be
  available in system and the core numbers in each group cannot
  have overlap.
- CPUSET and CPU_RATE_LIMIT are mutually exclusive. One cannot
  create a resource group with both CPUSET and CPU_RATE_LIMIT.
  But the CPUSET and CPU_RATE_LIMIT can be freely switched in
  one group by executing ALTER operation, that means if one
  feature has been set, the other is disabled.
- The cpu cores will be returned to GPDB, when the group has been
  dropped, or the CPUSET value has been changed, or the CPU_RATE_LIMIT
  has been set.
- If some of the cores have been allocated to the resource group,
  then the CPU_RATE_LIMIT in other groups only indicating the
  percentage of cpu resources of the left cpu cores.
- If the GPDB is busy, all the other cores which have not be
  allocated to any resource groups exclusively through CPUSET
  have already been run out, the cpu cores in CPUSET will still
  not be allocated.
- The cpu cores in CPUSET will be used exclusively only in GPDB
  level, the other non-GPDB processes in system may use them.
- Add test cases for this new feature, and the test environment
  must contain at least two cpu cores, so we upgrade the configuration
  of instance_type in resource_group jobs.

* - Compatible with the case that cgroup directory cpuset/gpdb
  does not exist
- Implement pg_dump for cpuset & memory_auditor
- Fix a typo
- Change default cpuset value from empty string to -1, for
  the code in 5X assume that all the default value in
  resource group is integer, a non-integer value will make the
  system fail to start

We used to implement the memory auditor feature differently on master
and 5X, on master the attribute is stored in pg_resgroup while on 5X
it's stored in pg_resgroupcapability.  This increases the maintenance
effort significantly.  So we refactor this feature on master to minimize
the difference between these two branches.

- Revert "resgroup: fix an access to uninitialized address."
  This reverts commit 56c20709.
- Revert "ic: Mark binary_swap_gpdb as optional input for resgroup jobs."
  This reverts commit 9b3d0cfc.
- Revert "resgroup: fix a boot failure when cgroup is not mounted."
  This reverts commit 4c8f28b0.
- Revert "resgroup: backward compatibility for memory auditor"
  This reverts commit f2f86174.
- Revert "Show memory statistics for cgroup audited resource group."
  This reverts commit d5fb628f.
- Revert "Fix resource group test failure."
  This reverts commit 78b885ec.
- Revert "Support cgroup memory auditor for resource group."
  This reverts commit 6b3d0f66.
- Apply "resgroup: backward compatibility for memory auditor"
  This cherry picks commit 23cd8b1e
- Apply "ic: Mark binary_swap_gpdb as optional input for resgroup jobs."
  This Cherry picks commit c86652e6
- Apply "resgroup: fix an access to uninitialized address."
  This Cherry picks commit b257b344

By default the memory management in resource group is
implemented by tracking every memory alloc/free in memory
context.

To support external components ( such as plcontainer) which
do not use memory context, cgroup memory auditor is added
into resource group.

Note that catalog table pg_resgroup is changed.

The gp_bloat_expected_pages.btdexppages column is numeric, but it was passed to
the function gp_bloat_diag() as integer in the definition of the view of the
same name gp_bloat_diag. This caused integer overflow errors when the number of
expected pages exceeded max integer limit for columns with very large widths.

This changes the function signature and call to use numeric for the
btxexppages paramter.

Adding a simple test to mimic the cutomer issue.

Author: Nadeem Ghani <nghani@pivotal.io>
Author: Shoaib Lari <slari@pivotal.io>

Also change a few regression tests to use the new syntax, instead of
gp_toolkit's __gp_localid and __gp_masterid functions.

This bumps the copyright years to the appropriate years after not
having been updated for some time.  Also reformats existing code
headers to match the upstream style to ensure consistency.

Change initial contents in pg_resgroupcapability:
* Remove memory_redzone_limit;
* Add memory_shared_quota, memory_spill_ratio;

Change resgroup concurrency range to [1, 'max_connections']:
* Original range is [0, 'max_connections'], and -1 means unlimited.
* Now the range is [1, 'max_connections'], and -1 is not supported.

Change resgroup limit type from float to int.

Changed below resgroup resource limit types from float to int percentage value:
* cpu_rate_limit;
* memory_limit;
* memory_shared_quota;
* memory_spill_ratio;

Add view gp_toolkit.gp_resgroup_status for resource group runtime
statistics information, such as cpu usage, memory usage, etc..

Example:

    -- query resource group runtime statistics information
    SELECT * FROM gp_toolkit.gp_resgroup_status;

A helper function pg_resgroup_get_status_kv() is also provided to
collect these information.

A new dir src/backend/utils/resgroup/ is created to hold the resgroup
source code.

As resgroup is not fully implemented yet there is only dummy output for
this view at the moment.

Add view gp_toolkit.gp_resgroup_config for resource group information,
such as concurrency, cpu rate limitation, etc..

Example:

    -- query resource group config information
    SELECT * FROM gp_toolkit.gp_resgroup_config;

No need to create a whole new library for this. It's included in every
installation anyway, as part of gp_toolkit. Might as well just include
it in the backend, and make the build slightly simpler.

This meant moving the version field from pg_appendonly to the
pg_aoseg_<oid> table (or pg_aocsseg_<oid>, for AOCS). We can still read
and write both formats, but new segments will always be created in the new
format (except if you set the test_appendonly_version_default GUC).

[#130666969]
Internally this is the shared memory variable used_segspace in
the view gp_toolkit.gp_workfile_mgr_used_diskspace.
Signed-off-by: NXin Zhang <xzhang@pivotal.io>

This includes code for sharedcache lookup and eviction.

We don't use workfile manager as a cache, we only keep workfiles around for
as long as we need them. So "clearing the cache" should always be a no-op.

If there is a schema in the database that the user does not have access to,
doing "select * from gp_toolkit.__gp_user_tables_readable" throws a
permission denied error on the schema. Use the relation-OID-variant of
has_table_privilege() function, to work around that.

Usage: minirepro <database> [options]

Options:
  --version             show program's version number and exit
  -?, --help            Show this help message and exit
  -h HOST, --host=HOST  Specify a remote host
  -p PORT, --port=PORT  Specify a port other than 5432
  -U USER, --user=USER  Connect as someone other than current user
  -q QUERY_FILE         file name that contains the query
  -f OUTPUT_FILE        minirepro output file name

Closes #157.

Use the standard initdb mechanism for it, instead of having bespoken code
in gpinitsystem.

Also remove the unnecessarily complicated logic to generate gp_toolkit.sql
from gp_toolkit.sql.in. All the JETPACK_* variables were constants, and
there is particular need in making the gp_toolkit functions relocatable to
different schemas or prefixes, so just hardcode them and remove the sed
magic.

Rewrite the file header of gp_toolkit.sql. The legal blurp is not needed,
make the header look more like that of information_schema.sql and
system_views.sql.

Remove compaction_info.sql. It was only needed for old in-place
upgrade from 4.3.x to 4.3.4.2, which is not relevant for master branch.

Fix the "jetpack" bugbuster regression test. It was originally written to
test the gp_toolkit views, but gp_toolkit was in fact not installed in
bugbuster, and the expected output just contained a bunch of "relation
does not exist" errors. Now it actually tests gp_toolkit again. Some of
the test queries didn't produce repeateable results, due to differing oids,
differences in table sizes, and other tables created by previous tests, so
fixed those too.

This has a small user-visible difference: gp_toolkit schema is now also
installed in template0. That caused the difference in the expected outputs:
pg_regress creates the "regression" database using template0 as the
template, so gp_toolkit used to not be installed in it, but it is now.