Also, added a unit test.
Signed-off-by: NNadeem Ghani <nghani@pivotal.io>

Signed-off-by: NLarry Hamel <lhamel@pivotal.io>

CREATE RESOURCE GROUP rg1 WITH (concurrency=1, cpu_rate_limit=10, memory_limit=10);
	CREATE ROLE r1 RESOURCE GROUP rg1;

	session 1:
	set role r1;
	BEGIN;

	session 2:
	BEGIN; <--- hang, and then cancel
	BEGIN; <--- assertion failure
Signed-off-by: NNing Yu <nyu@pivotal.io>

Memory usage statistic in resource group is defined as unsigned integer.
For subtraction 'a - b' on memory usage, the atomic subtraction function
'pg_atomic_sub_fetch_*' will return the value of 'a' before the subtraction.
Then this value is asserted to be no less than 'b'.

Change initial contents in pg_resgroupcapability:
* Remove memory_redzone_limit;
* Add memory_shared_quota, memory_spill_ratio;

Change resgroup concurrency range to [1, 'max_connections']:
* Original range is [0, 'max_connections'], and -1 means unlimited.
* Now the range is [1, 'max_connections'], and -1 is not supported.

Change resgroup limit type from float to int.

Changed below resgroup resource limit types from float to int percentage value:
* cpu_rate_limit;
* memory_limit;
* memory_shared_quota;
* memory_spill_ratio;

Currently, CommitPrepared xlog record, `xl_xact_commit_prepared` doesn't store
information for distributed transaction like distributed transaction id or
distributed timestamp. Its extremely helpful to have this information recorded
plus also needed to replay/redo the xlog record.

Currently, redo of CommitPrepared xlog record is not updating the distributed
commit log. Which as of now seems not possing any issues as recovery of primary
or failover to mirror will disconnect all existing connections but for
consistency its better to redo distributedlog commit as well during redo of
CommitPrepared.

segmentCount variable is unused in TMGXACT_CHECKPOINT structure hence loose it
out. Also, removing the union in fspc_agg_state, tspc_agg_state and
dbdir_agg_state structures as don't see reason for having the same.

The minipage attribute was using varbit without actually storing a
varbit datum in it. Change over to bytea since it makes reading the
value back easier, especially for pg_upgrade. This complements the
change in commit dce769fe which performed the same change
for the visimap attribute of the AO visimap relation.

Since the bitmap hack function is created in the pg_temp schema,
exempt it from Oid synchronization during binary upgrades to allow
creation. This fix applies to the visimap handling as well.

The vpinfo for `aocsvpinfo_decode()` had incorrect data type. Change it
from `varbit` to `bytea` to match the column type in pg_aocsseg table.

It's needed to compile with gpfdist.

Per github issue #2739, reported by @flochman.

Commit a8f956c6 removed the old SAN
failover code but left the catalogs in place due to catalog change
freeze. This removes the no longer used catalogs and the relevant
doc entries.

This adds the ability for the caller of pg_terminate_backend() or
pg_cancel_backend() to include an optional message to the process
which is being signalled. The message will be appended to the error
message returned to the killed process. The new syntax is overloaded
as:

    SELECT pg_terminate_backend(<pid> [, msg]);
    SELECT pg_cancel_backend(<pid> [, msg]);

The creation of filespaces, database, tablespaces, and relfilenodes is logged by
a MMXLOG record in the Xlog. Functionality is added in xlogdump to display these
records.

* Add libpq as database driver

* Add information how to add a sequence as default value

* Update Encoding information, add example

There are times where gpperfmon_log_alert_history scenario fails,
because there's no data in the log alert history table.
This might be due to us copying an empty csv file; gpperfmon writes to a
file for log alert in a cadence. We might be copying a file that has not
been written into yet -- possibly empty.

Make sure that we have something to copy first before proceeding to the
next step.
Signed-off-by: NMarbin Tan <mtan@pivotal.io>

Coefficient of Variation Calculation

Coefficient of variation is the standard deviation divided by the mean.

We're using the term skew very loosely in our description, as we're
actually calculating coefficient of variation.

With coefficient of variation, we can tell how dispersed the data points are across
the segments. The higher the coefficient of variation, the more
non-uniform the distribution of the data in cluster.

Coefficient of variation is unitless so it could be used for comparing
different clusters and how they are performing relative to each other.

CPU Skew calculation:
mean(cpu) = sum(per segment cpu cycle) / sum(segments)
variance(cpu) = sqrt( sum(((cpu(segment) - mean(cpu))^2) ... ) / sum(segments) )
std_dev(cpu) = sqrt(variance(cpu))
skew(cpu) = coeffecient of variation = std_dev(cpu)/mean(cpu)

Row out Skew calculation:
mean(row) = sum(per segment row) / sum(segments)
variance(row) = sqrt( sum(((row(segment) - mean(row))^2) ... ) / sum(segments) )
std_dev(row) = sqrt(variance(row))
skew(row) = coeffecient of variation = std_dev(row)/mean(row)
Signed-off-by: NTushar Dadlani <tdadlani@pivotal.io>

Doing any kind of intensive work should show up in gpperfmon for both
cpu skew and row skew, so putting them together as they can be tested at
the same time.
Signed-off-by: NMarbin Tan <mtan@pivotal.io>

sigar_proc_cpu_get is the function which takes a pid
and goes to the cpu and gets the cpu_elapsed value for that pid at a
given moment.

get_pid_metrics saves the information into a hash table within the
gpsmon process. That pid only gets wiped out of the hashtable when
gpmmon requests a dump command to gpsmon.

gpsmon process sends tcp packets with information about cpu_elapsed
(amount of cpu cycles spent on a given
slice/process). Doing a 'dump' clears out the gpsmon hashtables.

This could lead to scenarios where a query has ended,
the process has died, and the sigar_proc_cpu_get function can't find the pid
so it puts 0 into the struct which is updating the hashtable so the
hashtable has a 0 for cpu_elapsed and then this is what is in
queries_tail and so queries_history ends up having a 0 for cpu_elapsed
since it was the last entry in queries_tail

Fix: Ensure that we validate the functions that request pid metrics
from libsigar and log the issues if they occur.
Signed-off-by: NTushar Dadlani <tdadlani@pivotal.io>

Previously, if pg_statistic already contains statistics for
a given table column, attempting to insert those statistics
again during the restore would give a primary key error and
statistics would not be correctly restored.

Now, existing statistics are deleted just before inserting the
restore statistics, so there is no collision.
Signed-off-by: NJamie McAtamney <jmcatamney@pivotal.io>

gpload.cleanupSql() in the `finally` block may throw a `SystemExit`
exception, which will not be caught by `except Exception:` and exit
before cleaning up others like gpfdist processes.

This commit catches `SystemExit`, and do the stop_gpfdists() before
cleanupSql() to avoid this situation.
Signed-off-by: NAdam Lee <ali@pivotal.io>
Signed-off-by: NXiaoran Wang <xiwang@pivotal.io>

Signed-off-by: NChris Hajas <chajas@pivotal.io>

Signed-off-by: NDhanashree Kashid <dkashid@pivotal.io>

This is the final part of the backup and restore TINC to behave
migration. The MFR suite tests the managed file replication for Data
Domain.
Signed-off-by: NChris Hajas <chajas@pivotal.io>

These tests compare generated plans to catch significant plan changes
during subselect pull up for IN, EXISTS, NOT IN, NOT EXISTS related
queries.

- Currently the readme leaves out the  dependency
- Places all dependencies into one location in the readme