Signed-off-by: NMarbin Tan <mtan@pivotal.io>

Coefficient of Variation Calculation

Coefficient of variation is the standard deviation divided by the mean.

We're using the term skew very loosely in our description, as we're
actually calculating coefficient of variation.

With coefficient of variation, we can tell how dispersed the data points are across
the segments. The higher the coefficient of variation, the more
non-uniform the distribution of the data in cluster.

Coefficient of variation is unitless so it could be used for comparing
different clusters and how they are performing relative to each other.

CPU Skew calculation:
mean(cpu) = sum(per segment cpu cycle) / sum(segments)
variance(cpu) = sqrt( sum(((cpu(segment) - mean(cpu))^2) ... ) / sum(segments) )
std_dev(cpu) = sqrt(variance(cpu))
skew(cpu) = coeffecient of variation = std_dev(cpu)/mean(cpu)

Row out Skew calculation:
mean(row) = sum(per segment row) / sum(segments)
variance(row) = sqrt( sum(((row(segment) - mean(row))^2) ... ) / sum(segments) )
std_dev(row) = sqrt(variance(row))
skew(row) = coeffecient of variation = std_dev(row)/mean(row)
Signed-off-by: NTushar Dadlani <tdadlani@pivotal.io>

The behave test is intermittently failing because the substep is trying
to install gpperfmon eventhough gpperfmon is already installed. Trying
to figure out which part of failing the correct value is hard with the
current setup.
Decouple the checks to determine which part is actually failing.
Signed-off-by: NTushar Dadlani <tdadlani@pivotal.io>

Any http request to gpfdist with pipe will lead original pipe reader process hung.
In order to avoid the situation that random http request to the working gpfdist instance,
gpfdist will check request header.

Any manual request should run as:
wget --header='X-GP-PROTO:0' http://host:port/file

If QD receives a SIGINT and calls CHECK_FOR_INTERRUPTS after finishing Gang creation, but before recording this Gang in global variables like primaryWriterGang, this Gang would not be destroyed, hence next time QD wants to create a new writer Gang, it would find existing writer Gang on segments, and report snapshot collision error.

Signed-off-by: NDhanashree Kashid <dkashid@pivotal.io>

The pg_control change to bring in heap checksum from upstream is breaking
binary compatibility.  As soon as that is merged, binary swap test will start
failing.  Disable it now.  It will be re-enabled once a new beta tag is
generated.  Thereafter, binary swap test will verify binary compatibility
between the new beta tag and HEAD.

This patch pulls in the addition of checksum version information to pg_control
and a GUC to report the checksum version.  Heap data checksum feature will be
pulled in its entirety as subsequent patches.

Upstream commit that this patch pulls from:

commit 96ef3b8f
Author: Simon Riggs <simon@2ndQuadrant.com>
Date:   Fri Mar 22 13:54:07 2013 +0000

    Allow I/O reliability checks using 16-bit checksums

commit 44395174
Author: Simon Riggs <simon@2ndQuadrant.com>
Date:   Tue Apr 30 12:27:12 2013 +0100

    Record data_checksum_version in control file.
commit 5a7e75849cb595943fc605c4532716e9dd69f8a0
Author: Heikki Linnakangas <heikki.linnakangas@iki.fi>
Date:   Mon Sep 16 14:36:01 2013 +0300

    Add a GUC to report whether data page checksums are enabled.

* Update gpdemo documentation

Remove Solaris documentation
Update port numbers
Add environment variables

This was removed in upstream in c970292a,
and is one step forward to make a number of counters 64 bit safe.

* DOCS: Adding security guide source

* Proposed updates from review

Support ALTER RESOURCE GROUP SET CPU_RATE_LIMIT syntax.

The new cpu rate limit take effect immediately at end of transaction.

Example 1:

    CREATE RESOURCE GROUP g1
        WITH (cpu_rate_limit=0.1,memory_limit=0.1);
    ALTER RESOURCE GROUP g1 SET CPU_RATE_LIMIT 0.2;

The new cpu rate limit take effect immediately.

Example 2:

    BEGIN;
    ALTER RESOURCE GROUP g1 SET CPU_RATE_LIMIT 0.2;

The new cpu rate limit doesn't take effect unless the transaction is
committed.
Signed-off-by: NRichard Guo <riguo@pivotal.io>
Signed-off-by: NGang Xiong <gxiong@pivotal.io>

* Updated file for COPY ON SEGMENT

* Extra note about FROM and STDOUT

* Incorporate comments from David and Mel

* Revise COPY FROM note

* Copying note from line 65

- make distclean can not be called before configure

The gp_statsistics prefix was not included in the list of files to
restore from ddboost, causing restore to fail when gpdbrestore
--restore-stats was used.

This functionality was included with pg_dump, but was missing from
gpcrondump.

With --enable-segwalrep, mirror leverages replay of xl_mm_fs_obj records to
delete files. Code was not correctly handling appendonly tables as was calling
`smgrdounlink()`, which is for heap tables or indexes. For AO/CO tables, need to
perform drop of specific single file mentioned in xlog record, which is
performed by `MirroredAppendOnly_Drop()`. Without this code currently files like
<relfilenode>.127, <relfilenode>.129,... etc get left behind on mirror. The
problem was not seen so far as master never stores data for AO/CO tables hence
these files are not created on master. Only when now we start enabling wal
replication for segments this is required.

Helpful for debugging to set GUC Debug_print_qd_mirroring but the message were
with DEBUG1. Just enabling and disabling is guc enough to control the logging
don't need second level.

Incase of --enable-segwalrep, write-ahead logging should not be skipped for
anything, as it relies on that mechanism to construct the things on
mirror. Write-ahead logging for these pieces were only enabled performed for
master, with this commit gets enabled for segments as well.

Currently, get_filespaces_to_send() only works for QD. To enable pg_basebackup
and wal replication for QEs, this function must also work on QEs. This function
relies on pg_filespace_entry table to provide information, as its only available
on QD currently can't be leveraged. Hence just enabling basic support for QEs
for default filespace. Supporting user defined filespaces and non-default
transaction filespace, will be dealt incrementally later.

In this behave test, we delete some entries in pg_depend and in some
relative catalog tables to simulate a corruption around pg_depend. The
gpcheckcat tool should then flag these down.

The current gpcheckcat dependency check only checked for extra
pg_depend entries where a pg_depend entry's objid or refobjid did not
exist as an OID of any catalog table with hasoids set. We also need to
check the reverse scenario where a catalog entry is missing an entry
in pg_depend. This particular scenario is difficult to flag due to
catalog entries having multiple unique pg_depend references or are
created later from a query that may add dependency (e.g. granting
ownership of a database to a certain user). Therefore, we add a very
basic check only against catalog tables that immediately create
dependencies upon its relative query.

We did not check for missing or extra pg_depend entries across the
cluster during gpcheckcat. We would be unaware of scenarios where a
pg_depend entry went missing and the object that used that dependency
is dropped. Those scenarios could lead to leftover catalog entries and
prevent some simple CREATE statements.

As gpcheckcat builds its mapping of catalog issues, it can flag
objects whose parents no longer exist (e.g. a toast table left over
after dropping a table). When these get caught, gpcheckcat will
unfortunately error out on the reporting step. To prevent erroring
out, we just check for None in the RelationObject's vars during
reporting.

Another issue that is fixed is the repetitive reporting of issues on
the testing's current database following testing of a different
database. The catalog issues reported were invalid for the current
database and were actually issues from the previous database that was
checked. This was caused by the improper resetting of the GPObjects
and GPObjectGraph global dictionaries. To fix the issue, we properly
use the clear() function to reuse the global variables.

[#147538353]