The backport of the data checksum catalog changes backported the
relevant GUC from a version which has struct config_bool defined
differently than GPDB. The reason an extra NULL in the config_bool
array initialization wasn't causing a compilation failure is that
there is an extra bool member at the end which is only set during
runtime, reset_val. The extra NULL was "overflowing" into this
member and thus only raised a warning under -Wint-conversion:

    guc.c:1180:15: warning: incompatible pointer to integer
	               conversion initializing 'bool' (aka 'char')
				   with an expression of type 'void *’

Fix by removing the superflous NULL. Since it was setting reset_val
to NULL (and for a GUC which is yet to "do something") there should
be no effects by this.

Implement resgroup memory limit.

In a resgroup we divide the memory into several slots, the number
depends on the concurrency setting in the resgroup. Each slot has a
reserved quota of memory, all the slots also share some shared memory
which can be acquired preemptively.

Some GUCs and resgroup options are defined to adjust the exact allocation
policy:

resgroup options:
- memory_shared_quota
- memory_spill_ratio

GUCs:
- gp_resource_group_memory_limit
Signed-off-by: NNing Yu <nyu@pivotal.io>

Added cdbvars.h to be installed so extensions can import it.

This code is not correctly recording the data. There seems to be a bug
with how we're modifying the values. Most likely, that we're trying to
change the address instead of the actual value it's pointing to.

We will need to fully run the test this part of the code again and create a new PR.

This reverts commit 411d3c82.

This gpsegwalrep tool is meant for assisting in segment WAL
replication development which is why it is not placed in
gpMgmt/bin/. The tool is used to initialize, start, stop, and destroy
WAL replication mirror segments. It can also be said that this tool is
a rough example of what a segment walrep tool would look like in
Greenplum 6.x.

There are times where a developer may want to bring up a Greenplum
demo cluster without mirrors. This change should make this possible
now.

Example:
WITH_MIRRORS=false make create-demo-cluster

compiler is complaining about prototype not being there, even if it's
already there. It seems that adding a void into the prototype silences this
compiler warning.
Signed-off-by: NTushar Dadlani <tdadlani@pivotal.io>

Signed-off-by: NMarbin Tan <mtan@pivotal.io>

Coefficient of Variation Calculation

Coefficient of variation is the standard deviation divided by the mean.

We're using the term skew very loosely in our description, as we're
actually calculating coefficient of variation.

With coefficient of variation, we can tell how dispersed the data points are across
the segments. The higher the coefficient of variation, the more
non-uniform the distribution of the data in cluster.

Coefficient of variation is unitless so it could be used for comparing
different clusters and how they are performing relative to each other.

CPU Skew calculation:
mean(cpu) = sum(per segment cpu cycle) / sum(segments)
variance(cpu) = sqrt( sum(((cpu(segment) - mean(cpu))^2) ... ) / sum(segments) )
std_dev(cpu) = sqrt(variance(cpu))
skew(cpu) = coeffecient of variation = std_dev(cpu)/mean(cpu)

Row out Skew calculation:
mean(row) = sum(per segment row) / sum(segments)
variance(row) = sqrt( sum(((row(segment) - mean(row))^2) ... ) / sum(segments) )
std_dev(row) = sqrt(variance(row))
skew(row) = coeffecient of variation = std_dev(row)/mean(row)
Signed-off-by: NTushar Dadlani <tdadlani@pivotal.io>

The behave test is intermittently failing because the substep is trying
to install gpperfmon eventhough gpperfmon is already installed. Trying
to figure out which part of failing the correct value is hard with the
current setup.
Decouple the checks to determine which part is actually failing.
Signed-off-by: NTushar Dadlani <tdadlani@pivotal.io>

Any http request to gpfdist with pipe will lead original pipe reader process hung.
In order to avoid the situation that random http request to the working gpfdist instance,
gpfdist will check request header.

Any manual request should run as:
wget --header='X-GP-PROTO:0' http://host:port/file

If QD receives a SIGINT and calls CHECK_FOR_INTERRUPTS after finishing Gang creation, but before recording this Gang in global variables like primaryWriterGang, this Gang would not be destroyed, hence next time QD wants to create a new writer Gang, it would find existing writer Gang on segments, and report snapshot collision error.

Signed-off-by: NDhanashree Kashid <dkashid@pivotal.io>

The pg_control change to bring in heap checksum from upstream is breaking
binary compatibility.  As soon as that is merged, binary swap test will start
failing.  Disable it now.  It will be re-enabled once a new beta tag is
generated.  Thereafter, binary swap test will verify binary compatibility
between the new beta tag and HEAD.

This patch pulls in the addition of checksum version information to pg_control
and a GUC to report the checksum version.  Heap data checksum feature will be
pulled in its entirety as subsequent patches.

Upstream commit that this patch pulls from:

commit 96ef3b8f
Author: Simon Riggs <simon@2ndQuadrant.com>
Date:   Fri Mar 22 13:54:07 2013 +0000

    Allow I/O reliability checks using 16-bit checksums

commit 44395174
Author: Simon Riggs <simon@2ndQuadrant.com>
Date:   Tue Apr 30 12:27:12 2013 +0100

    Record data_checksum_version in control file.
commit 5a7e75849cb595943fc605c4532716e9dd69f8a0
Author: Heikki Linnakangas <heikki.linnakangas@iki.fi>
Date:   Mon Sep 16 14:36:01 2013 +0300

    Add a GUC to report whether data page checksums are enabled.

* Update gpdemo documentation

Remove Solaris documentation
Update port numbers
Add environment variables

This was removed in upstream in c970292a,
and is one step forward to make a number of counters 64 bit safe.

* DOCS: Adding security guide source

* Proposed updates from review

Support ALTER RESOURCE GROUP SET CPU_RATE_LIMIT syntax.

The new cpu rate limit take effect immediately at end of transaction.

Example 1:

    CREATE RESOURCE GROUP g1
        WITH (cpu_rate_limit=0.1,memory_limit=0.1);
    ALTER RESOURCE GROUP g1 SET CPU_RATE_LIMIT 0.2;

The new cpu rate limit take effect immediately.

Example 2:

    BEGIN;
    ALTER RESOURCE GROUP g1 SET CPU_RATE_LIMIT 0.2;

The new cpu rate limit doesn't take effect unless the transaction is
committed.
Signed-off-by: NRichard Guo <riguo@pivotal.io>
Signed-off-by: NGang Xiong <gxiong@pivotal.io>

* Updated file for COPY ON SEGMENT

* Extra note about FROM and STDOUT

* Incorporate comments from David and Mel

* Revise COPY FROM note

* Copying note from line 65

- make distclean can not be called before configure

The gp_statsistics prefix was not included in the list of files to
restore from ddboost, causing restore to fail when gpdbrestore
--restore-stats was used.

This functionality was included with pg_dump, but was missing from
gpcrondump.

With --enable-segwalrep, mirror leverages replay of xl_mm_fs_obj records to
delete files. Code was not correctly handling appendonly tables as was calling
`smgrdounlink()`, which is for heap tables or indexes. For AO/CO tables, need to
perform drop of specific single file mentioned in xlog record, which is
performed by `MirroredAppendOnly_Drop()`. Without this code currently files like
<relfilenode>.127, <relfilenode>.129,... etc get left behind on mirror. The
problem was not seen so far as master never stores data for AO/CO tables hence
these files are not created on master. Only when now we start enabling wal
replication for segments this is required.