Signed-off-by: NZhenghua Lyu <zlv@pivotal.io>

The user can config resgroup to make some query's query memory is zero.
In such cases, it will use work memory. And since query_mem's type is uint64,
we simply remove the assert in spi execution's code.

Signed-off-by: NXiaoran Wang <xiwang@pivotal.io>

Using gp_segment_configuration catalog table easily can find if mirrors exist or
not, do not need special table to communicate the same. Earlier
gp_fault_strategy used to convey 'n' for mirrorless system, 'f' for replication
and 's' for san mirrors. Since support for 's' was removed in 5.0 only purpose
gp_fault_strategy served was mirrored or not mirrored system. Hence deleting the
gp_fault_strategy table and at required places using gp_segment_configuration to
find the required info.

GPORCA should not spend time extracting column statistics that are not
needed for cardinality estimation. This commit eliminates this overhead
of requesting and generating the statistics for columns that are not
used in cardinality estimation unnecessarily.

E.g:
`CREATE TABLE foo (a int, b int, c int);`

For table foo, the query below only needs for stats for column `a` which
is the distribution column and column `c` which is the column used in
where clause.
`select * from foo where c=2;`

However, prior to that commit, the column statistics for column `b` is
also calculated and passed for the cardinality estimation. The only
information needed by the optimizer is the `width` of column `b`. For
this tiny information, we transfer every stats information for that
column.

This commit and its counterpart commit in GPORCA ensures that the column
width information is passed and extracted in the `dxl:Relation` metadata
information.

Preliminary results for short running queries provides up to 65x
performance improvement.
Signed-off-by: NJemish Patel <jpatel@pivotal.io>

* gpload hang due to non-reentrant function invoked in single handler

instead of using libapr, we register term signal in libevent, so that the signal handler
in the asynchronous model to avoid function non-reentrant problem.

1. copy *test/regress/*.pm file to install locaiton to support
regression test diff
2. set LIBPATH and GP_LIBPATH_FOR_PYTHON env for AIX
Signed-off-by: NPeifeng Qiu <pqiu@pivotal.io>

Although I'm not too familiar with SystemTap, I'm pretty sure that recent
versions can do user space tracing better. I don't think anyone is using
these hacks anymore, so remove them.

* resgroup: move MyResGroupSharedInfo into MyResGroupProcInfo.

MyResGroupSharedInfo is now replaced with MyResGroupProcInfo->group.

* resgroup: retire resGranted in PGPROC.

when resGranted == false we must have resSlotId == InvalidSlotId,
when resGranted != false we must have resSlotId != InvalidSlotId,
so we can retire resGranted and keep only resSlotId.

* resgroup: rename sharedInfo to group.

in resgroup.c there used to be both `group` and `sharedInfo` for the
same thing, now only use `group`.

* resgroup: rename MyResGroupProcInfo to self.

We want to use this variable directly so a short name is better.

running or waiting in resource group.

* Use built-in JSON parser for PXF fragments

Instead of relying on a external library, use the built-in JSON
parser in the backend for the PXF fragments parsing. Since this
replaces the current implementation with an event-based callback
parser, the code is more complicated, but dogfooding the parser
that we want extension writers to use is a good thing.

This removes the dependency on json-c from autoconf, and enables
building PXF on Travis for extra coverage.

* Use elog for internal errors, and ereport for user errors

Internal errors where we are interested in source filename should
use elog() which will decorate the error messages automatically
with this information. The connection error is interesting for the
user however, use ereport() instead there.

Migrations efforts for concourse 3 that are backwards compatible with
concourse 2.7.3

Includes using the image_resource in external task yamls for pipeline
jobs that use a sles based docker image. Currently unknown why the sles
image in particular causes image_resource issues.
Signed-off-by: NDivya Bhargov <dbhargov@pivotal.io>
Signed-off-by: NJim Doty <jdoty@pivotal.io>

This is the spec-compliant spelling, but GPDB has only allowed
"agg OVER (window)" so far. With this commit, the parens are still allowed,
for backwards-compatibility.

Change deparsing code to also use the non-parens syntax in view definitions
and EXPLAIN. Adjust expected output of regression tests accordingly.

We are now on postgres 8.4 which apparently activated some conditional
statements in describe.c. This one was missed from the first 8.4 merge
chunk which makes psql -l currently error out.

Reported by Brian Lu in the Greenplum Developers mailing list:
https://groups.google.com/a/greenplum.org/forum/#!topic/gpdb-dev/5l7J2j5yla8

Update README to explain prerequisites for
correct access to LSN fields of a page.
Independent chunk removed from checksums
patch to reduce size of patch.

(cherry picked from commit 1c563a2a)

'x' is eXtensions, 'E' is the External tables with the relstorage 'x'.

1) Give error meesage about proxy.
2) Change server_side_encryption default value from "none" to empty
string.
Signed-off-by: NXiaoran Wang <xiwang@pivotal.io>

    commit 3d009e45
    Author: Heikki Linnakangas <heikki.linnakangas@iki.fi>
    Date:   Wed Feb 27 18:17:21 2013 +0200

        Add support for piping COPY to/from an external program.

        This includes backend "COPY TO/FROM PROGRAM '...'" syntax, and corresponding
        psql \copy syntax. Like with reading/writing files, the backend version is
        superuser-only, and in the psql version, the program is run in the client.

        In the passing, the psql \copy STDIN/STDOUT syntax is subtly changed: if you
        the stdin/stdout is quoted, it's now interpreted as a filename. For example,
        "\copy foo from 'stdin'" now reads from a file called 'stdin', not from
        standard input. Before this, there was no way to specify a filename called
        stdin, stdout, pstdin or pstdout.

        This creates a new function in pgport, wait_result_to_str(), which can
        be used to convert the exit status of a process, as returned by wait(3),
        to a human-readable string.

        Etsuro Fujita, reviewed by Amit Kapila.
Signed-off-by: NAdam Lee <ali@pivotal.io>
Signed-off-by: NMing LI <mli@apache.org>

It seems that we can no longer use ${} in the env var fields so remove
them to get our Travis builds green again.

Reference:
https://github.com/travis-ci/travis-build/commit/a6779473124442fa748795fa9fd47afc529fc9d4

This utility used to confirm data transfered by doing a md5 digest. This
commit changes behavior to use sha256 instead.
Signed-off-by: NShoaib Lari <slari@pivotal.io>

Signed-off-by: NNadeem Ghani <nghani@pivotal.io>

Need flexibility to be able to control at what freqeuncy pipeline gets triggerd,
one such usecase is same pipleine code runs with asserts on every commit and
without asserts daily/weekly. So, adding a time resource via which can control
the triggering. Plus, parameterize the trigger to control the things.

Some backups with Data Domain contained an incorrect path in their
report file. When checking whether the backup timestamp was in a pre or
post content-id format, restore would fail since the path in the report
file didn't match the expected pattern. Instead, we now check for the
timestamp format in a more generalized way to account for this discrepancy.
Signed-off-by: NChris Hajas <chajas@pivotal.io>

It was noticed in CentOS 6 that the entries in the char* list would
become blank after freeing the raw string that is obtained from
setting the GUC wal_consistency_checking. We should free this string
only after we finish using it.

Authors: Jimmy Yih and Taylor Vesely

The MoveTransFilespaceLocally class in filespace.py used to verify that two directories
are equivalent by doing a md5 on their contents. This commit changes behavior to use
sha256 instead.
Signed-off-by: NNadeem Ghani <nghani@pivotal.io>

Signed-off-by: NShoaib Lari <slari@pivotal.io>

We no longer call contain_windowfunc() on raw parse trees, so we can
revert this to the way it is in the upstream.

This is the first merge commit, on our way to merging with PostgreSQL 8.4.

Nothing too exciting in this batch. Most of the included changes we had
in fact already backported earlier. But some things worth mentioning:

* This bumps up the PostgreSQL version number to 8.4devel. This might
  require 3rd party tools to be fixed to cope with that.

* Bumping the version number caused some \d queries in psql to fail,
  because they did a version check and tried to run queries that only work
  on PostgreSQL 8.4. This happens because the GPDB version of psql has been
  backported from 9.0. To make it work for now, I #ifdef'd out the version
  checks, and added GPDB_84_MERGE_FIXME comments to remind us to re-enable
  them again later, once we have merged the backend catalog changes that
  they need.

* Same for a some version checks in pg_upgrade.

* The build system now links the whole backend in one invocation, instead
  of using the per-directory intermediate SUBSYS.o files. We had already
  backported the Makefile changes for that, so this just flips the switch
  to start using it.

This has been a joint effort between Heikki Linnakangas, Daniel Gustafsson,
Jacob Champion and Tom Meyer.

The view gp_toolkit.gp_resgroup_status collect resgroup status
information on both QD and QEs, also before and after a 300ms delay to
measure the cpu usage. When there is concurrently resgroup drops it
might fail due to missing resgroups.

This is fixed by holding ExclusiveLock which blocks the drop operations.
Signed-off-by: NNing Yu <nyu@pivotal.io>

In some weak test agents, the shared semaphore is two small to support large
max_connections like 150, so refine test cases to make it work with
40 max_connections.

Previously in ResGroupSlotRelease(), we mark the resWaiting flag to
false out of the ResGroupLock, a race condition can happen as that
the waitProc that meant to be wake up is canceled just before it's
resWaiting is set to false and in the window resWaiting is set, the
waitProc run another query and wait for a free slot again, then
resWaiting is set and the waitProc will get another unexpected free slot.

eg: rg1 has concurrency = 1

s1: BEGIN;
s2: BEGIN; -- blocked
s1: END; -- set breakpoint just before resWaiting is set in ResGroupSlotRelease()
s2: cancel it;
s3: BEGIN; -- will get a free slot.
s2: BEGIN; -- run begin again and be blocked.
s1: continue;
s2: s2 will got unexpected free slot

To avoid leak of resgroup slot, we set resWaiting under ResGroupLock so the
waitProc have no window to re-fetch the slot before resWaiting is set.
It's hard to add a regression test, adding breakpoint in ResGroupSlotRelease
may block all other operations because it holds the ResGroupLock.

string overflow won't happen here because gpcheckcloud_newline was
checked before, but still, strncpy() looks better.
Signed-off-by: NXiaoran Wang <xiwang@pivotal.io>
Signed-off-by: NAdam Lee <ali@pivotal.io>