* resgroup: move MyResGroupSharedInfo into MyResGroupProcInfo.

MyResGroupSharedInfo is now replaced with MyResGroupProcInfo->group.

* resgroup: retire resGranted in PGPROC.

when resGranted == false we must have resSlotId == InvalidSlotId,
when resGranted != false we must have resSlotId != InvalidSlotId,
so we can retire resGranted and keep only resSlotId.

* resgroup: rename sharedInfo to group.

in resgroup.c there used to be both `group` and `sharedInfo` for the
same thing, now only use `group`.

* resgroup: rename MyResGroupProcInfo to self.

We want to use this variable directly so a short name is better.

running or waiting in resource group.

* Use built-in JSON parser for PXF fragments

Instead of relying on a external library, use the built-in JSON
parser in the backend for the PXF fragments parsing. Since this
replaces the current implementation with an event-based callback
parser, the code is more complicated, but dogfooding the parser
that we want extension writers to use is a good thing.

This removes the dependency on json-c from autoconf, and enables
building PXF on Travis for extra coverage.

* Use elog for internal errors, and ereport for user errors

Internal errors where we are interested in source filename should
use elog() which will decorate the error messages automatically
with this information. The connection error is interesting for the
user however, use ereport() instead there.

Migrations efforts for concourse 3 that are backwards compatible with
concourse 2.7.3

Includes using the image_resource in external task yamls for pipeline
jobs that use a sles based docker image. Currently unknown why the sles
image in particular causes image_resource issues.
Signed-off-by: NDivya Bhargov <dbhargov@pivotal.io>
Signed-off-by: NJim Doty <jdoty@pivotal.io>

This is the spec-compliant spelling, but GPDB has only allowed
"agg OVER (window)" so far. With this commit, the parens are still allowed,
for backwards-compatibility.

Change deparsing code to also use the non-parens syntax in view definitions
and EXPLAIN. Adjust expected output of regression tests accordingly.

We are now on postgres 8.4 which apparently activated some conditional
statements in describe.c. This one was missed from the first 8.4 merge
chunk which makes psql -l currently error out.

Reported by Brian Lu in the Greenplum Developers mailing list:
https://groups.google.com/a/greenplum.org/forum/#!topic/gpdb-dev/5l7J2j5yla8

Update README to explain prerequisites for
correct access to LSN fields of a page.
Independent chunk removed from checksums
patch to reduce size of patch.

(cherry picked from commit 1c563a2a)

'x' is eXtensions, 'E' is the External tables with the relstorage 'x'.

1) Give error meesage about proxy.
2) Change server_side_encryption default value from "none" to empty
string.
Signed-off-by: NXiaoran Wang <xiwang@pivotal.io>

    commit 3d009e45
    Author: Heikki Linnakangas <heikki.linnakangas@iki.fi>
    Date:   Wed Feb 27 18:17:21 2013 +0200

        Add support for piping COPY to/from an external program.

        This includes backend "COPY TO/FROM PROGRAM '...'" syntax, and corresponding
        psql \copy syntax. Like with reading/writing files, the backend version is
        superuser-only, and in the psql version, the program is run in the client.

        In the passing, the psql \copy STDIN/STDOUT syntax is subtly changed: if you
        the stdin/stdout is quoted, it's now interpreted as a filename. For example,
        "\copy foo from 'stdin'" now reads from a file called 'stdin', not from
        standard input. Before this, there was no way to specify a filename called
        stdin, stdout, pstdin or pstdout.

        This creates a new function in pgport, wait_result_to_str(), which can
        be used to convert the exit status of a process, as returned by wait(3),
        to a human-readable string.

        Etsuro Fujita, reviewed by Amit Kapila.
Signed-off-by: NAdam Lee <ali@pivotal.io>
Signed-off-by: NMing LI <mli@apache.org>

It seems that we can no longer use ${} in the env var fields so remove
them to get our Travis builds green again.

Reference:
https://github.com/travis-ci/travis-build/commit/a6779473124442fa748795fa9fd47afc529fc9d4

This utility used to confirm data transfered by doing a md5 digest. This
commit changes behavior to use sha256 instead.
Signed-off-by: NShoaib Lari <slari@pivotal.io>

Signed-off-by: NNadeem Ghani <nghani@pivotal.io>

Need flexibility to be able to control at what freqeuncy pipeline gets triggerd,
one such usecase is same pipleine code runs with asserts on every commit and
without asserts daily/weekly. So, adding a time resource via which can control
the triggering. Plus, parameterize the trigger to control the things.

Some backups with Data Domain contained an incorrect path in their
report file. When checking whether the backup timestamp was in a pre or
post content-id format, restore would fail since the path in the report
file didn't match the expected pattern. Instead, we now check for the
timestamp format in a more generalized way to account for this discrepancy.
Signed-off-by: NChris Hajas <chajas@pivotal.io>

It was noticed in CentOS 6 that the entries in the char* list would
become blank after freeing the raw string that is obtained from
setting the GUC wal_consistency_checking. We should free this string
only after we finish using it.

Authors: Jimmy Yih and Taylor Vesely

The MoveTransFilespaceLocally class in filespace.py used to verify that two directories
are equivalent by doing a md5 on their contents. This commit changes behavior to use
sha256 instead.
Signed-off-by: NNadeem Ghani <nghani@pivotal.io>

Signed-off-by: NShoaib Lari <slari@pivotal.io>

We no longer call contain_windowfunc() on raw parse trees, so we can
revert this to the way it is in the upstream.

This is the first merge commit, on our way to merging with PostgreSQL 8.4.

Nothing too exciting in this batch. Most of the included changes we had
in fact already backported earlier. But some things worth mentioning:

* This bumps up the PostgreSQL version number to 8.4devel. This might
  require 3rd party tools to be fixed to cope with that.

* Bumping the version number caused some \d queries in psql to fail,
  because they did a version check and tried to run queries that only work
  on PostgreSQL 8.4. This happens because the GPDB version of psql has been
  backported from 9.0. To make it work for now, I #ifdef'd out the version
  checks, and added GPDB_84_MERGE_FIXME comments to remind us to re-enable
  them again later, once we have merged the backend catalog changes that
  they need.

* Same for a some version checks in pg_upgrade.

* The build system now links the whole backend in one invocation, instead
  of using the per-directory intermediate SUBSYS.o files. We had already
  backported the Makefile changes for that, so this just flips the switch
  to start using it.

This has been a joint effort between Heikki Linnakangas, Daniel Gustafsson,
Jacob Champion and Tom Meyer.

The view gp_toolkit.gp_resgroup_status collect resgroup status
information on both QD and QEs, also before and after a 300ms delay to
measure the cpu usage. When there is concurrently resgroup drops it
might fail due to missing resgroups.

This is fixed by holding ExclusiveLock which blocks the drop operations.
Signed-off-by: NNing Yu <nyu@pivotal.io>

In some weak test agents, the shared semaphore is two small to support large
max_connections like 150, so refine test cases to make it work with
40 max_connections.

Previously in ResGroupSlotRelease(), we mark the resWaiting flag to
false out of the ResGroupLock, a race condition can happen as that
the waitProc that meant to be wake up is canceled just before it's
resWaiting is set to false and in the window resWaiting is set, the
waitProc run another query and wait for a free slot again, then
resWaiting is set and the waitProc will get another unexpected free slot.

eg: rg1 has concurrency = 1

s1: BEGIN;
s2: BEGIN; -- blocked
s1: END; -- set breakpoint just before resWaiting is set in ResGroupSlotRelease()
s2: cancel it;
s3: BEGIN; -- will get a free slot.
s2: BEGIN; -- run begin again and be blocked.
s1: continue;
s2: s2 will got unexpected free slot

To avoid leak of resgroup slot, we set resWaiting under ResGroupLock so the
waitProc have no window to re-fetch the slot before resWaiting is set.
It's hard to add a regression test, adding breakpoint in ResGroupSlotRelease
may block all other operations because it holds the ResGroupLock.

string overflow won't happen here because gpcheckcloud_newline was
checked before, but still, strncpy() looks better.
Signed-off-by: NXiaoran Wang <xiwang@pivotal.io>
Signed-off-by: NAdam Lee <ali@pivotal.io>

During NOT EXISTS sublink pullup, we create a one-time false filter when
the sublink contains aggregates without checking for limitcount. However
in situations where the sublink contains an aggregate with limit 0, we
should not generate such filter as it produces incorrect results.

Added regress test.

Also, initialize all the members of IncrementVarSublevelsUp_context
properly.
Signed-off-by: NDhanashree Kashid <dkashid@pivotal.io>

tqual.c is quite a long file, and there are plenty of legitimate
MPP-specific changes in it compared to upstream. Move these functions to
separate file, to reduce our diff footprint. This hopefully makes merging
easier in the future.

Run pgindent over the new file, and do some manual tidying up. Also, use
CStringGetTextDatum() instead of the more complicated dance with
DirectFunctionCall and textin(), to convert C strings into text Datums.

In the upstream, two different structs are used to represent a window
definition. WindowDef in the grammar, which is transformed into
WindowClause during parse analysis. In GPDB, we've been using the same
struct, WindowSpec, in both stages. Split it up, to match the upstream.

The representation of the window frame, i.e. "ROWS/RANGE BETWEEN ..." was
different between the upstream implementation and the GPDB one. We now use
the upstream frameOptions+startOffset+endOffset representation in raw
WindowDef parse node, but it's still converted to the WindowFrame
representation for the later stages, so WindowClause still uses that. I
will switch over the rest of the codebase to the upstream representation as
a separate patch.

Also, refactor WINDOW clause deparsing to be closer to upstream.

One notable difference is that the old WindowSpec.winspec field corresponds
to the winref field in WindowDef andWindowClause, except that the new
'winref' is 1-based, while the old field was 0-based.

Another noteworthy thing is that this forbids specifying "OVER (w
ROWS/RANGE BETWEEN ...", if the window "w" already specified a window frame,
i.e. a different ROWS/RANGE BETWEEN. There was one such case in the
regression suite, in window_views, and this updates the expected output of
that to be an error.

Previously, gpdbrestore would analyze all schemas in the database during
a schema-only restore. Now, only those schemas that were restored will
be analyzed.
Signed-off-by: NChris Hajas <chajas@pivotal.io>

With commit 387c485d winstar and winagg
fields were added in WindowRef Node, so this commit adds handling for
them in ORCA Translator.

I don't know what this was used for, but I don't see any code that would
send these messages anymore.

nMotionNodes tracks the number of Motion in a plan, and each
plan node maintains nMotionNodes. Counting number of Motions in a plan node by
traversing the tree and adding up nMotionNodes found in nested plans will give
incorrect number of Motion nodes. So instead of using nMotionNodes, use
a boolean flag to track if the subtree tree excluding the initplans
contains a motion node