Signed-off-by: NJim Doty <jdoty@pivotal.io>

There is an assert failure when Window's child is a HashJoin operator
and while filling its buffer Window receives a NULL tuple. In this case
HashJoin will call ExecEagerFreeHashJoin() since it is done returning
any tuples. However, Window once it has returned all the tuples in its
input buffer will call ExecProcNode() on HashJoin. This casues an assert
failure in HashJoin that states that ExecHashJoin() should not be called
if HashJoin's hashtable has already been released.

This commit fixes the above issue by setting a flag in WindowState when
Window encounters a null tuple while filling its buffer. This flag then
guards any subsequent call to ExecProcNode() from fetchCurrentRow()

This brought in postgres/postgres@44d5be0 pretty much wholesale, except:

1. We leave `WITH RECURSIVE` for a later commit. The code is brought in,
    but kept dormant by us bailing early at the parser whenever there is
    a recursive CTE.
2. We use `ShareInputScan` in the stead of `CteScan`. ShareInputScan is
    basically the parallel-capable `CteScan`. (See `set_cte_pathlist`
    and `create_ctescan_plan`)
3. Consequently we do not put the sub-plan for the CTE in a
    pseudo-initplan: it is directly present in the main plan tree
    instead, hence we disable `SS_process_ctes` inside
    `subquery_planner`
4. Another corollary is that all new operators (`CteScan`,
    `RecursiveUnion`, and `WorkTableScan`) are dead code right now. But
    they will come to live once we bring in parallel implementation of
    `WITH RECURSIVE`

In general this commit reduces the divergence between Greenplum and
upstream.

User visible changes:
The merge in parser enables a corner case previously treated as error:
you can now specify fewer columns in your `WITH` clause than the actual
projected columns in the body subquery of the `WITH`.

Original commit message:

> Implement SQL-standard WITH clauses, including WITH RECURSIVE.
>
> There are some unimplemented aspects: recursive queries must use UNION ALL
> (should allow UNION too), and we don't have SEARCH or CYCLE clauses.
> These might or might not get done for 8.4, but even without them it's a
> pretty useful feature.
>
> There are also a couple of small loose ends and definitional quibbles,
> which I'll send a memo about to pgsql-hackers shortly.  But let's land
> the patch now so we can get on with other development.
>
> Yoshiyuki Asaba, with lots of help from Tatsuo Ishii and Tom Lane
>

(cherry picked from commit 44d5be0e)

In an effort to make each secret have less access we are moving to use
git deploy keys to pull changes from github.

 For now GPDB and its submodules are public, so we should be able to
 pull without any keys.
Signed-off-by: NDivya Bhargov <dbhargov@pivotal.io>
Signed-off-by: NJim Doty <jdoty@pivotal.io>

Signed-off-by: NDivya Bhargov <dbhargov@pivotal.io>

* rewording/conditionalizing more Pivotal-specific info

* conditionalizing/editing/or getting rid of more Pivotal-specific references and info

This patch contains the changes to pipeline.yml to enable the testing of
segment wal replication feature on concourse.
Signed-off-by: NAbhijit Subramanya <asubramanya@pivotal.io>

The generate_ao_xlog/generate_aoco_xlog test would fail if there was a page
header or a continuation record in the XLOG sent by the sender since the
validation did not take them into consideration. This patch fixes the
validation of the received XLOG by detecting the page headers and continuation
records and making sure that it skips to the correct record boundary.
Signed-off-by: NAbhijit Subramanya <asubramanya@pivotal.io>

* Update text for deprecated LOG ERRORS <table>

* Remove deprecation note for gpdb5.

gpfdist and gpcloud were shifted to top level by below commits, should
be moved out of gpAux/Makefile.

    commit 6125ac85cae720f484d0d45042131f4b859779d2
    Author: Marbin Tan <mtan@pivotal.io>
    Date:   Fri Feb 5 11:23:06 2016 -0800

        Move gpfdist to gpdb core.

    commit 4e34d8bb
    Author: Adam Lee <ali@pivotal.io>
    Date:   Wed May 24 16:53:15 2017 +0800

        Add a build flag for gpcloud

Change tracking files are used to capture information on what's changed while
mirror was down, to help incrementally bring it back to sync. In some instances
mostly due to disk issues / full situations, if change tracking log was partial
written or got messed up, resulted in rolling PANIC of segment and there by DB
unavailable due to double fault. Only way out was manual intervention to remove
changetracking files and run full resync.

So, instead this commit now adds checksum protection to auto detect any problem
with change tracking files during recovery / incremental resync. If checksum
miss-match gets detected it takes preventive action to mark segment into
ChangeTrackingDisabled state and keeps DB available. Plus, explicitely enforces
only full recovery is allowed to bring mirror up in sync as changetracking info
doesn't exist. Any attenpt for incremental resync clearly communicates out full
resync has to be performed. So, this eliminates need for manual intervention to
get DB back to availble state if changetracking files get corrupted.

Removing unused function that got left over from cleanup.

CID 170478:  Control flow issues  (DEADCODE)
the "goto" function was dead code. We are closing the currently opened
file right before we open up a new file already, so the FILE pointer was
always NULL when "goto bail" occurred.

CID 170479: (Null pointer dereferenced in agg_put_qexec)

We removed too much in c0c1897f

Coverity detected that we were trying to access a pointer that is NULL
for this codepath. It would come up if the key value pair didn't yet
exist in the the apr_hash. Ensure that we allocate memory before doing
a memcpy.

Signed-off-by: NChris Hajas <chajas@pivotal.io>

We need to bump to CentOS 7 before we can undo this, because Red Hat
ships very old compilers :(

* Remove DDBoost Full and Incremental backup/restore TINC jobs
* Add behave backup/restore DDBoost job

The previous job took 3h, 50 mins and the new job takes 1h, 20 mins.

These tests were used to test backup/restore on Data Domain. They have
been replaced with behave tests.
Signed-off-by: NJamie McAtamney <jmcatamney@pivotal.io>

This is part of the effort to unify our backup/restore tests into a
single suite.

* Adds infrastructure to setup DDBoost on the client and cleanup server
* after completion
* Adds tests for DDBoost specific options
* Adds test coverage from TINC suite that was not included in behave
Signed-off-by: NChris Hajas <chajas@pivotal.io>

A test output in extprotocol is undeterministic in which case all
segments will report an error, you can not tell which segment will
report it firstly. Normally it's segment 0 first report that, but we
can not count on this.

Make sliceIndex, rootIndex, parentIndex and children near each other to see
slice debug output info easily.

Formerly, GPDB do dispatch/interconnect cleanup on executor level
which means once an error occurs within executor, it will be catched and
dispatch/interconnect will be cleaned. The problem is if an error occurs
after an executor started but before the executor run, dispatch/interconnect
has no chance to be cleaned up. A problem is that outbound UDP interconnect
packets still think the interconnect is active and will access the
memory that has been freed.

This commit add a few cleanup points on portal level, a higher call level
than executor to cover more cases shown as above. mppExecutorCleanup() is
reentrant, so it's ok to do double check on both level.

This statistic info is convenient for debugging and test cases

This reverts commit 9bea5037.

Formerly, GPDB do dispatch/interconnect cleanup on executor level
which means once an error occurs within executor, it will be catched and
dispatch/interconnect will be cleaned. The problem is if an error occurs
after an executor started but before the executor run, dispatch/interconnect
has no chance to be cleaned up. A problem is that outbound UDP interconnect
packets still think the interconnect is active and will access the
memory that has been freed.

This commit add a few cleanup points on portal level, a higher call level
than executor to cover more cases shown as above. mppExecutorCleanup() is
reentrant, so it's ok to do double check on both level.

Since Command creates a short-lived SSH session, we observe the PID given
a throw-away remote process. Assume that the PID is unused and available on
the remote in the near future.

This pid is no longer associated with a
running process and won't be recycled for long enough that tests
have finished.

Looking ahead introduces the risk of a time-of-check-time-of-use race
since the pid might have been allocated by the operating system by
the time the test would use the data.

because it was causing storage/basic/lock TINC test to fail.

This reverts commit 243121f9.

Signed-off-by: NTushar Dadlani <tdadlani@pivotal.io>

Signed-off-by: NLarry Hamel <lhamel@pivotal.io>

Signed-off-by: NTushar Dadlani <tdadlani@pivotal.io>

* docs: correct the gpperfmon_install command syntax

* Correct gpperfmon_install command syntax in help text

* Remove references to SHA-256-FIPS

* gpperfmon_install docs: Clarify options syntax
Signed-off-by: NC.J. Jameson <cjameson@pivotal.io>