* Docs - change default log_rotation_size to 1GB

- Also clarify how and when log rotation occurs, i.e., log_rotation_size triggers rotation, it's not a maximum file size.
- For log_rotation_age, log is rotated when the current log file reaches specified age, not "once a day".

* Update from review comments

* Updates from review

* docs - add content for lock_timeout guc

* add note about relationship to deadlock timeouts

This is the equivalent of
https://github.com/greenplum-db/gporca/pull/462,
"Normalize expression before inferring predicates".

Backported from master commit 2b78a302

Reported-by: Cyrille Lintz

Joining nested REURSIVE clauses is planned as a join between
two WorkTableScan nodes, which we currently cannot do. Detect
and disallow for now until we have the required infrastructure
to handle this class of queries. The below query is an example
of this:

  WITH RECURSIVE r1 AS (
      SELECT 1 AS a
      UNION ALL
       (
          WITH RECURSIVE r2 AS (
              SELECT 2 as b
              UNION ALL
              SELECT b FROM r1, r2
          )
          SELECT b FROM r2
      )
  )
  SELECT * FROM r1 LIMIT 1;

In upstream PostgreSQL, the resulting plan exhibits the same
behavior as in GPDB, but there is no restiction on WorkTableScan
on the inner side of joins in PostgreSQL:

                           QUERY PLAN
  -------------------------------------------------------------
   Limit
     CTE r1
       ->  Recursive Union
             ->  Result
             ->  CTE Scan on r2 r2_1
                   CTE r2
                     ->  Recursive Union
                           ->  Result
                           ->  Nested Loop
                                 ->  WorkTable Scan on r1 r1_1
                                 ->  WorkTable Scan on r2
     ->  CTE Scan on r1
  (12 rows)

Backported from master commit e9c0e77a

Upgrade Behave to 1.2.6

(cherry picked from commit 3dd52c8b)

wait_for_mirror_sync() uses contentid as argument, but the missing_xlog test
case uses dbid as input, so in this test wait_for_mirror_sync() is not really
useful - this makes the test flaky.
Co-authored-by: NHaozhou Wang <hawang@pivotal.io>

Reviewed-by: Jimmy Yih

unlogged_heap_tables test occasionally fails since during primary restarting
there is a possibility that fts switches the mirrors to primaries.  For
unlogged table, there is no data on the mirrors so select query could return
different results if there is or there is no switch, thus makes the test not
deterministic.

We disable fts probe in this test using gp fault injector to fix this issue.
Co-authored-by: NHaozhou Wang <hawang@pivotal.io>
Reviewed-by: NAshwin Agrawal <aagrawal@pivotal.io>
Reviewed-by: NJimmy Yih <jyih@pivotal.io>

Previously the test waits no more than 30 seconds until mirror promotion
finished, but 30 seconds are not enough in a real test environment. In this
patch, we increase the guc gp_gang_creation_retry_timer from 250 (ms) to 1000
(ms) so we that we wait for 2 minutes at most, given guc
gp_gang_creation_retry_count has been configured with its maximum value.
Co-authored-by: NHaozhou Wang <hawang@pivotal.io>

We've removed SRF-in-targetlist support for most node types in GPDB.
Instead, we insert Result nodes to evaluate the set-returning-functions.
This can be benefit to performance.

However, Result node cannot evaluate WindowFunc, which an WindowAgg's
target list usually has. So we need to support SRF-in-targetlist for
WindowAgg nodes.

This is the same logic as for AggState.
Reviewed-by: NHeikki Linnakangas <hlinnakangas@pivotal.io>
Reviewed-by: NTaylor Vesely <tvesely@pivotal.io>

It is observed that before gp_replica_check some mirrors are down according to
the 'gpstop -u' output for unknown reason. Not sure why. Probably there is bug
in previous test, or probably oom tests affects. In any case we should not
proceed if the cluster is not ok. Of course we could restart the cluster but
we'd error out instead so that people could find potential issues in previous
testing.
Co-authored-by: NHaozhou Wang <hawang@pivotal.io>

Reviewed-by: Jimmy Yih

It will assert fail at this

  mdcreate (reln=0x2ac5548, forkNum=INIT_FORKNUM, isRedo=0 '\000') at md.c:288
    ExceptionalCondition (conditionName=0xe7cc00 "!(reln->md_fd[forkNum] == ((void *)0))", errorType=0xe7cbbe "FailedAssertion", fileName=0xe7cbb9 "md.c", lineNumber=288) at assert.c:46

This fixes https://github.com/greenplum-db/gpdb/issues/7340

Reported-by: zhbp366@github
Reviewed-by: NJimmy Yih <jyih@pivotal.io>
Reviewed-by: NTaylor Vesely <tvesely@pivotal.io>

These files don't exist in the gpdb repo, nor the bin_gpdb tar.gz
resources in Concourse for 6 or 5 as far as we can tell.
Co-authored-by: NAmil Khanzada <akhanzada@pivotal.io>
Co-authored-by: NBen Christel <bchristel@pivotal.io>

User defined functions cache their plan, therefore if we modify the
plan during execution, we risk having invalid data during the next
execution of the cached plan.

ExecuteTruncate modifies the plan's relations when declaring partitions
that need truncation.

Instead, we copy the list of relations that need truncation and add the
partition relations to the copy.

(cherry picked from commit e3cf4f26)

Commit ea9df812 relaxes the requirement
that all LWLocks are in a single array, which broke the LOCK_DEBUG
code for trying to acquiring an LWLock. This is a meatball surgery
attempt at making it at least compile and run, without trying to
reinvent the old usecase.

Minor tweaking of the code is also done to make LOCK_DEBUG cases be
identified identically.

Backported from master 3a1ced8f
Reported-by: Simon Gao in Github issue #7251
Reviewed-by: Ashwin Agrawal

* Create a public dockerfile for compiling gpdb on ubuntu
* Create directory /usr/local/greenplum-db-devel for installing greenplum
* Move current ubuntu files to ubuntu16_ppa. Update readme

* docs - remove ALTER SYSTEM reference

* Restore source for ALTER SYSTEM ref, but leave out of maps.

* Add ALTER SYSTEM to list of unsupported PostgreSQL features

* Docs - removes replication port references

Also replaces gpstate -p command output with more recent version

* add missing line break to example

Commit 99360f54 extended the offset
numbers in the GIN posting list to 16 bits over upstream who use
11 bits. This however means that the assertion around ip_posid is
no longer valid as it's always true, causing a compiler warning:

ginpostinglist.c:89:24: warning: result of comparison of constant
                                 65536 with expression of type
								 'OffsetNumber' (aka 'unsigned short')
								 is always true
[-Wtautological-constant-out-of-range-compare]
Assert(iptr->ip_posid < (1 << MaxHeapTuplesPerPageBits));
~~~~~~~~~~~~~~~~~~~~~~^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
../../../../src/include/c.h:784:10: note: expanded from macro 'Assert'
Trap(!(condition), "FailedAssertion")
~~~~~~~^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
../../../../src/include/c.h:766:28: note: expanded from macro 'Trap'
if ((assert_enabled) && (condition))
^~~~~~~~~
1 warning generated.

Fix by commenting out the assertion with preprocessor blocks in an
attempt to make future merge conflicts easier to resolve.

A previous version of this patch was reviewed by Adam Berlin.

The level of TERSE or greater was set during development, we can lower
this to DEBUG now that this feature has stablized.

It fails and appears to not be maintained when:

--enable-gpfdist and --with-openssl are configured.

(cherry picked from commit 813528e7)

Because the gpinitstandby flag for passing in a standby master data directory
was changed from -F to -S, this commit changes -F to -S in gpinitsystem to
match.

-S was already in use for specifying that the cluster should have its mirrors
set up in spread configuration, so that flag is changed to --mirror-mode, which
accepts the mirror mode ('group' or 'spread') as an argument.
Co-authored-by: NJamie McAtamney <jmcatamney@pivotal.io>
Co-authored-by: NMark Sliva <msliva@pivotal.io>
(cherry picked from commit 6fa3bce8)

Despite my best efforts to avoid races on overloaded test containers,
test_join_and_indicate_progress_prints_dots_until_pool_is_done() has
been failing fairly often. Replace the simple-but-flaky time-based test
with an implementation that serializes the components of the race.

(cherry picked from commit 9f5384bc)

* REASSIGN OWNED. edits. Remove qualification that it doesn't change the database ownership

* ALTER FUNCTION. set from current clause description revision

* COMMENT. capitalize proper noun.

* COPY. describe PROGRAM option.

* CREATE AGGREGATE. Implements new syntax.

* CREATE FUNCTION. edits.

* DROP ROLE. small edit, link to other commands

* DROP USER. trivial edit.

* PREPARE. small edits

* REASSIGN OWNED. Trivial edits.

* REINDEX. trivial edit.

* pg_dump. use --quote-all-identifers for cross-version dumps

* Additional edits

* Updates from review

* Updates from review

6_Backport: show gprecoverseg progress

- Read PXF host/port from environment variables
- Add functions to get the PXF port and host
- Add unit tests for get_pxf_port get_pxf_host
We store the current value of the environment variable, then we set a
custom host/port, run the tests, and finally restore the value of the
environment variable.
- Handle error when parsing PXF_PORT

- The transaction ID and segment ID are only used by a subset of
  profiles. PXF should be able to append the transaction and segment IDs
  depending on the profile using the header information.
- Remove write_file_name, do not send path

Co-authored-by: NOliver Albertini <oalbertini@pivotal.io>
Co-authored-by: NBradford D. Boyle <bboyle@pivotal.io>

Previously was using master.
Co-authored-by: NOliver Albertini <oalbertini@pivotal.io>
Co-authored-by: NBradford D. Boyle <bboyle@pivotal.io>

Postgres optimizes the storage of the ItemPointer in the Gin Posting
List. It only stores 11 bits for the offset number because heap tables
only have enough tuples per block to fit in 11 bits. However,
Greenplum append-only tables store 16 bits worth of offset numbers.

Initially we thought we'd need to modify decode_varbyte() but it turns
out that it is OK.  It handles 48 bits already.
Co-authored-by: NAlexandra Wang <lewang@pivotal.com>
Co-authored-by: NAshwin Agrawal <aagrawal@pivotal.io>

(cherry picked from commit 5e918bf6)

* docs - remove gpsyncmaster and add pg_basebackup to backend program list.

* Updates from review

Make "demo_cluster" the default tag (ie: no tag). That is, when a
scenario does not specify a behave tag it uses the demo cluster
environment to run. Use the concourse_cluster tag if a scenario requires
a remote cluster. Concourse runs the scenarios with no tag that use a
demo cluster by filtering for --tags=~concourse_cluster.
Authored-by: NKalen Krempely <kkrempely@pivotal.io>
(cherry picked from commit 40028cf4)