test_RemoteOperation_logger_debug() has been flaking out on the CI
pipeline, and there's no indication of what is going wrong. Replace the
assertTrue() call, which gives no indication of the difference between
actual and expected, with mock.assert_has_calls(), which will tell us
exactly what the calls were in case of failure.

It's possible that this will fix the flake entirely. The previous test
implementation depended on logger.debug() to be called *first* with our
expected output, but given the poor isolation of our global logger
system, it's entirely possible that some other code occasionally calls
debug(). (That this is an issue at all indicates that this isn't really
a unit test, but that's not something to tackle here.) assert_has_calls()
doesn't mind how many other calls happen as long as the one we're
looking for is eventually made, and I think that matches the intent of
the test better anyway.

Backport to 6X_STABLE.

(cherry picked from commit 0fe07f49)

In the case where the user wants to change the port of a mirror, but
leave it in the same directory on the same host, gpmovemirrors was
previously deleting the directory as part of the cleanup because it
did not recognize that the old and new directories were the same.
This commit adds a check to prevent that.

This commit also logs a warning if the move_config_file contains
identical attributes(host,port,data directory) for the old and new
mirror.
Co-authored-by: NJamie McAtamney <jmcatamney@pivotal.io>
Co-authored-by: NShoaib Lari <slari@pivotal.io>
Co-authored-by: NDavid Krieger <dkrieger@pivotal.io>
(cherry picked from commit d15258a4)

While actual logic relating to filespaces was already removed from gpmovemirrors
in a previous commit, there were still some comments and dead code referencing
filespaces, which this commit removes.
Co-authored-by: NJamie McAtamney <jmcatamney@pivotal.io>
Co-authored-by: NShoaib Lari <slari@pivotal.io>
(cherry picked from commit b54840bc)

This corresponds to ORCA commit "Use prime number of buckets to avoid
hash collisions"
Authored-by: NChris Hajas <chajas@pivotal.io>

1.It's a known issue with gpperfmon that cpu skew and row skew information is inaccurate.
 We remove this flaky case for now.
2.GPDB6 rename table gp_distribution_policy colum attnum distkey.
Co-authored-by: NWenlin Zhang <wzhang@pivotal.io>
Co-authored-by: NBing Xu <bxu@pivotal.io>

We don't support Greenplum on these platforms.

Some files (e.g. Makefile.{hpux,solaris}) have been left in place
because they are upstream postgres files. Removing them isn't
worth the headache it would cause when merging commits from
postgres.

Cherry-picked from 52c37372Authored-by: NBen Christel <bchristel@pivotal.io>

* docs - change gpinitsystem -S option to --mirror-mode; change -F to -S

* sentence edit

* Address review comments about mirroring modes

* boldface variable portions of names for readability

* In params list, put short option before the long, consistently.

* Docs - change default log_rotation_size to 1GB

- Also clarify how and when log rotation occurs, i.e., log_rotation_size triggers rotation, it's not a maximum file size.
- For log_rotation_age, log is rotated when the current log file reaches specified age, not "once a day".

* Update from review comments

* Updates from review

* docs - add content for lock_timeout guc

* add note about relationship to deadlock timeouts

This is the equivalent of
https://github.com/greenplum-db/gporca/pull/462,
"Normalize expression before inferring predicates".

Backported from master commit 2b78a302

Reported-by: Cyrille Lintz

Joining nested REURSIVE clauses is planned as a join between
two WorkTableScan nodes, which we currently cannot do. Detect
and disallow for now until we have the required infrastructure
to handle this class of queries. The below query is an example
of this:

  WITH RECURSIVE r1 AS (
      SELECT 1 AS a
      UNION ALL
       (
          WITH RECURSIVE r2 AS (
              SELECT 2 as b
              UNION ALL
              SELECT b FROM r1, r2
          )
          SELECT b FROM r2
      )
  )
  SELECT * FROM r1 LIMIT 1;

In upstream PostgreSQL, the resulting plan exhibits the same
behavior as in GPDB, but there is no restiction on WorkTableScan
on the inner side of joins in PostgreSQL:

                           QUERY PLAN
  -------------------------------------------------------------
   Limit
     CTE r1
       ->  Recursive Union
             ->  Result
             ->  CTE Scan on r2 r2_1
                   CTE r2
                     ->  Recursive Union
                           ->  Result
                           ->  Nested Loop
                                 ->  WorkTable Scan on r1 r1_1
                                 ->  WorkTable Scan on r2
     ->  CTE Scan on r1
  (12 rows)

Backported from master commit e9c0e77a

Upgrade Behave to 1.2.6

(cherry picked from commit 3dd52c8b)

wait_for_mirror_sync() uses contentid as argument, but the missing_xlog test
case uses dbid as input, so in this test wait_for_mirror_sync() is not really
useful - this makes the test flaky.
Co-authored-by: NHaozhou Wang <hawang@pivotal.io>

Reviewed-by: Jimmy Yih

unlogged_heap_tables test occasionally fails since during primary restarting
there is a possibility that fts switches the mirrors to primaries.  For
unlogged table, there is no data on the mirrors so select query could return
different results if there is or there is no switch, thus makes the test not
deterministic.

We disable fts probe in this test using gp fault injector to fix this issue.
Co-authored-by: NHaozhou Wang <hawang@pivotal.io>
Reviewed-by: NAshwin Agrawal <aagrawal@pivotal.io>
Reviewed-by: NJimmy Yih <jyih@pivotal.io>

Previously the test waits no more than 30 seconds until mirror promotion
finished, but 30 seconds are not enough in a real test environment. In this
patch, we increase the guc gp_gang_creation_retry_timer from 250 (ms) to 1000
(ms) so we that we wait for 2 minutes at most, given guc
gp_gang_creation_retry_count has been configured with its maximum value.
Co-authored-by: NHaozhou Wang <hawang@pivotal.io>

We've removed SRF-in-targetlist support for most node types in GPDB.
Instead, we insert Result nodes to evaluate the set-returning-functions.
This can be benefit to performance.

However, Result node cannot evaluate WindowFunc, which an WindowAgg's
target list usually has. So we need to support SRF-in-targetlist for
WindowAgg nodes.

This is the same logic as for AggState.
Reviewed-by: NHeikki Linnakangas <hlinnakangas@pivotal.io>
Reviewed-by: NTaylor Vesely <tvesely@pivotal.io>

It is observed that before gp_replica_check some mirrors are down according to
the 'gpstop -u' output for unknown reason. Not sure why. Probably there is bug
in previous test, or probably oom tests affects. In any case we should not
proceed if the cluster is not ok. Of course we could restart the cluster but
we'd error out instead so that people could find potential issues in previous
testing.
Co-authored-by: NHaozhou Wang <hawang@pivotal.io>

Reviewed-by: Jimmy Yih

It will assert fail at this

  mdcreate (reln=0x2ac5548, forkNum=INIT_FORKNUM, isRedo=0 '\000') at md.c:288
    ExceptionalCondition (conditionName=0xe7cc00 "!(reln->md_fd[forkNum] == ((void *)0))", errorType=0xe7cbbe "FailedAssertion", fileName=0xe7cbb9 "md.c", lineNumber=288) at assert.c:46

This fixes https://github.com/greenplum-db/gpdb/issues/7340

Reported-by: zhbp366@github
Reviewed-by: NJimmy Yih <jyih@pivotal.io>
Reviewed-by: NTaylor Vesely <tvesely@pivotal.io>

These files don't exist in the gpdb repo, nor the bin_gpdb tar.gz
resources in Concourse for 6 or 5 as far as we can tell.
Co-authored-by: NAmil Khanzada <akhanzada@pivotal.io>
Co-authored-by: NBen Christel <bchristel@pivotal.io>

User defined functions cache their plan, therefore if we modify the
plan during execution, we risk having invalid data during the next
execution of the cached plan.

ExecuteTruncate modifies the plan's relations when declaring partitions
that need truncation.

Instead, we copy the list of relations that need truncation and add the
partition relations to the copy.

(cherry picked from commit e3cf4f26)

Commit ea9df812 relaxes the requirement
that all LWLocks are in a single array, which broke the LOCK_DEBUG
code for trying to acquiring an LWLock. This is a meatball surgery
attempt at making it at least compile and run, without trying to
reinvent the old usecase.

Minor tweaking of the code is also done to make LOCK_DEBUG cases be
identified identically.

Backported from master 3a1ced8f
Reported-by: Simon Gao in Github issue #7251
Reviewed-by: Ashwin Agrawal

* Create a public dockerfile for compiling gpdb on ubuntu
* Create directory /usr/local/greenplum-db-devel for installing greenplum
* Move current ubuntu files to ubuntu16_ppa. Update readme

* docs - remove ALTER SYSTEM reference

* Restore source for ALTER SYSTEM ref, but leave out of maps.

* Add ALTER SYSTEM to list of unsupported PostgreSQL features

* Docs - removes replication port references

Also replaces gpstate -p command output with more recent version

* add missing line break to example

Commit 99360f54 extended the offset
numbers in the GIN posting list to 16 bits over upstream who use
11 bits. This however means that the assertion around ip_posid is
no longer valid as it's always true, causing a compiler warning:

ginpostinglist.c:89:24: warning: result of comparison of constant
                                 65536 with expression of type
								 'OffsetNumber' (aka 'unsigned short')
								 is always true
[-Wtautological-constant-out-of-range-compare]
Assert(iptr->ip_posid < (1 << MaxHeapTuplesPerPageBits));
~~~~~~~~~~~~~~~~~~~~~~^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
../../../../src/include/c.h:784:10: note: expanded from macro 'Assert'
Trap(!(condition), "FailedAssertion")
~~~~~~~^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
../../../../src/include/c.h:766:28: note: expanded from macro 'Trap'
if ((assert_enabled) && (condition))
^~~~~~~~~
1 warning generated.

Fix by commenting out the assertion with preprocessor blocks in an
attempt to make future merge conflicts easier to resolve.

A previous version of this patch was reviewed by Adam Berlin.

The level of TERSE or greater was set during development, we can lower
this to DEBUG now that this feature has stablized.

It fails and appears to not be maintained when:

--enable-gpfdist and --with-openssl are configured.

(cherry picked from commit 813528e7)

Because the gpinitstandby flag for passing in a standby master data directory
was changed from -F to -S, this commit changes -F to -S in gpinitsystem to
match.

-S was already in use for specifying that the cluster should have its mirrors
set up in spread configuration, so that flag is changed to --mirror-mode, which
accepts the mirror mode ('group' or 'spread') as an argument.
Co-authored-by: NJamie McAtamney <jmcatamney@pivotal.io>
Co-authored-by: NMark Sliva <msliva@pivotal.io>
(cherry picked from commit 6fa3bce8)

Despite my best efforts to avoid races on overloaded test containers,
test_join_and_indicate_progress_prints_dots_until_pool_is_done() has
been failing fairly often. Replace the simple-but-flaky time-based test
with an implementation that serializes the components of the race.

(cherry picked from commit 9f5384bc)

* REASSIGN OWNED. edits. Remove qualification that it doesn't change the database ownership

* ALTER FUNCTION. set from current clause description revision

* COMMENT. capitalize proper noun.

* COPY. describe PROGRAM option.

* CREATE AGGREGATE. Implements new syntax.

* CREATE FUNCTION. edits.

* DROP ROLE. small edit, link to other commands

* DROP USER. trivial edit.

* PREPARE. small edits

* REASSIGN OWNED. Trivial edits.

* REINDEX. trivial edit.

* pg_dump. use --quote-all-identifers for cross-version dumps

* Additional edits

* Updates from review

* Updates from review

6_Backport: show gprecoverseg progress

- Read PXF host/port from environment variables
- Add functions to get the PXF port and host
- Add unit tests for get_pxf_port get_pxf_host
We store the current value of the environment variable, then we set a
custom host/port, run the tests, and finally restore the value of the
environment variable.
- Handle error when parsing PXF_PORT

- The transaction ID and segment ID are only used by a subset of
  profiles. PXF should be able to append the transaction and segment IDs
  depending on the profile using the header information.
- Remove write_file_name, do not send path