test_RemoteOperation_logger_debug() has been flaking out on the CI
pipeline, and there's no indication of what is going wrong. Replace the
assertTrue() call, which gives no indication of the difference between
actual and expected, with mock.assert_has_calls(), which will tell us
exactly what the calls were in case of failure.

It's possible that this will fix the flake entirely. The previous test
implementation depended on logger.debug() to be called *first* with our
expected output, but given the poor isolation of our global logger
system, it's entirely possible that some other code occasionally calls
debug(). (That this is an issue at all indicates that this isn't really
a unit test, but that's not something to tackle here.) assert_has_calls()
doesn't mind how many other calls happen as long as the one we're
looking for is eventually made, and I think that matches the intent of
the test better anyway.

Backport to 6X_STABLE.

The metadata_track suite was originally part of cdbfast, dating back
around 10-11 years. It was later moved into bugbuster around six years
ago, already then with doubts as to what it actually did. After the
open sourcing we scrapped bugbuster, moving anything worthwhile (or
just not analyzed for usefulness yet) into the normal regress schedule
which is where metadata_track remained till now. This all according to
memory and the old proprietary issue tracker.

Looking at metadata_track, it's entirely duplicative only issuing
lots of DDL already tested elsewhere without verifying the results,
most likely since it was originally testing the metadata tracking
of the operations. Since the latter part is no longer happening,
move parts of the test into the existing pg_stat_last_operation
test and remove the rest, as the remaining value of this quite slow
test (spending ~10-12 minutes serially in the pipeline) is highly
debatable.

The existing pg_stat_last_operation was, and I quote, "underwhelming"
so most of it is replaced herein. There is still work to be done in
order to boost metadata tracking test coverage, but this is at least
a start.

Reviewed-by: Jimmy Yih

* Create a public dockerfile for compiling gpdb on ubuntu
* Create directory /usr/local/greenplum-db-devel for installing greenplum
* Move current ubuntu files to ubuntu16_ppa. Update readme

* docs - remove ALTER SYSTEM reference

* Restore source for ALTER SYSTEM ref, but leave out of maps.

* Add ALTER SYSTEM to list of unsupported PostgreSQL features

* Docs - removes replication port references

Also replaces gpstate -p command output with more recent version

* add missing line break to example

Joining nested REURSIVE clauses is planned as a join between
two WorkTableScan nodes, which we currently cannot do. Detect
and disallow for now until we have the required infrastructure
to handle this class of queries. The below query is an example
of this:

  WITH RECURSIVE r1 AS (
      SELECT 1 AS a
      UNION ALL
       (
          WITH RECURSIVE r2 AS (
              SELECT 2 as b
              UNION ALL
              SELECT b FROM r1, r2
          )
          SELECT b FROM r2
      )
  )
  SELECT * FROM r1 LIMIT 1;

In upstream PostgreSQL, the resulting plan exhibits the same
behavior as in GPDB, but there is no restiction on WorkTableScan
on the inner side of joins in PostgreSQL:

                           QUERY PLAN
  -------------------------------------------------------------
   Limit
     CTE r1
       ->  Recursive Union
             ->  Result
             ->  CTE Scan on r2 r2_1
                   CTE r2
                     ->  Recursive Union
                           ->  Result
                           ->  Nested Loop
                                 ->  WorkTable Scan on r1 r1_1
                                 ->  WorkTable Scan on r2
     ->  CTE Scan on r1
  (12 rows)

Backport to 6X_STABLE as it's a live bug there.

pg_upgrade can potentially preassign quite a large number of Oids
when upgrading, so the List would be a bottleneck during lookups
of Oids allocated to non-preassigned objects. This replaces the
list with a Red-Black tree to achieve O(log n) lookups at the cost
of slower inserts.

Judging by the historical repos, it was never used anywhere.

Reviewed-by: Jimmy Yih

* REASSIGN OWNED. edits. Remove qualification that it doesn't change the database ownership

* ALTER FUNCTION. set from current clause description revision

* COMMENT. capitalize proper noun.

* COPY. describe PROGRAM option.

* CREATE AGGREGATE. Implements new syntax.

* CREATE FUNCTION. edits.

* DROP ROLE. small edit, link to other commands

* DROP USER. trivial edit.

* PREPARE. small edits

* REASSIGN OWNED. Trivial edits.

* REINDEX. trivial edit.

* pg_dump. use --quote-all-identifers for cross-version dumps

* Additional edits

* Updates from review

* Updates from review

It will assert fail at this

  mdcreate (reln=0x2ac5548, forkNum=INIT_FORKNUM, isRedo=0 '\000') at md.c:288
    ExceptionalCondition (conditionName=0xe7cc00 "!(reln->md_fd[forkNum] == ((void *)0))", errorType=0xe7cbbe "FailedAssertion", fileName=0xe7cbb9 "md.c", lineNumber=288) at assert.c:46

This fixes https://github.com/greenplum-db/gpdb/issues/7340

Reported-by: zhbp366@github
Reviewed-by: NJimmy Yih <jyih@pivotal.io>
Reviewed-by: NTaylor Vesely <tvesely@pivotal.io>

unlogged_heap_tables test occasionally fails since during primary restarting
there is a possibility that fts switches the mirrors to primaries.  For
unlogged table, there is no data on the mirrors so select query could return
different results if there is or there is no switch, thus makes the test not
deterministic.

We disable fts probe in this test using gp fault injector to fix this issue.
Co-authored-by: NHaozhou Wang <hawang@pivotal.io>
Reviewed-by: NAshwin Agrawal <aagrawal@pivotal.io>
Reviewed-by: NJimmy Yih <jyih@pivotal.io>

Because the gpinitstandby flag for passing in a standby master data directory
was changed from -F to -S, this commit changes -F to -S in gpinitsystem to
match.

-S was already in use for specifying that the cluster should have its mirrors
set up in spread configuration, so that flag is changed to --mirror-mode, which
accepts the mirror mode ('group' or 'spread') as an argument.
Co-authored-by: NJamie McAtamney <jmcatamney@pivotal.io>
Co-authored-by: NMark Sliva <msliva@pivotal.io>

The level of TERSE or greater was set during development, we can lower
this to DEBUG now that this feature has stablized.

- Read PXF host/port from environment variables
- Add functions to get the PXF port and host
- Add unit tests for get_pxf_port get_pxf_host
We store the current value of the environment variable, then we set a
custom host/port, run the tests, and finally restore the value of the
environment variable.
- Handle error when parsing PXF_PORT

- The transaction ID and segment ID are only used by a subset of
  profiles. PXF should be able to append the transaction and segment IDs
  depending on the profile using the header information.
- Remove write_file_name, do not send path

These files don't exist in the gpdb repo, nor the bin_gpdb tar.gz
resources in Concourse for 6 or 5 as far as we can tell.
Co-authored-by: NAmil Khanzada <akhanzada@pivotal.io>
Co-authored-by: NBen Christel <bchristel@pivotal.io>

* docs - remove gpsyncmaster and add pg_basebackup to backend program list.

* Updates from review

`create rule <...> do instead select * from t for update`
will dispatch a query with lockingclause node. Add
deserialization method for it to make things correct.

Commit 99360f54 extended the offset
numbers in the GIN posting list to 16 bits over upstream who use
11 bits. This however means that the assertion around ip_posid is
no longer valid as it's always true, causing a compiler warning:

ginpostinglist.c:89:24: warning: result of comparison of constant
                                 65536 with expression of type
								 'OffsetNumber' (aka 'unsigned short')
								 is always true
[-Wtautological-constant-out-of-range-compare]
Assert(iptr->ip_posid < (1 << MaxHeapTuplesPerPageBits));
~~~~~~~~~~~~~~~~~~~~~~^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
../../../../src/include/c.h:784:10: note: expanded from macro 'Assert'
Trap(!(condition), "FailedAssertion")
~~~~~~~^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
../../../../src/include/c.h:766:28: note: expanded from macro 'Trap'
if ((assert_enabled) && (condition))
^~~~~~~~~
1 warning generated.

Fix by commenting out the assertion with preprocessor blocks in an
attempt to make future merge conflicts easier to resolve.

A previous version of this patch was reviewed by Adam Berlin.

Move gp_segment_connect_timeout before gp_inject_fault in dispatch test to make the
ICW stable in a slow system. The previous gp_segment_connect_timeout is
set to 1 second, which may cause gp_inject_fault timeout if the system is slow.
Co-authored-by: NTang Pengzhou <ptang@pivotal.io>

eg. tar xzf "*.tar.gz" does not work.

  Remove the double quotes, it works
Authored-by: NBob Bao <bbao@pivotal.io>

We've removed SRF-in-targetlist support for most node types in GPDB.
Instead, we insert Result nodes to evaluate the set-returning-functions.
This can be benefit to performance.

However, Result node cannot evaluate WindowFunc, which an WindowAgg's
target list usually has. So we need to support SRF-in-targetlist for
WindowAgg nodes.

This is the same logic as for AggState.
Reviewed-by: NHeikki Linnakangas <hlinnakangas@pivotal.io>
Reviewed-by: NTaylor Vesely <tvesely@pivotal.io>

* docs - pxf jdbc server file-based config

* add apache ignite to list of dbs supported by jdbc connector

* misc edits requested by david

Each gpexpand test case need a new environment.
Previous code uses pkill to make sure no postgres
process remaining and then uses gpstop to double
confirm.

However, pkill to terminate cluster may over-kill
some other postgres process on the hosts. What's
more, the interface `Given a working directory of
the test as '/data/gpdata/gpexpand'` will remove
the directory which also remove the $MASTER_DATA_DIRECTORY,
so the following `gpstop` is useless because
$MASTER_DATA_DIRECTORY is gone. Also, this introduce
race conditon.

This commit removes pkill and use gpstop interface
to build a clean environment for each test case. Also
it makes sure that `gpstop` is invoked at the very
beginning of each test case.

Make "demo_cluster" the default tag (ie: no tag). That is, when a
scenario does not specify a behave tag it uses the demo cluster
environment to run. Use the concourse_cluster tag if a scenario requires
a remote cluster. Concourse runs the scenarios with no tag that use a
demo cluster by filtering for --tags=~concourse_cluster.
Authored-by: NKalen Krempely <kkrempely@pivotal.io>

Regenerate pipeline from CLI_BEHAVE_TESTS refactor.

Renamed to CLI_BEHAVE_TESTS for clarity

Tag existing tests with @demo_cluster and @concourse_cluster
annotations. The @demo_cluster tag denotes the scenario can run
locally. The @concourse_cluster tag denotes the scenario that requires
a remote cluster.
Co-authored-by: NJamie McAtamney <jmcatamney@pivotal.io>
Authored-by: NKalen Krempely <kkrempely@pivotal.io>

Now that the previous commits have placed these tests in CLI_TESTS,
we can remove references to this symbol.

Tag existing tests with @demo_cluster and @concourse_cluster
annotations. The @demo_cluster tag denotes the scenario can run
locally. The @concourse_cluster tag denotes the scenario that requires
a remote cluster.
Co-authored-by: NJamie McAtamney <jmcatamney@pivotal.io>
Authored-by: NKalen Krempely <kkrempely@pivotal.io>

Tag existing tests with @demo_cluster and @concourse_cluster
annotations. The @demo_cluster tag denotes the scenario can run
locally. The @concourse_cluster tag denotes the scenario that requires
a remote cluster.
Co-authored-by: NJamie McAtamney <jmcatamney@pivotal.io>
Authored-by: NKalen Krempely <kkrempely@pivotal.io>