Full joins are sub-optimal in ORCA as they are implemented as a UNION of
Left Outer Join AND Left Anti-Semi Join. However, GPDB provides a full
outer join operator. Therefore, until ORCA implements a more optimal
FULL JOIN, it will fall back to the Postgres legacy query optimizer for
queries with FULL JOINs.

Co-authored by: Sambitesh Dash sdash@pivotal.io
Co-authored by: Ashuka Xue axue@pivotal.io

Issue is encountered because XLogReaderState does not make any
guarantees to preserve the XLogRecord returned between calls to
ReadRecord. In this particular scenario we read the checkpoint and redo
records from the backup label. After reading the latter record we have
no guarantees that the former record is still pointing to unchanged
memory.

* docs - add hive filter pushdown note, correct xref

* just Disable

Rewrite test sql to verify partition numbers after drop partitions.
Co-authored-by: NWenlin Zhang <wzhang@pivotal.io>
Co-authored-by: NBing Xu <bxu@pivotal.io>

We've done the work to enable or disable specific combinations of
recursive CTEs. They are safe to use, and when they are not, there
will be an error message disallowing the query.

test_RemoteOperation_logger_debug() has been flaking out on the CI
pipeline, and there's no indication of what is going wrong. Replace the
assertTrue() call, which gives no indication of the difference between
actual and expected, with mock.assert_has_calls(), which will tell us
exactly what the calls were in case of failure.

It's possible that this will fix the flake entirely. The previous test
implementation depended on logger.debug() to be called *first* with our
expected output, but given the poor isolation of our global logger
system, it's entirely possible that some other code occasionally calls
debug(). (That this is an issue at all indicates that this isn't really
a unit test, but that's not something to tackle here.) assert_has_calls()
doesn't mind how many other calls happen as long as the one we're
looking for is eventually made, and I think that matches the intent of
the test better anyway.

Backport to 6X_STABLE.

(cherry picked from commit 0fe07f49)

In the case where the user wants to change the port of a mirror, but
leave it in the same directory on the same host, gpmovemirrors was
previously deleting the directory as part of the cleanup because it
did not recognize that the old and new directories were the same.
This commit adds a check to prevent that.

This commit also logs a warning if the move_config_file contains
identical attributes(host,port,data directory) for the old and new
mirror.
Co-authored-by: NJamie McAtamney <jmcatamney@pivotal.io>
Co-authored-by: NShoaib Lari <slari@pivotal.io>
Co-authored-by: NDavid Krieger <dkrieger@pivotal.io>
(cherry picked from commit d15258a4)

While actual logic relating to filespaces was already removed from gpmovemirrors
in a previous commit, there were still some comments and dead code referencing
filespaces, which this commit removes.
Co-authored-by: NJamie McAtamney <jmcatamney@pivotal.io>
Co-authored-by: NShoaib Lari <slari@pivotal.io>
(cherry picked from commit b54840bc)

This corresponds to ORCA commit "Use prime number of buckets to avoid
hash collisions"
Authored-by: NChris Hajas <chajas@pivotal.io>

1.It's a known issue with gpperfmon that cpu skew and row skew information is inaccurate.
 We remove this flaky case for now.
2.GPDB6 rename table gp_distribution_policy colum attnum distkey.
Co-authored-by: NWenlin Zhang <wzhang@pivotal.io>
Co-authored-by: NBing Xu <bxu@pivotal.io>

We don't support Greenplum on these platforms.

Some files (e.g. Makefile.{hpux,solaris}) have been left in place
because they are upstream postgres files. Removing them isn't
worth the headache it would cause when merging commits from
postgres.

Cherry-picked from 52c37372Authored-by: NBen Christel <bchristel@pivotal.io>

* docs - change gpinitsystem -S option to --mirror-mode; change -F to -S

* sentence edit

* Address review comments about mirroring modes

* boldface variable portions of names for readability

* In params list, put short option before the long, consistently.

* Docs - change default log_rotation_size to 1GB

- Also clarify how and when log rotation occurs, i.e., log_rotation_size triggers rotation, it's not a maximum file size.
- For log_rotation_age, log is rotated when the current log file reaches specified age, not "once a day".

* Update from review comments

* Updates from review

* docs - add content for lock_timeout guc

* add note about relationship to deadlock timeouts

This is the equivalent of
https://github.com/greenplum-db/gporca/pull/462,
"Normalize expression before inferring predicates".

Backported from master commit 2b78a302

Reported-by: Cyrille Lintz

Joining nested REURSIVE clauses is planned as a join between
two WorkTableScan nodes, which we currently cannot do. Detect
and disallow for now until we have the required infrastructure
to handle this class of queries. The below query is an example
of this:

  WITH RECURSIVE r1 AS (
      SELECT 1 AS a
      UNION ALL
       (
          WITH RECURSIVE r2 AS (
              SELECT 2 as b
              UNION ALL
              SELECT b FROM r1, r2
          )
          SELECT b FROM r2
      )
  )
  SELECT * FROM r1 LIMIT 1;

In upstream PostgreSQL, the resulting plan exhibits the same
behavior as in GPDB, but there is no restiction on WorkTableScan
on the inner side of joins in PostgreSQL:

                           QUERY PLAN
  -------------------------------------------------------------
   Limit
     CTE r1
       ->  Recursive Union
             ->  Result
             ->  CTE Scan on r2 r2_1
                   CTE r2
                     ->  Recursive Union
                           ->  Result
                           ->  Nested Loop
                                 ->  WorkTable Scan on r1 r1_1
                                 ->  WorkTable Scan on r2
     ->  CTE Scan on r1
  (12 rows)

Backported from master commit e9c0e77a

Upgrade Behave to 1.2.6

(cherry picked from commit 3dd52c8b)

wait_for_mirror_sync() uses contentid as argument, but the missing_xlog test
case uses dbid as input, so in this test wait_for_mirror_sync() is not really
useful - this makes the test flaky.
Co-authored-by: NHaozhou Wang <hawang@pivotal.io>

Reviewed-by: Jimmy Yih

unlogged_heap_tables test occasionally fails since during primary restarting
there is a possibility that fts switches the mirrors to primaries.  For
unlogged table, there is no data on the mirrors so select query could return
different results if there is or there is no switch, thus makes the test not
deterministic.

We disable fts probe in this test using gp fault injector to fix this issue.
Co-authored-by: NHaozhou Wang <hawang@pivotal.io>
Reviewed-by: NAshwin Agrawal <aagrawal@pivotal.io>
Reviewed-by: NJimmy Yih <jyih@pivotal.io>

Previously the test waits no more than 30 seconds until mirror promotion
finished, but 30 seconds are not enough in a real test environment. In this
patch, we increase the guc gp_gang_creation_retry_timer from 250 (ms) to 1000
(ms) so we that we wait for 2 minutes at most, given guc
gp_gang_creation_retry_count has been configured with its maximum value.
Co-authored-by: NHaozhou Wang <hawang@pivotal.io>

We've removed SRF-in-targetlist support for most node types in GPDB.
Instead, we insert Result nodes to evaluate the set-returning-functions.
This can be benefit to performance.

However, Result node cannot evaluate WindowFunc, which an WindowAgg's
target list usually has. So we need to support SRF-in-targetlist for
WindowAgg nodes.

This is the same logic as for AggState.
Reviewed-by: NHeikki Linnakangas <hlinnakangas@pivotal.io>
Reviewed-by: NTaylor Vesely <tvesely@pivotal.io>

It is observed that before gp_replica_check some mirrors are down according to
the 'gpstop -u' output for unknown reason. Not sure why. Probably there is bug
in previous test, or probably oom tests affects. In any case we should not
proceed if the cluster is not ok. Of course we could restart the cluster but
we'd error out instead so that people could find potential issues in previous
testing.
Co-authored-by: NHaozhou Wang <hawang@pivotal.io>

Reviewed-by: Jimmy Yih

It will assert fail at this

  mdcreate (reln=0x2ac5548, forkNum=INIT_FORKNUM, isRedo=0 '\000') at md.c:288
    ExceptionalCondition (conditionName=0xe7cc00 "!(reln->md_fd[forkNum] == ((void *)0))", errorType=0xe7cbbe "FailedAssertion", fileName=0xe7cbb9 "md.c", lineNumber=288) at assert.c:46

This fixes https://github.com/greenplum-db/gpdb/issues/7340

Reported-by: zhbp366@github
Reviewed-by: NJimmy Yih <jyih@pivotal.io>
Reviewed-by: NTaylor Vesely <tvesely@pivotal.io>

These files don't exist in the gpdb repo, nor the bin_gpdb tar.gz
resources in Concourse for 6 or 5 as far as we can tell.
Co-authored-by: NAmil Khanzada <akhanzada@pivotal.io>
Co-authored-by: NBen Christel <bchristel@pivotal.io>

User defined functions cache their plan, therefore if we modify the
plan during execution, we risk having invalid data during the next
execution of the cached plan.

ExecuteTruncate modifies the plan's relations when declaring partitions
that need truncation.

Instead, we copy the list of relations that need truncation and add the
partition relations to the copy.

(cherry picked from commit e3cf4f26)

Commit ea9df812 relaxes the requirement
that all LWLocks are in a single array, which broke the LOCK_DEBUG
code for trying to acquiring an LWLock. This is a meatball surgery
attempt at making it at least compile and run, without trying to
reinvent the old usecase.

Minor tweaking of the code is also done to make LOCK_DEBUG cases be
identified identically.

Backported from master 3a1ced8f
Reported-by: Simon Gao in Github issue #7251
Reviewed-by: Ashwin Agrawal

* Create a public dockerfile for compiling gpdb on ubuntu
* Create directory /usr/local/greenplum-db-devel for installing greenplum
* Move current ubuntu files to ubuntu16_ppa. Update readme

* docs - remove ALTER SYSTEM reference

* Restore source for ALTER SYSTEM ref, but leave out of maps.

* Add ALTER SYSTEM to list of unsupported PostgreSQL features

* Docs - removes replication port references

Also replaces gpstate -p command output with more recent version

* add missing line break to example

Commit 99360f54 extended the offset
numbers in the GIN posting list to 16 bits over upstream who use
11 bits. This however means that the assertion around ip_posid is
no longer valid as it's always true, causing a compiler warning:

ginpostinglist.c:89:24: warning: result of comparison of constant
                                 65536 with expression of type
								 'OffsetNumber' (aka 'unsigned short')
								 is always true
[-Wtautological-constant-out-of-range-compare]
Assert(iptr->ip_posid < (1 << MaxHeapTuplesPerPageBits));
~~~~~~~~~~~~~~~~~~~~~~^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
../../../../src/include/c.h:784:10: note: expanded from macro 'Assert'
Trap(!(condition), "FailedAssertion")
~~~~~~~^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
../../../../src/include/c.h:766:28: note: expanded from macro 'Trap'
if ((assert_enabled) && (condition))
^~~~~~~~~
1 warning generated.

Fix by commenting out the assertion with preprocessor blocks in an
attempt to make future merge conflicts easier to resolve.

A previous version of this patch was reviewed by Adam Berlin.

The level of TERSE or greater was set during development, we can lower
this to DEBUG now that this feature has stablized.

It fails and appears to not be maintained when:

--enable-gpfdist and --with-openssl are configured.

(cherry picked from commit 813528e7)

Because the gpinitstandby flag for passing in a standby master data directory
was changed from -F to -S, this commit changes -F to -S in gpinitsystem to
match.

-S was already in use for specifying that the cluster should have its mirrors
set up in spread configuration, so that flag is changed to --mirror-mode, which
accepts the mirror mode ('group' or 'spread') as an argument.
Co-authored-by: NJamie McAtamney <jmcatamney@pivotal.io>
Co-authored-by: NMark Sliva <msliva@pivotal.io>
(cherry picked from commit 6fa3bce8)

Despite my best efforts to avoid races on overloaded test containers,
test_join_and_indicate_progress_prints_dots_until_pool_is_done() has
been failing fairly often. Replace the simple-but-flaky time-based test
with an implementation that serializes the components of the race.

(cherry picked from commit 9f5384bc)