* docs - updates for changes to resgroup MAX_SPILL_RATIO/MEMORY_LIMIT

* some of the edits requested by hubert

* add RG info to statement_mem guc descript, RGs use max_statement_mem

* more misc edits

* new topic about reserving memory vs. using RG global shared mem pool

* more misc edits

* more misc edits

* change title

The test used to wait indefinitely for master to shutdown (PANIC) due to all
retries failing in the second phase of 2PC.  The indefinite wait would actually
be indefinite if the roles for content0 primary and mirror are flipped.  The
roles would flip because of a previously failing test.  Fix this by making the
wait bounded.  Also use new fault injector API, while at it.

Reviewed by Pengzhou Tang.

When '<content_id>U:' is encountered in isolation2 spec, the framework opens a
connection to master and obtains hostname and port of the primary segment with
specified content ID.  This connection to master should be in utility mode.  A
normal mode connection would wait for DTM recovery and if a test is in the
middle of testing DTM recovery, the connection may not even complete.

Reviewed by Pengzhou Tang

At some point, all existing tests need to be changed whole sale like this.

Reviewed by Jimmy Yih and Adam Berlin

The test is found to fail intermittently in CI when the gprecoverseg command is
fired and a mirror that was promoted by FTS is still undergoing promotion.  Fix
this by running a distributed query in a loop until the query runs
successfully.

Reviewed by Jimmy Yih and Adam Berlin

* docs - add info about per-greenplum-user pxf configuration

* misc easy edits requested by david

* remove file inadvertently added to the last commit :^(

* rename topic title and some sectiont titles, regorg

* more topic renames, add xrefs to connector cfg topics

* change remote back to external; address comments from francisco

This commits adds a new GUC that controls the xform:
optimizer_push_group_by_below_setop_threshold.

Also bump ORCA version to 3.49.0

In commit 0de5fc63 I renamed the upstream
version variables MAJORVERSION and VERSION to PG_ counterparts, to
clarify issues around Greenplum/PostgreSQL versioning at the time.
This has however been shown to cause issues when compiling extensions
which interrogate these variables during building.

Reported in https://github.com/greenplum-db/gpdb/issues/7916
Reviewed by Jimmy Yih

There's no reason to believe that the lock on the XID would not be
available, and if for reason it's not, it seems dangerous to just
continue as if nothing happened. So remove the GPDB-added code that
checked for that, reverting the function to the way it is in the upstream.

Because of the way the check was written, LockAcquire() always got called
twice, so we actually acquired the lock on the XID twice. That's usually
harmless, since both locks are released at the end of the transaction
anyway, except for one scenario: If you created a savepoint, and committed
it (e.g. with RELEASE SAVEPOINT), we would release the lock on the
subtransaction's XID only once, leaving the other lock still in place. So
we kept the lock on every committed subtransaction 'till the end of the
top transaction. That too was mostly harmless, the only ill effect was
that the locks would take some space in the global lock table, which is
limited in size, so you could run out of shared memory more easily if you
used subtransactions heavily.

Fixes github issue https://github.com/greenplum-db/gpdb/issues/7819

CString only has the length and data in the serialized structure, no
need to allocate spaces for the varlena header.

When creating WAL records for a twophase commit, we add the global
transaction info to distributed commit and distributed forget records
on the query dispatcher and to the commit prepared records on the
query executors. However, pg_xlogdump only prints this information for
the distributed commit and distributed forget records. We should do
the same for commit prepared records to more easily string together
transactions between QD and QE when comparing xlog dumps.

On query dispatcher:
distributed commit 2019-06-11 15:16:17.786319 PDT gid = 1560291226-0000000006, gxid = 6
distributed forget  gid = 1560291226-0000000006, gxid = 6

On query executor:
commit prepared 747: 2019-06-11 15:16:17.786660 PDT

Change query executor output to:
commit prepared 747: 2019-06-11 15:16:17.786660 PDT gid = 1560291226-0000000006 gxid = 6

Without this, default_tablespace GUC is only dispatched to QEs during explicit
SET command.  However, if the gang resets and a new gang is created, the GUC is
omitted and any tables / indexes created thereafter in that session will not be
created by QEs in the right tablespace.

Ashwin Agrawal raised this concern in a GitHub PR comment.

Reviewed by: Soumyadeep Chakraborty

This is cherry-pick of upstream commit
a1a789eb, slightly modified to work
with current GPDB code based of 9.4. Helps to fix the deadlocks seen
in CI for walreceiver.

-------------------
Author: Tom Lane <tgl@sss.pgh.pa.us>
Date:   Mon Apr 29 12:26:07 2019 -0400

    In walreceiver, don't try to do ereport() in a signal handler.

    This is quite unsafe, even for the case of ereport(FATAL) where we won't
    return control to the interrupted code, and despite this code's use of
    a flag to restrict the areas where we'd try to do it.  It's possible
    for example that we interrupt malloc or free while that's holding a lock
    that's meant to protect against cross-thread interference.  Then, any
    attempt to do malloc or free within ereport() will result in a deadlock,
    preventing the walreceiver process from exiting in response to SIGTERM.
    We hypothesize that this explains some hard-to-reproduce failures seen
    in the buildfarm.

    Hence, get rid of the immediate-exit code in WalRcvShutdownHandler,
    as well as the logic associated with WalRcvImmediateInterruptOK.
    Instead, we need to take care that potentially-blocking operations
    in the walreceiver's data transmission logic (libpqwalreceiver.c)
    will respond reasonably promptly to the process's latch becoming
    set and then call ProcessWalRcvInterrupts.  Much of the needed code
    for that was already present in libpqwalreceiver.c.  I refactored
    things a bit so that all the uses of PQgetResult use latch-aware
    waiting, but didn't need to do much more.

    These changes should be enough to ensure that libpqwalreceiver.c
    will respond promptly to SIGTERM whenever it's waiting to receive
    data.  In principle, it could block for a long time while waiting
    to send data too, and this patch does nothing to guard against that.
    I think that that hazard is mostly theoretical though: such blocking
    should occur only if we fill the kernel's data transmission buffers,
    and we don't generally send enough data to make that happen without
    waiting for input.  If we find out that the hazard isn't just
    theoretical, we could fix it by using PQsetnonblocking, but that
    would require more ticklish changes than I care to make now.

    This is a bug fix, but it seems like too big a change to push into
    the back branches without much more testing than there's time for
    right now.  Perhaps we'll back-patch once we have more confidence
    in the change.

    Patch by me; thanks to Thomas Munro for review.

    Discussion: https://postgr.es/m/20190416070119.GK2673@paquier.xyz
-------------------

In a UNION ALL query, the typmods of the individual queries might not be
the same, so that the typmod in the targetlist of the overall set operation
won't match all the individual queries. That could cause pulling up the
distribution key to fail.

Fixes github issue #7433.

CI reported much less failure after the commit
0639ee8e, so basically the commit works,
but it need retrying more times.

Changed the default retry time from 3 to 5.

* docs - pxf max tomcat threads now user-configurable

* misc edits

* Doc tablespace path limit of 100 characters

* Add restriction to content id path as well

The PostgreSQL semver regex did not account for developer versions
(e.g. 9.4.21 vs 9.5beta1). Relax the regex to make the test pass since
this was an oversight when the test was written.

After commit 679c10b3 broke backwards compatibility in 5X_STABLE, we
realized we didn't have a regression test to catch cases like that.

This commit causes the new binary to dump a (very minimal) view created
by an old binary. It would have caught something like 679c10b3. There
are more complexes cases that won't be caught by this. But here's a
start we can build on.
Co-authored-by: NJesse Zhang <sbjesse@gmail.com>

Commit 1315881c removed sync_tools from the build system.  This
patch adjusts PR pipeline accordingly.

When append-optimized table is scanned using bitmap indexscan, the variable
'need_recheck' is not initialized in BitmapAppendOnlyNext function, then the
value is always 'true'.  This results in unnecessary recheck of the qual
conditions at every tuple.

Fixes GitHub issue #7657

Authored-by: NBradford D. Boyle <bboyle@pivotal.io>

In previous versions of Greenplum, Ivy was used to resolve and download
dependencies for building the GPDB server. As we shift towards relying
on the build environment to provided the necessary dependencies, we no
longer require Ivy.

Previous work had shifted the dependencies out of Ivy for supported
platforms (i.e., RHEL and Ubuntu) but the Concourse pipeline was still
invoking Ivy (an effective no-op) as part of the sync_tools task. This
commit removes the sync_tools task and the remaining Ivy configuration
from inside of gpAux.

The responsibility for downloading gporca was moved into the
compile_gpdb Concourse task which is where xerces-c and orca are built.
Co-authored-by: NBradford D. Boyle <bboyle@pivotal.io>
Co-authored-by: NGoutam Tadi <gtadi@pivotal.io>

- convert print statements to function calls
- remove unused import
- fix a few minor PEP8 spacing lints
Authored-by: NBradford D. Boyle <bboyle@pivotal.io>

The warning was due to incorrect format specifier used for size_t.

as per review feedback.  Also remove the logic to report status of all faults.
There was a pre-existing bug in this logic.  Status of the fault most recently
encountered in the fault shash table would overwrite previously recorded
status.  I've removed this logic because the feature is not used anywhere.  If
needed, correct logic can be coded in future.

Reviewed by: Hao Wu

With this, a fault is now identified only by its name.  New workflow for adding
a fault:

1. Choose a name for your fault, say "brand_new_fault".  Use
   "git grep brand_new_fault" to check if the name is not already taken.

2. Identify the location in code and use SIMPLE_FAULT_INJECTOR("brand_new_fault")
   or more elaborate FaultInjector_InjectFaultIfSet("brand_new_fault", ...)
   function to setup the trigger.

3. Invoke gp_inject_fault("brand_new_fault", ...) from a regress, isolation or
   isolation2 test to enable the fault.

I made some cleanup along the way by marking a function as static and removing
an unused function.

Reviewed by: Hao Wu, Haozhou Wang, Ashwin Agrawal, Adam Berlin, Jacob Champion

Fixes GitHub issue #7788
Discussion: https://groups.google.com/a/greenplum.org/d/topic/gpdb-dev/zmCIxrN873c/discussion

* Add GPDB cloud technical recommendations doc

* Formatting fixes

* Review comments and a copy edit

* Address more review comments

* Fixes from reviews

* Add gpdb github release pipeline

Automatic creation of Github Tags and Releases
Authored-by: NTingfang Bao <bbao@pivotal.io>

* Fix read pg_distributedlog bug

In the function 'DistributedLog_AdvanceOldestXmin', it reads pg_distributedlog,
updates 'DistributedLogShared->oldestXmin' and truncates the logs whose xid is
smaller than 'DistributedLogShared->oldestXmin'. This looks good because it's
serialized under the lock DistributedLogControlLock. The problem is the read
function 'SimpleLruReadPage' may release and re-acquire the lock, and another
caller of DistributedLog_AdvanceOldestXmin may truncate the logs concurrently
and 'SimpleLruReadPage' will error out. DistributedLog_CommittedCheck may
read LRU page in a similar way, also need to be protected.

The number of pg_distributedlog files is about 32 times of pg_clog, truncate
pg_distributedlog based on the frozenxid might left too many files on the disk.
So we introduce a new LWLock to prevent truncating a file while someone else is
reading it.
Co-authored-by: NGang Xiong <gxiong@pivotal.io>
Co-authored-by: NHubert Zhang <hzhang@pivotal.io>

* Remove COPY query limitation for ON SEGMENT clause

* Feedback from Lisa

Now that gpcheck has been removed update the documentation as well.

Due to the number of platforms we now support and variety of customer
specific environments, it is not practical to update gpcheck to cover
all these cases.
Co-authored-by: NJamie McAtamney <jmcatamney@pivotal.io>

Authored-by: NBradford D. Boyle <bboyle@pivotal.io>

Postgres 9.5 commit 09d8d110 introduced a FLEXIBLE_ARRAY_NUMBER that is used in
oidvector. This means that `values` now has a flexible length rather than
hardcoded length of 1. However, when retrieving these args from the struct, we
were assuming no deep copy was necessary -- we simply assigned the results into
another oidvector. This meant that values were not properly copied and we got
invalid oids, which errored out on 9.5 merge branch with `Unrecognized type
name`.

In GPDB 6X, we would retrieve one value, which was enough to make
`ResolvePolymorphicTypes` not error out, but the values retrieved from
pgproctypes were still inaccurate (though it doesn't seem to cause any issues)

Now, we populate the correct arg values. This function was only used by
ORCA, which is why it only errored when ORCA was used.

We were not able to create a failing testcase on 6X and master, but the
wrong results retrieved from get_func_arg_types() seem risky enough that
they could cause issues.
Co-authored-by: NChris Hajas <chajas@pivotal.io>

It is possible that mirror/standby might not flush or not finish flushing the
related xlog soon after the tablespace drop/create command finishes, so using
'ls' or 'find' to check tablespace related directories in the test might fail.

gp_tablespace test failure was observed several times recently, due to this.
Adding UDFs to poll the status until timeout.

This patch does some minor cleanup for other files also.

Reviewed-by: Asim Praveen

We dispatch the create table utility before create toast table, so
we will get a "no pre-assigned OID" error. Fix it by call dispatch
function after create toast table.

Funtion insertResgroupCapabilities uses sprintf to
format capabilities string, which is not safe.
Using snprintf to replace sprintf.