`repair_frag()` should consult distributed snapshot
(`localXidSatisfiesAnyDistributedSnapshot()`) while following and moving chains
of updated tuples. Vacuum consults distributed snapshot
(`localXidSatisfiesAnyDistributedSnapshot()`) to find which tuples can be
deleted and not. For RECENTLY_DEAD tuples it used to make decision just based on
comparison with OldestXmin which is not sufficient and even there distributed
snapshot must be checked.

Fixes #4298

(cherry picked from commit 313ab24f)

Remove extra semicolon which breaks the query for copy-paste.
Backported from master.

Per report by Michael Mulcahy

-Move note in Examples section to emphasize container ID depends on configuration
-Add creating a docker group in Docker Install instructions for CentOS 6

* docs: update shared mem. info in GUCS max_fsm_pages and max_connections

Add info from support about shared memory conflicts.

PR for 5X_STABLE
Will be ported to MAIN

* docs: update shared mem. info in GUCS - review edits

Commit e924b46e removed all references
to the parameter tf-bucket-path because it is now hardcoded to clusters.

This reverts commit 508ffd48.

`repair_frag()` should consult distributed snapshot
(`localXidSatisfiesAnyDistributedSnapshot()`) while following and moving chains
of updated tuples. Vacuum consults distributed snapshot
(`localXidSatisfiesAnyDistributedSnapshot()`) to find which tuples can be
deleted and not. For RECENTLY_DEAD tuples it used to make decision just based on
comparison with OldestXmin which is not sufficient and even there distributed
snapshot must be checked.

Fixes #4298

(cherry picked from commit 313ab24f)

In scenarios where pg_statistic contains wrong statistic entry for an
attribute, or when the statistics on a particular attribute are broken,
for e.g the type of elements stored in stavalues<1/2/3> is different
than the actual attribute type or when there are holes in the attribute
numbers due to adding/dropping columns; following two APIs fail because
they relied on the attribute type sent by the caller:

- get_attstatsslot() : Extracts the contents (numbers/frequency array and
values array) of the requested statistic slot (MCV, HISTOGRAM etc). If the
attribute is pass-by-reference or if the attribute is of toastable type
(varlena types)then it returns a copy allocated with palloc()
- free_attstatsslot() : Frees any palloc'd data by get_attstatsslot()

This problem was fixed in upstream 8.3
(8c21b4e9) for get_attstatsslot(),
wherein the actual element type of the array will be used for
deconstructing it rather that using caller passed OID.
free_attstatsslot() still depends on the type oid sent by caller.

However the issue still exists for free_attstatsslot() where it crashes while
freeing the array. The crash happened because the caller sent type OID was of
type TEXT meaning this a varlena type and hence free_attstatsslot() attempted
to free the datum; however due to the broken slot the datums extracted from
values array were of fixed length type such as int. We considered the int value
as memory address and crashed while freeing it.

This commit brings in a following fix from upstream 10 which redesigns
get_attstatsslot()/free_attstatsslot() such than they robust to scenarios like
these.

commit 9aab83fc
Author: Tom Lane <tgl@sss.pgh.pa.us>
Date:   Sat May 13 15:14:39 2017 -0400

    Redesign get_attstatsslot()/free_attstatsslot() for more safety and speed.

    The mess cleaned up in commit da075960 is clear evidence that it's a
    bug hazard to expect the caller of get_attstatsslot()/free_attstatsslot()
    to provide the correct type OID for the array elements in the slot.
    Moreover, we weren't even getting any performance benefit from that,
    since get_attstatsslot() was extracting the real type OID from the array
    anyway.  So we ought to get rid of that requirement; indeed, it would
    make more sense for get_attstatsslot() to pass back the type OID it found,
    in case the caller isn't sure what to expect, which is likely in binary-
    compatible-operator cases.

    Another problem with the current implementation is that if the stats array
    element type is pass-by-reference, we incur a palloc/memcpy/pfree cycle
    for each element.  That seemed acceptable when the code was written because
    we were targeting O(10) array sizes --- but these days, stats arrays are
    almost always bigger than that, sometimes much bigger.  We can save a
    significant number of cycles by doing one palloc/memcpy/pfree of the whole
    array.  Indeed, in the now-probably-common case where the array is toasted,
    that happens anyway so this method is basically free.  (Note: although the
    catcache code will inline any out-of-line toasted values, it doesn't
    decompress them.  At the other end of the size range, it doesn't expand
    short-header datums either.  In either case, DatumGetArrayTypeP would have
    to make a copy.  We do end up using an extra array copy step if the element
    type is pass-by-value and the array length is neither small enough for a
    short header nor large enough to have suffered compression.  But that
    seems like a very acceptable price for winning in pass-by-ref cases.)

    Hence, redesign to take these insights into account.  While at it,
    convert to an API in which we fill a struct rather than passing a bunch
    of pointers to individual output arguments.  That will make it less
    painful if we ever want further expansion of what get_attstatsslot can
    pass back.

    It's certainly arguable that this is new development and not something to
    push post-feature-freeze.  However, I view it as primarily bug-proofing
    and therefore something that's better to have sooner not later.  Since
    we aren't quite at beta phase yet, let's put it in.

    Discussion: https://postgr.es/m/16364.1494520862@sss.pgh.pa.us

Most of the changes are same as the upstream commit with following additional
changes:
- Relcache translator changes in ORCA.
- Added a test that simulates the crash due to broken stats
- get_attstatsslot() contains an extra check for empty slot array which existed
in master but is not there in upstream.
Signed-off-by: NAbhijit Subramanya <asubramanya@pivotal.io>
(cherry picked from commit ae06d7b0)

1, pipes might not exist while in close_program_pipes(), check it.
  For instance, relation doesn't exist, copy workflow fails before
  executing the program, "cstate->program_pipes->pid" dereferences NULL.

2, the program might be still running or hang when copy exits, kill it.
  Cases like the program hangs, doesn't take signals, user is trying to
  cancel.
  Since it's already the end of copy, and the program was started by
  copy, it should be safe to kill to clean up.

(cherry picked from commit d6bd4ac4)

* docs - clarify some resource group information

* reword a bit to make intent clearer

DST law changes in Brazil, Sao Tome and Principe.  Historical corrections
for Bolivia, Japan, and South Sudan.  The "US/Pacific-New" zone has been
removed (it was only a link to America/Los_Angeles anyway).

This fixes the "ERROR:  record type has not been registered" error, when
a record-type variable is used in a query inside a PL/pgSQL function.
This is essentially the same problem we battled with in Motion nodes
in GPDB 5, and added the whole tuple remapper to deal with it. Only this
time, the problem is with record Datums being dispatched from QD to QE,
as Params, rather than with record Datums being transferred across a
Motion.

To fix, send the transient record type cache along with the query
parameters, if there are any of the parameters are transient record types.
This is a bit inefficient, as the transient record type cache can be quite
large. A more fine-grained approach would be to send only those record
types that are actually used in the parameters, but more code would be
required to figure that out. This will do for now.

Refactor the serialization and deserialization of the query parameters, to
leverage the outfast/readfast functions.

Backport to 5X_STABLE. This changes the wire format of query parameters, so
this requires the QD and QE to be on the same minor version. But this does
not change the on-disk format, or the numbering of existing Node tags.

Fixes github issue #4444.

The current behavior is to check for the existence of dumps dirs and, unless
run with the -f flag, error out if they do. This commit extends this
check to backups.

Author: Nadeem Ghani <nghani@pivotal.io>

Leverage perfmon stat sender procees to call a hook function for cluster
level info collection.

Without changing existing behavior of stat sender process, this commit
changed the interval in main loop of stat sender from 1 second to 100 ms
because cluster info collector requires a finer granularity of sampling.
With this change, stats sender process should be started when
gp_enable_query_metrics is on.

Author: Wang Hao <haowang@pivotal.io>
Author: Zhang Teng <tezhang@pivotal.io>

The hook is called for
 - each query Submit/Start/Finish/Abort/Error
 - each plan node, on executor Init/Start/Finish

Author: Wang Hao <haowang@pivotal.io>
Author: Zhang Teng <tezhang@pivotal.io>

On postmaster start, additional space in Shmem is allocated for Instrumentation
slots and a header. The number of slots is controlled by a cluster level GUC,
default is 5MB (approximate 30K slots). The default number is estimated by 250
concurrent queries * 120 nodes per query. If the slots are exhausted,
instruments are allocated in local memory as fallback.

These slots are organized as a free list:
  - Header points to the first free slot.
  - Each free slot points to next free slot.
  - The last free slot's next pointer is NULL.

ExecInitNode calls GpInstrAlloc to pick an empty slot from the free list:
  - The free slot pointed by the header is picked.
  - The picked slot's next pointer is assigned to the header.
  - A spin lock on the header to prevent concurrent writing.
  - When GUC gp_enable_query_metrics is off, Instrumentation will
    be allocated in local memory.

Slots are recycled by resource owner callback function.

Benchmark result with TPC-DS shows performance impact by this commit is less than 0.1%
To improve performance of instrumenting, following optimizations are added:
  - Introduce instrument_option to skip CDB info collection
  - Optimize tuplecount in Instrumentation from double to uint64
  - Replace instrument tuple entry/exit function with macro
  - Add need_timer to Instrumentation, to allow eliminating of timing overhead.
    This is porting part of upstream commit:
------------------------------------------------------------------------
commit af7914c6
Author: Robert Haas <rhaas@postgresql.org>
Date:   Tue Feb 7 11:23:04 2012 -0500

Add TIMING option to EXPLAIN, to allow eliminating of timing overhead.
------------------------------------------------------------------------

Author: Wang Hao <haowang@pivotal.io>
Author: Zhang Teng <tezhang@pivotal.io>

- Add libpython
- Add libevent
- Add libgcc_s

The flakyness is caused by the Orca version bump to 2.53.11, in which Orca
improved the equivclasses for inner/outer join plan. Previously Orca generated
bad plan, causing OOM. But as of 2.53.11, Orca generated better plan, may not
cause OOM. Due to the flakyness of this test, remove it.

- Stagger kickoff based on prod/dev and number of sections
- Also suggest dev pipeline command has a git-branch

Author: Jim Doty <jdoty@pivotal.io>
Author: C.J. Jameson <cjameson@pivotal.io>
Author: Shoaib Lari <slari@pivotal.io>
(cherry picked from commit f84ff8ce)

Partial cherry-pick of da4dc150

Author: C.J. Jameson <cjameson@pivotal.io>
Author: Shoaib Lari <slari@pivotal.io>

Resetting all the environment variables makes it difficult to use an
alternate Python installation (e.g. virtualenv). Make a copy of the
current env instead, and add our PIPELINE_FILE to that.

(cherry picked from commit df83e394)

This backports the below commit which moved from the source timezone
file to the newly introduced compact format.

  commit 097b24cea68ac167a82bb617eb1844c8be4eaf24
  Author: Tom Lane <tgl@sss.pgh.pa.us>
  Date:   Sat Nov 25 15:30:11 2017 -0500

    Replace raw timezone source data with IANA's new compact format.

    Traditionally IANA has distributed their timezone data in pure source
    form, replete with extensive historical comments.  As of release 2017c,
    they've added a compact single-file format that omits comments and
    abbreviates command keywords.  This form is way shorter than the pure
    source, even before considering its allegedly better compressibility.
    Hence, let's distribute the data in that form rather than pure source.

    I'm pushing this now, rather than at the next timezone database update,
    so that it's easy to confirm that this data file produces compiled zic
    output that's identical to what we were getting before.

    Discussion: https://postgr.es/m/1915.1511210334@sss.pgh.pa.us

Backported from master.

The timezone data in Greenplum are from the base version of
PostgreSQL that the current version of Greenplum is based on.
This cause issues since it means we are years behind on tz
changes that have happened. This pulls in the timezone data
and code from PostgreSQL 10.1 with as few changes to Greenplum
as possible to minimize merge conflicts. The goal is to gain
data rather than features, and for Greenplum for each release
to be able to stay current with the iana tz database as it is
imported into upstream PostgreSQL.

This removes a Greenplum specific test for the Yakutsk timezone
as it was made obsolete by upstream tz commit 1ac038c2c3f25f72.

Backported from master.

PR for 5X_STABLE only

This information can be easily derived from CDXLColRef member of
CDXLScalarIdent. This now mirrors what is done in the ORCA types
CScalarIdent and CColRef.
Signed-off-by: NEkta Khanna <ekhanna@pivotal.io>

Previously, we would include the DEFAULT argument modifier when
performing the ACL dump, which caused an error during restore. This is
dumping correctly in pg_dump, so we now use the same logic in gpcrondump.

Author: Chris Hajas <chajas@pivotal.io>