Centos 6 defaults nprocs as 1024 for users. Many times in CI seen failures as
cannot fork process, hence better to set reasonable limits (currently equivalent
to what's recommended by admin guide) while creating gpadmin user.

There is already a preset GUC for the PostgreSQL version number in
machine readable form: server_version_num, and human readable form:
server_version. This adds a Greenplum counterparts for these as
gp_server_version_num and gp_server_version.

Previously gporca translator was only pruning the non-visible system columns from
the table descriptor for non-partition `appendonly` tables or if the
paritition table is marked as `appendonly` at the root level.

If one of the leaf partitions in is marked as `appendonly` but the root
is not, the system columns still appears in the table descriptor.

This commit fixes the issue by checking if the root table has
`appendonly` paritions and pruning system columns if it has.

TINC discover would try to import the file instead of the class when
given something like this:
tinc.py discover -t tincrepo/mpp/gpdb/tests -s dispatch/queryfinish

The -t path is added to sys.path and the -s is imported. The -s needs
to be in dots for proper class importing.

Signed-off-by: NTom Meyer <tmeyer@pivotal.io>

docs: removing dita 'to-topic' chunking, which causes the subnav to collapse when the topic is selected (#1678)

doc: updating README to indicate that the full-easy-install version of DITA-OT is required.  Download links also provided (#1676)

When a pulse build is interrupted, fetching artifacts fails due to being
unavailable for a short bit. Retry so it fails more cleanly

Use slimmer docker image for pulse-cloud

* adding source for best practices doc

* removing pivotal copyright

* adding missing graphics files

The query in hashjoin_spill was choosing a different plan with ORCA, where
the "other" side of the join was hashed than expected, and therefore it did
not spill to disk as expected. Turn the join the query into an OUTER JOIN,
to force the join order.

In materialize_spill and sis_mat_sort tests, the test queries are tailored
to produce a particular plan with a Materialize node that spills. ORCA
chose a different plan that didn't do that. I was not able to quickly
formulate the query in a way that would produce the same result with and
without ORCA, so just force ORCA off for the test. The point of the test
is to test workfile behavior at runtime, in executor, so that should be OK.

Likewise, the sisc_sort_spill test was producing different plan with ORCA,
without a Sort, so it wasn't testing what it was supposed to test. Disable
ORCA for sisc_sort_spilll too.

The helper function in sisc_mat_sort tests were producing "insufficient
memory reserved for statement" errors with ORCA. Not sure why, but I bumped
statement_mem slightly to not hit that limit anymore. It still spills to
disk, which is the point.

There's no need to allocate ExecWorkFile struct in TopMemoryContext. We
don't rely on a callback to clean them up anymore, they will be closed
by the normal ResourceOwner and MemoryContext mechanisms. Otherwise,
the memory allocated for the struct is leaked on abort.

In the passing, make the error handling more robust.

Also modify tuplestorenew.c, to allocate everything in a MemoryContext,
rather than using gp_malloc() directly. ResourceOwner and MemoryContext
mechanisms will now take care of cleanup on abort, instead of the custom
end-of-xact callback.

Remove execWorkFile mock test. The only thing it tested was that
ExecWorkFile was allocated in TopMemoryContext. That didn't seem very
interesting to begin with, and the assumption is now false, anyway. We
could've changed the test to test that it's allocated in
CurrentMemoryContext instead, but I don't see the point.

Per report in private from Karthikeyan Rajaraman. Test cases also by him,
with some cleanup by me.

This commit cleans up some tinc.py calls, updates project names, and
splits the cs-walrep test into two targets for CI test
parallelization.

This is fixed with the commit d5463511.

The GPOS library has been imported into the GPORCA repository so
there is no need to install it separately anymore.

This fixes a self-deadlock scenario in filerep postmaster reset. We
encountered a situation in our testing where injecting PANIC on
primary to force a postmaster reset would cause the mirror to hang
while waiting for a child backend process to exit. This mirror backend
process sends a message to its postmaster to start terminating backend
child processes and then runs proc_exit. The mirror postmaster will
quickly act on the message and start sending exit signals to its
backend processes. In rare cases, the mirror backend process that
messages the mirror postmaster can encounter a self-deadlock where
authdie from SIGQUIT sent from its postmaster is called before its
natural proc_exit finishes.

This self-deadlock window can also occur when a psql connection is
started and the forked backend process gets a bad status from
ProcessStartupPacket. This will result in a proc_exit while SIGQUIT is
assigned to authdie instead of quickdie so a very precise timed
SIGQUIT could cause the process to self-deadlock.

The function `AppendOnlyCompaction_GetHideRatio()` incorrectly truncated the
fractional part while calculating the ratio of hidden tuples which caused the
file to not be compacted (for e.g if the calculated threshold was 10.1% and the
guc was set to 10%, the segfile would not be compacted). This patch correctly
rounds up the calculated value to fix this issue.

Context:
gp_fastsequence is used to generate and keep track of row numbers for AO and CO
tables. Row numbers for AO/CO tables act as a component to form TID, stored in
index tuples and used during index scans to lookup intended tuple. Hence this
number must be monotonically incrementing value. Also should not rollback
irrespective of insert/update transaction aborting for AO/CO table, as reusing
row numbers even across aborted transactions would yield wrong results for index
scans. Also, entries in gp_fastsequence only must exist for lifespan of the
corresponding table.

Change:
Given those special needs, now reserved entries in gp_fastsequence are created
as part of create table itself instead of deffering their creation to insert
time. Insert within same transaction as create table is the only scenario needs
coverage from these precreated entries, reserved entries above hence means entry
for segfile 0 (used by CTAS or ALTER) and segfile 1 (used by insert within same
transaction as create). Rest all entries continue to use frozen inserts to
gp_fastsequence as they can only happen after create table transaction has
committed.

With that change in logic to leverage MVCC to handle cleanup of entries for
gp_fastseqeunce, enables to get rid of special recovery and abort code
performing frozen deletes. With that code gone fixes issues like:
1] `REINDEX DATABASE` or `REINDEX TABLE pg_class` hang on segment nodes if
encounters error after Prepare Transaction.

2] Dangling gp_fastsequence in scenario, transaction created AO table, inserted
tuples and aborts after prepare phase is complete. To cleanup gp_fastsequence,
must open the relation and perform frozen heap delete to mark the entry as
invisible. But if the backend performing the abort prepared is not connected to
the same database, then delete operation cannot be done and leaves dangling
entries.

Output of helpful interaction with Heikki Linnakangas and Asim R P.
See discussion on gpdb-dev, thread 'reindex database abort hang':
https://groups.google.com/a/greenplum.org/forum/#!topic/gpdb-dev/ASml6lN0qRE

The old coding failed to apply the mutator to the children of a SubPlan.

While at it, try to explain in the comment *why* SubPlans need special
treatment in GPDB, while they don't in PostgreSQL. I'm not 100% I got the
reasoning correct, since I'm basically just reverse-engineering the
original thinking here. But seems plausible..

Fixes github issue #368.

Signed-off-by: NLarry Hamel <lhamel@pivotal.io>

* This exists in 5x but not in 43x

5x has external options while 43x does not. This causes a diff, so we
will ignore this.

* change inet and cidr from plain to main.

Author: Chumki Roy and Marbin Tan

Signed-off-by: NHaisheng Yuan <hyuan@pivotal.io>

This comes from a discussion on the GPDB mailing list about need of
the prefetch_inner parameter in Joins and the deadlock case it is
trying to prevent.

When the size of the inner tuple in a MergeJoin is large, planner
decides to place a Materialize on the top of the inner Sort operator.
Furthemore if there is any Motion below the MergeJoin,
mjnode->prefetch_inner is set to true.

During execution of plan that has a Motion underneath a Materialize
underneath a MJ, MJ retrieves one tuple from the Motion (via
Materialize) and then resets the stream (by calling ExecReScan). Since
the Materialize underneath is initialized with incorrect eflags, it
deletes its tuplestore, instead of rewinding its cursor.  This fix sets
the right eflags value during Init (see comments in ExecMaterialReScan).

Increase the timeout to start transactions after killing a primary in
the gprecoverseg test suite.

* Update kerberos init file to ignore gid value