Commit 4e655714 enabled autovacuum for template0, but didn't include any
tests to check that autovacuum really runs on template0. This commit adds
that.

To make this run faster, make debug_burn_xids burn 4096 XIDs instead of
1024.
Co-authored-by: NXin Zhang <xzhang@pivotal.io>

This reverts commit f7de81d6.
CLA was not signed

* Support replicated table in GPDB

Currently, tables are distributed across all segments by hash or random in GPDB. There
are requirements to introduce a new table type that all segments have the duplicate
and full table data called replicated table.

To implement it, we added a new distribution policy named POLICYTYPE_REPLICATED to mark
a replicated table and added a new locus type named CdbLocusType_SegmentGeneral to specify
the distribution of tuples of a replicated table.  CdbLocusType_SegmentGeneral implies
data is generally available on all segments but not available on qDisp, so plan node with
this locus type can be flexibly planned to execute on either single QE or all QEs. it is
similar with CdbLocusType_General, the only difference is that CdbLocusType_SegmentGeneral
node can't be executed on qDisp. To guarantee this, we try our best to add a gather motion
on the top of a CdbLocusType_SegmentGeneral node when planing motion for join even other
rel has bottleneck locus type, a problem is such motion may be redundant if the single QE
is not promoted to executed on qDisp finally, so we need to detect such case and omit the
redundant motion at the end of apply_motion(). We don't reuse CdbLocusType_Replicated since
it's always implies a broadcast motion bellow, it's not easy to plan such node as direct
dispatch to avoid getting duplicate data.

We don't support replicated table with inherit/partition by clause now, the main problem is
that update/delete on multiple result relations can't work correctly now, we can fix this
later.

* Allow spi_* to access replicated table on QE

Previously, GPDB didn't allow QE to access non-catalog table because the
data is incomplete,
we can remove this limitation now if it only accesses replicated table.

One problem is QE need to know if a table is replicated table,
previously, QE didn't maintain
the gp_distribution_policy catalog, so we need to pass policy info to QE
for replicated table.

* Change schema of gp_distribution_policy to identify replicated table

Previously, we used a magic number -128 in gp_distribution_policy table
to identify replicated table which is quite a hack, so we add a new column
in gp_distribution_policy to identify replicated table and partitioned
table.

This commit also abandon the old way that used 1-length-NULL list and
2-length-NULL list to identify DISTRIBUTED RANDOMLY and DISTRIBUTED
FULLY clause.

Beside, this commit refactor the code to make the decision-making of
distribution policy more clear.

* support COPY for replicated table

* Disable row ctid unique path for replicated table.
  Previously, GPDB use a special Unique path on rowid to address queries
  like "x IN (subquery)", For example:
  select * from t1 where t1.c2 in (select c2 from t3), the plan looks
  like:
   ->  HashAggregate
         Group By: t1.ctid, t1.gp_segment_id
            ->  Hash Join
                  Hash Cond: t2.c2 = t1.c2
                ->  Seq Scan on t2
                ->  Hash
                    ->  Seq Scan on t1

  Obviously, the plan is wrong if t1 is a replicated table because ctid
  + gp_segment_id can't identify a tuple, in replicated table, a logical
  row may have different ctid and gp_segment_id. So we disable such plan
  for replicated table temporarily, it's not the best way because rowid
  unique way maybe the cheapest plan than normal hash semi join, so
  we left a FIXME for later optimization.

* ORCA related fix
  Reported and added by Bhuvnesh Chaudhary <bchaudhary@pivotal.io>
  Fallback to legacy query optimizer for queries over replicated table

* Adapt pg_dump/gpcheckcat to replicated table
  gp_distribution_policy is no longer a master-only catalog, do
  same check as other catalogs.

* Support gpexpand on replicated table && alter the dist policy of replicated table

This seems like a more clear and robust way to do this, and matches the
upstream code more closely, too.

This test keeps breaking whenever any new columns are added or removed from
tables with more than 20 columns. That's a pain, we've had to change the
expected output of this test practically every time we merge. Now that I
look at this more closely, I don't see the point of this query, so let's
just remove it.

With the 8.4 merge, planner considers using HashAgg to implement
DISTINCT. At the end of planning, we replace the expressions in the
targetlist of certain operators (including Agg) into OUTER references
in targetlist of its lefttree (see set_plan_refs() >
set_upper_references()).
But, as per the code, in the case when grouping() or group_id() are
present in the target list of Agg, it skips the replacement and this is
problematic in case the Agg is implementing DISTINCT.

It seems that the Agg's targetlist need not compute grouping() or
group_id() when its lefttree is computing it. In that case, it may
simply refer to it. This would then also apply to other operators
WindowAgg, Result & PartitionSelector.

However, the Repeat node needs to compute these functions at each stage
because group_id is derived from RepeatState::repeat_count. Thus, it
connot be replaced by an OUTER reference.

Hence, this commit removes the special case for these functions for all
operators except Repeat. Then, a DISTINCT HashAgg produces the correct
results.
Signed-off-by: NShreedhar Hardikar <shardikar@pivotal.io>

Not passing the right gp_dbid, caused test running after these test to
fail, as gp_inject_fault checks dbid to set the fault or not. If
segment was started with gp_dbid=0, the fault fails to set in other
tests causing failures.

FTS went in infinite loop without this fix on probe if primary failed
to respond back to probe request. Encountered the issue using suspend
fault for fts message handler.

If primary has promote file and pg_basebackup copies over the same,
then due to existency of the same mirror gets auto-promoted which is
very dangerous. Hence avoid copying over promote file.

Not seeing reason for blocking visibility of this guc. Similar to all
other fts gucs, its useful.

If infinite_loop fault is set and shut-down is requested, bail-out
instead of waiting and needing forced kill. It was this way till
commit ae760e25 removed the
`IsFtsShudownRequested()` check. Didn't intentionaly change the
suspend fault as it never had shut-down check and don't wish to change
behavior of any tests using the same.

The DTX_STATE_FORCED_COMMITTED was identical to
DTX_STATE_INSERTED_COMMITTED.

Remove a log message to indicate if a QE reader is writing an XLOG
record.  Back in GPDB 4.3 when lazy XID feature didn't exist, a QE
reader would be assigned a valid transaction ID.  That could lead to
extending CLOG and generating XLOG.  This case no longer applies to
GPDB.

The command "COPY enumtest FROM stdin;" hit an infinite loop on merge
branch.  Code indicates that the issue can happen on master as well.
QD backend went into infinite loop when the connection was already
closed from QE end.  The TCP connection was in CLOSE_WAIT state.
Libpq connection status was CONNECTION_BAD and asyncStatus was
PGASYNC_BUSY.

Fix the infinite loop by checking libpq connection status in each
iteration.

Authored-by: NKaren Huddleston <khuddleston@pivotal.io>

This commit introduces a GUC `optimizer_enable_associativity` to enable
or disable join associativity. Join Associativity increases the search
space as it increases the numbers of groups to represent a join and its
associative counterpart, i.e (A X B) X C ~ A X (B X C).

This patch, by default disables join associativity transform, if
required the users can enable the transform. There are few plan changes
which are observed due to this change. However, further evaluation of
the plan changes revealed that even though the cost of the the resulting
plan has increased, the execution time went down by 1-2 seconds.

For the queries with plan changes, there are 3 tables which are joined,
i.e A, B and C. If we increase the number of tuples returned by the
subquery which forms A', we see the old plan. But if the tuples in
relation B and C is significantly higher, the plan changes with the
patch yeild faster execution times. This suggests that we may need to
tune the cost model to adapt to such cases.

The plan cost increase is 1000x as compared to the old plans, this 1000x
factor is due to the value of `optimizer_nestloop_factor=1024`, if you
set the value of the GUC `optimizer_nestloop_factor=1`, the plan before
or after the patch remains same.

Thank You Heikki for pointing out the presence of `gpxlogloc` data type to
compare xlog locations instead of exiting hacks in test.

When gpload finishes its query, it will send SIGTERM to gpfdist.
gpfdist handle SIGTERM with exit(1), which will invoke registered
apr handlers and cleanup all apr resources including apr_pool. If
this happens just during normal destruction of apr_pool in
do_close, gpfdist will hang.

Call _exit in gpfdist to avoid any cleanup handlers, and let gpload
send SIGKILL to perform hard kill.

 .Smaller image size
 .Change the role from gp to gpadmin
 .Remove libraries that are not needed

Test failed few times randomly in CI and newly added debug log revealed flaw
with text comparison of xlog location. For example text comparison considers
this "1/103BE80" xlog location as smaller than "1/FEE230" and hence test
fails. So, instead replaced the logic with other hack to convert the location to
hex and then compare, which should serve th purpose till we get to 9.4.

pg_dump is currently returning the following error as the parlevel
was not being used in the query for backend version 9.0:

pg_dump: column number -1 is out of range 0..20

Also, fix the query that was running on GPDB4.
Authored-by: NChris Hajas <chajas@pivotal.io>

docs: pl/container - note about OOM message displayed when PID 1 terminated on older docker installs (#4758)

- Use nocheck to skip `make check` in test
- Use parallel to use 6 parallel processes

Although standard SQL ignores the ORDER BYs in views (and sub-selects),
PostgreSQL and thus GPDB preserves them.

For the query in create_view.sql, the expected output will be sorted
according to the ORDER BY clause in the view definition. But, if the
rows come in the wrong order from the view then gpdiff.pl will not
report it as an error.

Remove the FIXME in this file, since there is no way to enforce the
order via gpdiff.pl.

Instead, to test the output row ordering, this commit modifies the
queries in gp_create_view.sql and adds an order-sensitive operator -
row_number().
Signed-off-by: NDhanashree Kashid <dkashid@pivotal.io>

* docs - add cancel/term backend content

* address comments from daniel

* use single l in cancel tenses

Signed-off-by: NSambitesh Dash <sdash@pivotal.io>
Signed-off-by: NJesse Zhang <sbjesse@gmail.com>

FTS starts probing as soon as fts process gets created and triggers
every minute. No reason to trigger and wait for one probe cycle on
first connection which calls initTM(). If for some reason fail to
dispatch command to segment, dispatcher will trigger probe anyways, so
why slow things down upfront.

I am not sure on usage of this GUC gp_set_read_only. When set it sets
FTS to read-only which is then used while starting any transaction and
converted to read-only. Seems better to achieve the same result using
GUC `default_transaction_read_only` or `transaction_read_only`, which
can be used at session-level as well instead of full system-wide
setting.

Previously, to avoid the leak of the gang if someone terminates
the query in the middle of gang creation, we added a global pointer
named CurrentGangCreating so the partially created gang can also be
destroyed at the end of the transaction. However, the memory context
named GangContext where CurrentGangCreating was created may be reset
before CurrentGangCreating is actually destroyed and a SIGSEGV may
occur. So this commit makes sure that CurrentGangCreating is destroyed
ahead of other created gangs and the reset of GangContext.

Allows the compile and icw tests for each platform to pass
and fail independently of other platforms.

The gate jobs were born out of necessity to handle infrastructure issues
with concourse. Now that infrastructure issues have been stabilized,
it's time to review the layout of the pipeline again.

This commit removes the icw_start_gate job that multiplexed a passing
condition from all of the compile jobs that sat in front of every icw
job. This was not desirable following a longer running
issue with the compilation on one platform, ubuntu16, that then blocked
icw tests on the remaining platforms.

Replacing the passing condition of icw_start_gate on each icw job is the
corresponding compilation job for the test, based on platform.

E.G.

This:
                    (blocks)          (blocks)
compile_gpdb_centos6  -> gate_icw_start -> icw_planner_centos6
compile_gpdb_ubuntu16 ->                -> icw_planner_ubuntu16

Is now this:

                    (blocks)
compile_gpdb_centos6  -> icw_planner_centos6
                    (blocks)
compile_gpdb_ubuntu16 -> icw_planner_ubuntu16
Signed-off-by: NTaylor Vesely <tvesely@pivotal.io>

Concurrent index builds are not supoorted in greenplum. Seems there exist GUC
gp_create_index_concurrently under which we still allow the creations. But still
with the same for bitmap indexes CONCURRENTLY cannot be supported.

This test keeps failing randomly in CI due to "WARNING: ignoring query cancel
request for synchronous replication to ensure cluster consistency." This happens
as during bitmap index creation it locally commits on each segment and then one
of the segment errors during index build with "ERROR: CONCURRENTLY is not
supported when creating bitmap indexes", issues cancelation message to other
segments.