Based on the experiments with tpch and tpcds, we found that 16MB would
be the optimal size of mdcache.
[#96490982].

Vacuum of AO tables happens in four phases: prepare, compact, drop and cleanup.
When there are concurrent transactions executing vacuum on AO tables, if one of
the vacuum transaction is executing a drop phase, the other transaction will
not be able to open the relation if it is also executing the drop phase. When
the relation open fails, vacuum simply exits without performing the cleanup
phase. The relfrozenxid in pg_class is updated during the cleanup phase and
hence the age of the table does not get reduced. This becomes a problem because
the age of the database does not reduce and eventually the xid wraparound issue
will be encountered.

This problem is fixed by performing the cleanup phase even if the open relation
fails during drop phase.

The column_to_scan function used to get the smallest eof column was
operating incorrectly. It would loop the segfiles until it found a
segfile with nonzero tuplecount, grab the first column that has a
nonzero eof, and return that value. The issue here is that it did not
iterate through all the columns to find the smallest nonzero eof. With
this commit, we now loop through all the columns of the segfile to
decide on a column to return. The unit tests for this function were
also incorrect so we fix that in this commit too.

Due to the different phases of VACUUM, compaction phase could have
committed but the drop was cancelled. This could leave some
aoseg/aocsseg entries in AOSEG_STATE_AWAITING_DROP state which could
then be cleaned up in a subsequent VACUUM as expected... but it
currently just skips. The master keeps a hash table (AppendOnlyHash)
and uses it during VACUUM to check tuple count, state, and
aborted/committed. Our issue here is that state checking was only
being done against state AWAITING_DROP_READY. It was not including
COMPACTED_AWAITING_DROP in the state check. This commit adds the check
and updates the detailed comment block on the state transitions in
appendonlywriter.h.

When doing an ADD COLUMN operation on an AOCO table, the vpinfo byte
array stored in the aocsseg auxiliary table for each column is updated
to add an extra vpinfo index to match the relnatts value in
pg_class. However, entries in the aocsseg auxiliary table with state 2
(AOSEG_STATE_AWAITING_DROP) will not have their vpinfo byte array
updated. Subsequent calls to getAOCSVPEntry function will then fail
due to the vpinfo size does not match the relnatts value in pg_class
(e.g. persistent table rebuild). Since we have the AccessExclusiveLock
on the table during ADD COLUMN, we might as well schedule the drop
before the vpinfos are updated to prevent any inconsistency issues.

We expect lazy VACUUM to skip drop phase if it cannot acquire the
relation's AccessExclusiveLock. This commit properly sets the dontWait
flag to True when calling try_relation_open during drop phase. We also
change the dontWait part of try_relation_open to not error out because
callers should be the ones to handle an invalid relation.

If GPDB is configured to build with orca, this commit adds checks to
ensure orca libraries are available.

We also chose more representative headers for the checks.

If gpdb is configured to build with orca it requires a Greenplum patched
version of Xerces library. This commit adds an explicit check for the
patched Xerces.

This also gives us an opportunity to be more idiomatic in doing
autoconf: we are replacing the manual manipulation of LIBS in
src/backend/Makefile with `AC_CHECK_LIB` in `configure.in`.

When the GPDB is build with GPORCA, `optimizer` GUC is set to on
by default. If GPDB is not build with GPORCA this guc is off by default.
Signed-off-by: NOmer Arap <oarap@pivotal.io>

This changes autoconf to enable orca by default. To be clear, the twin
flags of `--enable-orca` and `--disable-orca` are already available,
this commit merely changes the default (and help messages accordingly).
Signed-off-by: NOmer Arap <oarap@pivotal.io>

As of commit c7ff7663, subplans are initialized even they are unused. In some
case, the generated subplan is not used, but it has motionId = 0, which breaks
executor's sanity check. This patch fixes the issue by removing unused
subplans.

See discussions at:
https://github.com/greenplum-db/gpdb/issues/2383

To test if `ON SEGMENT` is compatible with all the other options.

Previously, we remove the shared memory object when drop resource group,
and restore it if the transaction aborts. The concurrently access to
shared memory object would fail in this way.
If the resource group is dropped, new transactions to this resource
group will be queued up until the drop transaction is finished.
Signed-off-by: NRichard Guo <riguo@pivotal.io>

Partition pruning test in ICW is failing because of `max_stack_depth` is
too small for sles11 environment. This commit increases the limit from
`100kB` to `500kB`.

Until now, people working on documentation would indicate that their
commits did not need to be run through automated testing by using the
`[ci skip]` annotation on their commit messages. This automates the same.

Signed-off-by: NDivya Bhargov <dbhargov@pivotal.io>

This change was made in order to facilitate the new git branching rules.
In order to use different credentials to pull in from our fork, we need
to be able to more finely control which git repo we are pulling from
when we clone the pipeline to smoke test a change.
Signed-off-by: NDivya Bhargov <dbhargov@pivotal.io>

Signed-off-by: NDivya Bhargov <dbhargov@pivotal.io>

The old implementation of function remove_unused_initplans is buggy, we can
come up with multiple ways to break it, because it doesn't utilize postgres
plan/expression tree walker correctly. Rewrite the whole function to resolve
found and potential bugs.

CID 129288, 129277:
Buffer not null terminated (BUFFER_SIZE_WARNING)
buffer_size_warning: Calling strncpy with a maximum size argument of 1024 bytes
on destination array m->path2 of size 1024 bytes might leave the destination
string unterminated.
This issue was detected in the `xlog_mm_redo()` function. Fixed by using
`strlcpy()` to ensure that the buffer is null terminated.

CID 129856: Argument cannot be negative (NEGATIVE_RETURNS)
negative_returns: fd is passed to a parameter that cannot be negative.
This issue was also found in the `xlog_mm_redo()` function.
Fixed the issue by making sure that `close()` is not called on negative fd.

CID 129947: Out-of-bounds write (OVERRUN)
overrun-local: Overrunning array buf of 80 bytes at byte offset 80 using index
nbytes (which evaluates to 80).
The above defect was found in `WalRcvUsr2Handler()` which is in gpdb only and
not in upstream.

CID 129271, 129270, 129269, 129268 : Function address comparison (BAD_COMPARE)
func_conv: This implicit conversion to a function pointer is suspicious:
The above defect was found in `WalReceiverMain()`. The comparison with NULL was
being performed against a function name and not a function pointer or variable
which is clearly incorrect. Since we do static linking of the functions, the
check was not necessary in the first place. Fixed the defect by removing the
check.

The problem is that if ExecPartitionSelector() gets no tuples from the child
node, it doesn't call InsertPidIntoDynamicTableScanInfo() at all. It should
call InsertPidIntoDynamicTableScanInfo() in that case, with InvalidOid as the
partition OID, to signal to PartSelectedExpr that it was executed and no
partitions are selected.

Now with this change it will call InsertPidIntoDynamicTableScanInfo() after the
last last tuple has been read, always, not only when there were no tuples at
all.

* gpdeletesystem and pg_dumpall ref page updates

* clarify that --no-gp-syntax removes table distribution clauses

Remove some unnecessary null-tests, and replace a goto-label construct
with an "if" block.

Michael Paquier, reviewed by me.

Coverity points out that mdc_finish returns a pointer to a local buffer
(which of course is gone as soon as the function returns), leaving open
a risk of misbehaviors possibly as bad as a stack overwrite.

In reality, the only possible call site is in process_data_packets()
which does not examine the returned pointer at all.  So there's no
live bug, but nonetheless the code is confusing and risky.  Refactor
to avoid the issue by letting process_data_packets() call mdc_finish()
directly instead of going through the pullf_read() API.

Although this is only cosmetic, it seems good to back-patch so that
the logic in pgp-decrypt.c stays in sync across all branches.

Marko Kreen

We were not storing sourceTag in CachedPlanSource, so if a cached plan
is used later, that field is broken, and would break resource queue
concurrency limit functionality for example.

Reported by issue #2308 when using JDBC/ODBC

* docs - add plan cache invalidation info

* use correct postgres version, remove reuse statement

Signed-off-by: NJesse Zhang <sbjesse@gmail.com>

CID 167715: Incorrect Expression (UNINTENDED_INTEGER_DIVISION)
Covertiy reported that a division operation of two integers is then
converted to a double. This would cause any fractional part of the
quotient to be lost. This is fixed by casting the divisor to a double
before the division.

CID 167714: Dereference before null check (REVERSE_INULL)
Coverity reported that a null check is unnecessary because the pointer
spill_file->file_info is dereferenced before this line in every code
path. This issue is fixed by removing the null pointer check

This commit fixes a bug where a SharedInputScan with a Redistribute
Motion in its subtree that is also shared within the same slice may not
execute the SharedInputScan and stop the Motion prematurely. To avoid
this issue squelchNodeWalker() short-circuits if it comes across a
ShareInputScan in its subtree that is shared within the same slice

0, default database doesn't exist in some env.
1, reports good even the cases failed.
2, init_file ignores too many lines.

This commit fixes these.

This is a partial backport of f9e9da66 from upstream needed
to get contrib/orca_debug building on macOS:

  commit f9e9da66
  Author: Tom Lane <tgl@sss.pgh.pa.us>
  Date:   Mon Jul 5 23:15:56 2010 +0000

	Fix a few single-file (MODULES, not MODULE_big) contrib makefiles that were
	supposing that they should set SHLIB_LINK rather than LDFLAGS_SL.  Since these
	don't go through Makefile.shlib that was a no-op on most platforms.  Also
	regularize the few platform-specific Makefiles that did pay attention to
	SHLIB_LINK: it seems that the real value of that is to pull in BE_DLLLIBS,
	so do that instead.  Per buildfarm failures on cygwin.

The orca_debug contrib module was in disrepair and no longer
compiled due to changes in the version strings. Fix the code
and also add the proper libraries to link against to give the
.so a fighting chance to compile. Override the compiler in the
Makefile to use a C++ compiler instead.

Functions already compiled in when building GPDB with Orca were
removed, missing prototypes added and an unused function removed.
This allows orca_debug to compile without warnings.