The AO implementation aligns with 8.4 and forward heap implementation, to write
the data during recovery and not fail.

Also, to note in case of AO the way seek it performed during replay, its not
going to fail if file doesn't have that much data yet. As the way seek works
irrespective of the length of file will seek to that offset from requested
position and write the data (sure file will have hole in it for this case) . But
it will not result in seek failure as such. We will write the data and if
truncation has happened then will happen again during recovery.

The lock level looks fine hence resolving the fixme's.

There are many tests which flow through this code path specifically in
alter_table.sql. Nothing exploded with the removal of the same and gpcheckcat
flagged nothing means its fine to delete the same.

- Introduce a new GUC gp_resource_group_bypass, when it is on,
  the query in this session will not be limited by resource group

* Extra docs for gp_external_enable_filter_pushdown

Add extra documentation for 'gp_external_enable_filter_pushdown' and the pushdown feature in PXF extension.

* Minor doc text fixes

Minor documentation text fixes, proposed by @dyozie.

* Clarify the pushdown support by PXF

Add the following information:
* List the PXF connectors that support pushdown;
* State that GPDB PXF extension supports pushdown;
* Add a list of conditions that need to be fulfilled for the pushdown feature to work when PXF protocol is used.

* Correct the list of PXF connectors with pushdown

* State that Hive and HBase PXF connectors support filter predicate pushdown;
* Remove references to JDBC and Apache Ignite PXF connectors, as proposed by @dyozie (these are not officially supported by Greenplum).

Number of fsync buffers synced to disk varies based on how hint-bits gets
updated and all. Like sometimes I see global table pg_tablespace buffer flushed
and sometimes not depending on what tests were executed before this test.

Greenplum added rd_isyscat to Relation structure. Only usage of the same is in
markDirty() to decide if buffer should be marked dirty or not. The setting of
rd_issyscat is based in checking if relation name starts with "pg_" then set it
else not. Which anyways is very loose.

Modified instead to just make check based on if oid < FirstNormalObjectId or to
cover for pg_aoseg tables RelationGetNamespace(relation) ==
PG_AOSEGMENT_NAMESPACE. So, this allows us to remove the extra variable.

This patch is not trying to change the intent of GUC
`gp_disable_tuple_hints`. That's all together different discussion.

The issue happens because of constant folding in the testexpr of the
SUBPLAN expression node. The testexpr may be reduced to a const and any
PARAMs, previous used in the testexpr, disappear, However, the subplan
still remains.

This behavior is similar in upstream Postgres 10 and may be of
performance consideration. Leaving that aside for now, the constant
folding produces an elog(ERROR)s when the plan has subplans and no
PARAMs are used. This check in `addRemoteExecParamsToParamList()` uses
`context.params` which computes the used PARAMs in the plan and `nIntPrm
= list_length(root->glob->paramlist`, which is the number of PARAMs
declared/created.
Given the ERROR messages generated, the above check makes no sense.
Especially since it won’t even trip for the InitPlan bug (mentioned in
the comments) as long as there is at least one PARAM in the query.

This commit removes this check since it doesn't correctly capture the
intent.

In theory, it could be be replaced by one specifically aimed at
InitPlans, that is, find all the params ids used by InitPlan and then
make sure they are used in the plan. But we already do this and
remove any unused initplans in `remove_unused_initplans()`. So I don’t
see the point of adding that.

Fixes #2839

To merge stats in incremental analyze for root partition, we use leaf
tables' statistics. In commit b28d0297,
we fixed an issue where child attnum do not match with a root table's
attnum for the same column. After we fixed that issue with a test, that
test also exposed the bug in analyze code.

This commit fixes the issue in analyze using the similar fix in
b28d0297.

This is related to the work we have done to fix the sles11 and windows
compilation failures.
Co-authored-by: NLisa Oakley <loakley@pivotal.io>
Co-authored-by: NAlexandra Wang <lewang@pivotal.io>

Previously, we would use the root table's information to acquire stats
from the `syscache` which return no result. The reason it does not
return any result is because we query syscache using `inh` field which
is set true for root table and false for the leaf tables.

Another issue which is not evident is the possibility of mismatching
`attnum`s for the root and leaf tables after running specific scenarios.
When we delete a column and then split a partition, unchanged partitions
and old partitions preserves the old attnums while newly created
partitions have increasing attnums with no gaps. If we query syscache
using the root's attnum for that column, we would be getting a wrong
stats for that specific column. Passing root's `inh` hide the issue of
having wrong stats.

This commit fixes the issue by getting the attribute name using the
root's attnume and use it to acquire correct attnum for the largest leaf
partition.

No need to have database scope analyze, only specific table needs to be
analyzed for the test.

GPDB only supports 1 replica currently. Need to adopt in FTS and all to support
1:n till then restrict max_wal_senders GUC to 1. Later when code can handle the
same the max value of guc can be changed.

Also, remove the setting of max_wal_senders in postmaster which was added
earlier for dealing with filerep/walrep co-existence.

These three was listed in common options,

```
FILL MISSING FIELDS
LOG ERRORS [SEGMENT REJECT LIMIT <replaceable class="parameter">count</replaceable> [ROWS | PERCENT] ]
IGNORE EXTERNAL PARTITIONS
```

But,

1, they are not working with both FROM and TO
2, FILL MISSING FIELDS should be [FILL_MISSING_FIELDS true | false] in
generic form, which is silly, the old syntax is better
3, SREH and IGNORE EXTERNAL PARTITIONS could not be specified as generic ones

Also documents the missing NEWLINE option.

Unloading doesn't need it, checking the distribution policy neither.

Also, remove dev generated pipeline and add prod generated pipeline
Co-authored-by: NLisa Oakley <loakley@pivotal.io>
Co-authored-by: NTrevor Yacovone <tyacovone@pivotal.io>

Co-authored-by: NLisa Oakley <loakley@pivotal.io>
Co-authored-by: NTrevor Yacovone <tyacovone@pivotal.io>

This is a common issue with sles11 - that it doesn't
currently include support for > TLS v1.1. Many upstream endpoints over
the last 6 months have started to enforce > TLS v1.1.

We resolved this by separating the sync_tools call into a separate task,
such that we could run this task from a centos docker image, prior to
compiling on the correct OS.

These changes were backported to other compile jobs. We are pushing this
change to resolve the sles11 blocker, but we are still experiencing
difficulty with windows.
Co-authored-by: NLisa Oakley <loakley@pivotal.io>
Co-authored-by: NTrevor Yacovone <tyacovone@pivotal.io>
Co-authored-by: NEd Espino <edespino@pivotal.io>

Some functions if had different return type or argument compared to upstream,
were modified with comment in pg_proc.h while few were moved completely to
pg_proc.sql. This difference causes confusion while merging and having single
consistent method for all would be better. So, with this commit now upstream
functions would be defined in pg_proc.h irrespective if there definitions differ
from upstream or not.

Note:
pg_proc.h is used for all upstream definitions and pg_proc.sql is used to
auto-generate gpdb added functions in Greenplum.

In markDirty() seems oversight in commit
8c8b5c39 to avoid calling MarkBufferDirtyHint()
for temp tables. Previous patch used relation->rd_istemp before calling
XLogSaveBufferForHint() in MarkBufferDirtyHint() that was unnecessary given it
already checks for BM_PERMANENT. So, call now MarkBufferDirtyHint()
unconditionally.

gp_dispatch = false on utility command is correct, as cannot dispatch the SET
command yet as not established the transaction yet on
QEs. transaction_deferrable is only useful with serializable isolation level as
per upstream docs, so added note to start dispatching the same when we support
serializable isolation level.

Currently gpdb wal rep code is mix of multiple versions, once we reach 9.3 get
opportunity to pain get in sync with upstream version. This will be taken care
of then till that time live with gpdb modified version of the
CheckPromoteSignal().

No reason to call `SyncRepWaitForLSN()` from walsender process itself. Some
existed in past seems which performed the same, even if walsender for whatever
reason needs to perform transaction shouldn't result in wrtitng
anything. Replaced the if with assertion instead to catch any viaolations of the
assumption.

Removing the Greenplum specific guc `Debug_xlog_insert_print`. Instead use
upstream guc `wal_debug` for the same. Also, remove some unneccessary
modifications vs upstream.

Upstream doesn't have it and not used anymore in Greenplum, so loose it.

Now, that wal replication is enabled for QD and QE the code must be enabled.

Upstream and for greenplum master if procdie is received while waiting for
replication, just WARNING is issued and transaction moves forward without
waiting for mirror. But that would cause inconsistency for QE if failover
happens to such mirror missing the commit-prepared record.

If only prepare is performed and primary is yet to process the commit-prepared,
gxact is present in memory. If commit-prepared processing is complete on primary
gxact is removed from memory. If gxact is found then we will flow through
regular commit-prepared flow, emit the xlog record and sync the same to
mirror. But if gxact is not found on primary, we used to return blindly success
to QD. Hence, modified the code to always call `SyncRepWaitForLSN()` before
replying to QD incase gxact is not found on primary.

It calls `SyncRepWaitForLSN()` with the lsn value of `flush` from
`xlogctl->LogwrtResult`, as there is no way to find-out the actual lsn value of
commit-prepared record for primary. Usage of that lsn is based on following
assumptions
	- WAL always is written serially forward
	- Synchronous mirror if has xlog record xyz must have xlog records before xyz
	- Not finding gxact entry in-memory on primary for commit-prepared retry
  	  from QD means it was for sure committed (completed) on primary

Since, the commit-prepared retry can be received if everything is done on
segment but failed on some other segment, under concurrency we may call
`SyncRepWaitForLSN()` with same lsn value multiple times given we are using
latest flush point. Hence in GPDB check in `SyncRepQueueIsOrderedByLSN()`
doesn't validate for unique entries but just validates the queue is sorted which
is required for correctness. Without the same during ICW tests can hit assertion
"!(SyncRepQueueIsOrderedByLSN(mode))".

Added new test job to the pipeline to certify GPHDFS with MAPR Hadoop distribution and renamed existing GPHDFS certification job to state that it tests with generic Hadoop. MAPR cluster consists of 1 node deployed by CCP scripts into GCE.

- MAPR 5.2
- Parquet 1.8.1
Co-authored-by: NAlexander Denissov <adenissov@pivotal.io>
Co-authored-by: NShivram Mani <smani@pivotal.io>
Co-authored-by: NFrancisco Guerrero <aguerrero@pivotal.io>

A check was added during the 9.1 merge to verify that the sequence
filepath to be created would not collide with an existing file. The
filepath that is constructed does not use the sequence OID value that
was just generated and uses whatever value is in that piece of memory
at the time. This would make the check go through usually, especially
in our CI testing, but occasionally a sequence would fail to be
created because the random filepath would exist. Fix the issue by
storing the generated OID in the RelFileNode var that will be passed
into the filepath construction.

Resource group capabilities could be missing in ALTER command, e.g.:

- create resgroup rg1 with v5.0 which does not support cpuset(cap=7);
- binary switch to v5.10 (suppose it supports cpuset);
- alter rg1's cpu_rate_limit, it will also `update` the cpuset cap;

Now as rg1 was created with v5.0 so there is no cap=7 row in the catalog
table pg_resgroupcapability, so the `update` operation will raise an
error as the expected tuple can not be found.

The proper behavior is to fallback to `insert` in such a case.

Test cases are not included as it is already covered by existing binary
swap test resgroup_current_3_queue.

(cherry picked from commit 95f215d9)

When the server is built with `--disable-orca`, we shouldn't (and used
not to) allow setting the option `optimizer=on`. Upstream Postgres 9.1
commit 2594cf0e introduced the check
hook in a GUC code refactoring, and it seems that we regressed in the
9.1 merge where we forgot to signal the calling code in
`call_bool_check_hook` to error out.

This commit fixes that by reintroducing the error.