They were mostly marked as static in the prototype already, so this is just
for readability.

cf7cddf7 has conflict with cc38f526, struct PQExpBufferData is
needed by structure SegmentDatabaseDescriptor, so bring gp-libpq-int.h back

The whole cdb directory was shipped to end users and all header files
that cdb*.h included are also need to be shipped to make checkinc.py
pass. However, exposing gp_libpq_fe/*.h will confuse customer because
they are almost the same as libpq/*, as Heikki's suggestion, we should
keep gp_libpq_fe/* unchanged. So to make system work, we include
gp-libpg-fe.h and gp-libpq-int.h directly in c files that need them

It was very difficult to verify if interconnect is stucked in resending
phase or if there is udp resending latency within interconnect. To improve
it, this commit record a debug message every Gp_interconnect_debug_retry_interval
times when gp_log_interconnect is set to DEBUG.

They don't exist in the upstream. All but one of the callers actually just
needed list_member_*().

This gets rid of the GPDB-specific "QE details" message, that was only sent
once at QE backend startup, to notify the QD about the motion listener port
of the QE backend. Use a standard ParameterStatus message instead, pretending
that there is a GUC called "qe_listener_port". This reduces the difference
between the gp_libpq_fe copy of libpq, and libpq proper. I have a dream that
one day we will start using the standard libpq also for QD-QE communication,
and get rid of the special gp_libpq_fe copy altogether, and this is a small
step in that direction.

In the passing, change the type of Gp_listener_port variable from signed to
unsigned. Gp_listener_port actually holds two values: the TCP and UDP
listener ports, and there is bit-shifting code to store those two 16-bit
port numbers in the single 32-bit integer. But the bit-shifting was a bit
iffy, on a signed integer. Making it unsigned makes it more clear what's
happening.

StringInfo is more appropriate in backend code. (Unless the buffer needs to
be used in a thread.)

In the passing, rename the 'conn' static variable in cdbfilerepconnclient.c.
It seemed overly generic.

Clean up after commits e40e78fa, 038aa959, 859d6592 and a21da89e.
Tut tut.

In commit 06f56fe8, we assigned a random segment to server as singleton
QE for load blance. An assertion failure "outputSegIdx[0] == gp_singleton_segindex"
occurs when plan contains initplans because apply_motion_mutator() did
not set segindex of FLOW_SINGLETON to gp_singleton_segindex. Although
segindex field did not affect the creation of singleton gang and also
did not affect QE sending data to the right upstream, we still need
making them consistent to silent the assertion and avoid potential issue.

We add the PT information in addition to the backup block for proper recovery.
In this case, during XLog redo, the blocks of the dropped tables will not be
restored after consulting PT.

In order to fetch the PT information, we have to pass Relation to
MarkBufferDirtyHint(). For that interface change, we refactored
MarkBufferDirtyHint() with additional Relation parameter. The relation
information is eventually passed to XLogSaveBufferForHint() to fetch the PT
information when preparing the XLOG_HINT record.

Validate every BufferPool page sent to the mirror by the
primary prior to writing.
Signed-off-by: NTaylor Vesely <tvesely@pivotal.io>

During resync, filerep copied block with out-of-date checksum over from primary
to mirror. This caused checksum verification failure later on the mirror side,
and also caused inconsistency between the two on disk images of primary and
mirror.

The fix introduced here will always recompute the checksum during resync.

The performance impact is very low, since we only recompute the checksum for
changed blocks. However, for the full copy, we will recompute checksum for all
the blocks to be copied. We have to do it, because there is no easy way to
gurantee there no other changes like hint bit change during resync, since it's
an online operation.

Also fixed wrong comments regarding to page lsn.
Signed-off-by: NXin Zhang <xzhang@pivotal.io>

Resource group memory spill is similar to 'statement_mem' in
resource queue, the difference is memory spill is calculated
according to the memory quota of the resource group.

The related GUCs, variables and functions shared by both resource
queue and resource group are moved to the namespace resource manager.

Also codes of resource queue relating to memory policy are refactored in this commit.
Signed-off-by: NPengzhou Tang <ptang@pivotal.io>
Signed-off-by: NNing Yu <nyu@pivotal.io>

This is a mistake introduced by 353a937d and it
flood the pg_log at a noticeable rate, change it back
to DEBUG1.

Support COPY statement that imports the data file on segments directly
parallel. It could be used to import data files generated by "COPY ...
to ... ON SEGMENT'.

This commit also supports all kinds of data file formats which "COPY ...
TO" supports, processes reject limit numbers and logs errors accordingly.

Key workflow:
   a) For COPY FROM, nothing changed by this commit, dispatch modified
   COPY command to segments at first, then read data file on master, and
   dispatch the data to relevant segment to process.

   b) For COPY FROM ON SEGMENT, on QD, read dummy data file, other parts
   keep unchanged, on QE, process the data stream (empty) dispatched
   from QD at first, then re-do the same workflow to read and process
   the local segment data file.
Signed-off-by: NMing LI <mli@pivotal.io>
Signed-off-by: NAdam Lee <ali@pivotal.io>
Signed-off-by: NHaozhou Wang <hawang@pivotal.io>
Signed-off-by: NXiaoran Wang <xiwang@pivotal.io>

The issue of hanging on recv() in internal_cancel() are reported
serveral times, the socket status is shown 'ESTABLISHED' on master,
while the peer process on the segment has already exit. We are not
sure how exactly dose this happen, but we are able to simulate this
hang issue by dropping packet or reboot the system on the segment.

This patch use poll() to do non-blocking recv() in internal_cancel();
the timeout of poll() is set to the max value of authentication_timeout
to make sure the process on segment has already exit before attempting
another retry; and we expect retry on connect() can detect network issue.
Signed-off-by: NNing Yu <nyu@pivotal.io>

Partition Selection is the process of determining at runtime ("execution
time") which leaf partitions we can skip scanning. Three types of Scan
operators benefit from partition selection: DynamicTableScan,
DynamicIndexScan, and BitmapTableScan.

Currently, there is a minimal amount of logging about what partitions
are selected, but they are scattered between DynamicIndexScan and
DynamicTableScan (and so we missed BitmapTableScan).

This commit moves the logging into the PartitionSelector operator
itself, when it exhausts its inputs. This also brings the nice side
effect of more granular information: the log now attributes the
partition selection to individual partition selectors.

* Remove PartOidExpr, it's not used in GPDB.

The target lists of DML nodes that ORCA generates includes a column for the
target partition OID. It can then be referenced by PartOidExprs. ORCA uses
these to allow sorting the tuples by partition, before inserting them to the
underlying table. That feature is used by HAWQ, where grouping tuples that
go to the same output partition is cheaper.

Since commit adfad608, which removed the gp_parquet_insert_sort GUC, we
don't do that in GPDB, however. GPDB can hold multiple result relations open
at the same time, so there is no performance benefit to grouping the tuples
first (or at least not enough benefit to counterbalance the cost of a sort).

So remove the now unused support for PartOidExpr in the executor.

* Bump ORCA version to 2.37
Signed-off-by: NEkta Khanna <ekhanna@pivotal.io>

* Removed acceptedLeaf
Signed-off-by: NEkta Khanna <ekhanna@pivotal.io>

When a standby is shut down and restarted, WAL recovery starts from
the last restartpoint. If we replay an AO write record which has a
following drop record, the WAL replay of the AO write record will find
that the segment file does not exist. To fix this, we piggyback on top
of the heap solution of tracking invalid pages in the invalid_page_tab
hash table. The hash table key struct uses a block number which, for
AO's sake, we pretend is the segment file number for AO/AOCO
tables. This solution will be revisited to possibly create a separate
hash table for AO/AOCO tables with a proper key struct.

Big thanks to Heikki for pointing out the issue.

We generate AO XLOG records when --enable-segwalrep is configured. We
should now replay those records on the mirror or during recovery. The
replay is only performed for standby mode since promotion will not
execute until after there are no more XLOG records to read from the
WAL stream.

Filerep resync logic to fetch changed blocks from changetracking (CT)
log is changed.  LSN is no longer used to filter out blocks from CT
log.  If a relation's changed blocks falls above the threshold number
of blocks that can be fetched at a time, the last fetched block number
is remembered and used to form subsequent batch.

The GUC gp_changetracking_max_rows replaces a compile time constant.  Resync
worker obtains at the most gp_changetracking_max_rows number of changed blocks
from changetracking log at one time.  Controling this with a GUC allows
exploiting bugs in resync logic around this area.

segmentCount variable is unused in TMGXACT_CHECKPOINT structure hence loose it
out. Also, removing the union in fspc_agg_state, tspc_agg_state and
dbdir_agg_state structures as don't see reason for having the same.

Implement resgroup memory limit.

In a resgroup we divide the memory into several slots, the number
depends on the concurrency setting in the resgroup. Each slot has a
reserved quota of memory, all the slots also share some shared memory
which can be acquired preemptively.

Some GUCs and resgroup options are defined to adjust the exact allocation
policy:

resgroup options:
- memory_shared_quota
- memory_spill_ratio

GUCs:
- gp_resource_group_memory_limit
Signed-off-by: NNing Yu <nyu@pivotal.io>

If QD receives a SIGINT and calls CHECK_FOR_INTERRUPTS after finishing Gang creation, but before recording this Gang in global variables like primaryWriterGang, this Gang would not be destroyed, hence next time QD wants to create a new writer Gang, it would find existing writer Gang on segments, and report snapshot collision error.

[ci skip]

The test used to validate that the tmlock is not held after completing the DTM
recovery. The root cause for not releasing the lock was that in case of an
error during recovery `elog_demote(WARNING)` was called which would demote the
error to a warning. This would cause the abort processing code to not get
executed and hence the lock would not be released. Adding a simple assert in
the code once DTM recovery is complete is sufficient to make sure that the lock
is released.

CID 170750: Logically dead code. In
ChangeTracking_GetLastChangeTrackingLogEndLoc() code can never be reached
because of a logical contradiction, hence removing the same.

"%m%s" format style is differ from upstream and broken the coverity
scan, and it did not give too much value to keep it, so remove it.

* CID 170493:  Null pointer dereferences  (REVERSE_INULL), not need to do
NULL check for transportStates
* CID 170492:  Resource leaks  (RESOURCE_LEAK), addrs in not freed
* CID 170481:  Performance inefficiencies, should pass parameter fds of type "mpp_fd_set" by pointer

Change tracking files are used to capture information on what's changed while
mirror was down, to help incrementally bring it back to sync. In some instances
mostly due to disk issues / full situations, if change tracking log was partial
written or got messed up, resulted in rolling PANIC of segment and there by DB
unavailable due to double fault. Only way out was manual intervention to remove
changetracking files and run full resync.

So, instead this commit now adds checksum protection to auto detect any problem
with change tracking files during recovery / incremental resync. If checksum
miss-match gets detected it takes preventive action to mark segment into
ChangeTrackingDisabled state and keeps DB available. Plus, explicitely enforces
only full recovery is allowed to bring mirror up in sync as changetracking info
doesn't exist. Any attenpt for incremental resync clearly communicates out full
resync has to be performed. So, this eliminates need for manual intervention to
get DB back to availble state if changetracking files get corrupted.