This reverts commit 4ba11904.

* fix gpload fial when capital letters in column name in merge mode

add double quotations for column names when create staging table
omit distributio
Co-authored-by: NXiaoxiaoHe <hxiaoxiao@vmware.com>

The GROUPING SETS statement with multiple canonical rollups working on
randomly distributed partitioned table causes rebuilding of
root->simple_rte_array and root->simple_rel_array on planning stage by
PostgreSQL optimizer. The rebuilding process initializes both RTEs and
RelOptInfos item by item and doesn't take into account that the
build_simple_rel() routine is recursive for inherited relations and
requires a full list of already initialized child RTEs inside
root->simple_rte_array. As a result, the call of build_simple_rel() fails
on the access to child RTEs.

The current fix segregates loops of building root->simple_rte_array and
root->simple_rel_array trying to leave a whole semantics unchanged.

When doing 'VACUUM FULL', 'swap_relation_files' updates the pg_class entry but
not increase the command counter, so the later 'vac_update_relstats' will
inplace update the 'relfrozenxid' and 'relhasindex' of the old tuple, when the
transaction is interrupted and aborted on the QE after this, the old entry is
corrupted.

This will make LISTEN and NOTIFY work on the QD node.

fix match column condition to resovle primary key conflict when using the gpload
merge mode to import data to the Multi-level partition table
fix fail when special char and capital letters in column names
Co-authored-by: NXiaoxiaoHe <hxiaoxiao@vmware.com>

Resource group used to access resGroupSlot in SessionState without
lock. This is correct when session only access resGroupSlot by itself.
But as we introduced runaway feature, we need to traverse the current
session array to find the top consumer session when redzone is reached.
This requires:
1. runaway detector should hold shared resgroup lock to avoid resGroupSlot
is detached from a session concurrently when redzone is reached.
2. normal session should hold exclusive lock when modifying resGroupSlot
in SessionState.

Also fix a compile warning.
Reviewed-by: NNing Yu <nyu@pivotal.io>

(cherry picked from commit a4cb06b4)

Proxy bgworker will become orphan process after postmaster is dead
due to the lack of checking pipe postmaster_alive_fds[POSTMASTER_FD_WATCH].
Epoll this pipe inside proxy bgworker main loop as well.
Reviewed-by: NNing Yu <nyu@pivotal.io>
(cherry picked from commit 9ce59d1a)

This approach special cases gp_segment_id enough to include the column
as a distributed column constraint. It also updates direct dispatch info
to be aware of gp_segment_id which represents the raw value of the
segment where the data resides. This is different than other columns
which hash the datum value to decide where the data resides.

After this change the following DDL shows Gather Motion from 2 segments
on a 3 segment demo cluster.

```
CREATE TABLE t(a int, b int) DISTRIBUTED BY (a);
EXPLAIN SELECT gp_segment_id, * FROM t WHERE gp_segment_id=1 or gp_segment_id=2;
                                  QUERY PLAN
-------------------------------------------------------------------------------
 Gather Motion 2:1  (slice1; segments: 2)  (cost=0.00..431.00 rows=1 width=12)
   ->  Seq Scan on t  (cost=0.00..431.00 rows=1 width=12)
         Filter: ((gp_segment_id = 1) OR (gp_segment_id = 2))
 Optimizer: Pivotal Optimizer (GPORCA)
(4 rows)

```

(cherry picked from commit 10e2b2d9)

When calculating safeChunksThreshold of runaway in resource group,
we used to divide by 100 to get the number of safe chunks. This may
lead to small chunk numbers to be rounded to zero. Fix it by storing
safeChunksThreshold100(100 times bigger than the real safe chunk) and
do the computation on the fly.
Reviewed-by: NNing Yu <nyu@pivotal.io>
(cherry picked from commit 757184f9)

MASTER_DATA_DIECTORY must be exported to before calling
gpconfig else it will fail to set the GUC

Allow setting DCA_VERSION_FILE to enable testing

Allow overriding the value of DCA_VERSION_FILE so that during testing it
can be manipulated. Also add a test for the same to ensure that DCA
configuration GUCs are set properly on the environment.

When converting semi-join to inner-join, a distinct agg on ctid is added
above the hash-join node. But the fake ctids generated in Values Scan
were invalid, with offset number 0, which caused an assertion failure.

This patch is based on commit d8886cf9, which fixed the same issue for
Function Scans.
Co-authored-by: Ndh-cloud <60729713+dh-cloud@users.noreply.github.com>
Co-authored-by: NJesse Zhang <sbjesse@gmail.com>

When we convert semi join to inner join, a distinct agg on ctid is added
above the hash join node. But tuples from function scan have no ctid. So
the assert check failed.

Set a synthetic ctid based on a fake ctid by call existed funciton
slot_set_ctid_from_fake.

(cherry picked from commit d8886cf9)

Glibc implementations are known to return inconsistent results for
strcoll() and strxfrm() on many platforms that can cause
unpredictable bugs. Because of that PostgreSQL disabled strxfrm()
by default since 9.5 at compile time by TRUST_STRXFRM definition.
Greenplum has its own mk sort implementation that can also use
strxfrm(). Hence mk sort can also be affected by strcoll() and
strxfrm() inconsistency (breaks merge joins). That is why strxfrm()
should be disabled by default with TRUST_STRXFRM_MK_SORT definition
for mk sort as well. We don't use PostgreSQL's TRUST_STRXFRM
definition as many users used Greenplum with strxfrm() enabled for
mk sort and disabled in PostgreSQL core. Keeping TRUST_STRXFRM_MK_SORT
as a separate definition allows these users not to reindex after
version upgrade.
Reviewed-by: NAsim R P <pasim@vmware.com>
Reviewed-by: NHeikki Linnakangas <linnakangash@vmware.com>
Reviewed-by: NHubert Zhang <hzhang@pivotal.io>

* Fix distclean of postgres_fdw and dblink

After crash recovery finishes, the startup process will create an
end-of-recovery checkpoint. The checkpoint will recycle/remove xlog files
according to orphaned prepared transaction LSNs, replication slot data, etc.
The orphaned prepared transaction LSN data (TwoPhaseState->prepXacts, etc) for
checkpoint are populated in the startup process RecoverPreparedTransactions(),
but the function is called after the end-of-recovery checkpoint creation so
xlog files with orphaned prepared transactions might be recycled/removed. This
can cause "requested WAL segment pg_xlog/000000010000000000000009 has already
been removed" kind of error when bringing up the crashed primary. For example
if you run 'gprecoverseg -a -v', you might be able to see failure with stack as
below. In more details, this happens when running the single mode postgres in
pg_rewind.
	2    0x5673a1 postgres <symbol not found> (xlogutils.c:572)
	3    0x567b2f postgres read_local_xlog_page (xlogutils.c:870)
	4    0x5658f3 postgres <symbol not found> (xlogreader.c:503)
	5    0x56518a postgres XLogReadRecord (xlogreader.c:226)
	6    0x54d725 postgres RecoverPreparedTransactions (twophase.c:1955)
	7    0x55b0b7 postgres StartupXLOG (xlog.c:7760)
	8    0xb0087f postgres InitPostgres (postinit.c:704)
	9    0x97a97c postgres PostgresMain (postgres.c:4829)

Fixing this by prescanning the prepared transaction data from xlogs directly
for checkpoint creation if it's still InRecovery (i.e. when
TwoPhaseState->prepXacts is not populated).

Please note that "orphaned" might be temporary (usually happens on an
environment with heavy write-operation load) or permanent unless master dtx
recovery (implies a product bug).

This is a gpdb 6 only issue. On gpdb 7, state files are used to store prepared
transactions during checkpoint so we do not keep the wal files with orphaned
prepared transactions and thus we won't encounter this issue.
Reviewed-by: NAsim R P <pasim@vmware.com>

Previous commit ab74e1c6, c7befb1d did not completely solve its race
condition, it did not test for last iteration of the while/for loop.
This could result in failed assertion in the following loop. The patch
moves the judgement to the ending of the for loop, it is safe, because
the first iteration will never trigger: Assert(activeWeight > 0.0).

Also, the other one race condition can trigger this assertion
Assert(gl->numFollowersActive > 0). Consider this situation:

    Backend A, B belong to the same statement.

    Timestamp1: backend A's leader is A, backend B's leader is B.

    Timestamp2: backend A's numFollowersActive remains zero due to timeout.

    Timestamp3: Sweeper calculates leader B's numFollowersActive to 1.

    Timestamp4: backend B changes it's leader to A even if A is inactive.

We stop sweeping for this race condition just like commit ab74e1c6 did.

Both Assert(activeWeight > 0.0) and Assert(gl->numFollowersActive > 0)
are removed.

(cherry picked from commit b1c19196)

* Fix dblink's libpq issue

When using dblink to connect to a postgres database, it reports the
following error:
unsupported frontend protocol 28675.0: server supports 2.0 to 3.0

Even if dblink.so is dynamically linked to libpq.so which is compiled
with the option -DFRONTEND, but when it's loaded in gpdb and run,
it will use the backend libpq which is compiled together with
Postgres program and reports the error.

This pr is almost the same as the pr:
https://github.com/greenplum-db/gpdb/pull/10617
which is used to fix postgres_fdw issue

* Disable dblink on MaxOS

--exclude-libs and -Bstatic are not supported on MaxOS

* Fix postgres_fdw's libpq issue

When using posgres_fdw, it reports the following error:
unsupported frontend protocol 28675.0: server supports 2.0 to 3.0

root cause: Even if postgres_fdw.so is dynamic linked to libpq.so
which is compiled with the option -DFRONTEND, but when it's loaded
in gpdb and run, it will use the backend libpq which is compiled together
with postgres program and reports the error.

We statically link libpq into postgres_fdw and hide all the symbols
of libpq.a with --exclude-libs=libpq.a to make it uses the frontend
libpq.

As postgres_fdw is compiled as a backend without -DFRONTEND, and linked
to libpq which is a frontend, but _PQconninfoOption's length is
different between backend and frontend as there is a macro in it.
The backend's _PQconninfoOption has field connofs, but the frontend
doesn't. This leads to the crash of postgres_fdw. One solution is to
remove the FRONTEND macro in struct PQconninfoOption to make PQconninfoOption
is same in the backend and frontend, but that brings an ABI change. To avoid that,
we add the FRONTEND macro on including libpq header files, so that postgres_fdw
can process the libpq's variables returned by libpq.a's functions as frontend.

* Disable postgres_fdw on MaxOS

* docs - add information on upgrading to PostGIS 2.5.4

Upgrade instructions 2.1.5 to different versions of 2.5.4

* docs - upgrade to PostGIS 2.5.4 review comments

* docs - more review comment updates.
reorder upgrade sections.
clarify removing PostGIS package, is for removing the gppkg

* docs - minor edit

* docs - review updates - more emphasis on removing PostGIS from a database deleting objects.
-Create separate paragraph in Upgrading section.
-Add warning in Removing PostGIS section

* docs - minor review comment update

* small edits
Co-authored-by: NDavid Yozie <dyozie@pivotal.io>

Commit 445fc7cc hardened some parts of analyzedb. However, it missed a
couple of cases.

1) When the statement to get the modcount from the pg_aoseg table failed
due to a dropped table, the transaction was also terminated. This caused
further modcount queries to fail and while those tables were analyzed,
it would error and not properly record the mod count. Therefore, we now
restart the transaction when it errors.

2) If the table is dropped and then recreated while analyzedb is running
(or some other mechanism that results in the table being successfully
analyzed, but the pg_aoseg table did not exist during the initial
check), the logic to update the modcount may fail. Now, we skip the
update for the table if this occurs. In this case, the modcount would
not be recorded and the next analyzedb run will consider the table
modified (or dirty) and re-analyze it, which is the desired behavior.

* allow NEq predicates with lossy casts for PS
* Updating some mdp files that were missing the PartitionTypes XML tag
* Update partition_pruning result file for planner
* adding new testcase for NEq
* move icg tests from partition_pruning to gporca
* add test for checking multilevel range/list PS
Co-authored-by: NHans Zeller <hzeller@vmware.com>

Raise a meaningful error message for this case.
GPDB doesn't support alter type on primary key and unique
constraint column. Because it requires to drop - recreate logic.
The drop currently only performs on master which lead error when
recreating index (since recreate index will dispatch to segments and
there still an old constraint index exists).

This fixes the issue https://github.com/greenplum-db/gpdb/issues/10561.
Reviewed-by: NHubert Zhang <hzhang@pivotal.io>
(cherry picked from commit 32446a32)

This is kept as a separate commit as an example of how to add additional
workloads.
Co-authored-by: NChris Hajas <chajas@vmware.com>

This creates an explain pipeline which is used to run query workloads
using explain and identify plan differences during feature development.

This previously existed on 5X, but with the Orca in GPDB effort the
architecture needed to be updated.

The new pipeline has the following features for use with slack commands:

- Comparing a SHA with a baseline SHA
- Identifying and outputting plan diffs
- Summarizing the results with planning time, any
fallbacks/crashes/timeouts, and the type of diff (cost, cardinality,
plan change)
- Enabling custom gucs (turning optimizer off, using a different cost
model, enabling/disabling a feature flag/join order)

Some use cases that we've come across:
- Checking whether a new join order algorithm influences optimization
time/dramatically changes plans

One of the main goals with this change is to make it easy to add
workloads. The slack command can be used in the following format:

\test_gpdb6_explain <SHA_remote> <SHA> <baseline_SHA_remote>
<baseline_SHA> <workload> [optional_gucs]

where workloads is either "all" or a list with no spaces of the names of
the workloads to run (eg: workload1,workload2,workload3).
and gucs, if specified, are in the format:
```
optimizer=off
enable_nestloop=on
```

There is a race condition between receiveFrom() and
dbms_pipe_session_A::createImplicitPipe(). Sleep an extra seconds
is not reliable. So wait until the data is available.
This PR changes the semantics of receiveFrom(), i.e. wait until
the data is available, no timeout happens.
Reviewed-by: NHubert Zhang <hzhang@pivotal.io>
(cherry picked from commit 2f170b36)

* docs - add information on table rewrites when executing ALTER TABLE commands.

Information is in Notes section of ALTER TABLE.
Also, made some minor edits.

* docs - updates based on review comments.
-simplified table listing ALTER TABLE commands that perform table rewrites.
-add links to table rewrite information.

* docs - clarify gpcopy supported greenplum versions

* use or later rather than +

ic-proxy supports to handle cancel signal during
backend connection setup now, so external_table test case
could be enabled.
Reviewed-by: NNing Yu <nyu@pivotal.io>
(cherry picked from commit 1789e437)

ic-proxy is developed with libuv, the minimal supported libuv version is
1.18.0. But in commit 608514, we introduce new API in libuv 1.19, which
break compatibility on os like Ubuntu 18.04 whose default libuv version
is 1.18.

We should keep our code base align with libuv 1.18, and replace the new
API in libuv 1.19. The API change is mainly about how to access data field
in uv handle and uv loop. The new API uses function interface like
`uv_handle_set_data` and `uv_handle_get_data` to access data filed, while
the old API in 1.18 access the data filed directly. Note that in the latest
libuv version 1.38.2, the old API and new API are both supported. And libuv
is stable enough to support the old API for a long time.

(cherry picked from commit ab36eb90)

Previously, when backends connect to a proxy, we need to setup
domain socket pipe and send HELLO message(recv ack message) in
a blocking and non-parallel way. This makes ICPROXY hard to introduce
check_for_interrupt during backend registeration.

By utilizing libuv loop, we could register backend in paralle. Note
that this is one of the step to replace all the ic_tcp backend logic
reused by ic_proxy currently. In future, we should use libuv to replace
all the backend logic, from registeration to send/recv data.
Co-authored-by: NNing Yu <nyu@pivotal.io>
(cherry picked from commit 608514c5)

1. SQL script in /gpcontrib/gp_internal_tools/gp_internal_tools--1.0.0.sql
and /gpcontrib/uninstall_gp_session_state.sql changes current search_path,
this commit set search_path to default at the end of the above two SQL scripts
2.fix typos in /gpcontrib/gp_internal_tools/gp_internal_tools--1.0.0.sql
3.fix the expect caused by this fix.

This fixes the issue: https://github.com/greenplum-db/gpdb/issues/10575.

(cherry picked from commit 9d9662f0)
Co-authored-by: NLittleWuCoding <31984989+LittleWuCoding@users.noreply.github.com>

Unlogged tables do not propagate to replica servers, skip them and their
initialization forks.

optarg is a global and is not required to be passed to
process_greenplum_option. Also, for bool parameters pg_strdup must not
be called else it will fail with cannot duplicate null pointer (internal error)

Add gpreload behave test

* gpreload test where a table (public.t1) and
  view (gpreload_schema.t1) have the same name.

Add gpreload CI test jobs.
Co-authored-by: NEd Espino <eespino@vmware.com>

After losing tcp connection between gpmmon and gpsmon and subsequent its
reestablishment the gpsmon enters to infinite loop inside libevent
routine of timeout event processing. The reason was in calling event_set
function under new connection establishment on currently working (not
released) tcp_event that rewrites internal flags of that event and puts
it into unconsistent state.

The current fix releases tcp_event from participating in event loop
(takes to unpending state) under connection closing from gpsmon by
timeout so that the next new connection have to set tpc_event in event
loop again.