The main change here is that we completely get rid of the ORCA prod
pipeline, as there's no need to create a tag/publish artifacts anymore.

We also now build and test the orca unit tests during the compile step.

Additionally, I've refactored the orca-dev "slack command" pipeline to
use the canonical compile_gpdb.bash script and based the pipeline off
the existing PR pipeline. This simplifies the `test_orca_pipeline.yml`
file significantly and it now doesn't rely on custom scripts.

There are still quite a few files in the concourse directory that are
used by the `test_explain_pipeline.yml` and a couple of pipelines that
are external to this repo. Some of these aren't actively being used and
need to be cleaned up/removed, but I'll leave that for a later time.

Also, uses https for xerces-c download and checks hash to ensure correct
download.

Previously, we used a config file that was modified by cmake. Instead,
use configure to modify macros. Additionally use a compile definition in
cmake so we don't need a separate config file in gpos.

This also enables compile-time warnings in the src/backend/gporca files. C++ files
in the translator never had these warnings enabled, and so are not
enabled in src/backend/gpopt. A block of unused code is removed to get
pass the compile warnings.

This simplifies some of the cmake files further.

Cmake should only build targets for use with gporca_test. GNU make
should build orca within GPDB. Remove the install target to decrease the
chance for confusion.

Also, add makefile to test directory so make clean works.

ORCA doesn't need a specific version anymore.

A query plan will now look like:
```
                QUERY PLAN
------------------------------------------
 Result  (cost=0.00..0.00 rows=1 width=1)
 Optimizer: Pivotal Optimizer (GPORCA)
(2 rows)
```

Update regression tests affected by this change.

Additionally, use g++ for compiling mock unit tests since we're linking
c++ files.

ORCA previously was built into a shared library using cmake. Now, we
build ORCA into the postgres binary.

We no longer need to check for ORCA libraries or version, as ORCA now
exists in the same repo and the concept of a version is not needed.

ORCA currently exists as a standalone repo. However, it requires
significant coordination to build GPDB with ORCA or make changes to
ORCA. With these commits, we get rid of the versioning concept, which
hopefully will make both building and developing on ORCA much easier
(and you won't hit version mismatch issues anymore)

The GPDB build process will now also build ORCA using make. If you want
to run the ORCA unit tests, those will continue to use cmake/ctest.
Cmake will not install ORCA or build ORCA with GPDB, it is only used to
run ORCA unit tests.

This merge commit includes all ORCA commits from when ORCA was
open-sourced until f6044123 (v3.97.1)

This commit was generated using `git subtree add -P src/backend/gporca
git@github.com:greenplum-db/gporca.git HEAD`.

Add 'src/backend/gporca/' from commit 'f6044123'
git-subtree-dir: src/backend/gporca
git-subtree-mainline: b6fbabdb
git-subtree-split: f6044123

Discussion: https://groups.google.com/a/greenplum.org/d/msg/gpdb-dev/q4g73r6Y_qA/OpZXDFFwAwAJ

The DistributionKey for path of FULL JOIN may contain multiple ECs. We
need to make sure each of them contains expr belonging to the given
list, to tell grouping on this given set of exprs can be done in place
without motion.

Fixes #9784.
Reviewed-by: NHeikki Linnakangas <hlinnakangas@pivotal.io>

* docs WIP - add some python3 info

* move data type mapping to pl/python page; add more py3 info

* add set-returning function example

* address comments from david

Co-authored-by: NRyan <ryan@chapterx.com>

Without this patch:

    select distinct x from xidtab;
    ERROR:  could not identify an ordering operator for type xid
    LINE 1: select distinct x from xidtab;
                            ^
    HINT:  Use an explicit ordering operator or modify the query.

There were two similar but distinct issues:

1. transformDistinctToGroupBy() called addTargetToSortList(), even though
   the expression might not have ordering operators. There's another
   function, addTargetToGroupList(), for that. Use that.

2. make_distribution_exprs_for_groupclause() tried to create a PathKey on
   the expression, even if it didn't have ordering operators.

Also update a comment to explain why we do the DISTINCT -> GROUP BY
transformation. The comment said that it was because DISTINCT doesn't
support hashing, but that was fixed in PostgreSQL 8.4 already. However,
the planner still doesn't know how to perform the "pre-unique"
optimization for DISTINCT, so you still get better plans with GROUP BY.

This came up when working on the v12 merge. A new test was added to
upstream, 'hash_func', which contained a query like this involving
'relacl' datatype. But it's a pre-existing issue.

Fixes https://github.com/greenplum-db/gpdb/issues/9855Reviewed-by: NPengzhou Tang <ptang@pivotal.io>

The related code path should never been run into else that implies a bug so no
need of worry about this bug much. I manually tested by hacking the code.

Spotted by Ashwin Agrawal

Also add note to not change GUC value.

With ORCA going into GPDB, we'll now be using the infrastructure scripts
in the gpdb5 repo instead of the gpdb master repo.

* Revert "Refactor Sharedsnapshot"

This reverts commit cfc911d8.

we create a DSM for sharedsnapshot synchronization between writer and reader. the DSM lifecycle is the same as a session, which we use dsm_ping_maping to mark dsm_segment.

However, in centos 6, the DSM will randomly be free after some test cases even the session does not exist. It seems to related to dsm itself.

Partially revert this part of the code. still left shared snapshot dump to DSM.

* Dump sharedsnapshot to DSM

Cursor writer gang dump snapshot using dsm instead of BufFile. A loop
buffer is kept in Sharedsnapshot to store dump dsm handle. Each cursor
declaration, gpdb dispatch to QE twice. The first dispatch, we force writer
gang dumping snapshot. The Second dispatch, we create a new reader gang to
execute cursor fetch and do not wait for reader gang finishes. Writer gang does
not know when reader gang sync the snapshot done, but must be in a short
period of time after the first dispatch done. The loop buffer is big enough,
so we can repeatedly using this space to store snapshot dump.

* docs - provide some config info for PAM authentication

* misc edit per david

* misc edits to genericize

On login shells SLES replaces PATH with a fixed known PATH. This was
causing compilation to fail by removing gcc from the PATH. Previously,
gcc was being set in /root/.bashrc, but that was removed in favor of
setting it with docker ENV.

This removes invoking compile_gpdb.bash with a login shell to retain the
PATH containing gcc on SLES.

This PR fixed the hang issue in PR(https://github.com/greenplum-db/gpdb/pull/9726).

The Postgres planner could optimize SelectLockingClause to acquire
RowShareLock instead of ExclusiveLock, but the orca doesn't optimize
it. So the following tests may hang with orca:
```
1: begin;
1: select * from t_lockmods where c<3 for share;
2: select * from t_lockmods for share;
2: select * from t_lockmods for update skip locked;
2: select * from t_lockmods where c>=3 for update nowait;
2: select * from t_lockmods for update nowait;
```
Reviewed-by: NHubert Zhang <hzhang@pivotal.io>

Correctly capitalize Zstandard, and remove multiple trailing periods
in sentences.
Reviewed-by: NMel Kiyama <mkiyama@pivotal.io>

The wait policy, NOWAIT or SKIP LOCKED, only affects how SELCT locks
rows as they are obtained from the table. They are used by the execution.
The SELECT statement with locking clause may upgrade the lock mode.
This PR removes the nowait logic in opening the relation in the parser analyze
stage. So, the SELECT statement with locking clause will wait for the table
lock if it can't hold the table lock.

Co-authored-by: Zhenghua Lyu zlv@pivotal.io

* docs - add PERSISTENTLY keyword to LOG ERRORS clause

Also add
--add functions gp_read_persisistent_error_log() and gp_truncate_persisistent_error_log()
--add information to run gpexterrorhandle.sql to install the functions

* docs - fix typos

* doc - review updates and fixes

* docs - updates to Encrypting Client/Server Connections

clarified/enhanced SSL configuration information.

* docs - minor edit

As @hidva pointed out, the prototype for pg_plan_query() in gpdbdefs.h
was incorrect. On closer inspection, the preprocess_query_optimizer()
prototype at least was also incorrect. But none of these prototypes in
gpdbdefs.h are actually needed. Some of them are not used in ORCA at all,
and others in the headers that are #included. Remove them all.

Also remove a few of the #includes that are not needed. Many more of them
could probably be removed, I didn't systematically go through all of them,
I just removed a few that seemed unlikely to be needed.

When calculating the statistics of a filter node, the histograms of
newly projected columns are set to empty. Such histograms are not "well
defined" and thus should not be used to derive cardinality. Instead, we
use the default cardinality in such cases.

However, there was a bug which calculated the cardinality from non "well
defined" histograms in the presence of a disjunction on newly projected
columns. This would result in gross underestimation of expected rows of
the filter.

This commit fixes this issue.
Co-authored-by: NAshuka Xue <axue@pivotal.io>
Co-authored-by: NShreedhar Hardikar <shardikar@pivotal.io>

Consider the case below:

create table foo (citext a, citext b); explain select min(a), count(distinct a)
from foo;

Today in GPDB, no combine function exists for a `min` on citext. So
`ExecInitAgg` will fail for top level aggregate. Aggs with no combine function
are call non-splittable.

So we should create multi-phase DQAs only if all participating aggs are
splittable.

If have_yaml is set to yes, the autoconf has already added -lyaml to
LIBS so there is no need to add it again.
Reviewed-by: NHeikki Linnakangas <hlinnakangas@pivotal.io>

While the unit test in question isn't inspecting the contents of the
array, not initializing caused a compiler warning for uninitialized
variable.

Reported by bzhaoopenstack on Github

WaitLatch() asserts timeout >= 0 when flag WL_TIMEOUT is set.