When constructing a DbURL without an explicit username, /usr/bin/id was
executed regardless of whether we needed its output or not. That seems
excessive; let's only use it if the environment doesn't already have
what we need.

In a classic example of "too many layers", chk_local_db_running() used
FileDirExists, which spins up a new Python interpreter, to check for the
existence of a file. Just use os.path.exists() directly.

This commit checks whether there are any active pg_rewind connections
when the mirror is down and displays an appropriate status message if
so.  To facilitate this, the pg_rewind call in gprecoverseg has been
modified to specify an application_name when connecting to the database.
Co-authored-by: NJacob Champion <pchampion@pivotal.io>
Co-authored-by: NJamie McAtamney <jmcatamney@pivotal.io>
Co-authored-by: NJimmy Yih <jyih@pivotal.io>

We had cherry-picked this from a later version of PostgreSQL in commit
e43797f7, and got the second copy in the PostgreSQL 9.4.20 merge.

And a stray reference to WORKFILE_HASHTABLE_NUM_PARTITIONS, which doesn't
exist anymore, from a comment.

It was only needed for EState. builtins.h is included from many different
places, so it seems good to not pull in executor.h. Add explicit #includes
into many places that were relying on the indirectly included executor.h.

Commit 7fa85902 removed TINC from the tree, but commit ad6b920f
accidentally re-introduced a few (dead) files. Remove them yet again.
Reviewed-by: NHeikki Linnakangas <hlinnakangas@pivotal.io>
Reviewed-by: NJimmy Yih <jyih@pivotal.io>
Reviewed-by: NGeorgios Kokolatos <gkokolatos@pivotal.io>

gpfdist tests have the same problem happened in sreh, increase the sleep.

	commit bb8575a9
	Author: Heikki Linnakangas <hlinnakangas@pivotal.io>
	Date:   Mon Nov 27 21:57:08 2017 +0200

	    Increase sleep between launching gpfdist and running test queries.

	    We've seen a lot of failures in the 'sreh' test in the pipeline, like this:

	    --- 263,269 ----
	      FORMAT 'text' (delimiter '|')
	      SEGMENT REJECT LIMIT 10000;
	      SELECT * FROM sreh_ext;
	    ! ERROR:  connection failed dummy_protocol://DUMMY_LOCATION
	      INSERT INTO sreh_target SELECT * FROM sreh_ext;
	      NOTICE:  Found 10 data formatting errors (10 or more input rows). Rejected related input data.
	      SELECT count(*) FROM sreh_target;

	    I don't really know, but I'm guessing it could be because it sometimes
	    takes more than one second for gpfdist to fully start up, if there's a lot
	    of disk or other activity. Increase the sleep time from 1 to 3 seconds;
	    we'll see if that helps.

Most of the "masking" code was removed in 9.3 merge already, and the
rest in commit c9dee15b. But these were left over.
Reviewed-by: NAshwin Agrawal <aagrawal@pivotal.io>

Reviewed-by: NHeikki Linnakangas <hlinnakangas@pivotal.io>

We can allow this even without any specific knowledge of the semantics
of the window function, so long as pushed-down quals will either accept
every row in a given window partition, or reject every such row.  Because
window functions act only within a partition, such a case can't result
in changing the window functions' outputs for any surviving row.
Eliminating entire partitions in this way obviously can reduce the cost
of the window-function computations substantially.

The fly in the ointment is that it's hard to be entirely sure whether
this is true for an arbitrary qual condition.  This patch allows pushdown
if (a) the qual references only partitioning columns, and (b) the qual
contains no volatile functions.  We are at risk of incorrect results if
the qual can produce different answers for values that the partitioning
equality operator sees as equal.  While it's not hard to invent cases
for which that can happen, it seems to seldom be a problem in practice,
since no one has complained about a similar assumption that we've had
for many years with respect to DISTINCT.  The potential performance
gains seem to be worth the risk.

David Rowley, reviewed by Vik Fearing; some credit is due also to
Thomas Mayer who did considerable preliminary investigation.

PostgreSQL and GPDB both have this change. We prefer to keep the same
with PostgreSQL, so remove this temporarily from GPDB and will
cherry-pick d222585a from PostgreSQL later.
Reviewed-by: NHeikki Linnakangas <hlinnakangas@pivotal.io>

Previously in qual_is_pushdown_safe(), we had this function
qual_is_pushdown_safe_set_operation() to check if the qual contains
references to WindowFunc node.

This is unnecessary because if a qual contains references to WindowFunc
node, its corresponding column would be marked as unsafe that prevents
the qual from being pushed down. So remove the related functions.
Reviewed-by: NHeikki Linnakangas <hlinnakangas@pivotal.io>

The gpstate utility opens up the postmaster.pid file to read and
display its contents to the user in a nice way.  One of those items
from postmaster.pid is the pid of the postmaster process.  When the
postmaster is in single-user mode (e.g. pg_rewind ensuring clean
shutdown), the pid in the postmaster.pid file is negative.  This is
intentional, but we should display it as positive to the user to avoid
any confusion.
Co-authored-by: NJimmy Yih <jyih@pivotal.io>
Co-authored-by: NShoaib Lari <slari@pivotal.io>

This reverts commit bf7836f0.

crash_recovery_dtm test has a scenario which intends to test if QE
undergoes crash recovery after writing prepare record but before
responding to QD, the abort processing is completed fine. For the same
test used GUC `debug_abort_after_segment_prepared` to PANIC all the QE
at that specific point for DELETE. Next test executes SELECT query to
validate the DELETE was aborted. But flakiness comes if this SELECT
query gets executed while PANIC processing is still underway as test
had no way to wait till PANIC and restart completed before running the
SELECT.

Now the test instead uses fault injector to sleep at intended point
and uses pg_ctl restart -w to make sure recovery is completed and only
after that the SELECT query will be executed.

So, as a result removing the test only GUC
`debug_abort_after_segment_prepared` and related code for it.

explain_format tests validate for memory related information. But the
printing for that information is not stable and varies based on orca vs
planner, assert enabled vs disabled, query reusing the slice vs run on
fresh session. Plus, future modifications not related to explain
formatting can cause this test to fail. Hence, only minimal explain
format validation which is found to be stable currently is being
retained for this test.

Better alternative needs to be found to perform for full validation for
explain formatting, seems sql way is too fragile for it.
Co-authored-by: NAshwin Agrawal <aagrawal@pivotal.io>

Currently exlusion constraints do not work correctly in MPP environment.
For example, if the exclusion constraint is on a column which is not the
table's distribution key, then it is possible to get wrong results.

Following statements should give 1 row because first tuple should
exclude the second.
```
CREATE TABLE t(a int, b int, EXCLUDE (b WITH =)) DISTIBUTED BY (a);
INSERT INTO t values (1, 1), (2, 1);
```

However, that is not currently the case if the distribution key hashes
to different segments. This commit removes exclusion constraints feature
entirely until there is a way to coordinate exclusion constriants
between the segments.
Co-authored-by: NAdam Berlin <aberlin@pivotal.io>
Co-authored-by: NMelanie Plageman <melanieplageman@gmail.com>
Co-authored-by: NDavid Kimura <dkimura@pivotal.io>

We noticed a case in CI where it seemed like it took longer than 30
seconds to promote the mirror during recovery.
Co-authored-by: NMelanie Plageman <mplageman@pivotal.io>

In plans with a Nested Loop join on the inner side of another Nested Loop
join, the planner could produce a plan where a Motion node was rescanned.
That produced an error at execution time:

ERROR:  illegal rescan of motion node: invalid plan (nodeMotion.c:1604)  (seg0 slice4 127.0.0.1:40000 pid=27206) (nodeMotion.c:1604)
HINT:  Likely caused by bad NL-join, try setting enable_nestloop to off

Make sure we add a Materialize node to shield the from rescanning in such
cases.

While we're at it, add an explicit flag to MaterialPaths and plans, to
indicate that the Material node was added to shield the child node from
rescanning. There was a weaker test in ExecInitMaterial itself, which just
checked if the immediately child was a Motion node, but that feels
sketchy; what if there's a Result node in between, for example? However, I
kept the direct check for a Motion node, too, because I'm not sure if there
are other places where we add Material nodes on top of Motions, aside from
the call in create_nestloop_path() that this fixes. ORCA probably does
that, at least.

Fixes https://github.com/greenplum-db/gpdb/issues/6769Reviewed-by: NPengzhou Tang <ptang@pivotal.io>
Reviewed-by: NPaul Guo <pguo@pivotal.io>

update_mergeclause_eclasses() asserts that 'left_ac' and 'right_ac' are
not NULL. Therefore, it makes no sense to call it when left_ac is NULL.
But we should call it unconditionally, like in other similar functions in
the same file.

I can't come up with a test case that would trip the assertion, the ECs
have been correctly set by the time we get here. And I haven't been able
to come up with a query where the lack of the call would matter, although
I think it is potentially a problem. So, just fix it.

Also remove redundant assertions on left_ac and right_ac being not NULL.
update_mergeclause_eclasses() asserts that, too.

Fixes https://github.com/greenplum-db/gpdb/issues/5372Reviewed-by: NMelanie Plageman <mplageman@pivotal.io>

These variables are since commit 4eaeb7bc no longer used by any
caller, so remove.
Reviewed-by: NAshwin Agrawal <aagrawal@pivotal.io>
Reviewed-by: NAdam Berlin <aberlin@pivotal.io>

cdb_util.sql was installed in "make install", but it was not loaded by
initdb.
Reviewed-by: NVenkatesh Raghavan <vraghavan@pivotal.io>
Reviewed-by: NDaniel Gustafsson <dgustafsson@pivotal.io>

* docs - appendoptimized is an alias for appendonly storage option

* option names lower case, add legacy, misc edits

Just checking gp_stat_replication is not enough at the end of the test,
we need to actually validate gp_segment_configuration reflects primary
and mirror are in sync.

* docs - update compression information
-Add a topic that lists features that support configuring compression.
-update gp_workfile_compression GUC. Remove requirement to install zstd.

* docs - update compression information based on review comments.
-changed description to point to feature/utility for specific compression information.
-added general, pivotal, and OSS specific information about compression requirements.

* docs - compression information - reivew edit.

missing_xlog brings the mirror down and fiddles with its state. Before
this commit it only validates mirror is up but misses to make its in
sync state. Tests following to this test make assumption of starting
when all the primaries and mirrors are in sync state. Hence, every
test should end leaving primaries and mirrors as up and sync.
Co-authored-by: NAlexandra Wang <lewang@pivotal.io>

Bump ORCA version to v3.27.0
Signed-off-by: NRahul Iyer <riyer@pivotal.io>

Test uao_crash_compaction_column needs to wait for record to be
replayed on mirror for checking. Depending on work load generated by
the tests this can take long time. Hence, moving the test to run at
start of schedule to reduce waiting for that time.

Also, to reduce the flakiness seen in failure increase the number of
retries in test to check if replay_location = flush_location on mirror
from 1000 to 5000.
Co-authored-by: NAlexandra Wang <lewang@pivotal.io>

Co-authored-by: NChris Hajas <chajas@pivotal.io>
Co-authored-by: NShreedhar Hardikar <shardikar@pivotal.io>

Co-authored-by: NDavid Sharp <dsharp@pivotal.io>
Co-authored-by: NBradford D. Boyle <bboyle@pivotal.io>

Co-authored-by: NNandish Jayaram <njayaram@pivotal.io>
Co-authored-by: NSambitesh Dash <sdash@pivotal.io>

This commit is to fix PANIC on mirorr with "WAL contains references to
invalid pages" on xlog replay for truncate record with offset 0.

Primary creates the file first and then writes the xlog record for the
creation for AO tables similar to heap.  Hence, file can get created
on primary without writing xlog record if failure happens on primary
just after creating the file. This creates situation where VACUUM can
generate truncate record based on aoseg entry with eof 0 and file
present on primary. Then during replay mirror may not have the file,
as was never created on mirror. So, avoid adding the entry to invalid
hash table for truncate at offset zero (EOF=0).  This avoids mirror
PANIC, as anyways truncate to zero is same as file not present.
Co-authored-by: NAlexandra Wang <lewang@pivotal.io>

This was spotted by Pengzhou Tang, during code inspection. So, fixing
this, I don't think had any ill effect but definitely is unnecessary.