The source file and line number is removed from the answer files because
of a code change that replaced elog(ERROR) with ereport(ERROR) in commit
89b4365e.

Also removed the step to create extension for fault injector from
setUp() method.  That was causing the check_connection() method to
report assert failure if master was recovering from PANIC.

This is a quick fix to make the CI green.  The tests need to be moved
from TINC to isolation2 but that's a separate commit.

Now that we have merged the upstream changes that introduced the
ModifyTable node, it seems like a good time to fix RETURNING. So, here
we go.

Internal errors, i.e. errors without an error code, have the source file
name and line number appended to the error message. That's troublesome for
regression tests, as the line number needs to be memorized in the expected
output. Use a proper error code, to avoid that, and because it's nice to
have proper error codes anyway.

test_reindex_scenarios takes long in tinc, the scenarios are pretty
simple so adding them in isolation2.

The fault was set incorrectly with tablename='None' invalidating the
test and also more importantly taking 6 mins instead of 36 secs. This
change brings it back to normal runtime. Still looking into moving it
out of TINC.

Previously, the validation was run on the pipeline after it was written to disk

Now, run validation on the in-memory representation of the pipeline. To make
this pattern more natural, bring the validate pipeline logic into the
gen_pipeline.py script. Previously, the validate pipeline script was run
separately, but I don't think it's used in isolation any more.
Authored-by: NC.J. Jameson <cjameson@pivotal.io>

Signed-off-by: NJesse Zhang <sbjesse@gmail.com>

ORCA has historically ignored type modifiers from databases that support
them, noticeably Postgres and Greenplum. This has led to surprises in a
few cases:

1. The output description over the wire (for Postgres protocol) will
lose the type modifier information, which often meant length. This
surprises code that expects a non-default type modifier, e.g. a JDBC
driver.

2. The executor in some cases -- notably DML -- expects a precise type
modifier. Because ORCA always erases the type modifiers and presents a
default, the executor is forced to find that information elsewhere.

After this commit, ORCA will be aware of type modifiers in table
columns, scalar identifiers, constants, and length-coercion casts.
Signed-off-by: NShreedhar Hardikar <shardikar@pivotal.io>

Signed-off-by: NJesse Zhang <sbjesse@gmail.com>

The gprecoverseg tool has been broken after filerep and persistent
tables were removed. This commit makes rebalance work again by using
full recovery instead of incremental. This commit should maybe be
reverted once pg_rewind is implemented since incremental would then be
usable.
Co-authored-by: NJimmy Yih <jyih@pivotal.io>
Co-authored-by: NDavid Kimura <dkimura@pivotal.io>

Author: C.J. Jameson <cjameson@pivotal.io>

Prior to this commit, Orca returned wrong results for queries with left
outer joins containing predicates with `IS DISTINCT FROM`. This has been
fixed as part of ORCA v2.54.4.These are the associated tests.
Signed-off-by: NEkta Khanna <ekhanna@pivotal.io>

very useful function to check AOWrite, now supports
AoHeaderKind_NonBulkDenseContent and AoHeaderKind_BulkDenseContent

Adding tests here for
- Setting the gp_default_storage_options with --masteronly or -m option is not supported
- Setting different seg and master value is expected to fail
- gpconfig -s gp_default_storage_options show show current settings
- CTAS
- Add/split partition to a partition table after the gp_default_operations changed with GUC value

This helps to remove lot of tinc tests testing various combinatios
around it.

commit a7cce539 first bumped
statement_mem for ICW to 250MB, to address the issue of gpcheckcat
needing more memory in the later stages of installcheck-world.
installcheck-good, didn't really need the extra memory back in July
2017. However, memory consumption has slowly crept up in the past half a
year and now we have to bump this up or some or the tests will fail
under planner, duh.

-- add timestamp

- Indentation and new line adjustments

flock function on AIX only permits exclusive lock on file opened
for writing. Exclusive lock on file opened for reading will fail
always. Disable this on AIX platform.

The referenced bug does not affect modern versions of macOS (tested
back to El Capitan), and Snow Leopard is EOL since long.  Further,
this workaround is not present/needed in PostgreSQL, with the code
running on Leopard as well as Snow Leopard in the buildfarm without
issues, so it seems that this problem has been fixed even in Snow
Leopard in the later patchlevels.  Remove the workarounds and align
the code with upstream.

This reverts commit 66faeec2.

Behave doesn't support table names with dots yet, will re-do this in the
upcoming PR which supports more special characters.

Take dot as a valid character, and quote the corresponding external
table's name.

This is the first batch of commits from PostgreSQL 9.0. This takes us almost
up to 8.5alpha3, to the point just before the Hot Standby commit. (The
development branch was labeled 8.5devel at the time, because it was decided
only later in the release cycle, to bump the version number to 9.0.)

Notable upstream changes in this batch, from a GPDB point of view:

* EXPLAIN code was heavily refactored. You can now specify JSON (or YAML
  or XML) format output. To make that work with all the GPDB addons and
  changes to EXPLAIN, cdbexplain.c was moved to
  src/backend/commands/explain_gp.c, and it is now #included from
  explain.c. In EXPLAIN ANALYZE, the actual costs and row counts are now
  printed on the same line as the estimated costs, like in upstream,
  instead of separate lines with "Rows in" and "Rows Out" etc. fields.
  A lot of the extra GPDB information printed in the text format, is not
  yet present in the JSON or other formats. Some GPDB specific information
  which was previously always printed, now requires EXPLAIN (VERBOSE). The
  work remaining on this refactoring is to convert the GPDB EXPLAIN GUCs to
  EXPLAIN options, and to ensure that the formats are valid when these
  options are turned on.

* EXPLAIN (COSTS OFF) option was added. We could start using this in all
  the GPDB-added tests too, that currently rely on 'gpdiff' to mask out the
  costs from the plans.

* Deferred unique constraints. These work, but if a deferred constraint
  violation is caught at COMMIT time, the error message is not very nice.
  To fix that, some work is required in the error handling of errors during
  distributed commit.

* PL/python test files were rearranged. This required moving around quite a
  few GPDB-added parts, too.

* So-called "flat files", used internally to store information about
  crucial catalog tables like pg_database and pg_authid, were removed. We
  had also used a flat file for the GPDB-specific 'pg_auth_time_constraint'
  catalog, so the code to enforce those time constraints had to be
  rewritten accordingly, to also not use a flat file. As a result, you no
  longer get the "WARNING: time constraints added on superuser role"
  message when you modify the pg_auth_time_constraints table directly DML
  commands, instead of the normal ALTER ROLE commands.

* Another side-effect of the "flat file" removal was that the
  in-shared-memory values for XID wraparound limits, xidWarnLimit and
  xidStopLimit, are no longer updated at postmaster startup, nor when a
  database is DROPped or CREATEd. They used to be updated whenever the flat
  file was updated, but now it's only done at a VACUUM. That required
  adding additional VACUUM commands to the 'xid_limits' TINC test. It's a
  bit questionable, whether the behaviour, when a new database is created
  from 'template0' is appropriate anymore. But refactoring all that is a
  bigger, and separate, job. (In the upstream, autovacuum takes care of
  vacuuming template0, and also takes care of updating the in-memory limits
  soon after postmaster startup.)

* DML commands are now handled by adding a ModifyTable node on top of the
  plan. We had quite a few GPDB-specific changes in the ExecInsert /
  ExecUpdate / ExecDelete code, that was now moved to be part of the
  ModifyTable implementation. The new ModifyTable node is very similar to
  the DML node used by ORCA-generated plans. In the future, we should
  remove the DML node type, and use the normal ModifyTable node in ORCA
  plans as well. Similarly, locking of rows in FOR SHARE/UPDATE queries, as
  well as actual UPDATEs, is now handled by the new LockRows node type.

* bytea_output default changed to 'hex'. We had already backported the
  bytea_output setting, but left the default to 'escape'. It is now
  changed to 'hex'. That caused quite a few changes in expected outputs.

* Whitespace is now significant in regression test output. This required
  fixing quite a expected output files. Most of this was done already as
  separate commits before this merge, but there was still plenty left here.

* psql now prints newlines in datums differently. We had already
  backported the code for the new format from 9.1, along with all the rest
  of psql code, but now the default is changed to the new format.

This merge also includes cherry-picks of some upstream commits from later
in the 9.0 series, or from later PostgreSQL versions, mostly to fix bugs
that the regression tests happened to hit:

* Update PL/perl tests for multi-dimensional arrays, to match PostgreSQL
  v10. We had backported the feature earlier, so let's keep the tests in
  sync. Also update one related error message to match upstream.

* To fix join removal bugs, src/backend/optimizer/plan/analyzejoins.c was
  cherry-picked in whole from the tip of Postgres REL9_0_STABLE branch.
  Also, commits b78f6264 and bab9f1d.

* Commit c45841f, to fix bug with whole-row references to append subplans.

* Commit 894fc6b7, to fix bug in aclexplode function.

* Commit ea46000a, to "Arrange for client authentication to occur before we
  select a specific database to connect to". We had backported this earlier
  already, this brings in more parts that are now applicable to GPDB. This
  allows us to remove some hacks in place that worked around the lack of
  BKI_ROWTYPE_OID, which we now have.

* Commit b220b5a, to avoid crashing when we have problems unlinking files
  post-commit.

Co-Author: Jacob Champion <pchampion@pivotal.io>
Co-Author: Daniel Gustafsson <dgustafsson@pivotal.io>
Co-Author: Max Yang <myang@pivotal.io>
Co-Author: Xiaoran Wang <xiwang@pivotal.io>

In the upcoming PostgreSQL 9.0 merge, validating the custom variable
classes was causing grief in with the new plpgsql.variable_conflict GUC.
If you set that GUC after loading plpgsql in the QD, plpgsql migh still not
be loaded in QE processes. If the GUC was changed in the QD, the QD would
dispatch the new value to QE processes, which threw an error because the
'plpgsql' custom variable class was not defined.

In principle, we would have the same problem with any other GUC added by
a loadable module.

To fix,  cherry-pick upstream commit from 9.2, to remove the GUC and the
validation of custom variable classes altogether. Might as well do that
now, rather than make some temporary work-arounds, since it's a sensible
change that we'll get eventually from upstream anyway.

Upstream commit:

commit 1a00c0ef
Author: Tom Lane <tgl@sss.pgh.pa.us>
Date:   Tue Oct 4 12:36:18 2011 -0400

    Remove the custom_variable_classes parameter.

    This variable provides only marginal error-prevention capability (since
    it can only check the prefix of a qualified GUC name), and the consensus
    is that that isn't worth the amount of hassle that maintaining the setting
    creates for DBAs.  So, let's just remove it.

    With this commit, the system will silently accept a value for any qualified
    GUC name at all, whether it has anything to do with any known extension or
    not.  (Unqualified names still have to match known built-in settings,
    though; and you will get a WARNING at extension load time if there's an
    unrecognized setting with that extension's prefix.)

    There's still some discussion ongoing about whether to tighten that up and
    if so how; but if we do come up with a solution, it's not likely to look
    anything like custom_variable_classes.

This removes much of the GPDB machinery to handle "deduplication paths"
within the planner. We will now use the upstream code to build JOIN_SEMI
paths, as well as paths where the outer side of the join is first
deduplicated (JOIN_UNIQUE_OUTER/INNER).

The old style "join first and deduplicate later" plans can be better in
some cases, however. To still be able to generate such plan, add new
JOIN_DEDUP_SEMI join type, which is transformed into JOIN_INNER followed
by the deduplication step after the join, during planning.

This new way of constructing these plans is simpler, and allows removing
a bunch of code, and reverting some more code to the way it is in the
upstream.

I'm not sure if this can generate the same plans that the old code could,
in all cases. In particular, I think the old "late deduplication"
mechanism could delay the deduplication further, all the way to the top of
the join tree. I'm not sure when that woud be useful, though, and the
regression suite doesn't seem to contain any such cases (with EXPLAIN). Or
maybe I misunderstood the old code. In any case, I think this is good
enough.

This was removed in PostgreSQL 8.4 already, but we failed to remove it
during the 8.4 merge, apparently. It wasn't used for anything anymore.

Commit ce3153fa, about to be merged from PostgreSQL 9.0 soon, removes
the -w option from pg_regress's "diff" invocation. That commit will fix
all the PostgreSQL regression tests to pass without it, but we need to
also fix all the GPDB tests. That's what this commit does.

I did much of this in commit 06a2bb64, but now that we're about to
actually merge that, more cases popped up.

Co-Author: Daniel Gustafsson <dgustafsson@pivotal.io>

Add sleep 1s after send cancel signal

It's a GUC, like any other. Testing that it's set correctly in every
possible context, when set to a user, when set on a database, etc., seems
excessive.

Also, if by some freak incident the GUC is set incorrectly, the
consequences are not very serious. The table just gets created with wrong
parameters, but it's still a valid table, and if you look at it with e.g.
\d, you will see the options actually used.

[ci skip]

Delete udf_exception_handling tinc tests.

Also remove tinc stage from CI `aocoalter_catalog_loaders`. Majority
of tests from this group has been moved to ICW and remaining ones are
now part of generic `make storage` target.

Simplified the tests by setting the GUCs at session level instead of
reloads. Also removed creating tables with index, sequence etc as not
needed for these tests.

So, same coverage is achieved with ICW in 15 secs vs older version
taking 15 mins in TINC.

Co-Author: Kevin Yeap <kyeap@pivotal.io>
Co-Author: Nadeem Ghani <nghani@pivotal.io>
Co-Author: Jim Doty <jdoty@pivotal.io>

This makes it a lot easier to diagnose problems after a failure.

This function is only used in memquota.c, and based on the API it
seems reasonable to keep it there, so declaring it static.