dbconn.connect() was using an odd pattern to set connection parameters
like client_encoding -- it would SET these parameters after the
connection was established and then issue a COMMIT. Even if your purpose
for connecting was read-only, you'd need to wait for a commit (which
also meant waiting for FTS, if a mirror was down). This is wasteful and
slow. Let's set those parameters during connection instead.

Now that we no longer wait for a transaction on connection, behave steps
that relied on those transactions need to be modified. Add unit tests to
cover the changed implementation.
Co-authored-by: NDavid Kimura <dkimura@pivotal.io>

In commit 48cb340d, we spawn a dedicated process to recovery distributed
transaction, newly requested connections will error out if dtx can't be
recovered in 4 seconds. This sounds correct to error out quickly because the
system is indeed unavailable, however, for some utilities like gpstart and
gprecoverseg, an error like this will be treated as a very scary issue and
will fail the utilities.

4 seconds is designed to notify the users as soon as possible, now it seems
better to keep the old behave before 48cb340d which is always waiting until
the dtx is recovered.

It tried to make sure gpfdist is ready by sleep 3 seconds before
queries, but that could not guarantee. This commit changes the sleep to
a retry until gpfdist is ready.

Signed-off-by: NSambitesh Dash <sdash@pivotal.io>

It fails and appears to not be maintained when:

--enable-gpfdist and --with-openssl are configured.

Co-authored-by: NJesse Zhang <jzhang@pivotal.io>

When given --print-failure-diffs, pg_regress prints the contents of
regression.diffs after a failure.

The seg contrib module has been missing the regress testfiles since
a merge in the past. This brings in the test files from 9.4 stable
such that we can again run installcheck on this module.
Reviewed-by: NJacob Champion <pchampion@pivotal.io>
Reviewed-by: NAshwin Agrawal <aagrawal@pivotal.io>

A later commit will remove V0 support.

Author: Andres Freund, with contributions by Craig Ringer
Reviewed-By: Peter Eisentraut, Craig Ringer
Discussion: https://postgr.es/m/20161208213441.k3mbno4twhg2qf7g@alap3.anarazel.de

Testing for equality in Python with 'is' tests for the variables
point to the same object, '==' tests for them containing the same
value. In this case we want '==' as we are comparing values.
Reviewed-by: NZhenghua Lyu <kainwen@gmail.com>

Previously, after system started in dispatcher mode, the first connection
is hijacked to do dtx recovery which has a few defects: Security issue that
any users can be promoted to superuser temporarily; The first connection
feels slow compared to rest of connection; Not all connections have enough
context to do dtx recovery, eg: background worker; Weried error message
for a user query if dtx recovery failed.

'dtx recovery process' is spawned to do the recovery of in-doubt distributed
transactions, it's a one-off process (no loop). before 'dtx recovery process'
exit normally, the system is considered in an inconsistent state, any background
worker that requires a "read-only" or "read-write" should not be scheduled and
any client connections will wait at most 2 seconds or then error out with
"System is recovering distributed transactions"
Co-authored-by: NGang Xiong <gxiong@pivotal.io>

Spotted by: Hubert Zhang <hzhang@pivotal.io>,
Idea from Ashwin Agrawal <aagrawal@pivotal.io>,
Idea from Heikki Linnakangas <hlinnakangas@pivotal.io>
Reviewed-by: NNing Yu <nyu@pivotal.io>
Reviewed-by: NAsim R P <apraveen@pivotal.io>

Discussion: https://groups.google.com/a/greenplum.org/d/msg/gpdb-dev/
od05RL6FGxc/IVtFjFPUBAAJA

In the past several weeks, gpload test case 22 failed two times.
It may report: Fatal Python error: GC object already tracked.
We have reproduced this issue locally and in dev pipeline for 3
days, but it can't be reproduced.
So we decide to disable it now, in order to not blocking other PRs.

Postgres optimizes the storage of the ItemPointer in the Gin Posting
List. It only stores 11 bits for the offset number because heap tables
only have enough tuples per block to fit in 11 bits. However,
Greenplum append-only tables store 16 bits worth of offset numbers.

Initially we thought we'd need to modify decode_varbyte() but it turns
out that it is OK.  It handles 48 bits already.
Co-authored-by: NAlexandra Wang <lewang@pivotal.com>
Co-authored-by: NAshwin Agrawal <aagrawal@pivotal.io>

We've seen frequent pg_rewind test failures due to promotion timeout. We
should increase the timeout. Also, add code to error out on timeout.
Co-authored-by: NNing Yu <nyu@pivotal.io>

Reviewed-by: Ashwin Agrawal

Any code dealing with replication ports was removed in previous commits; this
commit removes lingering references in comments and help text.
Co-authored-by: NJamie McAtamney <jmcatamney@pivotal.io>
Co-authored-by: NMark Sliva <msliva@pivotal.io>

* Add pipeline for running explain tests

This commit adds necessary files for creating a pipeline that tests a
custom GPDB and GPORCA commit against 5X_STABLE and gporca/master using
1TB test suite data in order to compare the plans generated.

For ORCA commit "Fix crash when merging group containing CTE consumer"
Authored-by: NChris Hajas <chajas@pivotal.io>

BuildPartitionNodeFromRoot will always use root relid no matter
what parameter is passed to it.

There is an optimization we might do: for some cases, like if user directly insert into
a child partition, then set estate->es_result_partitions to NULL to avoid dispatch it
since a huge partition table's partition info may cost much memory. We can spike this
later.

Previously the default value is 0, so normally log rotation happens each
day by default according to the other log related guc configurations.
That often makes a single log file rather large in real users'
environment.  Limiting the log file size so that people could easiler
open and search the log files.
Reviewed-by: NDaniel Gustafsson <dgustafsson@pivotal.io>

1) pm_launch_walreceiver was introduced in b824fe8f
   to avoid a potential deadlock when cluster is down after
   updating the new role in gp_segment_configuration but before
   sending promote signal. It is buggy now since some changes.
   We need to put pm_launch_walreceiver in shared memory since
   it's used in different processes.

2) The following assertion is not always true in a real scenario if
   the segment is being promoted. We could simply stop the connection
   for the !am_mirror case.

   ProcessStartupPacket()
     if (am_ftshandler)
     {
        Assert(am_mirror);

Ning Yu and we had some discussions for these bugs. He contributed for
issue 1 also.
Co-authored-by: NHaozhou Wang <hawang@pivotal.io>

Fts probe didn't report an error if it failed to resolve the DNS, so fts probe
can mark those segments as down. Previously, DNSLookupAsError was only set to
false in readCdbComponentInfoAndUpdateStatus(), however, codes keep coming and
evolving, fts probe may set DNSLookupAsError to true in other code paths.

To avoiding fts probe report a DNS error unexpectedly, use am_ftsprobe only to
determine it and hide the DNSLookupAsError from callers.

Commit ea9df812 relaxes the requirement
that all LWLocks are in a single array, which broke the LOCK_DEBUG
code for trying to acquiring an LWLock. This is a meatball surgery
attempt at making it at least compile and run, without trying to
reinvent the old usecase.

Minor tweaking of the code is also done to make LOCK_DEBUG cases be
identified identically.

Reported-by: Simon Gao in Github issue #7251
Reviewed-by: Ashwin Agrawal

The GPSMON_METRIC_MAX setting was only used for platforms where we
couldn't link with libsigar (namely Solaris, making these defines
even more redundant as they weren't applicable in the first case),
and support for that has long since been ripped out so remove the
useless defines too.

Also remove a commented out variable while in there.
Reviewed-by: NJimmy Yih <jyih@pivotal.io>
Reviewed-by: NHeikki Linnakangas <hlinnakangas@pivotal.io>

Reviewed-by: NDavid Yozie <dyozie@pivotal.io>

Since Greenplum is now based on PostgreSQL 9.4, update links to the
upstream documentation to refer to the 9.4 docset rather than older
versions. Also fix a few mentions of 8.3(.23) in the text along with
some spacing.
Reviewed-by: NDavid Yozie <dyozie@pivotal.io>

The search.cpan.org site has been discontinued and replaced with
metacpan.org, fix by replacing all links in the documentation.
There is an additional link in the testcode which will be fixed
as we merge with upstream.
Reviewed-by: NDavid Yozie <dyozie@pivotal.io>

In all but a few places we referred to the upstream documentation as
"PostgreSQL documentation", this fixes the few places where we had a
different spelling.
Reviewed-by: NDavid Yozie <dyozie@pivotal.io>

* docs - discuss pxf column projection

* add note about filter serialization

* column projection is always enabled

* edits requested by david

Related PR discussion:

kmacoskey:

Another piece of this can be removed with the same intention as the
other removal.

This line in the Makefile is related to a kfw snowflake dependency that
was being vendored into the Clients/Loaders package as brought in by
Ivy. Note the path, ext. It was dropped from Ivy by releng awhile ago,
2c6f4c9d#diff-271f398f0c07147024e7980737cddb7d . The cp here probably
should have been removed along with the change to Ivy, but it was
missed. It can be removed - but lets make sure the Clients team is
aware.

pf-qiu:

I think it's safe to do this now.
Co-authored-by: NBradford D. Boyle <bboyle@pivotal.io>
Co-authored-by: NBen Christel <bchristel@pivotal.io>

Co-authored-by: NSoumyadeep Chakraborty <sochakraborty@pivotal.io>
Co-authored-by: NBen Christel <bchristel@pivotal.io>

Since we have removed the Ivy dependency manager entirely from GPDB 6+,
we do not need this code anymore. It used to copy Ivy-managed libraries
into the Greenplum installation for packaging.
Co-authored-by: NSoumyadeep Chakraborty <sochakraborty@pivotal.io>
Co-authored-by: NBen Christel <bchristel@pivotal.io>