gp_dump_agent is used by management utils for backup/restore operation.
Apply similar patch for pg_dump to cdb_dump_agent.
Signed-off-by: NYuan Zhao <yuzhao@pivotal.io>

We will first support external table query execution on master segment
before we fully support task dispatch on master segment.

1. Modify grammar to enable ON clause for external table.
2. Modify ExtTableEntry catalog to record on clause.
3. Update legacy planner to support on clause for external table.
4. Update pg_dump.
5. Update regress tests.
Signed-off-by: NHaozhou Wang <hawang@pivotal.io>
Signed-off-by: NAdam Lee <ali@pivotal.io>
Signed-off-by: NYuan Zhao <yuzhao@pivotal.io>

* Update catalog version

For "Support ON MASTER clause for external table"

* Add on master regression tests for gpcloud

There was a rather ugly mechanism in fileam.c, to ensure that the
utl_fclose() function was called for any still-open external table, on
abort. That relied on a global variable and assumed that there can be only
one external table open at a time, which seems like a shaky assumption.

With this patch, utl_file() is no longer guaranteed to be called on abort.
Instead, make the UTL_FILE implementations responsible for cleaning up any
low-level resources, like file descriptors or handles to external libraries
(like libcurl), themselves. Use the ResourceOwner mechanism for that.

For curl- and EXECUTE-style external tables, register a new ResourceOwner
hook to close with the libcurl handles, and launched subprocesses, on
abort. For file external tables, modify fstream.c to open the underlying
file with OpenTransientFile(), so that it gets closed automatically on
abort.

This changes the url_fopen() API, so that it never returns a NULL. Instead
of returning NULL, and reporting an error in the *response_code/string
arguments, make url_fopen() always report errors directly with ereport().
That makes the callers simpler, and allows the implementations to flexibly
report whatever details of the error they have available.

-pthread is coming as part of PTHREAD_CFLAGS, no need to add it separately.

cdb_dumpall_agent.c is neither compiled, not does it contain anything
of interest. Kill it for now, if we want cdb_dumpall_agent it can be
resurrected or started on from scratch.

External tables cannot inherit attributes when acting as a partition
member in a partititioned table. Attribute inheritance was cleaned up
in upstream commit 02e64181 which was
part of the 8.3 merge, and as an effect of this external tables were
dumped as inheriting the partition parent attributes. Fix by always
setting the attributes as local for external tables and make it even
more explicit by altering the attribute check in dumpExternal to only
consider dropped attributes. Apply the fix to both pg_dump and
cdb_dump_agent albeit a tad differently as pg_dump has abstracted the
logic for printing column names.

Report and debugging by Chris Hajas, Lawrence Hamel and Karen Huddleston

There was previously no support added for CREATE OPERATOR in the
binary upgrade code, add Oid pre-assignment support for operators.

The binary-upgrade support was added with the pg_upgrade patch and
allows for the Oids in the old cluster to be pre-assigned to their
respective objects during pg_upgrade. The main motivation for this
refactoring is that we need to pre-assign all Oids before object
creation during restore into the new cluster since a restore can
otherwise allocate an Oid which is later preassigned. This brings
all pre-assignments into the head of the dumpfile such that they
happen before most object creations (pg_dumpall objects are still
written to the dumpfile before pg_dump has a chance).

The main contributions of this patch include:

* All binary-upgrade methods are moved to binary_upgrade.{c|h} for
  pg_dump and binary_upgradeall.{c[h} for pg_dumpall. This greatly
  reduce the diff wrt upstream PostgreSQL
* Oid preassign calls are now loaded into the backup archive as
  TOC entries which allows for them to be sorted to be output
  before object creation
* Avoid usage of PQExpBuffers for trivial string construction where
  a fixed buffer on the stack is sufficient, or simply outputting
  the string directly
* Binary upgrade dumping in pg_dump is moved to a separate function
  which loops over the dumpable objects instead of being mixed in
  with the main code
* Various simplifications of the code and cleanups where possible

Cherry-picked into GPDB ahead of merge to fix compiler warning from
backporting of bytea HEX encode/decode.

Coverity complained that the aoseg_namespace variable potentially
was uninitialized at usage, and while technically correct it would
require the pg_aoseg namespace to be missing in the source cluster
which seems far fatched. Fix by forcing initialization to invalid
and ensuring that the Oid has been changed to a valid Oid before
using.

While looking at the code I also realized it had been blindly
cargoculted to the same design pattern as the remaining queries
which here was pointless. Simplify the logic and avoid allocating
an extra PQExpBuffer.

While the type cache used in binary upgrades isn't a dumpable
object, it has a DO_ identifier associated to be able to use
the indexing code. Skip it when looking through the DO objects
to keep compilers and static analyzers happy.

This was already done in pg_dump, do so for cdb_dump_agent too
and expand the documentation on why.

This doesn't save too much time, it is just the pedant in me cringing
when I looked at why we had a mandatory proprietary dependency.

Alas this probably should have been done when we first introduced the
`--enable-netbackup` configure flag, but better late than never.

The four BSA agents seem to (inherently, as opposed to incidentally)
share the same dependencies. If that's true, we might as well declare
them together in one place.

Those binaries need the gpbsa shared object to build. This wasn't
exposed until EITHER

0. We changed the order we build them in the previous commit; OR
0. We ran a parallel build

This commit simply adds the proper dependency declaration to all the
offending targets.

This should not break anything given the declarative nature of Make. But
it does (especially noticeable if you are not running a parallel
`make`). Because we are not declaring a dependency on gpbsa libraries on
the the dump agents executables.

* Implement bytea HEX encode/decode
this is from upstream commit a2a8c7a6
The primary reason for back patching is to allow pgadmin4 to work
Unfortunately there are a few hacks to make this work without Enum Config GUCs
mostly around setting the GUC as the target is an integer but the guc value is a string

* fixed regression tests \
removed unused code

Constraints are not uniquely identified by namespace and name alone.
It's possible to have multiple constraints with the same name, but attached
to different tables or domains. So include parent relation and domain OIDs
in the key.

This was uncovered by a regression test in the TINC test suite. Move the
OID consistency tests from TINC into the normal regression test suite, to
have test coverage for this.

1. Add custom external table dependency with protocol to keep the dumped
   SQL ordered correctly. (create protocol must execute before creating custom
   external table)

2. Remove \" from dumped location URI

3. Support dump multiple locations
Signed-off-by: NPeifeng Qiu <pqiu@pivotal.io>
Signed-off-by: NYuan Zhao <yuzhao@pivotal.io>

Creating an external table with duplicate location URIs is a syntax
error so move the duplicate check to the parsing step. This simplifies
the external table command code as well as provides an error message
with context marker to the user.

Also fix up test sources to match the new error messages and remove
the duplicate test from gpfdist since we already test that in ICW.

In case user wanna specify parameters with OPTIONS other than config
file in LOCATION, it provides flexible and friendly grammar.

Usage:
CREATE EXTERNAL TABLE xxx LOCATION xxx FORMAT xxx
[OPTIONS ( [option_key 'option_value' [, ...]] )]
xxx;
Signed-off-by: NKuien Liu <kliu@pivotal.io>
Signed-off-by: NPeifeng Qiu <pqiu@pivotal.io>
Signed-off-by: NAdam Lee <ali@pivotal.io>
Signed-off-by: NYuan Zhao <yuzhao@pivotal.io>
Signed-off-by: NJesse Zhang <jzhang@pivotal.io>

* Backs up search path, optimizer settings, and gp_default_storage_options
* These settings will only be restored during database restores (`-e` option)

Commit 08d85beb introduced a doublefree on the PQExpBuffer in pg_dump
when populating the typecache for binary-upgrade mode. Fix by moving
all buffer management into the scope where it's used.

Ensure that we close all PQExpBuffers before closing scope. Spotted
by Coverity

Fixing incorret filename identifiers in the header comment blocks
around the code, mostly stemming from copy-paste it would seem.

-- previously, within a 60-second loop to test if a pqPing was
successful, there was a 2-second fail-fast test to make sure that
the postmaster.pid file was written to disk.

-- However, in extreme loading conditions, the underlying OS may be too
busy to accomplish that within 2 seconds. Removing the fail-fast means
that in that extreme loading case, pg_ctl will still be able to start,
assuming that the OS can write a file within 60 seconds.

-- This does mean that in cases where the pidfile cannot be written at
all (for instance, if the disk is full) the failure will take the full
60 seconds to be reported.

This works by building an indexarray containing all the pg_type
entries on the first lookup, once built the indexarray can be
queried and further SQL queries avoided. The corresponding array
type is cached together with the base type.

Backport another pg_upgrade-related change from upstream. Distinguishing
between binary-upgrade and "normal" utility mode turns out to be rather
handy.

CAUTION: This removes the GPDB-specific postmaster command-line options
-b and -C, becuase they clashed with the newly introduced -b flag, used
to choose binary-upgrade mode! In order to invoke the GPDB functionality
previously reached via -b -C, the long option forms need to be used. This
commit switches pg_ctl over to using long options.

This commit adds a substantial amount of Greenplum specific code to handle
binary-upgrade in the database backend. Below is a list of the changes:

- When restoring a schema dump on a master node, in binary-upgrade mode,
  we want to populate gp_distribution_policy, while in "normal" utility mode,
  we don't. In the passing, simplify the query used to fetch the entries from
  gp_distribution_policy in pg_dump.
- pg_dump will in binary upgrade mode preassign Oids for the Oid dispatcher
  to ensure that object Oids are stable across dump/restore. This requires
  the Oid dispatcher to handle binary upgrade mode. Dump files created with
  binary-upgrade mode will assign multiple Oids into the preassigned Oids list,
  sometimes for objects not created within the current transaction. Allow the
  preassigned_oids list to preserve the Oids across end-of-xact when in binary
  upgrade mode.
- The binary upgrade functions from pg_upgrade_support are used to preassign
  Oids during pg_dump in binary mode and are installed into a separate
  binary_upgrade schema. Due to a chicken-and-egg problem, installing these
  functions by themselves will however fail Oid assignment as they are required
  for Oid assignment. This reserves a block of OIDs for use with the
  binary_upgrade schema to ensure that the functions can be installed.

The original commit from upstream which has been heavily amended to:

commit 76dd09bb
Author: Bruce Momjian <bruce@momjian.us>
Date:   Mon Apr 25 12:00:21 2011 -0400

    Add postmaster/postgres undocumented -b option for binary upgrades.
    This option turns off autovacuum, prevents non-super-user connections,
    and enables oid setting hooks in the backend.  The code continues to use
    the old autoavacuum disable settings for servers with earlier catalog
    versions.

    This includes a catalog version bump to identify servers that support
    the -b option.

Heikki Linnakangas, Daniel Gustafsson and Dave Cramer

The binary-upgrade mode in pg_dump which was backported in a previous commit
is here extended to handle Greenplum clusters. The gist of the changes are
to extend the Oid dumping to cover all object types as upstream PostgreSQL
only covers types. The changes include:

Make "pg_dump --binary-upgrade" connect in utility mode, it's supposed to
be run against a single segment server. Also make pg_dump work against 4.3
clusters as the binary-upgrade mode will be used with pg_upgrade on the old
cluster in the upgrade.

Dump Oids for all object types in binary upgrade mode. In PostgreSQL only
type Oids are dumped in binary upgrade mode, for Greenplum however we need
to dump the Oids for all objects that require Oid dispatching (see
oid_dispatch.c for details). Inject calls in the dump file for the Oid
dispatcher to preassign Oids during restore.

For partitioned tables, the Oids for each child table as well as their
constraints are added to the dump file. The distributioncheck for
partitions is skipped when restoring during binary upgrade. The
distribution info is not available in the segments so during binary
upgrade we cant enforce this check.

Avoid creating array type for AO relation types, it's not useful for anything
and only grows the catalog for no use.

Heikki Linnakangas, Daniel Gustafsson and Dave Cramer

It is needed by pg_upgrade. Note that pg_upgrade runs the *new* version's
pg_dump binary, so there is no need to backport this to GPDB 4.3.

Original commit:

Author: Bruce Momjian <bruce@momjian.us>
Date:   Tue Feb 17 15:41:50 2009 +0000

    Add pg_dump --binary-upgrade flag to be used by binary upgrade
    utilities.

    The new code allows transfer of dropped column information to the
    upgraded server.

The different kinds of NOTICE messages regarding table distribution
were using a mix of upper and lower case for 'DISTRIBUTED BY'. Make
them consistent by using upper case for all messages and update the
test files, and atmsort regexes, to match.

Ensure to free any allocated content on premature exit. These potential
leaks are small but we might as well clean up to make static analysis
happy.

Per defect in Coverity

transformPassThroughParms() is hardcoded to always return true since
the code that could render false has been killed. No single callsite
further inspect the returnvalue (for good reasons). Remove return
value altogether and make function return void.

At this point we already know that pQry is non-NULL as we have
dereferenced it previous in the function. Remove useless check
to make the code clearer.

Per gripe by Coverity.

Remove commented out code which clearly hasn't been in use for some
time. Anyone interested in this code have the VCS archives at their
disposal.

This patch started out as a small refactor of executable discovery due to
the function having multiple leaks, but turned into a revamping of the way
the commandlines are constructed (since that's where the executable paths
were used). The changes introduced in this diff are:

    - Build the commandlines in PQExpBuffers instead of fixed size buffer
      believed to be large enough
    - Move commandline functions from cdb_dump_util into cdb_restore_agent
      as that's the only callsite, no need to link that code into other
      binaries. Also refactor the functions significantly as there were
      many functions doing essentially the same thing.
    - Remove code to find executables and just use find_other_exec() since
      all binaries are in the same folder
    - Change shellEscape() to not always resetting the PQExpBuffer, also
      use the same code as in src/backend/cdb/cdbbackup.c which handles
      quoting as well
    - Remove various small functions that only returned a static string
      and variables holding a static string
    - Use MAXPGPATH instead of locally defined macro for the same thing
    - Remove the highly uninteresting unittests for string concatenation

The code is commented out without any commented out callsites so
remove. This is what the vcs history is for