When user dumps database to Data Domain Boost server, storage
unit and backup directory must be already created and specified,
previously, we hard coded the storage unit to "GPDB" and user
had no option to use others.

This commit adds --ddboost-storage-unit option, which allows
user to dynamically specify storage unit for dump and restore.

This commits allows user to have storage unit information
statically saved into configure file in their cluster host.

This commit added storage unit option into gpmfr for replicating
and recovering dump copies, in which case it uses identical storage
unit and backup directory between primary and secondary DDBoost server.

--ddboost-storage-unit option takes higher priority than using
statically configured storage unit.

Authors:
Pengcheng Tang, Marbin Tan, Nikhil Kak
Lawrence Hamel, Stephen Wu, Chris Hajas, Chumki Roy

Now s3ext could recognize then decompress gzip encoded files
automatically, doesn't require any extra parameter, configuration or
extended filename.

I don't see anything in this test that would require a huge number of rows.
Half a million should be more than enough to show up in reltuples/relpages.

Almost all tests in bfv_legacy were also in qp_misc_rio. I kept the ones
in qp_misc_rio, except for a few that have different output with ORCA.
This way, only bfv_legacy needs to have an alternative expected output file
for ORCA.

Test 31 was identical to test 25.

This exact same test case, with some extra EXPLAINs, is in the
co_nestloop_idxscan regression test.

Instead of pushing the responsibility of rescanning down to each different
kind of external table implement rescanning in fileam.c in a generic
fashion, by closing and reopening the underlying "url". This gets us rescan
support for custom and EXECUTE-type external tables, which was missing
before, and also makes the code simpler.

There are no known cases where the rescan support is currently needed (hence
no test case included), because the planner puts Materialize nodes on top of
external scans, but in principle every plan node is supposed to be
rescannable. I tested this by reverting the previous patch that fixed
using external scans in a subplan; without that patch, an external table
in a subplan would get rescanned.

ParallelizeCorrelatedSubPlanMutator() turns each Scan on a base relation
into a "Result - Material - Broadcast - Scan" pattern, but it missed
ExternalScans. External tables are supposed to be treated as distributed,
i.e. each segment holds different part of the external table, so they
need to be treated like regular tables.

Long time ago, a hack was put in place in GPDB to use the "scan" slot,
instead of the "outer" slot which is used in the upstream, to hold the
result of an Agg or Window plan node's child. It's not clear to me why
that was done. There was even a comment in fix_upper_expr() saying we
wouldn't need it if we just fixed the executor to not contain that hack,
and there was also a TODO comment in CMappingColIdVarPlStmt.cpp about
that.

Everything seems to work without those hacks, so revert this thing back
to the way it works in the upstream. This is simpler in its own right,
and also reduces our diff vs. upstream, which will make merging easier
in the future.

This was raised in #691 and was identified as a bug in upstream as well. The
patch has now been committed to upstream, this is a backport with Greenplum
versions of Flex/Perl maintained. See below for upstream commit message.

    commit 7d7b1292
    Author: Tom Lane <tgl@sss.pgh.pa.us>
    Date:   Mon May 2 11:18:10 2016 -0400

    Fix configure's incorrect version tests for flex and perl.

    awk's equality-comparison operator is "==" not "=".  We got this right
    in many places, but not in configure's checks for supported version
    numbers of flex and perl.  It hadn't been noticed because unsupported
    versions are so old as to be basically extinct in the wild, and because
    the only consequence is whether or not a WARNING flies by during
    configure.

    Daniel Gustafsson noted the problem with respect to the test for flex,
    I found the other by reviewing other awk calls.

This closes #697

This closes #682

Without GUC_GPDB_ADOPT, the values for guc is not going to be dispatched
to the QE processes.

Specically persistent check:
  gp_persistent_relation_node   <=> filesystem

Authors: Marbin Tan & Larry Hamel

Batch size of 8 is too low and each cluster may have a different
system configuration, so we would like to dertermine a default batch
size before running gpcheckcat.

* Add unittest for batch size

* Truncate batch size to be, at maximum, the amount of primaries.
  Batch size can be no longer be larger than the amount of primaries.

* Refactor: Create method main() for gpcheckcat
  In order improve unit testing, move functionality from '__main__'
  to a method.

Authors: Marbin Tan, Larry Hamel, Nikhil Kak

* Remove warning when compiling aosegfiles.c

Use strlcpy instead of strncat to stop compiler complaining. The
original case "strncat(segnumArray, tmp, sizeof(tmp))" is simple
and safe, but the compiler complains much by law [-Wstrncat-size].

Changes to be committed:
	modified:   src/backend/access/aocs/aocssegfiles.c
	modified:   src/backend/access/appendonly/aosegfiles.c

* speed up a bit by replacing strncat with strlcpy

Files changed:
    modified:   src/backend/access/external/url.c
    modified:   src/backend/cdb/cdbhash.c
    modified:   src/backend/cdb/cdbmutate.c
    modified:   src/pl/plpgsql/src/pl_funcs.c

Skipped:
    catcoretable.c:104:26:CatCoreType_int4_array
	because it is an enumerated constant to ensure integrity

In a previous commit, f569c1d1,
gpcheckcat was modified to display a list of tables with missing
attributes.  This commit adds the ability to list tables with
extraneous attributes.

Authors: Chumki Roy and James McAtamney

Fixed a typo in the --with-codegen-prefix description and corrected
spelling in the CMake check error message. No functional changes.

atpxPartAddList() needs a CreateStmt that represents the parent table,
but instead of creating it already in the parser, and adding more details
to it in analyze.c, it's simpler to create it later, in atpxPartAddList(),
where it's actually needed.

The code in InitPostgres() was refactored in PostgreSQL 9.0 so that it
no longer uses FindMyDatabaseByOid() function. We had backpatched the
InitPostgres() changes already, so backpatch the removal of
FindMyDatabaseByOid() as well. Silences a compiler warning.

In _SPI_execute_plan, debug_query_string was set directly to plan->query which is not
memory context safe, it means debug_query_string have a chance to refer to a invalid
address when FATAL/PANIC level error occurs.

A common source of bugs has been that an object gets assigned a different
OID in the master and in segments. A segment should normally never have to
allocate an OID (for catalog objects) on its own, all OIDs should be
allocated in the master, and sent over to the segments. To make such bugs
easier to catch, add a WARNING if an OID is allocated in a segment.

There were some DEBUG1 elogs for the same thing in place already, but the
list of catalogs that need synchronized OIDS wasn't up-do-date, and this new
place for the elog() is less invasive anyway.

When I merged the operator family patch, I missed dispatching the new DDL
commands to segments. Because of that, the segments didn't have information
about operator families. Some operator families would be greated implicitly
by CREATE OPERATOR CLASS, but you wouldn't necessarily get the same
configuration of families and classes as in the master. Things worked pretty
well despite that, because operator families and classes are used for
planning, and planning happens in the master. Nevertheless, we really should
have the operator family information in segments too, in case you run
queries in maintenance mode directly on the segments, or if you execute
functions in segments that need to execute expression that depend on them.
Also, there were no regression tests for the new DDL commands.

If you do CREATE OPERATOR, with a commutator or negator operator that
doesn't exist yet, the system creates a "shell" entry for the non-existent
operator. But those shell operators didn't get the same OID in all segments,
which could lead to strange errors later. I couldn't find a test case
demonstrating actual bugs from that, but it sure seems sketchy. Given that
we take care to synchronize the OID of the primary created operator, surely
we should do the same for all operators.

The out/readfuncs.c support for AlterTableStmt.comptypeArrayOid was
missing. Because of that, the segments didn't get the OID of the composite
type's array type from master, and allocated it on their own.

We were using "complex" as a user defined type, which is no longer the
case once #605 was checked in.

tuple.
This closes #649

Squashed commit of the following:

commit 96f415bc1744533e96a74bcc85016ff5df65d06e
Author: Karthikeyan Jambu Rajaraman <karthi.jrk@gmail.com>
Date:   Thu Apr 21 15:08:49 2016 -0700

    Don't call prepare for execution if no generator is registered

commit 5318e9214a3bbc237c0d79017cdef02538c4e711
Author: Karthikeyan Jambu Rajaraman <karthi.jrk@gmail.com>
Date:   Thu Apr 21 11:35:22 2016 -0700

    Adding if codegen protection in starting active manager and while
    deleting manager

commit 12b0045d51a5b056b32a6cc9ef370e6d9af315fd
Author: Karthikeyan Jambu Rajaraman <karthi.jrk@gmail.com>
Date:   Fri Apr 15 14:33:55 2016 -0700

    Updating Readme for Codegen GPDB Function

commit d17119c5d8ce9a706b3321d5e7ad41f419fddbdc
Author: Karthikeyan Jambu Rajaraman <karthi.jrk@gmail.com>
Date:   Wed Apr 13 13:25:14 2016 -0700

    Add documentation for how to codegen in README.md

commit d711be4bb044592e7f97bf31ceb5f32337ad6d03
Author: Karthikeyan Jambu Rajaraman <karthi.jrk@gmail.com>
Date:   Wed Apr 13 11:19:03 2016 -0700

    Fixing doxygen namespace addtogroup

commit e34131bcc9bb3d36c5e8995e1a0efe11fbd55417
Author: Karthikeyan Jambu Rajaraman <karthi.jrk@gmail.com>
Date:   Wed Apr 13 11:17:32 2016 -0700

    Changing CodeGen to Codegen

commit bedfcca80b5f1ff796783ca7c2035d920ae86c4e
Author: Karthikeyan Jambu Rajaraman <karthi.jrk@gmail.com>
Date:   Wed Apr 13 10:51:39 2016 -0700

    Fix cpplint error

commit 32f89feedba655e1242529c78a54faf01ddf32bf
Author: Karthikeyan Jambu Rajaraman <karthi.jrk@gmail.com>
Date:   Wed Apr 13 10:48:49 2016 -0700

    Change regular version of generated to regular version of target in comment

commit d1da2824c08e34eb7139b08cd9d2aff19bf008e7
Author: Karthikeyan Jambu Rajaraman <karthi.jrk@gmail.com>
Date:   Wed Apr 13 10:42:09 2016 -0700

    Made GetFunctionPointer and CreateFunction to take typedef of function type as only public api

commit 9f48f77bd9fc50c7d29a4e8f7700452ef88dc79f
Author: Karthikeyan Jambu Rajaraman <karthi.jrk@gmail.com>
Date:   Tue Apr 12 17:18:24 2016 -0700

    Rename doCodegeneration to GenerateCodeInternal

commit 9884d07f4c6ab08d9ccca8d07b80974081cda1fa
Author: Karthikeyan Jambu Rajaraman <karthi.jrk@gmail.com>
Date:   Tue Apr 12 17:15:55 2016 -0700

    fix cpplint error

commit c21b3bee3e289675fc13017688269b40e26efa11
Author: Karthikeyan Jambu Rajaraman <karthi.jrk@gmail.com>
Date:   Tue Apr 12 17:08:16 2016 -0700

    Update comments

commit f016bb582af65e84def1b55917017008e292d45b
Author: Karthikeyan Jambu Rajaraman <karthi.jrk@gmail.com>
Date:   Tue Apr 12 17:01:19 2016 -0700

    Adding assert in static setToRegular function

commit 93a791a4ec415313933dd5b7c1ae17dbdd6193a6
Author: Karthikeyan Jambu Rajaraman <karthi.jrk@gmail.com>
Date:   Tue Apr 12 16:49:15 2016 -0700

    Removing decltype in Slotdeformtype codegen since we know the functype

commit ed95193d0991cb24fd68f6d9a473de630f297b0c
Author: Karthikeyan Jambu Rajaraman <karthi.jrk@gmail.com>
Date:   Tue Apr 12 16:45:49 2016 -0700

    Fixed compiler / unit test failure
    Changed InitCodeGen to InitCodegen

commit 3c95c9efdd4336e100a86748b0e87d79f07bcc11
Author: Karthikeyan Jambu Rajaraman <karthi.jrk@gmail.com>
Date:   Tue Apr 12 16:28:24 2016 -0700

    Changing CodeGen back to Codegen while cherry-picking

commit 4e24ef22bcf54edc00c2a0d1f9f9703364cdf5ba
Author: Karthikeyan Jambu Rajaraman <krajaraman@pivotal.io>
Date:   Tue Apr 12 11:08:42 2016 -0700

    Changing auto to actual type

commit 1d92fc190dda3ec866edc83d56d9eca9fe90a63a
Author: Karthikeyan Jambu Rajaraman <krajaraman@pivotal.io>
Date:   Tue Apr 12 10:44:49 2016 -0700

    fix indent in codegen_utils

commit ea6fa787a2332f94b8d20d24825ba1f0a240d02e
Author: Karthikeyan Jambu Rajaraman <krajaraman@pivotal.io>
Date:   Mon Apr 11 17:25:53 2016 -0700

    Adding codegen guc to wrapper function so related function not executed when codegen guc is disabled

commit f1cd7d80d8d81f3565f3ab12cb94a3c4cc58c44a
Author: Karthikeyan Jambu Rajaraman <krajaraman@pivotal.io>
Date:   Mon Apr 11 17:13:24 2016 -0700

    Changing bool to unsigned int in codegen_wrapper.h

commit 7ef462540a312d077b332c732c125c55dedb6b14
Author: Karthikeyan Jambu Rajaraman <krajaraman@pivotal.io>
Date:   Mon Apr 11 16:59:08 2016 -0700

    Fix based on lint complain

commit ddc55cec0d579a03cb9e2303efa6d74f5183979f
Author: Karthikeyan Jambu Rajaraman <krajaraman@pivotal.io>
Date:   Mon Apr 11 16:19:48 2016 -0700

    Removing commented code from unittest

commit 28b8886ed2bfdc854aa38f871a41536c1e6cef29
Author: Karthikeyan Jambu Rajaraman <krajaraman@pivotal.io>
Date:   Mon Apr 11 16:17:45 2016 -0700

    Chaging auto in for to actual type

commit ad8d78e07e9b789f4d8ae3c6ccc3cea1a05089c0
Author: Karthikeyan Jambu Rajaraman <krajaraman@pivotal.io>
Date:   Mon Apr 11 16:13:21 2016 -0700

    * Handle return value of SetToGenerated from PrepareForExecution
    * Change size_t to unsigned integer

commit d46ee8be414b3ac2a9ea7ef899f29cef3df145c7
Author: Karthikeyan Jambu Rajaraman <krajaraman@pivotal.io>
Date:   Mon Apr 11 15:30:40 2016 -0700

    Introducing template function to do enroll for any type of generator

commit 83428605f906e28d0fc994dcc311e0cc3e6d9e12
Author: Karthikeyan Jambu Rajaraman <krajaraman@pivotal.io>
Date:   Mon Apr 11 14:41:11 2016 -0700

    Changing manager to take module name as argument

commit b88aa40f89860e7c15503365ea304d2392f71cda
Author: Karthikeyan Jambu Rajaraman <krajaraman@pivotal.io>
Date:   Mon Apr 11 14:27:31 2016 -0700

    adding more comments to SlotDeformTupleCodeGenInfo and unique counter

commit df1b704548fcbc1b43524064eb2f024995eea865
Author: Karthikeyan Jambu Rajaraman <krajaraman@pivotal.io>
Date:   Mon Apr 11 14:08:34 2016 -0700

    Change test target to have ext .t and adding gitignore for that extension

commit 9ee3521ca687c4ea7f13d15893f47146ecdb5277
Author: Karthikeyan Jambu Rajaraman <krajaraman@pivotal.io>
Date:   Mon Apr 11 13:59:57 2016 -0700

    Add target_include_directories for test library header file

commit effb7dc165b4c7094731dc39e0872741b28b2e6a
Author: Karthikeyan Jambu Rajaraman <krajaraman@pivotal.io>
Date:   Mon Apr 11 13:36:50 2016 -0700

    Move all possible ifdef from gpdb to codegen_wrapper based on pull request comments

commit 989b96a91c1e843577b5f24623d1e4a6b810394c
Author: Karthikeyan Jambu Rajaraman <krajaraman@pivotal.io>
Date:   Fri Apr 8 10:43:17 2016 -0700

    fixing comments for GetOrigFuncName in codegen_interface

commit 665de065c34ea13abcec8c5f3a3336558dae501f
Author: Karthikeyan Jambu Rajaraman <krajaraman@pivotal.io>
Date:   Fri Apr 8 10:27:31 2016 -0700

    Disabling codegen guc by default

commit ec34513e86dea95bd36ee528724273c1c19e7062
Author: Karthikeyan Jambu Rajaraman <krajaraman@pivotal.io>
Date:   Thu Apr 7 16:28:43 2016 -0700

    Fixed comments space/tab so it aligned in different editor

commit 35a60fe2fd053a5babd1820302afad95c5c2635f
Author: Karthikeyan Jambu Rajaraman <krajaraman@pivotal.io>
Date:   Thu Apr 7 16:12:08 2016 -0700

    Fixing coding style based on cpplint

commit 9477dcf6c0a7ffcca77b9f2c62c7340d13f474aa
Author: Karthikeyan Jambu Rajaraman <krajaraman@pivotal.io>
Date:   Thu Apr 7 13:37:01 2016 -0700

    Adding USE_CODEGEN ifdef

commit 142ffc1416341a3ac1d0fe37a83577168f2ec166
Author: Shreedhar Hardikar <shardikar@pivotal.io>
Date:   Tue Apr 5 22:21:19 2016 +0000

    Make codegen work with GPDB unit test framework

commit 9dca9507dd933a5517bee3ca34615c09c23c1728
Author: Shreedhar Hardikar <shardikar@pivotal.io>
Date:   Mon Apr 4 22:10:21 2016 +0000

    Add gpcodegen_mock.c

commit 41d515ff0342d3df3b0853ad91731f6d1d75843c
Author: Shreedhar Hardikar <shardikar@pivotal.io>
Date:   Mon Apr 4 21:46:34 2016 +0000

    Remove unneccessary header file includes in codegen

commit 14c5e14b9834197693b1ef56efa96386cfe049c5
Author: Foyzur Rahman <frahman@gmail.com>
Date:   Tue Apr 12 14:49:51 2016 -0700

    Integrate codegen framework and make simple external call to slot deform tuple
Signed-off-by: NKarthikeyan Jambu Rajaraman <karthi.jrk@gmail.com>
Signed-off-by: NFoyzur Rahman <frahman@gmail.com>

Removed one test that was checking for a memory leak by attempting to run the system out of memory.
Refactored one test that was checking for a large insert into a table with a bitmap index.

Closes #660.

[#116336737, #117671863]

Commit 9d19664c ported a test from cdbfast to master, which uses gp_fault_inject mechanism to throw a SIGSEGV.
We remove this test because i) gp_fault_inject works only if USE_TEST_UTILS = 1, while when we test GPDB, we set USE_TEST_UTILS to 0, and ii) throwing segmentation faults in ICG is a wrong approach.

Fixes compiler warning for implicit declaration due to missing
headerfile include:

  copy.c:4143:6: warning: implicit declaration of function
     'reconstructTupleValues' is invalid in C99
     [-Wimplicit-function-declaration]

Backport from upstream and related commits are

    commit e5fe2e84
    Author: Tom Lane <tgl@sss.pgh.pa.us>
    Date:   Fri Nov 9 23:58:32 2007 +0000

        Recognize RETURN QUERY via a textual test, so that QUERY doesn't need to be
        a plpgsql keyword.  This avoids springing a new reserved word on plpgsql
        programmers.
        For consistency, handle RETURN NEXT the same way.

    commit b2b9b4d5
    Author: Neil Conway <neilc@samurai.com>
    Date:   Wed Jul 25 04:19:09 2007 +0000

        Implement RETURN QUERY for PL/PgSQL. This provides some convenient syntax
        sugar for PL/PgSQL set-returning functions that want to return the result
        of evaluating a query; it should also be more efficient than repeated
        RETURN NEXT statements. Based on an earlier patch from Pavel Stehule.

Committers: Kuien Liu and Haozhou Wang

The tests were added in commit 7d0325e4
and were not properly translated from Pivotal's internal tests to the
pg_regress tests. They now properly test CREATE RULE on AO/CO tables.

Thanks to Heikki Linnakangas for catching this issue.