There was previously no support added for CREATE OPERATOR in the
binary upgrade code, add Oid pre-assignment support for operators.

The binary-upgrade support was added with the pg_upgrade patch and
allows for the Oids in the old cluster to be pre-assigned to their
respective objects during pg_upgrade. The main motivation for this
refactoring is that we need to pre-assign all Oids before object
creation during restore into the new cluster since a restore can
otherwise allocate an Oid which is later preassigned. This brings
all pre-assignments into the head of the dumpfile such that they
happen before most object creations (pg_dumpall objects are still
written to the dumpfile before pg_dump has a chance).

The main contributions of this patch include:

* All binary-upgrade methods are moved to binary_upgrade.{c|h} for
  pg_dump and binary_upgradeall.{c[h} for pg_dumpall. This greatly
  reduce the diff wrt upstream PostgreSQL
* Oid preassign calls are now loaded into the backup archive as
  TOC entries which allows for them to be sorted to be output
  before object creation
* Avoid usage of PQExpBuffers for trivial string construction where
  a fixed buffer on the stack is sufficient, or simply outputting
  the string directly
* Binary upgrade dumping in pg_dump is moved to a separate function
  which loops over the dumpable objects instead of being mixed in
  with the main code
* Various simplifications of the code and cleanups where possible

Github auto discovers the project license via the licensee ruby gem
which in turn bases it's matching on the choosealicense.com database
of license texts. The matching is failing for our license so copy
the exact text from choosealicense in an attempt to make the matching
work. The diff pulls in the Appendix and changes whitespace on the
license name line, no changes whatsoever are made on the main license
text.

Signed-off-by: NHaozhou Wang <hawang@pivotal.io>

Only for custom protocols entry exists in pg_extprotocol. So, only for external
tables with custom protocols add dependency in pg_depend.

Fixes #1547.

Signed-off-by: NJingyi Mei <jmei@pivotal.io>
Signed-off-by: NTom Meyer <tmeyer@pivotal.io>

* pipelines were missing the new input from pulse trigger
  gpdb_src_behave_tarball. This is an update to fix that issue.

* Restructured/refactored the pipeline for better visibility for each
  job

- `optimizer_analyze_root_partition` GUC functionality is introduced.
When optimizer_analyze_root_partition is enabled, it will enable stats
collection on root partitions when a plain analyze is run on a root
partition table
- `optimizer_analyze_midlevel_partition` GUC functionality is introduced
When optimizer_analyze_midlevel_partition is enabled, it will enable
stats collection on midlevel partitions.

Expectation:
Case 1:
```sql
set optimizer_analyze_root_partition=off;
analyze tablename;
-- Stats should only be collected for the leaf tables
analyze rootpartition tablename;
-- Stats should be only collected for the root table
```

Case 2:
```sql
set optimizer_analyze_root_partition=on;
analyze tablename;
-- Stats should be collected for the root and the leaf table;
analyze rootpartition tablename;
-- Stats should be only collected for the root table
```
Signed-off-by: NOmer Arap <oarap@pivotal.io>

Uninitialized variable causes release and debug build to behave
differently for analyze. This commit fixes the issue.

* gptransfer used a sql command with "STRING_AGG" that combined all the attribute
* names and generated the distribution key. However, this resulted in
* an issue when there were more than 1 distribution key. In that case,
* the combined distribution key was escaped incorrectly.

* Instead, we now escape each attribute name separately via python.
Signed-off-by: NLarry Hamel <lhamel@pivotal.io>

With PostgreSQL 8.3, there's a new concept called "operator families".
An operator class is now part of an operator family, which can contain
cross-datatype operators that are "compatible" with each other.

ORCA doesn't know anything about that. This commit updates the
Translator files to refer to OpFamily instead of 'OpClasses'.

ORCA still doesn't take advantage of this, but at least we are using
operator families in operator classes' stead to make indexes work.
Signed-off-by: NHaisheng Yuan <hyuan@pivotal.io>

This is addressing the GPDB_83_MERGE_FIXME comment in xact.c:1081.

GPDB doesn't need `haveNonTemp` check, since GPDB doesn't allow data loss.

GPDB doesn't support the asynchronous commits from upstream because this might
cause data inconsistency across segments in a cluster.

We disable the support of async commit using macro IMPLEMENT_ASYNC_COMMIT.
And make the user GUC `synchronous_commit` as DEFUNCT_OPTIONS, so that its
setting will be ignored, and a WARNING is generated.

The original check for temp table in smgrGetPendingFileSysWork() is not valid
in GPDB, since GPDB temp table use shared buffers to support access across slices.

Once GPDB decide to support async commit, this macro can be removed.
Signed-off-by: NXin Zhang <xzhang@pivotal.io>

This allows writing only one .source file for a UAO test.  Create the
.source file in "input/uao*/" directory.  Place the answer file, also
named as .source, into corresponding "output/uao*/" directory.  The
.source files must contain the following header:

   create schema <filename_prefix>@orientation@;
   set search_path="$user",<filename_prefix>@orientation@,public;
   SET gp_default_storage_options='orientation=@orientation@';

Replace "<filename_prefix>" with the filename excluding the ".source"
extension.

Generated files are named as <filename_prefix>_row.sql and
<filename_prefix>_column.sql.  Add the generated filenames to schedule
files and run pg_regress as usual.

A new option "--ao-dir" is added to pg_regress.  To enable row/column test
generation, set it to the directory name containing generic UAO .source tests.
The directory should be created under src/test/regress/input.

* Enable to run behave tests with the installed binary instead of source
* code.

* This is an effort to enable us to run behave tests without relying on
* the source code and instead use the GPDB installed in the system as
* the behave tests are integration tests.

* If the version has non-digit characters, then the float casting fails
* for the minor version.

Author: Chumki Roy <croy@pivotal.io>
Date:   Mon Dec 5 11:06:45 2016 -0800

    Remove error line - Aborting PQO plan generation.
    * Fix ORCA error message and make it same as with Planner.

Author: Haisheng Yuan <hyuan@pivotal.io>
Date:   Wed Jul 20 14:30:38 2016 -0700

    Update kerberos tinc test answer files

Cherry-picked into GPDB ahead of merge to fix compiler warning from
backporting of bytea HEX encode/decode.

Coverity complained that the aoseg_namespace variable potentially
was uninitialized at usage, and while technically correct it would
require the pg_aoseg namespace to be missing in the source cluster
which seems far fatched. Fix by forcing initialization to invalid
and ensuring that the Oid has been changed to a valid Oid before
using.

While looking at the code I also realized it had been blindly
cargoculted to the same design pattern as the remaining queries
which here was pointless. Simplify the logic and avoid allocating
an extra PQExpBuffer.

While the type cache used in binary upgrades isn't a dumpable
object, it has a DO_ identifier associated to be able to use
the indexing code. Skip it when looking through the DO objects
to keep compilers and static analyzers happy.

This was already done in pg_dump, do so for cdb_dump_agent too
and expand the documentation on why.

When adding operators to an existing operator family, ensure to
dispatch the Oids of the created objects to the segments. This
also backports the upstream alt_opf17 test case from alter_generic.

Reported by Dhanashree Kashid

- Update answer file for
  tincrepo/mpp/gpdb/tests/catalog/basic/security_definer/pre_script.ans
- Index on pg_aoseg was removed. Update
  tincrepo/mpp/gpdb/tests/catalog/mpp25256/mpp25256.sql and its answer file to
  reflect that.
- UDF exception handling test cases:
  - Some of the answer files had to be updated for the (file: line no) diff
  - The error messages during gang creation has changed. A few answer files had
    to be changed to account for the diff
- AOCO alter table
  - Fix the diff in the \d+ output for aoco table with cidr and inet columns

Update the .ans file to conform to the test output for 5.0.

The patch aims to cleanup and fix tests in the
tincrepo/mpp/gpdb/tests/storage/basic folder.
- Fix the external table tests.
- Fix transaction management test and remove the max_prepared_transactions test
  since it is no longer valid.
- Fix the xidlimits compilation.
- Fix the partition ddl tests.

Update the .ans file to conform to the test output for 5.0.

If a CTAS query uses a query with an InitPlan, and the InitPlan contains
a call to a function that dispatches another query to the segments, the
list of Oids is dispatched to the segments prematurely, in the inner query.
As a result, when the CTAS query is dispatched, the OID of the new table
is not included, and you get a "no pre-assigned OID for relation" error.

To fix, attach the OID of the new table to the query descriptor before
executing the InitPlans.

Fixes github issue #1543, reported with test case by Abhijit Subramanya.

Currently, the walsender process is not properly stopped during
gpstop. The SignalSomeChildren function was changed in the Postgres
8.3 merge in commit 20ac0be1. It was
previously at a Postgres 9.2 state to support sending signals to
walsender as a part of our master and standby master replication.

We don't build this as part of the CI, and we still won't pass the PL/TCL
regression tests (doesn't seem like there's anything serious there at a
quick glance though), but let's at least make it compile.

Per github issue #1524

For simplicity. This is less error-prone, too, in the face of future changes
to ExprContext.

DynamicTableScanInfo is an extension of EState, so always allocate it in the
same memory context. The DynamicTableScanInfo.memoryContext field always
pointed to es_query_cxt, so that is what we in fact always did anyway, this
just removes the unnecessary abstraction, for simplicity.

Pass the EState that contains it to where it's needed, instead.