The seg contrib module has been missing the regress testfiles since
a merge in the past. This brings in the test files from 9.4 stable
such that we can again run installcheck on this module.
Reviewed-by: NJacob Champion <pchampion@pivotal.io>
Reviewed-by: NAshwin Agrawal <aagrawal@pivotal.io>

A later commit will remove V0 support.

Author: Andres Freund, with contributions by Craig Ringer
Reviewed-By: Peter Eisentraut, Craig Ringer
Discussion: https://postgr.es/m/20161208213441.k3mbno4twhg2qf7g@alap3.anarazel.de

This job is used to test online expand, it will create a cluster
with two segments, then expand to 3 and run all ICW to check
whether the cluster is OK after expansion.
As restart is forbidden in online expand test, so we exclude all
the case which contains restart opertaion. But if someone add a
new test with restart, the job may fail. Manual intervention to
exclude the test is needed.
So we move this job to our own dev pipeline to reduce the impact
on prod pipeline.

As we merge with upstream and by that keep refining the Postgres
planner, legacy planner is no longer a suitable name. This changes
all variations of the spelling (legacy planner, legacy optimizer,
legacy query optimizer etc) to say "Postgres" rather than "legacy".
Reviewed-by: NVenkatesh Raghavan <vraghavan@pivotal.io>
Reviewed-by: NDavid Yozie <dyozie@pivotal.io>
Reviewed-by: NGeorgios Kokolatos <gkokolatos@pivotal.io>

This commit is part of Add Partitioned Indexes #7047.

Until indexes can be upgraded drop all indexes. Drop from the root
partition instead of on intermediate/leaf partitions, since they now
cascade due to their INTERNAL_AUTO dependencies.
Co-authored-by: NKalen Krempely <kkrempely@pivotal.io>

The GUC which enables recursive CTEs is in the currently released
version called gp_recursive_cte_prototype, but in order to reflect
the current state of the code it's now renamed to gp_recursive_cte.
By default the GUC is still off, but that might change before we
ship the next release.

The previous GUC name is still supported, but marked as deprecated,
in order to make upgrades easier.
Reviewed-by: NIvan Novick <inovick@pivotal.io>
Reviewed-by: NGeorgios Kokolatos <gkokolatos@pivotal.io>

The citext test was altered in the past to add a distribution column
other than citext. Since we now allow for extension datatypes to be
distribution columns with the new hashing work, remove the workaround
and re-align the test with upstream.

This also updates the testdata from upstream to work in Greenplum.
Reviewed-by: NHeikki Linnakangas <hlinnakangas@pivotal.io>

All of these checks are done only when the old cluster version has
support for gphdfs.

1). skip checking for gphdfs tables for cluster versions where gphdfs
support is absent.
2). fail upon a successful check for gphdfs roles

NOTE: we do not special case the existence of the gphdfs.so library,
even in absense of gphdfs tables or roles.  In other words, the existing
library checks force the user to drop all gphdfs-dependent functions
before upgrade.
Co-authored-by: NFrancisco Guerrero <aguerrero@pivotal.io>

Minor changes are made to pg_upgrade to allow GPDB5 to be upgraded
to GPDB6:

1). pg_ctl arguments need to explicitly contain the dbid, contentid and
numcontents in order to start the GPDB5 cluster

2). the type of the attnum field of the gp_distribution_policy had
changed

3). gp_toolkit is a view, and hence datatypes it contains that have been
modified in GPDB6(such as name), need not be flagged as errors during
pg_upgrade's check of the old cluster.

4). index access method type 'bitmap' needs to be added to the exclusion
list to select only bpchar_pattern_ops index access methods
Co-authored-by: NJesse Zhang <jzhang@pivotal.io>

Commit 98627fe7 introduced a workaround
for Oid collisions stemming partition exchange of heap relations
that lack an array type. Commit 73434db2
introduced a preallocation check to the Oid dispatcher which now
is responsible for avoiding these collisions. Remove the workaround
from the pre-test script.
Reviewed-by: NJacob Champion <pchampion@pivotal.io>
Reviewed-by: NShaoqi Bai <sbai@pivotal.io>

We tweak the existing 6-6 upgrade test test_gpdb.sh to work
on a 5-6 upgrade.  This allows anyone to upgrade a 5 demo
cluster, with various user data, to a 6 demo cluster.
Co-authored-by: NJesse Zhang <jzhang@pivotal.io>

It was only needed for EState. builtins.h is included from many different
places, so it seems good to not pull in executor.h. Add explicit #includes
into many places that were relying on the indirectly included executor.h.

In Greenplum 5.X the recursive CTE feature was hidden behind a GUC as
it wasn't deemed of production quality just yet. Commit 20152cbf
removed that GUC in order to make stabilization work easier, there are
still enough rough edges to not consider recursive CTE a feature which
is on by default. This brings back the GUC using the same name in order
to be backwards compatible even though "prototype" is a bit misleading
as this point.

In order for the cluster, and the associated tools, to work this also
turns on/off the GUC as required when there are recursive queries in
the toolchain.

Also adds a test and tidies up a few comments in surrounding code.

This is another attempt at this, the previous coding being reverted
in ea57a7aa.
Reviewed-by: NVenkatesh Raghavan <vraghavan@pivotal.io>
Reviewed-by: NGeorgios Kokolatos <gkokolatos@pivotal.io>

gphdfs is being deprecated for GPDB 6. This commit removes the
following:
- Remove gphdfs from pipelines
- Remove gphdfs from gpcontrib
- Remove gphdfs from Makefile
- Remove tests using gphdfs
- Remove GUCs, code, and comments used for gphdfs
- Remove gphdfs from GPDB docs
- Remove avro and parquet formats

Replace the use of the built-in hashing support for built-in datatypes, in
cdbhash.c, with the normal PostgreSQL hash functions. Now is a good time
to do this, since we've already made the change to use jump consistent
hashing in GPDB 6, so we'll need to deal with the upgrade problems
associated with changing the hash functions, anyway.

It is no longer enough to track which columns/expressions are used to
distribute data. You also need to know the hash function used. For that,
a new field is added to gp_distribution_policy, to record the hash
operator class used for each distribution key column. In the planner,
a new opfamily field is added to DistributionKey, to track that throughout
the planning.

Normally, if you do "CREATE TABLE ... DISTRIBUTED BY (column)", the
default hash operator class for the datatype is used. But this patch
extends the syntax so that you can specify the operator class explicitly,
like "... DISTRIBUTED BY (column opclass)". This is similar to how an
operator class can be specified for each column in CREATE INDEX.

To support upgrade, the old hash functions have been converted to special
(non-default) operator classes, named cdbhash_*_ops. For example, if you
want to use the old hash function for an integer column, you could do
"DISTRIBUTED BY (intcol cdbhash_int4_ops)". The old hard-coded whitelist
of operators that have "compatible" cdbhash functions has been replaced
by putting the compatible hash opclasses in the same operator family. For
example, all legacy integer operator classes, cdbhash_int2_ops,
cdbhash_int4_ops and cdbhash_int8_ops, are all part of the
cdbhash_integer_ops operator family).

This removes the pg_database.hashmethod field. The hash method is now
tracked on a per-table and per-column basis, using the opclasses, so it's
not needed anymore.

To help with upgrade from GPDB 5, this introduces a new GUC called
'gp_use_legacy_hashops'. If it's set, CREATE TABLE uses the legacy hash
opclasses, instead of the default hash opclasses, if the opclass is not
specified explicitly. pg_upgrade will set the new GUC, to force the use of
legacy hashops, when restoring the schema dump. It will also set the GUC
on all upgraded databases, as a per-database option, so any new tables
created after upgrade will also use the legacy opclasses. It seems better
to be consistent after upgrade, so that collocation between old and new
tables work for example. The idea is that some time after the upgrade, the
admin can reorganize all tables to use the default opclasses instead. At
that point, he should also clear the GUC on the converted databases. (Or
rather, the automated tool that hasn't been written yet, should do that.)

ORCA doesn't know about hash operator classes, or the possibility that we
might need to use a different hash function for two columns with the same
datatype. Therefore, it cannot produce correct plans for queries that mix
different distribution hash opclasses for the same datatype, in the same
query. There are checks in the Query->DXL translation, to detect that
case, and fall back to planner. As long as you stick to the default
opclasses in all tables, we let ORCA to create the plan without any regard
to them, and use the default opclasses when translating the DXL plan to a
Plan tree. We also allow the case that all tables in the query use the
"legacy" opclasses, so that ORCA works after pg_upgrade. But a mix of the
two, or using any non-default opclasses, forces ORCA to fall back.

One curiosity with this is the "int2vector" and "aclitem" datatypes. They
have a hash opclass, but no b-tree operators. GPDB 4 used to allow them
as DISTRIBUTED BY columns, but we forbid that in GPDB 5, in commit
56e7c16b. Now they are allowed again, so you can specify an int2vector
or aclitem column in DISTRIBUTED BY, but it's still pretty useless,
because the planner still can't form EquivalenceClasses on it, and will
treat it as "strewn" distribution, and won't co-locate joins.

Abstime, reltime, tinterval datatypes don't have default hash opclasses.
They are being removed completely on PostgreSQL v12, and users shouldn't
be using them in the first place, so instead of adding hash opclasses for
them now, we accept that they can't be used as distribution key columns
anymore. Add a check to pg_upgrade, to refuse upgrade if they are used
as distribution keys in the old cluster. Do the same for 'money' datatype
as well, although that's not being removed in upstream.

The legacy hashing code for anyarray in GPDB 5 was actually broken. It
could produce a different hash value for two arrays that are considered
equal, according to the = operator, if there were differences in e.g.
whether the null bitmap was stored or not. Add a check to pg_upgrade, to
reject the upgrade if array types were used as distribution keys. The
upstream hash opclass for anyarray works, though, so it is OK to use
arrays as distribution keys in new tables. We just don't support binary
upgrading them from GPDB 5. (See github issue
https://github.com/greenplum-db/gpdb/issues/5467). The legacy hashing of
'anyrange' had the same problem, but that was new in GPDB 6, so we don't
need a pg_upgrade check for that.

This also tightens the checks ALTER TABLE ALTER COLUMN and CREATE UNIQUE
INDEX, so that you can no longer create a situation where a non-hashable
column becomes the distribution key. (Fixes github issue
https://github.com/greenplum-db/gpdb/issues/6317)

Discussion: https://groups.google.com/a/greenplum.org/forum/#!topic/gpdb-dev/4fZVeOpXllQCo-authored-by: NMel Kiyama <mkiyama@pivotal.io>
Co-authored-by: NAbhijit Subramanya <asubramanya@pivotal.io>
Co-authored-by: NPengzhou Tang <ptang@pivotal.io>
Co-authored-by: NChris Hajas <chajas@pivotal.io>
Reviewed-by: NBhuvnesh Chaudhary <bchaudhary@pivotal.io>
Reviewed-by: NNing Yu <nyu@pivotal.io>
Reviewed-by: NSimon Gao <sgao@pivotal.io>
Reviewed-by: NJesse Zhang <jzhang@pivotal.io>
Reviewed-by: NZhenghua Lyu <zlv@pivotal.io>
Reviewed-by: NMelanie Plageman <mplageman@pivotal.io>
Reviewed-by: NYandong Yao <yyao@pivotal.io>

Currently, gp_dbid and gp_contentid is passed as command line
arguments for starting QD and QE. Since, the values are stored in
master's catalog table, to get the right values, must start the master
first. Hence, hard-coded dbid=1 was used for starting the master in
admin mode always. This worked fine till dbid was not used for
anything on-disk. But given dbid is used for tablespace path in GPDB
6, startting the instance with wrong dbid, means inviting recovery
time failues, data corruption or data loss situations. Dbid=1 will go
wrong after failover to standby master as it has dbid != 1. This
commit hence eliminate the need of passing the gp_dbid and
gp_contentid on command line, instead while creating the instance the
values are stored in conf files for the instance.

This also helps to avoid passing gp_dbid as argument to pg_rewind,
which needs to start target instance in single user mode to complete
recovery before performing rewind operation.

Plus, this eases during development to just use pg_ctl start and not
require to correctly pass these values.

 - gp_contentid is stored in postgresql.conf file.

 - gp_dbid is stored in internal.auto.conf.

 - Introduce internal.auto.conf file created during
   initdb. internal.auto.conf is included from postgresql.conf file.

 - Separate file is chosen to write gp_dbid for ease handling during
   pg_rewind and pg_basebackup, as can exclude copying this file from
   primary to mirror, instead of trying to edit the contents of the
   same after copy during these operations. gp_contentid remains same
   for primary and mirror hence having it in postgresql.conf file
   makes senes. If gp_contentid is also stored in this new file
   internal.auto.conf then pg_basebackup needs to be passed contentid
   as well to write to this file.

 - pg_basebackup: write the gp_dbid after backup. Since, gp_dbid is
   unique for primary and mirror, pg_basebackup excludes copying
   internal.auto.conf file storing the gp_dbid. pg_basebackup explicit
   (over)writes the file with value passed as
   --target-gp-dbid. --target-gp-dbid due to this is mandatory
   argument to pg_basebackup now.

 - gpexpand: update gp_dbid and gp_contentid post directory copy.

 - pg_upgrade: retain all configuration files for
   segment. postgresql.auto.conf and internal.auto.conf are also
   internal configuration files which should be restored back after
   directory copy. Similar, change is required in gp_upgrade repo in
   restoreSegmentFiles() after copyMasterDirOverSegment().

 - Update tests to avoid passing gp_dbid and gp_contentid.
Co-authored-by: NAlexandra Wang <lewang@pivotal.io>

If the cluster is in expansion mode, then there must be some partial
tables which are not equal between numsegments and cluster size.
Now we check the expansion status by table's numsegments when we run
gpexpand, gpexpand will raise an error if there are partial tables.

Relkind 'm' stands for materialized view now. This addressed a
GPDB_93_MERGE_FIXME.
Co-authored-by: NAlexandra Wang <lewang@pivotal.io>
Co-authored-by: NJimmy Yih <jyih@pivotal.io>

Change the mapping AO segfilenum+rownum to an ItemPointer, so that we
avoid using ItemPointer.ip_posid values higher than 32768. Such offsets
are impossible on heap tables, because you can't fit that many tuples on
a page. In GiST, since PostgreSQL 9.1, we have taken advantage of that by
using 0xfffe (65534) to mark special "invalid" GiST tuples. We can tolerate
that, because those invalid tuples can only appear on internal pages, so
they cannot be confused with AO TIDs, which only appear on leaf pages. But
we will also use of those high values for other similar magic values, in
later version of PostgreSQL, so it seems better to keep clear of them, even
if we could make it work.

To allow binary upgrades of indexes that already contain AO tids with high
offsets, we still allow and handle those, too, in the code to fetch AO
tuples. Also relax the sanity check in GiST code, to not confuse those high
values with invalid tuples.

Fixes https://github.com/greenplum-db/gpdb/issues/6227Reviewed-by: NAshwin Agrawal <aagrawal@pivotal.io>

To make merging and diffing with upstream easier.

Back when the initial backport of pg_upgrade was added to Greenplum
we included a small dump of a 4.3.x cluster for testing purposes.
This cluster was inadequate for testing pg_upgrade back then, and is
even more so today. Remove to reduce potential confusion that these
files are actually part of the now quite extensive test harness for
pg_upgrade in Greenplum. The cluster also requires access to a 4.3
cluster which is a licensed product, so remove from the open source
repository.

Discussion: https://github.com/greenplum-db/gpdb/pull/6149Reviewed-by: NJim Doty <jdoty@pivotal.io>
Reviewed-by: NJacob Champion <pchampion@pivotal.io>

Our window aggregate's now work the same as in upstream, and this contrib
module's code is 100% identical to upstream too, except for this FIXME.
I also tried running the tsearch2 regression tests. It failed because of
some trivial-looking issues, but all the tests for the rewrite() aggregate
worked fine.

USER MAPPING doesn't have the mpp_execute option, pass the unchanged
options to its validator.

The values would be the default values for things of them.

For example:

If a foreign table has not set the mpp_execute, it follows the server,
if a server has not set the mpp_execute, it follows the wrapper,
if a wrapper has not set the mpp_execute, it chooses the default value,
which is 'master'.

This is a backport of the below commit from upstream PostgreSQL,
which was originally written for Greenplum and submitted as an
upstream-first feature. The commit didn't cherrypick as upstream
has moved pg_upgrade to src/bin and Greenplum has yet to merge
that.
Reviewed-by: NJacob Champion <pchampion@pivotal.io>

  commit 2d34ad84
  Author: Tom Lane <tgl@sss.pgh.pa.us>
  Date:   Sat Dec 1 15:45:11 2018 -0500

    Add a --socketdir option to pg_upgrade.

    This allows control of the directory in which the postmaster sockets
    are created for the temporary postmasters started by pg_upgrade.
    The default location remains the current working directory, which is
    typically fine, but if it is deeply nested then its pathname might
    be too long to be a socket name.

    In passing, clean up some messiness in pg_upgrade's option handling,
    particularly the confusing and undocumented way that configuration-only
    datadirs were handled.  And fix check_required_directory's substantially
    under-baked cleanup of directory pathnames.

    Daniel Gustafsson, reviewed by Hironobu Suzuki, some code cleanup by me

    Discussion: https://postgr.es/m/E72DD5C3-2268-48A5-A907-ED4B34BEC223@yesql.se

The Greenplum specific error handling via ereport()/elog() calls was
in need of a unification effort as some parts of the code was using a
different messaging style to others (and to upstream). This aims at
bringing many of the GPDB error calls in line with the upstream error
message writing guidelines and thus make the user experience of
Greenplum more consistent.

The main contributions of this patch are:

* errmsg() messages shall start with a lowercase letter, and not end
  with a period. errhint() and errdetail() shall be complete sentences
  starting with capital letter and ending with a period. This attempts
  to fix this on as many ereport() calls as possible, with too detailed
  errmsg() content broken up into details and hints where possible.

* Reindent ereport() calls to be more consistent with the common style
  used in upstream and most parts of Greenplum:

	ereport(ERROR,
			(errcode(<CODE>),
			 errmsg("short message describing error"),
			 errhint("Longer message as a complete sentence.")));

* Avoid breaking messages due to long lines since it makes grepping
  for error messages harder when debugging. This is also the de facto
  standard in upstream code.

* Convert a few internal error ereport() calls to elog(). There are
  no doubt more that can be converted, but the low hanging fruit has
  been dealt with. Also convert a few elog() calls which are user
  facing to ereport().

* Update the testfiles to match the new messages.

Spelling and wording is mostly left for a follow-up commit, as this was
getting big enough as it was. The most obvious cases have been handled
but there is work left to be done here.

Discussion: https://github.com/greenplum-db/gpdb/pull/6378Reviewed-by: NAshwin Agrawal <aagrawal@pivotal.io>
Reviewed-by: NHeikki Linnakangas <hlinnakangas@pivotal.io>

The copied system identifiers from the master need to be made unique so
that mirrors cannot accidently connect to the wrong primaries.  We do
this with the --system-identifier option to pg_resetxlog introduced in
the last commit.

We have also added a test to verify unique system identifiers are
assigned to the segments.
Co-authored-by: NShoaib Lari <slari@pivotal.io>
Co-authored-by: NJim Doty <jdoty@pivotal.io>

Given a database in which the "public" schema does not have its default
OID 2200 (e.g. it was dropped and recreated), pg_upgrade would ignore
the previous OID and use the built-in "public" schema at restoration.
This is because creation of the "public" schema is skipped in
dump / restore.

Add a binary_upgrade field to RestoreOptions, to teach pg_dump /
pg_restore not to skip the public schema during binary upgrade. This
adds a new "--binary-upgrade" option to pg_restore.
Co-authored-by: NJacob Champion <pchampion@pivotal.io>
Co-authored-by: NJesse Zhang <sbjesse@gmail.com>

Discussion: https://github.com/greenplum-db/gpdb/pull/6410
Reviewed-by: Heikki Linnakangas

Given the online gpexpand work, the gp_num_contents_in_cluster GUC is
unused. So, delete the same from code to avoid confusions and eliminate this
long argument required to start a postgres instance in gpdb.

This allows the data to be visible on the segments. The segements
should not interpret any transaction id from master during or after
upgrade.
Co-authored-by: NAsim R P <apraveen@pivotal.io>
Co-authored-by: NKalen Krempely <kkrempely@pivotal.io>

Thanks to "Remove restriction to INSERT/UPDATE/DELETE foreign relations
(#6202)", get the right upstream error message back.

Forbid `mpp_execute 'any'`, pass `on_segment` for 'all segments'.

* Replace ExecStoreTuple with ExecStoreHeapTuple in postgres_fdw, to make
  it compile.

* Teach plan_tree_walker() to recurse into ForeignScan.fdw_exprs field.
  This fixes a crash in the postgres_fdw regression tests.

This is enough to make postgres_fdw compile, and not crash. It's still
failing a lot of regression tests, though. The failures need further
investigation, but this is a start.
Reviewed-by: NFrancisco Guerrero <aguerrero@pivotal.io>

During an upgrade, there are numerous new objects created in the new
cluster which either doesn't have an Oids preallocated from the old,
or which never existed in the old cluster to begin with. These objects
are assigned Oids in the new cluster which may collide with Oids which
are preassigned from the old cluster, but which hasn't yet been used
in the restore process. This is because the restore process is doing
the preallocation just before the object creation.

To avoid collisions, the Oids which will be preassigned are tracked
during schema dump, and are injected into the Oid dispatch machinery
before any object is restored such that the list can be queried. This
requires a new mode of dumping in pg_dump where an object is recorded
first in the dependency chain but is dumped last. To support this, a
new API for amending ArchiveEntry objects has been added. Currently
it only supports changing the definition but it can easily be extended
to cover future usecases.

The performance of this patch is a TODO, passing all the Oids in an
array in a single call is unlikely to scale to real-world scenarios
but it's better to subject this to a wider audience sooner rather
than later.

Also fix up since-long outdated documentation comments in oid_dispatch.c
and some minor style nits while in there.
Reviewed-by: NJacob Champion <pchampion@pivotal.io>