Relfilenode is only unique within a tablespace. Across tablespaces same
relfilenode may be allocated within a database. Currently, gp_relation_node only
stores relfilenode and segment_num and has unique index using only those fields
without tablespace. So, it breaks for situations where same relfilenode gets
allocated to table within database.

This commit substantially rewrites pg_upgrade to handle upgrading a
Greenplum cluster from 4.3 to 5.0. The Greenplum specifics of pg_upgrade
are documented in contrib/pg_upgrade/README.gpdb. A summary of the
changes is listed below:

 - Make pg_upgrade to pass the pre-checks against GPDB 4.3.
 - Restore dumped schema in utility mode: pg_upgrade is executed on a
   single server in offline mode so ensure we are using utility mode.
 - Disable pg_upgrade checks that don't apply when upgrading to 8.3:
   When support for upgrading to Greenplum 6.0 is added the checks that
   make sense to backport will need to be readded.
 - Support AO/AOCS table: This bumps the AO table version number, and
   adds a conversion routine for numeric attributes. The on-disk format
   of numerics changed between PostgreSQL 8.3 and 8.4. With this commit,
   we can distinguish between AO segments created in the old format and
   the new, and read both formats. New AO segments are always created in
   the new format. Also performs a check for AO tables having NUMERIC
   attributes without free segfiles. Since AO table segments cannot be
   rewritten if there are no free segfiles, issue a warning if such a
   table is encountered during the upgrade.
 - Add code to convert heap pages offline: Bumps heap page format version
   number. While this isn't strictly necessary, when we're doing the
   conversion off-line, it reduces confusion if something goes wrong.
 - Add check for money datatype: the upgrade doesn't support the money
   datatype so check for it's presence and abort upgrade if found.
 - Create new Oid in QD and pass new Oids in dump for pg_upgrade on QE:
   When upgrading from GPDB4 to 5, we need to create new arraytypes for
   the base relation rowtypes in the QD, but we also need to dispatch
   these new OIDs to the QEs. Objects assigning InvalidOid in the Oid
   dispatcher will cause a new Oid to be assigned. Once the new cluster
   is restored, dump the new Oids into a separate dumpfile which isn't
   unlinked on exit. If this file is placed into the cwd of pg_upgrade
   on the QEs, it will be pulled into the db dump and used during
   restoring, thus "dispatching" the Oids from the QD even though they
   are offline. pg_upgrade doesn't at this point know if it's running
   at a QD or a QE so it will always dump this file and include the
   InvalidOid markers.
 - gp_relation_node is reset and rebuilt during upgrade once the data
   files from the old cluster are available to the new cluster. This
   change required altering how checkpoints are requested in the
   backend.
 - Mark indexes as invalid to ensure they are rebuilt in the new
   cluster.
 - Copy the pg_distributedlog from old to new during upgrade: We need
   the distributedlog in the new cluster to be able to start up once
   the upgrade has pulled over the clog.
 - Dont delete dumps when runnin with --debug: While not specific to
   Greenplum, this is a local addition which greatly helps testing
   and development of pg_upgrade.

For testing purposes, a small test cluster created with Greenplum 4.3
is included in contrib/pg_upgrade/test

Heikki Linnakangas, Daniel Gustafsson and Dave Cramer

The MemTupleNoOidSpace() wasn't implemented and did just do an
assertion rather than what it said on the tin. Remove since the
intention of the macro isn't really possible - there is nothing
like HEAP_HASOID for memtuples.

Segments that are still in an old format are treated as read-only. All new
data go to new segments, in new format. This allows us to eventually get
rid of the old format completely.

This is hypothetical until we have pg_upgrade working for GPDB 4.3 -> 5.0
upgrade, as you can't have old-format tables or segments at all in a
cluster that's initialized with 5.0. Stay tuned for pg_upgrade, but this
is preparatory work for that.

This meant moving the version field from pg_appendonly to the
pg_aoseg_<oid> table (or pg_aocsseg_<oid>, for AOCS). We can still read
and write both formats, but new segments will always be created in the new
format (except if you set the test_appendonly_version_default GUC).

I found these with "callcatcher", written by Caolán McNamara. Many thanks
for the tool! See https://www.skynet.ie/~caolan/Packages/callcatcher.html

SnapshotNow MUST be used during insert flow to fetch the latest EOF value from
aoseg table. Earlier usage of ActiveSnapshot in AO insert flow caused data
corruption as potentially could read incorrect or stale EOF value from
aoseg. The specific scenario this happens when entry from AppendOnlyHash table
gets evicted out. It invalidates (zeros out) the latestWriteXid and hence
check for usedByConcurrentTransaction cannot be performed. So, any transaction
later inserting data to same AO table, if it has aquired the snapshot and has
latestWriteXid listed in its in-progress distributed transaction list, gets
the same segfile to write. Based on DTM visibility rules using ActiveSnapshot
current transaction will not read the EOF value from previous inserted
transaction as its listed in its in-progress list and hence will overwrite the
data but SnapshotNow will see the latest and make sure to append.

Currently, during insert to AO/CO tables check is performed to validate
file-offset (EOF) found from pg_aoseg which would be used to start writing
the data to file. The offset to start writing needs to be greater than
currently marked EOF in persistent table. The validation is currently
performed for all other scenarios except EOF=0, so this commit is modifing to
enable it for EOF-0 case as well. As it was seen in field for yet to be known
reasons maybe indexing issues on pg_aoseg table, that incorrect EOF=0 was
fetched from aoseg and valid data was over-written. So, this protection will
prevent data-loss scenarios.

It's pretty much identical to Form_pg_appendonly, which is now available
in the relcache. No need to pass this additional struct around.

This way, you don't need to always fetch it from the system catalogs,
which makes things simpler, and is marginally faster too.

To make all the fields in pg_appendonly accessible by direct access to
the Form_pg_appendonly struct, change 'compresstype' field from text to
name. "No compression" is now represented by an empty string, rather than
NULL. I hope there are no applications out there that will get confused
by this.

The GetAppendOnlyEntry() function used to take a Snapshot as argument,
but that seems unnecessary. The data in pg_appendonly doesn't change for
a table after it's been created. Except when it's ALTERed, or rewritten
by TRUNCATE or CLUSTER, but those operations invalidate the relcache, and
we're never interested in the old version.

There's not much need for the AppendOnlyEntry struct and the
GetAppendOnlyEntry() function anymore; you can just as easily just access
the Form_pg_appendonly struct directly. I'll remove that as a separate
commit, though, to keep this one more readable.

Many headers in src/include/catalog have gotten this treatment in the
upstream. This makes life easier for the next commit, which adds a
reference to Form_pg_appendonly to RelationData. With this separation,
we can avoid pulling in a lot of other headers into rel.h, and avoid a
circular dependency,

The appendonlygettup() method fetches appendonly tuples, not heap
tuples, this is probably a copy-paste artefact from heapam.c which
has the same comment. While there also remove unused file header
tags.

This change applies only to column oriented tables (CO).  It reads
block directory entries for only those columns that appear in project
list (e.g. select clause).

Closes #369