If the directory of tablespace does not exist, we should got a
error on commit transaction. But error on commit transaction will
cause a panic. So the directory of tablespace should be checked
so that we can avoid panic.

This also applies the fix in efd76c4c to all callers of
force_mirrors_to_catch_up(), thus eliminating flakes such as:

```
-- Ensure that the mirrors including the standby master have removed the dboid dir under the target tablespace.
 SELECT gp_wait_until_triggered_fault('after_drop_database_directories', 1, dbid) FROM gp_segment_configuration WHERE role='m';
- gp_wait_until_triggered_fault
--------------------------------
- Success:
- Success:
- Success:
- Success:
-(4 rows)
-
+ERROR:  failed to inject fault: ERROR:  fault not triggered, fault name:'after_drop_database_directories' fault type:'wait_until_triggered' (gp_inject_fault.c:127)
+DETAIL:  DETAIL:  Timed-out as 10 minutes max wait happens until triggered.
```

These tests were invoking the new fault injector interface using its
temporary name - gp_inject_fault2().  Now the new interface is the
only one present in the repository, named as gp_inject_fault().

Co-authored-by: NSoumyadeep Chakraborty <sochakraborty@pivotal.io>

It is possible that mirror/standby might not flush or not finish flushing the
related xlog soon after the tablespace drop/create command finishes, so using
'ls' or 'find' to check tablespace related directories in the test might fail.

gp_tablespace test failure was observed several times recently, due to this.
Adding UDFs to poll the status until timeout.

This patch does some minor cleanup for other files also.

Reviewed-by: Asim Praveen

TAR has a limitation that symlink targets must be shorter than 100
characters. Base backup will truncate the symlink target path beyond
100 characters to match the TAR spec.

This change introduces a warning to the user while creating a
tablespace with some hints for how to avoid the problem.

This commit includes changes to the server to ensure that the utilities:
pg_rewind and pg_basebackup can be changed to support recovery in a
multi-segment-singular-host setting. We link pg_tblspc to a <dbid>
subdirectory of the tablespace, rather than to the path of the
tablespace directly, and we remove the <dbid> from the tablespace
version directory. At the same time, we have designed towards preserving
the response to pg_tablespace_location(<tablespace_oid>) such that it
does not return the dbid suffix. The design is such that it is the
responsibility of the utilities to append the dbid as and when required.

Before this commit:
  * the symlink to the tablespace directory looks like:
      pg_tblspc/spcoid/ -> /<tablespace_location>/
  * Under the symlink target, we would have the following:
    GPDB_MAJORVER_CATVER_db<dbid>/dboid/relfilenode
  * pg_tablespace_location(tsoid) returns: <tablespace_location>
e.g.
  * pg_tblspc/20981/ -> /data1/tsp1
  * Under /data1/tsp1: GPDB_6_201902061_db1/19849/192814
  * pg_tablespace_location(20981) returns: /data1/tsp1

After this commit:
  * the symlink to the tablespace directory looks like:
      pg_tblspc/spcoid/ -> /<tablespace_location>/<dbid>
  * Under the symlink target, we would have the following:
      GPDB_MAJORVER_CATVER/dboid/relfilenode
  * pg_tablespace_location(tsoid) returns: <tablespace_location>

e.g.
  * pg_tblspc/20981/ -> /data1/tsp1/1
  * Under /data1/tsp1/1: GPDB_6_201902061/19849/192814
  * pg_tablespace_location(20981) returns: /data1/tsp1

Motivation:

When tablespaces were aligned to upstream postgres, while removing
filespaces, we added the `tablespace_version_directory()` function to
supply each segment with a unique tablespace directory name. This was
accomplished by appending the 'magic' `GpIdentity.dbid` global variable
to the `GP_TABLESPACE_VERSION_DIRECTORY` in `tablespace_version_directory()`.

This is problematic for several reasons- but perhaps most severely is
the fact that in order to use any code in libpgcommon.so that references
this value, you need to first set the `GpIdentity.dbid` global,
otherwise any functions that deal with tablespaces will be broken in
unpredictable ways.

An example is pg_rewind- where `GetRelationPath()` will not return a valid
relation unless you repeatedly toggle the `GpIdentity.dbid` between the
value of the source or target segment dependant on the context of which
relfiles are being examined.

This commit bumps the catalog version here we have made breaking changes
in the tablespace filesystem layout.
Co-authored-by: NAdam Berlin <aberlin@pivotal.io>
Co-authored-by: NTaylor Vesely <tvesely@pivotal.io>
Co-authored-by: NSoumyadeep Chakraborty <sochakraborty@pivotal.io>

- Unlogged recovery behavior requires an init fork for appendonly table and its auxilliary tables.
They allow recovery to identify tables that are unlogged and need truncation.
Co-authored-by: NAsim R P <apraveen@pivotal.io>

Currently upstream supports two options: seq_page_cost and random_page_cost. In theory we possibly do not need to mpp-ify this utility stmt, but it is better to do that in case upstream or greenplum add some other option that affects QE.

PG9.4 starts to allow the WITH syntax to support options
in create tablespace. Greenplum previously used the OPTIONS
syntax to support per segment location. Let's union them all to
use the WITH syntax only, following upstream.

Note the greenplum specific OPTIONS exists in gpdb master only.

Alter tablespace needs to copy all underlying files of table from one tablespace
to other. For AO/CO tables this was implemented using full directory scan to
find files and copy when persistent tables were removed. This gets very
inefficient and varies in performance based on number of files present in the
directory. Instead use the same optimization logic used for `mdunlink_ao()`
leveraging known file layout for AO/CO tables.

Also, old logic had couple of bugs:
- missed coping the base or .0 file. Which means data loss if table was altered in past.
- xlogging even for temp tables

These are fixed as well with this patch. Additional tests added to cover for
those missing scenaiors. Also, moved the AO specific code to aomd.c, out of
tablecmds.c file to reduce conflicts to upstream.

The pg_tablespace_location function only returns the tablespace
location of the connected segment. For Greenplum, we need to be able
to easily see the tablespace locations for all segments. This is
especially needed because we support defining tablespace locations
per-segment.

The old filespace concept allowed for different physical location per
segment/mirror, whereas the tablespace definition only support a
single location.  Tablespaces are backed by a symlink which can be
changed without any changes required on the database level, as long as
the system is turned off at the time. To recreate the filespace
functionality, a process like this would be required:

  1. CREATE TABLESPACE foo LOCATION '/foo/bar';
  2. Shut down the cluster
  3. Move the symlinks to preferred location on each segment
  4. Start cluster back up

Since step 3 require accessing each segment in the cluster, it is
preferable to allow syntax do to this automatically. This commit adds
support for specifying per segment overrides from the default LOCATION
by using an optional OPTIONS clause to specify the segment's content
ID and desired non-default location:

  CREATE TABLESPACE foo LOCATION '/foo/bar'
    OPTIONS (content1 '/content1/bar', content2 '/content2/bar');

Any segment content ID not listed in the OPTIONS clause will get the
LOCATION as its tablespace symlink. The spclocation will be set to
LOCATION, but that column will shortly go away as it's no longer used.

Greenplum Developers mailing thread:
https://groups.google.com/a/greenplum.org/forum/#!msg/gpdb-dev/vmdRjm54on8/nj2SSns0AwAJCo-authored-by: NJimmy Yih <jyih@pivotal.io>