Most of these codes were set only by GpSegStart.__convertSegments, which
has been unused since ce4d96b6. Remove that function as well.

The final client of SEGSTART_ERROR_MIRRORING_FAILURE, gpstart, has been
simplified; the concept of a "mirroring failure" has not been supported
since the removal of filerep.

The -i flag is deprecated and only available on Linux systems.  On
MacOS and other BSD systems, xargs does not have the -i flag.  Since
we do not give an argument to -i, it is basically doing -I{} so change
it to be explicit and proper.

Co-authored-by: NAlexandra Wang <lewang@pivotal.io>

Currently, gp_dbid and gp_contentid is passed as command line
arguments for starting QD and QE. Since, the values are stored in
master's catalog table, to get the right values, must start the master
first. Hence, hard-coded dbid=1 was used for starting the master in
admin mode always. This worked fine till dbid was not used for
anything on-disk. But given dbid is used for tablespace path in GPDB
6, startting the instance with wrong dbid, means inviting recovery
time failues, data corruption or data loss situations. Dbid=1 will go
wrong after failover to standby master as it has dbid != 1. This
commit hence eliminate the need of passing the gp_dbid and
gp_contentid on command line, instead while creating the instance the
values are stored in conf files for the instance.

This also helps to avoid passing gp_dbid as argument to pg_rewind,
which needs to start target instance in single user mode to complete
recovery before performing rewind operation.

Plus, this eases during development to just use pg_ctl start and not
require to correctly pass these values.

 - gp_contentid is stored in postgresql.conf file.

 - gp_dbid is stored in internal.auto.conf.

 - Introduce internal.auto.conf file created during
   initdb. internal.auto.conf is included from postgresql.conf file.

 - Separate file is chosen to write gp_dbid for ease handling during
   pg_rewind and pg_basebackup, as can exclude copying this file from
   primary to mirror, instead of trying to edit the contents of the
   same after copy during these operations. gp_contentid remains same
   for primary and mirror hence having it in postgresql.conf file
   makes senes. If gp_contentid is also stored in this new file
   internal.auto.conf then pg_basebackup needs to be passed contentid
   as well to write to this file.

 - pg_basebackup: write the gp_dbid after backup. Since, gp_dbid is
   unique for primary and mirror, pg_basebackup excludes copying
   internal.auto.conf file storing the gp_dbid. pg_basebackup explicit
   (over)writes the file with value passed as
   --target-gp-dbid. --target-gp-dbid due to this is mandatory
   argument to pg_basebackup now.

 - gpexpand: update gp_dbid and gp_contentid post directory copy.

 - pg_upgrade: retain all configuration files for
   segment. postgresql.auto.conf and internal.auto.conf are also
   internal configuration files which should be restored back after
   directory copy. Similar, change is required in gp_upgrade repo in
   restoreSegmentFiles() after copyMasterDirOverSegment().

 - Update tests to avoid passing gp_dbid and gp_contentid.
Co-authored-by: NAlexandra Wang <lewang@pivotal.io>

pg_rewind --slot is mutual exclusive with --source-pgdata
Co-authored-by: NDavid Kimura <dkimura@pivotal.io>

gpstart calls gpstop, so honor the parallel parameter by passing it along to gpstop.
Authored-by: NKalen Krempely <kkrempely@pivotal.io>

The pass statement is a no-op intended to be used in empty classes
or other types of stubs. When following other statements in a block
it has no purpose. Remove all such occurrences.
Reviewed-by: NJacob Champion <pchampion@pivotal.io>

Use gpstart '-B' ('--parallel') option value to limit the maximum size
of a thread pool used by gpsegstart to create segments.

Put a limit on the number of workers in a pool created by gpstart.

When there are many segments on one server node, the unlimited number
of workers may open unlimited number of pipes.
This may lead to a failure of 'select()' call
in 'gpMgmt/bin/gppylib/gpsubprocess.py' and inability
to start the cluster at all.

This commit prevents the described problem from happening.

Closes https://github.com/greenplum-db/gpdb/issues/6176

Because expand copies the entire master directory, we need to zero out the
distributed_xlog since the segment is not yet part of the cluster, to
allow it to only see local transactions.  Fixed the log level to WARN
when expanding a cluster with unique indexes.
Co-authored-by: NDavid Krieger <dkrieger@pivotal.io>
Co-authored-by: NJim Doty <jdoty@pivotal.io>
Co-authored-by: NKalen Krempely <kkrempely@pivotal.io>

These debug logs are extremely helpful in debugging failures in our Python
utility execution calls which are usually enabled when passing the utility's
verbose flag. Specifically, Command objects should print out the result
stdout/stderror when validation fails and Segment object string representation
should contain the role and preferred role.
Co-authored-by: NPaul Guo <pguo@pivotal.io>

Now that pg_rewind is available, we can use it to incrementally recover a failed
primary segment that will be transitioning to mirror. Incremental recovery with
pg_rewind has a big performance advantage over full recovery. We also change
rebalance to use pg_rewind incremental recovery. However, if pg_rewind fails
during gprecoverseg, the user must run full recovery (gprecoverseg -F).
Co-authored-by: NPaul Guo <pguo@pivotal.io>

Given the online gpexpand work, the gp_num_contents_in_cluster GUC is
unused. So, delete the same from code to avoid confusions and eliminate this
long argument required to start a postgres instance in gpdb.

Here revert four commits

1. Remove Master/Standby SyncRepWait Greenplum hack: 7f6066ea
2. Add alter system synchronous_standby_names to * when gpinitstandby -n 1136f2fb
3. Hot fix in gpinitstandby behave test failure: b6c77b2f
4. Remove unused variables, to silence compiler warnings 88a185a5

The first commit should revert, because gpinitstandby did not change correctly comform to the this commit.
It will cause If the standby is down, and the synchronous_standby_names is *, the cluster will not start and hang on

The second, third, fourth should revert, because it is relavent to the first commit.

After applying that commit 22e04dc12df9e0577ba93a75dbef160c8c1ed258, the master will block when the standby master is down.

There are a couple things that need to be done to unblock the master.
1. Run gpinitstandby -n to start the standby master back up.
2. Run psql postgres -c "ALTER SYSTEM SET synchronous_standby_names = '';" and reload the master segment.

Note that the ALTER SYSTEM SET has to be called again to set synchronous_standby_names back to '*' (and master config reloaded) to enable synchronous replication again.
Thoughts are to make it 1 step combined into gpinitstandby -n instead of documenting a multi-step process.

What this commit is just to make it 1 step combined into gpinitstandby -n.
Co-authored-by: NNing Yu <nyu@pivotal.io>

* Protect catalog changes on master.

To allow gpexpand to do the job without restarting the cluster we need
to prevent concurrent catalog changes on master.  A catalog lock is
provided for this purpose, all insert/update/delete to catalog tables
should hold this lock in shared mode before making the changes; gpexpand
can hold this lock in exclusive mode to 1) wait for in-progress catalog
updates to commit/rollback and 2) prevent concurrent catalog updates.

Add UDF to hold catalog lock in exclusive mode.

Add test cases for the catalog lock.
Co-authored-by: NJialun Du <jdu@pivotal.io>
Co-authored-by: NNing Yu <nyu@pivotal.io>

* Numsegments management.

GpIdentity.numsegments is a global variable saving the cluster size in
each process. It is important for cluster. Changing of the cluster size
means the expansion is finished and new segments have taken effect.

FTS will count the number of primary segments from gp_segment_configuration
and record it in shared memory. Later, other processes including master,
fts, gdd, QD will update GpIdentity.numsegmentswith this information.

So far it's not easy to make old transactions running with new segments,
so for QD GpIdentity.numsegments can only be updated at beginning of
transactions. Old transactions can only see old segments. QD will dispatch
GpIdentity.numsegments to QEs, so they will see the same cluster size.

Catalog changes in old transactions are disallowed.

Consider below workflow:

        A: begin;
        B: gpexpand from N segments to M (M > N);
        A: create table t1 (c1 int);
        A: commit;
        C: drop table t1;

Transaction A began when cluster size was still N, all its commands are
dispatched to N segments only even after cluster being expanded to M
segments, so t1 was only created on N tables, not only the data
distribution but also the catalog records.  This will cause the later
DROP TABLE to fail on the new segments.

To prevent this issue we currently disable all catalog updates in old
transactions once the expansion is done.  New transactions can update
catalog as they are already running on all the M segments.
Co-authored-by: NJialun Du <jdu@pivotal.io>
Co-authored-by: NNing Yu <nyu@pivotal.io>

* Online gpexpand implementation.

Do not restart the cluster during expansion.
- Lock the catalog
- Create the new segments from master segment
- Add new segments to cluster
- Reload the cluster so new transaction and background processes
  can see new segments
- Unlock the catalog

Add test cases.
Co-authored-by: NJialun Du <jdu@pivotal.io>
Co-authored-by: NNing Yu <nyu@pivotal.io>

* New job to run ICW after expansion.

It is better to run ICW after online expansion to see if the
cluster works well. So we add a new job to create a cluster with
two segments first, then expand to three segments and run all the
ICW cases. Cluster restarting is forbidden for we must be sure
all the cases will be passed after online expansion. So any case
which contains cluster restarting must be removed from this job.

The new job is icw_planner_centos7_online_expand, it is same as
icw_planner_centos7 but contains two new params EXCLUDE_TESTS and
ONLINE_EXPAND. If ONLINE_EXPAND is set, the ICW shell will go to
different branch, creating cluster with size two, expanding, etc.
EXCLUDE_TESTS lists the cases which contain cluster restarting
or the test cases will fail without restarting test cases.

After the whole run, the pid of master will be checked, if it
changes, the cluster must be restarted, so the job will fail.
As a result any new restarting test cases should be added to
EXCLUDE_TESTS.

* Add README.
Co-authored-by: NJialun Du <jdu@pivotal.io>
Co-authored-by: NNing Yu <nyu@pivotal.io>

* Small changes per review comments.

- Delete confusing test case
- Change function name updateBackendGpIdentityNumsegments to
  updateSystemProcessGpIdentityNumsegments
- Fix some typos

* Remove useless Assert

These two asserts will cause failure in isolation2/uao/
compaction_utility_insert. This case will check AO table in utility
mode. But numsegments are meaningless in sole segment, so this
assert must be removed.

This reverts commit 4750e1b6.

This is the final batch of commits from PostgreSQL 9.2 development,
up to the point where the REL9_2_STABLE branch was created, and 9.3
development started on the PostgreSQL master branch.

Notable upstream changes:

* Index-only scan was included in the batch of upstream commits. It
  allows queries to retrieve data only from indexes, avoiding heap access.

* Group commit was added to work effectively under heavy load. Previously,
  batching of commits became ineffective as the write workload increased,
  because of internal lock contention.

* A new fast-path lock mechanism was added to reduce the overhead of
  taking and releasing certain types of locks which are taken and released
  very frequently but rarely conflict.

* The new "parameterized path" mechanism was added. It allows inner index
  scans to use values from relations that are more than one join level up
  from the scan. This can greatly improve performance in situations where
  semantic restrictions (such as outer joins) limit the allowed join orderings.

* SP-GiST (Space-Partitioned GiST) index access method was added to support
  unbalanced partitioned search structures. For suitable problems, SP-GiST can
  be faster than GiST in both index build time and search time.

* Checkpoints now are performed by a dedicated background process. Formerly
  the background writer did both dirty-page writing and checkpointing. Separating
  this into two processes allows each goal to be accomplished more predictably.

* Custom plan was supported for specific parameter values even when using
  prepared statements.

* API for FDW was improved to provide multiple access "paths" for their tables,
  allowing more flexibility in join planning.

* Security_barrier option was added for views to prevents optimizations that
  might allow view-protected data to be exposed to users.

* Range data type was added to store a lower and upper bound belonging to its
  base data type.

* CTAS (CREATE TABLE AS/SELECT INTO) is now treated as utility statement. The
  SELECT query is planned during the execution of the utility. To conform to
  this change, GPDB executes the utility statement only on QD and dispatches
  the plan of the SELECT query to QEs.
Co-authored-by: NAdam Lee <ali@pivotal.io>
Co-authored-by: NAlexandra Wang <lewang@pivotal.io>
Co-authored-by: NAshwin Agrawal <aagrawal@pivotal.io>
Co-authored-by: NAsim R P <apraveen@pivotal.io>
Co-authored-by: NDaniel Gustafsson <dgustafsson@pivotal.io>
Co-authored-by: NGang Xiong <gxiong@pivotal.io>
Co-authored-by: NHaozhou Wang <hawang@pivotal.io>
Co-authored-by: NHeikki Linnakangas <hlinnakangas@pivotal.io>
Co-authored-by: NJesse Zhang <sbjesse@gmail.com>
Co-authored-by: NJinbao Chen <jinchen@pivotal.io>
Co-authored-by: NJoao Pereira <jdealmeidapereira@pivotal.io>
Co-authored-by: NMelanie Plageman <mplageman@pivotal.io>
Co-authored-by: NPaul Guo <paulguo@gmail.com>
Co-authored-by: NRichard Guo <guofenglinux@gmail.com>
Co-authored-by: NShujie Zhang <shzhang@pivotal.io>
Co-authored-by: NTaylor Vesely <tvesely@pivotal.io>
Co-authored-by: NZhenghua Lyu <zlv@pivotal.io>

This was only used by gprepairmirrorseg, which was removed in a previous
commit.
Authored-by: NChris Hajas <chajas@pivotal.io>

The --file flag was broken because the command to read from segment
postgresql.conf was being run locally instead of remotely.
Co-authored-by: NJamie McAtamney <jmcatamney@pivotal.io>
Co-authored-by: NNadeem Ghani <nghani@pivotal.io>

Handling sequences in MPP stage is challenging. This patch refactors the same
mainly to eliminate the shortcomings/pitfalls in previous implementation. Lets
first glance at issues with older implementation.

  - required relfilenode == oid for all sequence relations
  - "denial of service" due to dedicated but single process running on QD to
    serve sequence values for all the tables in all the databases to all the QEs
  - sequence tables direct opened as a result instead of flowing throw relcache
  - divergence from upstream implementation

Many solutions were considered refer mailing discussion for the same before
settling on one in here. Newer implementation still leverages centralized place
QD to generated sequence values. It now leverages the existing QD backend
process connecting to QEs for the query to serve the nextval request. As a
result need for relfilenode == oid gets eliminated as based on oid, QD process
can now lookup relfilenode from catalog and also leverage relcache. No more
direct opens by single process across databases.

For communication between QD and QE for sequence nextval request, async notify
message is used (Notify messages are not used in GPDB for anything else
currently). QD process while waiting for results from QEs is sitting idle and
hence on seeing the nextval request, calls `nextval_internal()` and responds
back with the value.

Since the need for separate sequence server process went away, all the code for
the same is removed.

Discussion:
https://groups.google.com/a/greenplum.org/d/msg/gpdb-dev/hni7lS9xH4c/o_M3ddAeBgAJCo-authored-by: NAsim R P <apraveen@pivotal.io>
Co-authored-by: NAshwin Agrawal <aagrawal@pivotal.io>

Authored-by: NDavid Sharp <dsharp@pivotal.io>

This is needed during gprecoverseg full to preserve important files
such as pg_log files. We pass this flag down the call stack to prevent
other utilities such as gpinitstandby or gpaddmirror from using the
new flag. The new flag can be dangerous if not used properly and
should only be used when data directory file preservation is
necessary.

The pg_log directory has always been excluded using the pg_basebackup
exclude option (-E ./pg_log). With this change, we add it to the
static list inside of basebackup. Because of this change, we are able
to remove all instances of mkdir pg_log in our management
utilities. Previously, the utilities would always have to create the
pg_log directory after running pg_basebackup because the postmaster
does a validation check on the pg_log path existing.

This also helps us align better with upstream Postgres since the
pg_basebackup exclude option is Greenplum-specific and really not
needed at all. Our dynamic exclusion list hasn't changed for a very
long time (so it's pretty much static anyways) and is not maintained
in the utilities very well. We may actually remove the pg_basebackup
exclude option in the near future.

listen_addresses in postgresql.conf doesn't taken effect now, backend and
postmaster are listening on all addresses. From the point of security, we
should be able to let user specify the listen address.

gpstart did a cluster-wide check of heap_checksum settings and refused
to start the cluster if this setting was inconsistent. This meant a
round of ssh'ing across the cluster which was causing OOM errors with
large clusters.

This commit moves the heap_checksum validation to gpsegstart.py, and
changes the logic so that only those segments which have the same
heap_checksum setting as master are started.

Author: Jim Doty <jdoty@pivotal.io>
Author: Nadeem Ghani <nghani@pivotal.io>
Author: Shoaib Lari <slari@pivotal.io>

Revert the state machine and other logic in postmaster.c the way it is in
upstream. Remove some GUCs related to mirrored and non-mirrored mode. Remove
the -M, -x and -y postmaster options, and change management scripts to not
pass those options.

The current behavior is to check for the existence of dumps dirs and, unless
run with the -f flag, error out if they do. This commit extends this
check to backups.

Author: Nadeem Ghani <nghani@pivotal.io>

- Change gpdeletsystem to delete tablespaces before datadir
- Refactor SegmentStart.noWait to pg_ctrl_wait
- Create PgBaseBackup class
- Revert the change to default Mirror status
- Assorted typos and bugfixes

Author: Shoaib Lari <slari@pivotal.io>
Author: Jim Doty <jdoty@pivotal.io>
Author: Nadeem Ghani <nghani@pivotal.io>

The gpexpand utility and associated code are modified to work with the WALREP
code.  Previously, gpexpand only worked with the primaries and relied on Filerep
to populate the mirrors. We are changing gpexpand such that it initializes the
mirrors using pg_basebackup and set them up for WAL replication.

Also, since the WALREP branch removed Filespaces, so we have also done the same
and replaced references to Filespaces by the Data Directory of the segments.

Author: Marbin Tan <mtan@pivotal.io>
Author: Shoaib Lari <slari@pivotal.io>
Author: Nadeem Ghani <nghani@pivotal.io>

This reverts commit fe30f20aacd78cfc1b9ba587152b8c73fbb2d045.

We can start mirror with -w flag now.

Author: Ashwin Agrawal <aagrawal@pivotal.io>
Author: Jimmy Yih <jyih@pivotal.io>

It set the master_mirroring_administrator_disable variable, but it's not
used for anything. Remove the variable, and all the code to pass that flag
through.

It was now unused.

Remove the concept of filespaces, revert tablespaces to work the same as
in upstream.

There is some leftovers in management tools. I don't know how to test all
that, and I was afraid of touching things I can't run. Also, we may need
to create replacements for some of those things on top of tablespaces, to
make the management of tablespaces easier, and it might be easier to modify
the existing tools than write them from scratch. (Yeah, you could always
look at the git history, but still.)

Per the discussion on gpdb-dev mailing list, the plan is to cherry-pick
commit 16d8e594 from PostgreSQL 9.2, to make it possible to have a
different path for a tablespace in the primary and its mirror. But that's
not included in this commit yet.

TODO: Make temp_tablespaces work.
TODO: Make pg_dump do something sensible, when dumping from a GPDB 5 cluster
that uses filespaces. Same with pg_upgrade.

Discussion: https://groups.google.com/a/greenplum.org/d/msg/gpdb-dev/sON4lraPEqg/v3lkM587BAAJ