Because the gpinitstandby flag for passing in a standby master data directory
was changed from -F to -S, this commit changes -F to -S in gpinitsystem to
match.

-S was already in use for specifying that the cluster should have its mirrors
set up in spread configuration, so that flag is changed to --mirror-mode, which
accepts the mirror mode ('group' or 'spread') as an argument.
Co-authored-by: NJamie McAtamney <jmcatamney@pivotal.io>
Co-authored-by: NMark Sliva <msliva@pivotal.io>

This reintroduces commit ba3eb5b4, which was reverted in 659f0ee5.

After the WALrep changes, the previous -F (filespace) option was
co-opted to be the new standby data directory option. This isn't a
particularly obvious association.

Change the option to -S. (-D would have been better, but that's already
in use as a short alias for --debug.) Also document this option in the
official gpinitstandby help.

One of these changes appears to have possibly introduced a serious
performance regression in the master pipeline. To avoid destabilizing
work over the weekend, I'm reverting for now and we can investigate more
fully next week.

This reverts the following commits:
"gprecoverseg: Show progress of pg_basebackup on each segment"
    1b38c6e8
"Add gprecoverseg -s to show progress sequentially"
    9e89b5ad
"gpinitstandby: guide the user on single-host systems"
    c9c3c351
"gpinitstandby: rename -F to -S and document it"
    ba3eb5b4

After the WALrep changes, the previous -F (filespace) option was
co-opted to be the new standby data directory option. This isn't a
particularly obvious association.

Change the option to -S. (-D would have been better, but that's already
in use as a short alias for --debug.) Also document this option in the
official gpinitstandby help.

Co-authored-by: NAlexandra Wang <lewang@pivotal.io>

Currently, gp_dbid and gp_contentid is passed as command line
arguments for starting QD and QE. Since, the values are stored in
master's catalog table, to get the right values, must start the master
first. Hence, hard-coded dbid=1 was used for starting the master in
admin mode always. This worked fine till dbid was not used for
anything on-disk. But given dbid is used for tablespace path in GPDB
6, startting the instance with wrong dbid, means inviting recovery
time failues, data corruption or data loss situations. Dbid=1 will go
wrong after failover to standby master as it has dbid != 1. This
commit hence eliminate the need of passing the gp_dbid and
gp_contentid on command line, instead while creating the instance the
values are stored in conf files for the instance.

This also helps to avoid passing gp_dbid as argument to pg_rewind,
which needs to start target instance in single user mode to complete
recovery before performing rewind operation.

Plus, this eases during development to just use pg_ctl start and not
require to correctly pass these values.

 - gp_contentid is stored in postgresql.conf file.

 - gp_dbid is stored in internal.auto.conf.

 - Introduce internal.auto.conf file created during
   initdb. internal.auto.conf is included from postgresql.conf file.

 - Separate file is chosen to write gp_dbid for ease handling during
   pg_rewind and pg_basebackup, as can exclude copying this file from
   primary to mirror, instead of trying to edit the contents of the
   same after copy during these operations. gp_contentid remains same
   for primary and mirror hence having it in postgresql.conf file
   makes senes. If gp_contentid is also stored in this new file
   internal.auto.conf then pg_basebackup needs to be passed contentid
   as well to write to this file.

 - pg_basebackup: write the gp_dbid after backup. Since, gp_dbid is
   unique for primary and mirror, pg_basebackup excludes copying
   internal.auto.conf file storing the gp_dbid. pg_basebackup explicit
   (over)writes the file with value passed as
   --target-gp-dbid. --target-gp-dbid due to this is mandatory
   argument to pg_basebackup now.

 - gpexpand: update gp_dbid and gp_contentid post directory copy.

 - pg_upgrade: retain all configuration files for
   segment. postgresql.auto.conf and internal.auto.conf are also
   internal configuration files which should be restored back after
   directory copy. Similar, change is required in gp_upgrade repo in
   restoreSegmentFiles() after copyMasterDirOverSegment().

 - Update tests to avoid passing gp_dbid and gp_contentid.
Co-authored-by: NAlexandra Wang <lewang@pivotal.io>

To create mirrors, pg_basebackup needs to be performed. pg_basebackup
to correctly handle tablespaces needs dbid as argument. This
requirement exist because dbid is used in tablespace path.

dbid in master catalog to be in sync with what's used by mirror for
tablespace, need to add mirror to catalog first. Get the dbid and pass
the same to pg_basebackup for creating mirror.
Co-authored-by: NAlexandra Wang <lewang@pivotal.io>

Given the online gpexpand work, the gp_num_contents_in_cluster GUC is
unused. So, delete the same from code to avoid confusions and eliminate this
long argument required to start a postgres instance in gpdb.

Authored-by: NGoutam Tadi <gtadi@pivotal.io>

[#159742200]
Co-authored-by: NXin Zhang <xzhang@pivotal.io>
Co-authored-by: NGoutam Tadi <gtadi@pivotal.io>

Add behave test for FQDN_HBA flag support
Co-authored-by: NXin Zhang <xzhang@pivotal.io>
Co-authored-by: NGoutam Tadi <gtadi@pivotal.io>

- Behave tests for gpinitsystem with fqdn
Co-authored-by: NGoutam Tadi <gtadi@pivotal.io>
Co-authored-by: NJamie McAtamney <jmcatamney@pivotal.io>

There are a lot of blank lines in the 'gpinitsystem' log. These are
read directly from the configuration file using commands 'cat'. The
comments are turned into blank lines, but they are not removed:

  gpinitsystem:node:gp6-[INFO]:-Start Main
  gpinitsystem:node:gp6-[INFO]:-Command line options passed to utility = -c ../gpinitsystem_config
  gpinitsystem:node:gp6-[INFO]:-Start Function CHK_GPDB_ID
  ...
  gpinitsystem:node:gp6-[INFO]:-End Function CHK_FILE
  gpinitsystem:node:gp6-[INFO]:-Dumping gpinitsystem_config to logfile for reference

  ARRAY_NAME="EMC Greenplum DW"

  SEG_PREFIX=gpseg

  PORT_BASE=40300

This extends the exclusion regex used when appending to the logfile
to remove blank lines completely and to handle comments that doesn't
start on column zero.
Co-authored-by: NDaniel Gustafsson <dgustafsson@pivotal.io>
Reviewed-by: NDaniel Gustafsson <dgustafsson@pivotal.io>

Rather than hardcoding to require /bin/bash, move to using a lookup
via "/usr/bin/env bash" to allow for greater portability of the code.
This also changes the Bash test to checking if the current shell is
actually Bash, rather than looking if bash is available on the file
system (since we by the above mentioned changes no longer need that).

This patch helps cut-down 8 secs out of 30 secs (for single node gpinitsystem as
part of gpdemo), mostly more for multinode setups.

- Removes explicit 2 sec and 1 sleep.
- Removes explicit call to CHECKPOINT, not required.
- Avoid starting primaries after initdb. As they were started and then shutdown
  pretty soon to start back cluster with required cmdline arguments.
- Add -w to mirror start.

Still more room for improvement and speedup, this just gets us started.

listen_addresses in postgresql.conf doesn't taken effect now, backend and
postmaster are listening on all addresses. From the point of security, we
should be able to let user specify the listen address.

Revert the state machine and other logic in postmaster.c the way it is in
upstream. Remove some GUCs related to mirrored and non-mirrored mode. Remove
the -M, -x and -y postmaster options, and change management scripts to not
pass those options.

It just doesn't mean "filespace" anymore, just plain "datadir".

Remove the concept of filespaces, revert tablespaces to work the same as
in upstream.

There is some leftovers in management tools. I don't know how to test all
that, and I was afraid of touching things I can't run. Also, we may need
to create replacements for some of those things on top of tablespaces, to
make the management of tablespaces easier, and it might be easier to modify
the existing tools than write them from scratch. (Yeah, you could always
look at the git history, but still.)

Per the discussion on gpdb-dev mailing list, the plan is to cherry-pick
commit 16d8e594 from PostgreSQL 9.2, to make it possible to have a
different path for a tablespace in the primary and its mirror. But that's
not included in this commit yet.

TODO: Make temp_tablespaces work.
TODO: Make pg_dump do something sensible, when dumping from a GPDB 5 cluster
that uses filespaces. Same with pg_upgrade.

Discussion: https://groups.google.com/a/greenplum.org/d/msg/gpdb-dev/sON4lraPEqg/v3lkM587BAAJ

With file replication gone, gpinitsystem should no longer try to
initialize the cluster through filerep sequence.

The sequence now goes as follows:
1. Create and start master in master-only mode
2. Create primaries and register to master
3. Stop master.
4. Run gpstart to start master and primaries.
5. Create mirrors w/ pg_basebackup and register to master.
6. Start the mirrors and wait until primaries and mirrors sync.

Author: Jimmy Yih <jyih@pivotal.io>
Author: Marbin Tan <mtan@pivotal.io>

And the mechanism in initdb and gpinitsystem to set it. It's no longer
used for anything.

* Update gpinitsystem

edit commit message: Marbin Tan<mtan@pivotal.io>
Change the symbol to actual words to better convey the meaning.
Primary segment # -> Number of primary segments
Mirror segment # -> Number of mirror segments

Originally, gp_add_segment() is only used to add a primary segment. We
created a new gp_add_segment_primary() to keep the original
functionality, which automatically generate dbid and contentid.

We generalized the gp_add_segment() to be able to directly update
gp_segment_configuration and pg_filespace_entry for adding any type of
segments with full specification of segment and filespace mappings. In
this case, the new gp_add_segment() doesn't generate dbid and contentid
automatically, and rely on input parameters.

Originally there is separate code path in gp_add_segment_mirror() to
figure out primary dbid, such logic is actually common even with
gp_add_segment(), which can add both primary and mirror.

In that case, we refactor the primary dbid detection logic in common
function add_segment(), and refactor the gp_add_segment_mirror() to use
the add_segment() instead of add_segment_config_entry().

We update gpinitsystem to use the function gp_add_segment() instead of
update the gp_segment_configuration and pg_filespace_entry tables
directly via SQL.
Signed-off-by: NXin Zhang <xzhang@pivotal.io>

Using gp_segment_configuration catalog table easily can find if mirrors exist or
not, do not need special table to communicate the same. Earlier
gp_fault_strategy used to convey 'n' for mirrorless system, 'f' for replication
and 's' for san mirrors. Since support for 's' was removed in 5.0 only purpose
gp_fault_strategy served was mirrored or not mirrored system. Hence deleting the
gp_fault_strategy table and at required places using gp_segment_configuration to
find the required info.

Previously, during gpinitsystem, the standby was instantiated in the middle of
setting up the master. This ordering caused problems because
initializing the standby could cause an exit when an error
occurred. As a result of this early exit, the gp_toolkit and DCA gucs
were not set properly.

Instead, initialize the standby after the master is finished.
------------------------------------------
Previously the exit return code for gpinitsystem was always non-zero.
Now, it is non-zero only in an error or warning case.

The issue was due to SCAN_LOG interpretation of an empty string as a line count of one.
Fixed by changing to word count.
------------------------------------------
Initializing a standby can no longer cause gpinitsystem to exit early.
Added extra logging/output about standby master status.
Tell user at the end of gpinitsystem if gpinitstandby failed.
------------------------------------------
Signed-off-by: NMarbin Tan <mtan@pivotal.io>

We are hitting an issue where gpinitsystem sends a gpstop command that
has the data directory in all lower case. This causes an issue in
Linux, but is fine with macOS; linux platforms are case sensitive vs
macOS is not.

Remove the lower casing part of the code as it doesn't seem necessary.
gpinitsystem -c command eventually overwrites the SEG_PREFIX variable in a
different function to use the mixed case, while the gpinitsystem -I does not.

Fix: keep the mixed case segment names

gpinitsystem did not check for HEAP_CHECKSUM in the cluster
configuration file with a -c switch. This commit accepts the
HEAP_CHECKSUM setting, and additionally exports it to an
output_configuration_file when specified with the -O switch.

This commit also adds behave tests for the above, and for
reading the input_configuration_file with the -I switch.
Signed-off-by: NLarry Hamel <lhamel@pivotal.io>
Signed-off-by: NXin Zhang <xzhang@pivotal.io>

It's easy to lose log/error information if we print the usage when there is an
error with the cluster config and/or input config option.
Make it more direct by just showing the error to the user.

This is a new config option, on by default.
Signed-off-by: NMarbin Tan <mtan@pivotal.io>

    Added tests to make sure only one of them need to be specified for the
    gpinisystem to work.
Signed-off-by: NMarbin Tan <mtan@pivotal.io>
Signed-off-by: NC.J. Jameson <cjameson@pivotal.io>

- Sort arguments alphabetically
- Add -I and -O description

There is a lot of tab/space confusion in the code, this fixes some
of the worst offenders but there is lots of low-hanging fruit left.

There was a lot of unused code in the gpMgmt bash code that hasn't
been running for quite some time. On top of really dead code, some
codepaths were reachable, but useless (like printing a non-existing
version string). Remove the dead code and replace the version print
with working code which pulls the version from the canonical source.

Authors: Marbin Tan & Karen Huddleston

gp_dbid gp_num_contents_in_cluster gp_contentid
-b is being used upstream for binary upgrade mode.
The rest because of future conflict issues. The long options will always work

This will no longer spill logfiles from the demo cluster into
the users ~/gpAdminLogs directory. Also it makes it easier to
identify which logfile was created by the last regression test run.

Closes #689
Closes #523

since I get error when initsystem if the segment number >8
[FATAL]:-mismatch between content id and primary content id

I check that this is because, when matching mirror with primary has different order for some index.
e.g. I print the sort result:
primary is 0,10,11,12,13,1,14,15...
mirror is 0,10,11,12,13,14,1,15....

I believe it make more sense that we use sort -n , so I changed the code, and init system successfully.

Patch submitted by @byxiangfei