To create mirrors, pg_basebackup needs to be performed. pg_basebackup
to correctly handle tablespaces needs dbid as argument. This
requirement exist because dbid is used in tablespace path.

dbid in master catalog to be in sync with what's used by mirror for
tablespace, need to add mirror to catalog first. Get the dbid and pass
the same to pg_basebackup for creating mirror.
Co-authored-by: NAlexandra Wang <lewang@pivotal.io>

Given the online gpexpand work, the gp_num_contents_in_cluster GUC is
unused. So, delete the same from code to avoid confusions and eliminate this
long argument required to start a postgres instance in gpdb.

Authored-by: NGoutam Tadi <gtadi@pivotal.io>

[#159742200]
Co-authored-by: NXin Zhang <xzhang@pivotal.io>
Co-authored-by: NGoutam Tadi <gtadi@pivotal.io>

Add behave test for FQDN_HBA flag support
Co-authored-by: NXin Zhang <xzhang@pivotal.io>
Co-authored-by: NGoutam Tadi <gtadi@pivotal.io>

- Behave tests for gpinitsystem with fqdn
Co-authored-by: NGoutam Tadi <gtadi@pivotal.io>
Co-authored-by: NJamie McAtamney <jmcatamney@pivotal.io>

There are a lot of blank lines in the 'gpinitsystem' log. These are
read directly from the configuration file using commands 'cat'. The
comments are turned into blank lines, but they are not removed:

  gpinitsystem:node:gp6-[INFO]:-Start Main
  gpinitsystem:node:gp6-[INFO]:-Command line options passed to utility = -c ../gpinitsystem_config
  gpinitsystem:node:gp6-[INFO]:-Start Function CHK_GPDB_ID
  ...
  gpinitsystem:node:gp6-[INFO]:-End Function CHK_FILE
  gpinitsystem:node:gp6-[INFO]:-Dumping gpinitsystem_config to logfile for reference

  ARRAY_NAME="EMC Greenplum DW"

  SEG_PREFIX=gpseg

  PORT_BASE=40300

This extends the exclusion regex used when appending to the logfile
to remove blank lines completely and to handle comments that doesn't
start on column zero.
Co-authored-by: NDaniel Gustafsson <dgustafsson@pivotal.io>
Reviewed-by: NDaniel Gustafsson <dgustafsson@pivotal.io>

Rather than hardcoding to require /bin/bash, move to using a lookup
via "/usr/bin/env bash" to allow for greater portability of the code.
This also changes the Bash test to checking if the current shell is
actually Bash, rather than looking if bash is available on the file
system (since we by the above mentioned changes no longer need that).

This patch helps cut-down 8 secs out of 30 secs (for single node gpinitsystem as
part of gpdemo), mostly more for multinode setups.

- Removes explicit 2 sec and 1 sleep.
- Removes explicit call to CHECKPOINT, not required.
- Avoid starting primaries after initdb. As they were started and then shutdown
  pretty soon to start back cluster with required cmdline arguments.
- Add -w to mirror start.

Still more room for improvement and speedup, this just gets us started.

listen_addresses in postgresql.conf doesn't taken effect now, backend and
postmaster are listening on all addresses. From the point of security, we
should be able to let user specify the listen address.

Revert the state machine and other logic in postmaster.c the way it is in
upstream. Remove some GUCs related to mirrored and non-mirrored mode. Remove
the -M, -x and -y postmaster options, and change management scripts to not
pass those options.

It just doesn't mean "filespace" anymore, just plain "datadir".

Remove the concept of filespaces, revert tablespaces to work the same as
in upstream.

There is some leftovers in management tools. I don't know how to test all
that, and I was afraid of touching things I can't run. Also, we may need
to create replacements for some of those things on top of tablespaces, to
make the management of tablespaces easier, and it might be easier to modify
the existing tools than write them from scratch. (Yeah, you could always
look at the git history, but still.)

Per the discussion on gpdb-dev mailing list, the plan is to cherry-pick
commit 16d8e594 from PostgreSQL 9.2, to make it possible to have a
different path for a tablespace in the primary and its mirror. But that's
not included in this commit yet.

TODO: Make temp_tablespaces work.
TODO: Make pg_dump do something sensible, when dumping from a GPDB 5 cluster
that uses filespaces. Same with pg_upgrade.

Discussion: https://groups.google.com/a/greenplum.org/d/msg/gpdb-dev/sON4lraPEqg/v3lkM587BAAJ

With file replication gone, gpinitsystem should no longer try to
initialize the cluster through filerep sequence.

The sequence now goes as follows:
1. Create and start master in master-only mode
2. Create primaries and register to master
3. Stop master.
4. Run gpstart to start master and primaries.
5. Create mirrors w/ pg_basebackup and register to master.
6. Start the mirrors and wait until primaries and mirrors sync.

Author: Jimmy Yih <jyih@pivotal.io>
Author: Marbin Tan <mtan@pivotal.io>

And the mechanism in initdb and gpinitsystem to set it. It's no longer
used for anything.

* Update gpinitsystem

edit commit message: Marbin Tan<mtan@pivotal.io>
Change the symbol to actual words to better convey the meaning.
Primary segment # -> Number of primary segments
Mirror segment # -> Number of mirror segments

Originally, gp_add_segment() is only used to add a primary segment. We
created a new gp_add_segment_primary() to keep the original
functionality, which automatically generate dbid and contentid.

We generalized the gp_add_segment() to be able to directly update
gp_segment_configuration and pg_filespace_entry for adding any type of
segments with full specification of segment and filespace mappings. In
this case, the new gp_add_segment() doesn't generate dbid and contentid
automatically, and rely on input parameters.

Originally there is separate code path in gp_add_segment_mirror() to
figure out primary dbid, such logic is actually common even with
gp_add_segment(), which can add both primary and mirror.

In that case, we refactor the primary dbid detection logic in common
function add_segment(), and refactor the gp_add_segment_mirror() to use
the add_segment() instead of add_segment_config_entry().

We update gpinitsystem to use the function gp_add_segment() instead of
update the gp_segment_configuration and pg_filespace_entry tables
directly via SQL.
Signed-off-by: NXin Zhang <xzhang@pivotal.io>

Using gp_segment_configuration catalog table easily can find if mirrors exist or
not, do not need special table to communicate the same. Earlier
gp_fault_strategy used to convey 'n' for mirrorless system, 'f' for replication
and 's' for san mirrors. Since support for 's' was removed in 5.0 only purpose
gp_fault_strategy served was mirrored or not mirrored system. Hence deleting the
gp_fault_strategy table and at required places using gp_segment_configuration to
find the required info.

Previously, during gpinitsystem, the standby was instantiated in the middle of
setting up the master. This ordering caused problems because
initializing the standby could cause an exit when an error
occurred. As a result of this early exit, the gp_toolkit and DCA gucs
were not set properly.

Instead, initialize the standby after the master is finished.
------------------------------------------
Previously the exit return code for gpinitsystem was always non-zero.
Now, it is non-zero only in an error or warning case.

The issue was due to SCAN_LOG interpretation of an empty string as a line count of one.
Fixed by changing to word count.
------------------------------------------
Initializing a standby can no longer cause gpinitsystem to exit early.
Added extra logging/output about standby master status.
Tell user at the end of gpinitsystem if gpinitstandby failed.
------------------------------------------
Signed-off-by: NMarbin Tan <mtan@pivotal.io>

We are hitting an issue where gpinitsystem sends a gpstop command that
has the data directory in all lower case. This causes an issue in
Linux, but is fine with macOS; linux platforms are case sensitive vs
macOS is not.

Remove the lower casing part of the code as it doesn't seem necessary.
gpinitsystem -c command eventually overwrites the SEG_PREFIX variable in a
different function to use the mixed case, while the gpinitsystem -I does not.

Fix: keep the mixed case segment names

gpinitsystem did not check for HEAP_CHECKSUM in the cluster
configuration file with a -c switch. This commit accepts the
HEAP_CHECKSUM setting, and additionally exports it to an
output_configuration_file when specified with the -O switch.

This commit also adds behave tests for the above, and for
reading the input_configuration_file with the -I switch.
Signed-off-by: NLarry Hamel <lhamel@pivotal.io>
Signed-off-by: NXin Zhang <xzhang@pivotal.io>

It's easy to lose log/error information if we print the usage when there is an
error with the cluster config and/or input config option.
Make it more direct by just showing the error to the user.

This is a new config option, on by default.
Signed-off-by: NMarbin Tan <mtan@pivotal.io>

    Added tests to make sure only one of them need to be specified for the
    gpinisystem to work.
Signed-off-by: NMarbin Tan <mtan@pivotal.io>
Signed-off-by: NC.J. Jameson <cjameson@pivotal.io>

- Sort arguments alphabetically
- Add -I and -O description

There is a lot of tab/space confusion in the code, this fixes some
of the worst offenders but there is lots of low-hanging fruit left.

There was a lot of unused code in the gpMgmt bash code that hasn't
been running for quite some time. On top of really dead code, some
codepaths were reachable, but useless (like printing a non-existing
version string). Remove the dead code and replace the version print
with working code which pulls the version from the canonical source.

Authors: Marbin Tan & Karen Huddleston

gp_dbid gp_num_contents_in_cluster gp_contentid
-b is being used upstream for binary upgrade mode.
The rest because of future conflict issues. The long options will always work

This will no longer spill logfiles from the demo cluster into
the users ~/gpAdminLogs directory. Also it makes it easier to
identify which logfile was created by the last regression test run.

Closes #689
Closes #523

since I get error when initsystem if the segment number >8
[FATAL]:-mismatch between content id and primary content id

I check that this is because, when matching mirror with primary has different order for some index.
e.g. I print the sort result:
primary is 0,10,11,12,13,1,14,15...
mirror is 0,10,11,12,13,14,1,15....

I believe it make more sense that we use sort -n , so I changed the code, and init system successfully.

Patch submitted by @byxiangfei

This reverts commit a19e0e7e. I had misread
the code, thinking that GP_HOSTADDRESS is a constant throughout the script.
It's not; it's set by the SET_VAR function, so the lookups were all looking
up different addresses. I did not notice that while testing on a single
host.

I still think the "sleep 2" is unnecessary, but I'm now afraid of touching
this because I don't have a multi-host setup at hand to test with. So revert
the whole commit .

Spotted by Jimmy Yih.

* Remove an unnecessary sleep 2
* Perform the lookup of master hostname only once

The gpssh utility can have issues during the initial ssh connection
step due to different network configurations that may cause delay.
This commit exposes two key pxssh variables, delaybeforesend and
sync_multiplier, into configurables so that users can change them
as needed to ensure their gpssh works as intended in their network.

Authors: Christopher Hajas and Jimmy Yih

cleanup of syntax around inet4 v inet6 parsing to match

changing spacing in piped commands

fixing case semicolons

Adding freebsd IP detection commands

Use the standard initdb mechanism for it, instead of having bespoken code
in gpinitsystem.

Also remove the unnecessarily complicated logic to generate gp_toolkit.sql
from gp_toolkit.sql.in. All the JETPACK_* variables were constants, and
there is particular need in making the gp_toolkit functions relocatable to
different schemas or prefixes, so just hardcode them and remove the sed
magic.

Rewrite the file header of gp_toolkit.sql. The legal blurp is not needed,
make the header look more like that of information_schema.sql and
system_views.sql.

Remove compaction_info.sql. It was only needed for old in-place
upgrade from 4.3.x to 4.3.4.2, which is not relevant for master branch.

Fix the "jetpack" bugbuster regression test. It was originally written to
test the gp_toolkit views, but gp_toolkit was in fact not installed in
bugbuster, and the expected output just contained a bunch of "relation
does not exist" errors. Now it actually tests gp_toolkit again. Some of
the test queries didn't produce repeateable results, due to differing oids,
differences in table sizes, and other tables created by previous tests, so
fixed those too.

This has a small user-visible difference: gp_toolkit schema is now also
installed in template0. That caused the difference in the expected outputs:
pg_regress creates the "regression" database using template0 as the
template, so gp_toolkit used to not be installed in it, but it is now.