Use guc gp_role only now and replace the functionality of guc gp_session_role with it
also. Previously we have both gucs. The difference of the two gucs are (copied
from code comment):

 * gp_session_role
 *
 * - does not affect the operation of the backend, and
 * - does not change during the lifetime of PostgreSQL session.
 *
 * gp_role
 *
 * - determines the operating role of the backend, and
 * - may be changed by a superuser via the SET command.

This is not friendly for coding. For example, You could find Gp_role and
Gp_session_role are set as GP_ROLE_DISPATCH on Postmaster & many aux processes
on all nodes (even QE nodes) in a cluster, so you can see that to differ from
QD postmaster and QE postmaster, current gpdb uses an additional -E option in
postmaster arguments. These makes developers confusing when writing role branch
related code given we have three related variables.  Also some related code is
even buggy now (e.g. 'set gp_role' even FATAL quits).

With this patch we just have gp_role now. Some changes which might be
interesting in the patch are:

1. For postmaster, we should specify '-c gp_role=' (e.g. via pg_ctl argument) to
   determine the role else we assume the utility role.

2. For stand-alone backend, utility role is enforced (no need to specify by
   users).

3. Could still connect QE/QD nodes using utility mode with PGOPTIONS, etc as
   before.

4. Remove the '-E' gpdb hacking and align the '-E' usage with upstream.

5. Move pm_launch_walreceiver out of the fts related shmem given the later is
   not used on QE.
Reviewed-by: NBhuvnesh Chaudhary <bchaudhary@pivotal.io>
Reviewed-by: NGang Xiong <gxiong@pivotal.io>
Reviewed-by: NHao Wu <gfphoenix78@gmail.com>
Reviewed-by: NYandong Yao <yyao@pivotal.io>

Since the parse_long code is used for two options, the optionname
is passed as a parameter for error reporting. This was done through
automagic, but since it only applies to two callsites theres little
need for the complication, so hardcode the strings instead.
Reviewed-by: NJacob Champion <pchampion@pivotal.io>
Reviewed-by: NHeikki Linnakangas <hlinnakangas@pivotal.io>

This rearranges, and puts back, some code to align initdb with upstream
in cases where that can be done safely. One case of exit_nicely() was
decorated with break; in a switch statement, and that has been replaced
by exit(1) instead as the break served the purpose of avoiding compiler
warnings on fallthrough. This aligns us with what is happening in the
merge branch, as commit 3913a40f will
remove exit_nicely altogether.
Reviewed-by: NJacob Champion <pchampion@pivotal.io>
Reviewed-by: NHeikki Linnakangas <hlinnakangas@pivotal.io>

For some parts of postgresql.conf management, separate editing code
had been added in add_assignment where upstream otherwise has been
using replace_token. The main difference is that add_assignment
preserves the original line as a comment, whereas replace_token
replace the line altogether. Considering this is changing a brand
new file from the defaults, it hardly seems useful to carry that
diff, so remove add_assignment and instead use replace_token for
all config file editing.
Reviewed-by: NJacob Champion <pchampion@pivotal.io>
Reviewed-by: NHeikki Linnakangas <hlinnakangas@pivotal.io>

We had over time accumulated a few unused remains of hacks gone past.
Versionchecking is no longer done with a string comparison on a hard-
coded string, and the mirror creation support was already half-removed
with only dead code remaining (which in turn caused quite a diff to
upstream for no benefit). Remove it all.
Reviewed-by: NJacob Champion <pchampion@pivotal.io>
Reviewed-by: NHeikki Linnakangas <hlinnakangas@pivotal.io>

The --backend_output parameter was used to redirect output from initdb
executing the backend during probing to a separate logfile.  This log
was then appended to the main initdb log in case of errors.  Blindly
passing user parameters to system() is however problematic, as it will
execute any arbitrary commands passed.  The below example will redir
the output and also execute echo.

  ./bin/initdb -D data --backend_output="/tmp/out.log\" 2>&1; echo \"a"

There is no privilege escalation in initdb, so there is no security
angle to this, but it's still a less than desirable capability. Fix by
removing the parameter altogether (this feature does not exist in
upstream).
Reviewed-by: NJacob Champion <pchampion@pivotal.io>

GCC 7+ has a compiling option, -Wimplicit-fallthrough, which will
generate a warning/error if the code falls through cases(or default)
implicitly. The implicity may cause some bugs that are hardly to catch.

To enable this compile option, all switch-cases are not allowed to
fall through implicitly. To accomplish this target, 2 steps are done:
1. Append a comment line /* fallthrough */ or like at the end of case block.
2. Add break clause at the end of case block, if the last statement is
  ereport(ERROR) or like.

When -Wimplicit-fallthrough is enabled, -Werror=implicit-fallthrough
is also set. So fallthrough must be done explicitly.
Re-generate configure after modifying configure.in

No implicit-fallthrough for gpmapreduce and orafce
Reviewed-by: NHeikki Linnakangas <hlinnakangas@pivotal.io>
Reviewed-by: NGeorgios Kokolatos <gkokolatos@pivotal.io>
Reviewed-by: NPaul Guo <pguo@pivotal.io>

FileRep was replaced a long time ago, but a few leftovers had remained
in the codebase (mainly in comments and out-of-common-codepath places).
Fix by removing.
Reviewed-by: NAshwin Agrawal <aagrawal@pivotal.io>

Currently, gp_dbid and gp_contentid is passed as command line
arguments for starting QD and QE. Since, the values are stored in
master's catalog table, to get the right values, must start the master
first. Hence, hard-coded dbid=1 was used for starting the master in
admin mode always. This worked fine till dbid was not used for
anything on-disk. But given dbid is used for tablespace path in GPDB
6, startting the instance with wrong dbid, means inviting recovery
time failues, data corruption or data loss situations. Dbid=1 will go
wrong after failover to standby master as it has dbid != 1. This
commit hence eliminate the need of passing the gp_dbid and
gp_contentid on command line, instead while creating the instance the
values are stored in conf files for the instance.

This also helps to avoid passing gp_dbid as argument to pg_rewind,
which needs to start target instance in single user mode to complete
recovery before performing rewind operation.

Plus, this eases during development to just use pg_ctl start and not
require to correctly pass these values.

 - gp_contentid is stored in postgresql.conf file.

 - gp_dbid is stored in internal.auto.conf.

 - Introduce internal.auto.conf file created during
   initdb. internal.auto.conf is included from postgresql.conf file.

 - Separate file is chosen to write gp_dbid for ease handling during
   pg_rewind and pg_basebackup, as can exclude copying this file from
   primary to mirror, instead of trying to edit the contents of the
   same after copy during these operations. gp_contentid remains same
   for primary and mirror hence having it in postgresql.conf file
   makes senes. If gp_contentid is also stored in this new file
   internal.auto.conf then pg_basebackup needs to be passed contentid
   as well to write to this file.

 - pg_basebackup: write the gp_dbid after backup. Since, gp_dbid is
   unique for primary and mirror, pg_basebackup excludes copying
   internal.auto.conf file storing the gp_dbid. pg_basebackup explicit
   (over)writes the file with value passed as
   --target-gp-dbid. --target-gp-dbid due to this is mandatory
   argument to pg_basebackup now.

 - gpexpand: update gp_dbid and gp_contentid post directory copy.

 - pg_upgrade: retain all configuration files for
   segment. postgresql.auto.conf and internal.auto.conf are also
   internal configuration files which should be restored back after
   directory copy. Similar, change is required in gp_upgrade repo in
   restoreSegmentFiles() after copyMasterDirOverSegment().

 - Update tests to avoid passing gp_dbid and gp_contentid.
Co-authored-by: NAlexandra Wang <lewang@pivotal.io>

Also delete the unittest for set_gp_replication_config
Remove GpReplicationConfigFileLock
Co-authored-by: NJesse Zhang <jzhang@pivotal.io>
Co-authored-by: NShaoqi Bai <sbai@pivotal.io>

This removes a difference between the standard IsUnderPostmaster
execution environment and that of --boot and --single.  In a stand-alone
backend, "SELECT random()" always started at the same seed.

On a system capable of using posix shared memory, initdb could still
conclude "selecting dynamic shared memory implementation ... sysv".
Crashed --boot or --single postgres processes orphaned shared memory
objects having names that collided with the not-actually-random names
that initdb probed.  The sysv fallback appeared after ten crashes of
--boot or --single postgres.  Since --boot and --single are rare in
production use, systems used for PostgreSQL development are the
principal candidate to notice this symptom.

Back-patch to 9.3 (all supported versions).  PostgreSQL 9.4 introduced
dynamic shared memory, but 9.3 does share the "SELECT random()" problem.

Reviewed by Tom Lane and Kyotaro HORIGUCHI.

Discussion: https://postgr.es/m/20180915221546.GA3159382@rfd.leadboat.com

This reverts commit 4750e1b6.

This is the final batch of commits from PostgreSQL 9.2 development,
up to the point where the REL9_2_STABLE branch was created, and 9.3
development started on the PostgreSQL master branch.

Notable upstream changes:

* Index-only scan was included in the batch of upstream commits. It
  allows queries to retrieve data only from indexes, avoiding heap access.

* Group commit was added to work effectively under heavy load. Previously,
  batching of commits became ineffective as the write workload increased,
  because of internal lock contention.

* A new fast-path lock mechanism was added to reduce the overhead of
  taking and releasing certain types of locks which are taken and released
  very frequently but rarely conflict.

* The new "parameterized path" mechanism was added. It allows inner index
  scans to use values from relations that are more than one join level up
  from the scan. This can greatly improve performance in situations where
  semantic restrictions (such as outer joins) limit the allowed join orderings.

* SP-GiST (Space-Partitioned GiST) index access method was added to support
  unbalanced partitioned search structures. For suitable problems, SP-GiST can
  be faster than GiST in both index build time and search time.

* Checkpoints now are performed by a dedicated background process. Formerly
  the background writer did both dirty-page writing and checkpointing. Separating
  this into two processes allows each goal to be accomplished more predictably.

* Custom plan was supported for specific parameter values even when using
  prepared statements.

* API for FDW was improved to provide multiple access "paths" for their tables,
  allowing more flexibility in join planning.

* Security_barrier option was added for views to prevents optimizations that
  might allow view-protected data to be exposed to users.

* Range data type was added to store a lower and upper bound belonging to its
  base data type.

* CTAS (CREATE TABLE AS/SELECT INTO) is now treated as utility statement. The
  SELECT query is planned during the execution of the utility. To conform to
  this change, GPDB executes the utility statement only on QD and dispatches
  the plan of the SELECT query to QEs.
Co-authored-by: NAdam Lee <ali@pivotal.io>
Co-authored-by: NAlexandra Wang <lewang@pivotal.io>
Co-authored-by: NAshwin Agrawal <aagrawal@pivotal.io>
Co-authored-by: NAsim R P <apraveen@pivotal.io>
Co-authored-by: NDaniel Gustafsson <dgustafsson@pivotal.io>
Co-authored-by: NGang Xiong <gxiong@pivotal.io>
Co-authored-by: NHaozhou Wang <hawang@pivotal.io>
Co-authored-by: NHeikki Linnakangas <hlinnakangas@pivotal.io>
Co-authored-by: NJesse Zhang <sbjesse@gmail.com>
Co-authored-by: NJinbao Chen <jinchen@pivotal.io>
Co-authored-by: NJoao Pereira <jdealmeidapereira@pivotal.io>
Co-authored-by: NMelanie Plageman <mplageman@pivotal.io>
Co-authored-by: NPaul Guo <paulguo@gmail.com>
Co-authored-by: NRichard Guo <guofenglinux@gmail.com>
Co-authored-by: NShujie Zhang <shzhang@pivotal.io>
Co-authored-by: NTaylor Vesely <tvesely@pivotal.io>
Co-authored-by: NZhenghua Lyu <zlv@pivotal.io>

Make CREATE COLLATION and pg_import_system_collations() parallel aware by
dispatching collation creation to the QEs.

In order for collations to work correctly, we need to be sure that every
collation that is created on the QD is also installed on the QEs, and that the
OID matches in every database. We will take advantage of two phase commit to
prevent collations from being created if there is a problem adding it on any
QE. In upstream, collations are created during initdb, but this won't work for
GPDB, because while initdb is running there is no way to be sure that every
segment has the same locales installed.

We disable collation creation during initdb, and make it the responsibility of
the system administrator to initialize any needed collations by either running
a CREATE COLLATION command, or running the pg_import_system_collations() UDF.
Co-authored-by: NJim Doty <jdoty@pivotal.io>

Pull in more recent version of pg_import_system_collations() from
upstream. We have not pulled in the ICU collations, so wholesale
remove the sections of code that deal with them.

This commit is primarily a cherry-pick of 0b13b2a7, but also pulls
in prerequisite changes for CollationCreate().

	Rethink behavior of pg_import_system_collations().

	Marco Atzeri reported that initdb would fail if "locale -a" reported
	the same locale name more than once.  All previous versions of Postgres
	implicitly de-duplicated the results of "locale -a", but the rewrite
	to move the collation import logic into C had lost that property.
	It had also lost the property that locale names matching built-in
	collation names were silently ignored.

	The simplest way to fix this is to make initdb run the function in
	if-not-exists mode, which means that there's no real use-case for
	non if-not-exists mode; we might as well just drop the boolean argument
	and simplify the function's definition to be "add any collations not
	already known".  This change also gets rid of some odd corner cases
	caused by the fact that aliases were added in if-not-exists mode even
	if the function argument said otherwise.

	While at it, adjust the behavior so that pg_import_system_collations()
	doesn't spew "collation foo already exists, skipping" messages during a
	re-run; that's completely unhelpful, especially since there are often
	hundreds of them.  And make it return a count of the number of collations
	it did add, which seems like it might be helpful.

	Also, re-integrate the previous coding's property that it would make a
	deterministic selection of which alias to use if there were conflicting
	possibilities.  This would only come into play if "locale -a" reports
	multiple equivalent locale names, say "de_DE.utf8" and "de_DE.UTF-8",
	but that hardly seems out of the question.

	In passing, fix incorrect behavior in pg_import_system_collations()'s
	ICU code path: it neglected CommandCounterIncrement, which would result
	in failures if ICU returns duplicate names, and it would try to create
	comments even if a new collation hadn't been created.

	Also, reorder operations in initdb so that the 'ucs_basic' collation
	is created before calling pg_import_system_collations() not after.
	This prevents a failure if "locale -a" were to report a locale named
	that.  There's no reason to think that that ever happens in the wild,
	but the old coding would have survived it, so let's be equally robust.

	Discussion: https://postgr.es/m/20c74bc3-d6ca-243d-1bbc-12f17fa4fe9a@gmail.com
	(cherry picked from commit 0b13b2a7)

Move this logic out of initdb into a user-callable function.  This
simplifies the code and makes it possible to update the standard
collations later on if additional operating system collations appear.
Reviewed-by: NAndres Freund <andres@anarazel.de>
Reviewed-by: NEuler Taveira <euler@timbira.com.br>
(cherry picked from commit aa17c06f)

Commit ba5cfa93 introduced the upstream
code for managing timezones, but it failed to include the the below diff
to initdb for moving timezone handling from the postmaster to initdb.
This is a partial backport of the below commit, some hunks didn't apply
because due to requiring code introduced in PostgreSQL 9.1 and 9.2.

  commit ca4af308
  Author: Tom Lane <tgl@sss.pgh.pa.us>
  Date:   Fri Sep 9 17:59:11 2011 -0400

    Simplify handling of the timezone GUC by making initdb choose the default.

    We were doing some amazingly complicated things in order to avoid running
    the very expensive identify_system_timezone() procedure during GUC
    initialization.  But there is an obvious fix for that, which is to do it
    once during initdb and have initdb install the system-specific default into
    postgresql.conf, as it already does for most other GUC variables that need
    system-environment-dependent defaults.  This means that the timezone (and
    log_timezone) settings no longer have any magic behavior in the server.
    Per discussion.

This is a backport upstream commit 22817041, from PostgreSQL 9.0, which
added the server version number in the path. But in GPDB, also include the
gp_dbid in the path. This makes it possible to use the same tablespace path
on multiple servers running on the same host, without clashing.

Also includes cherry-pick of the small upstream cleanup commits 5c82ccb1
a6f56efc, and c282b36d.

Re-enable upstream 'tablespace' regression test. It works now, even when
all the nodes are running on same host.

And the mechanism in initdb and gpinitsystem to set it. It's no longer
used for anything.

Revert the code to open/read/write regular files, to the way it's in the
upstream.

WAL replication is the name of the game on this branch.

* Revert almost all the changes in smgr.c / md.c, to not go through
  the Mirrored* APIs.

* Remove mmxlog stuff. Use upstream "pending relation deletion" code
  instead.

* Get rid of multiple startup passes. Now it's just a single pass like
  in the upstream.

* Revert the way database drop/create are handled to the way it is in
  upstream. Doesn't use PT anymore, but accesses file system directly,
  and WAL-logs a single CREATE/DROP DATABASE WAL record.

* Get rid of MirroredLock

* Remove a few tests that were specific to persistent tables.

* Plus a lot of little removals and reverts to upstream code.

The intend of this extra configuration file is to control the
synchronization between primary and mirror for WALREP.

The gp_replication.conf is not designed to work with filerep, for
example, the scripts like gp_expand will fail since it directly modify
the configuration files instead of going through initdb.
Signed-off-by: NXin Zhang <xzhang@pivotal.io>

Signed-off-by: NXin Zhang <xzhang@pivotal.io>

We use this file to store the GUC value `sychronous_standby_names` in
order to control the blocking behavior between primary and mirrors as
used in upstream. When this GUC is on, the primary is blocked and
waiting for the commits propagated to mirrors regardless of mirror
status. When this GUC is off, primary just archive and won't wait for
mirrors.

gp_replication.conf is now read unconditionally by the GUC parsing logic
and needs to be set up by initdb. Refactor set_null_conf() to take a
filename so that we don't copy-paste more code.
Signed-off-by: NJacob Champion <pchampion@pivotal.io>

add_assignment previously didn't handle the case where an existing
config assignment was not found, and segfaulted. Fix that by inserting
brand-new assignments at the end of the config list.

Checksums are set immediately prior to flush out of shared buffers
and checked when pages are read in again. Hint bit setting will
require full page write when block is dirtied, which causes various
infrastructure changes. Extensive comments, docs and README.

WARNING message thrown if checksum fails on non-all zeroes page;
ERROR thrown but can be disabled with ignore_checksum_failure = on.

Feature enabled by an initdb option, since transition from option off
to option on is long and complex and has not yet been implemented.
Default is not to use checksums.

Checksum used is WAL CRC-32 truncated to 16-bits.

Simon Riggs, Jeff Davis, Greg Smith
Wide input and assistance from many community members. Thank you.

(cherry picked from commit 96ef3b8f)

This addresses the new warning types -Wformat-truncation
-Wformat-overflow that are part of -Wall, via -Wformat, in GCC 7.

We used to printed an "ok" after loading each SQL script from the
cdb_init.d directory. However, we only printed one "creating Greenplum
Database schema ..." line. That leads to funny output:

  creating information schema ... ok
  creating Greenplum Database schema ... ok
  ok
  ok
  vacuuming database template1 ... ok
  copying template1 to template0 ... ok
  copying template1 to postgres ... ok

To fix, only print one "ok", after loading all the scripts from cdb_init.d.

In many places where we had used a mixture of spaces and tabs for
indentation, new versions of gcc complained about misleading indentation,
because gcc doesn't know we're using tab width of 4. To fix, make the
indentation consistent in all the places where gcc gave a warning. Would
be nice to fix it all around, but that's a lot of work, so let's do it
in a piecemeal fashion whenever we run into issues or need to modify a
piece of code anyway.

For some files, especially the GPDB-specific ones, I ran pgindent over
the whole file. I used the pgindent from PostgreSQL master, which is
slightly different from what was used back 8.3 days, but that's what I had
easily available, and that's what we're heading to in the future anyway.
In some cases, I didn't commit the pgindented result if there were
funnily formatted code or comments that would need special treatment.

For other places, I fixed the indentation locally, just enough to make the
warnings go away.

I also did a tiny bit of other trivial cleanup, that I happened to spot
while working on this, although I tried to refrain from anything more
extensive.

Dump out the appropriate GRANT/REVOKE commands for databases and
tablespaces from pg_dumpall to replicate what the current state is.

This was broken during the changes to buildACLCommands for 9.6+
servers for pg_init_privs.

This was overlooked in commit 473b9328, which introduced DROP ACCESS
METHOD.  Although that command is restricted to superusers, we don't want
even superusers dropping the built-in methods; "DROP ACCESS METHOD btree"
in particular is unrecoverable from.  Pin these objects in the same way
that other initdb-created objects are pinned.

I chose to bump catversion for this fix.  That's not absolutely necessary
perhaps, but it will ensure that no 9.6 production systems are missing
the pin entries.

Use disallowed instead of reserved, cannot instead of can not, and
double quotes instead of single quotes.

Also add a test to cover the bug which started this discussion.

Per discussion with Tom.

As with CREATE ROLE, disallow users from specifying initial
superuser names which begin with 'pg_' in initdb.

Per discussion with Tom.

Create a "bsd" auth method that works the same as "password" so far as
clients are concerned, but calls the BSD Authentication service to
check the password.  This is currently only available on OpenBSD.

Marisa Emerson, reviewed by Thomas Munro

Now that all of the infrastructure exists, add in the ability to
dump out the ACLs of the objects inside of pg_catalog or the ACLs
for objects which are members of extensions, but only if they have
been changed from their original values.

The original values are tracked in pg_init_privs.  When pg_dump'ing
9.6-and-above databases, we will dump out the ACLs for all objects
in pg_catalog and the ACLs for all extension members, where the ACL
has been changed from the original value which was set during either
initdb or CREATE EXTENSION.

This should not change dumps against pre-9.6 databases.

Reviews by Alexander Korotkov, Jose Luis Tallon