[Lockfile](https://pypi.org/project/lockfile/) has not been maintained
since around 2015. Further, the functionality it provided seems poor - a
review of the code indicated that it used the presence of the PID file
itself as the lock - in Unix, using a file's existence followed by a
creation is not atomic, so a lock could be prone to race conditions.

The lockfile package also did not clean up after itself - a process
which was destroyed unexpectedly would not clear the created locks, so
some faulty logic was added to mainUtils.py, which checked to see if a
process with the same PID as the lockfile's creator was running. This
is obviously failure prone, as a new process might be assigned the same
PID as the old lockfile's owner, without actually being the same process.

(Of note, the SIG_DFL argument to os.kill() is not a signal at all, but
rather of type signal.handler. It appears that the python cast this
handler to the int 0, which, according to man 2 kill, leads to no signal
being sent, but existance and permission checks are still performed. So
it is a happy accident that this code worked at all)

This commit removes lockfile from the codebase entirely.

It also adds a "PIDLockFile" class which provides an atomic-guarenteed
lock via the mkdir and rmdir commands on Unix - thus, it is not safely
portable to Windows, but this should not be an issue as only Unix-based
utilities use the "simple_main()" function.

PIDLockFile provides API compatible classes to replace most of the
functionality from lockfile.PidLockFile, but does remove any timeout
logic as it was not used in any meaningful sense - a hard-coded timeout
of 1 second was used, but an immediate result of if the lock is held is
sufficient.

PIDLockFile also includes appropriate __enter__, __exit__, and __del__
attributes, so that, should we extend this class in the future, with
syntax is functional, and __del__ calls release, so a process reaped
unexpectedly should still clean its own locks as part of the garbage
collection process.
Authored-by: NTyler Ramer <tramer@pivotal.io>

Due to refactor of dbconn and newer versions of pygresql, using
`with dbconn.connect() as conn` no longer attempts to close a
connection, even if it did prior. Instead, this syntax uses the
connection itself as context and, as noted in execSQL, overrides the
autocommit functionality of execSQL.

Therefore, close the connection manually to ensure that execSQL is
auto-commited, and the connection is closed.
Co-authored-by: NTyler Ramer <tramer@pivotal.io>
Co-authored-by: NJamie McAtamney <jmcatamney@pivotal.io>

One reason pygresql was previously modified was that it did not handle
closing a connection very gracefully. In the process of updating
pygresql, we've wrapped the connection it provides with a
ClosingConnection function, which should handle gracefully closing the
connection when the "with dbconn.connect as conn" syntax is used.

This did, however, illustrate issues where a cursor might have been
created as the result of a dbconn.execSQL() call, which seems to hold
the connection open if not specifically closed.

It is therefore necessary to remove the ability to get a cursor from
dbconn.execSQL(). To highlight this difference, and to ensure that
future calls to this library is easy to use, I've cleaned up and
clarified the dbconn execution code, to include the following features.

- dbconn.execSQL() closes the cursor as part of the function. It returns
  no rows
- functions dbconn.query() is added, which behaves like dbconn.execSQL()
  except that it now returns a cursor
- function dbconn.execQueryforSingleton() is renamed
  dconn.querySingleton()
- function dbconn.execQueryforSingletonRow() is renamed
  dconn.queryRow()
Authored-by: NTyler Ramer <tramer@pivotal.io>

This commit updates pygresql from 4.0.0 to 5.1.2, which requires
numerous changes to take advantages of the major result syntax change
that pygresql5 implemented. Of note, cursors or query objects
automatically cast returned values as appropriate python types - list of
ints, for example, instead of a string like "{1,2}". This is the bulk of
the changes.

Updating to pygresql 5.1.2 provides numerous benfits, including the
following:

- CVE-2018-1058 was addressed in pygresql 5.1.1

- We can save notices in the pgdb module, rather than relying on importing
the pg module, thanks to the new "set_notices()"

- pygresql 5 supports python3

- Thanks to a change in the cursor, using a "with" syntax guarentees a
  "commit" on the close of the with block.

This commit is a starting point for additional changes, including
refactoring the dbconn module.

Additionally, since isolation2 uses pygresql, some pl/python scripts
were updated, and isolation2 SQL output is further decoupled from
pygresql. The output of a psql command should be similar enough to
isolation2's pg output that minimal or no modification is needed to
ensure gpdiff can recognize the output.
Co-Authored-by: NTyler Ramer <tramer@pivotal.io>
Co-authored-by: NJamie McAtamney <jmcatamney@pivotal.io>

Use guc gp_role only now and replace the functionality of guc gp_session_role with it
also. Previously we have both gucs. The difference of the two gucs are (copied
from code comment):

 * gp_session_role
 *
 * - does not affect the operation of the backend, and
 * - does not change during the lifetime of PostgreSQL session.
 *
 * gp_role
 *
 * - determines the operating role of the backend, and
 * - may be changed by a superuser via the SET command.

This is not friendly for coding. For example, You could find Gp_role and
Gp_session_role are set as GP_ROLE_DISPATCH on Postmaster & many aux processes
on all nodes (even QE nodes) in a cluster, so you can see that to differ from
QD postmaster and QE postmaster, current gpdb uses an additional -E option in
postmaster arguments. These makes developers confusing when writing role branch
related code given we have three related variables.  Also some related code is
even buggy now (e.g. 'set gp_role' even FATAL quits).

With this patch we just have gp_role now. Some changes which might be
interesting in the patch are:

1. For postmaster, we should specify '-c gp_role=' (e.g. via pg_ctl argument) to
   determine the role else we assume the utility role.

2. For stand-alone backend, utility role is enforced (no need to specify by
   users).

3. Could still connect QE/QD nodes using utility mode with PGOPTIONS, etc as
   before.

4. Remove the '-E' gpdb hacking and align the '-E' usage with upstream.

5. Move pm_launch_walreceiver out of the fts related shmem given the later is
   not used on QE.
Reviewed-by: NBhuvnesh Chaudhary <bchaudhary@pivotal.io>
Reviewed-by: NGang Xiong <gxiong@pivotal.io>
Reviewed-by: NHao Wu <gfphoenix78@gmail.com>
Reviewed-by: NYandong Yao <yyao@pivotal.io>

When introducing a new mirror, we need two steps:
1. start mirror segment
2. update gp_segment_configuration catalog

Previously gp_add_segment_mirror will be called to update
the catalog, but dbid is chosen by get_availableDbId() which
cannot ensure to be the same dbid in internal.auto.conf.
Reported by issue9837 
Reviewed-by: NPaul Guo <pguo@pivotal.io>

This reverts commit 1d04cab0.

This reverts commit b0b0b958.

This reverts commit bebc77af.

It is no loner needed, the correct approach is to install meta-only
index files on the new segments.

Which was introduced to exclude a large amount of paths.

Also changed the excluding logic of './db_dumps' and './promote'.  They
were excluded only when an empty 'excludePaths' was specified by the
caller, this is weird, so I changed the logic to always exclude these
two paths.

Gpexpand creates new primary segments by first creating a template from
the master datadir and then copying it to the new segments.  Some
catalog tables are only meaningful on master, such as
gp_segment_configuration, their content are then cleared on each new
segment with the "delete from ..." commands.

This works but is slow because we have to include the content of the
master-only tables in the archive, distribute them via network, and
clear them via the slow "delete from ..." commands -- the "truncate"
command is fast but it is disallowed on catalog tables as filenode must
not be changed for catalog tables.

To make it faster we now exclude these tables from the template
directly, so less data are transferred and there is no need to "delete
from" them explicitly.

Removed the duplicated 'gp_segment_configuration' entry in the
MASTER_ONLY_TABLES list.  Also sort the list in alphabetic order to
prevent dulicates in the future.

For ETL user scenarios, there are some cases that may frequently create
and drop the same external table. And once the external table gets dropped.
All errors stored in the error log will lose.

To enable error log persistent for external with the same
"dbname"."namespace"."table".
Bring in "error_log_persistent" external table option. If create the
external table with `OPTIONS (error_log_persistent 'true')` and `LOG ERROR`,
the external's error log will be name as "dbid_namespaceid_tablename"
under "errlogpersistent" directory.
And drop external table will ignore to delete the error log.

Since GPDB 5, 6 still use pg_exttable's options to mark
LOG ERRORS PERSISTENTLY, so keep the ability for loading from
OPTIONS(error_log_persistent 'true').

Create separate `gp_read_persistent_error_log` function to read
persistent error log.
If the external table gets deleted, only the namespace owner
has permission to delete the error log.

Create separate `gp_truncate_persistent_error_log` function to delete
persistent error log.
If the external table gets deleted. Only the namespace owner has
permission to delete the error log.
It also supports wildcard input to delete error logs
belong to a database or whole cluster.

If drop an external table create with `error_log_persistent`. And then
create the same "dbname"."namespace"."table" external table without
persistent error log. It'll write errors to the normal error log.
The persistent error log still exists.
Reviewed-by: NHaozhouWang <hawang@pivotal.io>
Reviewed-by: NAdam Lee <ali@pivotal.io>

It was only used for one message in gprecoverseg, and it doesn't seem
important.

The second argument to the function didn't do anything, since the removal
of email and SNMP alerts in commit 65822b80. And the NULL checks for the
arguments were pointless, because the function was marked as strict. But
rather than clean those up, let's just remove it altogether.
Reviewed-by: NAsim R P <apraveen@pivotal.io>
Reviewed-by: NJimmy Yih <jyih@pivotal.io>

1. When two gppkg packages have the same dependencies, gppkg utility
   will refuse to install the second gppkg package and throw an error.
   This patch fixes this issue and the second gppkg package can install
   successfully.

2. Fix install/uninstall issue if the master and standby master use the
   same node address.

The <> comparison operator is not supported in Python 3, and has been
deprecated for quite some time already. The 2.7 documentation states:

	!= can also be written <>, but this is an obsolete usage kept for
	backwards compatibility only. New code should always use !=.

Fix by replacing the old operators with the syntax.
Reviewed-by: NDaniel Gustafsson <dgustafsson@pivotal.io>
Reviewed-by: NJacob Champion <pchampion@pivotal.io>

This fixes multiple occurrences of duplicated words in sentences, like
"the the" and "is is" etc.
Reviewed-by: NMel Kiyama <mkiyama@pivotal.io>
Reviewed-by: NHeikki Linnakangas <hlinnakangas@pivotal.io>

Incremental recovery and rebalance operations involve running
pg_rewind against failed primaries.  This patch changes gprecoverseg
such that pg_rewind is invoked in parallel, using the WorkerPool
interface, for each affected segment in the cluster.  There is no
reason to rewind segments one after the other.

Fixes Github issue #9466

Reviewed by: Mark Sliva and Paul Guo

Remove the gpperfmon since we have the alternative metrics collector.

1, remove all gpperfmon codes, including gpmon, gpsmon, gpmmon,
gpperfmon and alert_log.
2, remove all gpperfmon GUCs, `gp_enable_gpperfmon`, `gp_perfmon_segment_interval`,
`gp_perfmon_print_packet_info`, `gpperfmon_port` and `gpperfmon_log_alert_level`.
3, remove the `perfmon` and `stats sender` processes/bgworkers.
4, remove the `apu` and `libsigar` dependencies.

Previously, gpstart could not start the cluster if a standby master host
was configured but currently down.  In order to check whether the
standby was supposed to be the acting master (and prevent the master
from being started if that was the case), gpstart needed to access the
standby host to retrieve the TimeLineID of the standby, and if the
standby host was down the master would not start.

This commit modifies gpstart to assume that the master host is the
acting master if the standby is unreachable, so that it never gets into
a state where neither the master nor the standby can be started.
Co-authored-by: NKalen Krempely <kkrempely@pivotal.io>
Co-authored-by: NMark Sliva <msliva@pivotal.io>
Co-authored-by: NAdam Berlin <aberlin@pivotal.io>

gpcheckcat hard-coded master dbid to 1 for various queries. This
assumption is flawed. There is no restriction master can only have dbid
1, it can be any value. For example, failover to standby and gpcheckat
is not usable with that assumption.

Hence, run-time find the value of master's dbid using the info that
it's content-id is always -1 and use the same.
Co-authored-by: NAlexandra Wang <lewang@pivotal.io>

This reverts commit e046859f.

Removed the duplicated 'gp_segment_configuration' entry in the
MASTER_ONLY_TABLES list.  Also sort the list in alphabetic order to
prevent dulicates in the future.

We update the pg_hba.conf file with replication entries for each
hostname/address to enable cross-subnet cluster expansion. There are no tests
for this change, but they can be added at a later time.
Co-authored-by: NJacob Champion <pchampion@pivotal.io>
Co-authored-by: NAdam Berlin <aberlin@pivotal.io>
Co-authored-by: NBhuvnesh Chaudhary <bchaudhary@pivotal.io>
Co-authored-by: NKalen Krempely <kkrempely@pivotal.io>
Co-authored-by: NDavid Krieger <dkrieger@pivotal.io>
Co-authored-by: NJamie McAtamney <jmcatamney@pivotal.io>

The four CM utilities gpinitsystem, gpinitstandby, gpaddmirrors, and
gpmovemirrors now have the relevant pg_hba.conf entries to allow WAL
replication to mirrors from their respective primaries across subnets.

There are two parts to this commit:
1). modify the CM utilities to add the pg_hba.conf entries to
allow WAL replication to mirrors across a subnet.
2). test the relevant CM utilities across subnets

The previous pg_hba.conf replication entry:
    'host replication $USER samenet trust'
does not allow WAL replication connections across subnets. We keep this entry
in order to support single-host development. We then add one replication line
for each primary and mirror interface address to new primaries and mirrors to
allow this. It looks like:
    'host replication $USER $IP_ADDRESS trust'
    or when HBA_HOSTNAMES=1
    'host replication $USER $HOSTNAME trust'
Further, if there is ever a failover and subsequent promotion,
replication connections can be made to the newly promoted primary from the host
on which the previous primary failed, because those addresses get copied over
to the new mirror during a pg_base_backup. We also add similar logic to support
cross-subnet replication between the master and standby. This behavior is
tested in the cross_subnet behave tests.

The cross_subnet behave tests assert that the replication connection is valid
by manually making the connection in addition to relying on segments being
synchronized, as a way to ensure that the pg_hba.conf file is being used.
Co-authored-by: NJacob Champion <pchampion@pivotal.io>
Co-authored-by: NAdam Berlin <aberlin@pivotal.io>
Co-authored-by: NBhuvnesh Chaudhary <bchaudhary@pivotal.io>
Co-authored-by: NKalen Krempely <kkrempely@pivotal.io>
Co-authored-by: NDavid Krieger <dkrieger@pivotal.io>
Co-authored-by: NJamie McAtamney <jmcatamney@pivotal.io>

gpdeletesystem uses GpDirsExist() to check if dump directories are
present to warn and avoid deleting the cluster. Only if "-f" option is
used allowed to delete the cluster with dump directories
present. Though this function incorrectly checks for files and
directories with name "*dump*" and not just directories.

So, gpdeletesystem started failing after commit eb036ac1. FTS writes
file with name of file as `gpsegconfig_dump`. GpDirsExist()
incorrectly reports this as backup directory present and fails. Fix
the same by only checking for directories and not files.

Fixes https://github.com/greenplum-db/gpdb/issues/8442Reviewed-by: NAsim R P <apraveen@pivotal.io>

This helps script handling by checking return values.
Reviewed-by: NAsim R P <apraveen@pivotal.io>

Follow-up to PR #8956 and its commit 12bb8165, which addressed gpstart's
failure to correctly display ignored segments. Add both a unit test for
the buggy implementation and an end-to-end test for the behavior itself.

This adds a new gpstart job to the CI pipeline.
Co-authored-by: NJacob Champion <pchampion@pivotal.io>

* Ship modified python module subprocess32 again

subprocess32 is preferred over subprocess according to python documentation.
In addition we long ago modified the code to use vfork() against fork() to
avoid some "Cannot allocate memory" kind of error (false alarm though - memory
is actually sufficient) on gpdb product environment that is usually with memory
overcommit disabled.  And we compiled and shipped it also but later it was just
compiled but not shipped somehow due to makefile change (maybe a regression).
Let's ship it again.

* Replace subprocess with our own subprocess32 in python code.

* Fix gpexpand entries in pg_hba.conf

Append into pg_hba configs of expanded segments the addresses of the
primary and mirror segments. The hba config is copied from segment 0.
We must update it with the new primary/mirror pair information otherwise
connections between the systems will fail. This connection is necessary
during pg_rewind.
Co-authored-by: NBhuvnesh Chaudhary <bhuvnesh2703@gmail.com>
Co-authored-by: NJesse Zhang <sbjesse@gmail.com>

Primary nodes should allow connections from its corresponding mirror
when pg_rewind tries to connect to it. This commit ensure that when
mirrors are added using gpaddmirrors the mirror entries are updated on
the primary pg_hba which is subsequently copied to the mirror node. This
allows connection between primary and mirror when the roles have
switched over.
Co-authored-by: NDavid Kimura <dkimura@pivotal.io>

Authored-by: NBradford D. Boyle <bboyle@pivotal.io>

As a continued follow-up in the CVE-2018-1058, ensure that the utilities
that connect using PyGreSQL primitives also have search_path protection.

We set an empty search_path for the connection in all cases except for
segment_reconfigurer.py, where a fully-qualified SELECT statement seemed
sufficient (there is only one query being performed on the connection,
and that seems unlikely to change).
Co-authored-by: NJamie McAtamney <jmcatamney@pivotal.io>
Co-authored-by: NJacob Champion <pchampion@pivotal.io>
Co-authored-by: NShoaib Lari <slari@pivotal.io>

The tree I was working off clearly had stale files, which led me to
include two utils which were removed some time ago: gpcheckutil.py
and gpcheck.py. Remove these two from their respective Makefiles.

Also fix a Bash error in the Stream symlink test, the logoical AND
requires [[ .. ]]; rather than [ .. ];.

Both of these spotted while repeatedly running make install with
trees in various states.

We have added tests to ensure that the search_path setting is taking effect. We
also are using a test database for the tests.
Co-authored-by: NJamie McAtamney <jmcatamney@pivotal.io>
Co-authored-by: NJacob Champion <pchampion@pivotal.io>
Co-authored-by: NNikolaos Kalampalikis <nkalampalikis@pivotal.io>