The level of TERSE or greater was set during development, we can lower
this to DEBUG now that this feature has stablized.

It fails and appears to not be maintained when:

--enable-gpfdist and --with-openssl are configured.

(cherry picked from commit 813528e7)

Because the gpinitstandby flag for passing in a standby master data directory
was changed from -F to -S, this commit changes -F to -S in gpinitsystem to
match.

-S was already in use for specifying that the cluster should have its mirrors
set up in spread configuration, so that flag is changed to --mirror-mode, which
accepts the mirror mode ('group' or 'spread') as an argument.
Co-authored-by: NJamie McAtamney <jmcatamney@pivotal.io>
Co-authored-by: NMark Sliva <msliva@pivotal.io>
(cherry picked from commit 6fa3bce8)

Despite my best efforts to avoid races on overloaded test containers,
test_join_and_indicate_progress_prints_dots_until_pool_is_done() has
been failing fairly often. Replace the simple-but-flaky time-based test
with an implementation that serializes the components of the race.

(cherry picked from commit 9f5384bc)

* REASSIGN OWNED. edits. Remove qualification that it doesn't change the database ownership

* ALTER FUNCTION. set from current clause description revision

* COMMENT. capitalize proper noun.

* COPY. describe PROGRAM option.

* CREATE AGGREGATE. Implements new syntax.

* CREATE FUNCTION. edits.

* DROP ROLE. small edit, link to other commands

* DROP USER. trivial edit.

* PREPARE. small edits

* REASSIGN OWNED. Trivial edits.

* REINDEX. trivial edit.

* pg_dump. use --quote-all-identifers for cross-version dumps

* Additional edits

* Updates from review

* Updates from review

6_Backport: show gprecoverseg progress

- Read PXF host/port from environment variables
- Add functions to get the PXF port and host
- Add unit tests for get_pxf_port get_pxf_host
We store the current value of the environment variable, then we set a
custom host/port, run the tests, and finally restore the value of the
environment variable.
- Handle error when parsing PXF_PORT

- The transaction ID and segment ID are only used by a subset of
  profiles. PXF should be able to append the transaction and segment IDs
  depending on the profile using the header information.
- Remove write_file_name, do not send path

Co-authored-by: NOliver Albertini <oalbertini@pivotal.io>
Co-authored-by: NBradford D. Boyle <bboyle@pivotal.io>

Previously was using master.
Co-authored-by: NOliver Albertini <oalbertini@pivotal.io>
Co-authored-by: NBradford D. Boyle <bboyle@pivotal.io>

Postgres optimizes the storage of the ItemPointer in the Gin Posting
List. It only stores 11 bits for the offset number because heap tables
only have enough tuples per block to fit in 11 bits. However,
Greenplum append-only tables store 16 bits worth of offset numbers.

Initially we thought we'd need to modify decode_varbyte() but it turns
out that it is OK.  It handles 48 bits already.
Co-authored-by: NAlexandra Wang <lewang@pivotal.com>
Co-authored-by: NAshwin Agrawal <aagrawal@pivotal.io>

(cherry picked from commit 5e918bf6)

* docs - remove gpsyncmaster and add pg_basebackup to backend program list.

* Updates from review

Make "demo_cluster" the default tag (ie: no tag). That is, when a
scenario does not specify a behave tag it uses the demo cluster
environment to run. Use the concourse_cluster tag if a scenario requires
a remote cluster. Concourse runs the scenarios with no tag that use a
demo cluster by filtering for --tags=~concourse_cluster.
Authored-by: NKalen Krempely <kkrempely@pivotal.io>
(cherry picked from commit 40028cf4)

Regenerate pipeline from CLI_BEHAVE_TESTS refactor.

(cherry picked from commit 5dc3bfaa)

Renamed to CLI_BEHAVE_TESTS for clarity

(cherry picked from commit 87c4a52c)

Tag existing tests with @demo_cluster and @concourse_cluster
annotations. The @demo_cluster tag denotes the scenario can run
locally. The @concourse_cluster tag denotes the scenario that requires
a remote cluster.
Co-authored-by: NJamie McAtamney <jmcatamney@pivotal.io>
Authored-by: NKalen Krempely <kkrempely@pivotal.io>
(cherry picked from commit 735da14f)

Now that the previous commits have placed these tests in CLI_TESTS,
we can remove references to this symbol.

(cherry picked from commit 55add2c5)

Tag existing tests with @demo_cluster and @concourse_cluster
annotations. The @demo_cluster tag denotes the scenario can run
locally. The @concourse_cluster tag denotes the scenario that requires
a remote cluster.
Co-authored-by: NJamie McAtamney <jmcatamney@pivotal.io>
Authored-by: NKalen Krempely <kkrempely@pivotal.io>
(cherry picked from commit ffce7734)

Tag existing tests with @demo_cluster and @concourse_cluster
annotations. The @demo_cluster tag denotes the scenario can run
locally. The @concourse_cluster tag denotes the scenario that requires
a remote cluster.
Co-authored-by: NJamie McAtamney <jmcatamney@pivotal.io>
Authored-by: NKalen Krempely <kkrempely@pivotal.io>
(cherry picked from commit d67ecc3f)

Tag existing tests with @demo_cluster and @concourse_cluster
annotations. The @demo_cluster tag denotes the scenario can run
locally. The @concourse_cluster tag denotes the scenario that requires
a remote cluster.
Authored-by: NKalen Krempely <kkrempely@pivotal.io>
(cherry picked from commit e3457667)

Tag existing tests with @demo_cluster and @concourse_cluster
annotations. The @demo_cluster tag denotes the scenario can run
locally. The @concourse_cluster tag denotes the scenario that requires
a remote cluster.
Authored-by: NKalen Krempely <kkrempely@pivotal.io>
(cherry picked from commit a5130acb)

Now that the previous commits have placed these tests in CLI_TESTS,
we can remove references to this symbol.

(cherry picked from commit 8988618e)

Tag existing tests with @demo_cluster and @concourse_cluster
annotations. The @demo_cluster tag denotes the scenario can run
locally. The @concourse_cluster tag denotes the scenario that requires
a remote cluster.
Co-authored-by: NJamie McAtamney <jmcatamney@pivotal.io>
Authored-by: NKalen Krempely <kkrempely@pivotal.io>
(cherry picked from commit f676d2cc)

Tag existing tests with @demo_cluster and @concourse_cluster
annotations. The @demo_cluster tag denotes the scenario can run
locally. The @concourse_cluster tag denotes the scenario that requires
a remote cluster.
Co-authored-by: NJamie McAtamney <jmcatamney@pivotal.io>
Authored-by: NKalen Krempely <kkrempely@pivotal.io>
(cherry picked from commit 37a09d64)

Tag existing tests with @demo_cluster and @concourse_cluster
annotations. The @demo_cluster tag denotes the scenario can run
locally. The @concourse_cluster tag denotes the scenario that requires
a remote cluster.
Co-authored-by: NJamie McAtamney <jmcatamney@pivotal.io>
Authored-by: NKalen Krempely <kkrempely@pivotal.io>
(cherry picked from commit 8bb12f08)

Tag existing tests with @demo_cluster and @concourse_cluster
annotations. The @demo_cluster tag denotes the scenario can run
locally. The @concourse_cluster tag denotes the scenario that requires
a remote cluster.

(cherry picked from commit e7b5f462)

Tag existing tests with @demo_cluster and @concourse_cluster
annotations. The @demo_cluster tag denotes the scenario can run
locally. The @concourse_cluster tag denotes the scenario that requires
a remote cluster.
Authored-by: NKalen Krempely <kkrempely@pivotal.io>
(cherry picked from commit 642c4c8f)

Tag existing tests with @demo_cluster and @concourse_cluster
annotations. The @demo_cluster tag denotes the scenario can run
locally. The @concourse_cluster tag denotes the scenario that requires
a remote cluster.
Co-authored-by: NJamie McAtamney <jmcatamney@pivotal.io>
Authored-by: NKalen Krempely <kkrempely@pivotal.io>
(cherry picked from commit 53aad9c3)

Templatize gpmovemirrors into a single concourse job to run both
concourse_cluster and demo_cluster tags separate tasks.
Co-Authored-By: NKalen Krempely <kkrempely@pivotal.io>
(cherry picked from commit 0df65394)

Each gpexpand test case need a new environment.
Previous code uses pkill to make sure no postgres
process remaining and then uses gpstop to double
confirm.

However, pkill to terminate cluster may over-kill
some other postgres process on the hosts. What's
more, the interface `Given a working directory of
the test as '/data/gpdata/gpexpand'` will remove
the directory which also remove the $MASTER_DATA_DIRECTORY,
so the following `gpstop` is useless because
$MASTER_DATA_DIRECTORY is gone. Also, this introduce
race conditon.

This commit removes pkill and use gpstop interface
to build a clean environment for each test case. Also
it makes sure that `gpstop` is invoked at the very
beginning of each test case.

It tried to make sure gpfdist is ready by sleep 3 seconds before
queries, but that could not guarantee. This commit changes the sleep to
a retry until gpfdist is ready.

(cherry picked from commit f616837a)

We've seen frequent pg_rewind test failures due to promotion timeout. We
should increase the timeout. Also, add code to error out on timeout.
Co-authored-by: NNing Yu <nyu@pivotal.io>

Reviewed-by: Ashwin Agrawal

Previously the default value is 0, so normally log rotation happens each
day by default according to the other log related guc configurations.
That often makes a single log file rather large in real users'
environment.  Limiting the log file size so that people could easiler
open and search the log files.

1) pm_launch_walreceiver was introduced in b824fe8f
   to avoid a potential deadlock when cluster is down after
   updating the new role in gp_segment_configuration but before
   sending promote signal. It is buggy now since some changes.
   We need to put pm_launch_walreceiver in shared memory since
   it's used in different processes.

2) The following assertion is not always true in a real scenario if
   the segment is being promoted. We could simply stop the connection
   for the !am_mirror case.

   ProcessStartupPacket()
     if (am_ftshandler)
     {
        Assert(am_mirror);

Ning Yu and we had some discussions for these bugs. He contributed for
issue 1 also.
Co-authored-by: NHaozhou Wang <hawang@pivotal.io>

Related PR discussion:

kmacoskey:

Another piece of this can be removed with the same intention as the
other removal.

This line in the Makefile is related to a kfw snowflake dependency that
was being vendored into the Clients/Loaders package as brought in by
Ivy. Note the path, ext. It was dropped from Ivy by releng awhile ago,
2c6f4c9d#diff-271f398f0c07147024e7980737cddb7d . The cp here probably
should have been removed along with the change to Ivy, but it was
missed. It can be removed - but lets make sure the Clients team is
aware.

pf-qiu:

I think it's safe to do this now.

(cherry-picked from 22646c10)
Co-authored-by: NBradford D. Boyle <bboyle@pivotal.io>
Co-authored-by: NBen Christel <bchristel@pivotal.io>