Signed-off-by: NLarry Hamel <lhamel@pivotal.io>
Signed-off-by: NShoaib Lari <slari@pivotal.io>

Previously, during gpinitsystem, the standby was instantiated in the middle of
setting up the master. This ordering caused problems because
initializing the standby could cause an exit when an error
occurred. As a result of this early exit, the gp_toolkit and DCA gucs
were not set properly.

Instead, initialize the standby after the master is finished.
------------------------------------------
Previously the exit return code for gpinitsystem was always non-zero.
Now, it is non-zero only in an error or warning case.

The issue was due to SCAN_LOG interpretation of an empty string as a line count of one.
Fixed by changing to word count.
------------------------------------------
Initializing a standby can no longer cause gpinitsystem to exit early.
Added extra logging/output about standby master status.
Tell user at the end of gpinitsystem if gpinitstandby failed.
------------------------------------------
Signed-off-by: NMarbin Tan <mtan@pivotal.io>

The data_checksums GUC setting should be the same as the master.

The existing test for gpinitstandby is modified to run on a single host.
Signed-off-by: NLarry Hamel <lhamel@pivotal.io>
Signed-off-by: NC.J. Jameson <cjameson@pivotal.io>

- validate consistent checksum settings
    - Make sure checksum settings for all segments are same as master
    - Add logging proxy to allow logging to file to have different contents
      than stdout
    - Do heap checksum only for when starting up all segments.

- Add option --skip-heap-checksum-validation
    - If this option is provided to gpstart, the cluster will start up without
      checking for matching "data_checksums" GUC between master and segments.
Signed-off-by: NNadeem Ghani <nghani@pivotal.io>
Signed-off-by: NMarbin Tan <mtan@pivotal.io>

Signed-off-by: NLarry Hamel <lhamel@pivotal.io>

Updating the log message to display the parameters gpconfig was called
with, both if the GUC was changed successfully or not.
Signed-off-by: NMarbin Tan <mtan@pivotal.io>
Signed-off-by: NShoaib Lari <slari@pivotal.io>
Signed-off-by: NLarry Hamel <lhamel@pivotal.io>

Follow up commit for f936c4f3, which
added quotes around gpconfig values.
Signed-off-by: NMarbin Tan <mtan@pivotal.io>

Signed-off-by: NChris Hajas <chajas@pivotal.io>

gpinitsystem did not check for HEAP_CHECKSUM in the cluster
configuration file with a -c switch. This commit accepts the
HEAP_CHECKSUM setting, and additionally exports it to an
output_configuration_file when specified with the -O switch.

This commit also adds behave tests for the above, and for
reading the input_configuration_file with the -I switch.
Signed-off-by: NLarry Hamel <lhamel@pivotal.io>
Signed-off-by: NXin Zhang <xzhang@pivotal.io>

Bug fix for a scenario with multiple analyzedb processes running
concurrently: the resulting report files were incorrect and/or
overwritten.

This commit adds a lock (a file semaphore) for synchronization between analyzedb processes.
Each process will acquire an exclusive lock, per database, read the most recent report
files (possibly written by concurrently running analyzedb processes) and
incorporate that latest information into its own report.
Signed-off-by: NLarry Hamel <lhamel@pivotal.io>

Refactor similar usage to share code with gpperfmon behave tests
Signed-off-by: NXin Zhang <xzhang@pivotal.io>

Add test for gppkg --migrate
 * gppkgs installed on the original master should be installed on the new master and all segments
Signed-off-by: NLarry Hamel <lhamel@pivotal.io>
Signed-off-by: NNadeem Ghani <nghani@pivotal.io>

Add a behave test to verify that the checksum configuration is preserved
after a segment is recovered using gprecoveseg.
Signed-off-by: NShoaib Lari <slari@pivotal.io>

This file contains a list of schema-qualified tablenames in the backup
set.  It is not used in the restore process; it is there solely to allow
users to determine which tables were dumped in that backup set.
Signed-off-by: NJamie McAtamney <jmcatamney@pivotal.io>
Signed-off-by: NChris Hajas <chajas@pivotal.io>

This is an internal utility called by gpdbrestore and should not have
specific tests (except for testing with valgrind).

If you have multiple segment hosts with failures during a "sync", we
used to only report the first issue.

Fix: accumulate all the failures before reporting to the user.
Add unit test
Signed-off-by: NMarbin Tan <mtan@pivotal.io>
Signed-off-by: NNadeem Ghani <nghani@pivotal.io>

This commit introduces BEHAVE_FLAGS as a new parameter in concourse.
This will help us run specific tests within a specific scenario, it used
to be just all or nothing. Now we can separate multi-host testing vs
single host testing.

Add tests for gppkg --clean for multi-host
 * gppkg --clean should install to the segment host with no gppkg
 * gppkg --clean should remove on all segment hosts when gppkg does not exist in master

Ensure that gppkg is doing an RPM to all hosts -- this is just a
backfill testing addition.

There are times where gpperfmon_log_alert_history scenario fails,
because there's no data in the log alert history table.
This might be due to us copying an empty csv file; gpperfmon writes to a
file for log alert in a cadence. We might be copying a file that has not
been written into yet -- possibly empty.

Make sure that we have something to copy first before proceeding to the
next step.
Signed-off-by: NMarbin Tan <mtan@pivotal.io>

This is the final part of the backup and restore TINC to behave
migration. The MFR suite tests the managed file replication for Data
Domain.
Signed-off-by: NChris Hajas <chajas@pivotal.io>

The behave test is intermittently failing because the substep is trying
to install gpperfmon eventhough gpperfmon is already installed. Trying
to figure out which part of failing the correct value is hard with the
current setup.
Decouple the checks to determine which part is actually failing.
Signed-off-by: NTushar Dadlani <tdadlani@pivotal.io>

In this behave test, we delete some entries in pg_depend and in some
relative catalog tables to simulate a corruption around pg_depend. The
gpcheckcat tool should then flag these down.

gpperfmon drop partition sql statement was syntactically incorrect, so
partition_age gpperfmon feature was not working.

We were using the rows in partitionrangestart column from
pg_partition to drop specific partitions. The row value from
partitionrangestart is reported, as such, '2017-02-01 00:00:00'::timestamp(0) without time zone.
The query was reporting an error "Not a constant expression".

Use only the first part of partitionrangestart to make our ALTER DROP
query work.

- Added behave test to confirm that it is now working
Signed-off-by: NNadeem Ghani <nghani@pivotal.io>
Signed-off-by: NMarbin Tan <mtan@pivotal.io>
Signed-off-by: NC.J. Jameson <cjameson@pivotal.io>

This is part of the effort to unify our backup/restore tests into a
single suite.

* Adds infrastructure to setup DDBoost on the client and cleanup server
* after completion
* Adds tests for DDBoost specific options
* Adds test coverage from TINC suite that was not included in behave
Signed-off-by: NChris Hajas <chajas@pivotal.io>

Since Command creates a short-lived SSH session, we observe the PID given
a throw-away remote process. Assume that the PID is unused and available on
the remote in the near future.

This pid is no longer associated with a
running process and won't be recycled for long enough that tests
have finished.

Looking ahead introduces the risk of a time-of-check-time-of-use race
since the pid might have been allocated by the operating system by
the time the test would use the data.

This is part of the effort to get all backup/restore tests using the
same test suite. Since the Netbackup tests take significantly longer, we
only run a subset of the regular test suite. We also tag scenarios to
allow parallel runs on separate hosts in CI.

This suite will take 1h, 40 mins after parallelization, down from the
current 2h, 20 mins.

If IPv6 is enabled, gpperfmon will complain that it can't connect to
the database because, gpmon does not have permission to access through
::1.

- Add behave test

This should work with both dbid and contentid backup filenames.
Signed-off-by: NKaren Huddleston <khuddleston@pivotal.io>

Signed-off-by: NChris Hajas <c.hajas@pivotal.io>

* add behave test for diskspace_history
* remove commented-out test (moved to new tracker story to redo)
* refactor test for qamode
Signed-off-by: NC.J. Jameson <cjameson@pivotal.io>

Signed-off-by: NMarbin Tan <mtan@pivotal.io>
Signed-off-by: NC.J. Jameson <cjameson@pivotal.io>

- We think that external tables previously broke with newlines. This is
  no longer the case, so removing space replacement logic.
Signed-off-by: NMarbin Tan <mtan@pivotal.io>
Signed-off-by: NLarry Hamel <lhamel@pivotal.io>

Signed-off-by: NLarry Hamel <lhamel@pivotal.io>

Modify previous behave tests to address test flakiness
-- Use gpstop instead of pkill
-- Reduce min_query_time to 0
Signed-off-by: NLarry Hamel <lhamel@pivotal.io>

Signed-off-by: NMelanie Plageman <mplageman@pivotal.io>

On our normal workstations, gpsmon comes up within 5 seconds very
consistently, but in other environments, it might need more time
Signed-off-by: NC.J. Jameson <cjameson@pivotal.io>