From e2082252c66a540446fa1ebf43db40920fb402c2 Mon Sep 17 00:00:00 2001 From: Zhenghua Lyu Date: Wed, 3 Apr 2019 10:41:39 +0800 Subject: [PATCH] Fix gpexpand flaky test cases Each gpexpand test case need a new environment. Previous code uses pkill to make sure no postgres process remaining and then uses gpstop to double confirm. However, pkill to terminate cluster may over-kill some other postgres process on the hosts. What's more, the interface `Given a working directory of the test as '/data/gpdata/gpexpand'` will remove the directory which also remove the $MASTER_DATA_DIRECTORY, so the following `gpstop` is useless because $MASTER_DATA_DIRECTORY is gone. Also, this introduce race conditon. This commit removes pkill and use gpstop interface to build a clean environment for each test case. Also it makes sure that `gpstop` is invoked at the very beginning of each test case. --- .../test/behave/mgmt_utils/gpexpand.feature | 102 +++++++----------- 1 file changed, 41 insertions(+), 61 deletions(-) diff --git a/gpMgmt/test/behave/mgmt_utils/gpexpand.feature b/gpMgmt/test/behave/mgmt_utils/gpexpand.feature index fd71ce687f..dc12f97eb3 100644 --- a/gpMgmt/test/behave/mgmt_utils/gpexpand.feature +++ b/gpMgmt/test/behave/mgmt_utils/gpexpand.feature @@ -4,10 +4,9 @@ Feature: expand the cluster by adding more segments @gpexpand_no_mirrors @gpexpand_timing Scenario: after resuming a duration interrupted redistribution, tables are restored - Given a working directory of the test as '/data/gpdata/gpexpand' - And the database is killed on hosts "mdw,sdw1" + Given the database is not running + And a working directory of the test as '/data/gpdata/gpexpand' And a temporary directory under "/data/gpdata/gpexpand/expandedData" to expand into - And the database is not running And a cluster is created with no mirrors on "mdw" and "sdw1" And the master pid has been saved And database "gptest" exists @@ -29,10 +28,9 @@ Feature: expand the cluster by adding more segments @gpexpand_timing @gpexpand_standby Scenario: after a duration interrupted redistribution, state file on standby matches master - Given a working directory of the test as '/data/gpdata/gpexpand' - And the database is killed on hosts "mdw,sdw1" + Given the database is not running + And a working directory of the test as '/data/gpdata/gpexpand' And a temporary directory under "/data/gpdata/gpexpand/expandedData" to expand into - And the database is not running And a cluster is created with no mirrors on "mdw" and "sdw1" And the user runs gpinitstandby with options " " And database "gptest" exists @@ -48,10 +46,9 @@ Feature: expand the cluster by adding more segments @gpexpand_no_mirrors @gpexpand_timing Scenario: after resuming an end time interrupted redistribution, tables are restored - Given a working directory of the test as '/data/gpdata/gpexpand' - And the database is killed on hosts "mdw,sdw1" + Given the database is not running + And a working directory of the test as '/data/gpdata/gpexpand' And a temporary directory under "/data/gpdata/gpexpand/expandedData" to expand into - And the database is not running And a cluster is created with no mirrors on "mdw" and "sdw1" And database "gptest" exists And there are no gpexpand_inputfiles @@ -70,10 +67,9 @@ Feature: expand the cluster by adding more segments @gpexpand_no_mirrors @gpexpand_segment Scenario: expand a cluster that has no mirrors - Given a working directory of the test as '/data/gpdata/gpexpand' - And the database is killed on hosts "mdw,sdw1" + Given the database is not running + And a working directory of the test as '/data/gpdata/gpexpand' And a temporary directory under "/data/gpdata/gpexpand/expandedData" to expand into - And the database is not running And a cluster is created with no mirrors on "mdw" and "sdw1" And database "gptest" exists And there are no gpexpand_inputfiles @@ -86,10 +82,9 @@ Feature: expand the cluster by adding more segments @gpexpand_no_mirrors @gpexpand_host Scenario: expand a cluster that has no mirrors with one new hosts - Given a working directory of the test as '/data/gpdata/gpexpand' - And the database is killed on hosts "mdw,sdw1,sdw2" + Given the database is not running + And a working directory of the test as '/data/gpdata/gpexpand' And a temporary directory under "/data/gpdata/gpexpand/expandedData" to expand into - And the database is not running And a cluster is created with no mirrors on "mdw" and "sdw1" And database "gptest" exists And there are no gpexpand_inputfiles @@ -103,10 +98,9 @@ Feature: expand the cluster by adding more segments @gpexpand_no_mirrors @gpexpand_host_and_segment Scenario: expand a cluster that has no mirrors with one new hosts - Given a working directory of the test as '/data/gpdata/gpexpand' - And the database is killed on hosts "mdw,sdw1,sdw2" + Given the database is not running + And a working directory of the test as '/data/gpdata/gpexpand' And a temporary directory under "/data/gpdata/gpexpand/expandedData" to expand into - And the database is not running And a cluster is created with no mirrors on "mdw" and "sdw1" And database "gptest" exists And there are no gpexpand_inputfiles @@ -120,10 +114,9 @@ Feature: expand the cluster by adding more segments @gpexpand_mirrors @gpexpand_segment Scenario: expand a cluster that has mirrors - Given a working directory of the test as '/data/gpdata/gpexpand' - And the database is killed on hosts "mdw,sdw1" + Given the database is not running + And a working directory of the test as '/data/gpdata/gpexpand' And a temporary directory under "/data/gpdata/gpexpand/expandedData" to expand into - And the database is not running And a cluster is created with mirrors on "mdw" and "sdw1" And database "gptest" exists And there are no gpexpand_inputfiles @@ -135,10 +128,9 @@ Feature: expand the cluster by adding more segments @gpexpand_mirrors @gpexpand_host Scenario: expand a cluster that has mirrors with one new hosts - Given a working directory of the test as '/data/gpdata/gpexpand' - And the database is killed on hosts "mdw,sdw1,sdw2,sdw3" + Given the database is not running + And a working directory of the test as '/data/gpdata/gpexpand' And a temporary directory under "/data/gpdata/gpexpand/expandedData" to expand into - And the database is not running And a cluster is created with mirrors on "mdw" and "sdw1" And database "gptest" exists And there are no gpexpand_inputfiles @@ -153,10 +145,9 @@ Feature: expand the cluster by adding more segments @gpexpand_host_and_segment @gpexpand_standby Scenario: expand a cluster that has mirrors with one new host - Given a working directory of the test as '/data/gpdata/gpexpand' - And the database is killed on hosts "mdw,sdw1,sdw2,sdw3" + Given the database is not running + And a working directory of the test as '/data/gpdata/gpexpand' And a temporary directory under "/data/gpdata/gpexpand/expandedData" to expand into - And the database is not running And a cluster is created with mirrors on "mdw" and "sdw1" And the user runs gpinitstandby with options " " And database "gptest" exists @@ -170,10 +161,9 @@ Feature: expand the cluster by adding more segments @gpexpand_verify_redistribution Scenario: Verify data is correctly redistributed after expansion - Given a working directory of the test as '/data/gpdata/gpexpand' - And the database is killed on hosts "localhost" + Given the database is not running + And a working directory of the test as '/data/gpdata/gpexpand' And a temporary directory under "/data/gpdata/gpexpand/expandedData" to expand into - And the database is not running And the cluster is generated with "1" primaries only And database "gptest" exists And the user connects to "gptest" with named connection "default" @@ -192,11 +182,10 @@ Feature: expand the cluster by adding more segments Then distribution information from table "public.redistribute" with data in "gptest" is verified against saved data @gpexpand_verify_writable_external_redistribution - Scenario: Verify policy of writable external table is correctly updated after redistribution - Given a working directory of the test as '/data/gpdata/gpexpand' - And the database is killed on hosts "localhost" + Scenario: Verify policy of writable external table is correctly updated after redistribution + Given the database is not running + And a working directory of the test as '/data/gpdata/gpexpand' And a temporary directory under "/data/gpdata/gpexpand/expandedData" to expand into - And the database is not running And the cluster is generated with "1" primaries only And database "gptest" exists And the user create a writable external table with name "ext_test" @@ -211,10 +200,9 @@ Feature: expand the cluster by adding more segments @gpexpand_icw_db_concourse Scenario: Use a dump of the ICW database for expansion - Given a working directory of the test as '/data/gpdata/gpexpand' - And the database is killed on hosts "mdw,sdw1,sdw2,sdw3" + Given the database is not running + And a working directory of the test as '/data/gpdata/gpexpand' And a temporary directory under "/data/gpdata/gpexpand/expandedData" to expand into - And the database is not running And a cluster is created with mirrors on "mdw" and "sdw1" And database "gptest" exists And the user runs psql with "-f /home/gpadmin/sqldump/dump.sql" against database "gptest" @@ -230,11 +218,10 @@ Feature: expand the cluster by adding more segments @gpexpand_no_restart @gpexpand_catalog_copied Scenario: expand a cluster without restarting db and catalog has been copied - Given a working directory of the test as '/data/gpdata/gpexpand' - And the database is killed on hosts "mdw,sdw1" + Given the database is not running + And a working directory of the test as '/data/gpdata/gpexpand' And the user runs command "rm -rf /data/gpdata/gpexpand/*" And a temporary directory under "/data/gpdata/gpexpand/expandedData" to expand into - And the database is not running And a cluster is created with no mirrors on "mdw" and "sdw1" And the master pid has been saved And database "gptest" exists @@ -255,11 +242,10 @@ Feature: expand the cluster by adding more segments @gpexpand_no_restart @gpexpand_conf_copied Scenario: expand a cluster without restarting db and conf has been copie - Given a working directory of the test as '/data/gpdata/gpexpand' - And the database is killed on hosts "mdw,sdw1" + Given the database is not running + And a working directory of the test as '/data/gpdata/gpexpand' And the user runs command "rm -rf /data/gpdata/gpexpand/*" And a temporary directory under "/data/gpdata/gpexpand/expandedData" to expand into - And the database is not running And a cluster is created with no mirrors on "mdw" and "sdw1" And the master pid has been saved And database "gptest" exists @@ -278,11 +264,10 @@ Feature: expand the cluster by adding more segments @gpexpand_no_restart @gpexpand_long_run_read_only Scenario: expand a cluster without restarting db with long-run read-only transaction - Given a working directory of the test as '/data/gpdata/gpexpand' - And the database is killed on hosts "mdw,sdw1" + Given the database is not running + And a working directory of the test as '/data/gpdata/gpexpand' And the user runs command "rm -rf /data/gpdata/gpexpand/*" And a temporary directory under "/data/gpdata/gpexpand/expandedData" to expand into - And the database is not running And a cluster is created with no mirrors on "mdw" and "sdw1" And the master pid has been saved And database "gptest" exists @@ -304,11 +289,10 @@ Feature: expand the cluster by adding more segments @gpexpand_no_restart @gpexpand_change_catalog_abort Scenario: expand a cluster without restarting db and any transaction which wants to change the catalog must be aborted - Given a working directory of the test as '/data/gpdata/gpexpand' - And the database is killed on hosts "mdw,sdw1" + Given the database is not running + And a working directory of the test as '/data/gpdata/gpexpand' And the user runs command "rm -rf /data/gpdata/gpexpand/*" And a temporary directory under "/data/gpdata/gpexpand/expandedData" to expand into - And the database is not running And a cluster is created with no mirrors on "mdw" and "sdw1" And the master pid has been saved And database "gptest" exists @@ -328,11 +312,10 @@ Feature: expand the cluster by adding more segments @gpexpand_no_restart @gpexpand_dml Scenario: expand a cluster without restarting db and any transaction which wants to do dml work well - Given a working directory of the test as '/data/gpdata/gpexpand' - And the database is killed on hosts "mdw,sdw1" + Given the database is not running + And a working directory of the test as '/data/gpdata/gpexpand' And the user runs command "rm -rf /data/gpdata/gpexpand/*" And a temporary directory under "/data/gpdata/gpexpand/expandedData" to expand into - And the database is not running And a cluster is created with no mirrors on "mdw" and "sdw1" And the master pid has been saved And database "gptest" exists @@ -355,11 +338,10 @@ Feature: expand the cluster by adding more segments @gpexpand_mirrors @gpexpand_rollback Scenario: inject a fail and test if rollback is ok - Given a working directory of the test as '/data/gpdata/gpexpand' - And the database is killed on hosts "mdw,sdw1" + Given the database is not running + And a working directory of the test as '/data/gpdata/gpexpand' And the user runs command "rm -rf /data/gpdata/gpexpand/*" And a temporary directory under "/data/gpdata/gpexpand/expandedData" to expand into - And the database is not running And a cluster is created with mirrors on "mdw" and "sdw1" And the user runs gpinitstandby with options " " And database "gptest" exists @@ -376,11 +358,10 @@ Feature: expand the cluster by adding more segments @gpexpand_no_mirrors @gpexpand_with_special_character Scenario: create database,schema,table with special character - Given a working directory of the test as '/tmp/gpexpand_behave' - And the database is killed on hosts "mdw,sdw1" + Given the database is not running + And a working directory of the test as '/tmp/gpexpand_behave' And the user runs command "rm -rf /tmp/gpexpand_behave/*" And a temporary directory under "/tmp/gpexpand_behave/expandedData" to expand into - And the database is not running And a cluster is created with no mirrors on "mdw" and "sdw1" And database "gptest" exists And create database schema table with special character @@ -397,11 +378,10 @@ Feature: expand the cluster by adding more segments @gpexpand_mirrors @gpexpand_retry_failing_work_in_phase1_after_releasing_catalog_lock Scenario: inject a fail and test if retry is ok - Given a working directory of the test as '/data/gpdata/gpexpand' - And the database is killed on hosts "mdw,sdw1" + Given the database is not running + And a working directory of the test as '/data/gpdata/gpexpand' And the user runs command "rm -rf /data/gpdata/gpexpand/*" And a temporary directory under "/data/gpdata/gpexpand/expandedData" to expand into - And the database is not running And a cluster is created with mirrors on "mdw" and "sdw1" And the user runs gpinitstandby with options " " And database "gptest" exists -- GitLab