diff --git a/docs/source/optional_plugins/varianter_cit.rst b/docs/source/optional_plugins/varianter_cit.rst index 6970926b031512bc0a7da14a0f6ccbcbf1eba1f1..88f5504559e0b7adfa6199e46f86c5d69a28b85d 100644 --- a/docs/source/optional_plugins/varianter_cit.rst +++ b/docs/source/optional_plugins/varianter_cit.rst @@ -4,49 +4,114 @@ CIT Varianter Plugin ==================== -This plugin is an implementation of a Combinatorial Interaction -Testing algorithm for the Avocado varianter functionality. It -generates an optimal number of variants, which in turn become -different test scenarios. - -.. note:: The publication by Ahmed, Bestoun S., Kamal Z. Zamli, and - Chee Peng Lim, entitled `“Application of particle swarm - optimization to uniform and variable strength covering array - construction” - `__, - Applied Soft Computing, 12(4), 2012, pp. 1330-1347, contains - the basis for the algorithm and implementation of this - feature. - -Files -===== +This plugin is an implementation of a "Combinatorial Interaction +Testing with Constraints" algorithm for the Avocado varianter +functionality. It generates an optimal number of variants, which in +turn become different test scenarios. + +Publications +============ + +The publication by Ahmed, Bestoun S., Kamal Z. Zamli, and Chee Peng +Lim, entitled `“Application of particle swarm optimization to uniform +and variable strength covering array construction” +`__, +Applied Soft Computing, 12(4), 2012, pp. 1330-1347, contains the basis +for the algorithm and implementation of this feature. + +Additionally, the publication by Bestoun S. Ahmed, Amador Pahim, +Cleber R. Rosa Junior, D. Richard Kuhn and Miroslav Bures, entitled +`"Towards an Automated Unified Framework to Run Applications for +Combinatorial Interaction Testing" +`__, contain a practical use +case of this software. + +Examples +======== + +Please refer to ``examples/varianter_cit/params.cit`` for an example +of a input file. + +Input file format +================= + +The following is the general structure of a input file:: + + PARAMETERS + Parameter_1 [Value_1, Value_2, Value_3, Value_4] + Parameter_2 [Value_1, Value_2, Value_3, Value_4] + Parameter_3 [Value_1, Value_2, Value_3, Value_4] + + CONSTRAINTS + Parameter_1 != Value_1 || Parameter_2 != Value_3 + Parameter_3 != Value_2 || Parameter_2 != Value_4 || Parameter_1 != Value_4 + +The input file has two parts, parameters and constraints. + +Parameters +---------- -- ``optional_plugins/varianter_cit/avocado_varianter_cit/__init__.py``: The - plugin implementation. +* Each line represent one parameter. -- ``examples/varianter_cit/params.ini``: An example parameters file. +* Each parameter has a name, and a list of values ​​inside brackets. + +Constraints: +------------ + +* Constraints have to be in Conjunctive normal form. + +* Constraints use these tree operands: ``!=, OR, AND`` + +* ``||`` represents operand ``OR`` and new line represents operand ``AND``. + +* In the example is this logic formula:: + ((P_1 != V1 OR P_2 != V_3) AND (P_3 != V_2 OR P_2 != V_4 OR P_1 != Value_4)) Usage ===== +.. note:: the algorithm employed here can be CPU intensive. If you + want more information on the progress of the combinatorial + calculation, add ``--debug`` to a command line, such as + ``avocado variants --debug --cit-parameter-file $PATH`` + +Cit varianter plugin runs with two parameters: + +- `--cit-parameter-file` with path to the input file +- `--cit-order-of-combinations` with strength of combination (default is 2) + To see the variants generated by this demo implementation, execute:: $ avocado variants --cit-parameter-file examples/varianter_cit/params.ini - CIT Variants (14): - Variant gold-triangle-solid-aluminum-cathodic: /run - Variant red-circle-gas-leather-cathodic: /run - Variant gold-square-gas-plastic-anodic: /run - Variant black-circle-liquid-aluminum-anodic: /run - Variant green-triangle-liquid-leather-anodic: /run - Variant green-circle-solid-plastic-cathodic: /run - Variant black-square-solid-leather-cathodic: /run - Variant red-square-liquid-aluminum-anodic: /run - Variant red-triangle-liquid-plastic-cathodic: /run - Variant black-triangle-gas-plastic-anodic: /run - Variant green-square-gas-aluminum-anodic: /run - Variant gold-circle-solid-leather-anodic: /run - Variant gold-triangle-liquid-plastic-anodic: /run - Variant red-circle-solid-aluminum-cathodic: /run + CIT Variants (28): + Variant red-square-solid-plastic-anodic-6-4-4-2: / + Variant green-circle-gas-leather-cathodic-7-5-4-1: / + Variant green-triangle-liquid-leather-anodic-5-4-1-3: / + Variant green-square-liquid-plastic-anodic-3-1-4-5: / + Variant red-triangle-solid-leather-anodic-5-2-4-1: / + Variant black-triangle-gas-leather-anodic-7-1-1-2: / + Variant green-circle-solid-aluminum-cathodic-7-1-5-4: / + Variant red-square-gas-plastic-cathodic-6-3-5-3: / + Variant gold-triangle-solid-leather-anodic-6-5-1-4: / + Variant gold-triangle-gas-leather-anodic-3-2-5-2: / + Variant gold-square-gas-plastic-cathodic-5-1-1-1: / + Variant red-circle-gas-plastic-anodic-1-1-3-3: / + Variant red-circle-gas-aluminum-cathodic-3-3-1-5: / + Variant black-triangle-solid-plastic-cathodic-5-5-5-5: / + Variant gold-triangle-gas-leather-anodic-7-4-2-5: / + Variant black-triangle-gas-aluminum-cathodic-6-1-2-1: / + Variant gold-square-liquid-leather-cathodic-3-5-2-3: / + Variant black-square-solid-aluminum-cathodic-7-2-4-3: / + Variant black-circle-liquid-aluminum-anodic-1-4-5-1: / + Variant black-triangle-gas-leather-cathodic-7-3-3-1: / + Variant green-square-solid-aluminum-cathodic-1-3-2-2: / + Variant gold-triangle-gas-aluminum-anodic-1-3-4-4: / + Variant red-square-liquid-plastic-anodic-7-2-2-4: / + Variant gold-circle-liquid-aluminum-anodic-5-5-3-2: / + Variant red-triangle-gas-leather-anodic-1-5-1-5: / + Variant gold-circle-liquid-aluminum-cathodic-5-3-2-4: / + Variant black-square-solid-plastic-cathodic-3-4-3-4: / + Variant green-circle-liquid-plastic-cathodic-6-2-3-5: / .. note:: The exact variants generated are not guaranteed to be the same across executions. @@ -54,133 +119,346 @@ To see the variants generated by this demo implementation, execute:: You can enable more verbosity, making each variant to show its content:: $ avocado variants --cit-parameter-file examples/varianter_cit/params.ini -c - CIT Variants (15): - - Variant black-square-solid-leather-cathodic: /run - /:coating => cathodic - /:color => black - /:material => leather - /:shape => square - /:state => solid - - Variant red-circle-gas-aluminum-cathodic: /run - /:coating => cathodic - /:color => red - /:material => aluminum - /:shape => circle - /:state => gas - - Variant red-triangle-liquid-plastic-anodic: /run - /:coating => anodic - /:color => red - /:material => plastic - /:shape => triangle - /:state => liquid - - Variant green-square-gas-aluminum-anodic: /run - /:coating => anodic - /:color => green - /:material => aluminum - /:shape => square - /:state => gas - - Variant gold-circle-solid-leather-anodic: /run - /:coating => anodic - /:color => gold - /:material => leather - /:shape => circle - /:state => solid - - Variant green-triangle-solid-plastic-cathodic: /run - /:coating => cathodic - /:color => green - /:material => plastic - /:shape => triangle - /:state => solid - - Variant gold-square-liquid-plastic-cathodic: /run - /:coating => cathodic - /:color => gold - /:material => plastic - /:shape => square - /:state => liquid - - Variant black-triangle-gas-leather-anodic: /run - /:coating => anodic - /:color => black - /:material => leather - /:shape => triangle - /:state => gas - - Variant black-circle-liquid-plastic-anodic: /run - /:coating => anodic - /:color => black - /:material => plastic - /:shape => circle - /:state => liquid - - Variant gold-triangle-gas-aluminum-anodic: /run - /:coating => anodic - /:color => gold - /:material => aluminum - /:shape => triangle - /:state => gas - - Variant green-circle-liquid-leather-anodic: /run - /:coating => anodic - /:color => green - /:material => leather - /:shape => circle - /:state => liquid - - Variant red-square-solid-aluminum-anodic: /run - /:coating => anodic - /:color => red - /:material => aluminum - /:shape => square - /:state => solid - - Variant black-triangle-liquid-aluminum-anodic: /run - /:coating => anodic - /:color => black - /:material => aluminum - /:shape => triangle - /:state => liquid - - Variant red-square-gas-plastic-anodic: /run - /:coating => anodic - /:color => red - /:material => plastic - /:shape => square - /:state => gas - - Variant red-square-liquid-leather-cathodic: /run - /:coating => cathodic - /:color => red - /:material => leather - /:shape => square - /:state => liquid + CIT Variants (28): + + Variant red-circle-solid-plastic-cathodic-6-3-3-1: / + /:coating => cathodic + /:color => red + /:material => plastic + /:p10 => 1 + /:p7 => 6 + /:p8 => 3 + /:p9 => 3 + /:shape => circle + /:state => solid + + Variant black-circle-liquid-aluminum-anodic-6-5-1-2: / + /:coating => anodic + /:color => black + /:material => aluminum + /:p10 => 2 + /:p7 => 6 + /:p8 => 5 + /:p9 => 1 + /:shape => circle + /:state => liquid + + Variant black-triangle-liquid-plastic-anodic-3-1-4-2: / + /:coating => anodic + /:color => black + /:material => plastic + /:p10 => 2 + /:p7 => 3 + /:p8 => 1 + /:p9 => 4 + /:shape => triangle + /:state => liquid + + Variant black-triangle-solid-plastic-cathodic-6-4-3-5: / + /:coating => cathodic + /:color => black + /:material => plastic + /:p10 => 5 + /:p7 => 6 + /:p8 => 4 + /:p9 => 3 + /:shape => triangle + /:state => solid + + Variant green-circle-solid-leather-cathodic-3-5-3-3: / + /:coating => cathodic + /:color => green + /:material => leather + /:p10 => 3 + /:p7 => 3 + /:p8 => 5 + /:p9 => 3 + /:shape => circle + /:state => solid + + Variant black-triangle-liquid-aluminum-cathodic-1-3-2-3: / + /:coating => cathodic + /:color => black + /:material => aluminum + /:p10 => 3 + /:p7 => 1 + /:p8 => 3 + /:p9 => 2 + /:shape => triangle + /:state => liquid + + Variant gold-square-gas-plastic-anodic-6-4-5-3: / + /:coating => anodic + /:color => gold + /:material => plastic + /:p10 => 3 + /:p7 => 6 + /:p8 => 4 + /:p9 => 5 + /:shape => square + /:state => gas + + Variant gold-triangle-solid-leather-cathodic-5-3-5-5: / + /:coating => cathodic + /:color => gold + /:material => leather + /:p10 => 5 + /:p7 => 5 + /:p8 => 3 + /:p9 => 5 + /:shape => triangle + /:state => solid + + Variant green-square-gas-aluminum-cathodic-5-2-3-2: / + /:coating => cathodic + /:color => green + /:material => aluminum + /:p10 => 2 + /:p7 => 5 + /:p8 => 2 + /:p9 => 3 + /:shape => square + /:state => gas + + Variant green-triangle-liquid-aluminum-cathodic-7-3-1-4: / + /:coating => cathodic + /:color => green + /:material => aluminum + /:p10 => 4 + /:p7 => 7 + /:p8 => 3 + /:p9 => 1 + /:shape => triangle + /:state => liquid + + Variant gold-square-solid-leather-anodic-5-5-2-4: / + /:coating => anodic + /:color => gold + /:material => leather + /:p10 => 4 + /:p7 => 5 + /:p8 => 5 + /:p9 => 2 + /:shape => square + /:state => solid + + Variant red-square-gas-leather-anodic-3-3-1-5: / + /:coating => anodic + /:color => red + /:material => leather + /:p10 => 5 + /:p7 => 3 + /:p8 => 3 + /:p9 => 1 + /:shape => square + /:state => gas + + Variant red-circle-liquid-aluminum-anodic-5-4-4-1: / + /:coating => anodic + /:color => red + /:material => aluminum + /:p10 => 1 + /:p7 => 5 + /:p8 => 4 + /:p9 => 4 + /:shape => circle + /:state => liquid + + Variant gold-circle-liquid-aluminum-cathodic-7-1-5-5: / + /:coating => cathodic + /:color => gold + /:material => aluminum + /:p10 => 5 + /:p7 => 7 + /:p8 => 1 + /:p9 => 5 + /:shape => circle + /:state => liquid + + Variant red-triangle-solid-plastic-anodic-1-5-5-2: / + /:coating => anodic + /:color => red + /:material => plastic + /:p10 => 2 + /:p7 => 1 + /:p8 => 5 + /:p9 => 5 + /:shape => triangle + /:state => solid + + Variant green-triangle-gas-plastic-cathodic-3-4-5-4: / + /:coating => cathodic + /:color => green + /:material => plastic + /:p10 => 4 + /:p7 => 3 + /:p8 => 4 + /:p9 => 5 + /:shape => triangle + /:state => gas + + Variant green-square-gas-leather-anodic-1-5-4-5: / + /:coating => anodic + /:color => green + /:material => leather + /:p10 => 5 + /:p7 => 1 + /:p8 => 5 + /:p9 => 4 + /:shape => square + /:state => gas + + Variant red-circle-solid-leather-anodic-1-1-3-4: / + /:coating => anodic + /:color => red + /:material => leather + /:p10 => 4 + /:p7 => 1 + /:p8 => 1 + /:p9 => 3 + /:shape => circle + /:state => solid + + Variant gold-circle-liquid-aluminum-anodic-3-2-2-5: / + /:coating => anodic + /:color => gold + /:material => aluminum + /:p10 => 5 + /:p7 => 3 + /:p8 => 2 + /:p9 => 2 + /:shape => circle + /:state => liquid + + Variant black-square-solid-plastic-cathodic-5-1-1-3: / + /:coating => cathodic + /:color => black + /:material => plastic + /:p10 => 3 + /:p7 => 5 + /:p8 => 1 + /:p9 => 1 + /:shape => square + /:state => solid + + Variant green-circle-gas-leather-cathodic-6-1-2-1: / + /:coating => cathodic + /:color => green + /:material => leather + /:p10 => 1 + /:p7 => 6 + /:p8 => 1 + /:p9 => 2 + /:shape => circle + /:state => gas + + Variant red-square-solid-aluminum-cathodic-7-2-4-3: / + /:coating => cathodic + /:color => red + /:material => aluminum + /:p10 => 3 + /:p7 => 7 + /:p8 => 2 + /:p9 => 4 + /:shape => square + /:state => solid + + Variant red-circle-gas-plastic-anodic-7-4-2-2: / + /:coating => anodic + /:color => red + /:material => plastic + /:p10 => 2 + /:p7 => 7 + /:p8 => 4 + /:p9 => 2 + /:shape => circle + /:state => gas + + Variant gold-square-liquid-leather-anodic-1-4-1-1: / + /:coating => anodic + /:color => gold + /:material => leather + /:p10 => 1 + /:p7 => 1 + /:p8 => 4 + /:p9 => 1 + /:shape => square + /:state => liquid + + Variant gold-square-liquid-leather-cathodic-6-3-4-2: / + /:coating => cathodic + /:color => gold + /:material => leather + /:p10 => 2 + /:p7 => 6 + /:p8 => 3 + /:p9 => 4 + /:shape => square + /:state => liquid + + Variant gold-square-liquid-leather-anodic-7-5-3-1: / + /:coating => anodic + /:color => gold + /:material => leather + /:p10 => 1 + /:p7 => 7 + /:p8 => 5 + /:p9 => 3 + /:shape => square + /:state => liquid + + Variant black-triangle-liquid-plastic-anodic-7-2-5-1: / + /:coating => anodic + /:color => black + /:material => plastic + /:p10 => 1 + /:p7 => 7 + /:p8 => 2 + /:p9 => 5 + /:shape => triangle + /:state => liquid + + Variant black-square-gas-leather-cathodic-6-2-4-4: / + /:coating => cathodic + /:color => black + /:material => leather + /:p10 => 4 + /:p7 => 6 + /:p8 => 2 + /:p9 => 4 + /:shape => square + /:state => gas To execute tests with those combinations use:: - $ avocado run passtest.py --cit-parameter-file examples/varianter_cit/params.ini + $ avocado run passtest.py --cit-parameter-file examples/varianter_cit/params.cit JOB ID : 6abd9e9f1ff9ed33a353ca8f3ef845cd4cc404a5 JOB LOG : $HOME/avocado/job-results/job-2018-07-23T08.46-6abd9e9/job.log - (01/15) passtest.py:PassTest.test;gold-circle-gas-plastic-cathodic: PASS (0.06 s) - (02/15) passtest.py:PassTest.test;green-square-solid-plastic-anodic: PASS (0.02 s) - (03/15) passtest.py:PassTest.test;black-triangle-liquid-aluminum-anodic: PASS (0.02 s) - (04/15) passtest.py:PassTest.test;red-triangle-solid-leather-cathodic: PASS (0.02 s) - (05/15) passtest.py:PassTest.test;red-square-liquid-aluminum-cathodic: PASS (0.02 s) - (06/15) passtest.py:PassTest.test;green-circle-gas-leather-anodic: PASS (0.02 s) - (07/15) passtest.py:PassTest.test;gold-circle-solid-aluminum-anodic: PASS (0.02 s) - (08/15) passtest.py:PassTest.test;black-square-gas-leather-cathodic: PASS (0.02 s) - (09/15) passtest.py:PassTest.test;red-circle-liquid-plastic-anodic: PASS (0.02 s) - (10/15) passtest.py:PassTest.test;green-triangle-gas-aluminum-cathodic: PASS (0.02 s) - (11/15) passtest.py:PassTest.test;gold-triangle-liquid-leather-cathodic: PASS (0.02 s) - (12/15) passtest.py:PassTest.test;black-circle-solid-plastic-anodic: PASS (0.02 s) - (13/15) passtest.py:PassTest.test;red-triangle-gas-plastic-cathodic: PASS (0.02 s) - (14/15) passtest.py:PassTest.test;gold-square-liquid-leather-anodic: PASS (0.02 s) - (15/15) passtest.py:PassTest.test;green-circle-liquid-aluminum-anodic: PASS (0.02 s) - RESULTS : PASS 15 | ERROR 0 | FAIL 0 | SKIP 0 | WARN 0 | INTERRUPT 0 | CANCEL 0 + (01/25) passtest.py:PassTest.test;black-circle-gas-plastic-anodic-3-3-5-5: PASS (0.04 s) + (02/25) passtest.py:PassTest.test;gold-square-liquid-leather-anodic-3-2-1-4: PASS (0.03 s) + (03/25) passtest.py:PassTest.test;green-square-gas-plastic-cathodic-3-5-4-1: PASS (0.04 s) + (04/25) passtest.py:PassTest.test;gold-circle-solid-leather-anodic-6-4-4-2: PASS (0.04 s) + (05/25) passtest.py:PassTest.test;green-triangle-liquid-aluminum-cathodic-7-4-5-1: PASS (0.04 s) + (06/25) passtest.py:PassTest.test;black-circle-gas-plastic-cathodic-1-4-3-4: PASS (0.04 s) + (07/25) passtest.py:PassTest.test;red-square-gas-leather-anodic-3-4-2-3: PASS (0.04 s) + (08/25) passtest.py:PassTest.test;gold-triangle-solid-leather-anodic-1-3-2-1: PASS (0.04 s) + (09/25) passtest.py:PassTest.test;green-circle-gas-plastic-cathodic-7-1-2-4: PASS (0.04 s) + (10/25) passtest.py:PassTest.test;green-triangle-gas-aluminum-cathodic-6-2-2-5: PASS (0.04 s) + (11/25) passtest.py:PassTest.test;black-circle-liquid-plastic-cathodic-5-5-2-2: PASS (0.03 s) + (12/25) passtest.py:PassTest.test;red-square-solid-aluminum-anodic-5-2-3-1: PASS (0.04 s) + (13/25) passtest.py:PassTest.test;gold-square-solid-leather-anodic-7-5-3-5: PASS (0.04 s) + (14/25) passtest.py:PassTest.test;green-triangle-solid-leather-anodic-1-5-1-3: PASS (0.04 s) + (15/25) passtest.py:PassTest.test;black-circle-liquid-leather-cathodic-6-1-1-1: PASS (0.04 s) + (16/25) passtest.py:PassTest.test;red-triangle-liquid-plastic-anodic-6-3-3-3: PASS (0.04 s) + (17/25) passtest.py:PassTest.test;green-triangle-solid-plastic-cathodic-5-3-4-4: PASS (0.04 s) + (18/25) passtest.py:PassTest.test;red-square-liquid-aluminum-anodic-6-5-5-4: PASS (0.04 s) + (19/25) passtest.py:PassTest.test;red-square-gas-aluminum-cathodic-7-3-1-2: PASS (0.04 s) + (20/25) passtest.py:PassTest.test;red-square-liquid-aluminum-anodic-1-1-4-5: PASS (0.04 s) + (21/25) passtest.py:PassTest.test;gold-circle-gas-plastic-anodic-5-4-1-5: PASS (0.04 s) + (22/25) passtest.py:PassTest.test;gold-circle-solid-leather-anodic-5-1-5-3: PASS (0.04 s) + (23/25) passtest.py:PassTest.test;red-circle-liquid-plastic-cathodic-1-2-5-2: PASS (0.04 s) + (24/25) passtest.py:PassTest.test;green-triangle-solid-aluminum-anodic-3-1-3-2: PASS (0.04 s) + (25/25) passtest.py:PassTest.test;black-circle-solid-aluminum-cathodic-7-2-4-3: PASS (0.03 s) + RESULTS : PASS 25 | ERROR 0 | FAIL 0 | SKIP 0 | WARN 0 | INTERRUPT 0 | CANCEL 0 JOB TIME : 1.21 s JOB HTML : $HOME/avocado/job-results/job-2018-07-23T08.46-6abd9e9/results.html diff --git a/examples/varianter_cit/params.cit b/examples/varianter_cit/params.cit index a20c3d4ec6c5e40b0af3c47cbc53daf8f0ccf096..06c8ef626bdbf9705fbc79bcce5d246fcf08b8a3 100644 --- a/examples/varianter_cit/params.cit +++ b/examples/varianter_cit/params.cit @@ -1,5 +1,5 @@ PARAMETERS -color [black, gold, red, green ] +color[black, gold, red, green ] shape[square, triangle, circle] state[liquid, solid, gas] material[leather, plastic, aluminum]