PICT is a free software (MIT licensed) tool that implements
combinatorial testing.  Information about it can be found here:

   https://github.com/Microsoft/pict/

This is a very simple, but functional varianter implementation.  It
performs the basic tasks of creating variants and making parameters
accessible to tests.  To test this version, it's recommended to
download PICT, build it (it only takes a C compiler such as GCC or
clang and make) and put it in a location included in your PATH (or use
`--pict-binary`).

To list the variants generated with the default order (2, that is,
do a pairwise idenpendent combinatorial testing):

 $ avocado variants --pict-parameter-file=examples/params.pict
 Pict Variants (11):
 Variant amd-scsi-rtl8139-windows-rhel6-acff:    /run
 ...
 Variant amd-ide-e1000-linux-rhel6-eb43:    /run

To list the variants generated with a 3-way combination:

 $ avocado variants --pict-parameter-file=examples/params.pict \
   --pict-order-of-combinations=3

 Pict Variants (28):
 Variant intel-ide-virtio-windows-rhel7-aea5:    /run
 ...
 Variant intel-scsi-e1000-linux-rhel7-9f61:    /run

It's also possible to run tests with variants generated by the same
command line switches.
Signed-off-by: NCleber Rosa <crosa@redhat.com>

Signed-off-by: NAmador Pahim <apahim@redhat.com>

Recent changes to remote runner use only stderr to get Avocado version,
but the docker plugin does not return separate stream for stdout and
stderr, therefor it always fails to get the version string. This patch
uses combination of stdeerr + stdout, which should be safe as we only
care about the first occurrence (so in case both stderr and stdout
produce version string, the stderr one will be used, anyway such
question is only hypothetical)
Signed-off-by: NLukáš Doktor <ldoktor@redhat.com>

Signed-off-by: NAmador Pahim <apahim@redhat.com>

The IMAGE_PROVIDERS stores possible classes to provide images, let's
avoid unnecessary multiple occurrences.
Signed-off-by: NLukáš Doktor <ldoktor@redhat.com>

Signed-off-by: NLukáš Doktor <ldoktor@redhat.com>

* https://github.com/avocado-framework/avocado:
  vmimage: make the list of providers expandable
  docs: create utils section

Public interface to append custom providers to the list iof providers
so users can add their private/internal image repositories.
Signed-off-by: NAmador Pahim <apahim@redhat.com>

We have many utils, but no documentation for them. Let's create an
'Utilities' section in our docs and add the first piece of content
there to document the vmimage module.
Signed-off-by: NAmador Pahim <apahim@redhat.com>

Signed-off-by: NCleber Rosa <crosa@redhat.com>

The operator is inverted since the beginning of times and no one ever
noticed. Until now.
Signed-off-by: NAmador Pahim <apahim@redhat.com>

Signed-off-by: NCleber Rosa <crosa@redhat.com>

Signed-off-by: NAmador Pahim <apahim@redhat.com>

Let's also package and build the recently introduced golang plugin.
Signed-off-by: NCleber Rosa <crosa@redhat.com>

There's a problem with the current SPEC file in that it doesn't
exclude some of the files part of the "plugins-loader-yaml" package.

This would cause problems if that plugin package attempts to be
installed, causing conflicts because they're packaged on two packages.
Signed-off-by: NCleber Rosa <crosa@redhat.com>

Signed-off-by: NAmador Pahim <apahim@redhat.com>

Signed-off-by: NAmador Pahim <apahim@redhat.com>

This prevents an error when the avocado golang plugin is installed,
but "go" itself is not.

This still allows tests to be listed without "go" because:
 * it is not a requirement for that
 * there's value in listing tests even when your system is (still?)
   not completely setup to run them
 * Avocado hasn't promised symmetry between listing and guaranteeing
   that tests will be able to run
 * There may be situations in which we can *not* predict if tests
   that we can list can be executed
Signed-off-by: NCleber Rosa <crosa@redhat.com>

Signed-off-by: NCleber Rosa <crosa@redhat.com>

Signed-off-by: NCleber Rosa <crosa@redhat.com>

The support for per-test-sysinfo collection is there ever since the
beginning. Let's add a `per_test` setting to enable the same sysinfo
collection as per-job per-test.
Signed-off-by: NLukáš Doktor <ldoktor@redhat.com>

The Daemons are suppose to output in `self.profile_dir` and not
`self.pre_dir`. Let's use the same handling as in
`start_job_collectibles` in `start_test_collectibles`.
Signed-off-by: NLukáš Doktor <ldoktor@redhat.com>

Signed-off-by: NCleber Rosa <crosa@redhat.com>

The output check functionality logs, on failures, the complete contents
of the both the actual output, and the expected one. Most often than not,
the most relevant information is the difference between both.

This patch logs the unified diffs and records the diff files
(`stdout.diff` and `stderr.diff`) containing the unified diff between
the expected outputs and the actual outputs.

Reference: https://trello.com/c/CXDP1khVSigned-off-by: NAmador Pahim <apahim@redhat.com>

Signed-off-by: NCleber Rosa <crosa@redhat.com>

In WritingTests documentation we mention `examples/tests`, but users
will probably get a better source of information in
`avocado-framework-tests` repository so let's include both here.
Signed-off-by: NLukáš Doktor <ldoktor@redhat.com>

The check output test heavily depends on test's datadir. Let's add a
pointer to where we define the test datadir as it's not just a single
file anymore.
Signed-off-by: NLukáš Doktor <ldoktor@redhat.com>

In most cases the single ` (italic) were meant as doube `` (code).
Signed-off-by: NLukáš Doktor <ldoktor@redhat.com>

The `avocado.fail_on` needs to be used as decorator, not a function.
Signed-off-by: NLukáš Doktor <ldoktor@redhat.com>

The --failfast off explanation is a bit hard to follow (for me),
hopefully this version will be easier to understand without the need to
look at it twice.
Signed-off-by: NLukáš Doktor <ldoktor@redhat.com>

Signed-off-by: NAmador Pahim <apahim@redhat.com>

Signed-off-by: NLukáš Doktor <ldoktor@redhat.com>

There is an extensive description on how INSTRUMENTED test datadir
hierarchy looks like, but it might be useful to mention SIMPLE test as
well to see the difference.
Signed-off-by: NLukáš Doktor <ldoktor@redhat.com>

As part of 0795831f, all occurrences of dict.iteritems() turned into
iteritems(dict).  The problem is that AvocadoParams doesn't implement
the items() method that six.iteritems() expects from Python 3.

So given that AvocadoParams doesn't provide it, we could either implement
items() just to satisfy iteritems(), or just use the existing interface.

This commit chooses the later, which means reverting a small part of
the changes (incorrect ones) from 0795831f.
Signed-off-by: NCleber Rosa <crosa@redhat.com>

On Python 2, that is allowed.  On Python 3, it raises a TypeError
with "'>' not supported between instances of 'NoneType' and 'int'".
Signed-off-by: NCleber Rosa <crosa@redhat.com>

The multiprocessing library in Python 3 has some differences from the
one in Python 2.  One of those, is the requirement of a context for a
SimpleQueue.

Since we're going to use that specific interface (get_context()),
let's check for it, and use it to get a context if it exists.

PS: This approach for checking the interface avoids having an external
library like `six` imported here just for the reason of checking the
Python version.
Signed-off-by: NCleber Rosa <crosa@redhat.com>

The 2to3 feature was enable to give Avocado a head start in the port to
Python 3.  But, while having positive characteristics, it has some
drawbacks, specially at development time, in which an extra step is
required.

The porting work done so far has not relied on 2to3, and it's been
having good results.  Let's then drop 2to3 completely, and allow the
same code base, without any massaging, to be used on both Python 2 and
Python 3.
Signed-off-by: NCleber Rosa <crosa@redhat.com>

The `must_exist` is part of the API and significantly changes the way
the datadirs are processed, therefor it deserves to be mentioned in the
documentation.
Signed-off-by: NLukáš Doktor <ldoktor@redhat.com>

Signed-off-by: NLukáš Doktor <ldoktor@redhat.com>

* https://github.com/avocado-framework/avocado:
  Runner: add option to log data directories for each test to be run
  selftests/unit/test_test.py: fix comment typo
  avocado.Test: use new data dirs for output check record feature
  Limit SimpleTest to "file" and "variant" data file sources
  avocado.Test: deprecate datadir property
  examples/tests/whiteboard.py: remove chdir into datadir
  avocado.Test: introduce get_data() method

While users could follow the documented rules for test directories,
it can be useful to log all of the options in advance.  With that
a test writer can create the data directories and put data files
at the right level.
Signed-off-by: NCleber Rosa <crosa@redhat.com>

Signed-off-by: NCleber Rosa <crosa@redhat.com>