This is an attempt to catch most (if not all) usages of open
that do not follow the context manager pattern.

Sometimes, for better readability, our own `genio` library is
used.
Signed-off-by: NCleber Rosa <crosa@redhat.com>

The default parameters mechanism proved to be confusing, overlapping
and thus was scheduled to be removed.  This implements the removal
of the default_params API.
Signed-off-by: NCleber Rosa <crosa@redhat.com>

The documentation and examples on the output check feature are
outdated or innaccurate.

For instance, the synctest.py example would produce different output
than recorded in the accompanying reference files because the API
parameters were not being respected.
Signed-off-by: NCleber Rosa <crosa@redhat.com>

Signed-off-by: NCleber Rosa <crosa@redhat.com>

On `make rpm`, docs build fails with:

    docs/source/optional_plugins/varianter_pict.rst:33: WARNING: Could
    not lex literal_block as "ini".

This patch changes the way to define a block of code for a
proven-to-be-good one.
Signed-off-by: NAmador Pahim <apahim@redhat.com>

PICT is a free software (MIT licensed) tool that implements
combinatorial testing.  Information about it can be found here:

   https://github.com/Microsoft/pict/

This is a very simple, but functional varianter implementation.  It
performs the basic tasks of creating variants and making parameters
accessible to tests.  To test this version, it's recommended to
download PICT, build it (it only takes a C compiler such as GCC or
clang and make) and put it in a location included in your PATH (or use
`--pict-binary`).

To list the variants generated with the default order (2, that is,
do a pairwise idenpendent combinatorial testing):

 $ avocado variants --pict-parameter-file=examples/params.pict
 Pict Variants (11):
 Variant amd-scsi-rtl8139-windows-rhel6-acff:    /run
 ...
 Variant amd-ide-e1000-linux-rhel6-eb43:    /run

To list the variants generated with a 3-way combination:

 $ avocado variants --pict-parameter-file=examples/params.pict \
   --pict-order-of-combinations=3

 Pict Variants (28):
 Variant intel-ide-virtio-windows-rhel7-aea5:    /run
 ...
 Variant intel-scsi-e1000-linux-rhel7-9f61:    /run

It's also possible to run tests with variants generated by the same
command line switches.
Signed-off-by: NCleber Rosa <crosa@redhat.com>

Let's document the best practices on subclassing Avocado.
Signed-off-by: NAmador Pahim <apahim@redhat.com>

This script can be used after one merges a PR to verify __all__ the
commits pass `make check` and publishes the results to github.
Signed-off-by: NLukáš Doktor <ldoktor@redhat.com>

The IMAGE_PROVIDERS stores possible classes to provide images, let's
avoid unnecessary multiple occurrences.
Signed-off-by: NLukáš Doktor <ldoktor@redhat.com>

Public interface to append custom providers to the list iof providers
so users can add their private/internal image repositories.
Signed-off-by: NAmador Pahim <apahim@redhat.com>

We have many utils, but no documentation for them. Let's create an
'Utilities' section in our docs and add the first piece of content
there to document the vmimage module.
Signed-off-by: NAmador Pahim <apahim@redhat.com>

Signed-off-by: NCleber Rosa <crosa@redhat.com>

The support for per-test-sysinfo collection is there ever since the
beginning. Let's add a `per_test` setting to enable the same sysinfo
collection as per-job per-test.
Signed-off-by: NLukáš Doktor <ldoktor@redhat.com>

The output check functionality logs, on failures, the complete contents
of the both the actual output, and the expected one. Most often than not,
the most relevant information is the difference between both.

This patch logs the unified diffs and records the diff files
(`stdout.diff` and `stderr.diff`) containing the unified diff between
the expected outputs and the actual outputs.

Reference: https://trello.com/c/CXDP1khVSigned-off-by: NAmador Pahim <apahim@redhat.com>

In WritingTests documentation we mention `examples/tests`, but users
will probably get a better source of information in
`avocado-framework-tests` repository so let's include both here.
Signed-off-by: NLukáš Doktor <ldoktor@redhat.com>

The check output test heavily depends on test's datadir. Let's add a
pointer to where we define the test datadir as it's not just a single
file anymore.
Signed-off-by: NLukáš Doktor <ldoktor@redhat.com>

In most cases the single ` (italic) were meant as doube `` (code).
Signed-off-by: NLukáš Doktor <ldoktor@redhat.com>

The `avocado.fail_on` needs to be used as decorator, not a function.
Signed-off-by: NLukáš Doktor <ldoktor@redhat.com>

The --failfast off explanation is a bit hard to follow (for me),
hopefully this version will be easier to understand without the need to
look at it twice.
Signed-off-by: NLukáš Doktor <ldoktor@redhat.com>

There is an extensive description on how INSTRUMENTED test datadir
hierarchy looks like, but it might be useful to mention SIMPLE test as
well to see the difference.
Signed-off-by: NLukáš Doktor <ldoktor@redhat.com>

The `must_exist` is part of the API and significantly changes the way
the datadirs are processed, therefor it deserves to be mentioned in the
documentation.
Signed-off-by: NLukáš Doktor <ldoktor@redhat.com>

This change deprecates the use of the of the `datadir` property,
and switches to `get_data()` instead.
Signed-off-by: NCleber Rosa <crosa@redhat.com>

This method allows to look for test data in ways that are more
flexible than the current "datadir" feature currently available.

The current "datadir" is bound to the test filename only, and
can not provide different data for different tests hosted within
a single test file.

The newly introduced get_data() method looks for test data in
various locations, to be able to provide custom data not only
to different tests hosted inside a single test file, but also
to different variants.

Note about this alternative implementation of `get_data()`: it does so
in a much more detached form.  It allows for other test types to
clearly change the way data files are stored and accessed.  It also
introduces the concept of named data sources.

In the previous implementation, the types of "data dirs" were hard
coded, while in this approach, they're defined at the class level and
it's up to the `get_data()` implementation to deal with it.
Consequently, the `get_data()` method gets a second (optional)
parameter that allows the user to specify from which data source the
data file should be retrieved.

This is necessary, for example, when we add configurability to sources
of data files, such as expected `stdout`, `stderr` and `output` files.
Signed-off-by: NCleber Rosa <crosa@redhat.com>

golang ships a builtin test runner, which is called by
`go test <opts> <package or path>`. This patch introduces the golang
optional plugin, which can be used to discover and run golang tests
using Avocado.
Signed-off-by: NAmador Pahim <apahim@redhat.com>

Instead of the iterator next method.
Signed-off-by: NCleber Rosa <crosa@redhat.com>

Signed-off-by: NCleber Rosa <crosa@redhat.com>

The issue is with ":", not with ";". We changed the wrong character.

This reverts commit 60a65d20.
Signed-off-by: NAmador Pahim <apahim@redhat.com>

Signed-off-by: NCleber Rosa <crosa@redhat.com>

The extra category for plugins looks confusing to me. This patch
slightly changes the wording and moves the links to optional plugins
directly into the PIP installation section.
Signed-off-by: NLukáš Doktor <ldoktor@redhat.com>

Let's update the command to install the packages dependencies to
the latest fedora. In this commit:

- Switch from 'yum' to 'dnf'.
- Add python2: latest Fedora does not ship python2 package in minimal
installation.
- Add libffi-devel and openssl-devel: not shipped in latest Fedora
minimal installation.
Signed-off-by: NAmador Pahim <apahim@redhat.com>

Don't create nodes directly, but first attempt to use OrderedDict. If
advanced features (tags, ...) are detected, declare the current mapping
TreeNode and follow as before. In the end create mapping which should
turn the directly connected OrderedDicts into TreeNodes while keep the
hidden user dicts as OrderedDicts.
Signed-off-by: NLukáš Doktor <ldoktor@redhat.com>

The keys are not smart-converted into detected types, but always used as
they are. Let's mention this in our documentation.
Signed-off-by: NLukáš Doktor <ldoktor@redhat.com>

The python unittests should be executable even now as a simple tests or
via external runner + contrib script. Let's create a new type and
discover python unittests as first-class test types in Avocado.

As Avocado tests are in fact python unittests, we need to manually skip
these by ignoring tests that are discovered as Avocado tests, but are
marked as not-to-be-executed, which required the additional change to
_find_avocado_tests method.
Signed-off-by: NLukáš Doktor <ldoktor@redhat.com>

Signed-off-by: NCleber Rosa <crosa@redhat.com>

It's currently not clear how users should go about installing the
optional plugins when using PyPI/pip.

Let's list the package names as they're available on PyPI.
Signed-off-by: NCleber Rosa <crosa@redhat.com>

The current TestName class records more than a test name (according
to the Avocado specification).  It records, and by default, presents
the Test ID (__str__() and __repr__()).

The Test Name, again according to the Avocado specifications, is
contained within a Test ID, so it's not necessary to have two
different classes IMO.

Finally, the str_filesystem() method was adapted to be coherent with
the fact that it returns a file system representation of the Test ID,
not the Test Name.  It also became a property, to match the same use
of str_uid and str_variant.
Signed-off-by: NCleber Rosa <crosa@redhat.com>

The current example, on Test Names, where being given on the Test ID
section.  Let's move them closer to the Test Name section, and while
at it, also give Test ID examples.
Signed-off-by: NCleber Rosa <crosa@redhat.com>

While a given resolver could return such a test name, I think this is
a bad example for a Test Name in this section.

The reason is that it's not possible, with the available test
resolvers/loaders to create a test such as that (it used to be).
Signed-off-by: NCleber Rosa <crosa@redhat.com>

Signed-off-by: NCleber Rosa <crosa@redhat.com>

Signed-off-by: NCleber Rosa <crosa@redhat.com>