Initial multiplex of configuration

In order for people to run all avocado tests at once,
move and reorganize tests under a directory called
selftests. For someone to run all self tests in
avocado, one would have to execute:

    selftests/run selftests/all

This is a shortcut for developers. .travis.yml
was also updated to run tests based on the new
directory structure.
Signed-off-by: NLucas Meneghel Rodrigues <lmr@redhat.com>

Signed-off-by: NLucas Meneghel Rodrigues <lmr@redhat.com>

Instead of using self.params.set_default() manually for
each default value, allow test writers to set a
default_params attribute with default params, and
set them at test class init time. This way we keep
things even shorter, while allowing avocado tests
to be used stand alone.
Signed-off-by: NLucas Meneghel Rodrigues <lmr@redhat.com>

Signed-off-by: NLucas Meneghel Rodrigues <lmr@redhat.com>

Signed-off-by: NLucas Meneghel Rodrigues <lmr@redhat.com>

Signed-off-by: NLucas Meneghel Rodrigues <lmr@redhat.com>

Make it possible to access param keys through attributes:

    self.params.sleep_length

instead of:

    self.params.get('sleep_length')

Also, introduce a params.set_default() method, that allows
to provide defaults to a given param attribute, in case
the configuration did not provide one (avocado tests running
as stand alone unittests).
Signed-off-by: NLucas Meneghel Rodrigues <lmr@redhat.com>

Tell the multiplex config file we're using floats as
the sleep_length value.
Signed-off-by: NLucas Meneghel Rodrigues <lmr@redhat.com>

When we specify the type of a variable in the multiplex config:

    sleep_length = 1.0
    sleep_length_type = float

Now we'll convert automatically the string 1.0 retrieved
from the multiplex system to a float in a transparent way.
This will save code by making people to avoid to manually
convert values (if not specified, the value is considered
a string).
Signed-off-by: NLucas Meneghel Rodrigues <lmr@redhat.com>

Signed-off-by: NLucas Meneghel Rodrigues <lmr@redhat.com>

Signed-off-by: NLucas Meneghel Rodrigues <lmr@redhat.com>

Signed-off-by: NLucas Meneghel Rodrigues <lmr@redhat.com>

Add a multiplexer parser, so people can try out their
multiplex files. Here is an example of usage

    $ scripts/avocado multiplex tests/sleeptest/sleeptest.mplx
    Dictionaries generated:
        dict 1:    sleeptest.short
        dict 2:    sleeptest.medium
        dict 3:    sleeptest.long

You can display dict contents by providing the flag --contents
(short -c).
Signed-off-by: NLucas Meneghel Rodrigues <lmr@redhat.com>

Modify sleeptest to pick any params from self.params.
Also, introduce a sample multiplex file, sleeptest.mplx,
an example of how to write a multiplex file to fulfill
matrix defining goals.
Signed-off-by: NLucas Meneghel Rodrigues <lmr@redhat.com>

With this, people will be able to introduce a --multiplex-file
(short -m) option to the run subcommand, that provides the
path to a multiplex file, that will be turned into a test
matrix (list of dicts) that will be run by the test runner.
Signed-off-by: NLucas Meneghel Rodrigues <lmr@redhat.com>

If one passes a multiplex file in the runner arguments,
we shall use it and generate test dicts for the entire
test matrix, and pass those dicts to each test instance.
Signed-off-by: NLucas Meneghel Rodrigues <lmr@redhat.com>

Introduce the possibility of using params dicts to
avocado tests. The params dicts lets tests to set
test tagged names based on the 'shortname' param,
among other features.
Signed-off-by: NLucas Meneghel Rodrigues <lmr@redhat.com>

Signed-off-by: NLucas Meneghel Rodrigues <lmr@redhat.com>

It is a common occurrence in automated testing
that we want to generate a test matrix, for example,
I want my python program to be tested under

1) Python 2.6
2) Python 2.7

In the following Linux Distributions

1) Fedora 20
2) Fedora 19
3) Ubuntu 12.04
4) Ubuntu 14.04

This commit introduces the multiplex config file
format, a format that lets you define this
test matrix in a compact way:

variants python:
    - py26:
        py_interpreter = 2.6
    - py27:
        py_interpreter = 2.7

variants distros:
    - fedora19:
        distro = Fedora19
    - fedora20:
        distro = Fedora20
    - ubuntu12_04:
        distro = Ubuntu12.04
    - ubuntu14_04:
        distro = Ubuntu14.04

Which generates a set of the 8 tests of this cartesian
product. Each test is in fact a python dict:

dict    1:  fedora19.py26
dict    2:  fedora19.py27
dict    3:  fedora20.py26
dict    4:  fedora20.py27
dict    5:  ubuntu12_04.py26
dict    6:  ubuntu12_04.py27
dict    7:  ubuntu14_04.py26
dict    8:  ubuntu14_04.py27

One given dict can be passed by to avocado tests, so they
can retrieve params from the dict. The next patches
focus on instrumenting avocado tests to make use of said
params dictionaries, and extending the test runner to
accept a multiplex file.
Signed-off-by: NLucas Meneghel Rodrigues <lmr@redhat.com>

Add a utility module that offers a Params class, a
Dict like class that offers some additional advantages
over a regular dictionary for testing purposes.
Signed-off-by: NLucas Meneghel Rodrigues <lmr@redhat.com>

avocado.job: Create entry point for  plugable test runners.

Add a new entry point to attach plugins when running tests with
test results.
Signed-off-by: NRuda Moura <rmoura@redhat.com>

Remove avocadoversion macro.
Signed-off-by: NRuda Moura <rmoura@redhat.com>

avocado.core.data_dir: Contemplate read only dirs

The avocado tests location can actually be a read
only location, so take that into account when looking
for test dirs.
Signed-off-by: NLucas Meneghel Rodrigues <lmr@redhat.com>

Add new avocado Functional Tests

They raise the following exceptions:

1) TestError (failure)
2) TestNAError (skip)
3) TestWarn (failure)

The corresponding functional tests for them go
in the Travis CI file.
Signed-off-by: NLucas Meneghel Rodrigues <lmr@redhat.com>

Since someone might want to be funny and raise those
exceptions, it is safer to make them to return "FAIL"
instead of "NEVER_RAISE_THIS", so avocado does not
crash.
Signed-off-by: NLucas Meneghel Rodrigues <lmr@redhat.com>

Since both can possibly fail and mask the actual test
results.
Signed-off-by: NLucas Meneghel Rodrigues <lmr@redhat.com>

It is easier to show when a test actually failed, or
we were not able to successfully set the test up.
Signed-off-by: NLucas Meneghel Rodrigues <lmr@redhat.com>

Fix stand alone tests rc

Signed-off-by: NLucas Meneghel Rodrigues <lmr@redhat.com>

When running stand alone tests, ie, tests without
the avocado runner, I surely do expect they return
exit codes according to wether it passed or failed.

This patch fixed that by making the TestModuleRunner
class to return appropriate return codes from the
Job.run() function.
Signed-off-by: NLucas Meneghel Rodrigues <lmr@redhat.com>

Fix test execution when running in standalone mode.

Fix errors:

* AttributeError: 'NoneType' object has no attribute 'unique_id'
* AttributeError: 'NoneType' object has no attribute 'keep_tmp_files'

When running tests standalone, like:

python tests/sleeptest/sleeptest.py
Signed-off-by: NRuda Moura <rmoura@redhat.com>

avocado.plugins.manager: Skip loading bad builtin plugins.

Signed-off-by: NLucas Meneghel Rodrigues <lmr@redhat.com>

Introduce Avocado journal replay utility