The test can be asked to report its state, or it can do it itself.
Given that the test knows when it enters each phase, let's ask it to
report it.

Because the *state* is now reported, it's possible for a test to
finish (for instance by killing itself) without setting a valid (user
visible) status, such as "PASS", "FAIL", etc.
Signed-off-by: NCleber Rosa <crosa@redhat.com>

The avocado test runner mostly "flies blind" when it comes to the
individual test phases, that is, it doesn't know if a test is
currently being initialized, running its setup, running the test
method itself or its teardown.

With this, the runner gets access to that information, and can act
differently based on this knowledge.  One use case is to allow for
different timeouts on different test phases.
Signed-off-by: NCleber Rosa <crosa@redhat.com>

This is a really bad example of a test, in the sense that it requires
a huge library just to draw an image.

Let's drop it, given that the remaining example works just as well.
Signed-off-by: NCleber Rosa <crosa@redhat.com>

The former command line option is an old version, and the "--show"
form is preferred.
Signed-off-by: NCleber Rosa <crosa@redhat.com>

This is actually an ancient option, and the preferred syntax has been
"--show=none" for a while.
Signed-off-by: NCleber Rosa <crosa@redhat.com>

The later is a pure Python library, designed for the specific purpose
of validating schema, while the former requires a compiler tool chain
to be installed in a number of scenarios.

Let's also pin the version of its direct depency, elementpath.
Signed-off-by: NCleber Rosa <crosa@redhat.com>

Rules such as "check" on our Makefile make sure that no temporary
directory created by Avocado is left behind, but, it's very hard to
tell who created a rogue directory.

This brings improvements to the naming of the temporary directory, so
that it's trivial to tell which test created in the first place.

Additionally, it improves the temporary directory creation, usually by
making use of a tearDown() test phase, instead of other more fragile
methods.
Signed-off-by: NCleber Rosa <crosa@redhat.com>

Which are only necessary under Python 2, and thus, not needed
anymore.
Signed-off-by: NCleber Rosa <crosa@redhat.com>

There's no need to keep the special conditions for Python 2, now that
it's not supported anymore.
Signed-off-by: NCleber Rosa <crosa@redhat.com>

This test actually works fine under Python 3, the skip was not
properly lifted before.
Signed-off-by: NCleber Rosa <crosa@redhat.com>

And the compatiblity helper six.  With Python 2 gone, a number of
package changes are also introduced, so that the packages are always
named python3-<suffix>.

Also, because it would be confusing not having an "avocado" script,
and to conform with the system wide change introduced by distros that
now offer Python 3 by default, the scripts are no longer called
avocado-3 (or avocado-3.x), but simply "avocado".
Signed-off-by: NCleber Rosa <crosa@redhat.com>

Each test has its own log file within a given job result dir,
located at "test-results/$(test-id)/debug.log".

If the test is an INSTRUMENTED test, using the standard Python logging
module/API, all produced content should be recorded into the test log
file.  This adds a handler that does exactly that to the logging
module root logger.
Signed-off-by: NCleber Rosa <crosa@redhat.com>

The fake-vmstat process should run for infinity. Using wait with 1s
timeout will eventually destroy the test after the timeout, but cleaner
solution is to explicitly terminate it and then use the wait to make
sure it finished (or eventually nuke it)
Signed-off-by: NLukáš Doktor <ldoktor@redhat.com>

Since it's not necessary to use proc.terminate() and call directly
process.wait() to get process finished, this change makes the proper
adjustments in test utils partition.
Signed-off-by: NCaio Carrara <ccarrara@redhat.com>

Recently we fixed bug where timeouts were not kept under certain
conditions, but the fix lacked selftests modification of those 2
affected selftests as they rely on test-postprocess timing.
Signed-off-by: NLukáš Doktor <ldoktor@redhat.com>

Based on some reports of failures to behave properly on
unittest.TestCase's assertRaises methods, let's improve the test
coverage on these situations.

References: https://github.com/avocado-framework/avocado/issues/2791Signed-off-by: NCleber Rosa <crosa@redhat.com>

Since the pystache project seems to be dead, without recent updates and
has not support planned for Python 3.7+, this change replace it by jinja
template engine.
Signed-off-by: NCaio Carrara <ccarrara@redhat.com>

In an effort to make jobs more self contained, and given that dry-run
jobs created temporary content that is not supposed to be kept after
the job is run, this introduces the following changes:

 * Jobs with "dry-run" enabled will now, by default, cleanup their
   own data.  Basically, this means that the "base log directory"
   created during job initialization, and that for non "dry-run"
   jobs is usually "~/avocado/job-results", will also be removed
   at the job's "cleanup()" phase.

 * A new "--dry-run-no-cleanup" command line option, that only makes
   sense when jobs with dry-run enabled start to clean up data for
   themselves

 * Because some result plugins will, by default, write to the job
   results directory, results plugins are now run before a job cleanup
   is done.
Signed-off-by: NCleber Rosa <crosa@redhat.com>

There are at least two places in which the same functionality is
present.  Let's make it a single utility function instead.
Signed-off-by: NCleber Rosa <crosa@redhat.com>

Given that the setUp() phase is not useful for test_disk_space, and the
custom cleanup done in the test is better had at the tearDown() phase.

Additionally, the following changes where made to make it more reliable:

 * Avoid the busy loop giving modprobe some time between attempts
 * Use "modprobe -r" instead of rmmod, which handlers module dependencies
 * Do an explicit check for status
 * If "modprobe -r" doesn't succeed, it's not a test failure, but a test
   a test error (given that the get_diskspace() function already ran
   and succeeded).

Ideally, the module removal would be done by the linux_modules.unload_module()
but there are some issues with that function that needs addressing first
(such as returning a meaningful result).
Signed-off-by: NCleber Rosa <crosa@redhat.com>

Signed-off-by: NCleber Rosa <crosa@redhat.com>

Signed-off-by: NCleber Rosa <crosa@redhat.com>

As we don't need to support Python < 2.7 anymore.
Signed-off-by: NCleber Rosa <crosa@redhat.com>

The newly released pylint version 2.2.0, changed its default behavior
(enabled) the following checks:

 * :unnecessary-pass (W0107): *Unnecessary pass statement*
   Used when a "pass" statement that can be avoided is encountered.

 * :unused-import (W0611): *Unused %s*
   Used when an imported module or variable is not used.

Let's apply fixes to those checks, and at the same time, pin the
pylint versions so that our CI environment is more stable (code that
test our code -- pylint -- doesn't change, while our code changes).

For Python 2, the latest release was 1.9.3, so let's use that.
Signed-off-by: NCleber Rosa <crosa@redhat.com>

The correct name of the exception is TestAbortError, and not
TestAbortedError.
Signed-off-by: NCleber Rosa <crosa@redhat.com>

Currently, the only way to pass parameters to Avocado is by using the
variants layer, that is, one of the existing varianter plugin
implementations.  This was based on earlier design decisions, that
we're trying to fix without breaking expectation and compatibility.

While it'd makes sense to support parameters created by the varianter
layer itself, together with non-varianter parameters, it seems that we
can start by adding support for one or the other being used at a time.

This commit does exactly that: add support for a method of passing
parameters via the command line for all tests in a job, without
having to resort to the varianter.  When the varianter is used,
it takes precedence and the core test parameter passing support
is ignored.

This should solve a number of use cases that require one-time
executions of tests with different parameters, especially
executions during test development and debugging.
Signed-off-by: NCleber Rosa <crosa@redhat.com>

The lv_utils functional tests, which usually don't run because
they require privileged execution, revealed a number of broken
expectations when it comes to bytes and text handling.

Let's fix both the module and the test.
Signed-off-by: NCleber Rosa <crosa@redhat.com>

There was a functional test on avocado that depended on html plugin.
This change moves this test to proper plugin instead of run the test
only if the plugin is available during test execution.

Reference: https://trello.com/c/EVggIZzjSigned-off-by: NCaio Carrara <ccarrara@redhat.com>

There was a functional test on avocado that depended on remote plugin.
This change moves this test to proper plugin instead of run the test
only if the plugin is available during test execution.

Reference: https://trello.com/c/NPEUj0wZSigned-off-by: NCaio Carrara <ccarrara@redhat.com>

Signed-off-by: NCaio Carrara <ccarrara@redhat.com>

I was trying to use the external runner, in a reasonably simple way:
from within a project (QEMU) build dir, I'd run:

 $ avocado run --external-runner=tests/qemu-iotests/check \
   --external-runner-chdir=runner -- 001 002

But, what should work, did not.  The problem is with the non-absolute
path to the external runner, that is needed because we're changing
directories.

Absolute paths won't hurt otherwise (when no chdir is done), so it's
always done here.
Signed-off-by: NCleber Rosa <crosa@redhat.com>

This pylint checks for possible redefining outer names. Some occurrences
was preserved with disable flag because they would broke some public
api, like method/functions names or parameters.
Signed-off-by: NCaio Carrara <ccarrara@redhat.com>

The W0613 check for unnused arguments. The most of the current cases
seems legitmate so the proper flag was added in each line so pylint can
ignore it.
Signed-off-by: NCaio Carrara <ccarrara@redhat.com>

To decode into the correct encoding and work properly across different
Python versions.

While at it, improve the docstring.
Signed-off-by: NCleber Rosa <crosa@redhat.com>

To the relevant code block that can, for a reason we expect, raise
such exception.
Signed-off-by: NCleber Rosa <crosa@redhat.com>

If the AVOCADO variable starts with the Python interpreter, the
command that will be logged by Avocado needs to be adjusted.
Signed-off-by: NCleber Rosa <crosa@redhat.com>

Instead of repeating those definitions on most test files, let's define
them once, and reuse them across tests.
Signed-off-by: NCleber Rosa <crosa@redhat.com>

When AVOCADO is set by the environment variable, it may or may not contain
a Python interpreter.  In any situation, Avocado itself won't know about
it, and will log the avocado test runner script in its logs.
Signed-off-by: NCleber Rosa <crosa@redhat.com>

Instead of trying locations on Python 2 and 3.  The same pattern is
used on other locations.
Signed-off-by: NCleber Rosa <crosa@redhat.com>

Instead of creating new temporary dirs and files, and handling their
removal in each test, let's just create additional dirs/files in the
already provided tmpdir, which will be removed by the test tearDown
method.
Signed-off-by: NCleber Rosa <crosa@redhat.com>