The test can be asked to report its state, or it can do it itself.
Given that the test knows when it enters each phase, let's ask it to
report it.

Because the *state* is now reported, it's possible for a test to
finish (for instance by killing itself) without setting a valid (user
visible) status, such as "PASS", "FAIL", etc.
Signed-off-by: NCleber Rosa <crosa@redhat.com>

The avocado test runner mostly "flies blind" when it comes to the
individual test phases, that is, it doesn't know if a test is
currently being initialized, running its setup, running the test
method itself or its teardown.

With this, the runner gets access to that information, and can act
differently based on this knowledge.  One use case is to allow for
different timeouts on different test phases.
Signed-off-by: NCleber Rosa <crosa@redhat.com>

This is a really bad example of a test, in the sense that it requires
a huge library just to draw an image.

Let's drop it, given that the remaining example works just as well.
Signed-off-by: NCleber Rosa <crosa@redhat.com>

The former command line option is an old version, and the "--show"
form is preferred.
Signed-off-by: NCleber Rosa <crosa@redhat.com>

This is actually an ancient option, and the preferred syntax has been
"--show=none" for a while.
Signed-off-by: NCleber Rosa <crosa@redhat.com>

The later is a pure Python library, designed for the specific purpose
of validating schema, while the former requires a compiler tool chain
to be installed in a number of scenarios.

Let's also pin the version of its direct depency, elementpath.
Signed-off-by: NCleber Rosa <crosa@redhat.com>

Rules such as "check" on our Makefile make sure that no temporary
directory created by Avocado is left behind, but, it's very hard to
tell who created a rogue directory.

This brings improvements to the naming of the temporary directory, so
that it's trivial to tell which test created in the first place.

Additionally, it improves the temporary directory creation, usually by
making use of a tearDown() test phase, instead of other more fragile
methods.
Signed-off-by: NCleber Rosa <crosa@redhat.com>

Which are only necessary under Python 2, and thus, not needed
anymore.
Signed-off-by: NCleber Rosa <crosa@redhat.com>

There's no need to keep the special conditions for Python 2, now that
it's not supported anymore.
Signed-off-by: NCleber Rosa <crosa@redhat.com>

And the compatiblity helper six.  With Python 2 gone, a number of
package changes are also introduced, so that the packages are always
named python3-<suffix>.

Also, because it would be confusing not having an "avocado" script,
and to conform with the system wide change introduced by distros that
now offer Python 3 by default, the scripts are no longer called
avocado-3 (or avocado-3.x), but simply "avocado".
Signed-off-by: NCleber Rosa <crosa@redhat.com>

Each test has its own log file within a given job result dir,
located at "test-results/$(test-id)/debug.log".

If the test is an INSTRUMENTED test, using the standard Python logging
module/API, all produced content should be recorded into the test log
file.  This adds a handler that does exactly that to the logging
module root logger.
Signed-off-by: NCleber Rosa <crosa@redhat.com>

Recently we fixed bug where timeouts were not kept under certain
conditions, but the fix lacked selftests modification of those 2
affected selftests as they rely on test-postprocess timing.
Signed-off-by: NLukáš Doktor <ldoktor@redhat.com>

Based on some reports of failures to behave properly on
unittest.TestCase's assertRaises methods, let's improve the test
coverage on these situations.

References: https://github.com/avocado-framework/avocado/issues/2791Signed-off-by: NCleber Rosa <crosa@redhat.com>

Since the pystache project seems to be dead, without recent updates and
has not support planned for Python 3.7+, this change replace it by jinja
template engine.
Signed-off-by: NCaio Carrara <ccarrara@redhat.com>

In an effort to make jobs more self contained, and given that dry-run
jobs created temporary content that is not supposed to be kept after
the job is run, this introduces the following changes:

 * Jobs with "dry-run" enabled will now, by default, cleanup their
   own data.  Basically, this means that the "base log directory"
   created during job initialization, and that for non "dry-run"
   jobs is usually "~/avocado/job-results", will also be removed
   at the job's "cleanup()" phase.

 * A new "--dry-run-no-cleanup" command line option, that only makes
   sense when jobs with dry-run enabled start to clean up data for
   themselves

 * Because some result plugins will, by default, write to the job
   results directory, results plugins are now run before a job cleanup
   is done.
Signed-off-by: NCleber Rosa <crosa@redhat.com>

There are at least two places in which the same functionality is
present.  Let's make it a single utility function instead.
Signed-off-by: NCleber Rosa <crosa@redhat.com>

The correct name of the exception is TestAbortError, and not
TestAbortedError.
Signed-off-by: NCleber Rosa <crosa@redhat.com>

Currently, the only way to pass parameters to Avocado is by using the
variants layer, that is, one of the existing varianter plugin
implementations.  This was based on earlier design decisions, that
we're trying to fix without breaking expectation and compatibility.

While it'd makes sense to support parameters created by the varianter
layer itself, together with non-varianter parameters, it seems that we
can start by adding support for one or the other being used at a time.

This commit does exactly that: add support for a method of passing
parameters via the command line for all tests in a job, without
having to resort to the varianter.  When the varianter is used,
it takes precedence and the core test parameter passing support
is ignored.

This should solve a number of use cases that require one-time
executions of tests with different parameters, especially
executions during test development and debugging.
Signed-off-by: NCleber Rosa <crosa@redhat.com>

I was trying to use the external runner, in a reasonably simple way:
from within a project (QEMU) build dir, I'd run:

 $ avocado run --external-runner=tests/qemu-iotests/check \
   --external-runner-chdir=runner -- 001 002

But, what should work, did not.  The problem is with the non-absolute
path to the external runner, that is needed because we're changing
directories.

Absolute paths won't hurt otherwise (when no chdir is done), so it's
always done here.
Signed-off-by: NCleber Rosa <crosa@redhat.com>

The W0613 check for unnused arguments. The most of the current cases
seems legitmate so the proper flag was added in each line so pylint can
ignore it.
Signed-off-by: NCaio Carrara <ccarrara@redhat.com>

Instead of repeating those definitions on most test files, let's define
them once, and reuse them across tests.
Signed-off-by: NCleber Rosa <crosa@redhat.com>

`examples/tests/simplewarning.sh` calls a generic avocado command,
which gets added to the path by the test code. That generic avocado
command is `scripts/avocado`, from the source repository, which
contains the unversioned `/usr/bin/env python`.

Under some environments, such as Fedora >= 29, there may be no
unversioned Python binary.  Let's respect the UNITTEST_AVOCADO_CMD
environment variable, and add the the directory containting that
binary to the PATH.
Signed-off-by: NCleber Rosa <crosa@redhat.com>

Signed-off-by: NMerlin Mathesius <mmathesi@redhat.com>

The 'avocado list -V' prints the list of tests as a matrix with
Type, Test and Tag(s) columns. Some lines may get trailing
whitespace chars due the method used to align the columns' value.

Changed the call to avocado.utils.astring.iter_tabular_output()
so that trailing whitespace are removed from each row.

Adjusted the PluginsTest.test_list_no_file_loader() test due lack
of whitespace on outputs to be compared.

Reference: https://trello.com/c/JWeay87z/1045-strip-tabularoutput-linesSigned-off-by: NWainer dos Santos Moschetta <wainersm@redhat.com>

It's not easy to find one definitive specification for the the JUnit /
Xunit formats, and a number of different schemas have been published
by different entities.

The current schema version we ship was based on a repository that is
no longer available.  Even though the format is exactly the same,
let's switch to the schema hosted by the JUnit project, and to what
is considered the Jenkins schema (as part of the JUnit 5 project).

Reference: https://junit.org/junit5/
Reference: https://raw.githubusercontent.com/junit-team/junit5/master/platform-tests/src/test/resources/jenkins-junit.xsdSigned-off-by: NCleber Rosa <crosa@redhat.com>

Which means it won't be executed on environments such as Travis, where
resources are limited and this test fails from time to time.
Signed-off-by: NCleber Rosa <crosa@redhat.com>

The "assertEquals" is set as deprecated in py3 in favor of
"assertEqual".
Signed-off-by: NLukáš Doktor <ldoktor@redhat.com>

Some tests execute avocado directly, which uses "/usr/bin/env python" by
default. On distributions with py2 and py3 installed we might end-up
running those tests using python2 while running the main selftest by
python3.

With this change we need to make sure to pass the correct python binary
in UNITTEST_AVOCADO_CMD as it's not doing this by default (to allow any
avocado command).
Signed-off-by: NLukáš Doktor <ldoktor@redhat.com>

The FDDrainer reads from process output and write to loggers.  The
test log, which is intended for humans, can indeed try to treat it as
text.  The stream logger, on the other hand, should deal with data as
binary content *only*, but this will be handled on a future change.

By having text content passed to the loggers, we avoid the occurrence
of log lines that look like:

  [stdout] b'command output'

Or output files (job-YYYY-MM-DDTHH.MM-id/tests/<test>/output) that
may literally contain:

  b'command output'

The encoding used is the one specified at the command execution.  With
this fix, a number of tests on Python 3 can now run successfully.
Signed-off-by: NCleber Rosa <crosa@redhat.com>

In the module global scope, "man echo" is executed to attempt to
determine if it's a GNU echo.  But, "man echo" in a locale such
as UTF-8 produces funky copyright characters:

  $ LANG=en_US.UTF-8 man echo | chardetect -
  <stdin>: windows-1252 with confidence 0.73

Which can not be decoded by the UTF-8 codec, producing
UnicodeDecodeErrors.  Even if that wasn't the case, it's better to
control the locale of command that can generate varied output, and
that we depend on.

  $ LANG=C man echo | chardetect -
  <stdin>: ascii with confidence 1.0
Signed-off-by: NCleber Rosa <crosa@redhat.com>

This change introduces the following requirement: the command to
be run by the avocado.utils.process functions should be given as unicode.
It brings a clearer interface, in which there's no need to deal with
encodings internally.

For simple test, it brings changes to respect the of passing unicode.
And, it changes the presentation of simple tests with unicode names
explicitly expreseed as such.

The way to deal with the process data that may be generated on
stdout/stderr is unchanged.
Signed-off-by: NCleber Rosa <crosa@redhat.com>

There are few issues (3 to my count) which affect a number of tests
on Python 3.  Instead of not executing any of the functional tests,
let's skip the broken ones.

This is important because it will prevent new issues from making
its way into the Avocado code.

The list of tests being skipped is being tracked on our planning board,
so that they are accounted for, and enabled when fixed.

Reference: https://trello.com/c/eFY9Vw1R/1282-python-3-functional-tests-checklistSigned-off-by: NCleber Rosa <crosa@redhat.com>

And that have been causing failures on Travis jobs, due to limited
on unreliable amount of resources present.

This is the full list of tests that won't be run anymore on Travis,
only on `make check-full`:

  selftests.functional.test_basic.RunnerOperationTest.test_early_latest_result
  selftests.functional.test_basic.RunnerSimpleTest.test_kill_stopped_sleep
  selftests.functional.test_interrupt.InterruptTest.test_well_behaved_sigint
  selftests.functional.test_interrupt.InterruptTest.test_well_behaved_sigterm
  selftests.functional.test_loader.LoaderTestFunctional.test_simple_using_main
  selftests.functional.test_loader.LoaderTestFunctional.test_sleep_a_lot
Signed-off-by: NCleber Rosa <crosa@redhat.com>

The newly W605 warning is caused by incorrect escaped chars, that are
automatically translated to '\\' but future versions should treat it as
SyntaxError.
Signed-off-by: NLukáš Doktor <ldoktor@redhat.com>

Signed-off-by: NLukáš Doktor <ldoktor@redhat.com>

The skip_location is suppose to use "skip_location" and not
"warn_location".
Signed-off-by: NLukáš Doktor <ldoktor@redhat.com>

The built-in regexp requires re.MULTILINE to work the same way as before
(allowing the match to be everywhere and not just in the first line).

As for the existing ones I believe users would benefit from allowing
MULTILINE as well, but there might be slight change in behavior.
Signed-off-by: NLukáš Doktor <ldoktor@redhat.com>

These warnings are described as:

 W0612: Used when a variable is defined but not used.
 W0622: Used when a variable or function override a built-in.

After a suggestion from Andrei Stepanov, we decided to immediately
enable these.

Rerefence: https://pagure.io/standard-test-roles/pull-request/144
Reference: https://trello.com/c/Ohh7Q6qj/1183-enable-w-in-travis-make-checkSigned-off-by: NCleber Rosa <crosa@redhat.com>

The latest jobs have been failing because of the following issues:

    ************* Module raise
    E1130: 48,30: Raise.test: bad operand type for unary -: NoneType
    ************* Module selftests.functional.test_basic
    I1101:1191,24: PluginsXunitTest.run_and_check: Module 'lxml.etree'
    has not 'XMLSchema' member, but source is unavailable. Consider
    adding this module to extension-pkg-whitelist if you want to
    perform analysis based on run-time introspection of living
    objects.
    I1101:1191,40: PluginsXunitTest.run_and_check: Module 'lxml.etree'
    has not 'parse' member, but source is unavailable. Consider adding
    this module to extension-pkg-whitelist if you want to perform
    analysis based on run-time introspection of living objects.
    I1101:1193,43: PluginsXunitTest.run_and_check: Module 'lxml.etree'
    has not 'parse' member, but source is unavailable. Consider adding
    this module to extension-pkg-whitelist if you want to perform
    analysis based on run-time introspection of living objects.
    ************* Module selftests.unit.test_xunit
    I1101: 91,24: xUnitSucceedTest.test_add_success: Module
    'lxml.etree' has not 'XMLSchema' member, but source is
    unavailable. Consider adding this module to
    extension-pkg-whitelist if you want to perform analysis based on
    run-time introspection of living objects.
    I1101: 91,40: xUnitSucceedTest.test_add_success: Module
    'lxml.etree' has not 'parse' member, but source is
    unavailable. Consider adding this module to
    extension-pkg-whitelist if you want to perform analysis based on
    run-time introspection of living objects.
    I1101: 92,43: xUnitSucceedTest.test_add_success: Module
    'lxml.etree' has not 'parse' member, but source is
    unavailable. Consider adding this module to
    extension-pkg-whitelist if you want to perform analysis based on
    run-time introspection of living objects.

We can't whitelist the `lxml.etree` module at this time (inspekt doesn't
have such an option) and the raise failure is a false positive.  Let's
ignore those issues then.
Signed-off-by: NCleber Rosa <crosa@redhat.com>

Under Python 3.4, which runs on our CI and is our lowest supported
version, json.loads() requires a string, while newer versions also
allows bytes.

    >>> json.loads(b'{}')
    Traceback (most recent call last):
      File "<stdin>", line 1, in <module>
      File "/home/cleber/.local/lib/python3.4/json/__init__.py", line 312, in loads s.__class__.__name__))
    TypeError: the JSON object must be str, not 'bytes'

Let's make that code compatible with Python 3.4.
Signed-off-by: NCleber Rosa <crosa@redhat.com>