Test tags (via docstring directives) have been supported in Avocado
for a number of releases, but up until now user would only be able to
see the tags on their tests by resorting to the source code.

This was already difficult when the tags where set only at the class
level, now that they can be set both on the class and method
docstring, users would have to do the "math" on their minds.

Let's display the test tags when the verbose switch is given to the
list command to make the life of users easier.
Signed-off-by: NCleber Rosa <crosa@redhat.com>

Introducing the --force-execution to create the Job regardless the fact
that some test references might not be resolved to tests.

Reference: https://trello.com/c/bDUJjhq0Signed-off-by: NAmador Pahim <apahim@redhat.com>

A finer level of granularity may help to extend our testing coverage
while reducing the amount of false positives.  This introduces the
three different levels, mapped like this:

   * level 0, AKA "make check"
   * level 2, AKA "make check-full"

There are no changes of test assignment, that is, tests previously
being run under "check-full" will continue to be run only at that
target.  The same is true for tests that would run under "make check"
before this.

The big change is that there's now a middle ground, that can be
activated by manually setting the AVOCADO_CHECK_LEVEL variable.  Level
1 is intended to be used be used on environments that are halfway
between a dedicated machine and a really low powered environment.
Signed-off-by: NCleber Rosa <crosa@redhat.com>

Due to the deprecation of the self.skip(), the availability of the skip
decorators and the new CANCEL status, some internal improvements are
important to keep our behaviour sane and our own code sound and clean.

- The test.SkipTest class was renamed to test.MockingTest to be even more
  generic, intending to be overridden by sub-classes that make the test
  to end both with SKIP or CANCEL status. To keep it generic,
  test.MockingTest class will not SKIP the test if used directly anymore.

- The test.TimeOutSkipTest and test.ReplaySkipTest classes now are using
  the skip decorators instead of raising an exception in setUp(), since
  'skipping' the test means 'don't execute anything', not even the
  setUp().

- The test.DryRunTest class, which is expected to log itself in setUp()
  and then abort the test execution, is now using self.cancel() (instead
  of raising a SKIP exception), being now compliant with the concept
  that a SKIP test cannot execute anything.

- Selftests were adjusted accordingly.
Signed-off-by: NAmador Pahim <apahim@redhat.com>

Currently there is a hardcoded test-postprocess timeout consisting of
two cycle-timeouts which is something between 1 and 2 seconds. This is
not sufficient on heavily loaded machines so this patch increases the
timeouts to:

 1s: when test was interrupted (ctrl+c/timeout)
10s: when the process died but the status was not yet delivered
60s: when test reported status but the process did not finish

taking into account the current available test/job timeout.

As those deadlines are significantly longer, users could have noticed a
frozen throbber, which is why I propagate the results_dispatcher and
handle the throbber also during the postprocess, which makes this patch
a bit more complicated than one would expect. I used "progress = False"
to distinguish between running test and postprocessing it.
Signed-off-by: NLukáš Doktor <ldoktor@redhat.com>

Signed-off-by: NLukáš Doktor <ldoktor@redhat.com>

As it has already been done to some other tests, two other tests have
been producing false positives.  This has been observed recently in
package builds in build farms.

Let's run them only when `make check-full` is called, which is
usually done at dedicated machines with plenty of resources.

Reference: https://kojipkgs.fedoraproject.org/work/tasks/9189/19059189/build.log
Reference: https://kojipkgs.fedoraproject.org/work/tasks/9445/19069445/build.logSigned-off-by: NCleber Rosa <crosa@redhat.com>

The MissingTest is yielded by TestLoaderProxy, but it was implemented
and described in FileLoader, which caused troubles when FileLoader was
not being used. This simple fix allows TestLoaderProxy to define basic
mappings and adds MissingTest to it.
Signed-off-by: NLukáš Doktor <ldoktor@redhat.com>

Currently the `selftests/run_coverage` only reports unit coverage. This
patch allows specifying custom `avocado` command in selftests and uses
it to run coverage to also include the functional tests to results.
Signed-off-by: NLukáš Doktor <ldoktor@redhat.com>

`system-out` should be in the same level as `failure`, but we are
creating it as a `failure` child.

This patch puts system-out in the same level as `failure` and adds the
functional test to validate all the xunit outputs against the
junit-4.xsd schema.
Signed-off-by: NAmador Pahim <apahim@redhat.com>

Recently we added a check for canceled test, let's also check the
"cancel" number of tests are reported correctly in json plugin.
Signed-off-by: NLukáš Doktor <ldoktor@redhat.com>

self.skip() is to be deprecated. This is the first step, adding a
message to the exception raised by the self.skip() and adjusting the
examples tests accordingly.
Signed-off-by: NAmador Pahim <apahim@redhat.com>

On some environments, such as when building RPM packages, very little
can be assumed to be available.  `man` is such as command that is
*not* available, so let's check for it and skip tests accordingly.
Signed-off-by: NCleber Rosa <crosa@redhat.com>

Signed-off-by: NLucas Meneghel Rodrigues <lookkas@gmail.com>

I'm not 100% sure, but I don't see a benefit in allowing people to
override test status from inside test. They are suppose to use
assertions, or `self.fail`-like methods. With this change it'd be harder
to manually override the test status (as can be seen on updated
unittests) but we minimize the accidental write into `self.status`
variable, which can lead to confusing results.

Note we can consider adding `set_status` method to allow setting this
property, but I'd like to avoid allowing direct `self.status = ` usage.
Signed-off-by: NLukáš Doktor <ldoktor@redhat.com>

The unittest standard library on Python 2.7 an later has everything
that the unittest2 backport is supposed to have.  Let's then drop all
the conditional imports of unittest2 and stick with unittest.
Signed-off-by: NCleber Rosa <crosa@redhat.com>

There's one functional test that is missing "--sysinfo=off".  I
happened to find this because lspci on my system hangs when running
under X.  That test would timeout at the sysinfo collection.
Signed-off-by: NCleber Rosa <crosa@redhat.com>

To provide support for passing arguments for simple tests in command
line, we were creating some cases where the test reference could not be
properly handled. Also, for test references with white-spaces, we have
an inconsistent behavior. See reference.

This patch removes the support for arguments in test references,
restoring the proper behavior, which is expected to be shell-like.

As consequence, we now handle correctly white-spaces in tests references
as well as Unicode characters.

Reference: https://www.redhat.com/archives/avocado-devel/2016-November/msg00011.html
Reference: https://trello.com/c/bfE9NBblSigned-off-by: NAmador Pahim <apahim@redhat.com>

We've been seen failures on resource constrained environments on some
tests that are either time sensitive and/or assume a some computing
resources will be available.

Environments such as TravisCI and RPM build servers may not have the
resources these tests expect, so let's move them to runs of `make
check-full`.  This will hopefully put an end to the false positives
we've been getting on TravisCI.
Signed-off-by: NCleber Rosa <crosa@redhat.com>

It has been some time now since replay feature accepts the job to be
identified by its results directory. Now it's time to remove the
--replay-data-dir option, since this feature became redundant.
Signed-off-by: NAmador Pahim <apahim@redhat.com>

Recent pylint update is more pedantic about module level spacing. There
are no changes to code, only couple of extra spaces to make it happy.
Signed-off-by: NLukáš Doktor <ldoktor@redhat.com>

It's been a while since the "Test ID RFC" came out:

https://www.redhat.com/archives/avocado-devel/2016-March/msg00024.html

Still, we've been referring to test URLs all around our code
and also on user visible strings.  This is an attempt to rename
the mentions of "URLs" that really should be "test references"
or simply "references" at this point.
Signed-off-by: NCleber Rosa <crosa@redhat.com>

Some functional tests, most of which execute a full avocado command,
were missing the `--job-results-dir`, pointing to a directory that
would be removed after the tests.  This has led to the pollution of
the standard job results directory (usually `~/avocado/job-results`)
when selftests are run.

This proposal fixes all but one set of functional tests, which has
to be fixed with a different set of changes (see reference).

Besides the functional tests, some unittests are also creating
job results that should be discarded.  It's believed that at least
some of this is caused by the relationship between tests and jobs,
but this is speculation and it's not attepted to be fixed on this
patch.

Reference: https://trello.com/c/I2ieqhR5Signed-off-by: NCleber Rosa <crosa@redhat.com>

The selftests.functional.test_basic.PluginsTest.test_plugin_order
executes the tests 3 times, for the last time only to check it does not
reports failure. Let's just add this missing plugin into the previous
check. If error occurs, one can always check manually which issue is
that (compare to thousands of executions where it goes well).
Signed-off-by: NLukáš Doktor <ldoktor@redhat.com>

Over the time some non-optimal "avocado run" executions accumulated.
This fixes (hopefully) all of them by adding `--sysinfo` and
`--job-results-dir`.
Signed-off-by: NLukáš Doktor <ldoktor@redhat.com>

The archive (ZIP) feature has no reason for reason for being
a core feature, as it maps nicely to the result plugin interface.

Let's turn that into a proper (and quite simple) plugin.

Also, let's add a functional test for both the archive plugin, and for
the ordered execution configuration using the zip archive for that.
Signed-off-by: NCleber Rosa <crosa@redhat.com>

Let's separate the check for requirements and the test code according to
the project's understanding of setup/test phases.

Reference: https://trello.com/c/mSrMLO0KSigned-off-by: NAmador Pahim <apahim@redhat.com>

Recently I fixed a bug where test which asks for input hangs for
infinity but I forgot to add unittest for it. Let's add it now.
Signed-off-by: NLukáš Doktor <ldoktor@redhat.com>

While the behavior of having a plugin enabled once it's installed
has suited Avocado reasonably well, there are valid use cases that
require plugins to be disabled.

While adding this mechanism, let's define what the Avocado fully
qualified name for a given plugin is.

While not an urgent feature, enabling and disabling plugins is
actually one requirement for the implementation of the Job API.  While
this change allows for disabling plugins globally, a refinement of the
same logic will allow a given Job to enable/disable some of Avocado's
functionality.
Signed-off-by: NCleber Rosa <crosa@redhat.com>

When no C compiler is available, these tests will produce errors.
Let's skip them when the requirements cannot be met.
Signed-off-by: NCleber Rosa <crosa@redhat.com>

The --dry-run replaces the test class with a special one instead. When
the original class used a different but "test" methodName this fails
during __init__ as unittest requires the methodName attribute to exists.
This patch always changes the methodName to "test", which is defined in
the fake dry-run class.
Signed-off-by: NLukáš Doktor <ldoktor@redhat.com>

Now replay jobs will respect the original job failfast option, when used. Also,
it is possible to disable failfast option on replay jobs with `--failfast off`.
Signed-off-by: NAmador Pahim <apahim@redhat.com>

With this option, a test that ends with a 'bad' status will make
the job to be interrupted.
Signed-off-by: NAmador Pahim <apahim@redhat.com>

Signed-off-by: NAmador Pahim <apahim@redhat.com>

There are two places, where runner can modify the test's status, each
handling it differently. Let's create a method to handle such occasions
in the same manner.
Signed-off-by: NLukáš Doktor <ldoktor@redhat.com>

Currently when the test reports status avocado does not check whether
the process finished, leaving open space for left-over test processes.
This patch unifies the test process handling in case the test reported
test status and allows the runner to destroy and report error in case
the test process did not finish.

Additionally it modifies the error message in case the test was
interrupted, because the original message is IMO not clear enough (I
spent quite a time analyzing how that happened and if that's an avocado
bug or supported result).
Signed-off-by: NLukáš Doktor <ldoktor@redhat.com>

The result-dir of the
test_basic.RunnerSimpleTest.test_kill_stopped_sleep depends on `sleep`
binary location. This patch uses the `astring.string_to_safe_path`
function to determine the location.
Signed-off-by: NLukáš Doktor <ldoktor@redhat.com>

Currently avocado adds custom handlers of SIGINT and SIGUSR1 in order to
produce traceback and notify about user interaction or timeout. This
could be missleading in case the test uses those signals and potentially
dangerous as some tests assume default behavior.

This patch removes the custom handling of SIGINT and SIGUSR1 and
reports the failure in `job.log`. Additionally it tries to inject the
error message in the test output, if status.logfile available.

In order to keep the useful traceback in case of interruption, this
patch overrides the default SIGTERM handler. The default behavior of
SIGTERM is to die, our custom handler raises SystemExit with info saying
the test was interrupted by sigterm, which should generate traceback and
finish. The runner then changes the result to INTERRUPTED, so even
when the test modifies the SIGTERM handler, we get the correct status.

Worth mentioning that in case test ignores SIGTERM, SIGKILL is emitted
by the runner, so this should be safe approach.
Signed-off-by: NLukáš Doktor <ldoktor@redhat.com>

This patch fixes and improves resolving of the test aliases (paths
relative to `data_dir.get_tests_dir`).

1. It removes the magic `.py` suffix, which prevented non-python files
   from being resolved (simple.sh was resolved to $tests/simple.sh.py)
2. It adds the feature to filter the class/method from the test
   reference by parsing $test_reference:$class.$method for implicit
   resolver. (previously this worked only for absolute path resolver)

Visible changes are:

* `avocado run passtest` => fails (used to work)
* `avocado run passtest.py` => works (did not work)
* `avocado run env_variables.sh` => works (did not work)
* `avocado run passtest.py:PassTest.test` => works (did not work)
Signed-off-by: NLukáš Doktor <ldoktor@redhat.com>

This release introduces a new versioning scheme, so the "big bump"
from 0.34.0 to 35.0 is not such a big bump.

Some additional changes with regards to versioning were added, such as
the way the remote runner checks for Avocado on the remote host.
Signed-off-by: NCleber Rosa <crosa@redhat.com>