The test_basic.RunnerOperationTest.test_early_latest_result waits for
link creation for 0.5s and then it simply fails, without waiting for the
process. The problem is that in such case the tmpdir can be cleaned
before it's used by the created process and therefor left behind.
Signed-off-by: NLukáš Doktor <ldoktor@redhat.com>

Basically switch to use the alias provided by six, with the Python
3 name.
Signed-off-by: NCleber Rosa <crosa@redhat.com>

This changes uses the six compatibility equivalents for iterkeys,
iteritems and itervalues.
Signed-off-by: NCleber Rosa <crosa@redhat.com>

Windows does not allow `<>:"/\|?*` characters in the filename. Let's
include these in our `astring.string_to_safe_path` method and use it
everywhere we create files.
Signed-off-by: NLukáš Doktor <ldoktor@redhat.com>

The html plugin uses entry-point "avocado_result_html", but the pkg name
is "avocado-framework-plugin-result-html" and that is what
"pkg_resources.require" accepts.

This required few tweaks related to recent changes to html plugin which
produces a single result file instead of multiple files.
Signed-off-by: NLukáš Doktor <ldoktor@redhat.com>

The basedir was mainly set correctly, but sometimes not used in some of
the tests. Let's move the `os.chdir` to `setUp` phase where it makes
sense and add new ones where they were not present.

Related to basedir the test_mux was using hardcoded path based on the
method used to execute the test, let's change it to `__file__` which
gives better results.
Signed-off-by: NLukáš Doktor <ldoktor@redhat.com>

These scripts rely on some Python library path manipulation to be
able to load from the source tree.  But, in fact that are a few
reasons for not having those, including:

 * All functionality in `avocado` relies on plugins, which will not be
   available from the source tree unless a `$ setup.py develop` is
   executed, which automatically makes the "import magic" unnecessary.

 * Most users will end up using a setuptools generated "entry point"
   instead.

For the sake of less code, I propose this (and all other?) magic to be
removed.
Signed-off-by: NCleber Rosa <crosa@redhat.com>

Found by the newest version of inspektor, 0.4.5.
Signed-off-by: NLucas Meneghel Rodrigues <lookkas@gmail.com>

Signed-off-by: NPaolo Bonzini <pbonzini@redhat.com>

There are bunch of really short, one-time-used variables defined
globally, which is not really necessary, nor easy to follow. Let's just
define those one/two liners directly to where they are used.
Signed-off-by: NLukáš Doktor <ldoktor@redhat.com>

The fail_reason is the instance of the actual exception. We do store it
only as string in case of old-type exception, but even new style
exceptions might be not available when sys.path is different causing
unpickle on runner part impossible.
Signed-off-by: NLukáš Doktor <ldoktor@redhat.com>

When no variants are provided Avocado sets the default variant, but
instead of list of mux paths it created just string "/run", which
processed into ["/", "r", "u", "n"]. Let's fix this by setting list.

Also the proper mux_path should be `/run/*` in order to include all
children nodes of the "/run".
Signed-off-by: NLukáš Doktor <ldoktor@redhat.com>

Just like `avocado list` presents the statistics for the test types,
it may be useful to also list the number of tests that have a
particular tag.
Signed-off-by: NCleber Rosa <crosa@redhat.com>

Test tags (via docstring directives) have been supported in Avocado
for a number of releases, but up until now user would only be able to
see the tags on their tests by resorting to the source code.

This was already difficult when the tags where set only at the class
level, now that they can be set both on the class and method
docstring, users would have to do the "math" on their minds.

Let's display the test tags when the verbose switch is given to the
list command to make the life of users easier.
Signed-off-by: NCleber Rosa <crosa@redhat.com>

Since too may reports of false positives have been generated during
RPM package builds, let's move all test sensitive tests into level 1.

This means that those tests will not be run on RPM package builds, but
will continue to run on Travis-CI.
Signed-off-by: NCleber Rosa <crosa@redhat.com>

Introducing the --force-execution to create the Job regardless the fact
that some test references might not be resolved to tests.

Reference: https://trello.com/c/bDUJjhq0Signed-off-by: NAmador Pahim <apahim@redhat.com>

A finer level of granularity may help to extend our testing coverage
while reducing the amount of false positives.  This introduces the
three different levels, mapped like this:

   * level 0, AKA "make check"
   * level 2, AKA "make check-full"

There are no changes of test assignment, that is, tests previously
being run under "check-full" will continue to be run only at that
target.  The same is true for tests that would run under "make check"
before this.

The big change is that there's now a middle ground, that can be
activated by manually setting the AVOCADO_CHECK_LEVEL variable.  Level
1 is intended to be used be used on environments that are halfway
between a dedicated machine and a really low powered environment.
Signed-off-by: NCleber Rosa <crosa@redhat.com>

Due to the deprecation of the self.skip(), the availability of the skip
decorators and the new CANCEL status, some internal improvements are
important to keep our behaviour sane and our own code sound and clean.

- The test.SkipTest class was renamed to test.MockingTest to be even more
  generic, intending to be overridden by sub-classes that make the test
  to end both with SKIP or CANCEL status. To keep it generic,
  test.MockingTest class will not SKIP the test if used directly anymore.

- The test.TimeOutSkipTest and test.ReplaySkipTest classes now are using
  the skip decorators instead of raising an exception in setUp(), since
  'skipping' the test means 'don't execute anything', not even the
  setUp().

- The test.DryRunTest class, which is expected to log itself in setUp()
  and then abort the test execution, is now using self.cancel() (instead
  of raising a SKIP exception), being now compliant with the concept
  that a SKIP test cannot execute anything.

- Selftests were adjusted accordingly.
Signed-off-by: NAmador Pahim <apahim@redhat.com>

Currently there is a hardcoded test-postprocess timeout consisting of
two cycle-timeouts which is something between 1 and 2 seconds. This is
not sufficient on heavily loaded machines so this patch increases the
timeouts to:

 1s: when test was interrupted (ctrl+c/timeout)
10s: when the process died but the status was not yet delivered
60s: when test reported status but the process did not finish

taking into account the current available test/job timeout.

As those deadlines are significantly longer, users could have noticed a
frozen throbber, which is why I propagate the results_dispatcher and
handle the throbber also during the postprocess, which makes this patch
a bit more complicated than one would expect. I used "progress = False"
to distinguish between running test and postprocessing it.
Signed-off-by: NLukáš Doktor <ldoktor@redhat.com>

Signed-off-by: NLukáš Doktor <ldoktor@redhat.com>

As it has already been done to some other tests, two other tests have
been producing false positives.  This has been observed recently in
package builds in build farms.

Let's run them only when `make check-full` is called, which is
usually done at dedicated machines with plenty of resources.

Reference: https://kojipkgs.fedoraproject.org/work/tasks/9189/19059189/build.log
Reference: https://kojipkgs.fedoraproject.org/work/tasks/9445/19069445/build.logSigned-off-by: NCleber Rosa <crosa@redhat.com>

The MissingTest is yielded by TestLoaderProxy, but it was implemented
and described in FileLoader, which caused troubles when FileLoader was
not being used. This simple fix allows TestLoaderProxy to define basic
mappings and adds MissingTest to it.
Signed-off-by: NLukáš Doktor <ldoktor@redhat.com>

Currently the `selftests/run_coverage` only reports unit coverage. This
patch allows specifying custom `avocado` command in selftests and uses
it to run coverage to also include the functional tests to results.
Signed-off-by: NLukáš Doktor <ldoktor@redhat.com>

`system-out` should be in the same level as `failure`, but we are
creating it as a `failure` child.

This patch puts system-out in the same level as `failure` and adds the
functional test to validate all the xunit outputs against the
junit-4.xsd schema.
Signed-off-by: NAmador Pahim <apahim@redhat.com>

Recently we added a check for canceled test, let's also check the
"cancel" number of tests are reported correctly in json plugin.
Signed-off-by: NLukáš Doktor <ldoktor@redhat.com>

self.skip() is to be deprecated. This is the first step, adding a
message to the exception raised by the self.skip() and adjusting the
examples tests accordingly.
Signed-off-by: NAmador Pahim <apahim@redhat.com>

On some environments, such as when building RPM packages, very little
can be assumed to be available.  `man` is such as command that is
*not* available, so let's check for it and skip tests accordingly.
Signed-off-by: NCleber Rosa <crosa@redhat.com>

Signed-off-by: NLucas Meneghel Rodrigues <lookkas@gmail.com>

I'm not 100% sure, but I don't see a benefit in allowing people to
override test status from inside test. They are suppose to use
assertions, or `self.fail`-like methods. With this change it'd be harder
to manually override the test status (as can be seen on updated
unittests) but we minimize the accidental write into `self.status`
variable, which can lead to confusing results.

Note we can consider adding `set_status` method to allow setting this
property, but I'd like to avoid allowing direct `self.status = ` usage.
Signed-off-by: NLukáš Doktor <ldoktor@redhat.com>

The unittest standard library on Python 2.7 an later has everything
that the unittest2 backport is supposed to have.  Let's then drop all
the conditional imports of unittest2 and stick with unittest.
Signed-off-by: NCleber Rosa <crosa@redhat.com>

There's one functional test that is missing "--sysinfo=off".  I
happened to find this because lspci on my system hangs when running
under X.  That test would timeout at the sysinfo collection.
Signed-off-by: NCleber Rosa <crosa@redhat.com>

To provide support for passing arguments for simple tests in command
line, we were creating some cases where the test reference could not be
properly handled. Also, for test references with white-spaces, we have
an inconsistent behavior. See reference.

This patch removes the support for arguments in test references,
restoring the proper behavior, which is expected to be shell-like.

As consequence, we now handle correctly white-spaces in tests references
as well as Unicode characters.

Reference: https://www.redhat.com/archives/avocado-devel/2016-November/msg00011.html
Reference: https://trello.com/c/bfE9NBblSigned-off-by: NAmador Pahim <apahim@redhat.com>

We've been seen failures on resource constrained environments on some
tests that are either time sensitive and/or assume a some computing
resources will be available.

Environments such as TravisCI and RPM build servers may not have the
resources these tests expect, so let's move them to runs of `make
check-full`.  This will hopefully put an end to the false positives
we've been getting on TravisCI.
Signed-off-by: NCleber Rosa <crosa@redhat.com>

It has been some time now since replay feature accepts the job to be
identified by its results directory. Now it's time to remove the
--replay-data-dir option, since this feature became redundant.
Signed-off-by: NAmador Pahim <apahim@redhat.com>

Recent pylint update is more pedantic about module level spacing. There
are no changes to code, only couple of extra spaces to make it happy.
Signed-off-by: NLukáš Doktor <ldoktor@redhat.com>

It's been a while since the "Test ID RFC" came out:

https://www.redhat.com/archives/avocado-devel/2016-March/msg00024.html

Still, we've been referring to test URLs all around our code
and also on user visible strings.  This is an attempt to rename
the mentions of "URLs" that really should be "test references"
or simply "references" at this point.
Signed-off-by: NCleber Rosa <crosa@redhat.com>

Some functional tests, most of which execute a full avocado command,
were missing the `--job-results-dir`, pointing to a directory that
would be removed after the tests.  This has led to the pollution of
the standard job results directory (usually `~/avocado/job-results`)
when selftests are run.

This proposal fixes all but one set of functional tests, which has
to be fixed with a different set of changes (see reference).

Besides the functional tests, some unittests are also creating
job results that should be discarded.  It's believed that at least
some of this is caused by the relationship between tests and jobs,
but this is speculation and it's not attepted to be fixed on this
patch.

Reference: https://trello.com/c/I2ieqhR5Signed-off-by: NCleber Rosa <crosa@redhat.com>

The selftests.functional.test_basic.PluginsTest.test_plugin_order
executes the tests 3 times, for the last time only to check it does not
reports failure. Let's just add this missing plugin into the previous
check. If error occurs, one can always check manually which issue is
that (compare to thousands of executions where it goes well).
Signed-off-by: NLukáš Doktor <ldoktor@redhat.com>

Over the time some non-optimal "avocado run" executions accumulated.
This fixes (hopefully) all of them by adding `--sysinfo` and
`--job-results-dir`.
Signed-off-by: NLukáš Doktor <ldoktor@redhat.com>

The archive (ZIP) feature has no reason for reason for being
a core feature, as it maps nicely to the result plugin interface.

Let's turn that into a proper (and quite simple) plugin.

Also, let's add a functional test for both the archive plugin, and for
the ordered execution configuration using the zip archive for that.
Signed-off-by: NCleber Rosa <crosa@redhat.com>