The variants file is called multiplex even though we changed the
terminology long ago. Let's support both names for now and slowly move
to the correct name.
Signed-off-by: NLukáš Doktor <ldoktor@redhat.com>

This patch adds to the loader the ability to recursively discover tests
when the docstring `🥑 recursive` is specified in the test class.

This recursive discovery will walk all the way up in the recursion
chain, adding the parents test methods to the child class information.

Reference: https://trello.com/c/GtFlhcOjSigned-off-by: NAmador Pahim <apahim@redhat.com>

We rely on garbage collector to run __del__s before we check all tmpdirs
were cleaned. Let's force-run gc.collect() and (just to be sure) run
sync.
Signed-off-by: NLukáš Doktor <ldoktor@redhat.com>

The suite order means the order in which the variants will be applied to
tests. Right now we run all variants of first test, then the same for
the second, third, ... tests. The new option "--execution-order" allows
to alternatively specify "tests-per-variant" mode where we run first
variant of all tests, then second variant, third, ...
Signed-off-by: NLukáš Doktor <ldoktor@redhat.com>

Currently the yaml_to_mux plugin is a core plugin without properly
defined requirements. Let's separate it to optional_plugins, define
dependencies and also build it separately in RPM.
Signed-off-by: NLukáš Doktor <ldoktor@redhat.com>

Let's report full result on doc build failure to be able to see what is
wrong.
Signed-off-by: NLukáš Doktor <ldoktor@redhat.com>

This patch adds the missing definition of `MODE_0644` in
`LoaderTestFunctional` class.
Signed-off-by: NAmador Pahim <apahim@redhat.com>

The format of the docstring directives has not been formally described
so far.  Because at least two different features are built on that,
and possibly more will come, it's a good idea to define a format.

This introduces a stricter format, basically adding a number of
requirements or rules.  To make the following description more
consistent let's call the "🥑" part of the docstring directive
"the marker" and what follows it the "content".

1) One or more spaces are required between the "🥑" marker and
   the content.  The following is a valid docstring directive:

   "🥑 enable"

   While this is invalid:

   ":avocado:enable"

2) A more limited set of characters are now allowed in the content,
   namely one from the following set: [a-zA-Z0-9_:,=].  The following
   is a valid docstring directive:

   "🥑 foo=bar,foz=baz:extra"

   While the following is invalid:

   "🥑 foo=bar!"

3) The first character of the content must be alphanumeric, so the
   following is a valid docstring directive:

   "🥑 foo=bar"

   While the following is invalid:

   "🥑 =bar"

4) An end of string (or end of line) is now required after the content
   (which itself may not contain white spaces).  The following is a
   valid docstring directive:

   "🥑 enable"

   While the following is invalid:

   "🥑 enable FOO"
Signed-off-by: NCleber Rosa <crosa@redhat.com>

with the new 'test_statuses.py', this test became redundant.
Signed-off-by: NAmador Pahim <apahim@redhat.com>

Currently, when something goes wrong in setUp(), tearDown() is not
executed. This patch changes the behaviour of our Test class, making the
tearDown() to be always executed, regardless what happens in setUp().

Reference: https://trello.com/c/zSZ4xFwpSigned-off-by: NAmador Pahim <apahim@redhat.com>

For couple of weeks the `inspekt lint` produces unwanted `Your code has
been rated at 10.00/10 (previous run: 10.00/10, +0.00)` messages
polluting the log too much for me to bear. I tried to remove those
messages, but so far I haven't found a solution, so let's just silence
it unless there is a lint failure.
Signed-off-by: NLukáš Doktor <ldoktor@redhat.com>

Just like `avocado list` presents the statistics for the test types,
it may be useful to also list the number of tests that have a
particular tag.
Signed-off-by: NCleber Rosa <crosa@redhat.com>

Test tags (via docstring directives) have been supported in Avocado
for a number of releases, but up until now user would only be able to
see the tags on their tests by resorting to the source code.

This was already difficult when the tags where set only at the class
level, now that they can be set both on the class and method
docstring, users would have to do the "math" on their minds.

Let's display the test tags when the verbose switch is given to the
list command to make the life of users easier.
Signed-off-by: NCleber Rosa <crosa@redhat.com>

Both for consistency and for Python 3 support.
Signed-off-by: NCleber Rosa <crosa@redhat.com>

Python 3 will not allow print statements, so let's import from future
and use it as function.
Signed-off-by: NCleber Rosa <crosa@redhat.com>

Since too may reports of false positives have been generated during
RPM package builds, let's move all test sensitive tests into level 1.

This means that those tests will not be run on RPM package builds, but
will continue to run on Travis-CI.
Signed-off-by: NCleber Rosa <crosa@redhat.com>

Introducing the --force-execution to create the Job regardless the fact
that some test references might not be resolved to tests.

Reference: https://trello.com/c/bDUJjhq0Signed-off-by: NAmador Pahim <apahim@redhat.com>

The Loader can report non-tests which are not intended for execution but
only for listing purposes. Let's avoid inheriting them from avocado Test
as it is not really necessary as the tests are basically dummy entities
which does not really do anything.
Signed-off-by: NLukáš Doktor <ldoktor@redhat.com>

Attempting to close a journal that wasn't initialized is certainly a
bug.  This will check for its initialization first.
Signed-off-by: NCleber Rosa <crosa@redhat.com>

The UI was showing information about the job, no matter if the job got
to run tests.

Even though the formal available statuses of a job are not too evolved
and documented, Avocado already sets a job status as "PASS" when it
finishes running the tests.  This involves most of the job phases.

Let's use that to check if the UI should print information about the
test results and job execution time.
Signed-off-by: NCleber Rosa <crosa@redhat.com>

We used to use increased verbosity to get name of the test which left
tmp dirs behind, but it is not necessary. This commit decreases the
verbosity and only logs the error as well as test name in case this
failure is detected.
Signed-off-by: NLukáš Doktor <ldoktor@redhat.com>

This is time sensitive (currently has a 2s timeout) and has failed
during package builds.

By having it as level 1, it will still be run on Travis-CI.

Reference: https://kojipkgs.fedoraproject.org/work/tasks/4583/19204583/build.logSigned-off-by: NCleber Rosa <crosa@redhat.com>

A finer level of granularity may help to extend our testing coverage
while reducing the amount of false positives.  This introduces the
three different levels, mapped like this:

   * level 0, AKA "make check"
   * level 2, AKA "make check-full"

There are no changes of test assignment, that is, tests previously
being run under "check-full" will continue to be run only at that
target.  The same is true for tests that would run under "make check"
before this.

The big change is that there's now a middle ground, that can be
activated by manually setting the AVOCADO_CHECK_LEVEL variable.  Level
1 is intended to be used be used on environments that are halfway
between a dedicated machine and a really low powered environment.
Signed-off-by: NCleber Rosa <crosa@redhat.com>

A long standing task is to show the job execution time, instead of the
sum of tests execution time.  After some major work, this is now
possible.

This change adds the job execution time to the human interface UI, in
addition to the test execution time.  Some users may prefer the job
execution time alone, while others may prefer all the time statistics
on a single line.

Since we cannot please everyone, this version just adds it.  I believe
UI tweaks can follow.
Signed-off-by: NCleber Rosa <crosa@redhat.com>

It may be useful to users of the Job class to have information on how
much time it took for it to run.  Since the goal for the Job class is
to be flexible when it comes to its usage, it may not be possible to
track the execution time if users follow a different path.

For now, the time accouting is done automatically if users use the
`run()` method.

If advanced users of the Job class choose to set the start time, end
time and elapsed time themselves, they are free to do it and `run()`
will never overwrite it.
Signed-off-by: NCleber Rosa <crosa@redhat.com>

The tests seems to be stable after the series of improvements applied to
our selftests procedures. Time to restore this check in travis.

Reference: https://trello.com/c/yXtgDRjkSigned-off-by: NAmador Pahim <apahim@redhat.com>

This commit adds "--replay-resume" argument, which is basically a
symlink to "--replay-test-status INTERRUPTED", therefor it executes only
the tests which were interrupted, or not executed (as we use INTERRUPTED
in Replay internally to represent not-executed-tests).

As the "results.json" used to get the list of tests might not be
available on a system crash, this also adds a fallback to use
"results.tap", which is a machine readable, streamline format. It does
not represent all available test statuses, but as a fallback it works
well.
Signed-off-by: NLukáš Doktor <ldoktor@redhat.com>

The "results.json" includes only executed tests in "tests" while the
non-executed tests are only included in the overall info like "total".
Let's append "UNKNOWN" tests at the end of the "tests" to represent
those non-executed tests and mark them as "INTERRUPTED".

According to our meanings they'd be "SKIP" tests, but this is local
representation for Replay only and we would not be able to distinguish
between SKIP and SKIP-AFTER-INTERRUPTION, which is super handy. To avoid
introducing new/artifical status, let's use INTERRUPTED which works for
our scenario and in the future we might expand it if needed.
Signed-off-by: NLukáš Doktor <ldoktor@redhat.com>

Currently we only support tags in test classes. This patch adds the
tags feature to the test methods. Test methods will iherit the tags from
its class.
Signed-off-by: NAmador Pahim <apahim@redhat.com>

Previous commit d5fd51cf attempted to remove all shebangs, as a result
of 143b574c that remove the executable bit of all tests.

There are still a few left, so this let's clean them up.
Signed-off-by: NCleber Rosa <crosa@redhat.com>

The "avocado multiplex" command is misleading as multiplexation is only
the generic concept used by one of our varianter plugins
implementations. Let's change the command name to "avocado variants"
which better reflects what is it used for.

As people depend on "avocado multiplex" command let's keep both of them
for now to give people time till Avocado 52.0 to migrate.
Signed-off-by: NLukáš Doktor <ldoktor@redhat.com>

Due to the deprecation of the self.skip(), the availability of the skip
decorators and the new CANCEL status, some internal improvements are
important to keep our behaviour sane and our own code sound and clean.

- The test.SkipTest class was renamed to test.MockingTest to be even more
  generic, intending to be overridden by sub-classes that make the test
  to end both with SKIP or CANCEL status. To keep it generic,
  test.MockingTest class will not SKIP the test if used directly anymore.

- The test.TimeOutSkipTest and test.ReplaySkipTest classes now are using
  the skip decorators instead of raising an exception in setUp(), since
  'skipping' the test means 'don't execute anything', not even the
  setUp().

- The test.DryRunTest class, which is expected to log itself in setUp()
  and then abort the test execution, is now using self.cancel() (instead
  of raising a SKIP exception), being now compliant with the concept
  that a SKIP test cannot execute anything.

- Selftests were adjusted accordingly.
Signed-off-by: NAmador Pahim <apahim@redhat.com>

Currently there is a hardcoded test-postprocess timeout consisting of
two cycle-timeouts which is something between 1 and 2 seconds. This is
not sufficient on heavily loaded machines so this patch increases the
timeouts to:

 1s: when test was interrupted (ctrl+c/timeout)
10s: when the process died but the status was not yet delivered
60s: when test reported status but the process did not finish

taking into account the current available test/job timeout.

As those deadlines are significantly longer, users could have noticed a
frozen throbber, which is why I propagate the results_dispatcher and
handle the throbber also during the postprocess, which makes this patch
a bit more complicated than one would expect. I used "progress = False"
to distinguish between running test and postprocessing it.
Signed-off-by: NLukáš Doktor <ldoktor@redhat.com>

The current 2s sleep is too-time-sensitive for travis from time to
time. Let's increase the test's sleep to 10s.
Signed-off-by: NLukáš Doktor <ldoktor@redhat.com>

Signed-off-by: NLukáš Doktor <ldoktor@redhat.com>

As it has already been done to some other tests, two other tests have
been producing false positives.  This has been observed recently in
package builds in build farms.

Let's run them only when `make check-full` is called, which is
usually done at dedicated machines with plenty of resources.

Reference: https://kojipkgs.fedoraproject.org/work/tasks/9189/19059189/build.log
Reference: https://kojipkgs.fedoraproject.org/work/tasks/9445/19069445/build.logSigned-off-by: NCleber Rosa <crosa@redhat.com>

The "astring.string_safe_encode" is a method, which tries to encode the
string no-matter how wrong it is. Let's also allow broken corrupted
strings by using "ignore" to skip the corrupted chars.
Signed-off-by: NLukáš Doktor <ldoktor@redhat.com>

The MissingTest is yielded by TestLoaderProxy, but it was implemented
and described in FileLoader, which caused troubles when FileLoader was
not being used. This simple fix allows TestLoaderProxy to define basic
mappings and adds MissingTest to it.
Signed-off-by: NLukáš Doktor <ldoktor@redhat.com>

When the tests references cannot be resolved to a test_suite, the
results plugins should not output anything.

Reference: https://trello.com/c/oMQsQHC6Signed-off-by: NAmador Pahim <apahim@redhat.com>

This commit enables the left-behind-temporary-dirs check between each
test. As all selftests are executed as a single process the BORG class
keeps the TMP dir of the unit-tests. To avoid results spoiled by this
fact this commit adds a force-cleanup of the tmp dir tracker overriding
the BORG internal state, which is not really nice, but for selftests
acceptable.

The method to clean it was especially named
"unittest_refresh_dir_tracker" to emphasize it should not be used
outside of unittest.
Signed-off-by: NLukáš Doktor <ldoktor@redhat.com>