The moment where we load the test in avocado is critical,
since it happens on a subprocess, a failure there is
opaque to the user, and the test early state never
gets populated, which makes avocado hang forever.

Handle this case better, and print an exception right
away to the user.

Changes from v1:
 * Improved error message, per Cleber's comments.
Signed-off-by: NLucas Meneghel Rodrigues <lmr@redhat.com>

Refactor the test code and create a library containing
stack trace related utilities for avocado. They'll be
useful in a later patch, that improves visibility of
errors in the test loading stage.

Changes from v1:
 * Changed module name from tb to stacktrace, per rmoura's suggestion.
Signed-off-by: NLucas Meneghel Rodrigues <lmr@redhat.com>

Signed-off-by: NLucas Meneghel Rodrigues <lmr@redhat.com>

plugins.remote: Improve json parsing

avocado: Add another branch in test detection heuristics

If we stop a stand alone test execution in the middle, we'll
have troubles since there's still no available test ID to
use in the json. Let's fetch that information in advance
from the file system.

Also, let's avoid a division by zero when no tests were
finished.

Changes from v1:

* Use the runtime module info (Cleber's work)
* Evaluate the pass rate in a cleaner fashion (Cleber's suggestion)
Signed-off-by: NCleber Rosa <crosa@redhat.com>
Signed-off-by: NLucas Meneghel Rodrigues <lmr@redhat.com>

We already have a reference in the runtime module to the currently
running test, but, even though we could reference the job from the
currently running test, the job may still be running after all tests
finish.

So, let's add an explicit reference to the currently running job.

Notice: IMHO, there's no need to define when a job finishes running
and clear it from the runtime module. It can be argued that another
job would only replace the current one if/when its run() method
would be called.
Signed-off-by: NCleber Rosa <crosa@redhat.com>

Avocado sometimes produces unexpected texts (failures) before
the actual json output. This patch tries to parse the output
for the correct line instead of using first line.

Note: Avocado should not produce any strings when --json -
specified. Anyway having this logic might be still useful.
Signed-off-by: NLukáš Doktor <ldoktor@redhat.com>

It occurred to us that we can tell apart a buggy avocado
test from a random buggy module since all avocado instrumented
tests will have an avocado import. Therefore, let's do a regexp
search on the module to see if we find an avocado import header.

Of course, this still has room for mistakes, but this is a
best effort enterprise anyway. Update the diagram with the
extra branch in the test resolution.
Signed-off-by: NLucas Meneghel Rodrigues <lmr@redhat.com>

docs: Improving test resolution diagram

Change the phrasing in some blocks of the test
resolution diagram so the meaning is more clear.

For example, 'Is it an importable module?' is more
aptly phrased as 'Can avocado import it?' and leaves
less room for confusion.
Signed-off-by: NLucas Meneghel Rodrigues <lmr@redhat.com>

[V2] Improve test load

Explain to the user how avocado resolves a path and decides
whether it is a test, and what kind of test it is. Use a diagram
to clarify the decision process used.
Signed-off-by: NLucas Meneghel Rodrigues <lmr@redhat.com>

This will do functional testing of avocado behaving
according to the use cases specified.
Signed-off-by: NLucas Meneghel Rodrigues <lmr@redhat.com>

Make sure avocado is behaving the way we expect on different
situations:

 * Simple executable test
 * Non executable (not a test)
 * Simple avocado test
 * Simple avocado test (non executable)
 * Buggy test
 * Buggy test (non executable)
 * Py module
 * Py module (executable)
Signed-off-by: NLucas Meneghel Rodrigues <lmr@redhat.com>

Now that we don't enforce a silly correspondence test
module name -> class name, we can call our test classes
whatever we want as long as it inherits from avocado.test.Test.

Let's then rename our tests to something more CamelCase.
Signed-off-by: NLucas Meneghel Rodrigues <lmr@redhat.com>

When avocado analyzes a path that looks like a python module,
first it'll try to import the module, then look for an appropriate
test class inside this module. Improve the test load heuristics
by looking for classes that inherit from avocado.test.Test, and
not the rudimentary detection, inspired by autotest.

If the module import went fine, but we simply can't find an
avocado test inside of it, if the file is executable we'll
try to execute it as a simple test. If it's not executable,
it'll be considered as not a test (same count as not_found tests).
If we can't import the module due to a test bug, once again we'll
check if the file is executable - if executable, we'll execute it
as a simple test, if not, we'll consider it a buggy test.

Unfortunately there's no way to tell apart a buggy avocado test
and a random py module with a bug, so both will be treated the same.
Adjust code to reflect this new status.
Signed-off-by: NLucas Meneghel Rodrigues <lmr@redhat.com>

The code was unnecessarily convoluted. Be simple and
straightforward.
Signed-off-by: NLucas Meneghel Rodrigues <lmr@redhat.com>

Fix compatibility problem with vm and remote plugins and htmlresult feature.

At the time it seemed a good idea to have an exit_code.AVOCADO_CRASH
integer, but since we also use it for option validation errors, it's
misleading a good deal of the time. Let's use a more generic
AVOCADO_FAIL, update usages and description.
Signed-off-by: NLucas Meneghel Rodrigues <lmr@redhat.com>

Due the changes on RemoteTest class, use the attributes start and end.
Signed-off-by: NRudá Moura <rmoura@redhat.com>

Fix compatibility with htmlresult on remote and vm plugins:

* Use time_start/start and time_end/end attributes.
* Move self.result.end_tests() close to tear_down() in order
  to have sysinfo files unarchived.
Signed-off-by: NRudá Moura <rmoura@redhat.com>

Signed-off-by: NRudá Moura <rmoura@redhat.com>

Signed-off-by: NRudá Moura <rmoura@redhat.com>

Component Isolation: Linux Distribution Support [v1]

[V1] Redirect output fix

Verify if we are not printing terminal codes inside
redirected outputs, when we in fact shouldn't.
Signed-off-by: NLucas Meneghel Rodrigues <lmr@redhat.com>

We were printing a throbber even though output was being
redirected, mainly because throbber was not following the
same rules as the rest of the colored output. Fix that
problem.
Signed-off-by: NLucas Meneghel Rodrigues <lmr@redhat.com>

Trivial misc license/doc commits

The file was introduced by commit e1363982, but
is missing a proper license header.

Minor improvement to the initial description in the man page
and update links to point to the (new) project page everywhere.
Signed-off-by: NAdemar de Souza Reis Jr <areis@redhat.com>

Avocado itself is released under GPLv2 (explicit version), but most of the code
is currently licensed under GPLv2+, so make that clear in the documentation.
Signed-off-by: NAdemar de Souza Reis Jr <areis@redhat.com>

avocado.runner: optimize the run of quick tests

Signed-off-by: NLucas Meneghel Rodrigues <lmr@redhat.com>

If the test is very quick, it may run in less than 0.1 secs, which is the
minimum time we wait for the result to be available in TestRunner::run_suite().

Since utils.wait_for() accepts a "first" parameter, documented as "Time to sleep
before first attempt", let's use it with a very aggressive low value, just
enough for the main process to be put to sleep by the scheduler and give time
for the other process to run.

Benchmark using multiplextest:
    $ TESTS=""; for f in $(seq 1 100); do TESTS="$TESTS examples/tests/multiplextest.py"; done
    $ time scripts/avocado run $TESTS > /dev/null

Before this change:
    real    0m12.114s
    user    0m1.549s
    sys     0m1.038s

After:
    real    0m2.639s
    user    0m1.065s
    sys     0m0.764s
Signed-off-by: NAdemar de Souza Reis Jr <areis@redhat.com>

Now that we gather values from the config files, 'Current'
becomes more proper term.
Signed-off-by: NLucas Meneghel Rodrigues <lmr@redhat.com>

As a subcommand (option, really) of the plugin command. The generated
distro file is supposed to be portable and be named canonically based
on the name, version, release and arch.

Changes from v0:
 * Return proper exit status when required command line arguments are
   missing
Signed-off-by: NCleber Rosa <crosa@redhat.com>