avocado.core.multiplexer: Assign tag to match multiplex variant

avocado.core.runner: Do better Ctrl+C handling

avocado: Two UI improvements

"avocado multiplex" and "avocado run" should share the same variant
numbers. This patch forces multiplexer to set the tag when generating
multiple variants. This way the tag is always set accordingly to the
variant.

When multiple tests of the same name are executed, the first argument is
the multiplexed variant number and additional `.$num` is added to
generate unique name.
Signed-off-by: NLukáš Doktor <ldoktor@redhat.com>

When the user disconnects from the GDB, test resumes and it might take
quite a long time. This patch notifies user that avocado is aware of him
leaving the GDB session and of resuming the test execution.
Signed-off-by: NLukáš Doktor <ldoktor@redhat.com>

The test log compounds of "avocado.test" and the "$ROOT" logger. We must
follow the same when displaying --show-job-log.
Signed-off-by: NLukáš Doktor <ldoktor@redhat.com>

Due to successive refactors in the test runner mechanism,
we did not notice that our Ctrl+C handling during job
execution went south. This patch fixes it by moving the
interruption handling to a place where we can actually
break free from the test looping. I also made the interrupt
messages cleaner, while keeping them informative.

The behavior "don't kill test subprocesses unless the user
really wants it" was also kept. You can test this by
running the multiplex test (large collection of tests):

$ scripts/avocado run examples/tests/multiplextest.py --multiplex examples/tests/multiplextest.py.data/multiplextest.yaml
JOB ID     : 783b06e69da4bbc964709748c12f014b65ef5b52
JOB LOG    : /home/lmr/avocado/job-results/job-2015-07-29T02.05-783b06e/job.log
JOB HTML   : /home/lmr/avocado/job-results/job-2015-07-29T02.05-783b06e/html/results.html
TESTS      : 1440
(1/1440) examples/tests/multiplextest.py:MultiplexTest.test: PASS (0.00 s)
...
(16/1440) examples/tests/multiplextest.py:MultiplexTest.test.15: PASS (0.00 s)
(17/1440) examples/tests/multiplextest.py:MultiplexTest.test.16:  ^C
Interrupt requested. Waiting 3 seconds for test to finish (ignoring new Ctrl+C until then)
RESULTS    : PASS 16 | ERROR 0 | FAIL 0 | SKIP 1424 | WARN 0 | INTERRUPT 0
TIME       : 0.12 s
Signed-off-by: NLucas Meneghel Rodrigues <lmr@redhat.com>

[V2] avocado: Shorter app output

avocado.utils.astring: Improve the tabular_output

This patch doesn't generate the trailing whitespaces rather then
removing them from the output afterwards, which was potentially
dangerous as the line might start with the whitespace and got displaced.
Signed-off-by: NLukáš Doktor <ldoktor@redhat.com>

Instead of putting outputs on separate lines, condensate
test results summary into a single line. The new output
looks like:

$ avocado run passtest
JOB ID     : f2f5060440bd57cba646c1f223ec8c40d03f539b
JOB LOG    : /home/user/avocado/job-results/job-2015-07-27T17.13-f2f5060/job.log
JOB HTML   : /home/user/avocado/job-results/job-2015-07-27T17.13-f2f5060/html/results.html
TESTS      : 1
(1/1) passtest.py:PassTest.test: PASS (0.00 s)
RESULTS    : PASS 1 | ERROR 0 | FAIL 0 | SKIP 0 | WARN 0 | INTERRUPT 0
TIME       : 0.00 s

We updated a few unittests in order to not depend on the
looks of the human output anymore, since unless we are
specifically testing for human output behavior, the unittests
should use machine readable output. Also, the documentation
was updated to reflect the new output layout.
Signed-off-by: NLucas Meneghel Rodrigues <lmr@redhat.com>

Use the new iter_tabular_output generator in the
avocado listing related commands, since displaying
each line of the iterator is already what we want
in the avocado listing commands.
Signed-off-by: NLucas Meneghel Rodrigues <lmr@redhat.com>

We did notice that listing commands have trailing whitespace
at their ends. Trailing whitespaces are bad for program output
(copy/paste, among other reasons), so let's fix it.

I did notice there was no reason to not implement the base
function for the tabular output as a generator, so I've
introduced iter_tabular_output, that is a generator, and
in order to not break API unnecessarily, implemented
tabular_output in terms of iter_tabular_output.
Signed-off-by: NLucas Meneghel Rodrigues <lmr@redhat.com>

This is a rather embarassing bug. We should be testing
for equality of outputs, so assertEqual instead of
assertTrue (which in effect makes the test always pass,
regardless of the avocado output). Let's fix it.
Signed-off-by: NLucas Meneghel Rodrigues <lmr@redhat.com>

avocado.core.loader: Refactor the loader to be consistent [v3]

When we changed the policy regarding skipping tests,
we forgot to change how the gdb plugin behaves on
undefined behavior - We are not supposed to be skipping
tests during its execution anymore.

So let's fix the problem by changing the undefined
behavior exception to a TestError instead of a TestNAError.
Signed-off-by: NLucas Meneghel Rodrigues <lmr@redhat.com>

Avocado defines own logging streams, all non-matching streams should
go to the job log.

This patch enables logging right after we setup the result direcotry to
allow the job to log early messages for example from test-discovery
directly into the job log.

Additionally this patch increases the module column in logging as lots
of module names are longer than 8 chars.

As the last change this removes the "avocado.linux" stream as it not
used anywhere.
Signed-off-by: NLukáš Doktor <ldoktor@redhat.com>

Previously the "file" loader was special and always matched the URL
which might be confusing and lead to problems when developing additional
plugins.

This commit tires to define the way loaders should work. They must never
fail (unless of uncaught exception which is reported and next available
plugins is used). Instead when the URL produces broken test or even a
single corrupted subtest, consider it as URL not for me and return empty
set. The loader proxy uses next plugin to resolve this URL. When no
plugin resolves the URL user is advised to run "avocado list -V" to see
the potential problems.

The order of loader plugins was also changed. Now the file-loader is
first by default and all remaining ones are loaded by the plugin
priority. The order can be changed in setting
"plugins/loader_plugins_priority". It uses "name" variable of the each
loader plugin. Not listed plugins are loaded afterwards.

Additionally the whole loader API was simplified. Now it only accepts
one parameter - list_tests - which is 3-state:

1. ALL       - all tests including broken/incorrect ones
2. AVAILABLE - available tests (for listing purposes)
2. DEFAULT   - default tests (to run when no url given)
Signed-off-by: NLukáš Doktor <ldoktor@redhat.com>

Test loader doesn't need to know anything about the job, the only useful
argument is `args`.

Second big change is the way loader is used. This patch uses list of
registered plugins on module-level (similarly to settings or logging)
instead of custom instance and plugins registered in args dict. This way
the plugins are loaded in the order they are registered (by plugin
priority) instead of random order.
Signed-off-by: NLukáš Doktor <ldoktor@redhat.com>

To improve readability (and avoid possible clashes) this patch clean-ups
the TestLoader class and moves all file-discovery code into separate
FileLoader class. Additionally it removes the dead code left when
TestLoaderProxy was introduced.
Signed-off-by: NLukáš Doktor <ldoktor@redhat.com>

This commit disables the `avocado.app.tracebacks` logging stream by
default. This stream is for app debugging and when someone needs it
he can enable it in the source code (until we have a proper way of
enabling/disabling streams on cmdline)
Signed-off-by: NLukáš Doktor <ldoktor@redhat.com>

.travis.yaml: Migrate to container-based infrastructure

Use addons to install packages instead of sudo.
Signed-off-by: NLukáš Doktor <ldoktor@redhat.com>

This patch adds 2 decorators:

* log_calls - decorates function and when the function is calles, it logs
  details into "avocado.debug"
* log_calls_class - decorates class to log it's (static) methods calls
Signed-off-by: NLukáš Doktor <ldoktor@redhat.com>

avocado.core.exceptions: Update fail_on_error and rename to fail_on

This patch renames the fail_on_error decorator to fail_on and adds
support to supply list of exceptions. This way the decorator can be used
not only to make tests fail on generic exceptions but also to specify
parts of code with expected failures of given type.
Signed-off-by: NLukáš Doktor <ldoktor@redhat.com>

Signed-off-by: NLucas Meneghel Rodrigues <lmr@redhat.com>

This fixes #628.

Add a couple of simple shell scripts that can be listed
and executed with avocado. Let's update our documentation
to use this new dir and avoid people confused with our
wrappers.
Signed-off-by: NLucas Meneghel Rodrigues <lmr@redhat.com>

Include our newest plugin into the list of functional tests.
Signed-off-by: NLucas Meneghel Rodrigues <lmr@redhat.com>

Give to test writers the fail_on_error decorator, which
should turn any generic exception into an avocado
TestFail.

This is important for some tests that can raise
any exception during a complex operation, and any of those
exceptions can be considered a test failure.
Signed-off-by: NLucas Meneghel Rodrigues <lmr@redhat.com>

Explain that skip() can only be called from inside setUp(),
and tell the expected avocado behavior when writers don't
follow that rule.
Signed-off-by: NLucas Meneghel Rodrigues <lmr@redhat.com>

Make sure avocado complains loudly when people try to call
skip() from outside setUp() - this shouldn't be allowed in
avocado tests. Let the user know of the problem and ask them
to fix their tests. Also, update the docstring for skip(),
which will reduce the chances of test writers using it in
inappropriate places.
Signed-off-by: NLucas Meneghel Rodrigues <lmr@redhat.com>

Have the clear definition of each test status in the
ReferenceGuide.
Signed-off-by: NLucas Meneghel Rodrigues <lmr@redhat.com>

In order to keep things simple and clear, enforce the
following test results rules:

 * `PASS`: the test passed, which means all conditions
   being tested have passed.
 * `FAIL`: the test failed, which means at least one
   condition being tested has failed. Ideally, it should
   mean a problem in the software being tested has been found.
 * `ERROR`: an error happened during the test execution.
   This can happen, for example, if there's a bug in the test
   runner, in its libraries or if a resource breaks unexpectedly.
   Uncaught exceptions in the test code will also result in this
   status.
 * `SKIP`: the test runner decided a requested test should not
   be executed. This can happen, for example, due to missing
   requirements in the test environment or when there's a job
   timeout.

In order to do that, let's not make the behavior on exceptions
other than TestFail to be configurable. Therefore, the config
value uncaught_exception_result must be removed and the status
of the test always be 'ERROR'.
Signed-off-by: NLucas Meneghel Rodrigues <lmr@redhat.com>