Job main loop: functional tests timing

Looks like test bots, such as travis, can have very little resources
at times and run things really slowly, example:

======================================================================
FAIL: Sleeptest is supposed to take 1 second, let's make a sandwich of
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/travis/build/avocado-framework/avocado/selftests/all/
functional/avocado/basic_tests.py", line 238, in test_runner_sleep_fail_sleep_timing
    self.assertLess(actual_time, 4.0)
AssertionError: 4.036144018173218 not less than 4.0

Let's increase the maximum expected times.
Signed-off-by: NCleber Rosa <crosa@redhat.com>

Test execution times async

Signed-off-by: NCleber Rosa <crosa@redhat.com>

As spotted by Ruda. Thanks!

CC: Ruda Moura <rmoura@redhat.com>
Signed-off-by: NCleber Rosa <crosa@redhat.com>

There's a bug in the current job mainloop that causes significant delays.
The test state is received from the test via the queue in an async fashion,
but if a proper state is received, we failt to mark the loop as done.

This patch, in conjunction with the previous reverted commit, puts the
queue timeout in sync with the UI timeout and puts the cycle timeout
back to 1s, without the adverse consequences.
Signed-off-by: NCleber Rosa <crosa@redhat.com>

This reverts commit 10e6af662844a4c9df6994bfc7536f0413fbc82d.
Signed-off-by: NCleber Rosa <crosa@redhat.com>

avocado: Add NOT_FOUND test status

To avoid confusion, when passing an invalid test ID
(typo, incorrect path, alias that could not be resolved),
introduce a new status, NOT_FOUND inside avocado, as
well as proper handling handling for such situations.

$ scripts/avocado run sbrubles
JOB ID    : 312b690173a8f5f261d9c5cefdb8699ac0798ded
JOB LOG   : /home/lmr/avocado/job-results/job-2014-08-14T10.02-312b690/job.log
TESTS     : 1
(1/1) sbrubles: NOT_FOUND (0.00 s)
PASS      : 0
ERROR     : 0
FAIL      : 0
SKIP      : 0
WARN      : 0
NOT FOUND : 1
TIME      : 0.00 s

Also, add a functional test for this new status.
Signed-off-by: NLucas Meneghel Rodrigues <lmr@redhat.com>

avocado.job: Give an error message on an empty test list

If an empty list of tests was passed, give an error validation
message and exit. Provide a functional test for it.
Signed-off-by: NLucas Meneghel Rodrigues <lmr@redhat.com>

New feature: silent (human) test result.

Implement new option --silent, to mute the human test result.
It has no effect with --json or --xunit options.

Example:

$ avocado --silent run failtest
$ echo $?
$ 1
Signed-off-by: NRuda Moura <rmoura@redhat.com>

Interrupt fixes

Turns out some subprocesses might misbehave (see tests
like trinity - run on a VM), so it's not enough to send
the interrupt signal. For a clean stop output wise,
SIGKILL is necessary.

As a side effect, the previous Ctrl+C can be resurrected,
this time renamed to _interrupt, since we're not really
simulating Ctrl+C.
Signed-off-by: NLucas Meneghel Rodrigues <lmr@redhat.com>

This way we avoid the bug that was fixed on the previous commit.
Signed-off-by: NLucas Meneghel Rodrigues <lmr@redhat.com>

This way we can correctly catch tests that just timed out.
Signed-off-by: NLucas Meneghel Rodrigues <lmr@redhat.com>

So we don't trigger multiprocess logging inadvertedly.
Just fill out the instance with appropriate data.
Signed-off-by: NLucas Meneghel Rodrigues <lmr@redhat.com>

avocado: Fix install and execution under virtual environment

This ends up making the subprocesses to receive the
SIGINT multiple times, leading to the multiprocess
logging to misbehave.

Also, remove the ctrl+c test, given that we couldn't
figure out a way to properly reproduce a SIGINT passed
to the process group of the running avocado instance.
Signed-off-by: NLucas Meneghel Rodrigues <lmr@redhat.com>

The use of virtual environments is a very useful feature
for developers and helps us to sanity check code instalability.
However, we were not able to use it because the data file
install has to be different for virt env and for regular
system wide install (RPMs).

This patch makes avocado aware of the execution mode,
and sets the install process so that it works differently
(and fine) for both normal install and virtenv install.

Additionally, this fixes a bug I introduced on installing
test files, that broke the avocado rpms.
Signed-off-by: NLucas Meneghel Rodrigues <lmr@redhat.com>

Sprint 11

Since we are not running short anytime of error code definitions, I
believe we can give a more descriptive return code when jobs are
actually interrupted.
Signed-off-by: NCleber Rosa <crosa@redhat.com>

That is, within a second, if the test process is running, the user
will see a "full cycle" of the throbber.
Signed-off-by: NCleber Rosa <crosa@redhat.com>

The command now is 'datadir'.
Signed-off-by: NLucas Meneghel Rodrigues <lmr@redhat.com>

Inside the job result instance dir, introduce a 'test-results'
directory that holds the results of all the other test instances.
This organizes things better and is a standard dir name,
since now we mimic a the origin test directory structure.

Update the docs accordingly.
Signed-off-by: NLucas Meneghel Rodrigues <lmr@redhat.com>

Signed-off-by: NLucas Meneghel Rodrigues <lmr@redhat.com>

Signed-off-by: NLucas Meneghel Rodrigues <lmr@redhat.com>

synctest fails on the CI VM, let's use the simpler,
bogus datadir test to do that functional verification.
Signed-off-by: NLucas Meneghel Rodrigues <lmr@redhat.com>

Create a test that uses files from the data dir, as
a functional test, so we notice when that functionality
is broken.
Signed-off-by: NLucas Meneghel Rodrigues <lmr@redhat.com>

Verify if plugins are correctly listed and none are
disabled.
Signed-off-by: NLucas Meneghel Rodrigues <lmr@redhat.com>

Order the available plugins and use colors to better
organize the information displayed:

$ scripts/avocado plugins
Plugins loaded:
    datadir - Implements the avocado 'datadir' subcommand (Enabled)
    json - JSON output (Enabled)
    journal - Test journal (Enabled)
    multiplexer - Implements the avocado 'multiplex' subcommand (Enabled)
    plugins_list - Implements the avocado 'plugins' subcommand (Enabled)
    run_vm - Run tests on a Virtual Machine (Enabled)
    sysinfo - Collect system information (Enabled)
    test_lister - Implements the avocado 'list' subcommand (Enabled)
    test_runner - Implements the avocado 'run' subcommand (Enabled)
    xunit - xUnit output (Enabled)
Signed-off-by: NLucas Meneghel Rodrigues <lmr@redhat.com>

Add a test that checks the behavior of the 'list' subcommand.
Signed-off-by: NLucas Meneghel Rodrigues <lmr@redhat.com>

Show test aliases and full paths on the output of
the 'list' command:

$ scripts/avocado list
Tests dir: /home/lmr/Code/avocado.lmr/tests
    Alias         Path
    gendata       /home/lmr/Code/avocado.lmr/tests/gendata.py
    synctest      /home/lmr/Code/avocado.lmr/tests/synctest.py
    linuxbuild    /home/lmr/Code/avocado.lmr/tests/linuxbuild.py
    fiotest       /home/lmr/Code/avocado.lmr/tests/fiotest.py
    skiptest      /home/lmr/Code/avocado.lmr/tests/skiptest.py
    warntest      /home/lmr/Code/avocado.lmr/tests/warntest.py
    doublefail    /home/lmr/Code/avocado.lmr/tests/doublefail.py
    timeouttest   /home/lmr/Code/avocado.lmr/tests/timeouttest.py
    doublefree    /home/lmr/Code/avocado.lmr/tests/doublefree.py
    trinity       /home/lmr/Code/avocado.lmr/tests/trinity.py
    multiplextest /home/lmr/Code/avocado.lmr/tests/multiplextest.py
    sleeptenmin   /home/lmr/Code/avocado.lmr/tests/sleeptenmin.py
    sleeptest     /home/lmr/Code/avocado.lmr/tests/sleeptest.py
    errortest     /home/lmr/Code/avocado.lmr/tests/errortest.py
    failtest      /home/lmr/Code/avocado.lmr/tests/failtest.py
    abort         /home/lmr/Code/avocado.lmr/tests/abort.py
    whiteboard    /home/lmr/Code/avocado.lmr/tests/whiteboard.py
Signed-off-by: NLucas Meneghel Rodrigues <lmr@redhat.com>

Execute a test that uses the data dir (synctest), both
with alias and non alias. It will help us to capture
when there is a regression using their data files.
Signed-off-by: NLucas Meneghel Rodrigues <lmr@redhat.com>

So avocado can correctly find data files for a test that
is referenced by its alias.
Signed-off-by: NLucas Meneghel Rodrigues <lmr@redhat.com>

Instead of starting tags at 1 and compulsorily putting
them on tests (even if it is the 1st execution of the test),
start with 0 and omit it if it is only the first test.

Also, reproduce the directory structure of tests in the
logs directory. This means that a test execution of

mydir/firstlevel.py
mydir/category/secondlevel.py

Would translate in the final directory results as:

mydir/firstlevel.py/
mydir/category/secondlevel.py/

Preserving the directory structure makes things more
consistent and intuitive, while preserving a unique
test ID.
Signed-off-by: NLucas Meneghel Rodrigues <lmr@redhat.com>

Signed-off-by: NLucas Meneghel Rodrigues <lmr@redhat.com>

Signed-off-by: NLucas Meneghel Rodrigues <lmr@redhat.com>

Use a job ID that is a SHA1, similar to git, instead
of the UUID. Also, use the first 7 characters of the
job ID in the directory structure, to mirror the git
short SHA1 identifier.
Signed-off-by: NLucas Meneghel Rodrigues <lmr@redhat.com>