New feature: silent (human) test result.

Implement new option --silent, to mute the human test result.
It has no effect with --json or --xunit options.

Example:

$ avocado --silent run failtest
$ echo $?
$ 1
Signed-off-by: NRuda Moura <rmoura@redhat.com>

Interrupt fixes

Turns out some subprocesses might misbehave (see tests
like trinity - run on a VM), so it's not enough to send
the interrupt signal. For a clean stop output wise,
SIGKILL is necessary.

As a side effect, the previous Ctrl+C can be resurrected,
this time renamed to _interrupt, since we're not really
simulating Ctrl+C.
Signed-off-by: NLucas Meneghel Rodrigues <lmr@redhat.com>

This way we avoid the bug that was fixed on the previous commit.
Signed-off-by: NLucas Meneghel Rodrigues <lmr@redhat.com>

This way we can correctly catch tests that just timed out.
Signed-off-by: NLucas Meneghel Rodrigues <lmr@redhat.com>

So we don't trigger multiprocess logging inadvertedly.
Just fill out the instance with appropriate data.
Signed-off-by: NLucas Meneghel Rodrigues <lmr@redhat.com>

avocado: Fix install and execution under virtual environment

This ends up making the subprocesses to receive the
SIGINT multiple times, leading to the multiprocess
logging to misbehave.

Also, remove the ctrl+c test, given that we couldn't
figure out a way to properly reproduce a SIGINT passed
to the process group of the running avocado instance.
Signed-off-by: NLucas Meneghel Rodrigues <lmr@redhat.com>

The use of virtual environments is a very useful feature
for developers and helps us to sanity check code instalability.
However, we were not able to use it because the data file
install has to be different for virt env and for regular
system wide install (RPMs).

This patch makes avocado aware of the execution mode,
and sets the install process so that it works differently
(and fine) for both normal install and virtenv install.

Additionally, this fixes a bug I introduced on installing
test files, that broke the avocado rpms.
Signed-off-by: NLucas Meneghel Rodrigues <lmr@redhat.com>

Sprint 11

Since we are not running short anytime of error code definitions, I
believe we can give a more descriptive return code when jobs are
actually interrupted.
Signed-off-by: NCleber Rosa <crosa@redhat.com>

That is, within a second, if the test process is running, the user
will see a "full cycle" of the throbber.
Signed-off-by: NCleber Rosa <crosa@redhat.com>

The command now is 'datadir'.
Signed-off-by: NLucas Meneghel Rodrigues <lmr@redhat.com>

Inside the job result instance dir, introduce a 'test-results'
directory that holds the results of all the other test instances.
This organizes things better and is a standard dir name,
since now we mimic a the origin test directory structure.

Update the docs accordingly.
Signed-off-by: NLucas Meneghel Rodrigues <lmr@redhat.com>

Signed-off-by: NLucas Meneghel Rodrigues <lmr@redhat.com>

Signed-off-by: NLucas Meneghel Rodrigues <lmr@redhat.com>

synctest fails on the CI VM, let's use the simpler,
bogus datadir test to do that functional verification.
Signed-off-by: NLucas Meneghel Rodrigues <lmr@redhat.com>

Create a test that uses files from the data dir, as
a functional test, so we notice when that functionality
is broken.
Signed-off-by: NLucas Meneghel Rodrigues <lmr@redhat.com>

Verify if plugins are correctly listed and none are
disabled.
Signed-off-by: NLucas Meneghel Rodrigues <lmr@redhat.com>

Order the available plugins and use colors to better
organize the information displayed:

$ scripts/avocado plugins
Plugins loaded:
    datadir - Implements the avocado 'datadir' subcommand (Enabled)
    json - JSON output (Enabled)
    journal - Test journal (Enabled)
    multiplexer - Implements the avocado 'multiplex' subcommand (Enabled)
    plugins_list - Implements the avocado 'plugins' subcommand (Enabled)
    run_vm - Run tests on a Virtual Machine (Enabled)
    sysinfo - Collect system information (Enabled)
    test_lister - Implements the avocado 'list' subcommand (Enabled)
    test_runner - Implements the avocado 'run' subcommand (Enabled)
    xunit - xUnit output (Enabled)
Signed-off-by: NLucas Meneghel Rodrigues <lmr@redhat.com>

Add a test that checks the behavior of the 'list' subcommand.
Signed-off-by: NLucas Meneghel Rodrigues <lmr@redhat.com>

Show test aliases and full paths on the output of
the 'list' command:

$ scripts/avocado list
Tests dir: /home/lmr/Code/avocado.lmr/tests
    Alias         Path
    gendata       /home/lmr/Code/avocado.lmr/tests/gendata.py
    synctest      /home/lmr/Code/avocado.lmr/tests/synctest.py
    linuxbuild    /home/lmr/Code/avocado.lmr/tests/linuxbuild.py
    fiotest       /home/lmr/Code/avocado.lmr/tests/fiotest.py
    skiptest      /home/lmr/Code/avocado.lmr/tests/skiptest.py
    warntest      /home/lmr/Code/avocado.lmr/tests/warntest.py
    doublefail    /home/lmr/Code/avocado.lmr/tests/doublefail.py
    timeouttest   /home/lmr/Code/avocado.lmr/tests/timeouttest.py
    doublefree    /home/lmr/Code/avocado.lmr/tests/doublefree.py
    trinity       /home/lmr/Code/avocado.lmr/tests/trinity.py
    multiplextest /home/lmr/Code/avocado.lmr/tests/multiplextest.py
    sleeptenmin   /home/lmr/Code/avocado.lmr/tests/sleeptenmin.py
    sleeptest     /home/lmr/Code/avocado.lmr/tests/sleeptest.py
    errortest     /home/lmr/Code/avocado.lmr/tests/errortest.py
    failtest      /home/lmr/Code/avocado.lmr/tests/failtest.py
    abort         /home/lmr/Code/avocado.lmr/tests/abort.py
    whiteboard    /home/lmr/Code/avocado.lmr/tests/whiteboard.py
Signed-off-by: NLucas Meneghel Rodrigues <lmr@redhat.com>

Execute a test that uses the data dir (synctest), both
with alias and non alias. It will help us to capture
when there is a regression using their data files.
Signed-off-by: NLucas Meneghel Rodrigues <lmr@redhat.com>

So avocado can correctly find data files for a test that
is referenced by its alias.
Signed-off-by: NLucas Meneghel Rodrigues <lmr@redhat.com>

Instead of starting tags at 1 and compulsorily putting
them on tests (even if it is the 1st execution of the test),
start with 0 and omit it if it is only the first test.

Also, reproduce the directory structure of tests in the
logs directory. This means that a test execution of

mydir/firstlevel.py
mydir/category/secondlevel.py

Would translate in the final directory results as:

mydir/firstlevel.py/
mydir/category/secondlevel.py/

Preserving the directory structure makes things more
consistent and intuitive, while preserving a unique
test ID.
Signed-off-by: NLucas Meneghel Rodrigues <lmr@redhat.com>

Signed-off-by: NLucas Meneghel Rodrigues <lmr@redhat.com>

Signed-off-by: NLucas Meneghel Rodrigues <lmr@redhat.com>

Use a job ID that is a SHA1, similar to git, instead
of the UUID. Also, use the first 7 characters of the
job ID in the directory structure, to mirror the git
short SHA1 identifier.
Signed-off-by: NLucas Meneghel Rodrigues <lmr@redhat.com>

Separate the code that generates a job SHA1 on its own
library. It'll make things easier for unittesting and other
needs.
Signed-off-by: NLucas Meneghel Rodrigues <lmr@redhat.com>

This is a workaround for the functional suite running
slow, until there's a more proper solution.

Reduce the polling time from 1 second to 0.01 second,
so short lived tests end quicker.
Signed-off-by: NLucas Meneghel Rodrigues <lmr@redhat.com>

Try as much as possible to turn timestamps into the
ISO format. In fact, iso requires hours and minutes
to be separated by a ':', but that makes URLs far
less convenient, requiring escape characters. So
let's replace it with a dot.

Also, put the short ID after the date, so date sort
can be properly done.
Signed-off-by: NLucas Meneghel Rodrigues <lmr@redhat.com>

Make some changes to bring the concept of job ID as a
user visible information:

* Make the human output to display the job ID
* Make the json output to have the job ID
* Record the job id in a file inside the results dir

Adapt functional tests to that new schema.
Signed-off-by: NLucas Meneghel Rodrigues <lmr@redhat.com>

Another change to clarify the purpose of each directory.
Signed-off-by: NLucas Meneghel Rodrigues <lmr@redhat.com>

Instead of using 'debug log' we can use 'job log', as it
is a human readable log of the job execution.
Signed-off-by: NLucas Meneghel Rodrigues <lmr@redhat.com>

We used 'run' as a prefix for avocado job output directories
after virt-test (basically, inertia). Let's use something
better:

job-[first 8 characters of job id]-timestamp

This way the job id starts to be more prominent in the
human interface as well.
Signed-off-by: NLucas Meneghel Rodrigues <lmr@redhat.com>

We can aim for a more terse output than what we have
today. This is my proposal to simplify things a bit,
while still being clear.
Signed-off-by: NLucas Meneghel Rodrigues <lmr@redhat.com>

Run avocado in a long test, interrupt it and verify if
the runner displays the appropriate behavior, in terms
of exit code and output.
Signed-off-by: NLucas Meneghel Rodrigues <lmr@redhat.com>

If we press Ctrl+C, the test that was being executed
still does not have a status, and that reflects in the
job log with a None printed in the final status.

So, if there's no status yet, we should safely (?)
assume that the test was interrupted.
Signed-off-by: NLucas Meneghel Rodrigues <lmr@redhat.com>

Handle KeyboardInterrupt inside avocado, making
Ctrl+C to terminate the program, instead of generating
a python backtrace.
Signed-off-by: NLucas Meneghel Rodrigues <lmr@redhat.com>