Signed-off-by: NCleber Rosa <crosa@redhat.com>

A finer level of granularity may help to extend our testing coverage
while reducing the amount of false positives.  This introduces the
three different levels, mapped like this:

   * level 0, AKA "make check"
   * level 2, AKA "make check-full"

There are no changes of test assignment, that is, tests previously
being run under "check-full" will continue to be run only at that
target.  The same is true for tests that would run under "make check"
before this.

The big change is that there's now a middle ground, that can be
activated by manually setting the AVOCADO_CHECK_LEVEL variable.  Level
1 is intended to be used be used on environments that are halfway
between a dedicated machine and a really low powered environment.
Signed-off-by: NCleber Rosa <crosa@redhat.com>

Signed-off-by: NLukáš Doktor <ldoktor@redhat.com>

* https://github.com/avocado-framework/avocado:
  Human interface: do not show tests execution time
  Human Interface: show job execution time
  Job: add time accounting
  Job Pre/Post plugins: move them to the appropriate location

This is a UI tweak, as suggested during review, to remove the tests
time.

It's kept separately, because it can be easily dropped if any reason
to keep it arises.
Signed-off-by: NCleber Rosa <crosa@redhat.com>

A long standing task is to show the job execution time, instead of the
sum of tests execution time.  After some major work, this is now
possible.

This change adds the job execution time to the human interface UI, in
addition to the test execution time.  Some users may prefer the job
execution time alone, while others may prefer all the time statistics
on a single line.

Since we cannot please everyone, this version just adds it.  I believe
UI tweaks can follow.
Signed-off-by: NCleber Rosa <crosa@redhat.com>

It may be useful to users of the Job class to have information on how
much time it took for it to run.  Since the goal for the Job class is
to be flexible when it comes to its usage, it may not be possible to
track the execution time if users follow a different path.

For now, the time accouting is done automatically if users use the
`run()` method.

If advanced users of the Job class choose to set the start time, end
time and elapsed time themselves, they are free to do it and `run()`
will never overwrite it.
Signed-off-by: NCleber Rosa <crosa@redhat.com>

The JobPre and JobPost plugin interfaces were intended to be run
before and after the job, but in reality they have been executed
at different points *within* the job execution.

This puts the execution of those plugins at their original intended
location:

 * Plugins implementing JobPre will be executed right after the Job
   instance is created, *before any other action*, such as the
   creation of the test suite.

 * Plugins implementing JobPost will be executed right after the Job
   instance `run()` method.

 * Plugins implementing `Result` will be executed after JobPost ones.

Users relying on the old location please take notice.
Signed-off-by: NCleber Rosa <crosa@redhat.com>

Signed-off-by: NCleber Rosa <crosa@redhat.com>

The tests seems to be stable after the series of improvements applied to
our selftests procedures. Time to restore this check in travis.

Reference: https://trello.com/c/yXtgDRjkSigned-off-by: NAmador Pahim <apahim@redhat.com>

Signed-off-by: NLukáš Doktor <ldoktor@redhat.com>

* https://github.com/avocado-framework/avocado:
  Extended vendors_map dict for Power8

Added a new item for vendors_map dic to def get_cpu_vendor_name():
Signed-off-by: NVenkat R B <vrbagal1@linux.vnet.ibm.com>

Signed-off-by: NAmador Pahim <apahim@redhat.com>

The recently added CANCEL color is not cleaned when we disable TERM
support.
Signed-off-by: NLukáš Doktor <ldoktor@redhat.com>

Signed-off-by: NAmador Pahim <apahim@redhat.com>

Signed-off-by: NLukáš Doktor <ldoktor@redhat.com>

* https://github.com/avocado-framework/avocado:
  run-kvm-unit-test.sh: enable options for configuring kvm-unit-tests

support configuring kvm-unit-test with supported arguments for enabling tests to
run in PowerPC.
Signed-off-by: NBalamuruhan S <bala24@linux.vnet.ibm.com>

This commit adds "--replay-resume" argument, which is basically a
symlink to "--replay-test-status INTERRUPTED", therefor it executes only
the tests which were interrupted, or not executed (as we use INTERRUPTED
in Replay internally to represent not-executed-tests).

As the "results.json" used to get the list of tests might not be
available on a system crash, this also adds a fallback to use
"results.tap", which is a machine readable, streamline format. It does
not represent all available test statuses, but as a fallback it works
well.
Signed-off-by: NLukáš Doktor <ldoktor@redhat.com>

Closing the file only flushes the file buffer, but does not sync the
data on disk. Let's ensure the job id, other jobdata and the results.tap
are force-flushed on disk.
Signed-off-by: NLukáš Doktor <ldoktor@redhat.com>

The "results.json" includes only executed tests in "tests" while the
non-executed tests are only included in the overall info like "total".
Let's append "UNKNOWN" tests at the end of the "tests" to represent
those non-executed tests and mark them as "INTERRUPTED".

According to our meanings they'd be "SKIP" tests, but this is local
representation for Replay only and we would not be able to distinguish
between SKIP and SKIP-AFTER-INTERRUPTION, which is super handy. To avoid
introducing new/artifical status, let's use INTERRUPTED which works for
our scenario and in the future we might expand it if needed.
Signed-off-by: NLukáš Doktor <ldoktor@redhat.com>

Signed-off-by: NCleber Rosa <crosa@redhat.com>

Signed-off-by: NAmador Pahim <apahim@redhat.com>

Currently we only support tags in test classes. This patch adds the
tags feature to the test methods. Test methods will iherit the tags from
its class.
Signed-off-by: NAmador Pahim <apahim@redhat.com>

Let's create a helper function to host the code that is often used to
generate the test methods list.

Also, the tags creation was moved up be be executed only once.

This is a preparation commit to the code that will introduce the
tags-per-function feature.
Signed-off-by: NAmador Pahim <apahim@redhat.com>

Signed-off-by: NCleber Rosa <crosa@redhat.com>

Signed-off-by: NCleber Rosa <crosa@redhat.com>

Signed-off-by: NCleber Rosa <crosa@redhat.com>

The real acronym that properly identifies Red Hat Enteprise Linux is
"rhel".  Let's use that as the result of the distro detection.
Signed-off-by: NCleber Rosa <crosa@redhat.com>

The current probe searches only for 'Red Hat'.  Since this is
clearly about Red Hat Enterprise Linux, let's include the full
string.
Signed-off-by: NCleber Rosa <crosa@redhat.com>

Variants other than the "Server" edition of the Red Hat Enterprise
Linux distro would fail detection because the current probe
specifically searches for "Server".

Other variants current in existence include "Client", "ComputeNode",
"Container", etc.  AFAICT, only a single word is used here.
Signed-off-by: NCleber Rosa <crosa@redhat.com>

Signed-off-by: NLukáš Doktor <ldoktor@redhat.com>

* https://github.com/avocado-framework/avocado:
  selftests: remove python shebangs

Previous commit d5fd51cf attempted to remove all shebangs, as a result
of 143b574c that remove the executable bit of all tests.

There are still a few left, so this let's clean them up.
Signed-off-by: NCleber Rosa <crosa@redhat.com>

The word "multiplex" is only valid as a multiplexation concept used by
yaml_to_mux. Everywhere else it should be either varianter or test
parameter system.
Signed-off-by: NLukáš Doktor <ldoktor@redhat.com>

The "avocado multiplex" command is misleading as multiplexation is only
the generic concept used by one of our varianter plugins
implementations. Let's change the command name to "avocado variants"
which better reflects what is it used for.

As people depend on "avocado multiplex" command let's keep both of them
for now to give people time till Avocado 52.0 to migrate.
Signed-off-by: NLukáš Doktor <ldoktor@redhat.com>

Now that the variant IDs are not a sequential number, we need this extra
information so users can easily know how many variants were generated.

Reference: https://trello.com/c/N7pB9E4OSigned-off-by: NAmador Pahim <apahim@redhat.com>

Signed-off-by: NAmador Pahim <apahim@redhat.com>

This adds the RFC on LTS to the documentation for future reference.
Signed-off-by: NCleber Rosa <crosa@redhat.com>

Signed-off-by: NCleber Rosa <crosa@redhat.com>

After proposals discussions on the mailing list, RFCs shape the
current or future status of Avocado.  It makes sense to archive
them for future reference.

Also some RFCs are constantly referred in other parts of the
documentation, so it makes sense to allow cross references between
them.
Signed-off-by: NCleber Rosa <crosa@redhat.com>

Signed-off-by: NCleber Rosa <crosa@redhat.com>

Signed-off-by: NLukáš Doktor <ldoktor@redhat.com>

* https://github.com/avocado-framework/avocado:
  skip exceptions cleanup
  skip classes: internal improvements