README.rst: mention make develop (fixes #1496)

Plugins: add mechanism to disable installed plugins

Handle result plugin failures / HTML result fix with older pystache

selftests.checkall: Adjust the behavior and add parallel check [v3]

Python 2.6 / EL6 fixes [v2]

Before you can run the script straight out of the run tree you need to
run a make develop step so let's make it clear.
Signed-off-by: NAlex Bennée <alex.bennee@linaro.org>

Using avocado.utils.process to run the journalctl commands makes avocado
to ignore the KeyboardInterrupt signal and consider the test as PASS due
to the later (sysinfo post) command executed successfully.

This patch changes the method to run the journalctl commands to
subprocess.check_output. Now Avocado seems to handle KeyboardInterrupt
nicely.

Reference: https://trello.com/c/dNgrjMOJSigned-off-by: NAmador Pahim <apahim@redhat.com>

Currently the fail-fast works per section (pylint style, unittests, ...)
but this patch enables it also inside the unittests, therefor it
finishes with the first failed unittest finishing even quicker than
before.
Note that this option is incompatible with the parallel check.
Signed-off-by: NLukáš Doktor <ldoktor@redhat.com>

This patch allows running the tests in parallel, when
AVOCADO_PARALLEL_CHECK env variable is set. It spawns 2xno_cpu workers
(or less when there are not enough tests), captures the output and shows
only output of failed workers. Before the exit it also reports the
number of executed tests in the same fashion as normal unittest
execution (without the number of failed tests, though).

On my laptop this decreases the duration from 125s to 28s.
Signed-off-by: NLukáš Doktor <ldoktor@redhat.com>

This patch adds summary at the end of the execution and uses colors to
mark start and finish of each section. This might produce some
unexpected characters on redirections/unsupported terminals at the
beginning and endings of the highlighted lines, but in most cases it
looks way better and those +3 chars can be ignored by humans.
Signed-off-by: NLukáš Doktor <ldoktor@redhat.com>

The setuptools entry point namespace, currently being show in the
output of the plugin listing, is not that relevant to users.

Let's use the plugin types (as documented in the section about the
fully qualified name of plugins), which is more relevant to Avocado
users.
Signed-off-by: NCleber Rosa <crosa@redhat.com>

While the behavior of having a plugin enabled once it's installed
has suited Avocado reasonably well, there are valid use cases that
require plugins to be disabled.

While adding this mechanism, let's define what the Avocado fully
qualified name for a given plugin is.

While not an urgent feature, enabling and disabling plugins is
actually one requirement for the implementation of the Job API.  While
this change allows for disabling plugins globally, a refinement of the
same logic will allow a given Job to enable/disable some of Avocado's
functionality.
Signed-off-by: NCleber Rosa <crosa@redhat.com>

The sleeptenmin example test contains `self.report_state` which should
not be called directly and it defines variable which is not used.
Signed-off-by: NLukáš Doktor <ldoktor@redhat.com>

We have been running the selftests during the build of the Fedora
packages, but have skipped that on EL6 (and EL7).  The reason is that
there were a few packages missing from EPEL.

They're now present on the EPEL 6 repos so let's enable checks while
building EL6 packages.

Some functional tests are failing on EL7, and fixing those and
enabling checks for EL7 packages will be the scope of further work.
Signed-off-by: NCleber Rosa <crosa@redhat.com>

When no C compiler is available, these tests will produce errors.
Let's skip them when the requirements cannot be met.
Signed-off-by: NCleber Rosa <crosa@redhat.com>

Older python-sphinx cannot handle newer inventory files (objects.inv).
While this is not fail proof, as there can be newer python-sphinx on
older Python, but it seems to be good enough as it works on all system
setups I tested.
Signed-off-by: NCleber Rosa <crosa@redhat.com>

Some older versions of sphinx do not ship with `sphinx-apidoc`.
Instead of breaking the document generation and selftests, let's just
skip the generation of API docs.
Signed-off-by: NCleber Rosa <crosa@redhat.com>

And not based on Python version.  It was observed that platforms based
on Python 2.6, such as EL6, do ship with setuptools that has support
for `--user`.  This eases considerably the development environment
on these platforms.
Signed-off-by: NCleber Rosa <crosa@redhat.com>

This will change the behavior one gets with `make develop` so that
optional (but in-tree) plugins will be enabled by default.  For now,
this means enabling the HTML result plugin.

This also means that `make check` will also test those by default,
as it's already been done on Travis CI.
Signed-off-by: NCleber Rosa <crosa@redhat.com>

Commit 3154794d broke the HTML report
when older some versions of pystache is used.  The current status is
that the HTML report generated on those systems is a one line report
containing the literal `utf8` string.

This proposal does not try to add support Unicode based inputs with
that version of pystache, because it's internally not well supported.
Internal methods such as `Template.render_tags()` will blindly try to
do conversions without respecting the encoding passed on `render()`.

Also, it's true that we're missing Unicode support in many other
places, such as documented here in this card:

 https://trello.com/c/bfE9NBbl

For now, let's try properly generate the report, and keep the error
handling already present, that let's users know there have been
encoding errors.

References: https://trello.com/c/8j3aVUlISigned-off-by: NCleber Rosa <crosa@redhat.com>

The original idea of the custom `map_methods` implementation
in the Job Pre/Post dispatcher was to overload the existing
`map_method` method.   But, I wrote it in the plural, and there's
no reason to do so.

Let's put them back to the more standard name.
Signed-off-by: NCleber Rosa <crosa@redhat.com>

Previously, a crash in one result plugin could abort the execution
of other plugins and consequently the last parts of the avocado
command line application execution.

Let's handle exceptions that could be raised there, as it's already
being done with Job Pre/Post plugins.
Signed-off-by: NCleber Rosa <crosa@redhat.com>

The `checkall` script should not try to develop/install/clean anything.
It's suppose to run the tests and it's upto the user to prepare the
environment they want to check.

This won't affect `make check` as it already cleans/develops before
executing checkall.
Signed-off-by: NLukáš Doktor <ldoktor@redhat.com>

Currently the `make check` goes through all checks and then reports
overall results. This might take a while and is not really `git bisect`
friendly. This patch changes it to report early unless
SELF_CHECK_CONTINUOUS is specified. This is used in `travis` as we are
interested in all failures.
Signed-off-by: NLukáš Doktor <ldoktor@redhat.com>

avocado.core.test: Always use "test" as methodName in dry-run

selftests: Use LC_ALL=C in tests which parse output

docs: Add note about args vs. positional args [v2]

Add link to the Avocado tests for Scylla Clusters

* https://github.com/avocado-framework/avocado:
  avocado/core/job.py: remove unused method _filter_test_suite

The filtering of test methods is now implemented at the loader module.
This is an unused leftover from older versions, so let's remove it.
Signed-off-by: NCleber Rosa <crosa@redhat.com>

* https://github.com/avocado-framework/avocado:
  Added version file and updated the respective files

Added version file and updated the respective files

Currently one of the test_output tests depends on a correct English
message, but the test runs under the system locales. Let's force the
LC_ALL for that execution.
Signed-off-by: NLukáš Doktor <ldoktor@redhat.com>

The --dry-run replaces the test class with a special one instead. When
the original class used a different but "test" methodName this fails
during __init__ as unittest requires the methodName attribute to exists.
This patch always changes the methodName to "test", which is defined in
the fake dry-run class.
Signed-off-by: NLukáš Doktor <ldoktor@redhat.com>

The python's argparse is quite clever and usually evaluates the
positionals and arguments correctly. But with subparsers or weird test
names it might falsely evaluate test name as argument, for example:

    avocado run --my_test
    avocado run passtest.py --show-job-log failtest.py

those will fail as --my_test looks like an invalid option and
--show-job-log is subparser's option not known to the main parser but as
the main parser already filled positionals it evaluates the
`failtest.py` as extra argument.

The most appropriate way to distinguish between args and positional args
is the `--` separator:

    avocado run -- --my_test
    avocado run --show-job-log -- passtest.py failtest.py
Signed-off-by: NLukáš Doktor <ldoktor@redhat.com>

* https://github.com/avocado-framework/avocado:
  selftests/checkall: skip `.git` directory on inspekt based checks

`.git` may hold content that ends up being checked by inspektor, such
as the `git-rerere` cache.  This may cause false positives, so let's
skip `.git` completetly from those checks.
Signed-off-by: NCleber Rosa <crosa@redhat.com>