Signed-off-by: NCleber Rosa <crosa@redhat.com>

New optional plugins are now available, and some in RPM package
format.  Let's include them where plugin RPM packages are listed.
Signed-off-by: NCleber Rosa <crosa@redhat.com>

The suite order means the order in which the variants will be applied to
tests. Right now we run all variants of first test, then the same for
the second, third, ... tests. The new option "--execution-order" allows
to alternatively specify "tests-per-variant" mode where we run first
variant of all tests, then second variant, third, ...
Signed-off-by: NLukáš Doktor <ldoktor@redhat.com>

Currently the yaml_to_mux plugin is a core plugin without properly
defined requirements. Let's separate it to optional_plugins, define
dependencies and also build it separately in RPM.
Signed-off-by: NLukáš Doktor <ldoktor@redhat.com>

Signed-off-by: NCleber Rosa <crosa@redhat.com>

Signed-off-by: NCleber Rosa <crosa@redhat.com>

Instead of current, let's stick to the default of using the word
default, and also the option value from optparse itself.
Signed-off-by: NCleber Rosa <crosa@redhat.com>

Signed-off-by: NCleber Rosa <crosa@redhat.com>

Signed-off-by: NCleber Rosa <crosa@redhat.com>

Currently all failures are ignorred, which makes it hard to actually
understand why our beloved test-reference is not a robot test. Let's
report !ROBOT line along with the exceptions details in such case.
Signed-off-by: NLukáš Doktor <ldoktor@redhat.com>

Signed-off-by: NCleber Rosa <crosa@redhat.com>

If state['params'] is nor present, let's generate the html report
anyway.
Signed-off-by: NAmador Pahim <apahim@redhat.com>

Currently we hardcode the "avocado.app" and "avocado.test" loggers
everywhere in Avocado, let's use the newly defined
`avocado.core.output.LOG_{UI,JOB}` variables everywhere in Avocado
except of `avocado.utils`, which shouldn't have used this interface in
the first place and will be subject of change in the future.
Signed-off-by: NLukáš Doktor <ldoktor@redhat.com>

The `exceptions` module is not imported even though it's suppose to be
used.
Signed-off-by: NLukáš Doktor <ldoktor@redhat.com>

Test ID has the test UID, the Test Name and the Variant. Let's split
those three information so users can order by any of them separately.
Signed-off-by: NAmador Pahim <apahim@redhat.com>

Convenience tooltip to expose the parameters available to the test.
Signed-off-by: NAmador Pahim <apahim@redhat.com>

Signed-off-by: NCleber Rosa <crosa@redhat.com>

All optional plugins require the installation (or presence) of
"avocado-framework", so let's do that to the newly introduced
resultsdb plugin too.
Signed-off-by: NCleber Rosa <crosa@redhat.com>

The ResultsDB expects the Testcase, not the Test ID. Even so, the
Testcase can be considered the combination of the test name plus the
variant.

While on this, this patch adds the parameters information to the
ResultsDB 'data' section.
Signed-off-by: NAmador Pahim <apahim@redhat.com>

When --html is used the original in-job-dir results are removed just
before the second render. This was probably due to typo as the original
location was (probably) intended for removal. This patch doesn't do that
because the user-defined dir can be shared dir with other user files so
let's just produce the results and keep the dirs intact (while keeping
the refresh in job-dir as that is guaranteed to be our directory and we
do know it can be removed).
Signed-off-by: NLukáš Doktor <ldoktor@redhat.com>

Propagate tests results to resultsdb.

Reference: https://trello.com/c/AcruJdgS/906-result-plugin-for-resultsdbSigned-off-by: NAmador Pahim <apahim@redhat.com>

This commit modifies the optional_plugins to depend on VERSION files
located in their main directories and adds "propagate-version" target to
Makefile to update the VERSION files for optional as well as "linkable"
plugins, if VERSION file was present.
Signed-off-by: NLukáš Doktor <ldoktor@redhat.com>

Signed-off-by: NCleber Rosa <crosa@redhat.com>

Signed-off-by: NCleber Rosa <crosa@redhat.com>

Simply because this file won't be available when being installed
from any method outside the source tree.

The downside is obvious: it requires extra steps when doing a new
release.  But that is still better than not being able to ship
these plugins to users using the standard Python mechanisms.
Signed-off-by: NCleber Rosa <crosa@redhat.com>

So users will now be able to find Avocado plugins on PyPI, and install
them via `pip`.  Let's try to be consistent with the naming on those
packages, so that users don't fall into naming traps such as
installing Django related extensions.

Also, those packages map more closely to their (RPM) packaged versions.
Signed-off-by: NCleber Rosa <crosa@redhat.com>

Fixing pylint messages for the robot plugin module.
Signed-off-by: NAmador Pahim <apahim@redhat.com>

When robot plugin cannot determine the tests from the test references we
should return an empty list instead of an exception.
Signed-off-by: NAmador Pahim <apahim@redhat.com>

We have to consider that `avocado list` may be executed without the test
references.
Signed-off-by: NAmador Pahim <apahim@redhat.com>

This patch creates an optional plugin to run Robot Framework tests
inside Avocado.
Signed-off-by: NAmador Pahim <apahim@redhat.com>

This is a simple collection of syntatic fixes that work across
Python 2.7 and 3.x.
Signed-off-by: NCleber Rosa <crosa@redhat.com>

We have received many requests to support skip tests from inside the
test method. As we can do that without breaking our own concepts, we
decided to introduce the CANCEL status and the corresponding
self.cancel() method to the Test class.

Reference: https://trello.com/c/viBJIEwISigned-off-by: NAmador Pahim <apahim@redhat.com>

Currently the variant format is (variant_id, (variant, mux_path)) which
is not really scalable, nor user friendly. Let's use a dictionary
instead.
Signed-off-by: NLukáš Doktor <ldoktor@redhat.com>

The fact that we are not raising exceptions when the remote job fails is
making the local job to keep executing as if nothing bad happened.

Let's raise an exception when the remote job fails so we can inform the
local job and also show to the user the remote error message.

Reference: https://trello.com/c/kn9WpdwuSigned-off-by: NAmador Pahim <apahim@redhat.com>

We are returning the stdout and the stderr of the fabric remote
execution all together inside the CmdResult.stdout

Let's separate them properly.
Signed-off-by: NAmador Pahim <apahim@redhat.com>

This method was copied (instead of moved) to the module itself. Let's
remove it at once and avoid duplicated/unused code.
Signed-off-by: NAmador Pahim <apahim@redhat.com>

The start_tests() method from results plugins is not used anymore. It
was replaced by the pre_tests entry point in plugins_interfaces.

This patch is just a cleanup.
Signed-off-by: NAmador Pahim <apahim@redhat.com>

Our users have pointed out that the basic Avocado test runner, and its
related packaged form, has too many dependencies.  They're absolutely
right, and the whole goal of having plugins in Avocado is that
dependencies can be made optional according to the features required
by a given user.

This specific change creates three new optional plugins, containing
the extended test runners: remote, vm and docker.
Signed-off-by: NCleber Rosa <crosa@redhat.com>

I tried to strip-down the bootstrap library in order to minimize
resources size. I used brute-force based on me not being able to find a
visible difference, so I can't guarantee everything is included, but I
tried hard and couldn't see a difference.
Signed-off-by: NLukáš Doktor <ldoktor@redhat.com>

When long fail_reason is detected, we add a on_click popup with the full
description, but somehow we included string "fail_reason" instead of the
fail_reason variable.
Signed-off-by: NLukáš Doktor <ldoktor@redhat.com>