This selftest makes sure test statuses are assessed correctly on the
test end inside jsonresult (which also verifies TestResult handling).

Additionally it makes sure jsonresult handles unknown statuses and
missing values in the status correctly.
Signed-off-by: NLukáš Doktor <ldoktor@redhat.com>

The TestResult baseclass keeps the count of executed tests and
additionally on test end it calls function associated with the finish
status. This is not used by any plugin as they usually don't care. The
only one which could benefit from it is xunit, but it uses simplified
version. I think it's time to say good bye to this and simplify the
TestResult class (along with minor speedup)
Signed-off-by: NLukáš Doktor <ldoktor@redhat.com>

Add test as "ERROR" when the status is unknown.
Signed-off-by: NLukáš Doktor <ldoktor@redhat.com>

The time related variables cause troubles, when the test is interrupted.
Let's initialize them to 0 so they can be used in result postprocessing
if the test is interrupted between early_status and finish.
Signed-off-by: NLukáš Doktor <ldoktor@redhat.com>

TestResult depends on the test status containing all information. The
problem is that on interruption the status might not be complete. Let's
provide understandable defaults for such cases.
Signed-off-by: NLukáš Doktor <ldoktor@redhat.com>

Currently all classes inherited from TestResult are storing the full
state per each executed test, although there is actually none using it.
Those who need additional information store them again in their
variables. Let's avoid wasting of memory and only store the counts
instead.
Signed-off-by: NLukáš Doktor <ldoktor@redhat.com>

Signed-off-by: NCleber Rosa <crosa@redhat.com>

Signed-off-by: NCleber Rosa <crosa@redhat.com>

avocado.spec: add python-lxml to build dependencies

Add short section about GPG signing (optional).
Signed-off-by: NLukáš Doktor <ldoktor@redhat.com>

Signed-off-by: NCleber Rosa <crosa@redhat.com>

avocado.core.remoter remote actions should fail on prompt [v3]

When avocado is run out of the source tree, it ignores the user local
configuration (~/.config/avocado/avocado.conf).  IMHO this is both
inconsistent and bad for development sessions.
Signed-off-by: NCleber Rosa <crosa@redhat.com>

Avocado is not interactive. Remote can prompt for interaction for
a number of reasons: unknown host key, password request and so on.

This patch makes Avocado to fail when interaction for remote access is
needed, forwarding the correspondent Fabric's message for the user.

Reference: https://trello.com/c/GEa8BY2jSigned-off-by: NAmador Pahim <apahim@redhat.com>

Commit 8dad284b introduced a requirement on python-lxml. Let's reflect
that on the SPEC file.
Signed-off-by: NCleber Rosa <crosa@redhat.com>

Signed-off-by: NCleber Rosa <crosa@redhat.com>

We're substituting `os.path` with flexmock, let's not forget to clean-up
properly.
Signed-off-by: NLukáš Doktor <ldoktor@redhat.com>

only the basename of the `self.filename` has to be shorter than
maxsize-5, not the full path.

Also the IOError in selftest is rather misleading as it relates to
missing source file. Let's just check the path is correctly set and
additionally verify `_record_reference_*` is not crashing when filename
not associated.
Signed-off-by: NLukáš Doktor <ldoktor@redhat.com>

Plugin Improvements

With the expansion of the types of plugins, and some of them only
used in specific circurstances, they are loaded at different times.

When a plugin fails to load, wherever it happens, Avocado should log
an error. Let's extract the function that does that from the
Application and make it a utility function to be reused elsewhere.
Signed-off-by: NCleber Rosa <crosa@redhat.com>

Given the upcoming expansion of plugin type, there may be no active
plugins for a given type of plugin.

If that's the case, let's not just print a header about the plugin
type, but also about the lack of active plugins for that given type.
Signed-off-by: NCleber Rosa <crosa@redhat.com>

Switch the two blocks of code for a loop around the list of plugin
dispatchers and descriptions. This will allow for easier printing
of extra plugin types when they are introduced.
Signed-off-by: NCleber Rosa <crosa@redhat.com>

We have a base Plugin class, so let's mark that one as an abstract
class, instead of individual inherited classes.

Also, remove useless __init__() methods.
Signed-off-by: NCleber Rosa <crosa@redhat.com>

Currently, exit codes cover all situations with one single value.

This patch makes exit codes to be ORable, so we can better represent
the variety of things that happens during an Avocado execution.

Reference: https://trello.com/c/SU5fixgHSigned-off-by: NAmador Pahim <apahim@redhat.com>

avocado.core.xunit fix XUnit results fail XSD validation [v2]

The xunit plugin claims to be compatible with jenkins-ci junit-4.xsd,
but it does not pass on the validation dur to the wrong key name 'skip'.

This patch changes the key name to 'skipped' and adds a unittest to validate
the xunit output against the junit-4.xsd schema.

The junit-4.xsd file is coming from:
https://svn.jenkins-ci.org/trunk/hudson/dtkit/dtkit-format/dtkit-junit-model/src/main/resources/com/thalesgroup/dtkit/junit/model/xsd/junit-4.xsd

Reference: https://trello.com/c/ZdEb0TblSigned-off-by: NAmador Pahim <apahim@redhat.com>

avocado.core.loader fix crash on loader exception [v4]

avocado.code.result fix results.json on job interruption

Currently UI shows coherent results, reporting tests that where not
executed due to an interruption as SKIP. This information is not in
line with results.json, which does not account un-executed tests.

This patch changes the _reconcile() call from HumanTestResult() class
to its parent class, TestResult(). That way all classes inheriting from
TestResult() will have proper results to report.

Reference: https://trello.com/c/8CXFl13MSigned-off-by: NAmador Pahim <apahim@redhat.com>

When an invalid python code is present in a test class, avocado crashes
badly. This patch adds a TestLoaderError class to be used in those cases,
so we use it to fake the invalid test class, fail the test with ERROR estatus
and report the exception in job log.

Reference: https://trello.com/c/3zDIjTuYSigned-off-by: NAmador Pahim <apahim@redhat.com>

Handle long filenames in avocado and fix unknown test status [v4]

The leading "." makes files invisible on unix systems, which might be
confusing to users. Let's replace with "_".
Signed-off-by: NLukáš Doktor <ldoktor@redhat.com>

This adds selftests regarding the long filenames fixes.
Signed-off-by: NLukáš Doktor <ldoktor@redhat.com>

When avocado fails early in the process (for example in
_setup_environment_variables) it crashes, because the test status is
None. Other example could be user-defined status, which is not inside
user-facing-statuses.

This patch modifies the "status" to "ERROR" when test reports
unsupported status and sets related values to provide details regarding
this failure.
Signed-off-by: NLukáš Doktor <ldoktor@redhat.com>