I'm not 100% sure, but I don't see a benefit in allowing people to
override test status from inside test. They are suppose to use
assertions, or `self.fail`-like methods. With this change it'd be harder
to manually override the test status (as can be seen on updated
unittests) but we minimize the accidental write into `self.status`
variable, which can lead to confusing results.

Note we can consider adding `set_status` method to allow setting this
property, but I'd like to avoid allowing direct `self.status = ` usage.
Signed-off-by: NLukáš Doktor <ldoktor@redhat.com>

The unittest standard library on Python 2.7 an later has everything
that the unittest2 backport is supposed to have.  Let's then drop all
the conditional imports of unittest2 and stick with unittest.
Signed-off-by: NCleber Rosa <crosa@redhat.com>

There's one functional test that is missing "--sysinfo=off".  I
happened to find this because lspci on my system hangs when running
under X.  That test would timeout at the sysinfo collection.
Signed-off-by: NCleber Rosa <crosa@redhat.com>

To provide support for passing arguments for simple tests in command
line, we were creating some cases where the test reference could not be
properly handled. Also, for test references with white-spaces, we have
an inconsistent behavior. See reference.

This patch removes the support for arguments in test references,
restoring the proper behavior, which is expected to be shell-like.

As consequence, we now handle correctly white-spaces in tests references
as well as Unicode characters.

Reference: https://www.redhat.com/archives/avocado-devel/2016-November/msg00011.html
Reference: https://trello.com/c/bfE9NBblSigned-off-by: NAmador Pahim <apahim@redhat.com>

We've been seen failures on resource constrained environments on some
tests that are either time sensitive and/or assume a some computing
resources will be available.

Environments such as TravisCI and RPM build servers may not have the
resources these tests expect, so let's move them to runs of `make
check-full`.  This will hopefully put an end to the false positives
we've been getting on TravisCI.
Signed-off-by: NCleber Rosa <crosa@redhat.com>

It has been some time now since replay feature accepts the job to be
identified by its results directory. Now it's time to remove the
--replay-data-dir option, since this feature became redundant.
Signed-off-by: NAmador Pahim <apahim@redhat.com>

Recent pylint update is more pedantic about module level spacing. There
are no changes to code, only couple of extra spaces to make it happy.
Signed-off-by: NLukáš Doktor <ldoktor@redhat.com>

It's been a while since the "Test ID RFC" came out:

https://www.redhat.com/archives/avocado-devel/2016-March/msg00024.html

Still, we've been referring to test URLs all around our code
and also on user visible strings.  This is an attempt to rename
the mentions of "URLs" that really should be "test references"
or simply "references" at this point.
Signed-off-by: NCleber Rosa <crosa@redhat.com>

Some functional tests, most of which execute a full avocado command,
were missing the `--job-results-dir`, pointing to a directory that
would be removed after the tests.  This has led to the pollution of
the standard job results directory (usually `~/avocado/job-results`)
when selftests are run.

This proposal fixes all but one set of functional tests, which has
to be fixed with a different set of changes (see reference).

Besides the functional tests, some unittests are also creating
job results that should be discarded.  It's believed that at least
some of this is caused by the relationship between tests and jobs,
but this is speculation and it's not attepted to be fixed on this
patch.

Reference: https://trello.com/c/I2ieqhR5Signed-off-by: NCleber Rosa <crosa@redhat.com>

The selftests.functional.test_basic.PluginsTest.test_plugin_order
executes the tests 3 times, for the last time only to check it does not
reports failure. Let's just add this missing plugin into the previous
check. If error occurs, one can always check manually which issue is
that (compare to thousands of executions where it goes well).
Signed-off-by: NLukáš Doktor <ldoktor@redhat.com>

Over the time some non-optimal "avocado run" executions accumulated.
This fixes (hopefully) all of them by adding `--sysinfo` and
`--job-results-dir`.
Signed-off-by: NLukáš Doktor <ldoktor@redhat.com>

The archive (ZIP) feature has no reason for reason for being
a core feature, as it maps nicely to the result plugin interface.

Let's turn that into a proper (and quite simple) plugin.

Also, let's add a functional test for both the archive plugin, and for
the ordered execution configuration using the zip archive for that.
Signed-off-by: NCleber Rosa <crosa@redhat.com>

Let's separate the check for requirements and the test code according to
the project's understanding of setup/test phases.

Reference: https://trello.com/c/mSrMLO0KSigned-off-by: NAmador Pahim <apahim@redhat.com>

Recently I fixed a bug where test which asks for input hangs for
infinity but I forgot to add unittest for it. Let's add it now.
Signed-off-by: NLukáš Doktor <ldoktor@redhat.com>

While the behavior of having a plugin enabled once it's installed
has suited Avocado reasonably well, there are valid use cases that
require plugins to be disabled.

While adding this mechanism, let's define what the Avocado fully
qualified name for a given plugin is.

While not an urgent feature, enabling and disabling plugins is
actually one requirement for the implementation of the Job API.  While
this change allows for disabling plugins globally, a refinement of the
same logic will allow a given Job to enable/disable some of Avocado's
functionality.
Signed-off-by: NCleber Rosa <crosa@redhat.com>

When no C compiler is available, these tests will produce errors.
Let's skip them when the requirements cannot be met.
Signed-off-by: NCleber Rosa <crosa@redhat.com>

The --dry-run replaces the test class with a special one instead. When
the original class used a different but "test" methodName this fails
during __init__ as unittest requires the methodName attribute to exists.
This patch always changes the methodName to "test", which is defined in
the fake dry-run class.
Signed-off-by: NLukáš Doktor <ldoktor@redhat.com>

Now replay jobs will respect the original job failfast option, when used. Also,
it is possible to disable failfast option on replay jobs with `--failfast off`.
Signed-off-by: NAmador Pahim <apahim@redhat.com>

With this option, a test that ends with a 'bad' status will make
the job to be interrupted.
Signed-off-by: NAmador Pahim <apahim@redhat.com>

Signed-off-by: NAmador Pahim <apahim@redhat.com>

There are two places, where runner can modify the test's status, each
handling it differently. Let's create a method to handle such occasions
in the same manner.
Signed-off-by: NLukáš Doktor <ldoktor@redhat.com>

Currently when the test reports status avocado does not check whether
the process finished, leaving open space for left-over test processes.
This patch unifies the test process handling in case the test reported
test status and allows the runner to destroy and report error in case
the test process did not finish.

Additionally it modifies the error message in case the test was
interrupted, because the original message is IMO not clear enough (I
spent quite a time analyzing how that happened and if that's an avocado
bug or supported result).
Signed-off-by: NLukáš Doktor <ldoktor@redhat.com>

The result-dir of the
test_basic.RunnerSimpleTest.test_kill_stopped_sleep depends on `sleep`
binary location. This patch uses the `astring.string_to_safe_path`
function to determine the location.
Signed-off-by: NLukáš Doktor <ldoktor@redhat.com>

Currently avocado adds custom handlers of SIGINT and SIGUSR1 in order to
produce traceback and notify about user interaction or timeout. This
could be missleading in case the test uses those signals and potentially
dangerous as some tests assume default behavior.

This patch removes the custom handling of SIGINT and SIGUSR1 and
reports the failure in `job.log`. Additionally it tries to inject the
error message in the test output, if status.logfile available.

In order to keep the useful traceback in case of interruption, this
patch overrides the default SIGTERM handler. The default behavior of
SIGTERM is to die, our custom handler raises SystemExit with info saying
the test was interrupted by sigterm, which should generate traceback and
finish. The runner then changes the result to INTERRUPTED, so even
when the test modifies the SIGTERM handler, we get the correct status.

Worth mentioning that in case test ignores SIGTERM, SIGKILL is emitted
by the runner, so this should be safe approach.
Signed-off-by: NLukáš Doktor <ldoktor@redhat.com>

This patch fixes and improves resolving of the test aliases (paths
relative to `data_dir.get_tests_dir`).

1. It removes the magic `.py` suffix, which prevented non-python files
   from being resolved (simple.sh was resolved to $tests/simple.sh.py)
2. It adds the feature to filter the class/method from the test
   reference by parsing $test_reference:$class.$method for implicit
   resolver. (previously this worked only for absolute path resolver)

Visible changes are:

* `avocado run passtest` => fails (used to work)
* `avocado run passtest.py` => works (did not work)
* `avocado run env_variables.sh` => works (did not work)
* `avocado run passtest.py:PassTest.test` => works (did not work)
Signed-off-by: NLukáš Doktor <ldoktor@redhat.com>

This release introduces a new versioning scheme, so the "big bump"
from 0.34.0 to 35.0 is not such a big bump.

Some additional changes with regards to versioning were added, such as
the way the remote runner checks for Avocado on the remote host.
Signed-off-by: NCleber Rosa <crosa@redhat.com>

This commit implements the serialized test ids described in the
Introduce proper test IDs RFC.

https://www.redhat.com/archives/avocado-devel/2016-March/msg00024.html

It implements `TestName` class, which contains the test uid, test name
and the variant uid and allows querying for the file-system-friendly
name.

The workflow is:

1. tests are discovered, name is translated to "Test Name" by loader
2. test_suite is passed to the runner (new) along with the number of
   tests+variants to be executed (used to get number of digits)
3. the Mux (params generator) yields the template + (new) variant id
4. the runner replaces template['name'] to TestName(uid, test_name,
   variant_id); where uid is currently no executed tests, test_name is
   the original name from Loader and variant_id is either None or the
   variant index.

This commit makes the `tag` argument unused. To avoid problems a warning
is issued on it's usage so we can remove it in the upcoming releases.
Signed-off-by: NLukáš Doktor <ldoktor@redhat.com>

The xunit plugin claims to be compatible with jenkins-ci junit-4.xsd,
but it does not pass on the validation dur to the wrong key name 'skip'.

This patch changes the key name to 'skipped' and adds a unittest to validate
the xunit output against the junit-4.xsd schema.

The junit-4.xsd file is coming from:
https://svn.jenkins-ci.org/trunk/hudson/dtkit/dtkit-format/dtkit-junit-model/src/main/resources/com/thalesgroup/dtkit/junit/model/xsd/junit-4.xsd

Reference: https://trello.com/c/ZdEb0TblSigned-off-by: NAmador Pahim <apahim@redhat.com>

When an invalid python code is present in a test class, avocado crashes
badly. This patch adds a TestLoaderError class to be used in those cases,
so we use it to fake the invalid test class, fail the test with ERROR estatus
and report the exception in job log.

Reference: https://trello.com/c/3zDIjTuYSigned-off-by: NAmador Pahim <apahim@redhat.com>

When avocado fails early in the process (for example in
_setup_environment_variables) it crashes, because the test status is
None. Other example could be user-defined status, which is not inside
user-facing-statuses.

This patch modifies the "status" to "ERROR" when test reports
unsupported status and sets related values to provide details regarding
this failure.
Signed-off-by: NLukáš Doktor <ldoktor@redhat.com>

When a job is timed out during a test execution, we fail the test
and put status ERROR in the test. The job then exits with the rc
AVOCADO_TESTS_FAIL. This patch fixes this, making the test status
INTERRUPTED and the job to exit with AVOCADO_JOB_INTERRUPTED.

Also, when a job is timed out before a test, the test is skipped
and the job exits with rc AVOCADO_ALL_OK. For that case, this patch
makes the job to exit with AVOCADO_JOB_INTERRUPTED instead, keeping
the test status as SKIP.

Given this change, now we have the following combinations of test status
and job return code:

Case1:
    - Test1: PASS
    - Test2: SKIP (TestTimeoutSkip)

    Job RC: AVOCADO_JOB_INTERRUPTED

Case2:
    - Test1: PASS
    - Test2: INTERRUPTED (TestTimeoutInterrupted)
    - Test3: SKIP (TestTimeoutSkip)

    Job RC: AVOCADO_JOB_INTERRUPTED

Case3:
    - Test1: PASS
    - Test2: FAIL
    - Test3: INTERRUPTED (TestTimeoutInterrupted)
    - Test4: SKIP (TestTimeoutSkip)

    Job RC: AVOCADO_JOB_INTERRUPTED
Signed-off-by: NAmador Pahim <apahim@redhat.com>

In case no tests are found we advice users to run `avocado list -V`
followed by list of urls, which does not really work with no-urls given.
This patch adds second message in case the user does not provide urls
(eg. is using the `--vt-config` option).
Signed-off-by: NLukáš Doktor <ldoktor@redhat.com>

A test base on the external runner feature depends on both the
external runner and the test URL. These two are always combined to
generate the complete command that will be executed by Avocado.

But, the Avocado command line application does not require URLs to be
given on the command line, because these may come from other sources
on loaders other than the basic FileLoader.

So, let's also explicitly require a URL to be given to to the External
Runner loader to return a test factory (class and parameters).
Signed-off-by: NCleber Rosa <crosa@redhat.com>

While working on the avocado logging system we thought it'd be
consistent and better to log certain streams to sys.stderr rather than
sys.stdout. It turned out to be a bit confusing and unexpected, so after
a discussion this commit adjusts the streams to:

 * avocado.app - loglevel <= info goes to stdout
               - loglevel > info goes to stderr
 * other streams - go always into stdout
 * sys.stdout - goes to sys.stdout when enabled
 * sys.stderr - goes to sys.stderr when enabled
              - goes to sys.stdout when enabled as test output
Signed-off-by: NLukáš Doktor <ldoktor@redhat.com>

Given the last CI jobs, RunnerSimpleTest:test_kill_stopped_sleep() has
been failing often. The obvious reason, given the nature of the test
and the heavily shared and loaded computing resources they run on, is
the lack of resources.

One example of such as failure:

   ======================================================================
   FAIL: test_kill_stopped_sleep (selftests.functional.test_basic.RunnerSimpleTest)
   ----------------------------------------------------------------------
   Traceback (most recent call last):
     File "/home/travis/build/avocado-framework/avocado/selftests/functional/test_basic.py", line 549, in test_kill_stopped_sleep
       % proc.get_output())
   AssertionError: Avocado process still alive 1s after job-timeout:
   JOB ID     : 5c65a06b4ab15e69d6c4a1fc762368e0e99cbc1f
   JOB LOG    : /tmp/avocado_selftests.functional.test_basicHIyhV4/job-2016-03-09T15.08-5c65a06/job.log
   TESTS      : 1
    (1/1) /bin/sleep 60:
   ctrl+z pressed, stopping test (5046)
   ERROR

So, let's give 4 extra seconds for the process to be
terminated. Hopefully failures in the timeout handling will still be
revealed given the relatively small buffer.
Signed-off-by: NCleber Rosa <crosa@redhat.com>

The configurations related to the "data_dir" are currently read at
module load time. This means that changes in the configuration between
module load time and changes to the settings (such as parsing an
additional config file) is never seen.

Since the avocado command line application parses extra configuration
files (given with `--config`) after the module is loaded, these extra
configuration files are never applied to the data_dir configuration.

There's one more issue with regards to the the settings usage: because
once references to the `settings` instance singleton are grabbed, they
will always point to the same settings objects. The data_dir unittests
exercise changes to the settings objects by replacing that
instance. So, let's refer to the settings using the full location
(module.attribute), which will always give back the "current" (be it
original or replaced) settings instance.
Signed-off-by: NCleber Rosa <crosa@redhat.com>

This check "avocado -v" produces correct output and returns 0.
Signed-off-by: NLukáš Doktor <ldoktor@redhat.com>

The `--silent` argument is valid for all commands, let's add it to
the core avocado args. For backward compatibility keep supporting it in
"run" subcommand too.
Signed-off-by: NLukáš Doktor <ldoktor@redhat.com>

This commit changes the idiom on (hopefully all) except statements,
from:
   except foo, details:

to:
   except foo as details:

Which is compatible with both Python 2 and 3.
Signed-off-by: NCleber Rosa <crosa@redhat.com>

This patch changes the way logging is initialized in avocado app:

1. stdout/stderr/logging is replaced with in-memory handlers
2. when arg parsing is done, logging is reconfigured accordingly to
   setting and re-logs stored messages (when streams are enabled)
3. when job starts and "test" stream is enabled, "" and "avocado.test"
   streams are enabled
4. when job finishes and "test" stream enabled the additional outputs,
   the additional outputs are disabled again.

The 3 and 4 is necessarily as the "" logger contains all logs including
avocado initialization (stevedore, PIL, ...). This way the "test" stream
still outputs everything, but only during the job-execution and not
during avocado initialization and cleanup. (one can use "early" log to
see those).

Last but not least it keeps "avocado.app" output in stdout (and stderr),
but it moves all other outputs to stderr instead. This is IMO better
handling as one can distinguish between debug and "additional" output.
Signed-off-by: NLukáš Doktor <ldoktor@redhat.com>