This patch fixes and improves resolving of the test aliases (paths
relative to `data_dir.get_tests_dir`).

1. It removes the magic `.py` suffix, which prevented non-python files
   from being resolved (simple.sh was resolved to $tests/simple.sh.py)
2. It adds the feature to filter the class/method from the test
   reference by parsing $test_reference:$class.$method for implicit
   resolver. (previously this worked only for absolute path resolver)

Visible changes are:

* `avocado run passtest` => fails (used to work)
* `avocado run passtest.py` => works (did not work)
* `avocado run env_variables.sh` => works (did not work)
* `avocado run passtest.py:PassTest.test` => works (did not work)
Signed-off-by: NLukáš Doktor <ldoktor@redhat.com>

This release introduces a new versioning scheme, so the "big bump"
from 0.34.0 to 35.0 is not such a big bump.

Some additional changes with regards to versioning were added, such as
the way the remote runner checks for Avocado on the remote host.
Signed-off-by: NCleber Rosa <crosa@redhat.com>

This commit implements the serialized test ids described in the
Introduce proper test IDs RFC.

https://www.redhat.com/archives/avocado-devel/2016-March/msg00024.html

It implements `TestName` class, which contains the test uid, test name
and the variant uid and allows querying for the file-system-friendly
name.

The workflow is:

1. tests are discovered, name is translated to "Test Name" by loader
2. test_suite is passed to the runner (new) along with the number of
   tests+variants to be executed (used to get number of digits)
3. the Mux (params generator) yields the template + (new) variant id
4. the runner replaces template['name'] to TestName(uid, test_name,
   variant_id); where uid is currently no executed tests, test_name is
   the original name from Loader and variant_id is either None or the
   variant index.

This commit makes the `tag` argument unused. To avoid problems a warning
is issued on it's usage so we can remove it in the upcoming releases.
Signed-off-by: NLukáš Doktor <ldoktor@redhat.com>

The xunit plugin claims to be compatible with jenkins-ci junit-4.xsd,
but it does not pass on the validation dur to the wrong key name 'skip'.

This patch changes the key name to 'skipped' and adds a unittest to validate
the xunit output against the junit-4.xsd schema.

The junit-4.xsd file is coming from:
https://svn.jenkins-ci.org/trunk/hudson/dtkit/dtkit-format/dtkit-junit-model/src/main/resources/com/thalesgroup/dtkit/junit/model/xsd/junit-4.xsd

Reference: https://trello.com/c/ZdEb0TblSigned-off-by: NAmador Pahim <apahim@redhat.com>

When an invalid python code is present in a test class, avocado crashes
badly. This patch adds a TestLoaderError class to be used in those cases,
so we use it to fake the invalid test class, fail the test with ERROR estatus
and report the exception in job log.

Reference: https://trello.com/c/3zDIjTuYSigned-off-by: NAmador Pahim <apahim@redhat.com>

When avocado fails early in the process (for example in
_setup_environment_variables) it crashes, because the test status is
None. Other example could be user-defined status, which is not inside
user-facing-statuses.

This patch modifies the "status" to "ERROR" when test reports
unsupported status and sets related values to provide details regarding
this failure.
Signed-off-by: NLukáš Doktor <ldoktor@redhat.com>

When a job is timed out during a test execution, we fail the test
and put status ERROR in the test. The job then exits with the rc
AVOCADO_TESTS_FAIL. This patch fixes this, making the test status
INTERRUPTED and the job to exit with AVOCADO_JOB_INTERRUPTED.

Also, when a job is timed out before a test, the test is skipped
and the job exits with rc AVOCADO_ALL_OK. For that case, this patch
makes the job to exit with AVOCADO_JOB_INTERRUPTED instead, keeping
the test status as SKIP.

Given this change, now we have the following combinations of test status
and job return code:

Case1:
    - Test1: PASS
    - Test2: SKIP (TestTimeoutSkip)

    Job RC: AVOCADO_JOB_INTERRUPTED

Case2:
    - Test1: PASS
    - Test2: INTERRUPTED (TestTimeoutInterrupted)
    - Test3: SKIP (TestTimeoutSkip)

    Job RC: AVOCADO_JOB_INTERRUPTED

Case3:
    - Test1: PASS
    - Test2: FAIL
    - Test3: INTERRUPTED (TestTimeoutInterrupted)
    - Test4: SKIP (TestTimeoutSkip)

    Job RC: AVOCADO_JOB_INTERRUPTED
Signed-off-by: NAmador Pahim <apahim@redhat.com>

In case no tests are found we advice users to run `avocado list -V`
followed by list of urls, which does not really work with no-urls given.
This patch adds second message in case the user does not provide urls
(eg. is using the `--vt-config` option).
Signed-off-by: NLukáš Doktor <ldoktor@redhat.com>

A test base on the external runner feature depends on both the
external runner and the test URL. These two are always combined to
generate the complete command that will be executed by Avocado.

But, the Avocado command line application does not require URLs to be
given on the command line, because these may come from other sources
on loaders other than the basic FileLoader.

So, let's also explicitly require a URL to be given to to the External
Runner loader to return a test factory (class and parameters).
Signed-off-by: NCleber Rosa <crosa@redhat.com>

While working on the avocado logging system we thought it'd be
consistent and better to log certain streams to sys.stderr rather than
sys.stdout. It turned out to be a bit confusing and unexpected, so after
a discussion this commit adjusts the streams to:

 * avocado.app - loglevel <= info goes to stdout
               - loglevel > info goes to stderr
 * other streams - go always into stdout
 * sys.stdout - goes to sys.stdout when enabled
 * sys.stderr - goes to sys.stderr when enabled
              - goes to sys.stdout when enabled as test output
Signed-off-by: NLukáš Doktor <ldoktor@redhat.com>

Given the last CI jobs, RunnerSimpleTest:test_kill_stopped_sleep() has
been failing often. The obvious reason, given the nature of the test
and the heavily shared and loaded computing resources they run on, is
the lack of resources.

One example of such as failure:

   ======================================================================
   FAIL: test_kill_stopped_sleep (selftests.functional.test_basic.RunnerSimpleTest)
   ----------------------------------------------------------------------
   Traceback (most recent call last):
     File "/home/travis/build/avocado-framework/avocado/selftests/functional/test_basic.py", line 549, in test_kill_stopped_sleep
       % proc.get_output())
   AssertionError: Avocado process still alive 1s after job-timeout:
   JOB ID     : 5c65a06b4ab15e69d6c4a1fc762368e0e99cbc1f
   JOB LOG    : /tmp/avocado_selftests.functional.test_basicHIyhV4/job-2016-03-09T15.08-5c65a06/job.log
   TESTS      : 1
    (1/1) /bin/sleep 60:
   ctrl+z pressed, stopping test (5046)
   ERROR

So, let's give 4 extra seconds for the process to be
terminated. Hopefully failures in the timeout handling will still be
revealed given the relatively small buffer.
Signed-off-by: NCleber Rosa <crosa@redhat.com>

The configurations related to the "data_dir" are currently read at
module load time. This means that changes in the configuration between
module load time and changes to the settings (such as parsing an
additional config file) is never seen.

Since the avocado command line application parses extra configuration
files (given with `--config`) after the module is loaded, these extra
configuration files are never applied to the data_dir configuration.

There's one more issue with regards to the the settings usage: because
once references to the `settings` instance singleton are grabbed, they
will always point to the same settings objects. The data_dir unittests
exercise changes to the settings objects by replacing that
instance. So, let's refer to the settings using the full location
(module.attribute), which will always give back the "current" (be it
original or replaced) settings instance.
Signed-off-by: NCleber Rosa <crosa@redhat.com>

This check "avocado -v" produces correct output and returns 0.
Signed-off-by: NLukáš Doktor <ldoktor@redhat.com>

The `--silent` argument is valid for all commands, let's add it to
the core avocado args. For backward compatibility keep supporting it in
"run" subcommand too.
Signed-off-by: NLukáš Doktor <ldoktor@redhat.com>

This commit changes the idiom on (hopefully all) except statements,
from:
   except foo, details:

to:
   except foo as details:

Which is compatible with both Python 2 and 3.
Signed-off-by: NCleber Rosa <crosa@redhat.com>

This patch changes the way logging is initialized in avocado app:

1. stdout/stderr/logging is replaced with in-memory handlers
2. when arg parsing is done, logging is reconfigured accordingly to
   setting and re-logs stored messages (when streams are enabled)
3. when job starts and "test" stream is enabled, "" and "avocado.test"
   streams are enabled
4. when job finishes and "test" stream enabled the additional outputs,
   the additional outputs are disabled again.

The 3 and 4 is necessarily as the "" logger contains all logs including
avocado initialization (stevedore, PIL, ...). This way the "test" stream
still outputs everything, but only during the job-execution and not
during avocado initialization and cleanup. (one can use "early" log to
see those).

Last but not least it keeps "avocado.app" output in stdout (and stderr),
but it moves all other outputs to stderr instead. This is IMO better
handling as one can distinguish between debug and "additional" output.
Signed-off-by: NLukáš Doktor <ldoktor@redhat.com>

When we abort the test and the test ignores SIGTERM, avocado hangs for
infinity. Let's wait a while and send SIGKILL.
Signed-off-by: NLukáš Doktor <ldoktor@redhat.com>

There is an issue when running SIMPLE tests with non absolute
paths, example:

 $ echo "#!/bin/bash" > /tmp/test.sh
 $ echo "exit 0" >> /tmp/test.sh
 $ chmod +x /tmp/test.sh
 $ cd /tmp
 $ avocado run test.sh

The previous work on the simple test runner, adding the filename
property, actually fixes that bug, so let's add a functional test to
avoid an regression.

Reference: https://trello.com/c/nbhyILO0Signed-off-by: NCleber Rosa <crosa@redhat.com>

The "replay_map" allows to substitute the test class with custom class.
The problem is, that the "methodName" defined in "test_params" might not
exist in this class.

We have two choices, either leave it up to the user, to substitute with
a compatible class (eg. by creating the "methodName" in "__init__"
before calling the super.__init__), or we can require them to always use
given "methodName". This patch uses the second approach as it seems less
magical to me.
Signed-off-by: NLukáš Doktor <ldoktor@redhat.com>

Using an unknown option in a given subcommand leads argparse to
show the main avocado usage instructions.

This patch makes the argparse show the subcommand help instead.

Also, argparse error exit code now matches error_codes.AVOCADO_FAIL,
which makes more sense than error_codes.AVOCADO_JOB_FAIL.

Reference:https://trello.com/c/o0TpH9aeSigned-off-by: NAmador Pahim <apahim@redhat.com>

There are a few leftover references to the old Plugin classes in
the functional tests. Let's remove them.
Signed-off-by: NCleber Rosa <crosa@redhat.com>

If a test has local imports (a fairly common use case),
we have to append the base test dir to sys.path. Otherwise,
we end up having a horrible traceback as a result:

Exception loading test
Traceback (most recent call last):
  File "/home/lmr/Code/avocado.lmr/avocado/core/runner.py", line 200, in _run_test
    instance = loader.load_test(test_factory)
  File "/home/lmr/Code/avocado.lmr/avocado/core/loader.py", line 262, in load_test
    test_module = imp.load_module(module_name, f, p, d)
  File "scylla_longevity.py", line 5, in <module>
    from sdcm.tester import ScyllaClusterTester
ImportError: No module named sdcm.tester

Which is clearly not wanted. We had this problem before
and now we have a regression. It's time for a functional
test to make sure we don't regress again.
Signed-off-by: NLucas Meneghel Rodrigues <lookkas@gmail.com>
Signed-off-by: NCleber Rosa <crosa@redhat.com>

Signed-off-by: NCleber Rosa <crosa@redhat.com>

Signed-off-by: NCleber Rosa <crosa@redhat.com>

Yes, too many plugin mentions, in this commit title, but that's what
it really is, so please bear with it.

Now that the plugin types have a finer granularity, let's show each
plugin type (based on the namespace) as a separate section.
Signed-off-by: NLucas Meneghel Rodrigues <lmr@scylladb.com>
Signed-off-by: NCleber Rosa <crosa@redhat.com>

Signed-off-by: NCleber Rosa <crosa@redhat.com>

Signed-off-by: NCleber Rosa <crosa@redhat.com>

Signed-off-by: NCleber Rosa <crosa@redhat.com>

Signed-off-by: NCleber Rosa <crosa@redhat.com>

The Avocado command line application has "subcommands". So far,
to execute these commands, a special method, named "dispatch"
was set on the command line parser result object (aka Namespace)
pointing to the `run` method of the same class.

In truthfullness, the command line parser doesn't have too much
to do with these methods. Also, the `set_defaults` method of the
command line argument parser is not exactly intended for that use.
So, let's ask the subcommands subparser to formally save the chosen
subcommand, at well, `subcommand`, and nothing more.

Since the application knows about the command plugins (the ones that
inherit from CLICmdBase), and their interface now *require* a method
called `run`, let's use the extension entry-point name to locate the
underlying object and its `run` method.
Signed-off-by: NCleber Rosa <crosa@redhat.com>

At this point, no plugin support exists in Avocado. This is in
preparation for the new plugin code to be cleanly introduced.

The plugins that play a role in the plugin architecture have
been removed, since they wouldn't be functional under the new
plugin management code. The ones that are add extra functionality
to Avocado have been kept, and will be ported to the new
architecture.

Also, most of the functional tests have been temporarily disabled.
The reason is that most of them run avocado, which depends on the
run command, which in turn, depends on the plugin archicture code.
Signed-off-by: NCleber Rosa <crosa@redhat.com>

This shows a more succint version of the help message (the so called
"usage" information). It's a certainly a matter of taste and change
of behavior, so this is marked as an RFC.

It also avoids a hackish injection of the print_help method as the
default dispatch method.
Signed-off-by: NCleber Rosa <crosa@redhat.com>

Signed-off-by: NLukáš Doktor <ldoktor@redhat.com>

Since we define the meaning of each exit code, we should be using that
ourselves.

Functional tests have been coded using literal values, and there may
be a good reason for that: proving that the API stands.  But there's a
big issue: we have not declared those codes as some kind of an
external API and guarantee stability. Also the subsequent changes in
this series actually change the meaning and values of those.
Signed-off-by: NCleber Rosa <crosa@redhat.com>

It's prone to mistakes maintain two separate ways to query for tests
(running them and displaying available ones).

This patch implements --dry-run, which follows the full "run" process,
only instead of running the test it skips them.

Additionally the --dry-run uses some tweaks to tell them apart from
normal executions, namely:

    1. No sysinfo gathering
    2. Unique id is 0000000000000000000000000000000000000000
    3. Unless user provides custom log location, fresh tmpdir is used
Signed-off-by: NLukáš Doktor <ldoktor@redhat.com>

In preparation for the large plugin refactor, let's simplify the code ASAP
and drop related (custom) functionality that is going to be replaced.

Standard setuptools entrypoints is going to provide better capability then
our custom code that supports external 'avocado_*.py'module loading.
Signed-off-by: NCleber Rosa <crosa@redhat.com>

This patch doesn't change any functionality, it only renames the
inner_runner to external_runner (the plugin is called simply external)
Signed-off-by: NLukáš Doktor <ldoktor@redhat.com>

Autopep8 style cleanup + alphabetic import + import cleanup in
avocado.core.{loader,test} and selftests.functional.test_basic.
Signed-off-by: NLukáš Doktor <ldoktor@redhat.com>

Currently we can only specify the loader plugins order via settings.
This patch adds "--loaders" argument and improves the granularity.
Now it's possible to set either @plugin_name, or TEST_TYPE, where
TEST_TYPE is the mapped name of the plugin (eg. SIMPLE, VT, ...)

There is one special name "DEFAULT", which injects all unspecified
loaders in the position of the "DEFAULT" keyword. When "DEFAULT" is
not specified, not-listed plugins/test_types are not evaluated.
(therefor it's possible to restrict some test loaders or only some
test types).

As the usecase changed slightly, the settings option was renamed
from plugins.loader_plugins_priority to plugins.loaders.
Signed-off-by: NLukáš Doktor <ldoktor@redhat.com>

Make sure we have avocado_ + __name__ prefixes to the dirs
created with tempfile, so that we have an easy way to spot
when we are not cleaning up directories properly. This mostly
concerns unittests, although the kernel_build lib and the
iso9660 lib also uses that API.
Signed-off-by: NLucas Meneghel Rodrigues <lmr@redhat.com>