There are a few leftover references to the old Plugin classes in
the functional tests. Let's remove them.
Signed-off-by: NCleber Rosa <crosa@redhat.com>

If a test has local imports (a fairly common use case),
we have to append the base test dir to sys.path. Otherwise,
we end up having a horrible traceback as a result:

Exception loading test
Traceback (most recent call last):
  File "/home/lmr/Code/avocado.lmr/avocado/core/runner.py", line 200, in _run_test
    instance = loader.load_test(test_factory)
  File "/home/lmr/Code/avocado.lmr/avocado/core/loader.py", line 262, in load_test
    test_module = imp.load_module(module_name, f, p, d)
  File "scylla_longevity.py", line 5, in <module>
    from sdcm.tester import ScyllaClusterTester
ImportError: No module named sdcm.tester

Which is clearly not wanted. We had this problem before
and now we have a regression. It's time for a functional
test to make sure we don't regress again.
Signed-off-by: NLucas Meneghel Rodrigues <lookkas@gmail.com>
Signed-off-by: NCleber Rosa <crosa@redhat.com>

Signed-off-by: NCleber Rosa <crosa@redhat.com>

Signed-off-by: NCleber Rosa <crosa@redhat.com>

Signed-off-by: NCleber Rosa <crosa@redhat.com>

Signed-off-by: NCleber Rosa <crosa@redhat.com>

Yes, too many plugin mentions, in this commit title, but that's what
it really is, so please bear with it.

Now that the plugin types have a finer granularity, let's show each
plugin type (based on the namespace) as a separate section.
Signed-off-by: NLucas Meneghel Rodrigues <lmr@scylladb.com>
Signed-off-by: NCleber Rosa <crosa@redhat.com>

Signed-off-by: NCleber Rosa <crosa@redhat.com>

Signed-off-by: NCleber Rosa <crosa@redhat.com>

Signed-off-by: NCleber Rosa <crosa@redhat.com>

Signed-off-by: NCleber Rosa <crosa@redhat.com>

Signed-off-by: NCleber Rosa <crosa@redhat.com>

Signed-off-by: NCleber Rosa <crosa@redhat.com>

Signed-off-by: NCleber Rosa <crosa@redhat.com>

The Avocado command line application has "subcommands". So far,
to execute these commands, a special method, named "dispatch"
was set on the command line parser result object (aka Namespace)
pointing to the `run` method of the same class.

In truthfullness, the command line parser doesn't have too much
to do with these methods. Also, the `set_defaults` method of the
command line argument parser is not exactly intended for that use.
So, let's ask the subcommands subparser to formally save the chosen
subcommand, at well, `subcommand`, and nothing more.

Since the application knows about the command plugins (the ones that
inherit from CLICmdBase), and their interface now *require* a method
called `run`, let's use the extension entry-point name to locate the
underlying object and its `run` method.
Signed-off-by: NCleber Rosa <crosa@redhat.com>

At this point, no plugin support exists in Avocado. This is in
preparation for the new plugin code to be cleanly introduced.

The plugins that play a role in the plugin architecture have
been removed, since they wouldn't be functional under the new
plugin management code. The ones that are add extra functionality
to Avocado have been kept, and will be ported to the new
architecture.

Also, most of the functional tests have been temporarily disabled.
The reason is that most of them run avocado, which depends on the
run command, which in turn, depends on the plugin archicture code.
Signed-off-by: NCleber Rosa <crosa@redhat.com>

This shows a more succint version of the help message (the so called
"usage" information). It's a certainly a matter of taste and change
of behavior, so this is marked as an RFC.

It also avoids a hackish injection of the print_help method as the
default dispatch method.
Signed-off-by: NCleber Rosa <crosa@redhat.com>

This fixes the recently introduced commit 5f7bafd0, running the skip
function instead of passing the function itself to `unittest.skipIf`.
Signed-off-by: NCleber Rosa <crosa@redhat.com>

The `test_gendata` test of `OutputTest` runs the example test
`gendata.py`, but the `gendata.py` itself has a conditional in one of
the tests that make use of the PIL (aka Pillow package) library and
does nothing if it's missing.

PIL (Pillow) is highly dependent on C compilers and libraries to be
able to build itself. This makes systems it harder to run the full
Avocado self tests on more diverse (usually minimalist) systems.

So, let's make that functional test optional, if the libraries are not
present.
Signed-off-by: NCleber Rosa <crosa@redhat.com>

There are a couple of functional unittests that assume that a given
output was produced by the regular UI, that is, the Human output
format.

That is fine, since they're functional tests. But, some situations
lead to test errors, instead of pointing failures. Let's turn those
into explicit checks and fail properly instead of producing errors.

Note: These errors were visible when the human output was disabled, in
the plugin overhaul work, but the fixes still make sense independently
of that work.
Signed-off-by: NCleber Rosa <crosa@redhat.com>

While going through the dependency list (requirements*.txt files) and
performing our self tests out of virtual environments, I noticed that
some tests are run outside the virtual environments.

The reason is that, even though the virtual environment is activated
for the test session (and say, `which python` gives `/venv/bin/python`),
we have hard coded `/usr/bin/python` in most places.

According to the some discussions on the virtualenv project itself[1],
a quick solution is to revert to the also common `/usr/bin/env python`
way of pointing to the Python interpreter.

[1] - https://github.com/pypa/virtualenv/issues/124Signed-off-by: NCleber Rosa <crosa@redhat.com>

Signed-off-by: NLukáš Doktor <ldoktor@redhat.com>

Since we define the meaning of each exit code, we should be using that
ourselves.

Functional tests have been coded using literal values, and there may
be a good reason for that: proving that the API stands.  But there's a
big issue: we have not declared those codes as some kind of an
external API and guarantee stability. Also the subsequent changes in
this series actually change the meaning and values of those.
Signed-off-by: NCleber Rosa <crosa@redhat.com>

It's prone to mistakes maintain two separate ways to query for tests
(running them and displaying available ones).

This patch implements --dry-run, which follows the full "run" process,
only instead of running the test it skips them.

Additionally the --dry-run uses some tweaks to tell them apart from
normal executions, namely:

    1. No sysinfo gathering
    2. Unique id is 0000000000000000000000000000000000000000
    3. Unless user provides custom log location, fresh tmpdir is used
Signed-off-by: NLukáš Doktor <ldoktor@redhat.com>

In preparation for the large plugin refactor, let's simplify the code ASAP
and drop related (custom) functionality that is going to be replaced.

Standard setuptools entrypoints is going to provide better capability then
our custom code that supports external 'avocado_*.py'module loading.
Signed-off-by: NCleber Rosa <crosa@redhat.com>

This is an automated mass import cleanup across all
avocado source files:

1) Imports follow the order:
 * Standard library imports
 * Non standard library external imports
 * Internal imports
 All separated by a single line
2) One line between imports and the rest of the code
3) Remove unused imports in the process
Signed-off-by: NLucas Meneghel Rodrigues <lmr@redhat.com>

This patch doesn't change any functionality, it only renames the
inner_runner to external_runner (the plugin is called simply external)
Signed-off-by: NLukáš Doktor <ldoktor@redhat.com>

Autopep8 style cleanup + alphabetic import + import cleanup in
avocado.core.{loader,test} and selftests.functional.test_basic.
Signed-off-by: NLukáš Doktor <ldoktor@redhat.com>

The BUGGY test concept relies on loading/executing the Python file
containing the tests and failing to do so. Since Avocado won't
load/execute Python test files anymore, it's not possible or desirable
to keep the BUGGY tests around.
Signed-off-by: NCleber Rosa <crosa@redhat.com>

There's no real reason why a single Python file can not hold multiple
test classes, each one with its own tests.

This test adds a simple Python test file with two classes, each one
with its own test, and expects the loader to find two instrumented
tests.
Signed-off-by: NCleber Rosa <crosa@redhat.com>

The test loader should only load (import, thus execute) Python test
modules at test execution time. During, for example, test discovery
and listing time, the Python modules should never be imported (thus
executed).

This introduces a test called "sleep eleven" (because "ten" is boring)
that "accidentally" has a `time.sleep(11)` at the global scope. This
test is listed by executing `avocado list -V sleepeleven.py` and is
given 3 seconds to finish. If the loader actually loads/executes the
Python code, it will fail because it will take at least a little bit
more then 11 seconds.
Signed-off-by: NCleber Rosa <crosa@redhat.com>

Currently we can only specify the loader plugins order via settings.
This patch adds "--loaders" argument and improves the granularity.
Now it's possible to set either @plugin_name, or TEST_TYPE, where
TEST_TYPE is the mapped name of the plugin (eg. SIMPLE, VT, ...)

There is one special name "DEFAULT", which injects all unspecified
loaders in the position of the "DEFAULT" keyword. When "DEFAULT" is
not specified, not-listed plugins/test_types are not evaluated.
(therefor it's possible to restrict some test loaders or only some
test types).

As the usecase changed slightly, the settings option was renamed
from plugins.loader_plugins_priority to plugins.loaders.
Signed-off-by: NLukáš Doktor <ldoktor@redhat.com>

Make sure we have avocado_ + __name__ prefixes to the dirs
created with tempfile, so that we have an easy way to spot
when we are not cleaning up directories properly. This mostly
concerns unittests, although the kernel_build lib and the
iso9660 lib also uses that API.
Signed-off-by: NLucas Meneghel Rodrigues <lmr@redhat.com>

Let's use a simple passtest to check for latest result,
and let's wait for avocado to finish. That should accomplish
the same result in a shorter and nicer way.
Signed-off-by: NLucas Meneghel Rodrigues <lmr@redhat.com>

The Xunit and JSON test cases inherit from PluginsTest. This test defines
some hardcoded-values tests, which are then inherited and run for each
inherited class with the exact same values. This patch creates abstract
class, which all 3 cases share and makes them inherit from it, rather
than from the actual testcase.
Signed-off-by: NLukáš Doktor <ldoktor@redhat.com>

The name was not correct as the test should error-out.
Signed-off-by: NLukáš Doktor <ldoktor@redhat.com>

Signed-off-by: NLucas Meneghel Rodrigues <lmr@redhat.com>

Fedora 21 ships psutils 0.6, that has the API psutil.get_pid_list()
instead of psutil.pids(). Let's make both compatible, for the sake
of running unittests on COPR builds.
Signed-off-by: NLucas Meneghel Rodrigues <lmr@redhat.com>

Currently when one passes exception, which is not instance of Exception,
avocado proceeds and returns "INTERRUPTED" test instead of "ERROR". This
for example happens when old-style-class is used as exception. This
patch uses pure "except" without any argument to catch such exceptions
and then it wraps them in "exceptions.TestError" instead.

The same applies to "setUp" and "tearDown", they only use different
class as a wrapper.
Signed-off-by: NLukáš Doktor <ldoktor@redhat.com>