At this point, no plugin support exists in Avocado. This is in
preparation for the new plugin code to be cleanly introduced.

The plugins that play a role in the plugin architecture have
been removed, since they wouldn't be functional under the new
plugin management code. The ones that are add extra functionality
to Avocado have been kept, and will be ported to the new
architecture.

Also, most of the functional tests have been temporarily disabled.
The reason is that most of them run avocado, which depends on the
run command, which in turn, depends on the plugin archicture code.
Signed-off-by: NCleber Rosa <crosa@redhat.com>

This shows a more succint version of the help message (the so called
"usage" information). It's a certainly a matter of taste and change
of behavior, so this is marked as an RFC.

It also avoids a hackish injection of the print_help method as the
default dispatch method.
Signed-off-by: NCleber Rosa <crosa@redhat.com>

This fixes the recently introduced commit 5f7bafd0, running the skip
function instead of passing the function itself to `unittest.skipIf`.
Signed-off-by: NCleber Rosa <crosa@redhat.com>

The `test_gendata` test of `OutputTest` runs the example test
`gendata.py`, but the `gendata.py` itself has a conditional in one of
the tests that make use of the PIL (aka Pillow package) library and
does nothing if it's missing.

PIL (Pillow) is highly dependent on C compilers and libraries to be
able to build itself. This makes systems it harder to run the full
Avocado self tests on more diverse (usually minimalist) systems.

So, let's make that functional test optional, if the libraries are not
present.
Signed-off-by: NCleber Rosa <crosa@redhat.com>

There are a couple of functional unittests that assume that a given
output was produced by the regular UI, that is, the Human output
format.

That is fine, since they're functional tests. But, some situations
lead to test errors, instead of pointing failures. Let's turn those
into explicit checks and fail properly instead of producing errors.

Note: These errors were visible when the human output was disabled, in
the plugin overhaul work, but the fixes still make sense independently
of that work.
Signed-off-by: NCleber Rosa <crosa@redhat.com>

While going through the dependency list (requirements*.txt files) and
performing our self tests out of virtual environments, I noticed that
some tests are run outside the virtual environments.

The reason is that, even though the virtual environment is activated
for the test session (and say, `which python` gives `/venv/bin/python`),
we have hard coded `/usr/bin/python` in most places.

According to the some discussions on the virtualenv project itself[1],
a quick solution is to revert to the also common `/usr/bin/env python`
way of pointing to the Python interpreter.

[1] - https://github.com/pypa/virtualenv/issues/124Signed-off-by: NCleber Rosa <crosa@redhat.com>

We used to support subtests filter to load only some of the avocado
tests, but the functionality is broken. This patch brings this
functionality back. It uses "re.search" so people can use partial
matches. Additionally it includes selftests to prevent future breakages.
Signed-off-by: NLukáš Doktor <ldoktor@redhat.com>

This patch adds unittest for loading simple tests with arguments.
Signed-off-by: NLukáš Doktor <ldoktor@redhat.com>

Signed-off-by: NLucas Meneghel Rodrigues <lookkas@gmail.com>

shlex.split() can fail on some strange commands (eg. on an unfinished
quotation). This patch fallbacks to simple string.split() in such cases,
which might return incorrect results, but is safer to use. Although it
logs warning about this notifying the user about possible issue.
Signed-off-by: NLukáš Doktor <ldoktor@redhat.com>

It has been reported that avocado crashes when a malformed command
is attempted to be executed, even when using `shell=True`.

The reason is that `shlex.split` will raise `ValueError` on those
commands. Since improving `shlex.split` may be too complex, and
possibly not the best place to "fix" this, let's just assume safely
that this command should be run on GDB.
Signed-off-by: NCleber Rosa <crosa@redhat.com>

Signed-off-by: NLukáš Doktor <ldoktor@redhat.com>

Since we define the meaning of each exit code, we should be using that
ourselves.

Functional tests have been coded using literal values, and there may
be a good reason for that: proving that the API stands.  But there's a
big issue: we have not declared those codes as some kind of an
external API and guarantee stability. Also the subsequent changes in
this series actually change the meaning and values of those.
Signed-off-by: NCleber Rosa <crosa@redhat.com>

It's prone to mistakes maintain two separate ways to query for tests
(running them and displaying available ones).

This patch implements --dry-run, which follows the full "run" process,
only instead of running the test it skips them.

Additionally the --dry-run uses some tweaks to tell them apart from
normal executions, namely:

    1. No sysinfo gathering
    2. Unique id is 0000000000000000000000000000000000000000
    3. Unless user provides custom log location, fresh tmpdir is used
Signed-off-by: NLukáš Doktor <ldoktor@redhat.com>

Improve the `AvocadoParams.iteritems()` to return unique items only
in the same fashion `multiplexer` does it.
Signed-off-by: NLukáš Doktor <ldoktor@redhat.com>

In preparation for the large plugin refactor, let's simplify the code ASAP
and drop related (custom) functionality that is going to be replaced.

Standard setuptools entrypoints is going to provide better capability then
our custom code that supports external 'avocado_*.py'module loading.
Signed-off-by: NCleber Rosa <crosa@redhat.com>

From the Python `tempfile.mkstemp()` docs:

The default directory is chosen from a platform-dependent list, but
the user of the application can control the directory location by
setting the TMPDIR, TEMP or TMP environment variables.

So, let's let `tempfile.mkstemp()` itself choose the temp dir, based
on what the use may have set in environment variables.
Signed-off-by: NCleber Rosa <crosa@redhat.com>

This is an automated mass import cleanup across all
avocado source files:

1) Imports follow the order:
 * Standard library imports
 * Non standard library external imports
 * Internal imports
 All separated by a single line
2) One line between imports and the rest of the code
3) Remove unused imports in the process
Signed-off-by: NLucas Meneghel Rodrigues <lmr@redhat.com>

Since the test loader now attempts to find Avocado instrumented tests
by using a parser and doesn't really load/execute test files, it can't
tell if a given class inherits from avocado.Test.

For those cases, a developer may signal, using a docstring tag, that
either:

 * A test class that doesn't look like an Avocado test class is indeed
   one ("🥑 enable")
 * A test class that looks like an Avocado test class should not be
   treated as such ("🥑 disable")

Also included is a a simple optimization by breaking earlier from the
parser loop.

Changes from v1:
   * Added two more tests and suggestions by Lukáš Doktor

Changes from v0:
   * Added fix reported by Lukáš Doktor
Signed-off-by: NCleber Rosa <crosa@redhat.com>
CC: Lukáš Doktor <ldoktor@redhat.com>

Inside test classes there may docstrings which may contain tags that
give special meaning. The tags are defined by "🥑 <value>" in a
single line. For now the only two "official" Avocado tags values
exist:

 * enable: the class is an Avocado INSTRUMENTED test, even if it
   doesn't "look like" one.
 * disable: the class is *NOT* an Avocado INSTRUMENTED test, even if
   looks like one.

Changes from v0:
 * Use unittest2 on Python 2.6
Signed-off-by: NCleber Rosa <crosa@redhat.com>

Sphinx will attempt to fetch the Python objects inventory during the build
process. If for some reason, this test is being run on a machine that can
not access that address simply because of network restrictions (or the
developer may simply be on a plane) then it's better to SKIP the test than
to give a false positive.

Changes from v0:
 * Use unittest2 on Python 2.6
Signed-off-by: NCleber Rosa <crosa@redhat.com>

This patch doesn't change any functionality, it only renames the
inner_runner to external_runner (the plugin is called simply external)
Signed-off-by: NLukáš Doktor <ldoktor@redhat.com>

Autopep8 style cleanup + alphabetic import + import cleanup in
avocado.core.{loader,test} and selftests.functional.test_basic.
Signed-off-by: NLukáš Doktor <ldoktor@redhat.com>

When one execute 2 jobs with first 7 chars of unique id at the same
time, they end-up in the same logdir.

This patch makes the creation safer. First it creates the main
directory, then it tries to create the directory with 7 chars of the
unique id. On failure it adds another char until it finds empty dir.

In case you spawn 2 jobs with the same id at the same time, it adds
a number starting with 0.

If you spawn more tham 1000 jobs at the same time with the exact same
id, it fails with IOError.
Signed-off-by: NLukáš Doktor <ldoktor@redhat.com>

With the changes that now do not load the test at discovery time,
the test class is a string with the class name, and not a class
itself.

Therefore, this discovery test is only going to check for the test
class name. Other test may be added to do the real module load and
test class instantiation.
Signed-off-by: NCleber Rosa <crosa@redhat.com>

Since loading/executing tests are not desirable before actually
running them, these tests must be dropped.
Signed-off-by: NCleber Rosa <crosa@redhat.com>

The BUGGY test concept relies on loading/executing the Python file
containing the tests and failing to do so. Since Avocado won't
load/execute Python test files anymore, it's not possible or desirable
to keep the BUGGY tests around.
Signed-off-by: NCleber Rosa <crosa@redhat.com>

There's no real reason why a single Python file can not hold multiple
test classes, each one with its own tests.

This test adds a simple Python test file with two classes, each one
with its own test, and expects the loader to find two instrumented
tests.
Signed-off-by: NCleber Rosa <crosa@redhat.com>

The test loader should only load (import, thus execute) Python test
modules at test execution time. During, for example, test discovery
and listing time, the Python modules should never be imported (thus
executed).

This introduces a test called "sleep eleven" (because "ten" is boring)
that "accidentally" has a `time.sleep(11)` at the global scope. This
test is listed by executing `avocado list -V sleepeleven.py` and is
given 3 seconds to finish. If the loader actually loads/executes the
Python code, it will fail because it will take at least a little bit
more then 11 seconds.
Signed-off-by: NCleber Rosa <crosa@redhat.com>

Currently we can only specify the loader plugins order via settings.
This patch adds "--loaders" argument and improves the granularity.
Now it's possible to set either @plugin_name, or TEST_TYPE, where
TEST_TYPE is the mapped name of the plugin (eg. SIMPLE, VT, ...)

There is one special name "DEFAULT", which injects all unspecified
loaders in the position of the "DEFAULT" keyword. When "DEFAULT" is
not specified, not-listed plugins/test_types are not evaluated.
(therefor it's possible to restrict some test loaders or only some
test types).

As the usecase changed slightly, the settings option was renamed
from plugins.loader_plugins_priority to plugins.loaders.
Signed-off-by: NLukáš Doktor <ldoktor@redhat.com>

Add a check for leftover tmp dirs after our unittests run.
This will let us to catch mistakes made during their development
(for example, forgetting to clean up temporary dirs created during
setup stage). Add that to `make check` and Travis CI jobs.
Signed-off-by: NLucas Meneghel Rodrigues <lmr@redhat.com>

Make sure we have avocado_ + __name__ prefixes to the dirs
created with tempfile, so that we have an easy way to spot
when we are not cleaning up directories properly. This mostly
concerns unittests, although the kernel_build lib and the
iso9660 lib also uses that API.
Signed-off-by: NLucas Meneghel Rodrigues <lmr@redhat.com>

Let's use a simple passtest to check for latest result,
and let's wait for avocado to finish. That should accomplish
the same result in a shorter and nicer way.
Signed-off-by: NLucas Meneghel Rodrigues <lmr@redhat.com>

And fix/improve the current implementation by:

 - Properly passing the fabric environment parameter (named "timeout",
   and not "connection_timeout").
 - Set different timeouts for Remote (60 seconds) and VM (120
   seconds). The reasoning is that for VM it may be needed to start
   the VM itself.
 - The timeout passed to fabric is now the timeout the user gives,
   divided by the number of attempts.
 - Change the default attempts from 3 to 10 so that the overall
   timeout is broken in smaller pieces and user hopefully waits less
   time for a successful outcome.

This was tested using an EL7 VM using TCG (non-kvm).
Signed-off-by: NCleber Rosa <crosa@redhat.com>

It turns out we were basing the links of the URL report
in the 'url' attribute, when we should be using 'test'.
This fixes an issue that came up with the new multiplexed
tests in VM and remote plugins, where the HTML report
link was invalid. Fix the unittest, that had the wrong
behavior built into it.
Signed-off-by: NLucas Meneghel Rodrigues <lmr@redhat.com>

The Xunit and JSON test cases inherit from PluginsTest. This test defines
some hardcoded-values tests, which are then inherited and run for each
inherited class with the exact same values. This patch creates abstract
class, which all 3 cases share and makes them inherit from it, rather
than from the actual testcase.
Signed-off-by: NLukáš Doktor <ldoktor@redhat.com>

The name was not correct as the test should error-out.
Signed-off-by: NLukáš Doktor <ldoktor@redhat.com>

Signed-off-by: NLucas Meneghel Rodrigues <lmr@redhat.com>