At this point, no plugin support exists in Avocado. This is in
preparation for the new plugin code to be cleanly introduced.

The plugins that play a role in the plugin architecture have
been removed, since they wouldn't be functional under the new
plugin management code. The ones that are add extra functionality
to Avocado have been kept, and will be ported to the new
architecture.

Also, most of the functional tests have been temporarily disabled.
The reason is that most of them run avocado, which depends on the
run command, which in turn, depends on the plugin archicture code.
Signed-off-by: NCleber Rosa <crosa@redhat.com>

This shows a more succint version of the help message (the so called
"usage" information). It's a certainly a matter of taste and change
of behavior, so this is marked as an RFC.

It also avoids a hackish injection of the print_help method as the
default dispatch method.
Signed-off-by: NCleber Rosa <crosa@redhat.com>

Signed-off-by: NLukáš Doktor <ldoktor@redhat.com>

Since we define the meaning of each exit code, we should be using that
ourselves.

Functional tests have been coded using literal values, and there may
be a good reason for that: proving that the API stands.  But there's a
big issue: we have not declared those codes as some kind of an
external API and guarantee stability. Also the subsequent changes in
this series actually change the meaning and values of those.
Signed-off-by: NCleber Rosa <crosa@redhat.com>

It's prone to mistakes maintain two separate ways to query for tests
(running them and displaying available ones).

This patch implements --dry-run, which follows the full "run" process,
only instead of running the test it skips them.

Additionally the --dry-run uses some tweaks to tell them apart from
normal executions, namely:

    1. No sysinfo gathering
    2. Unique id is 0000000000000000000000000000000000000000
    3. Unless user provides custom log location, fresh tmpdir is used
Signed-off-by: NLukáš Doktor <ldoktor@redhat.com>

In preparation for the large plugin refactor, let's simplify the code ASAP
and drop related (custom) functionality that is going to be replaced.

Standard setuptools entrypoints is going to provide better capability then
our custom code that supports external 'avocado_*.py'module loading.
Signed-off-by: NCleber Rosa <crosa@redhat.com>

This patch doesn't change any functionality, it only renames the
inner_runner to external_runner (the plugin is called simply external)
Signed-off-by: NLukáš Doktor <ldoktor@redhat.com>

Autopep8 style cleanup + alphabetic import + import cleanup in
avocado.core.{loader,test} and selftests.functional.test_basic.
Signed-off-by: NLukáš Doktor <ldoktor@redhat.com>

Currently we can only specify the loader plugins order via settings.
This patch adds "--loaders" argument and improves the granularity.
Now it's possible to set either @plugin_name, or TEST_TYPE, where
TEST_TYPE is the mapped name of the plugin (eg. SIMPLE, VT, ...)

There is one special name "DEFAULT", which injects all unspecified
loaders in the position of the "DEFAULT" keyword. When "DEFAULT" is
not specified, not-listed plugins/test_types are not evaluated.
(therefor it's possible to restrict some test loaders or only some
test types).

As the usecase changed slightly, the settings option was renamed
from plugins.loader_plugins_priority to plugins.loaders.
Signed-off-by: NLukáš Doktor <ldoktor@redhat.com>

Make sure we have avocado_ + __name__ prefixes to the dirs
created with tempfile, so that we have an easy way to spot
when we are not cleaning up directories properly. This mostly
concerns unittests, although the kernel_build lib and the
iso9660 lib also uses that API.
Signed-off-by: NLucas Meneghel Rodrigues <lmr@redhat.com>

Let's use a simple passtest to check for latest result,
and let's wait for avocado to finish. That should accomplish
the same result in a shorter and nicer way.
Signed-off-by: NLucas Meneghel Rodrigues <lmr@redhat.com>

The Xunit and JSON test cases inherit from PluginsTest. This test defines
some hardcoded-values tests, which are then inherited and run for each
inherited class with the exact same values. This patch creates abstract
class, which all 3 cases share and makes them inherit from it, rather
than from the actual testcase.
Signed-off-by: NLukáš Doktor <ldoktor@redhat.com>

Even though I may be the one to blame about coming up with this
"simple import magic", I believe it was a mistake and we should get
rid of them.

There are a couple of other ways to actually do development out of
of a source tree, including running unittests that do not require
this amount of boiler plate code. Examples include just setting
the PYTHONPATH environment variable to actually run (setuptools based)
`python setup.py develop`.

The little bits of what looks like the import magic that was left, is
not really import magic. It's just that functional tests need to locate
the Avocado source tree base directory to run the test runner from it.
Signed-off-by: NCleber Rosa <crosa@redhat.com>

To be honest, our test code could still be kept in the same directories and
have the same names. But I think we can improve two things here:

1) Flatten a little bit the directory structure of selftests. Two path
components are being dropped here: "all" and "avocado". So that
"selftests/all/functional/avocado" becomes simply
"selftests/functional".

2) File names match what is, by default, recognized by unittest (the
Standard Library module) based discovery of tests. That means that
doc_build_test.py becomes test_doc_build.py. Not a big deal IMHO.
Signed-off-by: NCleber Rosa <crosa@redhat.com>

Add some functional tests that ensure that we don't
get caught by mistakes made when modifying the human
test reporting code again (or at least, not by the
obvious ones).
Signed-off-by: NLucas Meneghel Rodrigues <lmr@redhat.com>

In some external test suites, what is defined as a "test" can not be
run by itself and need a specific runner script/tool.

This introduces the concept of an "inner runner", that is, a custom
test runner or tool that can deal with custom tests that do follow
the Avocado SIMPLE test definition.

This implements the Trello card:

 https://trello.com/c/TpXecE2n/486-introduce-inner-runner-to-avocado-run

More information can be found in the man page section about it.

Changes from v1:
 * removed "level" from "inner level runner" mentions
 * replicated docs from man page to "main" docs
Signed-off-by: NCleber Rosa <crosa@redhat.com>

Instead of creating at the end of the job. This allows users running
long (time wise) jobs to be monitored by "following" the
$AVOCADO_RESULT/latest/job.log file.

This implements the card described at:

 https://trello.com/c/QJ4wCD8x/447-reconsider-when-the-latest-log-symlink-is-createdSigned-off-by: NCleber Rosa <crosa@redhat.com>

On multiple places we modify the sys.path to make avocado work from
sources. Instead of `append` we should `insert` the path as when running
from avocado sources directory, we want to use the modules from here,
rather than from installed version.
Signed-off-by: NLukáš Doktor <ldoktor@redhat.com>

Previously the "file" loader was special and always matched the URL
which might be confusing and lead to problems when developing additional
plugins.

This commit tires to define the way loaders should work. They must never
fail (unless of uncaught exception which is reported and next available
plugins is used). Instead when the URL produces broken test or even a
single corrupted subtest, consider it as URL not for me and return empty
set. The loader proxy uses next plugin to resolve this URL. When no
plugin resolves the URL user is advised to run "avocado list -V" to see
the potential problems.

The order of loader plugins was also changed. Now the file-loader is
first by default and all remaining ones are loaded by the plugin
priority. The order can be changed in setting
"plugins/loader_plugins_priority". It uses "name" variable of the each
loader plugin. Not listed plugins are loaded afterwards.

Additionally the whole loader API was simplified. Now it only accepts
one parameter - list_tests - which is 3-state:

1. ALL       - all tests including broken/incorrect ones
2. AVAILABLE - available tests (for listing purposes)
2. DEFAULT   - default tests (to run when no url given)
Signed-off-by: NLukáš Doktor <ldoktor@redhat.com>

This patch renames the fail_on_error decorator to fail_on and adds
support to supply list of exceptions. This way the decorator can be used
not only to make tests fail on generic exceptions but also to specify
parts of code with expected failures of given type.
Signed-off-by: NLukáš Doktor <ldoktor@redhat.com>

Give to test writers the fail_on_error decorator, which
should turn any generic exception into an avocado
TestFail.

This is important for some tests that can raise
any exception during a complex operation, and any of those
exceptions can be considered a test failure.
Signed-off-by: NLucas Meneghel Rodrigues <lmr@redhat.com>

Make sure avocado complains loudly when people try to call
skip() from outside setUp() - this shouldn't be allowed in
avocado tests. Let the user know of the problem and ask them
to fix their tests. Also, update the docstring for skip(),
which will reduce the chances of test writers using it in
inappropriate places.
Signed-off-by: NLucas Meneghel Rodrigues <lmr@redhat.com>

On unhandeled unknown exceptions the test should finish with ERROR
rather than FAIL as the test developer didn't think it might happen.

This behavior is configurable in config when the user have a different
opinion.
Signed-off-by: NLukáš Doktor <ldoktor@redhat.com>

It was previously not possible for a test to skip itself (due to say,
missing requirements) during the setUp() phase. Any exception caught
at this point was reflected as a test failure.
Signed-off-by: NCleber Rosa <crosa@redhat.com>
Signed-off-by: NLucas Meneghel Rodrigues <lmr@redhat.com>

Add the --paginator (on|off) option to 'avocado config'.
This option has been missing from that command, which
is inconsistent with the other listing options we have.
Signed-off-by: NLucas Meneghel Rodrigues <lmr@redhat.com>

Move avocado.plugins to avocado.core.plugins, in conformance to our
purpose of keeping things not related to test developers,
inside avocado.core, isolated from users.
Signed-off-by: NRudá Moura <rmoura@redhat.com>

So that we clean up results after we are done with unittests.

Changes from v1:
 * Used super() instead of simply calling the method of the
   parent class, given that the the unittests are new object
   style classes, per ldoktor's suggestion.
Signed-off-by: NLucas Meneghel Rodrigues <lmr@redhat.com>

We're going one step ahead to unittest  compatibility.
Now we obsolete action() method and use runTest().
Then what is runTest() will be run().

Update the selftests and tests regarding this modification.
Signed-off-by: NRudá Moura <rmoura@redhat.com>

This patch is initial support for people using custom bash scripts
with avocado. Tests should use:

    PATH=$(avocado "exec-path"):$PATH

and then they can utilize the helpers.

This version contain functions to write to Test.log the same way it's
possible from python including failing the test with TestWarn in case
avocado_warn was used.
Signed-off-by: NLukáš Doktor <ldoktor@redhat.com>

We can't install libvirt-python using pip since the
VM used by Travis runs Ubuntu 12.04 LTS (libvirt headers
are too old), so we have to settle for not requiring
all plugins (VM) to be available. It will still be tested
on 2.7, though.

Changes from v1:
 * Per ldoktor's suggestion, change check of exact python
   version being 2.7.x to python version >= 2.7.0.
Signed-off-by: NLucas Meneghel Rodrigues <lmr@redhat.com>

Signed-off-by: NLucas Meneghel Rodrigues <lmr@redhat.com>

Verify that errors in the list command really go to stderr.
Signed-off-by: NLucas Meneghel Rodrigues <lmr@redhat.com>

More checks to make sure we are not outputting errors to
the wrong loggers.
Signed-off-by: NLucas Meneghel Rodrigues <lmr@redhat.com>

Create an additional logging handler for the avocado application,
that outputs to stderr, and two filters, that filter messages
per logging level. This way, normal information (app output)
goes to stdout and error messages go to stderr. The unittests
were updated accordingly.
Signed-off-by: NLucas Meneghel Rodrigues <lmr@redhat.com>

Instead of delaying the parsing of the test list to code that handles
the jobs, let's handle that early, and in the usual Python way, simply
by relying on argparse.

For other, possibly more complex command line requirements, custom code
can be written, but it should still be placed at the argparse level.
Signed-off-by: NCleber Rosa <crosa@redhat.com>

Add support to inspect and find tests inside a directory, recursively,
by using the method `discover_url`. It takes a url (path) and
returns the tests parameters it discovers.

The loader object has the .discover() method, that can be used
to inspect the location, and the .validate() method, that will
discover any problems on the user's input.

We are also changing the behavior of avocado to error out on
invalid inputs (missing paths or files that are not avocado
tests). Therefore, the concept of MISSING or NOT_A_TEST both
disappear from the UI. Unittests were updated to reflect the
new status quo.
Signed-off-by: NRudá Moura <rmoura@redhat.com>
Signed-off-by: NLucas Meneghel Rodrigues <lmr@redhat.com>

* Use --sysinfo=off in the current functional tests. It will
drop the total execution time of the tests.

* Add a new functional test `sysinfo_tests.py` which will actually
test if the sysinfo directory is being created or not, depending on
the proper command line option.

OK, I did it. This has got to be the most stupid commit ever on this repo.
Still, "print 'foo'" annoys me, and if want to have Avocado running on
Python 3 anytime, we would have to do this anyway.
Signed-off-by: NCleber Rosa <crosa@redhat.com>

Self test files used to be executable, but that changed on 143b574c.
So, it doesn't make much sense to keep the shebangs + python commands
on them.
Signed-off-by: NCleber Rosa <crosa@redhat.com>