The Xunit and JSON test cases inherit from PluginsTest. This test defines
some hardcoded-values tests, which are then inherited and run for each
inherited class with the exact same values. This patch creates abstract
class, which all 3 cases share and makes them inherit from it, rather
than from the actual testcase.
Signed-off-by: NLukáš Doktor <ldoktor@redhat.com>

Even though I may be the one to blame about coming up with this
"simple import magic", I believe it was a mistake and we should get
rid of them.

There are a couple of other ways to actually do development out of
of a source tree, including running unittests that do not require
this amount of boiler plate code. Examples include just setting
the PYTHONPATH environment variable to actually run (setuptools based)
`python setup.py develop`.

The little bits of what looks like the import magic that was left, is
not really import magic. It's just that functional tests need to locate
the Avocado source tree base directory to run the test runner from it.
Signed-off-by: NCleber Rosa <crosa@redhat.com>

To be honest, our test code could still be kept in the same directories and
have the same names. But I think we can improve two things here:

1) Flatten a little bit the directory structure of selftests. Two path
components are being dropped here: "all" and "avocado". So that
"selftests/all/functional/avocado" becomes simply
"selftests/functional".

2) File names match what is, by default, recognized by unittest (the
Standard Library module) based discovery of tests. That means that
doc_build_test.py becomes test_doc_build.py. Not a big deal IMHO.
Signed-off-by: NCleber Rosa <crosa@redhat.com>

Add some functional tests that ensure that we don't
get caught by mistakes made when modifying the human
test reporting code again (or at least, not by the
obvious ones).
Signed-off-by: NLucas Meneghel Rodrigues <lmr@redhat.com>

In some external test suites, what is defined as a "test" can not be
run by itself and need a specific runner script/tool.

This introduces the concept of an "inner runner", that is, a custom
test runner or tool that can deal with custom tests that do follow
the Avocado SIMPLE test definition.

This implements the Trello card:

 https://trello.com/c/TpXecE2n/486-introduce-inner-runner-to-avocado-run

More information can be found in the man page section about it.

Changes from v1:
 * removed "level" from "inner level runner" mentions
 * replicated docs from man page to "main" docs
Signed-off-by: NCleber Rosa <crosa@redhat.com>

Instead of creating at the end of the job. This allows users running
long (time wise) jobs to be monitored by "following" the
$AVOCADO_RESULT/latest/job.log file.

This implements the card described at:

 https://trello.com/c/QJ4wCD8x/447-reconsider-when-the-latest-log-symlink-is-createdSigned-off-by: NCleber Rosa <crosa@redhat.com>

On multiple places we modify the sys.path to make avocado work from
sources. Instead of `append` we should `insert` the path as when running
from avocado sources directory, we want to use the modules from here,
rather than from installed version.
Signed-off-by: NLukáš Doktor <ldoktor@redhat.com>

Previously the "file" loader was special and always matched the URL
which might be confusing and lead to problems when developing additional
plugins.

This commit tires to define the way loaders should work. They must never
fail (unless of uncaught exception which is reported and next available
plugins is used). Instead when the URL produces broken test or even a
single corrupted subtest, consider it as URL not for me and return empty
set. The loader proxy uses next plugin to resolve this URL. When no
plugin resolves the URL user is advised to run "avocado list -V" to see
the potential problems.

The order of loader plugins was also changed. Now the file-loader is
first by default and all remaining ones are loaded by the plugin
priority. The order can be changed in setting
"plugins/loader_plugins_priority". It uses "name" variable of the each
loader plugin. Not listed plugins are loaded afterwards.

Additionally the whole loader API was simplified. Now it only accepts
one parameter - list_tests - which is 3-state:

1. ALL       - all tests including broken/incorrect ones
2. AVAILABLE - available tests (for listing purposes)
2. DEFAULT   - default tests (to run when no url given)
Signed-off-by: NLukáš Doktor <ldoktor@redhat.com>

This patch renames the fail_on_error decorator to fail_on and adds
support to supply list of exceptions. This way the decorator can be used
not only to make tests fail on generic exceptions but also to specify
parts of code with expected failures of given type.
Signed-off-by: NLukáš Doktor <ldoktor@redhat.com>

Give to test writers the fail_on_error decorator, which
should turn any generic exception into an avocado
TestFail.

This is important for some tests that can raise
any exception during a complex operation, and any of those
exceptions can be considered a test failure.
Signed-off-by: NLucas Meneghel Rodrigues <lmr@redhat.com>

Make sure avocado complains loudly when people try to call
skip() from outside setUp() - this shouldn't be allowed in
avocado tests. Let the user know of the problem and ask them
to fix their tests. Also, update the docstring for skip(),
which will reduce the chances of test writers using it in
inappropriate places.
Signed-off-by: NLucas Meneghel Rodrigues <lmr@redhat.com>

On unhandeled unknown exceptions the test should finish with ERROR
rather than FAIL as the test developer didn't think it might happen.

This behavior is configurable in config when the user have a different
opinion.
Signed-off-by: NLukáš Doktor <ldoktor@redhat.com>

It was previously not possible for a test to skip itself (due to say,
missing requirements) during the setUp() phase. Any exception caught
at this point was reflected as a test failure.
Signed-off-by: NCleber Rosa <crosa@redhat.com>
Signed-off-by: NLucas Meneghel Rodrigues <lmr@redhat.com>

Add the --paginator (on|off) option to 'avocado config'.
This option has been missing from that command, which
is inconsistent with the other listing options we have.
Signed-off-by: NLucas Meneghel Rodrigues <lmr@redhat.com>

Move avocado.plugins to avocado.core.plugins, in conformance to our
purpose of keeping things not related to test developers,
inside avocado.core, isolated from users.
Signed-off-by: NRudá Moura <rmoura@redhat.com>

So that we clean up results after we are done with unittests.

Changes from v1:
 * Used super() instead of simply calling the method of the
   parent class, given that the the unittests are new object
   style classes, per ldoktor's suggestion.
Signed-off-by: NLucas Meneghel Rodrigues <lmr@redhat.com>

We're going one step ahead to unittest  compatibility.
Now we obsolete action() method and use runTest().
Then what is runTest() will be run().

Update the selftests and tests regarding this modification.
Signed-off-by: NRudá Moura <rmoura@redhat.com>

This patch is initial support for people using custom bash scripts
with avocado. Tests should use:

    PATH=$(avocado "exec-path"):$PATH

and then they can utilize the helpers.

This version contain functions to write to Test.log the same way it's
possible from python including failing the test with TestWarn in case
avocado_warn was used.
Signed-off-by: NLukáš Doktor <ldoktor@redhat.com>

We can't install libvirt-python using pip since the
VM used by Travis runs Ubuntu 12.04 LTS (libvirt headers
are too old), so we have to settle for not requiring
all plugins (VM) to be available. It will still be tested
on 2.7, though.

Changes from v1:
 * Per ldoktor's suggestion, change check of exact python
   version being 2.7.x to python version >= 2.7.0.
Signed-off-by: NLucas Meneghel Rodrigues <lmr@redhat.com>

Signed-off-by: NLucas Meneghel Rodrigues <lmr@redhat.com>

Verify that errors in the list command really go to stderr.
Signed-off-by: NLucas Meneghel Rodrigues <lmr@redhat.com>

More checks to make sure we are not outputting errors to
the wrong loggers.
Signed-off-by: NLucas Meneghel Rodrigues <lmr@redhat.com>

Create an additional logging handler for the avocado application,
that outputs to stderr, and two filters, that filter messages
per logging level. This way, normal information (app output)
goes to stdout and error messages go to stderr. The unittests
were updated accordingly.
Signed-off-by: NLucas Meneghel Rodrigues <lmr@redhat.com>

Instead of delaying the parsing of the test list to code that handles
the jobs, let's handle that early, and in the usual Python way, simply
by relying on argparse.

For other, possibly more complex command line requirements, custom code
can be written, but it should still be placed at the argparse level.
Signed-off-by: NCleber Rosa <crosa@redhat.com>

Add support to inspect and find tests inside a directory, recursively,
by using the method `discover_url`. It takes a url (path) and
returns the tests parameters it discovers.

The loader object has the .discover() method, that can be used
to inspect the location, and the .validate() method, that will
discover any problems on the user's input.

We are also changing the behavior of avocado to error out on
invalid inputs (missing paths or files that are not avocado
tests). Therefore, the concept of MISSING or NOT_A_TEST both
disappear from the UI. Unittests were updated to reflect the
new status quo.
Signed-off-by: NRudá Moura <rmoura@redhat.com>
Signed-off-by: NLucas Meneghel Rodrigues <lmr@redhat.com>

* Use --sysinfo=off in the current functional tests. It will
drop the total execution time of the tests.

* Add a new functional test `sysinfo_tests.py` which will actually
test if the sysinfo directory is being created or not, depending on
the proper command line option.

OK, I did it. This has got to be the most stupid commit ever on this repo.
Still, "print 'foo'" annoys me, and if want to have Avocado running on
Python 3 anytime, we would have to do this anyway.
Signed-off-by: NCleber Rosa <crosa@redhat.com>

Self test files used to be executable, but that changed on 143b574c.
So, it doesn't make much sense to keep the shebangs + python commands
on them.
Signed-off-by: NCleber Rosa <crosa@redhat.com>

When avocado analyzes a path that looks like a python module,
first it'll try to import the module, then look for an appropriate
test class inside this module. Improve the test load heuristics
by looking for classes that inherit from avocado.test.Test, and
not the rudimentary detection, inspired by autotest.

If the module import went fine, but we simply can't find an
avocado test inside of it, if the file is executable we'll
try to execute it as a simple test. If it's not executable,
it'll be considered as not a test (same count as not_found tests).
If we can't import the module due to a test bug, once again we'll
check if the file is executable - if executable, we'll execute it
as a simple test, if not, we'll consider it a buggy test.

Unfortunately there's no way to tell apart a buggy avocado test
and a random py module with a bug, so both will be treated the same.
Adjust code to reflect this new status.
Signed-off-by: NLucas Meneghel Rodrigues <lmr@redhat.com>

Since the 'config' plugin supersedes functionality
in the 'datadir' plugin, let's remove it from avocado.
Update the configuration accordingly.
Signed-off-by: NLucas Meneghel Rodrigues <lmr@redhat.com>

Due the concept of "drop-in" tests didn't match the actual
purpose of running any script or binary as a test in Avocado, we're
going to use the term "simple test" (defined in SimpleTest class)
to contrast with "native test", which are Python modules that
uses Avocado's API.
Signed-off-by: NRudá Moura <rmoura@redhat.com>

Put the sysinfo and test_lister plugins into separate
plugin files. Although not strictly necessary, the
neat 1 plugin - 1 source file relation makes things
organized and easy to understand.
Signed-off-by: NLucas Meneghel Rodrigues <lmr@redhat.com>

Signed-off-by: NLucas Meneghel Rodrigues <lmr@redhat.com>

Remove the license text from files inside docs, examples and selftest,
to keep the license text only in relevant parts of the project.

References:

* https://fedoraproject.org/wiki/Packaging:LicensingGuidelines
* https://www.gnu.org/licenses/gpl-howto.htmlSigned-off-by: NRudá Moura <rmoura@redhat.com>

This patch replaces most of the calls to sleeptest in our functional
test with passtest, in an attempt to reduce, without invalidating the
purporse, the time it takes to run them.
Signed-off-by: NCleber Rosa <crosa@redhat.com>

Introduce avocado.utils.script module to handle the creation
of scripts stored in the file system.

The module includes a flexible class for creating generic scripts and
 includes utilities functions too. Some examples:

    CONTENT = """
    #!/bin/sh
    echo "Hi There!"
    """

    x = avocado.utils.script.Script('/var/tmp/test.sh', CONTENT)
    x.save()
    ... do something with x, like run it ...
    x.remove()

    t = avocado.utils.script.make_temp_script('test.sh', CONTENT)
    ... do something with y, like run it ...

Avocado functional tests makes use of script creation,
so we will handle these creation using this proper module.
Signed-off-by: NRudá Moura <rmoura@redhat.com>

Now, introducing .notify() methods that do not reference
the UI directly, and migrate all plugins to use it.
Signed-off-by: NLucas Meneghel Rodrigues <lmr@redhat.com>

Signed-off-by: NLucas Meneghel Rodrigues <lmr@redhat.com>

We have tests/ and selftests/ in avocado (the test runner).
The former includes examples of tests that make use of the
avocado API, and are also used to test avocado via CI.

This is confusing, as it doesn't make it clear to users that
tests are not part of the avocado framework. Since we have a
directory called examples with some other files (multiplexer,
plugins), move the tests to there (and keep using them in the CI).
Signed-off-by: NLucas Meneghel Rodrigues <lmr@redhat.com>

In the XML output, we can't use an exclusive attribute "not_found"
for tests that has been not found. So we're going to count
"not found" as error and remove the attribute "not_found".
Signed-off-by: NRuda Moura <rmoura@redhat.com>