It turns out we were basing the links of the URL report
in the 'url' attribute, when we should be using 'test'.
This fixes an issue that came up with the new multiplexed
tests in VM and remote plugins, where the HTML report
link was invalid. Fix the unittest, that had the wrong
behavior built into it.
Signed-off-by: NLucas Meneghel Rodrigues <lmr@redhat.com>

The Xunit and JSON test cases inherit from PluginsTest. This test defines
some hardcoded-values tests, which are then inherited and run for each
inherited class with the exact same values. This patch creates abstract
class, which all 3 cases share and makes them inherit from it, rather
than from the actual testcase.
Signed-off-by: NLukáš Doktor <ldoktor@redhat.com>

The name was not correct as the test should error-out.
Signed-off-by: NLukáš Doktor <ldoktor@redhat.com>

Signed-off-by: NLucas Meneghel Rodrigues <lmr@redhat.com>

This patch adds support for using multiplex files and remote plugin
together. It copies all of the specified multiplex files and then
uses them in execution.
Signed-off-by: NLukáš Doktor <ldoktor@redhat.com>

Fedora 21 ships psutils 0.6, that has the API psutil.get_pid_list()
instead of psutil.pids(). Let's make both compatible, for the sake
of running unittests on COPR builds.
Signed-off-by: NLucas Meneghel Rodrigues <lmr@redhat.com>

Currently when one passes exception, which is not instance of Exception,
avocado proceeds and returns "INTERRUPTED" test instead of "ERROR". This
for example happens when old-style-class is used as exception. This
patch uses pure "except" without any argument to catch such exceptions
and then it wraps them in "exceptions.TestError" instead.

The same applies to "setUp" and "tearDown", they only use different
class as a wrapper.
Signed-off-by: NLukáš Doktor <ldoktor@redhat.com>

The way we built the unittest runner will happily
ignore errors, which is bad from a CI standpoint.
Let's ensure that errors don't go unnoticed by
analyzing the results object and see if it's got
errors or failures.
Signed-off-by: NLucas Meneghel Rodrigues <lmr@redhat.com>

Add simple script to see the "core" and "utils" unittest code coverage.
Signed-off-by: NLukáš Doktor <ldoktor@redhat.com>

If between psutil.procs() and the instantiation of the Process
object the given pid has terminated, we'll get a NoSuchProcess
error from psutils. Let's expand the list comprehension to avoid
getting caught by such a condition.
Signed-off-by: NLucas Meneghel Rodrigues <lmr@redhat.com>

All other tests (unit and functional) are based on unittest. Let's do
the same to this one, so that the default unittest discover code finds
it and runs it.
Signed-off-by: NCleber Rosa <crosa@redhat.com>

Even though I may be the one to blame about coming up with this
"simple import magic", I believe it was a mistake and we should get
rid of them.

There are a couple of other ways to actually do development out of
of a source tree, including running unittests that do not require
this amount of boiler plate code. Examples include just setting
the PYTHONPATH environment variable to actually run (setuptools based)
`python setup.py develop`.

The little bits of what looks like the import magic that was left, is
not really import magic. It's just that functional tests need to locate
the Avocado source tree base directory to run the test runner from it.
Signed-off-by: NCleber Rosa <crosa@redhat.com>

The unittest module, and unittest2 backport on Python 2.6, is capable of
finding unittests and running them. So, effectively, we do not need nose
at all.

This patch removes the dependency on nose, replacing the run script
with a version based solely on the unittest module.

One change of functionality is that run now looks and runs tests in the
standard selftests directories (unit, functional, doc), and does not
accept command line arguments. If one wants to run a subset of them,
it's pretty easy to just use the unittest module for that:

 $ python -m unittest discover -s selftests/unit
Signed-off-by: NCleber Rosa <crosa@redhat.com>

When running this functional test with the standard unittest
discovery/runner it produces output that conflicts with the runner
output.

It looks like this output is not necessary for the test itself, and
not that helpful for debugging purposes, so, let's just silence it.
Signed-off-by: NCleber Rosa <crosa@redhat.com>

To be honest, our test code could still be kept in the same directories and
have the same names. But I think we can improve two things here:

1) Flatten a little bit the directory structure of selftests. Two path
components are being dropped here: "all" and "avocado". So that
"selftests/all/functional/avocado" becomes simply
"selftests/functional".

2) File names match what is, by default, recognized by unittest (the
Standard Library module) based discovery of tests. That means that
doc_build_test.py becomes test_doc_build.py. Not a big deal IMHO.
Signed-off-by: NCleber Rosa <crosa@redhat.com>

Add functional automated tests for the Ctrl+C feature.
They use aexpect and psutil, so add their dependencies
to the relevant requirement files.
Signed-off-by: NLucas Meneghel Rodrigues <lmr@redhat.com>

Add some functional tests that ensure that we don't
get caught by mistakes made when modifying the human
test reporting code again (or at least, not by the
obvious ones).
Signed-off-by: NLucas Meneghel Rodrigues <lmr@redhat.com>

In some external test suites, what is defined as a "test" can not be
run by itself and need a specific runner script/tool.

This introduces the concept of an "inner runner", that is, a custom
test runner or tool that can deal with custom tests that do follow
the Avocado SIMPLE test definition.

This implements the Trello card:

 https://trello.com/c/TpXecE2n/486-introduce-inner-runner-to-avocado-run

More information can be found in the man page section about it.

Changes from v1:
 * removed "level" from "inner level runner" mentions
 * replicated docs from man page to "main" docs
Signed-off-by: NCleber Rosa <crosa@redhat.com>

Let's be practical and simply destroy (pull the plug on the VM) that
will have its state (snapshot reverted).

Not shutting down (or powering off/destroying) the VM has the side
effect of not reverting the state of the (still running) VM on a
couple of platforms, including EL7.

This fixed issue #396.
Signed-off-by: NCleber Rosa <crosa@redhat.com>

Instead of creating at the end of the job. This allows users running
long (time wise) jobs to be monitored by "following" the
$AVOCADO_RESULT/latest/job.log file.

This implements the card described at:

 https://trello.com/c/QJ4wCD8x/447-reconsider-when-the-latest-log-symlink-is-createdSigned-off-by: NCleber Rosa <crosa@redhat.com>

On multiple places we modify the sys.path to make avocado work from
sources. Instead of `append` we should `insert` the path as when running
from avocado sources directory, we want to use the modules from here,
rather than from installed version.
Signed-off-by: NLukáš Doktor <ldoktor@redhat.com>

The sysinfo.post collects "/var/log/messages" by default. This can fail
when not running as superuser. Adjust the unittests to handle booth
variants.
Signed-off-by: NLukáš Doktor <ldoktor@redhat.com>

This library adds utility functions for linux kernel modules.
The functionality covers listing, loading and removing
loadable modules.
Signed-off-by: NLucas Meneghel Rodrigues <lmr@redhat.com>

Add avocado.utils.output, a library with utility functions
for user friendly display of information.
Signed-off-by: NLucas Meneghel Rodrigues <lmr@redhat.com>

This library is used to control the linux boxes init systems,
(sysvinit vs systemd). It's useful for some tests to be able
to stop/start services, so that we transplanted it from
autotest.
Signed-off-by: NLucas Meneghel Rodrigues <lmr@redhat.com>

Instead of putting outputs on separate lines, condensate
test results summary into a single line. The new output
looks like:

$ avocado run passtest
JOB ID     : f2f5060440bd57cba646c1f223ec8c40d03f539b
JOB LOG    : /home/user/avocado/job-results/job-2015-07-27T17.13-f2f5060/job.log
JOB HTML   : /home/user/avocado/job-results/job-2015-07-27T17.13-f2f5060/html/results.html
TESTS      : 1
(1/1) passtest.py:PassTest.test: PASS (0.00 s)
RESULTS    : PASS 1 | ERROR 0 | FAIL 0 | SKIP 0 | WARN 0 | INTERRUPT 0
TIME       : 0.00 s

We updated a few unittests in order to not depend on the
looks of the human output anymore, since unless we are
specifically testing for human output behavior, the unittests
should use machine readable output. Also, the documentation
was updated to reflect the new output layout.
Signed-off-by: NLucas Meneghel Rodrigues <lmr@redhat.com>

We did notice that listing commands have trailing whitespace
at their ends. Trailing whitespaces are bad for program output
(copy/paste, among other reasons), so let's fix it.

I did notice there was no reason to not implement the base
function for the tabular output as a generator, so I've
introduced iter_tabular_output, that is a generator, and
in order to not break API unnecessarily, implemented
tabular_output in terms of iter_tabular_output.
Signed-off-by: NLucas Meneghel Rodrigues <lmr@redhat.com>

This is a rather embarassing bug. We should be testing
for equality of outputs, so assertEqual instead of
assertTrue (which in effect makes the test always pass,
regardless of the avocado output). Let's fix it.
Signed-off-by: NLucas Meneghel Rodrigues <lmr@redhat.com>

Previously the "file" loader was special and always matched the URL
which might be confusing and lead to problems when developing additional
plugins.

This commit tires to define the way loaders should work. They must never
fail (unless of uncaught exception which is reported and next available
plugins is used). Instead when the URL produces broken test or even a
single corrupted subtest, consider it as URL not for me and return empty
set. The loader proxy uses next plugin to resolve this URL. When no
plugin resolves the URL user is advised to run "avocado list -V" to see
the potential problems.

The order of loader plugins was also changed. Now the file-loader is
first by default and all remaining ones are loaded by the plugin
priority. The order can be changed in setting
"plugins/loader_plugins_priority". It uses "name" variable of the each
loader plugin. Not listed plugins are loaded afterwards.

Additionally the whole loader API was simplified. Now it only accepts
one parameter - list_tests - which is 3-state:

1. ALL       - all tests including broken/incorrect ones
2. AVAILABLE - available tests (for listing purposes)
2. DEFAULT   - default tests (to run when no url given)
Signed-off-by: NLukáš Doktor <ldoktor@redhat.com>

Test loader doesn't need to know anything about the job, the only useful
argument is `args`.

Second big change is the way loader is used. This patch uses list of
registered plugins on module-level (similarly to settings or logging)
instead of custom instance and plugins registered in args dict. This way
the plugins are loaded in the order they are registered (by plugin
priority) instead of random order.
Signed-off-by: NLukáš Doktor <ldoktor@redhat.com>

This patch renames the fail_on_error decorator to fail_on and adds
support to supply list of exceptions. This way the decorator can be used
not only to make tests fail on generic exceptions but also to specify
parts of code with expected failures of given type.
Signed-off-by: NLukáš Doktor <ldoktor@redhat.com>

Give to test writers the fail_on_error decorator, which
should turn any generic exception into an avocado
TestFail.

This is important for some tests that can raise
any exception during a complex operation, and any of those
exceptions can be considered a test failure.
Signed-off-by: NLucas Meneghel Rodrigues <lmr@redhat.com>

Make sure avocado complains loudly when people try to call
skip() from outside setUp() - this shouldn't be allowed in
avocado tests. Let the user know of the problem and ask them
to fix their tests. Also, update the docstring for skip(),
which will reduce the chances of test writers using it in
inappropriate places.
Signed-off-by: NLucas Meneghel Rodrigues <lmr@redhat.com>

subprocess.check_output is a new API starting in python 2.7.
So let's use our in house subprocess lib here to make it
compatible with 2.6.
Signed-off-by: NLucas Meneghel Rodrigues <lmr@redhat.com>

run_suite() actually happens earlier, and is only called once, so it seems
to be a better place for the check of Avocado on the remote system. If this
condition holds true, then copy the requested tests to the remote system.
Signed-off-by: NCleber Rosa <crosa@redhat.com>

The result.setup() method is now only used to setup the communication
channel, and the copy of the tests will be made conditional, based on
the presence of Avocado on the remote machine.

On the unittest side, I'm quite adverse to heavy mocking, so I favored
removing all these painful, and IMHO of little value checks.
Signed-off-by: NCleber Rosa <crosa@redhat.com>

In order to troubleshoot more effectively problems
happening in remote/vm operations, make all output
coming from fabric and paramiko to a 'remote.log'
file in the top level job directory.
Signed-off-by: NLucas Meneghel Rodrigues <lmr@redhat.com>

The gendata is a nice example of multiple-subtests test. Add
a functional test that checks gendata creates the files
correctly and that two subtests are spawned.
Signed-off-by: NLukáš Doktor <ldoktor@redhat.com>

The runTest method has been obsoleted and everyone should be using
"test*" method(s) instead. This patch removes the support and replaces
the default and example entry points to "test".
Signed-off-by: NLukáš Doktor <ldoktor@redhat.com>

On unhandeled unknown exceptions the test should finish with ERROR
rather than FAIL as the test developer didn't think it might happen.

This behavior is configurable in config when the user have a different
opinion.
Signed-off-by: NLukáš Doktor <ldoktor@redhat.com>