Add new 'aiostress' test based on autotest.
Signed-off-by: NAmador Pahim <apahim@redhat.com>

Add new 'compilebench' test based on autotest.
Signed-off-by: NAmador Pahim <apahim@redhat.com>

Add new 'bonnie' test based on autotest.
Signed-off-by: NAmador Pahim <apahim@redhat.com>

With the same one that comes from the test documentation.
Signed-off-by: NCleber Rosa <crosa@redhat.com>

Add new 'stress' test based on autotest.
Signed-off-by: NAmador Pahim <apahim@redhat.com>

While going through the dependency list (requirements*.txt files) and
performing our self tests out of virtual environments, I noticed that
some tests are run outside the virtual environments.

The reason is that, even though the virtual environment is activated
for the test session (and say, `which python` gives `/venv/bin/python`),
we have hard coded `/usr/bin/python` in most places.

According to the some discussions on the virtualenv project itself[1],
a quick solution is to revert to the also common `/usr/bin/env python`
way of pointing to the Python interpreter.

[1] - https://github.com/pypa/virtualenv/issues/124Signed-off-by: NCleber Rosa <crosa@redhat.com>

This example file should have been remove together with commit
7b969527. Better late than never,
so let's do it now.
Signed-off-by: NCleber Rosa <crosa@redhat.com>

To sync with the latest changes in Avocado.
Signed-off-by: NCleber Rosa <crosa@redhat.com>

This is an automated mass import cleanup across all
avocado source files:

1) Imports follow the order:
 * Standard library imports
 * Non standard library external imports
 * Internal imports
 All separated by a single line
2) One line between imports and the rest of the code
3) Remove unused imports in the process
Signed-off-by: NLucas Meneghel Rodrigues <lmr@redhat.com>

Now we don't need to explicitly specify --vm-hostname.
Signed-off-by: NLucas Meneghel Rodrigues <lmr@redhat.com>

Make sure we have avocado_ + __name__ prefixes to the dirs
created with tempfile, so that we have an easy way to spot
when we are not cleaning up directories properly. This mostly
concerns unittests, although the kernel_build lib and the
iso9660 lib also uses that API.
Signed-off-by: NLucas Meneghel Rodrigues <lmr@redhat.com>

The name was not correct as the test should error-out.
Signed-off-by: NLukáš Doktor <ldoktor@redhat.com>

We did change the short name identifiers that are displayed
in `avocado plugins`. Let's update so people don't get lost.
Signed-off-by: NLucas Meneghel Rodrigues <lmr@redhat.com>

Currently when one passes exception, which is not instance of Exception,
avocado proceeds and returns "INTERRUPTED" test instead of "ERROR". This
for example happens when old-style-class is used as exception. This
patch uses pure "except" without any argument to catch such exceptions
and then it wraps them in "exceptions.TestError" instead.

The same applies to "setUp" and "tearDown", they only use different
class as a wrapper.
Signed-off-by: NLukáš Doktor <ldoktor@redhat.com>

With ldoktor's fix 136a78b6 this is unnecessary and we
risk it being confused for people looking at the
examples.
Signed-off-by: NLucas Meneghel Rodrigues <lmr@redhat.com>

GNU make (mis)understands paths that contain ":". The reason is that they look
like static patterns:

 https://www.gnu.org/software/make/manual/html_node/Static-Usage.html#Static-Usage

The exact error message one would see would be similar to:

 Makefile:XXX: *** multiple target patterns.  Stop.
Signed-off-by: NCleber Rosa <crosa@redhat.com>

The `avocado.utils.kernel_build.KernelBuild` utility class can now
build a Linux kernel without an external configuration file, so let's
add this possibility to the test itself.
Signed-off-by: NCleber Rosa <crosa@redhat.com>

And let's use a generated config by default, that is, let's drop the
custom config file we ship with the test.
Signed-off-by: NCleber Rosa <crosa@redhat.com>

Since it's not possible (yet?) to depend on existing virtual machines
to run tests for some of the Avocado features, including many of the
VM Plugin features, let's add a test as an example that can be run
manually instead. While they will not be part (at this time?) of the
functional test suite, it may serve as a reference/example in the
future.

The approach chosen here is to have a first run of Avocado making sure that
a flag file does not exist. A second run of Avocado, now executed with an
additional `--vm-cleanup` parameter runs a test that creates the flag file.
Finally, the third and last execution of re-runs the same test as the first
run, checking that the flag file does *not* exist, that is, the clean up of
the VM did work.

Because this test requires a libvirt domain and the hostname/address
of that virtual machine, these parameters assume no defaults for
safety and/or security causes. The test will be skipped if these are
not configured properly.  Please edit the vm-cleanup.yaml file with
the appropriate parameters.
Signed-off-by: NCleber Rosa <crosa@redhat.com>

The logs from the trinity test are full of ANSI characters that add
colors to messages, which makes it difficult to read/manipulate
them.

Fix that by passing -m to trinity. From trinity --help:

   '--monochrome,-m: don't output ANSI codes'

Avocado-server now uses only django selftests. Let's change the
functional test to `curl ...`.
Signed-off-by: NLukáš Doktor <ldoktor@redhat.com>

Signed-off-by: NLukáš Doktor <ldoktor@redhat.com>

Signed-off-by: NLukáš Doktor <ldoktor@redhat.com>

This patch renames the fail_on_error decorator to fail_on and adds
support to supply list of exceptions. This way the decorator can be used
not only to make tests fail on generic exceptions but also to specify
parts of code with expected failures of given type.
Signed-off-by: NLukáš Doktor <ldoktor@redhat.com>

This fixes #628.

Add a couple of simple shell scripts that can be listed
and executed with avocado. Let's update our documentation
to use this new dir and avoid people confused with our
wrappers.
Signed-off-by: NLucas Meneghel Rodrigues <lmr@redhat.com>

Include our newest plugin into the list of functional tests.
Signed-off-by: NLucas Meneghel Rodrigues <lmr@redhat.com>

Give to test writers the fail_on_error decorator, which
should turn any generic exception into an avocado
TestFail.

This is important for some tests that can raise
any exception during a complex operation, and any of those
exceptions can be considered a test failure.
Signed-off-by: NLucas Meneghel Rodrigues <lmr@redhat.com>

Make sure avocado complains loudly when people try to call
skip() from outside setUp() - this shouldn't be allowed in
avocado tests. Let the user know of the problem and ask them
to fix their tests. Also, update the docstring for skip(),
which will reduce the chances of test writers using it in
inappropriate places.
Signed-off-by: NLucas Meneghel Rodrigues <lmr@redhat.com>

In order to keep things simple and clear, enforce the
following test results rules:

 * `PASS`: the test passed, which means all conditions
   being tested have passed.
 * `FAIL`: the test failed, which means at least one
   condition being tested has failed. Ideally, it should
   mean a problem in the software being tested has been found.
 * `ERROR`: an error happened during the test execution.
   This can happen, for example, if there's a bug in the test
   runner, in its libraries or if a resource breaks unexpectedly.
   Uncaught exceptions in the test code will also result in this
   status.
 * `SKIP`: the test runner decided a requested test should not
   be executed. This can happen, for example, due to missing
   requirements in the test environment or when there's a job
   timeout.

In order to do that, let's not make the behavior on exceptions
other than TestFail to be configurable. Therefore, the config
value uncaught_exception_result must be removed and the status
of the test always be 'ERROR'.
Signed-off-by: NLucas Meneghel Rodrigues <lmr@redhat.com>

This was added to the Travis CI job, so it doesn't need
to be explicitly executed. Let's drop it.
Signed-off-by: NLucas Meneghel Rodrigues <lmr@redhat.com>

Add a couple new tests to release.json:

1) Test that avocado-virt can assign values to the
   command line correctly.
2) Run the avocado migration test, as it's a fairly
   complex test that exercises a lot of functionality
   in avocado-virt.
Signed-off-by: NLucas Meneghel Rodrigues <lmr@redhat.com>

Summarize the test params to help people understand and run and
adjust test variants via multiplexer easily.
Signed-off-by: NLukáš Doktor <ldoktor@redhat.com>

The gendata is a nice example of multiple-subtests test. Add
a functional test that checks gendata creates the files
correctly and that two subtests are spawned.
Signed-off-by: NLukáš Doktor <ldoktor@redhat.com>

The runTest method has been obsoleted and everyone should be using
"test*" method(s) instead. This patch removes the support and replaces
the default and example entry points to "test".
Signed-off-by: NLukáš Doktor <ldoktor@redhat.com>

On unhandeled unknown exceptions the test should finish with ERROR
rather than FAIL as the test developer didn't think it might happen.

This behavior is configurable in config when the user have a different
opinion.
Signed-off-by: NLukáš Doktor <ldoktor@redhat.com>

Introduced with commit d6abdcd4,
avocado tests no longer report correctly to the unittest runner.
The reason is that there is a structure that must be correctly
updated by the run() method (the result instance), that was not
being updated in the new implementation.

Therefore, one possible fix, presented here, is to once again
*not* implement run() at all. this way we avoid messing around
with internal data structures, and when avocado is executed
under nosetests, or the default unittest runner, we get the
expected behavior of the method. The downside is that we have
less perceived integration among both classes.
Signed-off-by: NLucas Meneghel Rodrigues <lmr@redhat.com>

It was previously not possible for a test to skip itself (due to say,
missing requirements) during the setUp() phase. Any exception caught
at this point was reflected as a test failure.
Signed-off-by: NCleber Rosa <crosa@redhat.com>
Signed-off-by: NLucas Meneghel Rodrigues <lmr@redhat.com>

Instead of playing with core exceptions or downgrading those exceptions
to the base Test API, let's allow users to do so with convenience
methods.

The example tests now make use of those, and can now remain in the
examples directory instead of the former proposal of turning them
into "functional test assets" and moving them to the selftests
directory.
Signed-off-by: NCleber Rosa <crosa@redhat.com>

This means also moving its dependencies, namely spark and gdbmi_parser,
to a place under avocado/utils. I've chosen to add another level here
to clearly identify those deps as 3rd party external modules that are
reused in Avocado.
Signed-off-by: NCleber Rosa <crosa@redhat.com>

The initial idea about the avocado.restclient namespace was that
eventually users could write scripts using that as an API to
interact with avocado-server using Python code.

While the idea is still valid, the avocado-server interface is
a simple and standards compliant RESTful interface. Thus, users
can still easily interact with it without us having to support
a formal Python API at this point.

In the future, if need arises, we can improve the Python REST
client API and promote it to a public namespace.
Signed-off-by: NCleber Rosa <crosa@redhat.com>