The name was not correct as the test should error-out.
Signed-off-by: NLukáš Doktor <ldoktor@redhat.com>

Currently when one passes exception, which is not instance of Exception,
avocado proceeds and returns "INTERRUPTED" test instead of "ERROR". This
for example happens when old-style-class is used as exception. This
patch uses pure "except" without any argument to catch such exceptions
and then it wraps them in "exceptions.TestError" instead.

The same applies to "setUp" and "tearDown", they only use different
class as a wrapper.
Signed-off-by: NLukáš Doktor <ldoktor@redhat.com>

With ldoktor's fix 136a78b6 this is unnecessary and we
risk it being confused for people looking at the
examples.
Signed-off-by: NLucas Meneghel Rodrigues <lmr@redhat.com>

GNU make (mis)understands paths that contain ":". The reason is that they look
like static patterns:

 https://www.gnu.org/software/make/manual/html_node/Static-Usage.html#Static-Usage

The exact error message one would see would be similar to:

 Makefile:XXX: *** multiple target patterns.  Stop.
Signed-off-by: NCleber Rosa <crosa@redhat.com>

The `avocado.utils.kernel_build.KernelBuild` utility class can now
build a Linux kernel without an external configuration file, so let's
add this possibility to the test itself.
Signed-off-by: NCleber Rosa <crosa@redhat.com>

And let's use a generated config by default, that is, let's drop the
custom config file we ship with the test.
Signed-off-by: NCleber Rosa <crosa@redhat.com>

Since it's not possible (yet?) to depend on existing virtual machines
to run tests for some of the Avocado features, including many of the
VM Plugin features, let's add a test as an example that can be run
manually instead. While they will not be part (at this time?) of the
functional test suite, it may serve as a reference/example in the
future.

The approach chosen here is to have a first run of Avocado making sure that
a flag file does not exist. A second run of Avocado, now executed with an
additional `--vm-cleanup` parameter runs a test that creates the flag file.
Finally, the third and last execution of re-runs the same test as the first
run, checking that the flag file does *not* exist, that is, the clean up of
the VM did work.

Because this test requires a libvirt domain and the hostname/address
of that virtual machine, these parameters assume no defaults for
safety and/or security causes. The test will be skipped if these are
not configured properly.  Please edit the vm-cleanup.yaml file with
the appropriate parameters.
Signed-off-by: NCleber Rosa <crosa@redhat.com>

The logs from the trinity test are full of ANSI characters that add
colors to messages, which makes it difficult to read/manipulate
them.

Fix that by passing -m to trinity. From trinity --help:

   '--monochrome,-m: don't output ANSI codes'

This patch renames the fail_on_error decorator to fail_on and adds
support to supply list of exceptions. This way the decorator can be used
not only to make tests fail on generic exceptions but also to specify
parts of code with expected failures of given type.
Signed-off-by: NLukáš Doktor <ldoktor@redhat.com>

Give to test writers the fail_on_error decorator, which
should turn any generic exception into an avocado
TestFail.

This is important for some tests that can raise
any exception during a complex operation, and any of those
exceptions can be considered a test failure.
Signed-off-by: NLucas Meneghel Rodrigues <lmr@redhat.com>

Make sure avocado complains loudly when people try to call
skip() from outside setUp() - this shouldn't be allowed in
avocado tests. Let the user know of the problem and ask them
to fix their tests. Also, update the docstring for skip(),
which will reduce the chances of test writers using it in
inappropriate places.
Signed-off-by: NLucas Meneghel Rodrigues <lmr@redhat.com>

In order to keep things simple and clear, enforce the
following test results rules:

 * `PASS`: the test passed, which means all conditions
   being tested have passed.
 * `FAIL`: the test failed, which means at least one
   condition being tested has failed. Ideally, it should
   mean a problem in the software being tested has been found.
 * `ERROR`: an error happened during the test execution.
   This can happen, for example, if there's a bug in the test
   runner, in its libraries or if a resource breaks unexpectedly.
   Uncaught exceptions in the test code will also result in this
   status.
 * `SKIP`: the test runner decided a requested test should not
   be executed. This can happen, for example, due to missing
   requirements in the test environment or when there's a job
   timeout.

In order to do that, let's not make the behavior on exceptions
other than TestFail to be configurable. Therefore, the config
value uncaught_exception_result must be removed and the status
of the test always be 'ERROR'.
Signed-off-by: NLucas Meneghel Rodrigues <lmr@redhat.com>

Summarize the test params to help people understand and run and
adjust test variants via multiplexer easily.
Signed-off-by: NLukáš Doktor <ldoktor@redhat.com>

The gendata is a nice example of multiple-subtests test. Add
a functional test that checks gendata creates the files
correctly and that two subtests are spawned.
Signed-off-by: NLukáš Doktor <ldoktor@redhat.com>

The runTest method has been obsoleted and everyone should be using
"test*" method(s) instead. This patch removes the support and replaces
the default and example entry points to "test".
Signed-off-by: NLukáš Doktor <ldoktor@redhat.com>

On unhandeled unknown exceptions the test should finish with ERROR
rather than FAIL as the test developer didn't think it might happen.

This behavior is configurable in config when the user have a different
opinion.
Signed-off-by: NLukáš Doktor <ldoktor@redhat.com>

Introduced with commit d6abdcd4,
avocado tests no longer report correctly to the unittest runner.
The reason is that there is a structure that must be correctly
updated by the run() method (the result instance), that was not
being updated in the new implementation.

Therefore, one possible fix, presented here, is to once again
*not* implement run() at all. this way we avoid messing around
with internal data structures, and when avocado is executed
under nosetests, or the default unittest runner, we get the
expected behavior of the method. The downside is that we have
less perceived integration among both classes.
Signed-off-by: NLucas Meneghel Rodrigues <lmr@redhat.com>

It was previously not possible for a test to skip itself (due to say,
missing requirements) during the setUp() phase. Any exception caught
at this point was reflected as a test failure.
Signed-off-by: NCleber Rosa <crosa@redhat.com>
Signed-off-by: NLucas Meneghel Rodrigues <lmr@redhat.com>

Instead of playing with core exceptions or downgrading those exceptions
to the base Test API, let's allow users to do so with convenience
methods.

The example tests now make use of those, and can now remain in the
examples directory instead of the former proposal of turning them
into "functional test assets" and moving them to the selftests
directory.
Signed-off-by: NCleber Rosa <crosa@redhat.com>

This means also moving its dependencies, namely spark and gdbmi_parser,
to a place under avocado/utils. I've chosen to add another level here
to clearly identify those deps as 3rd party external modules that are
reused in Avocado.
Signed-off-by: NCleber Rosa <crosa@redhat.com>

And while doing that, also remove avocado.linux for good, since no one else
lives there.
Signed-off-by: NCleber Rosa <crosa@redhat.com>

Only avocado.Test is of general interest to test writers. For that reason
the entire test module has been moved to the avocado.core namespace.

The recommended way to go about writing instrumented tests is something like:

   from avocado import Test
   class MyTest(Test):
      def test_foo(self):
         do_stuff()

Changes from v2:
* Fixed references in docstrings
* Fixed refrences of avocado.test in Writing Test Guide docs

Changes from v1:
* Fixed comment typo on commit message
* Fixed old reference of avocado.test (then test.Test) instead of
  avocado then Test in GDB docs.
Signed-off-by: NCleber Rosa <crosa@redhat.com>

Because of the parameter behaviour change this test was failing to get
the name of the tarball to use during the test.

Once that was fixed, there was a couple of build errors. Instead of
debugging the build of the 1.4 version, I tried version 1.5, which
built without any problems. Let's use this oppotunity to fix the
test (wrapper) and update the (original test) tarball.
Signed-off-by: NCleber Rosa <crosa@redhat.com>

This patch adds documentation of the "new params API" and removes the
compatility layer. From now on only the API with paths is valid:

    self.params.get(key, path='*', default=None)

where the default path '*' matches anything from --mux-entry path.
Default path is `/test`.
Signed-off-by: NLukáš Doktor <ldoktor@redhat.com>

This patch modifies all example tests to use the new API. It uses the
default path, which is `*`, which matches the mux-entry paths ONLY.
In order to get the values it's necessarily to inject the YAML files
into the mux-entry path, by default `/test`, eg:

    avocado run sleeptest -m /test:examples/tests/sleeptest.py.data/
    sleeptest.yaml

It might seen as a bit more complicated to execute, but it's definitely
easier for maintaining and combining existing YAML files. To execute
matrix of tests you simply inject them into 2 different `/test` children,
eg: `-m /test/by_length:lengths.yaml /test/by_method:methods.yaml`.
Signed-off-by: NLukáš Doktor <ldoktor@redhat.com>

This patch modifies the multiplexer logic to support full recursive
multiplexation, unlike the old version which only flattened the
multiplexed nodes. This allows greater flexibility.

Additionally it removes the !join flag and inverts the logic to use !mux
instead. By default leaves are not multiplexed and all are part of the
current params. Users can specify exact nodes which get multiplexed by
using !mux keyword.
Signed-off-by: NLukáš Doktor <ldoktor@redhat.com>

This reverts commit 71d47611.

The idea in the mentioned commit were sent to the upstream mailing
list and a better solution was proposed, so it makes no sense to
maintain support in Avocado for something that will never make into
GDB itself.

Export avocado.core.job.main as avocado.main, so that make it look
like an API for test developers to use.
Signed-off-by: NRudá Moura <rmoura@redhat.com>

Move avocado.job to avocado.core.job, in conformance to our
purpose of keeping things not related to test developers,
inside avocado.core, isolated from users.
Signed-off-by: NRudá Moura <rmoura@redhat.com>

There are python traps (they can't be called scripts) which on import
modify stdin/out/err. This even on discovery makes avocado behave very
badly and it's hard to discover what and when it happened. This patch
provides dummy stdin/out/err when discovering the tests.

This solution only takes care of stdin/out/err related issues, safer
method would be to spawn another process and gather the info there.
Still we can't avoid modifying the OS during the discovery, but that's
IMO acceptable for a test-framework.

The examples/tests/failtest_ugly.py is used in basic_tests unittest
using "avocado list" (without this patch it waits for input, than it
prints the test's outputs and exits without listing the tests).
Signed-off-by: NLukáš Doktor <ldoktor@redhat.com>

After all, this is not a unittest of avocado per se, but
an example test. Besides, when executing it as a unittest, we
create an undesirable job directory in ~/avocado/job-results.
Signed-off-by: NLucas Meneghel Rodrigues <lmr@redhat.com>

Multiplextest contains in /guest/os contains Linux and Windows branches.
These branches are mutually exclusive thus they need to be !join-ed.

Additionally I reworked the /host/kernel_config and split the "numa" vs.
"page_size" variants and added "default". This IMO reflects the
real-world usage better.
Signed-off-by: NLukáš Doktor <ldoktor@redhat.com>

This is a testing version of params with paths support. By default
it acts as the old Params, but you can use params.get(key, path=....)
to utilize new-params handling.

For now the PATH for relative paths is hardcoded to "/test/*" but it
will be specifiable on the command line. I just need to pass it to the
test which I'd like to do over "metadata" (see below) rather then adding
it temporarily to the params and risk colisions.

The code is not optimized and in order to be compatible it uses old
concepts. Some of them I'd like to get rid of in near future:

* default_params => currently used only to set default "metadata" like
  test timeout. I'd like to replace it for RW Test.metadata, which would
  be overwritten durint __init__ if params contain the keys in specific
  paths. IMO they shouldn't be shared with test params.
* test_factory's params are currently dict with some metadata like
  wether this test is executed from directory or by uri. Again, they are
  metadata and should be handled separately. Some of them might be reused
  to params if needed, but not generaly.
* create separated multiplexer plugin generate multiple variants
* reconsider the need for "objects", "object_params" and "object_counts"
* couple of others I forget to mention...

There is one real change I made, I got rid of ${key}_${type} to specify
type of the argument. Yaml supports any type we might like and by
our convention all tests should be written to work without any params.
Thus this is IMO more confusing, than beneficial. But if you insist
I can copy&paste the support for it.
Signed-off-by: NLukáš Doktor <ldoktor@redhat.com>

We decided to use params.get(key, default) rather than flat dictionary
in test header. This way should be easier to read to code without the
need to look up and also use different defaults for different namespaces
(or even code branches).
Signed-off-by: NLukáš Doktor <ldoktor@redhat.com>

We're going one step ahead to unittest  compatibility.
Now we obsolete action() method and use runTest().
Then what is runTest() will be run().

Update the selftests and tests regarding this modification.
Signed-off-by: NRudá Moura <rmoura@redhat.com>

Use -e to exit on first failure.
Signed-off-by: NLukáš Doktor <ldoktor@redhat.com>

This patch is initial support for people using custom bash scripts
with avocado. Tests should use:

    PATH=$(avocado "exec-path"):$PATH

and then they can utilize the helpers.

This version contain functions to write to Test.log the same way it's
possible from python including failing the test with TestWarn in case
avocado_warn was used.
Signed-off-by: NLukáš Doktor <ldoktor@redhat.com>

The NastyException error is back:

class NastyException(Exception):
    def __init__(self, msg):
        self.msg = msg
    def __str__(self):
        return self.msg

crashes Avocado. The problem is we are trying to re-raise the exceptions
and then read the traceback again. Even reading of the details of this
very nasty written exception causes critical failure.

What this patch does instead of handling issues wherever we produce
them, it waits till the exception bubbles to the entry-point and it
tries to read it. If it fails, we log the original traceback
(information about the failure) and then user-friendly message saying
we're unable to retrieve the original failure.

Additionally I had to modify the xunit output as in case of very nasty
inputs it's unable to reproduce the values.
Signed-off-by: NLukáš Doktor <ldoktor@redhat.com>

Change behavior to expect the test to pass, by using test asserts.
Signed-off-by: NRudá Moura <rmoura@redhat.com>

This patch simplifies the concept of WARN test. In case user uses the
self.log.warn or self.log.warning methods, test is marked as dirty and
in case everything else worked fine it finishes with TestWarn exception.
Signed-off-by: NLukáš Doktor <ldoktor@redhat.com>