Avocado defines avocado.core.result.TestResult, so that alternative
test result classes can inherit from it. When a user writes a new
test result class, he's left wondering how to activate that.

First, for consistency, Avocado should define proper interfaces
for "test result plugins". This is on sight, and will ultimately
allow users to register new test result implementations simply by
registering the setuptools entry points, just as it's currently
done (on versions >= 0.31.0) for command line related plugins.

Until then, let's at least improve the current state of things.
These changes believe that the following improvements are given here:

* avocado.core.job.Job._set_output_plugins now won't look at every
  possible key in the application arguments, and won't silence
  possible errors while trying to instantiate test result classes.
* test result writers can now use a documented method for registering
  their test result classes.

The same approach was not chosen to be done with the test runner
classes because there can only by one test runner per Avocado job.
Signed-off-by: NCleber Rosa <crosa@redhat.com>

There is an issue when running SIMPLE tests with non absolute
paths, example:

 $ echo "#!/bin/bash" > /tmp/test.sh
 $ echo "exit 0" >> /tmp/test.sh
 $ chmod +x /tmp/test.sh
 $ cd /tmp
 $ avocado run test.sh

The previous work on the simple test runner, adding the filename
property, actually fixes that bug, so let's add a functional test to
avoid an regression.

Reference: https://trello.com/c/nbhyILO0Signed-off-by: NCleber Rosa <crosa@redhat.com>

The "replay_map" allows to substitute the test class with custom class.
The problem is, that the "methodName" defined in "test_params" might not
exist in this class.

We have two choices, either leave it up to the user, to substitute with
a compatible class (eg. by creating the "methodName" in "__init__"
before calling the super.__init__), or we can require them to always use
given "methodName". This patch uses the second approach as it seems less
magical to me.
Signed-off-by: NLukáš Doktor <ldoktor@redhat.com>

I noticed during a run of the self test suite that system information
was being collected (`rpm -qa` and the like can usually be seen using
quite a few resources).

It looks like this little typo was causing that.
Signed-off-by: NCleber Rosa <crosa@redhat.com>

Because the current implementation of avocado.main() creates a job
and runs "sys.argv[0]" to implement standalone mode, it ends up
running itself over and over.

This simple proposed fix, prevents avocado.main() from running
itself again if called from itself. Since they are on different
processes, the mechanism chosen to do this is to set an environment
variable, that will be seen by the next process.

Also, by exiting from main() with an error code, the test first
level test will fail. This will let the user know that the chosen
approach (SIMPLE tests written in Python and calling main()) are
not worthy of a PASS.

The functional tests make use of Python's standard library utilities
(subprocess module) directly for running Avocado because of current
issues with Avocado's own process utility module.

This adresses issue #961.
Signed-off-by: NCleber Rosa <crosa@redhat.com>

Instead of assuming we have administrative privileges to
run service manager related commands, use sudo in commands
that do require root. We now assume you're running the
test as a user that has sudo configuration such that no
password will be prompted to execute that command [1].

A place where we need such a provision is when running on
Amazon AWS instances (we can't run as root, but the base
user of the image has the appropriate sudo configuration).

[1] Running as root means that sudo won't be necessary,
but figuring that out is responsibility of the process
APIs, that were modified to take a sudo parameter and
figure things out based on that.
Signed-off-by: NLucas Meneghel Rodrigues <lookkas@gmail.com>

Some commands do require admin privileges to run, but we
can't use the 'root' user in some testing scenarios [1].

So let's add a 'sudo' parameter to process.run(), process.system()
and process.system_output() interfaces, that will append
a sudo command to the original command string, causing it
to be executed under sudo. The basic assumption here is
that the user executing the tests was added to the sudoers
file in such a way that it can run administrative commands
without a password.

[1] One specific environment where we can't use the root
user is when avocado runs on an AWS (Amazon Web Services)
instance.
Signed-off-by: NLucas Meneghel Rodrigues <lookkas@gmail.com>

Using an unknown option in a given subcommand leads argparse to
show the main avocado usage instructions.

This patch makes the argparse show the subcommand help instead.

Also, argparse error exit code now matches error_codes.AVOCADO_FAIL,
which makes more sense than error_codes.AVOCADO_JOB_FAIL.

Reference:https://trello.com/c/o0TpH9aeSigned-off-by: NAmador Pahim <apahim@redhat.com>

Turns out I thought out a1afd91b poorly, I needed to use
a more encompassing try/except block, and replicate the
block in the two tests that do psutil shenanigans.
Signed-off-by: NLucas Meneghel Rodrigues <lookkas@gmail.com>

Currently replay is using status.mapping as the list of valid status
for the --replay-test-status option. But status.mapping has some internal
status that users should not have to deal with.

This patch creates a user facing status which is a subset with only
the options relevant for the users and makes the job replay feature to take
advantage of it.
Signed-off-by: NAmador Pahim <apahim@redhat.com>

This patch is merging the concepts of TEST_NA and SKIP, making
the consequent status more relevant for the users.
Signed-off-by: NAmador Pahim <apahim@redhat.com>

In InterruptTest.test_well_behaved, sometimes it
takes a bit extra time for avocado to end, causing
a psutil.NoSuchProcess exception by the time the test
tries to access .cmdline() in the Process() object.

That condition means the process is already out of
the process list, which means the test passed. Let's
handle the NoSuchProcess exception accordingly.
Signed-off-by: NLucas Meneghel Rodrigues <lookkas@gmail.com>

Basic functional tests for the job replay feature.
Signed-off-by: NAmador Pahim <apahim@redhat.com>

As per report on issue #952, Avocado currently finds multiple tests
when multiple test names exist in a given Python class. When the
test code itself is loaded and the test class instantiated, only
one test method will remain (others will have been overloaded by
that one).

So, to mimic this in the static, AST based, test discovery, let's
make the test methods unique and try to keep the order in which the
AST parser found them.
Signed-off-by: NCleber Rosa <crosa@redhat.com>

There are a few leftover references to the old Plugin classes in
the functional tests. Let's remove them.
Signed-off-by: NCleber Rosa <crosa@redhat.com>

If a test has local imports (a fairly common use case),
we have to append the base test dir to sys.path. Otherwise,
we end up having a horrible traceback as a result:

Exception loading test
Traceback (most recent call last):
  File "/home/lmr/Code/avocado.lmr/avocado/core/runner.py", line 200, in _run_test
    instance = loader.load_test(test_factory)
  File "/home/lmr/Code/avocado.lmr/avocado/core/loader.py", line 262, in load_test
    test_module = imp.load_module(module_name, f, p, d)
  File "scylla_longevity.py", line 5, in <module>
    from sdcm.tester import ScyllaClusterTester
ImportError: No module named sdcm.tester

Which is clearly not wanted. We had this problem before
and now we have a regression. It's time for a functional
test to make sure we don't regress again.
Signed-off-by: NLucas Meneghel Rodrigues <lookkas@gmail.com>
Signed-off-by: NCleber Rosa <crosa@redhat.com>

It was noticed that the Python standard library provides os.ttyname(),
and, while it only returns the Pseudo Terminal file path when given
the slave file descriptor, it looks to be sufficient for most, if not
all, use cases.

This reverts commit 003bfaed.

CC: Lukáš Doktor <ldoktor@redhat.com>
Signed-off-by: NCleber Rosa <crosa@redhat.com>

Based on previous experiences (GDB terminal redirection attempts),
it's common for applications that need to allocate a pseudo to have
the name of the allocated file. This is, unfortunately, something that
the Python standard library doesn't give for free.

This simple utility module wraps and extends the Python standard
library `pty` module, providing a `openpty` function that, besides
allocating the pseudo terminal, also returns its name.
Signed-off-by: NCleber Rosa <crosa@redhat.com>

Including the old base plugin class definition.
Signed-off-by: NCleber Rosa <crosa@redhat.com>

Signed-off-by: NCleber Rosa <crosa@redhat.com>

Signed-off-by: NCleber Rosa <crosa@redhat.com>

Signed-off-by: NCleber Rosa <crosa@redhat.com>

Signed-off-by: NCleber Rosa <crosa@redhat.com>

Yes, too many plugin mentions, in this commit title, but that's what
it really is, so please bear with it.

Now that the plugin types have a finer granularity, let's show each
plugin type (based on the namespace) as a separate section.
Signed-off-by: NLucas Meneghel Rodrigues <lmr@scylladb.com>
Signed-off-by: NCleber Rosa <crosa@redhat.com>

Signed-off-by: NCleber Rosa <crosa@redhat.com>

Signed-off-by: NCleber Rosa <crosa@redhat.com>

Signed-off-by: NCleber Rosa <crosa@redhat.com>

Signed-off-by: NCleber Rosa <crosa@redhat.com>

Signed-off-by: NCleber Rosa <crosa@redhat.com>

Signed-off-by: NCleber Rosa <crosa@redhat.com>

Signed-off-by: NCleber Rosa <crosa@redhat.com>

The Avocado command line application has "subcommands". So far,
to execute these commands, a special method, named "dispatch"
was set on the command line parser result object (aka Namespace)
pointing to the `run` method of the same class.

In truthfullness, the command line parser doesn't have too much
to do with these methods. Also, the `set_defaults` method of the
command line argument parser is not exactly intended for that use.
So, let's ask the subcommands subparser to formally save the chosen
subcommand, at well, `subcommand`, and nothing more.

Since the application knows about the command plugins (the ones that
inherit from CLICmdBase), and their interface now *require* a method
called `run`, let's use the extension entry-point name to locate the
underlying object and its `run` method.
Signed-off-by: NCleber Rosa <crosa@redhat.com>

At this point, no plugin support exists in Avocado. This is in
preparation for the new plugin code to be cleanly introduced.

The plugins that play a role in the plugin architecture have
been removed, since they wouldn't be functional under the new
plugin management code. The ones that are add extra functionality
to Avocado have been kept, and will be ported to the new
architecture.

Also, most of the functional tests have been temporarily disabled.
The reason is that most of them run avocado, which depends on the
run command, which in turn, depends on the plugin archicture code.
Signed-off-by: NCleber Rosa <crosa@redhat.com>

This shows a more succint version of the help message (the so called
"usage" information). It's a certainly a matter of taste and change
of behavior, so this is marked as an RFC.

It also avoids a hackish injection of the print_help method as the
default dispatch method.
Signed-off-by: NCleber Rosa <crosa@redhat.com>

This fixes the recently introduced commit 5f7bafd0, running the skip
function instead of passing the function itself to `unittest.skipIf`.
Signed-off-by: NCleber Rosa <crosa@redhat.com>

The `test_gendata` test of `OutputTest` runs the example test
`gendata.py`, but the `gendata.py` itself has a conditional in one of
the tests that make use of the PIL (aka Pillow package) library and
does nothing if it's missing.

PIL (Pillow) is highly dependent on C compilers and libraries to be
able to build itself. This makes systems it harder to run the full
Avocado self tests on more diverse (usually minimalist) systems.

So, let's make that functional test optional, if the libraries are not
present.
Signed-off-by: NCleber Rosa <crosa@redhat.com>

There are a couple of functional unittests that assume that a given
output was produced by the regular UI, that is, the Human output
format.

That is fine, since they're functional tests. But, some situations
lead to test errors, instead of pointing failures. Let's turn those
into explicit checks and fail properly instead of producing errors.

Note: These errors were visible when the human output was disabled, in
the plugin overhaul work, but the fixes still make sense independently
of that work.
Signed-off-by: NCleber Rosa <crosa@redhat.com>

While going through the dependency list (requirements*.txt files) and
performing our self tests out of virtual environments, I noticed that
some tests are run outside the virtual environments.

The reason is that, even though the virtual environment is activated
for the test session (and say, `which python` gives `/venv/bin/python`),
we have hard coded `/usr/bin/python` in most places.

According to the some discussions on the virtualenv project itself[1],
a quick solution is to revert to the also common `/usr/bin/env python`
way of pointing to the Python interpreter.

[1] - https://github.com/pypa/virtualenv/issues/124Signed-off-by: NCleber Rosa <crosa@redhat.com>

We used to support subtests filter to load only some of the avocado
tests, but the functionality is broken. This patch brings this
functionality back. It uses "re.search" so people can use partial
matches. Additionally it includes selftests to prevent future breakages.
Signed-off-by: NLukáš Doktor <ldoktor@redhat.com>

This patch adds unittest for loading simple tests with arguments.
Signed-off-by: NLukáš Doktor <ldoktor@redhat.com>