This patch changes the way logging is initialized in avocado app:

1. stdout/stderr/logging is replaced with in-memory handlers
2. when arg parsing is done, logging is reconfigured accordingly to
   setting and re-logs stored messages (when streams are enabled)
3. when job starts and "test" stream is enabled, "" and "avocado.test"
   streams are enabled
4. when job finishes and "test" stream enabled the additional outputs,
   the additional outputs are disabled again.

The 3 and 4 is necessarily as the "" logger contains all logs including
avocado initialization (stevedore, PIL, ...). This way the "test" stream
still outputs everything, but only during the job-execution and not
during avocado initialization and cleanup. (one can use "early" log to
see those).

Last but not least it keeps "avocado.app" output in stdout (and stderr),
but it moves all other outputs to stderr instead. This is IMO better
handling as one can distinguish between debug and "additional" output.
Signed-off-by: NLukáš Doktor <ldoktor@redhat.com>

The function "add_console_handler" is quite useful. This patch extends
it's capabilities and renames it to fit the new purpose.

Because the new default logging level for "add_log_handler" is INFO,
we add the "avocado.test" and root logger to level DEBUG.

Then finally a few adjustments to the tests that rely on specific
log format and level:

1) selftests/functional/test_multiplex.py: adapt to changes in
test logger That means that extra prefixes ('avocado.test') are
printed out on each line when avocado is run with '--show-job-log'.
While at it, be more precise and check for the specific params at the
specific run with the given variant, instead of looking for a given
param in the output of the three tests variants.

2) selftests/functional/test_standalone.py: adapt to the change of
log prefix ('avocado.test') and extra line because of DEBUG level.
Signed-off-by: NLukáš Doktor <ldoktor@redhat.com>
Signed-off-by: NCleber Rosa <crosa@redhat.com>

The "start_file_logging" is used when job results are created to add
job file-handlers to logging.
Signed-off-by: NLukáš Doktor <ldoktor@redhat.com>

Options --multiplex-files and/or urls are incompatible with
--replay-test-status, since the resulting variants will differ from the
original job.

This patch adds a test to fail avocado when those incompatible options are
used togheter.

Also, some minor fixes in replay messages; and in return code when --remote
is used.

Reference: https://trello.com/c/OJpRv36iSigned-off-by: NAmador Pahim <apahim@redhat.com>

The functionality to run tests on remote machines and on VMs are
implemented using a mixture of TestRunner and TestResult classes.

In all fairness, it looks like the TestResult classes were abused
because non-standard attributes were used as a foundation for that
functionality. Now that `setup` and `tear_down` have been added
to the TestRunner interface, let's move those custom features from
the TestResult classes to the TestRunner ones.
Signed-off-by: NCleber Rosa <crosa@redhat.com>
Signed-off-by: NLukáš Doktor <ldoktor@redhat.com>

This unittest patches find_command to return /usr/bin/sudo and then runs
it and expects some results. This works only with correctly set sudo,
but it does not work without sudo (as the binary is not found) or with
incorrectly set sudo. Let's use "/bin/true" instead.
Signed-off-by: NLukáš Doktor <ldoktor@redhat.com>

When we abort the test and the test ignores SIGTERM, avocado hangs for
infinity. Let's wait a while and send SIGKILL.
Signed-off-by: NLukáš Doktor <ldoktor@redhat.com>

This function is useful, when one executes shell=True commands where
he sets values and then executes command. He can use this function
to get the actual binary path instead of the first variable=value
assignment.
Signed-off-by: NLukáš Doktor <ldoktor@redhat.com>

Avocado defines avocado.core.result.TestResult, so that alternative
test result classes can inherit from it. When a user writes a new
test result class, he's left wondering how to activate that.

First, for consistency, Avocado should define proper interfaces
for "test result plugins". This is on sight, and will ultimately
allow users to register new test result implementations simply by
registering the setuptools entry points, just as it's currently
done (on versions >= 0.31.0) for command line related plugins.

Until then, let's at least improve the current state of things.
These changes believe that the following improvements are given here:

* avocado.core.job.Job._set_output_plugins now won't look at every
  possible key in the application arguments, and won't silence
  possible errors while trying to instantiate test result classes.
* test result writers can now use a documented method for registering
  their test result classes.

The same approach was not chosen to be done with the test runner
classes because there can only by one test runner per Avocado job.
Signed-off-by: NCleber Rosa <crosa@redhat.com>

There is an issue when running SIMPLE tests with non absolute
paths, example:

 $ echo "#!/bin/bash" > /tmp/test.sh
 $ echo "exit 0" >> /tmp/test.sh
 $ chmod +x /tmp/test.sh
 $ cd /tmp
 $ avocado run test.sh

The previous work on the simple test runner, adding the filename
property, actually fixes that bug, so let's add a functional test to
avoid an regression.

Reference: https://trello.com/c/nbhyILO0Signed-off-by: NCleber Rosa <crosa@redhat.com>

The "replay_map" allows to substitute the test class with custom class.
The problem is, that the "methodName" defined in "test_params" might not
exist in this class.

We have two choices, either leave it up to the user, to substitute with
a compatible class (eg. by creating the "methodName" in "__init__"
before calling the super.__init__), or we can require them to always use
given "methodName". This patch uses the second approach as it seems less
magical to me.
Signed-off-by: NLukáš Doktor <ldoktor@redhat.com>

I noticed during a run of the self test suite that system information
was being collected (`rpm -qa` and the like can usually be seen using
quite a few resources).

It looks like this little typo was causing that.
Signed-off-by: NCleber Rosa <crosa@redhat.com>

Because the current implementation of avocado.main() creates a job
and runs "sys.argv[0]" to implement standalone mode, it ends up
running itself over and over.

This simple proposed fix, prevents avocado.main() from running
itself again if called from itself. Since they are on different
processes, the mechanism chosen to do this is to set an environment
variable, that will be seen by the next process.

Also, by exiting from main() with an error code, the test first
level test will fail. This will let the user know that the chosen
approach (SIMPLE tests written in Python and calling main()) are
not worthy of a PASS.

The functional tests make use of Python's standard library utilities
(subprocess module) directly for running Avocado because of current
issues with Avocado's own process utility module.

This adresses issue #961.
Signed-off-by: NCleber Rosa <crosa@redhat.com>

Instead of assuming we have administrative privileges to
run service manager related commands, use sudo in commands
that do require root. We now assume you're running the
test as a user that has sudo configuration such that no
password will be prompted to execute that command [1].

A place where we need such a provision is when running on
Amazon AWS instances (we can't run as root, but the base
user of the image has the appropriate sudo configuration).

[1] Running as root means that sudo won't be necessary,
but figuring that out is responsibility of the process
APIs, that were modified to take a sudo parameter and
figure things out based on that.
Signed-off-by: NLucas Meneghel Rodrigues <lookkas@gmail.com>

Some commands do require admin privileges to run, but we
can't use the 'root' user in some testing scenarios [1].

So let's add a 'sudo' parameter to process.run(), process.system()
and process.system_output() interfaces, that will append
a sudo command to the original command string, causing it
to be executed under sudo. The basic assumption here is
that the user executing the tests was added to the sudoers
file in such a way that it can run administrative commands
without a password.

[1] One specific environment where we can't use the root
user is when avocado runs on an AWS (Amazon Web Services)
instance.
Signed-off-by: NLucas Meneghel Rodrigues <lookkas@gmail.com>

Using an unknown option in a given subcommand leads argparse to
show the main avocado usage instructions.

This patch makes the argparse show the subcommand help instead.

Also, argparse error exit code now matches error_codes.AVOCADO_FAIL,
which makes more sense than error_codes.AVOCADO_JOB_FAIL.

Reference:https://trello.com/c/o0TpH9aeSigned-off-by: NAmador Pahim <apahim@redhat.com>

Turns out I thought out a1afd91b poorly, I needed to use
a more encompassing try/except block, and replicate the
block in the two tests that do psutil shenanigans.
Signed-off-by: NLucas Meneghel Rodrigues <lookkas@gmail.com>

Currently replay is using status.mapping as the list of valid status
for the --replay-test-status option. But status.mapping has some internal
status that users should not have to deal with.

This patch creates a user facing status which is a subset with only
the options relevant for the users and makes the job replay feature to take
advantage of it.
Signed-off-by: NAmador Pahim <apahim@redhat.com>

This patch is merging the concepts of TEST_NA and SKIP, making
the consequent status more relevant for the users.
Signed-off-by: NAmador Pahim <apahim@redhat.com>

In InterruptTest.test_well_behaved, sometimes it
takes a bit extra time for avocado to end, causing
a psutil.NoSuchProcess exception by the time the test
tries to access .cmdline() in the Process() object.

That condition means the process is already out of
the process list, which means the test passed. Let's
handle the NoSuchProcess exception accordingly.
Signed-off-by: NLucas Meneghel Rodrigues <lookkas@gmail.com>

Basic functional tests for the job replay feature.
Signed-off-by: NAmador Pahim <apahim@redhat.com>

As per report on issue #952, Avocado currently finds multiple tests
when multiple test names exist in a given Python class. When the
test code itself is loaded and the test class instantiated, only
one test method will remain (others will have been overloaded by
that one).

So, to mimic this in the static, AST based, test discovery, let's
make the test methods unique and try to keep the order in which the
AST parser found them.
Signed-off-by: NCleber Rosa <crosa@redhat.com>

There are a few leftover references to the old Plugin classes in
the functional tests. Let's remove them.
Signed-off-by: NCleber Rosa <crosa@redhat.com>

If a test has local imports (a fairly common use case),
we have to append the base test dir to sys.path. Otherwise,
we end up having a horrible traceback as a result:

Exception loading test
Traceback (most recent call last):
  File "/home/lmr/Code/avocado.lmr/avocado/core/runner.py", line 200, in _run_test
    instance = loader.load_test(test_factory)
  File "/home/lmr/Code/avocado.lmr/avocado/core/loader.py", line 262, in load_test
    test_module = imp.load_module(module_name, f, p, d)
  File "scylla_longevity.py", line 5, in <module>
    from sdcm.tester import ScyllaClusterTester
ImportError: No module named sdcm.tester

Which is clearly not wanted. We had this problem before
and now we have a regression. It's time for a functional
test to make sure we don't regress again.
Signed-off-by: NLucas Meneghel Rodrigues <lookkas@gmail.com>
Signed-off-by: NCleber Rosa <crosa@redhat.com>

It was noticed that the Python standard library provides os.ttyname(),
and, while it only returns the Pseudo Terminal file path when given
the slave file descriptor, it looks to be sufficient for most, if not
all, use cases.

This reverts commit 003bfaed.

CC: Lukáš Doktor <ldoktor@redhat.com>
Signed-off-by: NCleber Rosa <crosa@redhat.com>

Based on previous experiences (GDB terminal redirection attempts),
it's common for applications that need to allocate a pseudo to have
the name of the allocated file. This is, unfortunately, something that
the Python standard library doesn't give for free.

This simple utility module wraps and extends the Python standard
library `pty` module, providing a `openpty` function that, besides
allocating the pseudo terminal, also returns its name.
Signed-off-by: NCleber Rosa <crosa@redhat.com>

Including the old base plugin class definition.
Signed-off-by: NCleber Rosa <crosa@redhat.com>

Signed-off-by: NCleber Rosa <crosa@redhat.com>

Signed-off-by: NCleber Rosa <crosa@redhat.com>

Signed-off-by: NCleber Rosa <crosa@redhat.com>

Signed-off-by: NCleber Rosa <crosa@redhat.com>

Yes, too many plugin mentions, in this commit title, but that's what
it really is, so please bear with it.

Now that the plugin types have a finer granularity, let's show each
plugin type (based on the namespace) as a separate section.
Signed-off-by: NLucas Meneghel Rodrigues <lmr@scylladb.com>
Signed-off-by: NCleber Rosa <crosa@redhat.com>

Signed-off-by: NCleber Rosa <crosa@redhat.com>

Signed-off-by: NCleber Rosa <crosa@redhat.com>

Signed-off-by: NCleber Rosa <crosa@redhat.com>

Signed-off-by: NCleber Rosa <crosa@redhat.com>

Signed-off-by: NCleber Rosa <crosa@redhat.com>

Signed-off-by: NCleber Rosa <crosa@redhat.com>

Signed-off-by: NCleber Rosa <crosa@redhat.com>

The Avocado command line application has "subcommands". So far,
to execute these commands, a special method, named "dispatch"
was set on the command line parser result object (aka Namespace)
pointing to the `run` method of the same class.

In truthfullness, the command line parser doesn't have too much
to do with these methods. Also, the `set_defaults` method of the
command line argument parser is not exactly intended for that use.
So, let's ask the subcommands subparser to formally save the chosen
subcommand, at well, `subcommand`, and nothing more.

Since the application knows about the command plugins (the ones that
inherit from CLICmdBase), and their interface now *require* a method
called `run`, let's use the extension entry-point name to locate the
underlying object and its `run` method.
Signed-off-by: NCleber Rosa <crosa@redhat.com>