The unittest standard library on Python 2.7 an later has everything
that the unittest2 backport is supposed to have.  Let's then drop all
the conditional imports of unittest2 and stick with unittest.
Signed-off-by: NCleber Rosa <crosa@redhat.com>

Recent pylint update is more pedantic about module level spacing. There
are no changes to code, only couple of extra spaces to make it happy.
Signed-off-by: NLukáš Doktor <ldoktor@redhat.com>

While working on the avocado logging system we thought it'd be
consistent and better to log certain streams to sys.stderr rather than
sys.stdout. It turned out to be a bit confusing and unexpected, so after
a discussion this commit adjusts the streams to:

 * avocado.app - loglevel <= info goes to stdout
               - loglevel > info goes to stderr
 * other streams - go always into stdout
 * sys.stdout - goes to sys.stdout when enabled
 * sys.stderr - goes to sys.stderr when enabled
              - goes to sys.stdout when enabled as test output
Signed-off-by: NLukáš Doktor <ldoktor@redhat.com>

Those two are interconnected, let's keep them in one file instead of
local imports.
Signed-off-by: NLukáš Doktor <ldoktor@redhat.com>

This patch changes the way logging is initialized in avocado app:

1. stdout/stderr/logging is replaced with in-memory handlers
2. when arg parsing is done, logging is reconfigured accordingly to
   setting and re-logs stored messages (when streams are enabled)
3. when job starts and "test" stream is enabled, "" and "avocado.test"
   streams are enabled
4. when job finishes and "test" stream enabled the additional outputs,
   the additional outputs are disabled again.

The 3 and 4 is necessarily as the "" logger contains all logs including
avocado initialization (stevedore, PIL, ...). This way the "test" stream
still outputs everything, but only during the job-execution and not
during avocado initialization and cleanup. (one can use "early" log to
see those).

Last but not least it keeps "avocado.app" output in stdout (and stderr),
but it moves all other outputs to stderr instead. This is IMO better
handling as one can distinguish between debug and "additional" output.
Signed-off-by: NLukáš Doktor <ldoktor@redhat.com>

The function "add_console_handler" is quite useful. This patch extends
it's capabilities and renames it to fit the new purpose.

Because the new default logging level for "add_log_handler" is INFO,
we add the "avocado.test" and root logger to level DEBUG.

Then finally a few adjustments to the tests that rely on specific
log format and level:

1) selftests/functional/test_multiplex.py: adapt to changes in
test logger That means that extra prefixes ('avocado.test') are
printed out on each line when avocado is run with '--show-job-log'.
While at it, be more precise and check for the specific params at the
specific run with the given variant, instead of looking for a given
param in the output of the three tests variants.

2) selftests/functional/test_standalone.py: adapt to the change of
log prefix ('avocado.test') and extra line because of DEBUG level.
Signed-off-by: NLukáš Doktor <ldoktor@redhat.com>
Signed-off-by: NCleber Rosa <crosa@redhat.com>

Since we define the meaning of each exit code, we should be using that
ourselves.

Functional tests have been coded using literal values, and there may
be a good reason for that: proving that the API stands.  But there's a
big issue: we have not declared those codes as some kind of an
external API and guarantee stability. Also the subsequent changes in
this series actually change the meaning and values of those.
Signed-off-by: NCleber Rosa <crosa@redhat.com>

The name was not correct as the test should error-out.
Signed-off-by: NLukáš Doktor <ldoktor@redhat.com>

Currently when one passes exception, which is not instance of Exception,
avocado proceeds and returns "INTERRUPTED" test instead of "ERROR". This
for example happens when old-style-class is used as exception. This
patch uses pure "except" without any argument to catch such exceptions
and then it wraps them in "exceptions.TestError" instead.

The same applies to "setUp" and "tearDown", they only use different
class as a wrapper.
Signed-off-by: NLukáš Doktor <ldoktor@redhat.com>

Even though I may be the one to blame about coming up with this
"simple import magic", I believe it was a mistake and we should get
rid of them.

There are a couple of other ways to actually do development out of
of a source tree, including running unittests that do not require
this amount of boiler plate code. Examples include just setting
the PYTHONPATH environment variable to actually run (setuptools based)
`python setup.py develop`.

The little bits of what looks like the import magic that was left, is
not really import magic. It's just that functional tests need to locate
the Avocado source tree base directory to run the test runner from it.
Signed-off-by: NCleber Rosa <crosa@redhat.com>

To be honest, our test code could still be kept in the same directories and
have the same names. But I think we can improve two things here:

1) Flatten a little bit the directory structure of selftests. Two path
components are being dropped here: "all" and "avocado". So that
"selftests/all/functional/avocado" becomes simply
"selftests/functional".

2) File names match what is, by default, recognized by unittest (the
Standard Library module) based discovery of tests. That means that
doc_build_test.py becomes test_doc_build.py. Not a big deal IMHO.
Signed-off-by: NCleber Rosa <crosa@redhat.com>

On multiple places we modify the sys.path to make avocado work from
sources. Instead of `append` we should `insert` the path as when running
from avocado sources directory, we want to use the modules from here,
rather than from installed version.
Signed-off-by: NLukáš Doktor <ldoktor@redhat.com>

Make sure avocado complains loudly when people try to call
skip() from outside setUp() - this shouldn't be allowed in
avocado tests. Let the user know of the problem and ask them
to fix their tests. Also, update the docstring for skip(),
which will reduce the chances of test writers using it in
inappropriate places.
Signed-off-by: NLucas Meneghel Rodrigues <lmr@redhat.com>

The flag -r tells to the stand alone tests to clean their
temporary results at the end of the run, so that we don't
leave traces of their execution.

Changes from v1:
 * Wrote an actual commit message, explaining the purpose
   of the commit, per ldoktor's suggestion.
Signed-off-by: NLucas Meneghel Rodrigues <lmr@redhat.com>

It is more natural [1] that output of --show-job-log goes to
stdout than stderr. Let's change that behavior and update
unittests accordingly.

[1] Natural in this sense is defined by the user's expectation -
    when setting avocado to --show-job-log, you expect that the
    job log *is* in the application's stdout.

Changes from v1:
 * Python 2.6 constructor param for StreamHandler is strm,
   that was changed in 2.7 to be stream. Let's keep compatibility
   by using positional arguments.
Signed-off-by: NLucas Meneghel Rodrigues <lmr@redhat.com>

Signed-off-by: NLucas Meneghel Rodrigues <lmr@redhat.com>

The NastyException error is back:

class NastyException(Exception):
    def __init__(self, msg):
        self.msg = msg
    def __str__(self):
        return self.msg

crashes Avocado. The problem is we are trying to re-raise the exceptions
and then read the traceback again. Even reading of the details of this
very nasty written exception causes critical failure.

What this patch does instead of handling issues wherever we produce
them, it waits till the exception bubbles to the entry-point and it
tries to read it. If it fails, we log the original traceback
(information about the failure) and then user-friendly message saying
we're unable to retrieve the original failure.

Additionally I had to modify the xunit output as in case of very nasty
inputs it's unable to reproduce the values.
Signed-off-by: NLukáš Doktor <ldoktor@redhat.com>

We declared that WARN is only a special case of PASS,
we should probably return exit code 0.
Signed-off-by: NLukáš Doktor <ldoktor@redhat.com>

Self test files used to be executable, but that changed on 143b574c.
So, it doesn't make much sense to keep the shebangs + python commands
on them.
Signed-off-by: NCleber Rosa <crosa@redhat.com>

Remove the license text from files inside docs, examples and selftest,
to keep the license text only in relevant parts of the project.

References:

* https://fedoraproject.org/wiki/Packaging:LicensingGuidelines
* https://www.gnu.org/licenses/gpl-howto.htmlSigned-off-by: NRudá Moura <rmoura@redhat.com>

This patch replaces most of the calls to sleeptest in our functional
test with passtest, in an attempt to reduce, without invalidating the
purporse, the time it takes to run them.
Signed-off-by: NCleber Rosa <crosa@redhat.com>

We have tests/ and selftests/ in avocado (the test runner).
The former includes examples of tests that make use of the
avocado API, and are also used to test avocado via CI.

This is confusing, as it doesn't make it clear to users that
tests are not part of the avocado framework. Since we have a
directory called examples with some other files (multiplexer,
plugins), move the tests to there (and keep using them in the CI).
Signed-off-by: NLucas Meneghel Rodrigues <lmr@redhat.com>

Signed-off-by: NLucas Meneghel Rodrigues <lmr@redhat.com>

In order for people to run all avocado tests at once,
move and reorganize tests under a directory called
selftests. For someone to run all self tests in
avocado, one would have to execute:

    selftests/run selftests/all

This is a shortcut for developers. .travis.yml
was also updated to run tests based on the new
directory structure.
Signed-off-by: NLucas Meneghel Rodrigues <lmr@redhat.com>