The xunit plugin claims to be compatible with jenkins-ci junit-4.xsd,
but it does not pass on the validation dur to the wrong key name 'skip'.

This patch changes the key name to 'skipped' and adds a unittest to validate
the xunit output against the junit-4.xsd schema.

The junit-4.xsd file is coming from:
https://svn.jenkins-ci.org/trunk/hudson/dtkit/dtkit-format/dtkit-junit-model/src/main/resources/com/thalesgroup/dtkit/junit/model/xsd/junit-4.xsd

Reference: https://trello.com/c/ZdEb0TblSigned-off-by: NAmador Pahim <apahim@redhat.com>

The leading "." makes files invisible on unix systems, which might be
confusing to users. Let's replace with "_".
Signed-off-by: NLukáš Doktor <ldoktor@redhat.com>

This adds selftests regarding the long filenames fixes.
Signed-off-by: NLukáš Doktor <ldoktor@redhat.com>

When avocado fails early in the process (for example in
_setup_environment_variables) it crashes, because the test status is
None. Other example could be user-defined status, which is not inside
user-facing-statuses.

This patch modifies the "status" to "ERROR" when test reports
unsupported status and sets related values to provide details regarding
this failure.
Signed-off-by: NLukáš Doktor <ldoktor@redhat.com>

When a job is timed out during a test execution, we fail the test
and put status ERROR in the test. The job then exits with the rc
AVOCADO_TESTS_FAIL. This patch fixes this, making the test status
INTERRUPTED and the job to exit with AVOCADO_JOB_INTERRUPTED.

Also, when a job is timed out before a test, the test is skipped
and the job exits with rc AVOCADO_ALL_OK. For that case, this patch
makes the job to exit with AVOCADO_JOB_INTERRUPTED instead, keeping
the test status as SKIP.

Given this change, now we have the following combinations of test status
and job return code:

Case1:
    - Test1: PASS
    - Test2: SKIP (TestTimeoutSkip)

    Job RC: AVOCADO_JOB_INTERRUPTED

Case2:
    - Test1: PASS
    - Test2: INTERRUPTED (TestTimeoutInterrupted)
    - Test3: SKIP (TestTimeoutSkip)

    Job RC: AVOCADO_JOB_INTERRUPTED

Case3:
    - Test1: PASS
    - Test2: FAIL
    - Test3: INTERRUPTED (TestTimeoutInterrupted)
    - Test4: SKIP (TestTimeoutSkip)

    Job RC: AVOCADO_JOB_INTERRUPTED
Signed-off-by: NAmador Pahim <apahim@redhat.com>

People tend to mix unicode, strings and encoded strings. This patch adds
"avocado.utils.astring.string_safe_encode" function, which tries to
allow all of the listed inputs and it assumes "utf-8" is in use for
encoded strings.

This function is then used in the tabular_output to avoid
"UnicodeDecodeErrors" for all listed input types.
Signed-off-by: NLukáš Doktor <ldoktor@redhat.com>

In case no tests are found we advice users to run `avocado list -V`
followed by list of urls, which does not really work with no-urls given.
This patch adds second message in case the user does not provide urls
(eg. is using the `--vt-config` option).
Signed-off-by: NLukáš Doktor <ldoktor@redhat.com>

The --show stream:level is actually broken as the "name" variable is not
tuple, but only the stream name. The variable was renamed to
"stream_level" instead.
Signed-off-by: NLukáš Doktor <ldoktor@redhat.com>

Xunit/XML CDATA/attributes must not contain non-printable characters.
This patch escapes such characters using hex value.
Signed-off-by: NLukáš Doktor <ldoktor@redhat.com>

A test base on the external runner feature depends on both the
external runner and the test URL. These two are always combined to
generate the complete command that will be executed by Avocado.

But, the Avocado command line application does not require URLs to be
given on the command line, because these may come from other sources
on loaders other than the basic FileLoader.

So, let's also explicitly require a URL to be given to to the External
Runner loader to return a test factory (class and parameters).
Signed-off-by: NCleber Rosa <crosa@redhat.com>

And deprecate the --multiplex-files option, which was actually
the complete name of the option, but was not being used formally
by our documentation.
Signed-off-by: NCleber Rosa <crosa@redhat.com>

This gives the streams behaviors that were discussed and defined a
formal set of tests.
Signed-off-by: NCleber Rosa <crosa@redhat.com>
Signed-off-by: NLukáš Doktor <ldoktor@redhat.com>

While working on the avocado logging system we thought it'd be
consistent and better to log certain streams to sys.stderr rather than
sys.stdout. It turned out to be a bit confusing and unexpected, so after
a discussion this commit adjusts the streams to:

 * avocado.app - loglevel <= info goes to stdout
               - loglevel > info goes to stderr
 * other streams - go always into stdout
 * sys.stdout - goes to sys.stdout when enabled
 * sys.stderr - goes to sys.stderr when enabled
              - goes to sys.stdout when enabled as test output
Signed-off-by: NLukáš Doktor <ldoktor@redhat.com>

Given the last CI jobs, RunnerSimpleTest:test_kill_stopped_sleep() has
been failing often. The obvious reason, given the nature of the test
and the heavily shared and loaded computing resources they run on, is
the lack of resources.

One example of such as failure:

   ======================================================================
   FAIL: test_kill_stopped_sleep (selftests.functional.test_basic.RunnerSimpleTest)
   ----------------------------------------------------------------------
   Traceback (most recent call last):
     File "/home/travis/build/avocado-framework/avocado/selftests/functional/test_basic.py", line 549, in test_kill_stopped_sleep
       % proc.get_output())
   AssertionError: Avocado process still alive 1s after job-timeout:
   JOB ID     : 5c65a06b4ab15e69d6c4a1fc762368e0e99cbc1f
   JOB LOG    : /tmp/avocado_selftests.functional.test_basicHIyhV4/job-2016-03-09T15.08-5c65a06/job.log
   TESTS      : 1
    (1/1) /bin/sleep 60:
   ctrl+z pressed, stopping test (5046)
   ERROR

So, let's give 4 extra seconds for the process to be
terminated. Hopefully failures in the timeout handling will still be
revealed given the relatively small buffer.
Signed-off-by: NCleber Rosa <crosa@redhat.com>

Record the args and reload loader options on job replay.

Reference: https://trello.com/c/N3UCnR05Signed-off-by: NAmador Pahim <apahim@redhat.com>

The configurations related to the "data_dir" are currently read at
module load time. This means that changes in the configuration between
module load time and changes to the settings (such as parsing an
additional config file) is never seen.

Since the avocado command line application parses extra configuration
files (given with `--config`) after the module is loaded, these extra
configuration files are never applied to the data_dir configuration.

There's one more issue with regards to the the settings usage: because
once references to the `settings` instance singleton are grabbed, they
will always point to the same settings objects. The data_dir unittests
exercise changes to the settings objects by replacing that
instance. So, let's refer to the settings using the full location
(module.attribute), which will always give back the "current" (be it
original or replaced) settings instance.
Signed-off-by: NCleber Rosa <crosa@redhat.com>

The current implementation of _get_bogus_settings() solves a partial need
of the setUp() method. Let's combine both into a single (private) utility
method (_get_temporary_dirs_mapping_and_config()) that can be re-used.

The major goal is to re-use it when testing how data_dir reacts to changes
in the settings at run time.
Signed-off-by: NCleber Rosa <crosa@redhat.com>

This check "avocado -v" produces correct output and returns 0.
Signed-off-by: NLukáš Doktor <ldoktor@redhat.com>

The `--silent` argument is valid for all commands, let's add it to
the core avocado args. For backward compatibility keep supporting it in
"run" subcommand too.
Signed-off-by: NLukáš Doktor <ldoktor@redhat.com>

The "View" concept was developed to abstract the messages and events.
The problem is it was only abused to pass the status to other places, to
colorize the messages and to allow creating paginated view.

There already is a nice class hooked to all important events, the
"TestResult".

For (not only human readable) messages this commit uses the standard
python logging as it's pretty well known, widely used and very scalable.
The colored output is handled by already existing
"ProgressStreamHandler", which maps: DEBUG,INFO,WARNING and >=ERROR
messages to previously existing: minor, message, warning and error event
types.

The paginator was unified and is initialized during logging reconfigure.
During reconfigure all previously logged messages are re-logged into the
output so one does not lose those messages. Another great difference is
that the Paginator is cleaned at exit by avocado and does not require
complex handling to avoid broken console. To use paginator one just
enables it in "args" and writes to any available stream/stdout/stderr
Signed-off-by: NLukáš Doktor <ldoktor@redhat.com>

Both "stream" and "args" are present in the job. Let's only pass
the "job" object and let the Result class to take the needed objects.

Additionally allow overriding the output for json/xunit/html in order
to allow users to specify different value than the one from args.
Signed-off-by: NLukáš Doktor <ldoktor@redhat.com>

Find for files in multiple locations, caching it when successfully fetched.
Available as fetch_asset() method in avocado.Test().

Reference: https://trello.com/c/KTeMIx0uSigned-off-by: NAmador Pahim <apahim@redhat.com>

All *SkipTests are now inherited from "SkipTest" baseclass. This
baseclass contains improved __init__ which removes any unsupported
parameters from the Test.__init__ super call.

This is because those classes are used to substitute other avocado test
classes, for example "avocado-vt" tests which contain "vt_params"
argument unsupported by Test.__init__.
Signed-off-by: NLukáš Doktor <ldoktor@redhat.com>

The tree/multiplex modules have clear conditionals when the system
is not multiplex capable, namely because of the lack of YAML libraries.

Let's reflect that on their tests.
Signed-off-by: NCleber Rosa <crosa@redhat.com>

This commit changes the idiom on (hopefully all) except statements,
from:
   except foo, details:

to:
   except foo as details:

Which is compatible with both Python 2 and 3.
Signed-off-by: NCleber Rosa <crosa@redhat.com>

This change introduces symbolic names for various filesystem
permission modes, instead of the literal octals with which
Python 3 shows no compassion.
Signed-off-by: NCleber Rosa <crosa@redhat.com>

Those two are interconnected, let's keep them in one file instead of
local imports.
Signed-off-by: NLukáš Doktor <ldoktor@redhat.com>

This patch changes the way logging is initialized in avocado app:

1. stdout/stderr/logging is replaced with in-memory handlers
2. when arg parsing is done, logging is reconfigured accordingly to
   setting and re-logs stored messages (when streams are enabled)
3. when job starts and "test" stream is enabled, "" and "avocado.test"
   streams are enabled
4. when job finishes and "test" stream enabled the additional outputs,
   the additional outputs are disabled again.

The 3 and 4 is necessarily as the "" logger contains all logs including
avocado initialization (stevedore, PIL, ...). This way the "test" stream
still outputs everything, but only during the job-execution and not
during avocado initialization and cleanup. (one can use "early" log to
see those).

Last but not least it keeps "avocado.app" output in stdout (and stderr),
but it moves all other outputs to stderr instead. This is IMO better
handling as one can distinguish between debug and "additional" output.
Signed-off-by: NLukáš Doktor <ldoktor@redhat.com>

The function "add_console_handler" is quite useful. This patch extends
it's capabilities and renames it to fit the new purpose.

Because the new default logging level for "add_log_handler" is INFO,
we add the "avocado.test" and root logger to level DEBUG.

Then finally a few adjustments to the tests that rely on specific
log format and level:

1) selftests/functional/test_multiplex.py: adapt to changes in
test logger That means that extra prefixes ('avocado.test') are
printed out on each line when avocado is run with '--show-job-log'.
While at it, be more precise and check for the specific params at the
specific run with the given variant, instead of looking for a given
param in the output of the three tests variants.

2) selftests/functional/test_standalone.py: adapt to the change of
log prefix ('avocado.test') and extra line because of DEBUG level.
Signed-off-by: NLukáš Doktor <ldoktor@redhat.com>
Signed-off-by: NCleber Rosa <crosa@redhat.com>

The "start_file_logging" is used when job results are created to add
job file-handlers to logging.
Signed-off-by: NLukáš Doktor <ldoktor@redhat.com>

This implements a "make selfcheck" that developers can use to run
the Avocado selftests in itself. It also keeps the old "make check"
(for a while) so that we can compare to the old approach if any
regressions occur in the near future.
Signed-off-by: NCleber Rosa <crosa@redhat.com>

It's more common in our code (and pretty much around all Python
tests) to use the TestCase utility methods such as `assertEqual`
instead of doing `assert foo == bar`. Let's remove the few
occurrences of the `assert` idiom for cohesiveness.
Signed-off-by: NCleber Rosa <crosa@redhat.com>

One of the unittests, namely `test_parse_lsmod_is_empty`, is implemented
as a static method, which should be fine because it doesn't use any class
attribute. But, there are a couple of problems.

First, it looks like the Python unittest library, while able to "discover"
such tests (implemented as static methods):

   $ python test_utils_linux_modules.py -v
   test_parse_lsmod (__main__.TestLsmod) ... ok
   test_parse_lsmod_is_empty (__main__.TestLsmod) ... ok
   test_parse_lsmod_no_submodules (__main__.TestLsmod) ... ok
   test_parse_lsmod_single_submodules (__main__.TestLsmod) ... ok

   ----------------------------------------------------------------------
   Ran 4 tests in 0.001s

   OK

It can not deal with it if its name is given explicitly. Example:

   $ python -m unittest test_utils_linux_modules.TestLsmod.test_parse_lsmod_is_empty
   Traceback (most recent call last):
     File "/usr/lib64/python2.7/runpy.py", line 162, in _run_module_as_main
       "__main__", fname, loader, pkg_name)
     File "/usr/lib64/python2.7/runpy.py", line 72, in _run_code
       exec code in run_globals
     File "/usr/lib64/python2.7/unittest/__main__.py", line 12, in <module>
       main(module=None)
     File "/usr/lib64/python2.7/unittest/main.py", line 94, in __init__
       self.parseArgs(argv)
     File "/usr/lib64/python2.7/unittest/main.py", line 149, in parseArgs
       self.createTests()
     File "/usr/lib64/python2.7/unittest/main.py", line 158, in createTests
       self.module)
     File "/usr/lib64/python2.7/unittest/loader.py", line 130, in loadTestsFromNames
       suites = [self.loadTestsFromName(name, module) for name in names]
     File "/usr/lib64/python2.7/unittest/loader.py", line 122, in loadTestsFromName
       (obj, test))
   TypeError: calling <function test_parse_lsmod_is_empty at 0x7fb3f58129b0> returned None, not a test

While a bug should probably be filed on upstream CPython, we can't neither
should wait on a possible fix.

Second, it's much more common, and arguably more Python, to have non-static
methods as tests, so that unittest.TestCase methods can be called, such as
"assertEquals" instead of "assert foo == bar".

So, let's change this method to be a non-static one.
Signed-off-by: NCleber Rosa <crosa@redhat.com>

Signed-off-by: NCleber Rosa <crosa@redhat.com>

Signed-off-by: NCleber Rosa <crosa@redhat.com>

The idea is to hold utilities that are used to implement the safe,
AKA ast-based, test loader.
Signed-off-by: NCleber Rosa <crosa@redhat.com>

Options --multiplex-files and/or urls are incompatible with
--replay-test-status, since the resulting variants will differ from the
original job.

This patch adds a test to fail avocado when those incompatible options are
used togheter.

Also, some minor fixes in replay messages; and in return code when --remote
is used.

Reference: https://trello.com/c/OJpRv36iSigned-off-by: NAmador Pahim <apahim@redhat.com>

The functionality to run tests on remote machines and on VMs are
implemented using a mixture of TestRunner and TestResult classes.

In all fairness, it looks like the TestResult classes were abused
because non-standard attributes were used as a foundation for that
functionality. Now that `setup` and `tear_down` have been added
to the TestRunner interface, let's move those custom features from
the TestResult classes to the TestRunner ones.
Signed-off-by: NCleber Rosa <crosa@redhat.com>
Signed-off-by: NLukáš Doktor <ldoktor@redhat.com>

This unittest patches find_command to return /usr/bin/sudo and then runs
it and expects some results. This works only with correctly set sudo,
but it does not work without sudo (as the binary is not found) or with
incorrectly set sudo. Let's use "/bin/true" instead.
Signed-off-by: NLukáš Doktor <ldoktor@redhat.com>

When we abort the test and the test ignores SIGTERM, avocado hangs for
infinity. Let's wait a while and send SIGKILL.
Signed-off-by: NLukáš Doktor <ldoktor@redhat.com>