Some tests execute avocado directly, which uses "/usr/bin/env python" by
default. On distributions with py2 and py3 installed we might end-up
running those tests using python2 while running the main selftest by
python3.

With this change we need to make sure to pass the correct python binary
in UNITTEST_AVOCADO_CMD as it's not doing this by default (to allow any
avocado command).
Signed-off-by: NLukáš Doktor <ldoktor@redhat.com>

There are few issues (3 to my count) which affect a number of tests
on Python 3.  Instead of not executing any of the functional tests,
let's skip the broken ones.

This is important because it will prevent new issues from making
its way into the Avocado code.

The list of tests being skipped is being tracked on our planning board,
so that they are accounted for, and enabled when fixed.

Reference: https://trello.com/c/eFY9Vw1R/1282-python-3-functional-tests-checklistSigned-off-by: NCleber Rosa <crosa@redhat.com>

And that have been causing failures on Travis jobs, due to limited
on unreliable amount of resources present.

This is the full list of tests that won't be run anymore on Travis,
only on `make check-full`:

  selftests.functional.test_basic.RunnerOperationTest.test_early_latest_result
  selftests.functional.test_basic.RunnerSimpleTest.test_kill_stopped_sleep
  selftests.functional.test_interrupt.InterruptTest.test_well_behaved_sigint
  selftests.functional.test_interrupt.InterruptTest.test_well_behaved_sigterm
  selftests.functional.test_loader.LoaderTestFunctional.test_simple_using_main
  selftests.functional.test_loader.LoaderTestFunctional.test_sleep_a_lot
Signed-off-by: NCleber Rosa <crosa@redhat.com>

The newly W605 warning is caused by incorrect escaped chars, that are
automatically translated to '\\' but future versions should treat it as
SyntaxError.
Signed-off-by: NLukáš Doktor <ldoktor@redhat.com>

Signed-off-by: NLukáš Doktor <ldoktor@redhat.com>

The skip_location is suppose to use "skip_location" and not
"warn_location".
Signed-off-by: NLukáš Doktor <ldoktor@redhat.com>

The built-in regexp requires re.MULTILINE to work the same way as before
(allowing the match to be everywhere and not just in the first line).

As for the existing ones I believe users would benefit from allowing
MULTILINE as well, but there might be slight change in behavior.
Signed-off-by: NLukáš Doktor <ldoktor@redhat.com>

These warnings are described as:

 W0612: Used when a variable is defined but not used.
 W0622: Used when a variable or function override a built-in.

After a suggestion from Andrei Stepanov, we decided to immediately
enable these.

Rerefence: https://pagure.io/standard-test-roles/pull-request/144
Reference: https://trello.com/c/Ohh7Q6qj/1183-enable-w-in-travis-make-checkSigned-off-by: NCleber Rosa <crosa@redhat.com>

The latest jobs have been failing because of the following issues:

    ************* Module raise
    E1130: 48,30: Raise.test: bad operand type for unary -: NoneType
    ************* Module selftests.functional.test_basic
    I1101:1191,24: PluginsXunitTest.run_and_check: Module 'lxml.etree'
    has not 'XMLSchema' member, but source is unavailable. Consider
    adding this module to extension-pkg-whitelist if you want to
    perform analysis based on run-time introspection of living
    objects.
    I1101:1191,40: PluginsXunitTest.run_and_check: Module 'lxml.etree'
    has not 'parse' member, but source is unavailable. Consider adding
    this module to extension-pkg-whitelist if you want to perform
    analysis based on run-time introspection of living objects.
    I1101:1193,43: PluginsXunitTest.run_and_check: Module 'lxml.etree'
    has not 'parse' member, but source is unavailable. Consider adding
    this module to extension-pkg-whitelist if you want to perform
    analysis based on run-time introspection of living objects.
    ************* Module selftests.unit.test_xunit
    I1101: 91,24: xUnitSucceedTest.test_add_success: Module
    'lxml.etree' has not 'XMLSchema' member, but source is
    unavailable. Consider adding this module to
    extension-pkg-whitelist if you want to perform analysis based on
    run-time introspection of living objects.
    I1101: 91,40: xUnitSucceedTest.test_add_success: Module
    'lxml.etree' has not 'parse' member, but source is
    unavailable. Consider adding this module to
    extension-pkg-whitelist if you want to perform analysis based on
    run-time introspection of living objects.
    I1101: 92,43: xUnitSucceedTest.test_add_success: Module
    'lxml.etree' has not 'parse' member, but source is
    unavailable. Consider adding this module to
    extension-pkg-whitelist if you want to perform analysis based on
    run-time introspection of living objects.

We can't whitelist the `lxml.etree` module at this time (inspekt doesn't
have such an option) and the raise failure is a false positive.  Let's
ignore those issues then.
Signed-off-by: NCleber Rosa <crosa@redhat.com>

Under Python 3.4, which runs on our CI and is our lowest supported
version, json.loads() requires a string, while newer versions also
allows bytes.

    >>> json.loads(b'{}')
    Traceback (most recent call last):
      File "<stdin>", line 1, in <module>
      File "/home/cleber/.local/lib/python3.4/json/__init__.py", line 312, in loads s.__class__.__name__))
    TypeError: the JSON object must be str, not 'bytes'

Let's make that code compatible with Python 3.4.
Signed-off-by: NCleber Rosa <crosa@redhat.com>

The pystache library on Python 3 will generated an HTML file with entity
references for the single quote char, instead of the literal char.

Let's check for either one of them in the functional test
test_basic.RunnerSimpleTest.test_fs_unfriendly_run().
Signed-off-by: NCleber Rosa <crosa@redhat.com>

The config file generated and used on this test was missing a
separator between the section name and the first key/value line.

This adds a new line and makes the resulting file aesthetically
pleasing and able to be parsed.
Signed-off-by: NCleber Rosa <crosa@redhat.com>

Signed-off-by: NCleber Rosa <crosa@redhat.com>

Signed-off-by: NCleber Rosa <crosa@redhat.com>

Signed-off-by: NCleber Rosa <crosa@redhat.com>

The "url" and "test" fields are just legacy and duplicated fields
that carry the same data as the "id" field.  Let's remove those.
Signed-off-by: NCleber Rosa <crosa@redhat.com>

This brings a change in behaviour, in which the stdout/stderr of the
executed process will now be of bytes type, instead of a string type.

Two new attributes, which are implemented as properties, have been
added to the CmdResult class, `stdout_text` and `stderr_text`.  Those
are convenience methods that will return the same content that is in
`stdout` and `stderr`, reespectively, but decoded on the fly[1].

With regards to encoding, if one is not provided, the result of
`sys.getdefaultencoding()` will be used ("utf-8" for Python 3 and
"ascii" for Python 2).

Applications and/or tests using the APIs that return a CmdResult
should, to the best of my knowledge, set a default encoding themselves
so a stable behavior across Python versions.  But that if left to
users of this API.

A different tradeoff/design decision has to do with the tests modified
here.  One option is to have "text" (as in sequences of human readable
glyphs) as being of Python type "str".  On Python 2, "str" can be
compared to "bytes" because a conversion will happen on demand.  That
is, the following is fine on Python 2:

   >>> result = process.run("command")
   >>> "expected" in process.stdout

Where `expected` is of type "str" and `process.stdout` is of type
"bytes".  This is not true of Python 3, so either the types must match
or a conversion must be done explicitly.  The solutions to that are:

1) have these "text" as (of type) "bytes" in the source code itself,
   and avoid the conversion whenever possible
2) have "strings" in the source code itself, and use the conversion
   provided by `CmdResult.stdout_text` and `CmdResult.stderr_text`.

The approach chosen here is to avoid conversion if possible, that is,
use "byte" types, given the fact that the source code encoding is by
default 'ascii' and most of the "text" dealt with here can be
represented in 'ascii' too.  This is equivalent of doing:

   result = process.run("command")
   b"expected" in process.stdout
   "errors: %s" % 0 in process.stderr_text

[1] The obvious alternative, instead of decoding these on the fly
    would be to have multiple copies of the "same" data.  This assumes
    that binary data produced on the stdout/stderr will usually be
    larger than textual data.
Signed-off-by: NCleber Rosa <crosa@redhat.com>

This is similar to the xunit unittest changes in 353fed04, but now
applied to the functional tests which happen to use the same features.
Signed-off-by: NCleber Rosa <crosa@redhat.com>

This is similar to the change in 418bf92a, but now applied to the
functional tests that deal with xunit.
Signed-off-by: NCleber Rosa <crosa@redhat.com>

Avocado used to provide support for "test aliases", that is, one could
call "avocado run synctest" when the file containing the test was
named "synctest.py".  See e65ff614 for an example.

Since this "feature" has been long removed, let's remove the tests
also.
Signed-off-by: NCleber Rosa <crosa@redhat.com>

Remove variable assigments that are used just once and never actually
used.
Signed-off-by: NCleber Rosa <crosa@redhat.com>

Currently simple tests have a limited status API, being able to set only
the WARN status by generating an output string in a very specific and
hard-coded format.

This patch, while respects the current behaviour, creates configuration
keys, allowing users to provide regular expressions to search for in the
test outputs, aiming to set the final test status to either WARN or SKIP
when the test finishes with exit code 0.
Signed-off-by: NAmador Pahim <apahim@redhat.com>

A lot of the core Avocado code still refers to "mux" based terminology.
For instance, what was previously "mux_path" is now "parameter paths",
and what was previously "mux" is now really variants.
Signed-off-by: NCleber Rosa <crosa@redhat.com>

This is an attempt to catch most (if not all) usages of open
that do not follow the context manager pattern.

Sometimes, for better readability, our own `genio` library is
used.
Signed-off-by: NCleber Rosa <crosa@redhat.com>

The test_basic.RunnerOperationTest.test_early_latest_result waits for
link creation for 0.5s and then it simply fails, without waiting for the
process. The problem is that in such case the tmpdir can be cleaned
before it's used by the created process and therefor left behind.
Signed-off-by: NLukáš Doktor <ldoktor@redhat.com>

Basically switch to use the alias provided by six, with the Python
3 name.
Signed-off-by: NCleber Rosa <crosa@redhat.com>

This changes uses the six compatibility equivalents for iterkeys,
iteritems and itervalues.
Signed-off-by: NCleber Rosa <crosa@redhat.com>

Windows does not allow `<>:"/\|?*` characters in the filename. Let's
include these in our `astring.string_to_safe_path` method and use it
everywhere we create files.
Signed-off-by: NLukáš Doktor <ldoktor@redhat.com>

The html plugin uses entry-point "avocado_result_html", but the pkg name
is "avocado-framework-plugin-result-html" and that is what
"pkg_resources.require" accepts.

This required few tweaks related to recent changes to html plugin which
produces a single result file instead of multiple files.
Signed-off-by: NLukáš Doktor <ldoktor@redhat.com>

The basedir was mainly set correctly, but sometimes not used in some of
the tests. Let's move the `os.chdir` to `setUp` phase where it makes
sense and add new ones where they were not present.

Related to basedir the test_mux was using hardcoded path based on the
method used to execute the test, let's change it to `__file__` which
gives better results.
Signed-off-by: NLukáš Doktor <ldoktor@redhat.com>

These scripts rely on some Python library path manipulation to be
able to load from the source tree.  But, in fact that are a few
reasons for not having those, including:

 * All functionality in `avocado` relies on plugins, which will not be
   available from the source tree unless a `$ setup.py develop` is
   executed, which automatically makes the "import magic" unnecessary.

 * Most users will end up using a setuptools generated "entry point"
   instead.

For the sake of less code, I propose this (and all other?) magic to be
removed.
Signed-off-by: NCleber Rosa <crosa@redhat.com>

Found by the newest version of inspektor, 0.4.5.
Signed-off-by: NLucas Meneghel Rodrigues <lookkas@gmail.com>

Signed-off-by: NPaolo Bonzini <pbonzini@redhat.com>

There are bunch of really short, one-time-used variables defined
globally, which is not really necessary, nor easy to follow. Let's just
define those one/two liners directly to where they are used.
Signed-off-by: NLukáš Doktor <ldoktor@redhat.com>

The fail_reason is the instance of the actual exception. We do store it
only as string in case of old-type exception, but even new style
exceptions might be not available when sys.path is different causing
unpickle on runner part impossible.
Signed-off-by: NLukáš Doktor <ldoktor@redhat.com>

When no variants are provided Avocado sets the default variant, but
instead of list of mux paths it created just string "/run", which
processed into ["/", "r", "u", "n"]. Let's fix this by setting list.

Also the proper mux_path should be `/run/*` in order to include all
children nodes of the "/run".
Signed-off-by: NLukáš Doktor <ldoktor@redhat.com>

Just like `avocado list` presents the statistics for the test types,
it may be useful to also list the number of tests that have a
particular tag.
Signed-off-by: NCleber Rosa <crosa@redhat.com>

Test tags (via docstring directives) have been supported in Avocado
for a number of releases, but up until now user would only be able to
see the tags on their tests by resorting to the source code.

This was already difficult when the tags where set only at the class
level, now that they can be set both on the class and method
docstring, users would have to do the "math" on their minds.

Let's display the test tags when the verbose switch is given to the
list command to make the life of users easier.
Signed-off-by: NCleber Rosa <crosa@redhat.com>

Since too may reports of false positives have been generated during
RPM package builds, let's move all test sensitive tests into level 1.

This means that those tests will not be run on RPM package builds, but
will continue to run on Travis-CI.
Signed-off-by: NCleber Rosa <crosa@redhat.com>

Introducing the --force-execution to create the Job regardless the fact
that some test references might not be resolved to tests.

Reference: https://trello.com/c/bDUJjhq0Signed-off-by: NAmador Pahim <apahim@redhat.com>