There are few issues (3 to my count) which affect a number of tests
on Python 3.  Instead of not executing any of the functional tests,
let's skip the broken ones.

This is important because it will prevent new issues from making
its way into the Avocado code.

The list of tests being skipped is being tracked on our planning board,
so that they are accounted for, and enabled when fixed.

Reference: https://trello.com/c/eFY9Vw1R/1282-python-3-functional-tests-checklistSigned-off-by: NCleber Rosa <crosa@redhat.com>

And that have been causing failures on Travis jobs, due to limited
on unreliable amount of resources present.

This is the full list of tests that won't be run anymore on Travis,
only on `make check-full`:

  selftests.functional.test_basic.RunnerOperationTest.test_early_latest_result
  selftests.functional.test_basic.RunnerSimpleTest.test_kill_stopped_sleep
  selftests.functional.test_interrupt.InterruptTest.test_well_behaved_sigint
  selftests.functional.test_interrupt.InterruptTest.test_well_behaved_sigterm
  selftests.functional.test_loader.LoaderTestFunctional.test_simple_using_main
  selftests.functional.test_loader.LoaderTestFunctional.test_sleep_a_lot
Signed-off-by: NCleber Rosa <crosa@redhat.com>

Some tests might produce too much logs which might result in too big
xunit results, that might be hard/impossible to publish. Let's allow
setting a limit to number of characters to those files and embed only
1/2 of the limit from the beginning and 1/2 of the limit from the end of
the log.
Signed-off-by: NLukáš Doktor <ldoktor@redhat.com>

Signed-off-by: NLukáš Doktor <ldoktor@redhat.com>

The newly W605 warning is caused by incorrect escaped chars, that are
automatically translated to '\\' but future versions should treat it as
SyntaxError.
Signed-off-by: NLukáš Doktor <ldoktor@redhat.com>

We are using `;` in the test name, before the Variant ID. That character
is unsafe to be used as a fs path and, as such, need to be replaced for
a `_`.
Signed-off-by: NAmador Pahim <apahim@redhat.com>

As it was done in d2b4edc3, Python 3.4 needs a string to be passed to
json.loads() (and, indirectly, to json.load()).  Different from
d2b4edc3, the content we fetch externally may be able to tell us the
encoding that is used.

Let's use that information, if available, to decode the stream of
bytes into a string that is acceptable to json.loads().
Signed-off-by: NCleber Rosa <crosa@redhat.com>

Signed-off-by: NLukáš Doktor <ldoktor@redhat.com>

The skip_location is suppose to use "skip_location" and not
"warn_location".
Signed-off-by: NLukáš Doktor <ldoktor@redhat.com>

The built-in regexp requires re.MULTILINE to work the same way as before
(allowing the match to be everywhere and not just in the first line).

As for the existing ones I believe users would benefit from allowing
MULTILINE as well, but there might be slight change in behavior.
Signed-off-by: NLukáš Doktor <ldoktor@redhat.com>

And not the hardcoded path "./scripts/avocado".
Signed-off-by: NMerlin Mathesius <mmathesi@redhat.com>
Signed-off-by: NCleber Rosa <crosa@redhat.com>

And only if the method is not available, attempt to fallback to
the stdout itself if it is a string.

Before 30617c1e, the following kind of check was done:

   >>> type(bytes('foo')) in string_types
   False

And after it, the check was changed to:

   >>> isinstance(bytes('foo'), string_types)
   True

That made the bytes decoding not to be performed at times.  While the
new check is correct, we should favor the attempt to decode, and
only if not available, fallback to the original stdout if it looks
like a string.
Signed-off-by: NCleber Rosa <crosa@redhat.com>

Signed-off-by: NCleber Rosa <crosa@redhat.com>

Pylint describes this warning as:

   Used when a module is reimported multiple times.

Let's enable it and fix the one occurrence it finds.
Signed-off-by: NCleber Rosa <crosa@redhat.com>

Signed-off-by: NCleber Rosa <crosa@redhat.com>

Each Job creates several dirs/files and uses certain handlers to setup
the environment, let's be explicit and instead of preparing everything
magically on __init__ and cleaning most of it after "run" let users to
"setup" and "cleanup" the Job handlers explicitly and to simplify the
usage also support __enter__ and __exit__ methods to be able to simply
use:

    # No job
    with Job(...) as job:
        # do your stuff with temporary handlers set
    # Job temporary handlers cleaned
Signed-off-by: NLukáš Doktor <ldoktor@redhat.com>

Currently we use global "avocado.core.data_dir.get_tmp_dir" temporary
directory for Job execution, which does not allow coexistence of
multiple jobs. Let's only use that location as base_tmpdir and create a
per-job dir there.
Signed-off-by: NLukáš Doktor <ldoktor@redhat.com>

These warnings are described as:

 W0612: Used when a variable is defined but not used.
 W0622: Used when a variable or function override a built-in.

After a suggestion from Andrei Stepanov, we decided to immediately
enable these.

Rerefence: https://pagure.io/standard-test-roles/pull-request/144
Reference: https://trello.com/c/Ohh7Q6qj/1183-enable-w-in-travis-make-checkSigned-off-by: NCleber Rosa <crosa@redhat.com>

That is, to fix this pylint condition:

   W0622:263,0: Exception: Redefining built-in 'Exception'

We need to rename the Exception test.  To make things consistent,
let's rename all of the tests accordingly.
Signed-off-by: NCleber Rosa <crosa@redhat.com>

Sometimes, the command to be executed is a string encoded a specific
encoding, different than the system or hard coded default.

Let's make the encoding configurable, and it will apply also to the
content produced by the command.
Signed-off-by: NCleber Rosa <crosa@redhat.com>

The `path.find_command` raises exception in case it fails to find the
binary. Let's use the same approach as in
`selftests.functional.test_basic` to get binary or skip individual
tests.
Signed-off-by: NLukáš Doktor <ldoktor@redhat.com>

The latest jobs have been failing because of the following issues:

    ************* Module raise
    E1130: 48,30: Raise.test: bad operand type for unary -: NoneType
    ************* Module selftests.functional.test_basic
    I1101:1191,24: PluginsXunitTest.run_and_check: Module 'lxml.etree'
    has not 'XMLSchema' member, but source is unavailable. Consider
    adding this module to extension-pkg-whitelist if you want to
    perform analysis based on run-time introspection of living
    objects.
    I1101:1191,40: PluginsXunitTest.run_and_check: Module 'lxml.etree'
    has not 'parse' member, but source is unavailable. Consider adding
    this module to extension-pkg-whitelist if you want to perform
    analysis based on run-time introspection of living objects.
    I1101:1193,43: PluginsXunitTest.run_and_check: Module 'lxml.etree'
    has not 'parse' member, but source is unavailable. Consider adding
    this module to extension-pkg-whitelist if you want to perform
    analysis based on run-time introspection of living objects.
    ************* Module selftests.unit.test_xunit
    I1101: 91,24: xUnitSucceedTest.test_add_success: Module
    'lxml.etree' has not 'XMLSchema' member, but source is
    unavailable. Consider adding this module to
    extension-pkg-whitelist if you want to perform analysis based on
    run-time introspection of living objects.
    I1101: 91,40: xUnitSucceedTest.test_add_success: Module
    'lxml.etree' has not 'parse' member, but source is
    unavailable. Consider adding this module to
    extension-pkg-whitelist if you want to perform analysis based on
    run-time introspection of living objects.
    I1101: 92,43: xUnitSucceedTest.test_add_success: Module
    'lxml.etree' has not 'parse' member, but source is
    unavailable. Consider adding this module to
    extension-pkg-whitelist if you want to perform analysis based on
    run-time introspection of living objects.

We can't whitelist the `lxml.etree` module at this time (inspekt doesn't
have such an option) and the raise failure is a false positive.  Let's
ignore those issues then.
Signed-off-by: NCleber Rosa <crosa@redhat.com>

The AssertionError exception is intended to be used when an internal
condition that must exist, does not exist.  When dealing with values
received from an outer layer, that we can't control, it's better to
use the more appropriate ValueError.

Another reason is that code such as:

    def run_avocado(self):
        self._setup_environment_variables()
        try:
            self._tag_start()
            self._run_avocado()
    ...
        except AssertionError as detail:
             self.__status = 'FAIL'
             self.__fail_class = detail.__class__.__name__
             self.__fail_reason = str(detail)
             self.__traceback = stacktrace.prepare_exc_info(sys.exc_info())
    ...

Can be afected by the AssertionErrors, and return a test failure
instead of a test error.

Reference: https://docs.python.org/3/library/exceptions.html#ValueError
Reference: https://docs.python.org/3/library/exceptions.html#AssertionErrorSigned-off-by: NCleber Rosa <crosa@redhat.com>

Under Python 3.4, which runs on our CI and is our lowest supported
version, json.loads() requires a string, while newer versions also
allows bytes.

    >>> json.loads(b'{}')
    Traceback (most recent call last):
      File "<stdin>", line 1, in <module>
      File "/home/cleber/.local/lib/python3.4/json/__init__.py", line 312, in loads s.__class__.__name__))
    TypeError: the JSON object must be str, not 'bytes'

Let's make that code compatible with Python 3.4.
Signed-off-by: NCleber Rosa <crosa@redhat.com>

The FDDrainer can fail to flush due to some other conditions,
including when the stream handler is a plain StreamHandler,
and not a FileHandler.

Let's add tests that cover this situation, and another one that's
already fixed that deals with closed streams.

Reference: https://github.com/avocado-framework/avocado/pull/2512Signed-off-by: NCleber Rosa <crosa@redhat.com>

The items of a dictionary are not ordered by default, and between
Python 2 and 3 they always come out different in my experience.

Since this test is trying out the representation of a JSON built from
dicts, the same applies.  Let's make the checks a bit more flexible
(and a bit less accurate to be honest) and check for the pieces of
info inside the JSON file content.
Signed-off-by: NCleber Rosa <crosa@redhat.com>

On Python 3, the calculation of a value for a unit will return
a float.  Let's make things predictable accross Python 2 and 3
and force an integer instead.

Reference: https://github.com/avocado-framework/avocado/pull/2498Signed-off-by: NCleber Rosa <crosa@redhat.com>

The skip() method has been deprecated for quite some time.  Let's
remove it for good and have users using the decorators and cancel()
instead.
Signed-off-by: NCleber Rosa <crosa@redhat.com>

Now that we have a second use case for it, let's finally create the
DataSize object.
Signed-off-by: NAmador Pahim <apahim@redhat.com>

In situations where a user of the CmdResult class manually sets the
stdout/stderr attributes to a string, instead of the recommended and
documented content of type "bytes", it won't be possible to "decode"
the bytes into a string of a given encoding, and we'll end up with
a crash.

Since the goal of std{out,err}_text is to return a text version
of stdout, if itself already is of such a type, let's just return
it.

Additionally, if the data cannot be "decode()d", let's raise an
explicit TypeError on this location, rather than later in the code
when its value is attempted to be used.
Signed-off-by: NCleber Rosa <crosa@redhat.com>

Depending on the number of parameters, generating the Variants can be
compute intensive. This patch adds the ability to dump/load the variants
to/from a JSON file so the Variants generation can be offloaded from the
machine under test.

Reference: https://trello.com/c/llZJKBdiSigned-off-by: NAmador Pahim <apahim@redhat.com>

Looks like the Python 3 module import code behaves differently than
Python 2 with regards to where it looks modules by name:

   File "/home/cleber/src/avocado/avocado/avocado/core/test.py", line 932, in _run_avocado
      raise test_exception
   File "/home/cleber/src/avocado/avocado/avocado/core/test.py", line 819, in _run_avocado
      testMethod()
   File "/home/cleber/src/avocado/avocado/avocado/core/test.py", line 1371, in test
      self.error(self.exception)
   File "/home/cleber/src/avocado/avocado/avocado/core/test.py", line 1041, in error
      raise exceptions.TestError(message)\navocado.core.exceptions.TestError: Traceback (most recent call last):
   File "/usr/lib64/python3.6/imp.py", line 297, in find_module
      raise ImportError(_ERR_MSG.format(name), name=name)
   ImportError: No module named \'whiteboard\'\n\n'

Effectively, this means when running this test, on Python 3, on the
parallel mode, the change of directory implemented there
(on selftests/checkall) will make Python 3 fail to find this module,
and the test will fail.

This makes the reference to the module path absolute, so no matter
where this test is run from, it should be able to find the module.
Signed-off-by: NCleber Rosa <crosa@redhat.com>

This removes the exception of allowing failures on the Travis CI jobs
with Python 3, while, at the same time, temporary restricts Python 3
jobs to run only unittests and the documentation build for the Avocado
core.  That is, it excludes the functional tests, and the optional
plugin's tests.

While it may look like a step back, it's actually a step forward in the
Python 3 port, because now any regression on the unittests will be caught.
And, when the additional fixes are applied to address the functional tests
and plugin's tests, the whole set of tests will be enabled for Python 3.
Signed-off-by: NCleber Rosa <crosa@redhat.com>

The pystache library on Python 3 will generated an HTML file with entity
references for the single quote char, instead of the literal char.

Let's check for either one of them in the functional test
test_basic.RunnerSimpleTest.test_fs_unfriendly_run().
Signed-off-by: NCleber Rosa <crosa@redhat.com>

The content on the file is being read as bytes, so let's use byte
literals in the comparison.

The error message received some adjustments, mainly because it could
produce weird stuff text such as "1th" or "2th".  To be honest, I'd
remove this error message completely, the assertion error is pretty
easy to spot IMO.
Signed-off-by: NCleber Rosa <crosa@redhat.com>

This fixes the functional test
test_loader.LoaderTestFunctional.test_yaml_loader_run.
Signed-off-by: NCleber Rosa <crosa@redhat.com>

Instead of as bytes in assertions, specially because of a part of it
is going to be used as a path.  This fixes the following issue:

    ======================================================================
    ERROR: test_default_enabled_plugins (test_output.OutputPluginTest)
    ----------------------------------------------------------------------
    Traceback (most recent call last):
      File "/home/cleber/src/avocado/avocado/selftests/functional/test_output.py", line 509, in test_default_enabled_plugins
        self.check_output_files(debug_log)
      File "/home/cleber/src/avocado/avocado/selftests/functional/test_output.py", line 286, in check_output_files
        json_output_path = os.path.join(base_dir, 'results.json')
      File "/usr/lib64/python3.6/posixpath.py", line 92, in join
        genericpath._check_arg_types('join', a, *p)
      File "/usr/lib64/python3.6/genericpath.py", line 151, in _check_arg_types
        raise TypeError("Can't mix strings and bytes in path components") from None
    TypeError: Can't mix strings and bytes in path components
Signed-off-by: NCleber Rosa <crosa@redhat.com>

This fixes the functional test "test_invalid_values" (from
test_job_timeout.JobTimeOutTest).
Signed-off-by: NCleber Rosa <crosa@redhat.com>

Signed-off-by: NCleber Rosa <crosa@redhat.com>

The config file generated and used on this test was missing a
separator between the section name and the first key/value line.

This adds a new line and makes the resulting file aesthetically
pleasing and able to be parsed.
Signed-off-by: NCleber Rosa <crosa@redhat.com>