When things go wrong, the Avocado command line app will write a file
with the backtrace that caused the trash.  Depending on its content,
it may not be possible to encode that in an file encoded as ASCII.
Let's default to utf-8 then.

Also, this is done for portability reasons, as it brings a consistent
behavior (one explicit encoding type) across Python 2 and 3, which
have different default encodings.
Signed-off-by: NCleber Rosa <crosa@redhat.com>

On Python 3, trying to copy a FilterSet, using the instance.copy()
method, will return a plain set() object.  This influences a number of
code, including the fingerprint generation, and the variant ID which
will be based on the hash of the fingerprint.

Let's make things predictable and copy using Python's standard module
copy.

This fixes the functional test in module `test_getdata` under Python
3, as the variant IDs are now the same on jobs run under Python 2 and
3.
Signed-off-by: NCleber Rosa <crosa@redhat.com>

Since system_output() returns bytes (from CmdResult.stdout), the
split operation should also be done with bytes.
Signed-off-by: NCleber Rosa <crosa@redhat.com>

In situations where a user of the CmdResult class manually sets the
stdout/stderr attributes to a string, instead of the recommended and
documented content of type "bytes", it won't be possible to "decode"
the bytes into a string of a given encoding, and we'll end up with
a crash.

Since the goal of std{out,err}_text is to return a text version
of stdout, if itself already is of such a type, let's just return
it.

Additionally, if the data cannot be "decode()d", let's raise an
explicit TypeError on this location, rather than later in the code
when its value is attempted to be used.
Signed-off-by: NCleber Rosa <crosa@redhat.com>

Commit 7d0e6a44 effectively changed the supposed value of stdout and
stderr attributes to being of type "bytes".  But, the parameters with
the same names, and their defaults values was not updated.

Let's make stdout and stderr into bytes by default, both the
parameters and attribute values when no other is given.
Signed-off-by: NCleber Rosa <crosa@redhat.com>

Signed-off-by: NLukáš Doktor <ldoktor@redhat.com>

* https://github.com/avocado-framework/avocado:
  Support variants dump/load
  core varianter: implement __len__

Depending on the number of parameters, generating the Variants can be
compute intensive. This patch adds the ability to dump/load the variants
to/from a JSON file so the Variants generation can be offloaded from the
machine under test.

Reference: https://trello.com/c/llZJKBdiSigned-off-by: NAmador Pahim <apahim@redhat.com>

So one can call len(Varianter()).
Signed-off-by: NAmador Pahim <apahim@redhat.com>

Signed-off-by: NLukáš Doktor <ldoktor@redhat.com>

* https://github.com/avocado-framework/avocado:
  selftests/unit/test_runner_queue.py: use absolute path for module
  Python 3: do not allow failures on CI, but only run unittests and doc

Signed-off-by: NLukáš Doktor <ldoktor@redhat.com>

* https://github.com/avocado-framework/avocado:
  selftests/functional/test_basic.py: also check for HTML entity references
  selftests/functional/test_output.py: use bytes in comparison
  selftests/functional/test_loader.py: use text in comparisons
  avocado/core/test.py: use AvocadoParams's iteritems()
  selftests/functional/test_output.py: use command output as text
  selftests/functional/test_job_timeout.py: use bytes in comparison

Signed-off-by: NLukáš Doktor <ldoktor@redhat.com>

* https://github.com/avocado-framework/avocado:
  avocado/plugins/run.py: use ValueError.args, as Python 3 has no message attribute
  Avocado config: keep the file ascii only
  selftests/functional/test_loader.py: use regexp on text
  Mux: use six.iterkeys()
  test_mux.TestMuxTree.test_basic_functions: get a string on Python 3
  test_mux.TestMuxTree.test_basic: Use AvocadoParams's iteritems()

Looks like the Python 3 module import code behaves differently than
Python 2 with regards to where it looks modules by name:

   File "/home/cleber/src/avocado/avocado/avocado/core/test.py", line 932, in _run_avocado
      raise test_exception
   File "/home/cleber/src/avocado/avocado/avocado/core/test.py", line 819, in _run_avocado
      testMethod()
   File "/home/cleber/src/avocado/avocado/avocado/core/test.py", line 1371, in test
      self.error(self.exception)
   File "/home/cleber/src/avocado/avocado/avocado/core/test.py", line 1041, in error
      raise exceptions.TestError(message)\navocado.core.exceptions.TestError: Traceback (most recent call last):
   File "/usr/lib64/python3.6/imp.py", line 297, in find_module
      raise ImportError(_ERR_MSG.format(name), name=name)
   ImportError: No module named \'whiteboard\'\n\n'

Effectively, this means when running this test, on Python 3, on the
parallel mode, the change of directory implemented there
(on selftests/checkall) will make Python 3 fail to find this module,
and the test will fail.

This makes the reference to the module path absolute, so no matter
where this test is run from, it should be able to find the module.
Signed-off-by: NCleber Rosa <crosa@redhat.com>

This removes the exception of allowing failures on the Travis CI jobs
with Python 3, while, at the same time, temporary restricts Python 3
jobs to run only unittests and the documentation build for the Avocado
core.  That is, it excludes the functional tests, and the optional
plugin's tests.

While it may look like a step back, it's actually a step forward in the
Python 3 port, because now any regression on the unittests will be caught.
And, when the additional fixes are applied to address the functional tests
and plugin's tests, the whole set of tests will be enabled for Python 3.
Signed-off-by: NCleber Rosa <crosa@redhat.com>

The pystache library on Python 3 will generated an HTML file with entity
references for the single quote char, instead of the literal char.

Let's check for either one of them in the functional test
test_basic.RunnerSimpleTest.test_fs_unfriendly_run().
Signed-off-by: NCleber Rosa <crosa@redhat.com>

The content on the file is being read as bytes, so let's use byte
literals in the comparison.

The error message received some adjustments, mainly because it could
produce weird stuff text such as "1th" or "2th".  To be honest, I'd
remove this error message completely, the assertion error is pretty
easy to spot IMO.
Signed-off-by: NCleber Rosa <crosa@redhat.com>

This fixes the functional test
test_loader.LoaderTestFunctional.test_yaml_loader_run.
Signed-off-by: NCleber Rosa <crosa@redhat.com>

Because it doesn't provide an items() method, on which six.iteritems
relies when running on Python 3.

This fixes the following issue with a functional test:

    ======================================================================
    ERROR: test_dry_run (test_basic.RunnerOperationTest)
    ----------------------------------------------------------------------
    Traceback (most recent call last):
      File "/home/cleber/src/avocado/avocado/selftests/functional/test_basic.py", line 508, in test_dry_run
        result = json.loads(process.run(cmd).stdout)
      File "/home/cleber/src/avocado/avocado/avocado/utils/process.py", line 1238, in run
        raise CmdError(cmd, sp.result)
    avocado.utils.process.CmdError: Command './scripts/avocado run --sysinfo=off passtest.py failtest.py gendata.py --json - --mux-inject foo:1 bar:2 baz:3 foo:foo:a foo:bar:b foo:baz:c bar:bar:bar --dry-run' failed (rc=1)
Signed-off-by: NCleber Rosa <crosa@redhat.com>

Instead of as bytes in assertions, specially because of a part of it
is going to be used as a path.  This fixes the following issue:

    ======================================================================
    ERROR: test_default_enabled_plugins (test_output.OutputPluginTest)
    ----------------------------------------------------------------------
    Traceback (most recent call last):
      File "/home/cleber/src/avocado/avocado/selftests/functional/test_output.py", line 509, in test_default_enabled_plugins
        self.check_output_files(debug_log)
      File "/home/cleber/src/avocado/avocado/selftests/functional/test_output.py", line 286, in check_output_files
        json_output_path = os.path.join(base_dir, 'results.json')
      File "/usr/lib64/python3.6/posixpath.py", line 92, in join
        genericpath._check_arg_types('join', a, *p)
      File "/usr/lib64/python3.6/genericpath.py", line 151, in _check_arg_types
        raise TypeError("Can't mix strings and bytes in path components") from None
    TypeError: Can't mix strings and bytes in path components
Signed-off-by: NCleber Rosa <crosa@redhat.com>

This fixes the functional test "test_invalid_values" (from
test_job_timeout.JobTimeOutTest).
Signed-off-by: NCleber Rosa <crosa@redhat.com>

This fixes occurrences of crashes such as:

    Traceback (most recent call last):
      File "/tmp/py3/bin/avocado", line 6, in <module>
        exec(compile(open(__file__).read(), __file__, 'exec'))
      File "/home/cleber/src/avocado/avocado/scripts/avocado", line 75, in <module>
        sys.exit(app.run())
      File "/home/cleber/src/avocado/avocado/avocado/core/app.py", line 91, in run
        return method(self.parser.args)
      File "/home/cleber/src/avocado/avocado/avocado/plugins/run.py", line 208, in run
        LOG_UI.error(e.message)
    AttributeError: 'ValueError' object has no attribute 'message'
Signed-off-by: NCleber Rosa <crosa@redhat.com>

Further code would be required to make the configparser code load the
configuration file in an encoding different than ascii.  Since we don't
really need, at this time, any non-ascii content in the config file,
let's replace the only character that was not ascii in it.

This fixes the following issue:

    /bin/sh: whacky-unknown-command: command not found
    Error in sys.excepthook:
    Traceback (most recent call last):
      File "./scripts/avocado", line 53, in handle_exception
        os.write(2, msg + '\n')
    TypeError: a bytes-like object is required, not 'str'

    Original exception was:
    Traceback (most recent call last):
      File "./scripts/avocado", line 61, in <module>
        from avocado.core.app import AvocadoApp    # pylint: disable=E0611
      File "/home/cleber/src/avocado/avocado/avocado/__init__.py", line 28, in <module>
        from avocado.core.job import main
      File "/home/cleber/src/avocado/avocado/avocado/core/job.py", line 34, in <module>
        from . import data_dir
      File "/home/cleber/src/avocado/avocado/avocado/core/data_dir.py", line 37, in <module>
        from . import settings
      File "/home/cleber/src/avocado/avocado/avocado/core/settings.py", line 302, in <module>
        settings = Settings()
      File "/home/cleber/src/avocado/avocado/avocado/core/settings.py", line 187, in __init__
        self.process_config_path(config_path_intree)
      File "/home/cleber/src/avocado/avocado/avocado/core/settings.py", line 217, in process_config_path
        read_configs = self.config.read(pth)
      File "/usr/lib64/python3.6/configparser.py", line 697, in read
        self._read(fp, filename)
      File "/usr/lib64/python3.6/configparser.py", line 1015, in _read
        for lineno, line in enumerate(fp, start=1):
      File "/usr/lib64/python3.6/encodings/ascii.py", line 26, in decode
        return codecs.ascii_decode(input, self.errors)[0]
    UnicodeDecodeError: 'ascii' codec can't decode byte 0xe2 in position 2011: ordinal not in range(128)

Which in turn fixes the functional test
test_output.OutputPluginTest.test_broken_pipe on Python 3 that
resulted in a failure like this:

    Traceback (most recent call last):
      File "/home/cleber/src/avocado/avocado/selftests/functional/test_output.py", line 632, in test_broken_pipe
        self.assertEqual(len(result.stderr.splitlines()), 1)
    AssertionError: 30 != 1

Because of the traceback produced on stderr.
Signed-off-by: NCleber Rosa <crosa@redhat.com>

Signed-off-by: NCleber Rosa <crosa@redhat.com>

Because there won't be a iterkeys() method on the value object under
Python 3.

This fixes the `test_mux.TestMuxTree.test_advanced_yaml()` test under
Python 3.
Signed-off-by: NCleber Rosa <crosa@redhat.com>

And not bytes.  What happens here is that
`avocado.core.tree.tree_view()` will return either bytes with 'utf-8'
or 'ascii' depending on the given parameter.

On Python 2, it's fine to compare the result with strings.  On Python
3, we have to get a string from the bytes returned, that is, we need
to decode the bytes.

While at it, let's fix a typo (s/asci/ascii).
Signed-off-by: NCleber Rosa <crosa@redhat.com>

Because it doesn't provide an items() method, on which six.iteritems
relies when running on Python 3.
Signed-off-by: NCleber Rosa <crosa@redhat.com>

Signed-off-by: NAmador Pahim <apahim@redhat.com>

The config file generated and used on this test was missing a
separator between the section name and the first key/value line.

This adds a new line and makes the resulting file aesthetically
pleasing and able to be parsed.
Signed-off-by: NCleber Rosa <crosa@redhat.com>

Signed-off-by: NCleber Rosa <crosa@redhat.com>

Signed-off-by: NCleber Rosa <crosa@redhat.com>

Signed-off-by: NCleber Rosa <crosa@redhat.com>

Signed-off-by: NCleber Rosa <crosa@redhat.com>

Signed-off-by: NCleber Rosa <crosa@redhat.com>

Signed-off-by: NLukáš Doktor <ldoktor@redhat.com>

* https://github.com/avocado-framework/avocado:
  avocado/utils/kernel.py: support other major versions of the Linux kernel

Signed-off-by: NLukáš Doktor <ldoktor@redhat.com>

* https://github.com/avocado-framework/avocado:
  JSON Result Plugin: remove legacy (duplicated) fields

Signed-off-by: NLukáš Doktor <ldoktor@redhat.com>

* https://github.com/avocado-framework/avocado:
  DummyLoader: move it out of core

This loader is supposed to not perform anything locally, and it's
actually a hack to bypass the loader when tests are to be resolved and
run remotely.

Its only users is the remote runner.  To avoid pollution in core,
let's move it to that module.
Signed-off-by: NCleber Rosa <crosa@redhat.com>

Currently we have hardcoded the version 3 series in the URL used to
fetch the source code.  Let's use the major version, from the full
version number, as the URL prefix instead of the hardcoded "3.x".
Signed-off-by: NCleber Rosa <crosa@redhat.com>

The "url" and "test" fields are just legacy and duplicated fields
that carry the same data as the "id" field.  Let's remove those.
Signed-off-by: NCleber Rosa <crosa@redhat.com>

Signed-off-by: NLukáš Doktor <ldoktor@redhat.com>

* https://github.com/avocado-framework/avocado:
  avocado/core/parser.py: remove legacy default_avocado_params
  Makefile: remove cleanup of eggs generated on 36lts
  Travis-CI: remove 36lts branch from automated tests
  avocado.utils.make: drop legacy interface
  avocado/utils/build.py: fix docstring
  varianter: remove _process_default_params() method
  varianter: remove compatiblity code for 36lts
  mux: remove _get_variant_ids() method
  mux: remove compatiblity code for 36lts

Signed-off-by: NAmador Pahim <apahim@redhat.com>