Under Python 3.4, which runs on our CI and is our lowest supported
version, json.loads() requires a string, while newer versions also
allows bytes.

    >>> json.loads(b'{}')
    Traceback (most recent call last):
      File "<stdin>", line 1, in <module>
      File "/home/cleber/.local/lib/python3.4/json/__init__.py", line 312, in loads s.__class__.__name__))
    TypeError: the JSON object must be str, not 'bytes'

Let's make that code compatible with Python 3.4.
Signed-off-by: NCleber Rosa <crosa@redhat.com>

This fixes the functional test
test_loader.LoaderTestFunctional.test_yaml_loader_run.
Signed-off-by: NCleber Rosa <crosa@redhat.com>

Signed-off-by: NCleber Rosa <crosa@redhat.com>

This brings a change in behaviour, in which the stdout/stderr of the
executed process will now be of bytes type, instead of a string type.

Two new attributes, which are implemented as properties, have been
added to the CmdResult class, `stdout_text` and `stderr_text`.  Those
are convenience methods that will return the same content that is in
`stdout` and `stderr`, reespectively, but decoded on the fly[1].

With regards to encoding, if one is not provided, the result of
`sys.getdefaultencoding()` will be used ("utf-8" for Python 3 and
"ascii" for Python 2).

Applications and/or tests using the APIs that return a CmdResult
should, to the best of my knowledge, set a default encoding themselves
so a stable behavior across Python versions.  But that if left to
users of this API.

A different tradeoff/design decision has to do with the tests modified
here.  One option is to have "text" (as in sequences of human readable
glyphs) as being of Python type "str".  On Python 2, "str" can be
compared to "bytes" because a conversion will happen on demand.  That
is, the following is fine on Python 2:

   >>> result = process.run("command")
   >>> "expected" in process.stdout

Where `expected` is of type "str" and `process.stdout` is of type
"bytes".  This is not true of Python 3, so either the types must match
or a conversion must be done explicitly.  The solutions to that are:

1) have these "text" as (of type) "bytes" in the source code itself,
   and avoid the conversion whenever possible
2) have "strings" in the source code itself, and use the conversion
   provided by `CmdResult.stdout_text` and `CmdResult.stderr_text`.

The approach chosen here is to avoid conversion if possible, that is,
use "byte" types, given the fact that the source code encoding is by
default 'ascii' and most of the "text" dealt with here can be
represented in 'ascii' too.  This is equivalent of doing:

   result = process.run("command")
   b"expected" in process.stdout
   "errors: %s" % 0 in process.stderr_text

[1] The obvious alternative, instead of decoding these on the fly
    would be to have multiple copies of the "same" data.  This assumes
    that binary data produced on the stdout/stderr will usually be
    larger than textual data.
Signed-off-by: NCleber Rosa <crosa@redhat.com>

Test to cover the fixes in subtests filter. The filter
should work in a directory discovery for both INSTRUMENTED
and SIMPLE tests.
Signed-off-by: NAmador Pahim <apahim@redhat.com>

The use of PathInspector checks too much and too little at the same time.
For once, it attempts to find Python code inside a file that doesn't have
a '.py' suffix.

This means that a file named "test" instead of "test.py" can be recognized
as an INSTRUMENTED test.  A file no ending in ".py" can not be currently
loaded as a Python module by Avocado, and it results in a test execution
ERROR.

This changes the behavior for files with Python code, but not ending
in '.py' that can not be loaded as INSTRUMENTED tests, and will now
result in NOT_A_TEST instead.
Signed-off-by: NCleber Rosa <crosa@redhat.com>

The python unittests should be executable even now as a simple tests or
via external runner + contrib script. Let's create a new type and
discover python unittests as first-class test types in Avocado.

As Avocado tests are in fact python unittests, we need to manually skip
these by ignoring tests that are discovered as Avocado tests, but are
marked as not-to-be-executed, which required the additional change to
_find_avocado_tests method.
Signed-off-by: NLukáš Doktor <ldoktor@redhat.com>

Signed-off-by: NPaolo Bonzini <pbonzini@redhat.com>

Use the new ability to report full mappings after the discovery and
allow using `test_reference_resolver_class` to specify the resolver
class intended for test discovery.

To allow sufficient flexibility also support
`test_reference_resolver_args` to override resolver arguments and
`test_reference_resolver_extra` to extend the resolver extra_params.

Note the overridden arguments apply only to the resolver, not to whole
Avocado so it's not possible to override wrappers or such just for a
single test execution. Anyway it is possible to set `vt_config` just for
that single loader, which is more-than useful.
Signed-off-by: NLukáš Doktor <ldoktor@redhat.com>

Let's add yaml_testsuite loader which allows parsing YAML file and
producing testsuite out of it.

At this point due to Avocado limitation it only supports
FileLoader-based test discovery, but it should be possible with greater
changes to extend the scope of this loader for just about any Avocado
loader.
Signed-off-by: NLukáš Doktor <ldoktor@redhat.com>

This patch adds the missing definition of `MODE_0644` in
`LoaderTestFunctional` class.
Signed-off-by: NAmador Pahim <apahim@redhat.com>

Both for consistency and for Python 3 support.
Signed-off-by: NCleber Rosa <crosa@redhat.com>

Since too may reports of false positives have been generated during
RPM package builds, let's move all test sensitive tests into level 1.

This means that those tests will not be run on RPM package builds, but
will continue to run on Travis-CI.
Signed-off-by: NCleber Rosa <crosa@redhat.com>

A finer level of granularity may help to extend our testing coverage
while reducing the amount of false positives.  This introduces the
three different levels, mapped like this:

   * level 0, AKA "make check"
   * level 2, AKA "make check-full"

There are no changes of test assignment, that is, tests previously
being run under "check-full" will continue to be run only at that
target.  The same is true for tests that would run under "make check"
before this.

The big change is that there's now a middle ground, that can be
activated by manually setting the AVOCADO_CHECK_LEVEL variable.  Level
1 is intended to be used be used on environments that are halfway
between a dedicated machine and a really low powered environment.
Signed-off-by: NCleber Rosa <crosa@redhat.com>

As it has already been done to some other tests, two other tests have
been producing false positives.  This has been observed recently in
package builds in build farms.

Let's run them only when `make check-full` is called, which is
usually done at dedicated machines with plenty of resources.

Reference: https://kojipkgs.fedoraproject.org/work/tasks/9189/19059189/build.log
Reference: https://kojipkgs.fedoraproject.org/work/tasks/9445/19069445/build.logSigned-off-by: NCleber Rosa <crosa@redhat.com>

Currently the `selftests/run_coverage` only reports unit coverage. This
patch allows specifying custom `avocado` command in selftests and uses
it to run coverage to also include the functional tests to results.
Signed-off-by: NLukáš Doktor <ldoktor@redhat.com>

The unittest standard library on Python 2.7 an later has everything
that the unittest2 backport is supposed to have.  Let's then drop all
the conditional imports of unittest2 and stick with unittest.
Signed-off-by: NCleber Rosa <crosa@redhat.com>

This change introduces symbolic names for various filesystem
permission modes, instead of the literal octals with which
Python 3 shows no compassion.
Signed-off-by: NCleber Rosa <crosa@redhat.com>

Because the current implementation of avocado.main() creates a job
and runs "sys.argv[0]" to implement standalone mode, it ends up
running itself over and over.

This simple proposed fix, prevents avocado.main() from running
itself again if called from itself. Since they are on different
processes, the mechanism chosen to do this is to set an environment
variable, that will be seen by the next process.

Also, by exiting from main() with an error code, the test first
level test will fail. This will let the user know that the chosen
approach (SIMPLE tests written in Python and calling main()) are
not worthy of a PASS.

The functional tests make use of Python's standard library utilities
(subprocess module) directly for running Avocado because of current
issues with Avocado's own process utility module.

This adresses issue #961.
Signed-off-by: NCleber Rosa <crosa@redhat.com>

As per report on issue #952, Avocado currently finds multiple tests
when multiple test names exist in a given Python class. When the
test code itself is loaded and the test class instantiated, only
one test method will remain (others will have been overloaded by
that one).

So, to mimic this in the static, AST based, test discovery, let's
make the test methods unique and try to keep the order in which the
AST parser found them.
Signed-off-by: NCleber Rosa <crosa@redhat.com>

Signed-off-by: NCleber Rosa <crosa@redhat.com>

At this point, no plugin support exists in Avocado. This is in
preparation for the new plugin code to be cleanly introduced.

The plugins that play a role in the plugin architecture have
been removed, since they wouldn't be functional under the new
plugin management code. The ones that are add extra functionality
to Avocado have been kept, and will be ported to the new
architecture.

Also, most of the functional tests have been temporarily disabled.
The reason is that most of them run avocado, which depends on the
run command, which in turn, depends on the plugin archicture code.
Signed-off-by: NCleber Rosa <crosa@redhat.com>

While going through the dependency list (requirements*.txt files) and
performing our self tests out of virtual environments, I noticed that
some tests are run outside the virtual environments.

The reason is that, even though the virtual environment is activated
for the test session (and say, `which python` gives `/venv/bin/python`),
we have hard coded `/usr/bin/python` in most places.

According to the some discussions on the virtualenv project itself[1],
a quick solution is to revert to the also common `/usr/bin/env python`
way of pointing to the Python interpreter.

[1] - https://github.com/pypa/virtualenv/issues/124Signed-off-by: NCleber Rosa <crosa@redhat.com>

The BUGGY test concept relies on loading/executing the Python file
containing the tests and failing to do so. Since Avocado won't
load/execute Python test files anymore, it's not possible or desirable
to keep the BUGGY tests around.
Signed-off-by: NCleber Rosa <crosa@redhat.com>

There's no real reason why a single Python file can not hold multiple
test classes, each one with its own tests.

This test adds a simple Python test file with two classes, each one
with its own test, and expects the loader to find two instrumented
tests.
Signed-off-by: NCleber Rosa <crosa@redhat.com>

The test loader should only load (import, thus execute) Python test
modules at test execution time. During, for example, test discovery
and listing time, the Python modules should never be imported (thus
executed).

This introduces a test called "sleep eleven" (because "ten" is boring)
that "accidentally" has a `time.sleep(11)` at the global scope. This
test is listed by executing `avocado list -V sleepeleven.py` and is
given 3 seconds to finish. If the loader actually loads/executes the
Python code, it will fail because it will take at least a little bit
more then 11 seconds.
Signed-off-by: NCleber Rosa <crosa@redhat.com>

Make sure we have avocado_ + __name__ prefixes to the dirs
created with tempfile, so that we have an easy way to spot
when we are not cleaning up directories properly. This mostly
concerns unittests, although the kernel_build lib and the
iso9660 lib also uses that API.
Signed-off-by: NLucas Meneghel Rodrigues <lmr@redhat.com>

Even though I may be the one to blame about coming up with this
"simple import magic", I believe it was a mistake and we should get
rid of them.

There are a couple of other ways to actually do development out of
of a source tree, including running unittests that do not require
this amount of boiler plate code. Examples include just setting
the PYTHONPATH environment variable to actually run (setuptools based)
`python setup.py develop`.

The little bits of what looks like the import magic that was left, is
not really import magic. It's just that functional tests need to locate
the Avocado source tree base directory to run the test runner from it.
Signed-off-by: NCleber Rosa <crosa@redhat.com>

To be honest, our test code could still be kept in the same directories and
have the same names. But I think we can improve two things here:

1) Flatten a little bit the directory structure of selftests. Two path
components are being dropped here: "all" and "avocado". So that
"selftests/all/functional/avocado" becomes simply
"selftests/functional".

2) File names match what is, by default, recognized by unittest (the
Standard Library module) based discovery of tests. That means that
doc_build_test.py becomes test_doc_build.py. Not a big deal IMHO.
Signed-off-by: NCleber Rosa <crosa@redhat.com>

On multiple places we modify the sys.path to make avocado work from
sources. Instead of `append` we should `insert` the path as when running
from avocado sources directory, we want to use the modules from here,
rather than from installed version.
Signed-off-by: NLukáš Doktor <ldoktor@redhat.com>

Previously the "file" loader was special and always matched the URL
which might be confusing and lead to problems when developing additional
plugins.

This commit tires to define the way loaders should work. They must never
fail (unless of uncaught exception which is reported and next available
plugins is used). Instead when the URL produces broken test or even a
single corrupted subtest, consider it as URL not for me and return empty
set. The loader proxy uses next plugin to resolve this URL. When no
plugin resolves the URL user is advised to run "avocado list -V" to see
the potential problems.

The order of loader plugins was also changed. Now the file-loader is
first by default and all remaining ones are loaded by the plugin
priority. The order can be changed in setting
"plugins/loader_plugins_priority". It uses "name" variable of the each
loader plugin. Not listed plugins are loaded afterwards.

Additionally the whole loader API was simplified. Now it only accepts
one parameter - list_tests - which is 3-state:

1. ALL       - all tests including broken/incorrect ones
2. AVAILABLE - available tests (for listing purposes)
2. DEFAULT   - default tests (to run when no url given)
Signed-off-by: NLukáš Doktor <ldoktor@redhat.com>

The runTest method has been obsoleted and everyone should be using
"test*" method(s) instead. This patch removes the support and replaces
the default and example entry points to "test".
Signed-off-by: NLukáš Doktor <ldoktor@redhat.com>

Only avocado.Test is of general interest to test writers. For that reason
the entire test module has been moved to the avocado.core namespace.

The recommended way to go about writing instrumented tests is something like:

   from avocado import Test
   class MyTest(Test):
      def test_foo(self):
         do_stuff()

Changes from v2:
* Fixed references in docstrings
* Fixed refrences of avocado.test in Writing Test Guide docs

Changes from v1:
* Fixed comment typo on commit message
* Fixed old reference of avocado.test (then test.Test) instead of
  avocado then Test in GDB docs.
Signed-off-by: NCleber Rosa <crosa@redhat.com>

Export avocado.core.job.main as avocado.main, so that make it look
like an API for test developers to use.
Signed-off-by: NRudá Moura <rmoura@redhat.com>

Move avocado.job to avocado.core.job, in conformance to our
purpose of keeping things not related to test developers,
inside avocado.core, isolated from users.
Signed-off-by: NRudá Moura <rmoura@redhat.com>

Signed-off-by: NLucas Meneghel Rodrigues <lmr@redhat.com>

We're going one step ahead to unittest  compatibility.
Now we obsolete action() method and use runTest().
Then what is runTest() will be run().

Update the selftests and tests regarding this modification.
Signed-off-by: NRudá Moura <rmoura@redhat.com>

Signed-off-by: NLucas Meneghel Rodrigues <lmr@redhat.com>

More checks to make sure we are not outputting errors to
the wrong loggers.
Signed-off-by: NLucas Meneghel Rodrigues <lmr@redhat.com>

Create an additional logging handler for the avocado application,
that outputs to stderr, and two filters, that filter messages
per logging level. This way, normal information (app output)
goes to stdout and error messages go to stderr. The unittests
were updated accordingly.
Signed-off-by: NLucas Meneghel Rodrigues <lmr@redhat.com>