As it has already been done to some other tests, two other tests have
been producing false positives.  This has been observed recently in
package builds in build farms.

Let's run them only when `make check-full` is called, which is
usually done at dedicated machines with plenty of resources.

Reference: https://kojipkgs.fedoraproject.org/work/tasks/9189/19059189/build.log
Reference: https://kojipkgs.fedoraproject.org/work/tasks/9445/19069445/build.logSigned-off-by: NCleber Rosa <crosa@redhat.com>

Currently the `selftests/run_coverage` only reports unit coverage. This
patch allows specifying custom `avocado` command in selftests and uses
it to run coverage to also include the functional tests to results.
Signed-off-by: NLukáš Doktor <ldoktor@redhat.com>

The unittest standard library on Python 2.7 an later has everything
that the unittest2 backport is supposed to have.  Let's then drop all
the conditional imports of unittest2 and stick with unittest.
Signed-off-by: NCleber Rosa <crosa@redhat.com>

This change introduces symbolic names for various filesystem
permission modes, instead of the literal octals with which
Python 3 shows no compassion.
Signed-off-by: NCleber Rosa <crosa@redhat.com>

Because the current implementation of avocado.main() creates a job
and runs "sys.argv[0]" to implement standalone mode, it ends up
running itself over and over.

This simple proposed fix, prevents avocado.main() from running
itself again if called from itself. Since they are on different
processes, the mechanism chosen to do this is to set an environment
variable, that will be seen by the next process.

Also, by exiting from main() with an error code, the test first
level test will fail. This will let the user know that the chosen
approach (SIMPLE tests written in Python and calling main()) are
not worthy of a PASS.

The functional tests make use of Python's standard library utilities
(subprocess module) directly for running Avocado because of current
issues with Avocado's own process utility module.

This adresses issue #961.
Signed-off-by: NCleber Rosa <crosa@redhat.com>

As per report on issue #952, Avocado currently finds multiple tests
when multiple test names exist in a given Python class. When the
test code itself is loaded and the test class instantiated, only
one test method will remain (others will have been overloaded by
that one).

So, to mimic this in the static, AST based, test discovery, let's
make the test methods unique and try to keep the order in which the
AST parser found them.
Signed-off-by: NCleber Rosa <crosa@redhat.com>

Signed-off-by: NCleber Rosa <crosa@redhat.com>

At this point, no plugin support exists in Avocado. This is in
preparation for the new plugin code to be cleanly introduced.

The plugins that play a role in the plugin architecture have
been removed, since they wouldn't be functional under the new
plugin management code. The ones that are add extra functionality
to Avocado have been kept, and will be ported to the new
architecture.

Also, most of the functional tests have been temporarily disabled.
The reason is that most of them run avocado, which depends on the
run command, which in turn, depends on the plugin archicture code.
Signed-off-by: NCleber Rosa <crosa@redhat.com>

While going through the dependency list (requirements*.txt files) and
performing our self tests out of virtual environments, I noticed that
some tests are run outside the virtual environments.

The reason is that, even though the virtual environment is activated
for the test session (and say, `which python` gives `/venv/bin/python`),
we have hard coded `/usr/bin/python` in most places.

According to the some discussions on the virtualenv project itself[1],
a quick solution is to revert to the also common `/usr/bin/env python`
way of pointing to the Python interpreter.

[1] - https://github.com/pypa/virtualenv/issues/124Signed-off-by: NCleber Rosa <crosa@redhat.com>

The BUGGY test concept relies on loading/executing the Python file
containing the tests and failing to do so. Since Avocado won't
load/execute Python test files anymore, it's not possible or desirable
to keep the BUGGY tests around.
Signed-off-by: NCleber Rosa <crosa@redhat.com>

There's no real reason why a single Python file can not hold multiple
test classes, each one with its own tests.

This test adds a simple Python test file with two classes, each one
with its own test, and expects the loader to find two instrumented
tests.
Signed-off-by: NCleber Rosa <crosa@redhat.com>

The test loader should only load (import, thus execute) Python test
modules at test execution time. During, for example, test discovery
and listing time, the Python modules should never be imported (thus
executed).

This introduces a test called "sleep eleven" (because "ten" is boring)
that "accidentally" has a `time.sleep(11)` at the global scope. This
test is listed by executing `avocado list -V sleepeleven.py` and is
given 3 seconds to finish. If the loader actually loads/executes the
Python code, it will fail because it will take at least a little bit
more then 11 seconds.
Signed-off-by: NCleber Rosa <crosa@redhat.com>

Make sure we have avocado_ + __name__ prefixes to the dirs
created with tempfile, so that we have an easy way to spot
when we are not cleaning up directories properly. This mostly
concerns unittests, although the kernel_build lib and the
iso9660 lib also uses that API.
Signed-off-by: NLucas Meneghel Rodrigues <lmr@redhat.com>

Even though I may be the one to blame about coming up with this
"simple import magic", I believe it was a mistake and we should get
rid of them.

There are a couple of other ways to actually do development out of
of a source tree, including running unittests that do not require
this amount of boiler plate code. Examples include just setting
the PYTHONPATH environment variable to actually run (setuptools based)
`python setup.py develop`.

The little bits of what looks like the import magic that was left, is
not really import magic. It's just that functional tests need to locate
the Avocado source tree base directory to run the test runner from it.
Signed-off-by: NCleber Rosa <crosa@redhat.com>

To be honest, our test code could still be kept in the same directories and
have the same names. But I think we can improve two things here:

1) Flatten a little bit the directory structure of selftests. Two path
components are being dropped here: "all" and "avocado". So that
"selftests/all/functional/avocado" becomes simply
"selftests/functional".

2) File names match what is, by default, recognized by unittest (the
Standard Library module) based discovery of tests. That means that
doc_build_test.py becomes test_doc_build.py. Not a big deal IMHO.
Signed-off-by: NCleber Rosa <crosa@redhat.com>

On multiple places we modify the sys.path to make avocado work from
sources. Instead of `append` we should `insert` the path as when running
from avocado sources directory, we want to use the modules from here,
rather than from installed version.
Signed-off-by: NLukáš Doktor <ldoktor@redhat.com>

Previously the "file" loader was special and always matched the URL
which might be confusing and lead to problems when developing additional
plugins.

This commit tires to define the way loaders should work. They must never
fail (unless of uncaught exception which is reported and next available
plugins is used). Instead when the URL produces broken test or even a
single corrupted subtest, consider it as URL not for me and return empty
set. The loader proxy uses next plugin to resolve this URL. When no
plugin resolves the URL user is advised to run "avocado list -V" to see
the potential problems.

The order of loader plugins was also changed. Now the file-loader is
first by default and all remaining ones are loaded by the plugin
priority. The order can be changed in setting
"plugins/loader_plugins_priority". It uses "name" variable of the each
loader plugin. Not listed plugins are loaded afterwards.

Additionally the whole loader API was simplified. Now it only accepts
one parameter - list_tests - which is 3-state:

1. ALL       - all tests including broken/incorrect ones
2. AVAILABLE - available tests (for listing purposes)
2. DEFAULT   - default tests (to run when no url given)
Signed-off-by: NLukáš Doktor <ldoktor@redhat.com>

The runTest method has been obsoleted and everyone should be using
"test*" method(s) instead. This patch removes the support and replaces
the default and example entry points to "test".
Signed-off-by: NLukáš Doktor <ldoktor@redhat.com>

Only avocado.Test is of general interest to test writers. For that reason
the entire test module has been moved to the avocado.core namespace.

The recommended way to go about writing instrumented tests is something like:

   from avocado import Test
   class MyTest(Test):
      def test_foo(self):
         do_stuff()

Changes from v2:
* Fixed references in docstrings
* Fixed refrences of avocado.test in Writing Test Guide docs

Changes from v1:
* Fixed comment typo on commit message
* Fixed old reference of avocado.test (then test.Test) instead of
  avocado then Test in GDB docs.
Signed-off-by: NCleber Rosa <crosa@redhat.com>

Export avocado.core.job.main as avocado.main, so that make it look
like an API for test developers to use.
Signed-off-by: NRudá Moura <rmoura@redhat.com>

Move avocado.job to avocado.core.job, in conformance to our
purpose of keeping things not related to test developers,
inside avocado.core, isolated from users.
Signed-off-by: NRudá Moura <rmoura@redhat.com>

Signed-off-by: NLucas Meneghel Rodrigues <lmr@redhat.com>

We're going one step ahead to unittest  compatibility.
Now we obsolete action() method and use runTest().
Then what is runTest() will be run().

Update the selftests and tests regarding this modification.
Signed-off-by: NRudá Moura <rmoura@redhat.com>

Signed-off-by: NLucas Meneghel Rodrigues <lmr@redhat.com>

More checks to make sure we are not outputting errors to
the wrong loggers.
Signed-off-by: NLucas Meneghel Rodrigues <lmr@redhat.com>

Create an additional logging handler for the avocado application,
that outputs to stderr, and two filters, that filter messages
per logging level. This way, normal information (app output)
goes to stdout and error messages go to stderr. The unittests
were updated accordingly.
Signed-off-by: NLucas Meneghel Rodrigues <lmr@redhat.com>

Having avocado_loader_unittest as the prefix here was
basically a copy/paste mistake.
Signed-off-by: NLucas Meneghel Rodrigues <lmr@redhat.com>

Add support to inspect and find tests inside a directory, recursively,
by using the method `discover_url`. It takes a url (path) and
returns the tests parameters it discovers.

The loader object has the .discover() method, that can be used
to inspect the location, and the .validate() method, that will
discover any problems on the user's input.

We are also changing the behavior of avocado to error out on
invalid inputs (missing paths or files that are not avocado
tests). Therefore, the concept of MISSING or NOT_A_TEST both
disappear from the UI. Unittests were updated to reflect the
new status quo.
Signed-off-by: NRudá Moura <rmoura@redhat.com>
Signed-off-by: NLucas Meneghel Rodrigues <lmr@redhat.com>

* Use --sysinfo=off in the current functional tests. It will
drop the total execution time of the tests.

* Add a new functional test `sysinfo_tests.py` which will actually
test if the sysinfo directory is being created or not, depending on
the proper command line option.

OK, I did it. This has got to be the most stupid commit ever on this repo.
Still, "print 'foo'" annoys me, and if want to have Avocado running on
Python 3 anytime, we would have to do this anyway.
Signed-off-by: NCleber Rosa <crosa@redhat.com>

Self test files used to be executable, but that changed on 143b574c.
So, it doesn't make much sense to keep the shebangs + python commands
on them.
Signed-off-by: NCleber Rosa <crosa@redhat.com>

This will do functional testing of avocado behaving
according to the use cases specified.
Signed-off-by: NLucas Meneghel Rodrigues <lmr@redhat.com>