For a long time we have resorted to a hack, let's call it "all commits
hack", to test all commits in a branch on Travis-CI.  While it clear
had a positive side, it also had its annoyances.

Now, more and more often, we're faced with the obligation of breaking
PRs in smaller pieces just because of the limitations that the free
tier of Travis-CI imposes (for instance max job time is 50 minutes).

For instance, as this is written, the status of the master branch
is broken, because of the max job time.  And there's no way to
waive it or tweak it.

Also, the hack caused some really interesting problems, and caused
a lot of time wasted, for instance, because it masks the merge commit
that Travis-CI creates, and consequently, makes the signedoff-check.sh
fail on the real HEAD of the merged branch.  Basically, the "all
commits hack" is a requirement to currently have signedoff-check.sh
passing on Travis-CI.
Signed-off-by: NCleber Rosa <crosa@redhat.com>

Signed-off-by: NPraveen K Pandey <praveen@linux.vnet.ibm.com>

The current Python unittest discovery is too permissive, in the sense
that it will match any "test*" method on any class, wether it inherits
from the TestCase class of the unittest module or not.

This new implementation is more strict, but a lot more capable too.
This makes the Python unittest detection very similar (in code and
capabilities) to the Avocado instrumented test detection.
Signed-off-by: NCleber Rosa <crosa@redhat.com>

So that the specific Avocado checks are kept in its own function, and
other types of checks for other types of tests can be implemented more
cleanly.
Signed-off-by: NCleber Rosa <crosa@redhat.com>

The disabled variable should contain a set of the disabled tests, and
a list was being defined as a fallback if `iter_classes()` doesn't
yield anything.

Let's also make the naming of the variable uniform.
Signed-off-by: NCleber Rosa <crosa@redhat.com>

Signed-off-by: NCleber Rosa <crosa@redhat.com>

When looking for tests, the base class and the module where it
comes from, is of uttermost importance.  But, they have so far
been implicitly defined.

Let's make them explicit, which makes improves readability and
will help to make the utility functions reusable.
Signed-off-by: NCleber Rosa <crosa@redhat.com>

When a module with more than one level is imported, such as:

   import unittest.mock

All levels of the module will be available, that is, it's possible to
write code that makes use of whathever is in `unittest`, for instance,
`unittest.TestCase`.

Let's keep track of all intermediary levels imported to improve the
behavior of PythonModule.
Signed-off-by: NCleber Rosa <crosa@redhat.com>

The real goal of the test_imports attribute is to keep track of
classes that were imported.  The naming made sense given that original
this class (PythonModule) was called AvocadoModule and was hardcoded
to look for Test classes from the avocado module.
Signed-off-by: NCleber Rosa <crosa@redhat.com>

Signed-off-by: NCleber Rosa <crosa@redhat.com>

And those that don't.
Signed-off-by: NCleber Rosa <crosa@redhat.com>

A base class exists in this module, and number of the test classes
inherit from it, from the single supposed benefit of sharing a
(static) method.

Let's flatten the inheritance, and make the static method a utility
function, because, if not for anything else, It improves code
readability.

Note: this was actually observed while working on the safeloader
Python unittest detection improvements, meaning that in theory the
safeloader improvements would make this unnecessary for the actual
loading support.  Still, I believe there are benefits (readability,
maintenance) with these changes.
Signed-off-by: NCleber Rosa <crosa@redhat.com>

Signed-off-by: NCleber Rosa <crosa@redhat.com>

On `_examine_class()`, the original assigment of the avocado tags to
the `cl_tags` variables is not necessary because when it's used in the
first occurrence, a `continue` statement follows the assigment, so
it's never reused.  On the other hand, it may be assigned without ever
being used.

On `find_avocado_tests()`, it's just an opmization because `cl_tags`
was only being used once in the function, and at the same time matches
the style used in the previous function.
Signed-off-by: NCleber Rosa <crosa@redhat.com>

By renaming it to PythonModule, and making the module and klass names
configurable, it should be reusable for other types of Python modules
and classes, including unittest.TestCase.
Signed-off-by: NCleber Rosa <crosa@redhat.com>

By adding internal methods to deal with import from and plain
import statements.
Signed-off-by: NCleber Rosa <crosa@redhat.com>

Just like the next block is doing.
Signed-off-by: NCleber Rosa <crosa@redhat.com>

The module path only get's updated on the AvocadoModule constructor,
so after that, it won't change and the `path` variable is thus
unnecessary.
Signed-off-by: NCleber Rosa <crosa@redhat.com>

Adding info about the missing `is_avocado` parameter, and remove
docstring for non existing parameter (most probably a leftover from
_examine_class()).
Signed-off-by: NCleber Rosa <crosa@redhat.com>

Signed-off-by: NJan Richter <jarichte@redhat.com>

Merge branch 'selftests_extract_from_checkall' of https://github.com/clebergnu/avocado into clebergnu-selftests_extract_from_checkall

Selftests: extract from checkall

Signed-off-by: NCleber Rosa <crosa@redhat.com>

Signed-off-by: NCleber Rosa <crosa@redhat.com>

Signed-off-by: NCleber Rosa <crosa@redhat.com>

Signed-off-by: NCleber Rosa <crosa@redhat.com>

Signed-off-by: NCleber Rosa <crosa@redhat.com>

Signed-off-by: NCleber Rosa <crosa@redhat.com>

Signed-off-by: NCleber Rosa <crosa@redhat.com>

And the requirement target for "check", given that it's already
called from the selftests/checkall script.
Signed-off-by: NCleber Rosa <crosa@redhat.com>

These are superior than the ones defined in the test, and using them
removes the duplicate code.
Signed-off-by: NCleber Rosa <crosa@redhat.com>

The network utility used to (attempt to) get a free port defaults to
picking sequential ports.  But, that's not a requirement of this test.
When running tests in parallel, the same port may get allocated when a
sequential approach is used, and that is greatly minimized when random
ports are requested.
Signed-off-by: NCleber Rosa <crosa@redhat.com>

When working on a private fork and topic branches, it's helpful to not
have this limitation, and have builds triggered for them.

For the main repo, it should not a problem given that branches are not
often (only on LTS releases), and those are also desirable to have
builds run.
Signed-off-by: NCleber Rosa <crosa@redhat.com>

Signed-off-by: NCleber Rosa <crosa@redhat.com>

Signed-off-by: NCleber Rosa <crosa@redhat.com>

Instead of one large test method, let's have one for each.  Also,
let's define the base_logdir from the calling side, as with that
we can be sure that the test won't do any funky business, and
the tearDown() won't be trying to recursively remove what it
shouldn't.
Signed-off-by: NCleber Rosa <crosa@redhat.com>

Before and after variants get inserted.  This should make it easier to
both follow the code, and eventually to make the test factory a
documented structure.
Signed-off-by: NCleber Rosa <crosa@redhat.com>

On all tested Python 3 versions, the SimpleQueue is available from
the multiprocessing module, so let's drop the conditional import.
Signed-off-by: NCleber Rosa <crosa@redhat.com>

A "from <module> import <symbol>" statement will not always have
a module name set, one example is:

   from . import BASEDIR

In these situations, the statement's module attribute will be None.
Signed-off-by: NCleber Rosa <crosa@redhat.com>

Which are only necessary under Python 2, and thus, not needed
anymore.
Signed-off-by: NCleber Rosa <crosa@redhat.com>

The previous working dropping Python 2, removed the explicit
requirement of the six module, but because stevedore depends
on it, it would still be installed.

Now, every trace of explicit six usage has been removed.
Signed-off-by: NCleber Rosa <crosa@redhat.com>