As it has already been done to some other tests, two other tests have
been producing false positives.  This has been observed recently in
package builds in build farms.

Let's run them only when `make check-full` is called, which is
usually done at dedicated machines with plenty of resources.

Reference: https://kojipkgs.fedoraproject.org/work/tasks/9189/19059189/build.log
Reference: https://kojipkgs.fedoraproject.org/work/tasks/9445/19069445/build.logSigned-off-by: NCleber Rosa <crosa@redhat.com>

The MissingTest is yielded by TestLoaderProxy, but it was implemented
and described in FileLoader, which caused troubles when FileLoader was
not being used. This simple fix allows TestLoaderProxy to define basic
mappings and adds MissingTest to it.
Signed-off-by: NLukáš Doktor <ldoktor@redhat.com>

When the tests references cannot be resolved to a test_suite, the
results plugins should not output anything.

Reference: https://trello.com/c/oMQsQHC6Signed-off-by: NAmador Pahim <apahim@redhat.com>

This commit enables the left-behind-temporary-dirs check between each
test. As all selftests are executed as a single process the BORG class
keeps the TMP dir of the unit-tests. To avoid results spoiled by this
fact this commit adds a force-cleanup of the tmp dir tracker overriding
the BORG internal state, which is not really nice, but for selftests
acceptable.

The method to clean it was especially named
"unittest_refresh_dir_tracker" to emphasize it should not be used
outside of unittest.
Signed-off-by: NLukáš Doktor <ldoktor@redhat.com>

In some tests we override the job workflow, let's make sure the
post_tests is executed whenever pre_test was.
Signed-off-by: NLukáš Doktor <ldoktor@redhat.com>

Remove the temporary script files which are not located in tmpdirs.
Signed-off-by: NLukáš Doktor <ldoktor@redhat.com>

Use the combination of path-ordered leaves names and a hash of combined
fingerprint as variant_id. This should improve human-readability of the
results and also it should produce machine-friendly unique names, which
change when values in variant are modified.

Because the leaves names are ordered by path, the order of yaml file
does not affect the variant_id. On the other hand any modified/added
value changes the fingerprint and therefor variant_id.

Important feature is also that the variant still contains the
default_params, but the variant-id is not affected by it.
Signed-off-by: NLukáš Doktor <ldoktor@redhat.com>

The TreeNode.fingerprint() generates string, which represents the value
of the node. The idea is to represent the content of the node and it's
environment and such for the purpose of evaluating, whether other node
is alike.
Signed-off-by: NLukáš Doktor <ldoktor@redhat.com>

Using more than one docstrig directive makes Avocado to use only the
first one. This patch makes the safeloader to handle multiple docstring
directives (i.e. `🥑 enable` and `🥑 tags=foo`).

Reference: https://trello.com/c/6FpySXuCSigned-off-by: NAmador Pahim <apahim@redhat.com>

Currently the `selftests/run_coverage` only reports unit coverage. This
patch allows specifying custom `avocado` command in selftests and uses
it to run coverage to also include the functional tests to results.
Signed-off-by: NLukáš Doktor <ldoktor@redhat.com>

`system-out` should be in the same level as `failure`, but we are
creating it as a `failure` child.

This patch puts system-out in the same level as `failure` and adds the
functional test to validate all the xunit outputs against the
junit-4.xsd schema.
Signed-off-by: NAmador Pahim <apahim@redhat.com>

Let's move the junit schema file to .data so we can reuse it for
functional tests.
Signed-off-by: NAmador Pahim <apahim@redhat.com>

This adds the support to define per-node filters, which take part of
environment and are evaluated during multiplexation.

The basic idea is to define relations ({could be only used | could
not be used} with other branch), which makes it a bit different from
what we know from `--mux-filter-{only|out}`:

1. They are applied per complete variant (because the filters are
   compound of per-leaf-filters)
2. They don't remove/keep the nodes, but they only evaluate whether
   this resulting variant makes sense (whether it passes the filters)

See the updated documentation for details.
Signed-off-by: NLukáš Doktor <ldoktor@redhat.com>

Recently we added a check for canceled test, let's also check the
"cancel" number of tests are reported correctly in json plugin.
Signed-off-by: NLukáš Doktor <ldoktor@redhat.com>

self.skip() is to be deprecated. This is the first step, adding a
message to the exception raised by the self.skip() and adjusting the
examples tests accordingly.
Signed-off-by: NAmador Pahim <apahim@redhat.com>

self.cancel() should be accepted everywhere, including setUp()
and tearDown().
Signed-off-by: NAmador Pahim <apahim@redhat.com>

Skip decorators are intended to skip the test. Skipping the test means
also no setUp() and no tearDown(). This patch makes the setUp() and
tearDown() to be also skipped when using the skip decorators.
Signed-off-by: NAmador Pahim <apahim@redhat.com>

By mapping to urlparse* and urllib* libraries from appropriate
locations.
Signed-off-by: NCleber Rosa <crosa@redhat.com>

This is a simple collection of syntatic fixes that work across
Python 2.7 and 3.x.
Signed-off-by: NCleber Rosa <crosa@redhat.com>

We have received many requests to support skip tests from inside the
test method. As we can do that without breaking our own concepts, we
decided to introduce the CANCEL status and the corresponding
self.cancel() method to the Test class.

Reference: https://trello.com/c/viBJIEwISigned-off-by: NAmador Pahim <apahim@redhat.com>

To make sure the seltests does not leave job results behind,
we introduced a check which compares the job-results directory
content from before the selftets with the content after the
selftests. But using `AVOCADO_SELF_CHECK=1 make check`, we
expect at least one extra result in the job-results directory,
making the content of the job-results to not be the same after
the tests.

This patch changes the avocado command used for self check
in `make check`, pointing the results to a temporary directory.

Reference: https://trello.com/c/y57xZSUSSigned-off-by: NAmador Pahim <apahim@redhat.com>

On some environments, such as when building RPM packages, very little
can be assumed to be available.  `man` is such as command that is
*not* available, so let's check for it and skip tests accordingly.
Signed-off-by: NCleber Rosa <crosa@redhat.com>

The correct package name doesn't contain "python-"
Signed-off-by: NAmos Kong <amos@scylladb.com>

The definition of AVOCADO_JOB_FAIL was "Something went wrong with the
Job itself, by explicit avocado.core.exceptions.JobError exception",
while AVOCADO_FAIL is defined as "Something else went wrong and
avocado failed (or crashed). Commonly used on command line validation
errors".

With those in mind, I support that AVOCADO_JOB_FAIL should be slightly
adjusted (see changes in the diff), to include other types of
failures, but limited to job *execution* failures.  AVOCADO_FAIL is
already generic enough, so there's no need to change its definition.
This is important, IMHO, because users could quickly understand if a
job was executed, and failed, or if some earlier type of failure (such
as command line validation) happened.

With this updated definition, I spotted a few places where AVOCADO_FAIL
and AVOCADO_JOB_FAIL usage should be changed.  Examples:

 * In `avocado.plugins.diff` the resultsdir is attempted to be read, but
   in my book that still doesn't qualify as a job failure, because a job
   is not being executed.

 * In `avocado.plugins.replay` while there's functionality to run a
   (replayed) job, many of the occurrences of AVOCADO_JOB_FAIL are
   similar to the `avocado.plugins.diff` and happen while trying to load
   jobdata from the previous job resultsdir.

 * In `avocado.plugins.multiplex` the situation is a bit more tricky.  The
   same code is executed when multiple avocado subcommands are executed.
   If the currently executed command is `run`, it would mean that a job is
   being attempted to be executed.  In fact, a job *is* already executing
   at this point, so a AVOCADO_JOB_FAIL is appropriate.  In other scenarios,
   such as when running `$ avocado multiplex`, these failures have nothing
   to do with jobs, so AVOCADO_FAIL is appropriate.
Signed-off-by: NCleber Rosa <crosa@redhat.com>

macOS doesn't support the extended attributes of the files
in the zip file created for that unittest.
Signed-off-by: NLucas Meneghel Rodrigues <lookkas@gmail.com>

macOS doesn't have any of the system files necessary
for the partition code to work, so let's skip the tests
for that platform.
Signed-off-by: NLucas Meneghel Rodrigues <lookkas@gmail.com>

macOS doesn't have utilities to handle logical volumes out
of the box, so let's skip the lv_utils tests entirely on
that platform.
Signed-off-by: NLucas Meneghel Rodrigues <lookkas@gmail.com>

On macOS, when trying to access certain processes as
a regular user, a PermissionDenied exception will
be thrown. So let's catch that exception to avoid
errors executing test_interrupt.
Signed-off-by: NLucas Meneghel Rodrigues <lookkas@gmail.com>

Signed-off-by: NLucas Meneghel Rodrigues <lookkas@gmail.com>

Some Unixes won't have /bin/true, so let's make life
easier for those other OS to run the unittests.
Signed-off-by: NLucas Meneghel Rodrigues <lookkas@gmail.com>

Currently the variant format is (variant_id, (variant, mux_path)) which
is not really scalable, nor user friendly. Let's use a dictionary
instead.
Signed-off-by: NLukáš Doktor <ldoktor@redhat.com>

The "ignore_new_data" was used by "replay" to override Varianter and
tell it to stop accepting new data by other plugins (like yaml_to_mux).
With the new changes the plugins are part of the Varianter, so when
"replay" replaces it with the old (parsed) one, it will not parse itself
again, therefor the default_params and variants stays untouched.

To make sure this works well a unittest is included.
Signed-off-by: NLukáš Doktor <ldoktor@redhat.com>

This patch replaces the hardcoded mux in Varianter for list of Varianter
plugins, iterating through all of them to produce all specified
variants. The interface is described in `avocado.core.plugin_interfaces`
and we also provide base-implementation for Mux-like plugins in
`avocado.core.mux`.

Mainly this PR is about defining the stevedore entry points, moving the
code around and wrapping the code into a class according the defined
stevedore interface. Doing so allowed us to simplify the `MuxTree`
object which does not need to keep track of the `root` of the tree,
because the root is stored in the pluggin's instance.

There is still one potential issue in Varianter which is probably even
more visible now, which is that we are pickling it in jobdata in order
to be able to reproduce the variants when re-running/diffing the job.
Before this was prone to code-changes but now it can be even worse as
multiple plugins can be enabled and can break loading of the previous
state. To avoid the potential breakage this version simply initializes
only the available extensions which should produce only the supported
variants.
Signed-off-by: NLukáš Doktor <ldoktor@redhat.com>

It is possible to set relative paths in settings, but once we evaluate
them we should turn them into absolute paths to avoid incorrect
representation on `os.chdir` (or so)
Signed-off-by: NLukáš Doktor <ldoktor@redhat.com>

This should help beginners to identify the issue when accidentally
overriding test's public variables. Also add a unittest to support this
note.
Signed-off-by: NLukáš Doktor <ldoktor@redhat.com>

I'm not 100% sure, but I don't see a benefit in allowing people to
override test status from inside test. They are suppose to use
assertions, or `self.fail`-like methods. With this change it'd be harder
to manually override the test status (as can be seen on updated
unittests) but we minimize the accidental write into `self.status`
variable, which can lead to confusing results.

Note we can consider adding `set_status` method to allow setting this
property, but I'd like to avoid allowing direct `self.status = ` usage.
Signed-off-by: NLukáš Doktor <ldoktor@redhat.com>

checkall runs `ls` twice in the job-results dir. Once before the
selftests and once again after the tests, comparing the outputs to make
sure there's no new results there after the selftests.

When the job-results does not exist in advance (the case in travis
environments), the message 'ls: cannot access /home/travis/avocado/job-results:
No such file or directory' is shown in the logs. The message is harmless
and does not affect the check. Let's just suppress the stderr from the
`ls` command in that check.

Reference: https://trello.com/c/IBoLAJjfSigned-off-by: NAmador Pahim <apahim@redhat.com>

Previously we were directly using the yaml.Loader, which is shared
across all `yaml` instances. This patch creates a `_BaseLoader` with the
basic set of constructors. That one is copied and updated of the
run-time data (debug/non-debug) during the `_create_from_yaml` to allow
loading debug and non-debug files in a single execution without
overriding the same loader.
Signed-off-by: NLukáš Doktor <ldoktor@redhat.com>

We are returning the stdout and the stderr of the fabric remote
execution all together inside the CmdResult.stdout

Let's separate them properly.
Signed-off-by: NAmador Pahim <apahim@redhat.com>

The job has the number of tests, but not the total of test executions.
This information comes from the result attribute 'tests_total', updated
by the runner.

This patch changes the tap plugin so it can use the correct information.

Reference: https://trello.com/c/tiusKNNjSigned-off-by: NAmador Pahim <apahim@redhat.com>