As it has already been done to some other tests, two other tests have
been producing false positives.  This has been observed recently in
package builds in build farms.

Let's run them only when `make check-full` is called, which is
usually done at dedicated machines with plenty of resources.

Reference: https://kojipkgs.fedoraproject.org/work/tasks/9189/19059189/build.log
Reference: https://kojipkgs.fedoraproject.org/work/tasks/9445/19069445/build.logSigned-off-by: NCleber Rosa <crosa@redhat.com>

The MissingTest is yielded by TestLoaderProxy, but it was implemented
and described in FileLoader, which caused troubles when FileLoader was
not being used. This simple fix allows TestLoaderProxy to define basic
mappings and adds MissingTest to it.
Signed-off-by: NLukáš Doktor <ldoktor@redhat.com>

When the tests references cannot be resolved to a test_suite, the
results plugins should not output anything.

Reference: https://trello.com/c/oMQsQHC6Signed-off-by: NAmador Pahim <apahim@redhat.com>

Remove the temporary script files which are not located in tmpdirs.
Signed-off-by: NLukáš Doktor <ldoktor@redhat.com>

Use the combination of path-ordered leaves names and a hash of combined
fingerprint as variant_id. This should improve human-readability of the
results and also it should produce machine-friendly unique names, which
change when values in variant are modified.

Because the leaves names are ordered by path, the order of yaml file
does not affect the variant_id. On the other hand any modified/added
value changes the fingerprint and therefor variant_id.

Important feature is also that the variant still contains the
default_params, but the variant-id is not affected by it.
Signed-off-by: NLukáš Doktor <ldoktor@redhat.com>

Currently the `selftests/run_coverage` only reports unit coverage. This
patch allows specifying custom `avocado` command in selftests and uses
it to run coverage to also include the functional tests to results.
Signed-off-by: NLukáš Doktor <ldoktor@redhat.com>

`system-out` should be in the same level as `failure`, but we are
creating it as a `failure` child.

This patch puts system-out in the same level as `failure` and adds the
functional test to validate all the xunit outputs against the
junit-4.xsd schema.
Signed-off-by: NAmador Pahim <apahim@redhat.com>

Recently we added a check for canceled test, let's also check the
"cancel" number of tests are reported correctly in json plugin.
Signed-off-by: NLukáš Doktor <ldoktor@redhat.com>

self.skip() is to be deprecated. This is the first step, adding a
message to the exception raised by the self.skip() and adjusting the
examples tests accordingly.
Signed-off-by: NAmador Pahim <apahim@redhat.com>

self.cancel() should be accepted everywhere, including setUp()
and tearDown().
Signed-off-by: NAmador Pahim <apahim@redhat.com>

Skip decorators are intended to skip the test. Skipping the test means
also no setUp() and no tearDown(). This patch makes the setUp() and
tearDown() to be also skipped when using the skip decorators.
Signed-off-by: NAmador Pahim <apahim@redhat.com>

By mapping to urlparse* and urllib* libraries from appropriate
locations.
Signed-off-by: NCleber Rosa <crosa@redhat.com>

This is a simple collection of syntatic fixes that work across
Python 2.7 and 3.x.
Signed-off-by: NCleber Rosa <crosa@redhat.com>

We have received many requests to support skip tests from inside the
test method. As we can do that without breaking our own concepts, we
decided to introduce the CANCEL status and the corresponding
self.cancel() method to the Test class.

Reference: https://trello.com/c/viBJIEwISigned-off-by: NAmador Pahim <apahim@redhat.com>

On some environments, such as when building RPM packages, very little
can be assumed to be available.  `man` is such as command that is
*not* available, so let's check for it and skip tests accordingly.
Signed-off-by: NCleber Rosa <crosa@redhat.com>

The definition of AVOCADO_JOB_FAIL was "Something went wrong with the
Job itself, by explicit avocado.core.exceptions.JobError exception",
while AVOCADO_FAIL is defined as "Something else went wrong and
avocado failed (or crashed). Commonly used on command line validation
errors".

With those in mind, I support that AVOCADO_JOB_FAIL should be slightly
adjusted (see changes in the diff), to include other types of
failures, but limited to job *execution* failures.  AVOCADO_FAIL is
already generic enough, so there's no need to change its definition.
This is important, IMHO, because users could quickly understand if a
job was executed, and failed, or if some earlier type of failure (such
as command line validation) happened.

With this updated definition, I spotted a few places where AVOCADO_FAIL
and AVOCADO_JOB_FAIL usage should be changed.  Examples:

 * In `avocado.plugins.diff` the resultsdir is attempted to be read, but
   in my book that still doesn't qualify as a job failure, because a job
   is not being executed.

 * In `avocado.plugins.replay` while there's functionality to run a
   (replayed) job, many of the occurrences of AVOCADO_JOB_FAIL are
   similar to the `avocado.plugins.diff` and happen while trying to load
   jobdata from the previous job resultsdir.

 * In `avocado.plugins.multiplex` the situation is a bit more tricky.  The
   same code is executed when multiple avocado subcommands are executed.
   If the currently executed command is `run`, it would mean that a job is
   being attempted to be executed.  In fact, a job *is* already executing
   at this point, so a AVOCADO_JOB_FAIL is appropriate.  In other scenarios,
   such as when running `$ avocado multiplex`, these failures have nothing
   to do with jobs, so AVOCADO_FAIL is appropriate.
Signed-off-by: NCleber Rosa <crosa@redhat.com>

macOS doesn't have utilities to handle logical volumes out
of the box, so let's skip the lv_utils tests entirely on
that platform.
Signed-off-by: NLucas Meneghel Rodrigues <lookkas@gmail.com>

On macOS, when trying to access certain processes as
a regular user, a PermissionDenied exception will
be thrown. So let's catch that exception to avoid
errors executing test_interrupt.
Signed-off-by: NLucas Meneghel Rodrigues <lookkas@gmail.com>

Signed-off-by: NLucas Meneghel Rodrigues <lookkas@gmail.com>

The "ignore_new_data" was used by "replay" to override Varianter and
tell it to stop accepting new data by other plugins (like yaml_to_mux).
With the new changes the plugins are part of the Varianter, so when
"replay" replaces it with the old (parsed) one, it will not parse itself
again, therefor the default_params and variants stays untouched.

To make sure this works well a unittest is included.
Signed-off-by: NLukáš Doktor <ldoktor@redhat.com>

It is possible to set relative paths in settings, but once we evaluate
them we should turn them into absolute paths to avoid incorrect
representation on `os.chdir` (or so)
Signed-off-by: NLukáš Doktor <ldoktor@redhat.com>

I'm not 100% sure, but I don't see a benefit in allowing people to
override test status from inside test. They are suppose to use
assertions, or `self.fail`-like methods. With this change it'd be harder
to manually override the test status (as can be seen on updated
unittests) but we minimize the accidental write into `self.status`
variable, which can lead to confusing results.

Note we can consider adding `set_status` method to allow setting this
property, but I'd like to avoid allowing direct `self.status = ` usage.
Signed-off-by: NLukáš Doktor <ldoktor@redhat.com>

The job has the number of tests, but not the total of test executions.
This information comes from the result attribute 'tests_total', updated
by the runner.

This patch changes the tap plugin so it can use the correct information.

Reference: https://trello.com/c/tiusKNNjSigned-off-by: NAmador Pahim <apahim@redhat.com>

On broken/unresponsive machines it might happen that some of the sysinfo
commands don't finish. This allows setting "commands_timeout" in
[sysinfo.collect] section to prevent hangs.

Note this is not for all sysinfo collection, only for the commands
execution, which is currently the issue.
Signed-off-by: NLukáš Doktor <ldoktor@redhat.com>

The unittest standard library on Python 2.7 an later has everything
that the unittest2 backport is supposed to have.  Let's then drop all
the conditional imports of unittest2 and stick with unittest.
Signed-off-by: NCleber Rosa <crosa@redhat.com>

There has been intermitent errors on this specific test while running
on Travis.  We should not let this impact our work, so until we
identify the cause of the failure, let's skip this test *on Travis*
only.  This test will still be run as part of `make check-full`.

There's also a Trello card to not let us forget about this test.

Reference: https://trello.com/c/yXtgDRjkSigned-off-by: NCleber Rosa <crosa@redhat.com>

The SIMPLE test used here contains SHELL code, so it doesn't make sense
to name with a .py extension.
Signed-off-by: NCleber Rosa <crosa@redhat.com>

Avocado does not support self.skip() to be called from inside a test.
This is a design decision that users have to comply with. On the other
hand, users keep requesting alternative ways to skip tests.

This patch creates the decorators skip, skipIf and skipUnless to be
used in test functions. Corresponding exceptions will be raised in
those cases.

Reference: https://trello.com/c/JK5Z5dqlSigned-off-by: NAmador Pahim <apahim@redhat.com>

This PR adjusts our tests to avoid producing results in the default
results directory (usually caused by instantiating Job or Test without a
results dir).

One of the tests still produces test results as it is essential for the
test execution, but it goes through the produced tests and removes the
base directory.
Signed-off-by: NLukáš Doktor <ldoktor@redhat.com>

There's one functional test that is missing "--sysinfo=off".  I
happened to find this because lspci on my system hangs when running
under X.  That test would timeout at the sysinfo collection.
Signed-off-by: NCleber Rosa <crosa@redhat.com>

To provide support for passing arguments for simple tests in command
line, we were creating some cases where the test reference could not be
properly handled. Also, for test references with white-spaces, we have
an inconsistent behavior. See reference.

This patch removes the support for arguments in test references,
restoring the proper behavior, which is expected to be shell-like.

As consequence, we now handle correctly white-spaces in tests references
as well as Unicode characters.

Reference: https://www.redhat.com/archives/avocado-devel/2016-November/msg00011.html
Reference: https://trello.com/c/bfE9NBblSigned-off-by: NAmador Pahim <apahim@redhat.com>

Let's add a check that teststmpdir is shared across different jobs when
set manually via environment variables. This also contains a little
improvement in one of the used tests.
Signed-off-by: NLukáš Doktor <ldoktor@redhat.com>

The "AVOCADO_TESTS_COMMON_TMPDIR" is an environmental variable name used
to store the location of a common tmpdir. To avoid issues with
typos/name changes let's use the variable in `avocado.core.test` to
share the env variable name everywhere.
Signed-off-by: NLukáš Doktor <ldoktor@redhat.com>

The concept of multiplexer is about to be split into an abstract API for
variants generation, multiplexer which will be in-core implementation of
one of the variants generation plugins and yaml_to_mux plugin which uses
this multiplexer implementation to create variants from a yaml file.
Signed-off-by: NLukáš Doktor <ldoktor@redhat.com>

This commit creates a job pre/post plugin that generates a temporary
directory intended to be consumed by the tests. The temporary directory
will be created in 'pre', available via environment variable and removed
in 'post. We are also exposing the tests temporary directory in the Test
API.

Reference: https://trello.com/c/P1JOtVY7Signed-off-by: NAmador Pahim <apahim@redhat.com>

Default bootstrap library contains a custom font we don't really use.
Let's generate a version without it from:

    http://getbootstrap.com/customize/

Note I only removed the font, it still can be minimized...
Signed-off-by: NLukáš Doktor <ldoktor@redhat.com>

We do produce `results.xml` and `results.json`, let's also include
`results.tap` by default and allow disabling this by `--tap-job-result`.

Note it's impossible to initialize the default result.tap in `__init__`
as we only get `args` and not the job itself, therefor this patch adds
it as first thing inside the `pre_tests` step.
Signed-off-by: NLukáš Doktor <ldoktor@redhat.com>

We've been seen failures on resource constrained environments on some
tests that are either time sensitive and/or assume a some computing
resources will be available.

Environments such as TravisCI and RPM build servers may not have the
resources these tests expect, so let's move them to runs of `make
check-full`.  This will hopefully put an end to the false positives
we've been getting on TravisCI.
Signed-off-by: NCleber Rosa <crosa@redhat.com>

We aim to mark more tests that are not suitable to run everywhere, so
it looks like a good idea to rename this to check-full.
Signed-off-by: NCleber Rosa <crosa@redhat.com>

It has been some time now since replay feature accepts the job to be
identified by its results directory. Now it's time to remove the
--replay-data-dir option, since this feature became redundant.
Signed-off-by: NAmador Pahim <apahim@redhat.com>