To make sure the seltests does not leave job results behind,
we introduced a check which compares the job-results directory
content from before the selftets with the content after the
selftests. But using `AVOCADO_SELF_CHECK=1 make check`, we
expect at least one extra result in the job-results directory,
making the content of the job-results to not be the same after
the tests.

This patch changes the avocado command used for self check
in `make check`, pointing the results to a temporary directory.

Reference: https://trello.com/c/y57xZSUSSigned-off-by: NAmador Pahim <apahim@redhat.com>

Signed-off-by: NAmador Pahim <apahim@redhat.com>

Signed-off-by: NJun Guo <blackhumour.gj@gmail.com>

Signed-off-by: NAmador Pahim <apahim@redhat.com>

On some environments, such as when building RPM packages, very little
can be assumed to be available.  `man` is such as command that is
*not* available, so let's check for it and skip tests accordingly.
Signed-off-by: NCleber Rosa <crosa@redhat.com>

This is a step towards reducing the delta between Avocado "upstream"
packages, that is, the packages offered in our own custom repo, and
packages provided by "dowsntream" distros such as Fedora and (EP)EL.

The package rename is probably the change that could affect users
the most, so let's try to do this early (sprint-wise).  The proper
"Provides" and "Obsoletes" tags should, though, make the update
seamless.
Signed-off-by: NCleber Rosa <crosa@redhat.com>

Signed-off-by: NAmador Pahim <apahim@redhat.com>

Signed-off-by: NAmador Pahim <apahim@redhat.com>

The correct package name doesn't contain "python-"
Signed-off-by: NAmos Kong <amos@scylladb.com>

And not `order()` as mentioned in the docstring.
Signed-off-by: NCleber Rosa <crosa@redhat.com>

Signed-off-by: NAmador Pahim <apahim@redhat.com>

The definition of AVOCADO_JOB_FAIL was "Something went wrong with the
Job itself, by explicit avocado.core.exceptions.JobError exception",
while AVOCADO_FAIL is defined as "Something else went wrong and
avocado failed (or crashed). Commonly used on command line validation
errors".

With those in mind, I support that AVOCADO_JOB_FAIL should be slightly
adjusted (see changes in the diff), to include other types of
failures, but limited to job *execution* failures.  AVOCADO_FAIL is
already generic enough, so there's no need to change its definition.
This is important, IMHO, because users could quickly understand if a
job was executed, and failed, or if some earlier type of failure (such
as command line validation) happened.

With this updated definition, I spotted a few places where AVOCADO_FAIL
and AVOCADO_JOB_FAIL usage should be changed.  Examples:

 * In `avocado.plugins.diff` the resultsdir is attempted to be read, but
   in my book that still doesn't qualify as a job failure, because a job
   is not being executed.

 * In `avocado.plugins.replay` while there's functionality to run a
   (replayed) job, many of the occurrences of AVOCADO_JOB_FAIL are
   similar to the `avocado.plugins.diff` and happen while trying to load
   jobdata from the previous job resultsdir.

 * In `avocado.plugins.multiplex` the situation is a bit more tricky.  The
   same code is executed when multiple avocado subcommands are executed.
   If the currently executed command is `run`, it would mean that a job is
   being attempted to be executed.  In fact, a job *is* already executing
   at this point, so a AVOCADO_JOB_FAIL is appropriate.  In other scenarios,
   such as when running `$ avocado multiplex`, these failures have nothing
   to do with jobs, so AVOCADO_FAIL is appropriate.
Signed-off-by: NCleber Rosa <crosa@redhat.com>

Signed-off-by: NAmador Pahim <apahim@redhat.com>

Many process related APIs that avocado has are tied to
the GNU version of the 'ps' command, and therefore, do
not directly port to MacOS.

The correct solution would be to reimplement those APIs
using the 'psutil' module, but then people would complain
about avocado requiring EPEL. I don't have a good plan
on how to solve it just yet, so let's just document those
as TODO list items. We can fix those on later Pull Requests.
Signed-off-by: NLucas Meneghel Rodrigues <lookkas@gmail.com>

macOS doesn't support the extended attributes of the files
in the zip file created for that unittest.
Signed-off-by: NLucas Meneghel Rodrigues <lookkas@gmail.com>

macOS doesn't have any of the system files necessary
for the partition code to work, so let's skip the tests
for that platform.
Signed-off-by: NLucas Meneghel Rodrigues <lookkas@gmail.com>

macOS doesn't have utilities to handle logical volumes out
of the box, so let's skip the lv_utils tests entirely on
that platform.
Signed-off-by: NLucas Meneghel Rodrigues <lookkas@gmail.com>

On macOS, when trying to access certain processes as
a regular user, a PermissionDenied exception will
be thrown. So let's catch that exception to avoid
errors executing test_interrupt.
Signed-off-by: NLucas Meneghel Rodrigues <lookkas@gmail.com>

Signed-off-by: NLucas Meneghel Rodrigues <lookkas@gmail.com>

The message obtained trying to execute doublefree will
be different when using other C libraries. Let's expand
the list of accepted outputs by adding the macOS C library
message.
Signed-off-by: NLucas Meneghel Rodrigues <lookkas@gmail.com>

Some Unixes won't have /bin/true, so let's make life
easier for those other OS to run the unittests.
Signed-off-by: NLucas Meneghel Rodrigues <lookkas@gmail.com>

The filesystem name in Mac OS is 'cd9660', so we can
easily adapt the mounting code to that OS.
Signed-off-by: NLucas Meneghel Rodrigues <lookkas@gmail.com>

Signed-off-by: NCleber Rosa <crosa@redhat.com>

Signed-off-by: NAmador Pahim <apahim@redhat.com>

Signed-off-by: NCleber Rosa <crosa@redhat.com>

All the plugin interfaces defined in `avocado.core.plugin_interfaces`
ommit the "plugin" term, simply because this is already qualified in
the module name.

Let's also standardize the Varianter interface to "plugin-less" name.
Signed-off-by: NCleber Rosa <crosa@redhat.com>

variants.itertests() changed the format of each variant from a tuple to
a dictionary. After that change, the runner was not updated to provide
the mux_path considering the new format.

This patch adds the mux_path to the factory params in _iter_variants(),
fixing the Test.self.__params resolution.
Signed-off-by: NAmador Pahim <apahim@redhat.com>

Signed-off-by: NCleber Rosa <crosa@redhat.com>

Signed-off-by: NCleber Rosa <crosa@redhat.com>

Here the spell fixes for the v47 test plan.
Signed-off-by: NAmador Pahim <apahim@redhat.com>

man is quite outdated. This patch updates the man to reflect the
current options and UI.

Reference: https://trello.com/c/UB6hG2mDSigned-off-by: NAmador Pahim <apahim@redhat.com>

Signed-off-by: NCleber Rosa <crosa@redhat.com>

There are some old docstrings which are not much helpful, let's improve
them.
Signed-off-by: NLukáš Doktor <ldoktor@redhat.com>

Finally after all the refactoring, refining and modifying here is the
up-to-date documentation which follows the current structure and tries
to describe both theoretically and practically how are test's parameters
produced and how they can be used.
Signed-off-by: NLukáš Doktor <ldoktor@redhat.com>

Currently the variant format is (variant_id, (variant, mux_path)) which
is not really scalable, nor user friendly. Let's use a dictionary
instead.
Signed-off-by: NLukáš Doktor <ldoktor@redhat.com>

The "ignore_new_data" was used by "replay" to override Varianter and
tell it to stop accepting new data by other plugins (like yaml_to_mux).
With the new changes the plugins are part of the Varianter, so when
"replay" replaces it with the old (parsed) one, it will not parse itself
again, therefor the default_params and variants stays untouched.

To make sure this works well a unittest is included.
Signed-off-by: NLukáš Doktor <ldoktor@redhat.com>

The "mux-skip-defaults" is a generic concept and should be renamed to
"variants-skip-defaults" as the whole generic concept is called
variant/varianter. This change is only internal and not visible to
users.
Signed-off-by: NLukáš Doktor <ldoktor@redhat.com>

This patch replaces the hardcoded mux in Varianter for list of Varianter
plugins, iterating through all of them to produce all specified
variants. The interface is described in `avocado.core.plugin_interfaces`
and we also provide base-implementation for Mux-like plugins in
`avocado.core.mux`.

Mainly this PR is about defining the stevedore entry points, moving the
code around and wrapping the code into a class according the defined
stevedore interface. Doing so allowed us to simplify the `MuxTree`
object which does not need to keep track of the `root` of the tree,
because the root is stored in the pluggin's instance.

There is still one potential issue in Varianter which is probably even
more visible now, which is that we are pickling it in jobdata in order
to be able to reproduce the variants when re-running/diffing the job.
Before this was prone to code-changes but now it can be even worse as
multiple plugins can be enabled and can break loading of the previous
state. To avoid the potential breakage this version simply initializes
only the available extensions which should produce only the supported
variants.
Signed-off-by: NLukáš Doktor <ldoktor@redhat.com>

The idea behind default_params is, that some of the params are
considered the default_params, they can change, but are not part of
variant, nor the variant-id.

This implementation replaces the `Mux.data_inject` with
`Mux.add_default_param` which also accepts `name` to map data to
different plugins/groups in order to allow ignoring only some of them.

During parsing it extends multiplex variants of the defaults (merges
multiplex variants into defaults, therefor the mux overrides the values)
in order to provide the combination while yielding variants. Then when
there are variants, it yields the variant (compound of the variant and
the default_params) and variant-id. If there are no variants, it yields
only the default_params with no variant-id (None).
Signed-off-by: NLukáš Doktor <ldoktor@redhat.com>

Currently the Varianter itself calculates the number of tests, but in
the future we want to allow each variants plugin to report the number of
tests. Let's add support for `__len__` to the Mux plugin and utilize it
in Varianter instead of hardcoded iteration.
Signed-off-by: NLukáš Doktor <ldoktor@redhat.com>