The variants file is called multiplex even though we changed the
terminology long ago. Let's support both names for now and slowly move
to the correct name.
Signed-off-by: NLukáš Doktor <ldoktor@redhat.com>

Previous commit d5fd51cf attempted to remove all shebangs, as a result
of 143b574c that remove the executable bit of all tests.

There are still a few left, so this let's clean them up.
Signed-off-by: NCleber Rosa <crosa@redhat.com>

Currently the `selftests/run_coverage` only reports unit coverage. This
patch allows specifying custom `avocado` command in selftests and uses
it to run coverage to also include the functional tests to results.
Signed-off-by: NLukáš Doktor <ldoktor@redhat.com>

The definition of AVOCADO_JOB_FAIL was "Something went wrong with the
Job itself, by explicit avocado.core.exceptions.JobError exception",
while AVOCADO_FAIL is defined as "Something else went wrong and
avocado failed (or crashed). Commonly used on command line validation
errors".

With those in mind, I support that AVOCADO_JOB_FAIL should be slightly
adjusted (see changes in the diff), to include other types of
failures, but limited to job *execution* failures.  AVOCADO_FAIL is
already generic enough, so there's no need to change its definition.
This is important, IMHO, because users could quickly understand if a
job was executed, and failed, or if some earlier type of failure (such
as command line validation) happened.

With this updated definition, I spotted a few places where AVOCADO_FAIL
and AVOCADO_JOB_FAIL usage should be changed.  Examples:

 * In `avocado.plugins.diff` the resultsdir is attempted to be read, but
   in my book that still doesn't qualify as a job failure, because a job
   is not being executed.

 * In `avocado.plugins.replay` while there's functionality to run a
   (replayed) job, many of the occurrences of AVOCADO_JOB_FAIL are
   similar to the `avocado.plugins.diff` and happen while trying to load
   jobdata from the previous job resultsdir.

 * In `avocado.plugins.multiplex` the situation is a bit more tricky.  The
   same code is executed when multiple avocado subcommands are executed.
   If the currently executed command is `run`, it would mean that a job is
   being attempted to be executed.  In fact, a job *is* already executing
   at this point, so a AVOCADO_JOB_FAIL is appropriate.  In other scenarios,
   such as when running `$ avocado multiplex`, these failures have nothing
   to do with jobs, so AVOCADO_FAIL is appropriate.
Signed-off-by: NCleber Rosa <crosa@redhat.com>

The "ignore_new_data" was used by "replay" to override Varianter and
tell it to stop accepting new data by other plugins (like yaml_to_mux).
With the new changes the plugins are part of the Varianter, so when
"replay" replaces it with the old (parsed) one, it will not parse itself
again, therefor the default_params and variants stays untouched.

To make sure this works well a unittest is included.
Signed-off-by: NLukáš Doktor <ldoktor@redhat.com>

The unittest standard library on Python 2.7 an later has everything
that the unittest2 backport is supposed to have.  Let's then drop all
the conditional imports of unittest2 and stick with unittest.
Signed-off-by: NCleber Rosa <crosa@redhat.com>

The concept of multiplexer is about to be split into an abstract API for
variants generation, multiplexer which will be in-core implementation of
one of the variants generation plugins and yaml_to_mux plugin which uses
this multiplexer implementation to create variants from a yaml file.
Signed-off-by: NLukáš Doktor <ldoktor@redhat.com>

It has been some time now since replay feature accepts the job to be
identified by its results directory. Now it's time to remove the
--replay-data-dir option, since this feature became redundant.
Signed-off-by: NAmador Pahim <apahim@redhat.com>

Since we're renaming all of the mentions from URL to Test References, it also
makes sense to rename the `jobdata/urls` file to `jobdata/test_references`.

As compatibility measures, this tries to read the `urls` file when
`test_references` is not available (newer avocado and older jobdata) and
also adds a symlink from `test_referneces` to a link called `urls`, when
older avocado finds a newer jobdata.
Signed-off-by: NCleber Rosa <crosa@redhat.com>

It's been a while since the "Test ID RFC" came out:

https://www.redhat.com/archives/avocado-devel/2016-March/msg00024.html

Still, we've been referring to test URLs all around our code
and also on user visible strings.  This is an attempt to rename
the mentions of "URLs" that really should be "test references"
or simply "references" at this point.
Signed-off-by: NCleber Rosa <crosa@redhat.com>

Recently we created separate `yaml2mux` plugin, which keeps using the
`--multiplex` argument. The connection between yaml2mux and multiplex is
not really obvious and the separation was meant to bring the order to
the avocado params system. This is another step which replaces the
`--multiplex|-m` with `--mux-yaml|-m` and makes it more obvious that
it's related to `yaml` files and it's also related to `multiplexation`.

You might notice that the old `-m` was kept. It's very convenient and
also allows coexisting of old and new versions (useful eg. in remote
executions).
Signed-off-by: NLukáš Doktor <ldoktor@redhat.com>

Currently `replay` has hardcoded treating of `--multiplex`, which is not
systematic and even now allows some unhandled errors (like --mux-filter
along with replay-test-status). With this patch I propose to ignore all
multiplex related options and use the one from the job and process the
mux only when the user uses `--replay-ignore mux`. This means it does
not allow mixing of previous job's mux with new options, on the other
hand it's simple and easily understandable:

1. no `--replay-ignore mux` => all --mux options are ignored and the
   original Mux is used
2. `--replay-ignore mux` => old Mux is ignored and full new Mux is used
   including system-wide args, --mux options and the yaml parser (-m)

To make things clear this patch always logs a warning notifying the user
about the current situation.
Signed-off-by: NLukáš Doktor <ldoktor@redhat.com>

The module name changed from 'replay' to 'jobdata'. Let's use a directory
name that reflects the new module name.

This patch renames the job data directory from 'replay' to 'jobdata'.
To keep the compatibility, we try to retrieve the data from the 'replay'
directory when the 'jobdata' directory is not available.
Signed-off-by: NAmador Pahim <apahim@redhat.com>

This patch makes the avocado.core.jobdata code more clean and
efficient.
Signed-off-by: NAmador Pahim <apahim@redhat.com>

Now users can execute 'avocado run --replay latest' to have the latest
job replayed. No need to remember the job id.

Reference: https://trello.com/c/C0YCTT6dSigned-off-by: NAmador Pahim <apahim@redhat.com>

This patch fixes and improves resolving of the test aliases (paths
relative to `data_dir.get_tests_dir`).

1. It removes the magic `.py` suffix, which prevented non-python files
   from being resolved (simple.sh was resolved to $tests/simple.sh.py)
2. It adds the feature to filter the class/method from the test
   reference by parsing $test_reference:$class.$method for implicit
   resolver. (previously this worked only for absolute path resolver)

Visible changes are:

* `avocado run passtest` => fails (used to work)
* `avocado run passtest.py` => works (did not work)
* `avocado run env_variables.sh` => works (did not work)
* `avocado run passtest.py:PassTest.test` => works (did not work)
Signed-off-by: NLukáš Doktor <ldoktor@redhat.com>

And deprecate the --multiplex-files option, which was actually
the complete name of the option, but was not being used formally
by our documentation.
Signed-off-by: NCleber Rosa <crosa@redhat.com>

Record the args and reload loader options on job replay.

Reference: https://trello.com/c/N3UCnR05Signed-off-by: NAmador Pahim <apahim@redhat.com>

The "View" concept was developed to abstract the messages and events.
The problem is it was only abused to pass the status to other places, to
colorize the messages and to allow creating paginated view.

There already is a nice class hooked to all important events, the
"TestResult".

For (not only human readable) messages this commit uses the standard
python logging as it's pretty well known, widely used and very scalable.
The colored output is handled by already existing
"ProgressStreamHandler", which maps: DEBUG,INFO,WARNING and >=ERROR
messages to previously existing: minor, message, warning and error event
types.

The paginator was unified and is initialized during logging reconfigure.
During reconfigure all previously logged messages are re-logged into the
output so one does not lose those messages. Another great difference is
that the Paginator is cleaned at exit by avocado and does not require
complex handling to avoid broken console. To use paginator one just
enables it in "args" and writes to any available stream/stdout/stderr
Signed-off-by: NLukáš Doktor <ldoktor@redhat.com>

Options --multiplex-files and/or urls are incompatible with
--replay-test-status, since the resulting variants will differ from the
original job.

This patch adds a test to fail avocado when those incompatible options are
used togheter.

Also, some minor fixes in replay messages; and in return code when --remote
is used.

Reference: https://trello.com/c/OJpRv36iSigned-off-by: NAmador Pahim <apahim@redhat.com>

I noticed during a run of the self test suite that system information
was being collected (`rpm -qa` and the like can usually be seen using
quite a few resources).

It looks like this little typo was causing that.
Signed-off-by: NCleber Rosa <crosa@redhat.com>

Using an unknown option in a given subcommand leads argparse to
show the main avocado usage instructions.

This patch makes the argparse show the subcommand help instead.

Also, argparse error exit code now matches error_codes.AVOCADO_FAIL,
which makes more sense than error_codes.AVOCADO_JOB_FAIL.

Reference:https://trello.com/c/o0TpH9aeSigned-off-by: NAmador Pahim <apahim@redhat.com>

Currently replay is using status.mapping as the list of valid status
for the --replay-test-status option. But status.mapping has some internal
status that users should not have to deal with.

This patch creates a user facing status which is a subset with only
the options relevant for the users and makes the job replay feature to take
advantage of it.
Signed-off-by: NAmador Pahim <apahim@redhat.com>

This patch is merging the concepts of TEST_NA and SKIP, making
the consequent status more relevant for the users.
Signed-off-by: NAmador Pahim <apahim@redhat.com>

Basic functional tests for the job replay feature.
Signed-off-by: NAmador Pahim <apahim@redhat.com>