Adding here some some possible use case of the current API. So far I
notice that we have two big cases: automatic job discovery vs
manual/custom job. I understand that there is room for improvements but
I would like to keep things consistent and intuitive here, because of
that this work is separating the logic behind the constructors and the
from_config() methods for both: Job() and TestSuite().
Signed-off-by: NBeraldo Leal <bleal@redhat.com>

A lot of changes was made inside job and suite modules. This change will
update the docstrings to map with the current code.
Signed-off-by: NBeraldo Leal <bleal@redhat.com>

Before this patch directories are not being created properly because
uuid1 was not being called. Took the oportunity to replace to uuid4.
Signed-off-by: NBeraldo Leal <bleal@redhat.com>

With .get() we don't need this, since we have default values.
Signed-off-by: NBeraldo Leal <bleal@redhat.com>

This is a natural move since with this flexibility would be nice to have
test_parameters per test suite.
Signed-off-by: NBeraldo Leal <bleal@redhat.com>

If job.config is present when creating a test suite then we will extend
this configuration. This will make it possible to passa a very long job
config and only small/specific configs for each test suite with the
changes.
Signed-off-by: NBeraldo Leal <bleal@redhat.com>

Since we have one variant per suite now, this plugin needs to get the
list of variants instead of only one.
Signed-off-by: NBeraldo Leal <bleal@redhat.com>

Signed-off-by: NBeraldo Leal <bleal@redhat.com>

Now variants is for each Test Suite, we should log from there. Also
jobdata.record only needs the job object pointer.
Signed-off-by: NBeraldo Leal <bleal@redhat.com>

Since that we are going to a more decoupled way, users should create
they test suites using TestSuite() class.
Signed-off-by: NBeraldo Leal <bleal@redhat.com>

This will return a sum of all len(test_suite.tests) of the current job.
Signed-off-by: NBeraldo Leal <bleal@redhat.com>

This property will hold a list of TestSuite objects. I'm adding a
test_suite property here for backward compatibility. But it will be
removed soon.
Signed-off-by: NBeraldo Leal <bleal@redhat.com>

This method could be useful for other modules and since that is not
related to the job, IMO, makes sense to move to a more generic module.
Signed-off-by: NBeraldo Leal <bleal@redhat.com>

Signed-off-by: NCleber Rosa <crosa@redhat.com>

To make sure that all interface checks, current and future, will be
run with all the existing runner implementations.

Reference: https://github.com/avocado-framework/avocado/issues/4058Signed-off-by: NCleber Rosa <crosa@redhat.com>

Signed-off-by: NCleber Rosa <crosa@redhat.com>

Signed-off-by: NCleber Rosa <crosa@redhat.com>

Signed-off-by: NCleber Rosa <crosa@redhat.com>

The logging system inside avocado doesn't expect changes of configuration
between jobs. This commit brings feature for enabling and disabling logging
configurations of the system. It saves the old configurations to the list for
later use.

Reference: #3997
Signed-off-by: NJan Richter <jarichte@redhat.com>

By using the correct keys related to the filtering by tags
configuration.

Fixes: https://github.com/avocado-framework/avocado/issues/4054Signed-off-by: NCleber Rosa <crosa@redhat.com>

And reuse the core verbose option, like many other subcommands do.
Signed-off-by: NCleber Rosa <crosa@redhat.com>

Signed-off-by: NCleber Rosa <crosa@redhat.com>

Under the 5 seconds timeout that is specified in the job.  The best
way we can control the environment in which those tests run is to skip
them on general `make check` runs, and tag them so that they are
executed one at a time on
`selftests/pre_release/jobs/timesensitive.py`.

Fixes: https://github.com/avocado-framework/avocado/issues/3689Signed-off-by: NCleber Rosa <crosa@redhat.com>

Signed-off-by: NCleber Rosa <crosa@redhat.com>

Signed-off-by: NJan Richter <jarichte@redhat.com>

avocado run command line can accept options --loaders and --external-runner in
one time. But when we use external runner there is a specific loader and we
don't use values from option --loaders. When user uses both these options, we
have to warn him about this behaviour.

Reference: #3899
Signed-off-by: NJan Richter <jarichte@redhat.com>

Signed-off-by: NCleber Rosa <crosa@redhat.com>

In ecf253c4 we removed the default parameters, but forgot to remove
all the documentation.
Signed-off-by: NCleber Rosa <crosa@redhat.com>

Signed-off-by: NCleber Rosa <crosa@redhat.com>

Signed-off-by: NCleber Rosa <crosa@redhat.com>

Signed-off-by: NCleber Rosa <crosa@redhat.com>

Signed-off-by: NCleber Rosa <crosa@redhat.com>

Signed-off-by: NCleber Rosa <crosa@redhat.com>

Handling yes for vgcreate command, since vg creation can fail if
a filesystem is already present. Using the force parameter for this.
Reported-by: NWen Xiong <wenxiong@linux.vnet.ibm.com>
Signed-off-by: NNarasimhan V <sim@linux.vnet.ibm.com>

So that the features they use, end up being tested as a whole.

Note: these may and should become irrelevant once we have complete Job
API feature coverage, but it should catch and avoid breakages for now.
Signed-off-by: NCleber Rosa <crosa@redhat.com>

Like the other example jobs.
Signed-off-by: NCleber Rosa <crosa@redhat.com>

With the introduction of the TestSuite class, this plugin need to use
the right attributes, like `tests` and `size`.
Signed-off-by: NCleber Rosa <crosa@redhat.com>

Currently we pass/show the raw CmdError as the failure information, resulting
in output like this:

   JOB ID     : 989cf0b0c8d90fb593eb5bc6ff9291aa3248955c
   JOB LOG    : /home/cleber/avocado/job-results/job-2020-08-02T16.31-989cf0b/job.log
    (1/1) /bin/false: FAIL: Command '/bin/false' failed.\nstdout: b''\nstderr: b''\nadditional_info: None (0.06 s)
   RESULTS    : PASS 0 | ERROR 0 | FAIL 1 | SKIP 0 | WARN 0 | INTERRUPT 0 | CANCEL 0
   JOB HTML   : /home/cleber/avocado/job-results/job-2020-08-02T16.31-989cf0b/results.html
   JOB TIME   : 0.14 s

Which contains a duplicate of the command path (already in the test
name) an always empty and useless "additional_info", needless new
lines, and very bad formatting.  This changes turns it into:

   JOB ID     : 9cde24d36988d6441f755c14f79b2e908ae533ef
   JOB LOG    : /home/cleber/avocado/job-results/job-2020-08-02T16.58-9cde24d/job.log
    (1/1) /bin/false: FAIL: Exited with status: '1', stdout: '' stderr: '' (0.08 s)
   RESULTS    : PASS 0 | ERROR 0 | FAIL 1 | SKIP 0 | WARN 0 | INTERRUPT 0 | CANCEL 0
   JOB HTML   : /home/cleber/avocado/job-results/job-2020-08-02T16.58-9cde24d/results.html
   JOB TIME   : 0.16 s

Removing the superfluous, and adding the execution exit status, and
better formatting.
Signed-off-by: NCleber Rosa <crosa@redhat.com>

Currently, when running `avocado run`, the 'HTML RESULT' line is
printed, even if the job had no tests in the test suite and did not
really run.

The reason is that on 856c3b57 the TestSuite() class was introduced,
and with that, a different exception started to being raised when an
empty test suite was created.

When the new exception is handled, the job is set to FAILed, instead
of ERRORed.  We could change the exception or their handling, but I
think this side effect is actually a good thing.  A Job with an empty
test suite failed to proceed, but did not really ERRORed.

ERROR is currently reserved, and used in most cases, when things that
are not expected happen, and say, Avocado crashes.
Signed-off-by: NCleber Rosa <crosa@redhat.com>

With the mechanism (and requirement) for plugins to update the default
parameters gone, let's remove the default parameters themselves from
the variants.
Signed-off-by: NCleber Rosa <crosa@redhat.com>