We have many utils, but no documentation for them. Let's create an
'Utilities' section in our docs and add the first piece of content
there to document the vmimage module.
Signed-off-by: NAmador Pahim <apahim@redhat.com>

Signed-off-by: NCleber Rosa <crosa@redhat.com>

The operator is inverted since the beginning of times and no one ever
noticed. Until now.
Signed-off-by: NAmador Pahim <apahim@redhat.com>

Signed-off-by: NCleber Rosa <crosa@redhat.com>

Signed-off-by: NAmador Pahim <apahim@redhat.com>

Let's also package and build the recently introduced golang plugin.
Signed-off-by: NCleber Rosa <crosa@redhat.com>

There's a problem with the current SPEC file in that it doesn't
exclude some of the files part of the "plugins-loader-yaml" package.

This would cause problems if that plugin package attempts to be
installed, causing conflicts because they're packaged on two packages.
Signed-off-by: NCleber Rosa <crosa@redhat.com>

Signed-off-by: NAmador Pahim <apahim@redhat.com>

Signed-off-by: NAmador Pahim <apahim@redhat.com>

This prevents an error when the avocado golang plugin is installed,
but "go" itself is not.

This still allows tests to be listed without "go" because:
 * it is not a requirement for that
 * there's value in listing tests even when your system is (still?)
   not completely setup to run them
 * Avocado hasn't promised symmetry between listing and guaranteeing
   that tests will be able to run
 * There may be situations in which we can *not* predict if tests
   that we can list can be executed
Signed-off-by: NCleber Rosa <crosa@redhat.com>

Signed-off-by: NCleber Rosa <crosa@redhat.com>

Signed-off-by: NCleber Rosa <crosa@redhat.com>

The support for per-test-sysinfo collection is there ever since the
beginning. Let's add a `per_test` setting to enable the same sysinfo
collection as per-job per-test.
Signed-off-by: NLukáš Doktor <ldoktor@redhat.com>

The Daemons are suppose to output in `self.profile_dir` and not
`self.pre_dir`. Let's use the same handling as in
`start_job_collectibles` in `start_test_collectibles`.
Signed-off-by: NLukáš Doktor <ldoktor@redhat.com>

Signed-off-by: NCleber Rosa <crosa@redhat.com>

The output check functionality logs, on failures, the complete contents
of the both the actual output, and the expected one. Most often than not,
the most relevant information is the difference between both.

This patch logs the unified diffs and records the diff files
(`stdout.diff` and `stderr.diff`) containing the unified diff between
the expected outputs and the actual outputs.

Reference: https://trello.com/c/CXDP1khVSigned-off-by: NAmador Pahim <apahim@redhat.com>

Signed-off-by: NCleber Rosa <crosa@redhat.com>

In WritingTests documentation we mention `examples/tests`, but users
will probably get a better source of information in
`avocado-framework-tests` repository so let's include both here.
Signed-off-by: NLukáš Doktor <ldoktor@redhat.com>

The check output test heavily depends on test's datadir. Let's add a
pointer to where we define the test datadir as it's not just a single
file anymore.
Signed-off-by: NLukáš Doktor <ldoktor@redhat.com>

In most cases the single ` (italic) were meant as doube `` (code).
Signed-off-by: NLukáš Doktor <ldoktor@redhat.com>

The `avocado.fail_on` needs to be used as decorator, not a function.
Signed-off-by: NLukáš Doktor <ldoktor@redhat.com>

The --failfast off explanation is a bit hard to follow (for me),
hopefully this version will be easier to understand without the need to
look at it twice.
Signed-off-by: NLukáš Doktor <ldoktor@redhat.com>

Signed-off-by: NAmador Pahim <apahim@redhat.com>

Signed-off-by: NLukáš Doktor <ldoktor@redhat.com>

There is an extensive description on how INSTRUMENTED test datadir
hierarchy looks like, but it might be useful to mention SIMPLE test as
well to see the difference.
Signed-off-by: NLukáš Doktor <ldoktor@redhat.com>

As part of 0795831f, all occurrences of dict.iteritems() turned into
iteritems(dict).  The problem is that AvocadoParams doesn't implement
the items() method that six.iteritems() expects from Python 3.

So given that AvocadoParams doesn't provide it, we could either implement
items() just to satisfy iteritems(), or just use the existing interface.

This commit chooses the later, which means reverting a small part of
the changes (incorrect ones) from 0795831f.
Signed-off-by: NCleber Rosa <crosa@redhat.com>

On Python 2, that is allowed.  On Python 3, it raises a TypeError
with "'>' not supported between instances of 'NoneType' and 'int'".
Signed-off-by: NCleber Rosa <crosa@redhat.com>

The multiprocessing library in Python 3 has some differences from the
one in Python 2.  One of those, is the requirement of a context for a
SimpleQueue.

Since we're going to use that specific interface (get_context()),
let's check for it, and use it to get a context if it exists.

PS: This approach for checking the interface avoids having an external
library like `six` imported here just for the reason of checking the
Python version.
Signed-off-by: NCleber Rosa <crosa@redhat.com>

The 2to3 feature was enable to give Avocado a head start in the port to
Python 3.  But, while having positive characteristics, it has some
drawbacks, specially at development time, in which an extra step is
required.

The porting work done so far has not relied on 2to3, and it's been
having good results.  Let's then drop 2to3 completely, and allow the
same code base, without any massaging, to be used on both Python 2 and
Python 3.
Signed-off-by: NCleber Rosa <crosa@redhat.com>

The `must_exist` is part of the API and significantly changes the way
the datadirs are processed, therefor it deserves to be mentioned in the
documentation.
Signed-off-by: NLukáš Doktor <ldoktor@redhat.com>

Signed-off-by: NLukáš Doktor <ldoktor@redhat.com>

* https://github.com/avocado-framework/avocado:
  Runner: add option to log data directories for each test to be run
  selftests/unit/test_test.py: fix comment typo
  avocado.Test: use new data dirs for output check record feature
  Limit SimpleTest to "file" and "variant" data file sources
  avocado.Test: deprecate datadir property
  examples/tests/whiteboard.py: remove chdir into datadir
  avocado.Test: introduce get_data() method

While users could follow the documented rules for test directories,
it can be useful to log all of the options in advance.  With that
a test writer can create the data directories and put data files
at the right level.
Signed-off-by: NCleber Rosa <crosa@redhat.com>

Signed-off-by: NCleber Rosa <crosa@redhat.com>

This changes the output check record functionality, from using the
file-level directory only containing std{out,err}.expected files, to
one that respects leaves to the specific implementations on a test
class.

When checking if the output matches the expected one, it just uses the
new public get_data() method.  When recording output data, it either
uses the most specific source provided by the test class.  For
INSTREMENTED tests, it means a variant-level when a variant is being
used, or the test-level directory when a variant is not being used.

A tool that could be added on top of this change of behavior is one
that consolidates recorded output in higher-levels.  Suppose that a
test run with a thousand variants generate the same output.  It's a
waste to have one thousand files (not counting the directories) when a
single file at the file-level data directory would be enough.  But,
this can be implemented later, I suppose.
Signed-off-by: NCleber Rosa <crosa@redhat.com>

Simple tests are defined as an executable that can either return 0
for success, or non-zero for failures.  This means that it doesn't
support multiple tests on a single file.  But, it can be run with
variants.  This means that the possible sources for data files
are "variants" and "file".
Signed-off-by: NCleber Rosa <crosa@redhat.com>

This change deprecates the use of the of the `datadir` property,
and switches to `get_data()` instead.
Signed-off-by: NCleber Rosa <crosa@redhat.com>

The whiteboard test is an example test, and, since, we're about to
deprecate the datadir attribute, let's remote this call to chdir.

If we believe that's necessary to stress this part of the API, let's
write a (non-example) test for that.
Signed-off-by: NCleber Rosa <crosa@redhat.com>

This method allows to look for test data in ways that are more
flexible than the current "datadir" feature currently available.

The current "datadir" is bound to the test filename only, and
can not provide different data for different tests hosted within
a single test file.

The newly introduced get_data() method looks for test data in
various locations, to be able to provide custom data not only
to different tests hosted inside a single test file, but also
to different variants.

Note about this alternative implementation of `get_data()`: it does so
in a much more detached form.  It allows for other test types to
clearly change the way data files are stored and accessed.  It also
introduces the concept of named data sources.

In the previous implementation, the types of "data dirs" were hard
coded, while in this approach, they're defined at the class level and
it's up to the `get_data()` implementation to deal with it.
Consequently, the `get_data()` method gets a second (optional)
parameter that allows the user to specify from which data source the
data file should be retrieved.

This is necessary, for example, when we add configurability to sources
of data files, such as expected `stdout`, `stderr` and `output` files.
Signed-off-by: NCleber Rosa <crosa@redhat.com>

Signed-off-by: NLukáš Doktor <ldoktor@redhat.com>

* https://github.com/avocado-framework/avocado:
  Python 3 port: use six.string_types on isinstance checks
  Python 3 port: adapt to range differences
  Python 3 port: use next() builtin function
  Python 3 port: fix "unclosed file" conditions in avocado.core.{safe,}loader
  selftests/unit/test_test.py: only create fake tests where used
  selftests/unit/test_utils_stacktrace.py: extract raise exception test
  avocado/core/jobdata_compat_36_to_52.py: remove unused code
  avocado/core/loader.py: simplify and improve detection of Python files
  avocado/utils/path.py: optmize is_python method

Signed-off-by: NLukáš Doktor <ldoktor@redhat.com>

* https://github.com/avocado-framework/avocado:
  Introducing the golang plugin