Signed-off-by: NCleber Rosa <crosa@redhat.com>

Signed-off-by: NCleber Rosa <crosa@redhat.com>

The support for per-test-sysinfo collection is there ever since the
beginning. Let's add a `per_test` setting to enable the same sysinfo
collection as per-job per-test.
Signed-off-by: NLukáš Doktor <ldoktor@redhat.com>

The Daemons are suppose to output in `self.profile_dir` and not
`self.pre_dir`. Let's use the same handling as in
`start_job_collectibles` in `start_test_collectibles`.
Signed-off-by: NLukáš Doktor <ldoktor@redhat.com>

Signed-off-by: NCleber Rosa <crosa@redhat.com>

The output check functionality logs, on failures, the complete contents
of the both the actual output, and the expected one. Most often than not,
the most relevant information is the difference between both.

This patch logs the unified diffs and records the diff files
(`stdout.diff` and `stderr.diff`) containing the unified diff between
the expected outputs and the actual outputs.

Reference: https://trello.com/c/CXDP1khVSigned-off-by: NAmador Pahim <apahim@redhat.com>

Signed-off-by: NCleber Rosa <crosa@redhat.com>

In WritingTests documentation we mention `examples/tests`, but users
will probably get a better source of information in
`avocado-framework-tests` repository so let's include both here.
Signed-off-by: NLukáš Doktor <ldoktor@redhat.com>

The check output test heavily depends on test's datadir. Let's add a
pointer to where we define the test datadir as it's not just a single
file anymore.
Signed-off-by: NLukáš Doktor <ldoktor@redhat.com>

In most cases the single ` (italic) were meant as doube `` (code).
Signed-off-by: NLukáš Doktor <ldoktor@redhat.com>

The `avocado.fail_on` needs to be used as decorator, not a function.
Signed-off-by: NLukáš Doktor <ldoktor@redhat.com>

The --failfast off explanation is a bit hard to follow (for me),
hopefully this version will be easier to understand without the need to
look at it twice.
Signed-off-by: NLukáš Doktor <ldoktor@redhat.com>

Signed-off-by: NAmador Pahim <apahim@redhat.com>

Signed-off-by: NLukáš Doktor <ldoktor@redhat.com>

There is an extensive description on how INSTRUMENTED test datadir
hierarchy looks like, but it might be useful to mention SIMPLE test as
well to see the difference.
Signed-off-by: NLukáš Doktor <ldoktor@redhat.com>

The `must_exist` is part of the API and significantly changes the way
the datadirs are processed, therefor it deserves to be mentioned in the
documentation.
Signed-off-by: NLukáš Doktor <ldoktor@redhat.com>

Signed-off-by: NLukáš Doktor <ldoktor@redhat.com>

* https://github.com/avocado-framework/avocado:
  Runner: add option to log data directories for each test to be run
  selftests/unit/test_test.py: fix comment typo
  avocado.Test: use new data dirs for output check record feature
  Limit SimpleTest to "file" and "variant" data file sources
  avocado.Test: deprecate datadir property
  examples/tests/whiteboard.py: remove chdir into datadir
  avocado.Test: introduce get_data() method

While users could follow the documented rules for test directories,
it can be useful to log all of the options in advance.  With that
a test writer can create the data directories and put data files
at the right level.
Signed-off-by: NCleber Rosa <crosa@redhat.com>

Signed-off-by: NCleber Rosa <crosa@redhat.com>

This changes the output check record functionality, from using the
file-level directory only containing std{out,err}.expected files, to
one that respects leaves to the specific implementations on a test
class.

When checking if the output matches the expected one, it just uses the
new public get_data() method.  When recording output data, it either
uses the most specific source provided by the test class.  For
INSTREMENTED tests, it means a variant-level when a variant is being
used, or the test-level directory when a variant is not being used.

A tool that could be added on top of this change of behavior is one
that consolidates recorded output in higher-levels.  Suppose that a
test run with a thousand variants generate the same output.  It's a
waste to have one thousand files (not counting the directories) when a
single file at the file-level data directory would be enough.  But,
this can be implemented later, I suppose.
Signed-off-by: NCleber Rosa <crosa@redhat.com>

Simple tests are defined as an executable that can either return 0
for success, or non-zero for failures.  This means that it doesn't
support multiple tests on a single file.  But, it can be run with
variants.  This means that the possible sources for data files
are "variants" and "file".
Signed-off-by: NCleber Rosa <crosa@redhat.com>

This change deprecates the use of the of the `datadir` property,
and switches to `get_data()` instead.
Signed-off-by: NCleber Rosa <crosa@redhat.com>

The whiteboard test is an example test, and, since, we're about to
deprecate the datadir attribute, let's remote this call to chdir.

If we believe that's necessary to stress this part of the API, let's
write a (non-example) test for that.
Signed-off-by: NCleber Rosa <crosa@redhat.com>

This method allows to look for test data in ways that are more
flexible than the current "datadir" feature currently available.

The current "datadir" is bound to the test filename only, and
can not provide different data for different tests hosted within
a single test file.

The newly introduced get_data() method looks for test data in
various locations, to be able to provide custom data not only
to different tests hosted inside a single test file, but also
to different variants.

Note about this alternative implementation of `get_data()`: it does so
in a much more detached form.  It allows for other test types to
clearly change the way data files are stored and accessed.  It also
introduces the concept of named data sources.

In the previous implementation, the types of "data dirs" were hard
coded, while in this approach, they're defined at the class level and
it's up to the `get_data()` implementation to deal with it.
Consequently, the `get_data()` method gets a second (optional)
parameter that allows the user to specify from which data source the
data file should be retrieved.

This is necessary, for example, when we add configurability to sources
of data files, such as expected `stdout`, `stderr` and `output` files.
Signed-off-by: NCleber Rosa <crosa@redhat.com>

Signed-off-by: NLukáš Doktor <ldoktor@redhat.com>

* https://github.com/avocado-framework/avocado:
  Python 3 port: use six.string_types on isinstance checks
  Python 3 port: adapt to range differences
  Python 3 port: use next() builtin function
  Python 3 port: fix "unclosed file" conditions in avocado.core.{safe,}loader
  selftests/unit/test_test.py: only create fake tests where used
  selftests/unit/test_utils_stacktrace.py: extract raise exception test
  avocado/core/jobdata_compat_36_to_52.py: remove unused code
  avocado/core/loader.py: simplify and improve detection of Python files
  avocado/utils/path.py: optmize is_python method

Signed-off-by: NLukáš Doktor <ldoktor@redhat.com>

* https://github.com/avocado-framework/avocado:
  Introducing the golang plugin

golang ships a builtin test runner, which is called by
`go test <opts> <package or path>`. This patch introduces the golang
optional plugin, which can be used to discover and run golang tests
using Avocado.
Signed-off-by: NAmador Pahim <apahim@redhat.com>

Signed-off-by: NAmador Pahim <apahim@redhat.com>

Signed-off-by: NCleber Rosa <crosa@redhat.com>

This basically adds portability improvements, while at the same
time improving some of the checks that were done using the "str"
type instead of "basestring".
Signed-off-by: NCleber Rosa <crosa@redhat.com>

Basically switch to use the alias provided by six, with the Python
3 name.
Signed-off-by: NCleber Rosa <crosa@redhat.com>

Instead of the iterator next method.
Signed-off-by: NCleber Rosa <crosa@redhat.com>

The current code, which doesn't close the file, produces
"ResourceWarning: unclosed file" in Python 3.
Signed-off-by: NCleber Rosa <crosa@redhat.com>

This test currently creates some Avocado test instances (as part of
the test itself) that doesn't get used.  Let's limit the creation to
where they're used.
Signed-off-by: NCleber Rosa <crosa@redhat.com>

The test that makes sure an exception is raised is completely isolated
from the other checks, and can be made a test of its own.
Signed-off-by: NCleber Rosa <crosa@redhat.com>

The filter attribute and the del statement do not seem to be
necessary here.
Signed-off-by: NCleber Rosa <crosa@redhat.com>

The use of PathInspector checks too much and too little at the same time.
For once, it attempts to find Python code inside a file that doesn't have
a '.py' suffix.

This means that a file named "test" instead of "test.py" can be recognized
as an INSTRUMENTED test.  A file no ending in ".py" can not be currently
loaded as a Python module by Avocado, and it results in a test execution
ERROR.

This changes the behavior for files with Python code, but not ending
in '.py' that can not be loaded as INSTRUMENTED tests, and will now
result in NOT_A_TEST instead.
Signed-off-by: NCleber Rosa <crosa@redhat.com>

Signed-off-by: NCleber Rosa <crosa@redhat.com>

Trivial typo correction for the Makefile.
Signed-off-by: NPo-Hsu Lin <po-hsu.lin@canonical.com>

Signed-off-by: NCleber Rosa <crosa@redhat.com>