This is similar to the change in 418bf92a, but now applied to the
functional tests that deal with xunit.
Signed-off-by: NCleber Rosa <crosa@redhat.com>

That's specific to Python 3, but works fine on Python 2.
Signed-off-by: NCleber Rosa <crosa@redhat.com>

Which includes the module name, and the different treatment of
exceptions that do not inherit from BaseExcpetion.
Signed-off-by: NCleber Rosa <crosa@redhat.com>

Signed-off-by: NCleber Rosa <crosa@redhat.com>

Avocado used to provide support for "test aliases", that is, one could
call "avocado run synctest" when the file containing the test was
named "synctest.py".  See e65ff614 for an example.

Since this "feature" has been long removed, let's remove the tests
also.
Signed-off-by: NCleber Rosa <crosa@redhat.com>

Remove variable assigments that are used just once and never actually
used.
Signed-off-by: NCleber Rosa <crosa@redhat.com>

Currently simple tests have a limited status API, being able to set only
the WARN status by generating an output string in a very specific and
hard-coded format.

This patch, while respects the current behaviour, creates configuration
keys, allowing users to provide regular expressions to search for in the
test outputs, aiming to set the final test status to either WARN or SKIP
when the test finishes with exit code 0.
Signed-off-by: NAmador Pahim <apahim@redhat.com>

Which is an optional plugin, and the test statuses feature is at the
innermost core of Avocado.  While the original test indeed saved a lot
of duplicated code, it's code that serves a specific testing purpose,
and it's better to be simpler and more verbose and with the right (no)
dependencies.

This issue came up during the Python 3 port work.  Now that all the
unittests (selftests/unit/*) were passing (locally, there's one other
PR pending to be accepted), the next logical step is getting the
"core" functional (selftests/functional/*) tests passing.  Because
the 'varianter_yaml_to_mux' optional plugins port is not done, the
"core" functional tests fail and the current work flow is broken.
Signed-off-by: NCleber Rosa <crosa@redhat.com>

And drop the notion of a "multiplex capable" Avocado.  If the plugin is
installed, then we assume the dependencies have been fulfilled.  The plugin
presence is used then for the creation of the test suite, including the
plugin's own tests.
Signed-off-by: NCleber Rosa <crosa@redhat.com>

This simple test, after the s390x version, checks the proper parsing
of and x86_64 /proc/cpuinfo.
Signed-off-by: NCleber Rosa <crosa@redhat.com>

A lot of the core Avocado code still refers to "mux" based terminology.
For instance, what was previously "mux_path" is now "parameter paths",
and what was previously "mux" is now really variants.
Signed-off-by: NCleber Rosa <crosa@redhat.com>

The sysinfo code replicates some of the process handling code
already in avocado.utils.process.  Let's use our library instead
and remove code duplication.

This also marks the sysinfo interruption tests as "time and resource
sensitive", and will not run on environments such as Travis-CI, in
a similar way that other interruption based tests are currently set.
Signed-off-by: NCleber Rosa <crosa@redhat.com>

This is an attempt to catch most (if not all) usages of open
that do not follow the context manager pattern.

Sometimes, for better readability, our own `genio` library is
used.
Signed-off-by: NCleber Rosa <crosa@redhat.com>

The avocado.utils.process APIs offer the `allow_output_check`
parameter which exists at a function call level, and thus should be
applied only during the scope of that function.

While it's questionable if a test process should be able to skip
some of its generated output (by means of an `avocado.utils.process`
function parameter), the fact is that the utility libraries offer
that at a lower level.
Signed-off-by: NCleber Rosa <crosa@redhat.com>

In commit 4514a8ad, the default value from the command line option
`--output-check-record` was being used to set the mode of the
`avocado.utils.process` APIs with regards to output record mode.

The problem with that is it makes it impossible for the API to
distinguish between "no explicit behavior requested" and "explicitly
being asked to disable recording".
Signed-off-by: NCleber Rosa <crosa@redhat.com>

Avocado recently introduced the support to record the combined
output generated by a test.  Now, this introduces support for
checking against a previously record combined output.

If an 'output.expected' file exists, it'll take precedence
over the `stdout.expected` and `stderr.expected` files.

Also it's necessary to switch the default operation mode for
the `avocado.utils.process` functions, since to compare with
an `output.expected` file, the record mode for the currently
running test has to be the same.
Signed-off-by: NCleber Rosa <crosa@redhat.com>

The same code is run on multiple other tests.  It's a waste to run
it yet another time by itself.
Signed-off-by: NCleber Rosa <crosa@redhat.com>

Test to cover the fixes in subtests filter. The filter
should work in a directory discovery for both INSTRUMENTED
and SIMPLE tests.
Signed-off-by: NAmador Pahim <apahim@redhat.com>

And include jobdata results and a test for 52.0 (LTS).

The 52.0 release, an LTS release, was meant to allow transitions from
the previous LTS version (36.0lts) and it did so including the ability
to replay jobs created with the older version.

Now, we're 6+ months into the 52.0 release, and as promised, we'll
stop maintaining 36.0lts.  To keep to code clean, let's remove all
this compatibility with versions before 52.0.
Signed-off-by: NCleber Rosa <crosa@redhat.com>

It looks like Travis does some different handling of process, and
we need to manually force the writes to the stdout.
Signed-off-by: NCleber Rosa <crosa@redhat.com>

And the support for combined mode.  So far, there was no need to
determine how processes would be run, and how their output would be
collected: the stdout and stderr would always be collcted separately.

Now, with the need to support the "combined" mode, the
`--output-check-record` command line switch not only has to control
which files are populated (stdout.expected, stderr.expected) but also
it needs to control the execution of process, because it's what will
generate the "raw material" for a possible "output.expected" file.
Signed-off-by: NCleber Rosa <crosa@redhat.com>

The test_basic.RunnerOperationTest.test_early_latest_result waits for
link creation for 0.5s and then it simply fails, without waiting for the
process. The problem is that in such case the tmpdir can be cleaned
before it's used by the created process and therefor left behind.
Signed-off-by: NLukáš Doktor <ldoktor@redhat.com>

The output check functionality logs, on failures, the complete contents
of the both the actual output, and the expected one. Most often than not,
the most relevant information is the difference between both.

This patch logs the unified diffs and records the diff files
(`stdout.diff` and `stderr.diff`) containing the unified diff between
the expected outputs and the actual outputs.

Reference: https://trello.com/c/CXDP1khVSigned-off-by: NAmador Pahim <apahim@redhat.com>

This method allows to look for test data in ways that are more
flexible than the current "datadir" feature currently available.

The current "datadir" is bound to the test filename only, and
can not provide different data for different tests hosted within
a single test file.

The newly introduced get_data() method looks for test data in
various locations, to be able to provide custom data not only
to different tests hosted inside a single test file, but also
to different variants.

Note about this alternative implementation of `get_data()`: it does so
in a much more detached form.  It allows for other test types to
clearly change the way data files are stored and accessed.  It also
introduces the concept of named data sources.

In the previous implementation, the types of "data dirs" were hard
coded, while in this approach, they're defined at the class level and
it's up to the `get_data()` implementation to deal with it.
Consequently, the `get_data()` method gets a second (optional)
parameter that allows the user to specify from which data source the
data file should be retrieved.

This is necessary, for example, when we add configurability to sources
of data files, such as expected `stdout`, `stderr` and `output` files.
Signed-off-by: NCleber Rosa <crosa@redhat.com>

Basically switch to use the alias provided by six, with the Python
3 name.
Signed-off-by: NCleber Rosa <crosa@redhat.com>

The use of PathInspector checks too much and too little at the same time.
For once, it attempts to find Python code inside a file that doesn't have
a '.py' suffix.

This means that a file named "test" instead of "test.py" can be recognized
as an INSTRUMENTED test.  A file no ending in ".py" can not be currently
loaded as a Python module by Avocado, and it results in a test execution
ERROR.

This changes the behavior for files with Python code, but not ending
in '.py' that can not be loaded as INSTRUMENTED tests, and will now
result in NOT_A_TEST instead.
Signed-off-by: NCleber Rosa <crosa@redhat.com>

The 4235b329 changed the
stdout/stderr/output logging handlers to not to add tailing '\n's, but
these are processed by LoggingFile, which is designed to remove the
'\n's and to logged them per line. As a result the unittest was
incorrectly changed as the first outputs actually add the '\n' which was
removed in the expected output.

This patch keeps the raw approach as it is necessary for the last
buffered line and instead of buffering puts all lines including the
tailing '\n' and not '\n' for the remaining last line (if it's
available). Consequently this should improve the situation where output
is being produced by stderr and stdout concurrently as no buffering of
the non-tailing-newline output is done on LoggingFile level.
Signed-off-by: NLukáš Doktor <ldoktor@redhat.com>

This changes uses the six compatibility equivalents for iterkeys,
iteritems and itervalues.
Signed-off-by: NCleber Rosa <crosa@redhat.com>

Avocado process module has a routine to drain the std* buffer with a
per-line basis loop. This loop makes the new-line character to be
dropped.

On the other hand, the Python logging system is hard-coded to add a new-line
character at the end of the msg string.

The problem takes place when `process` writes a line which originally
had new-line character at the end but the logging system still adds the
new-line character to that line.

In this patch:
- avocado.utils.process to include the new-line character in the lines
  that originally had one.
- avocado.core.test to use for raw outputs a custom logging FileHandler
  that does not add a new-line character at the end of each line.

Verified with (checking `.../test-results/1-.../[stdout|stderr|output]`):

    $ avocado run --external-runner '/bin/echo' 'foo'

    $ avocado run --external-runner '/bin/echo -n' 'foo'

    $ avocado run --external-runner '/bin/ls' '.'

    $ avocado run --external-runner '/bin/ls' 'nonexistingfile

Reference: https://trello.com/c/dH7FLT75Signed-off-by: NAmador Pahim <apahim@redhat.com>

Windows does not allow `<>:"/\|?*` characters in the filename. Let's
include these in our `astring.string_to_safe_path` method and use it
everywhere we create files.
Signed-off-by: NLukáš Doktor <ldoktor@redhat.com>

The `self.logf` is only used as output location, let's use
`astring.string_to_safe_path` to make sure it can be written. As this
makes sure the name is fs friendly, we don't need any hacks like
`replace(" ", "_")` anymore, which also affects the html plugin.
Signed-off-by: NLukáš Doktor <ldoktor@redhat.com>

The html plugin uses entry-point "avocado_result_html", but the pkg name
is "avocado-framework-plugin-result-html" and that is what
"pkg_resources.require" accepts.

This required few tweaks related to recent changes to html plugin which
produces a single result file instead of multiple files.
Signed-off-by: NLukáš Doktor <ldoktor@redhat.com>

The issue is with ":", not with ";". We changed the wrong character.

This reverts commit 60a65d20.
Signed-off-by: NAmador Pahim <apahim@redhat.com>

The python unittests should be executable even now as a simple tests or
via external runner + contrib script. Let's create a new type and
discover python unittests as first-class test types in Avocado.

As Avocado tests are in fact python unittests, we need to manually skip
these by ignoring tests that are discovered as Avocado tests, but are
marked as not-to-be-executed, which required the additional change to
_find_avocado_tests method.
Signed-off-by: NLukáš Doktor <ldoktor@redhat.com>

The filelock test heavily depends on the storage speed. Worse this
dependency is not linear as we issue 1000 attemps in parallel which is
quite a lot simultaneous writes. On my laptop with ssd vs. in-mem
location formula (time of 100 iterations) + 2e-5*no_players worked the
best.
Signed-off-by: NLukáš Doktor <ldoktor@redhat.com>

The basedir was mainly set correctly, but sometimes not used in some of
the tests. Let's move the `os.chdir` to `setUp` phase where it makes
sense and add new ones where they were not present.

Related to basedir the test_mux was using hardcoded path based on the
method used to execute the test, let's change it to `__file__` which
gives better results.
Signed-off-by: NLukáš Doktor <ldoktor@redhat.com>

Signed-off-by: NCleber Rosa <crosa@redhat.com>

Now that the teardown() is always called, we have to explicitly flag
when it should be skipped. That's the case when someone decorates the
setUp() with a skip decorator or when someone calls self.skip() in
setUp().

Reference: https://trello.com/c/B6DJe0JSSigned-off-by: NAmador Pahim <apahim@redhat.com>

Some tests that rely on output check may also rely on the combined
generated output (stdout+stderr).  The first step is to record that
in the same order it was generated.
Signed-off-by: NCleber Rosa <crosa@redhat.com>

These scripts rely on some Python library path manipulation to be
able to load from the source tree.  But, in fact that are a few
reasons for not having those, including:

 * All functionality in `avocado` relies on plugins, which will not be
   available from the source tree unless a `$ setup.py develop` is
   executed, which automatically makes the "import magic" unnecessary.

 * Most users will end up using a setuptools generated "entry point"
   instead.

For the sake of less code, I propose this (and all other?) magic to be
removed.
Signed-off-by: NCleber Rosa <crosa@redhat.com>