Signed-off-by: NCleber Rosa <crosa@redhat.com>

This is an attempt to catch most (if not all) usages of open
that do not follow the context manager pattern.

Sometimes, for better readability, our own `genio` library is
used.
Signed-off-by: NCleber Rosa <crosa@redhat.com>

The avocado.utils.process APIs offer the `allow_output_check`
parameter which exists at a function call level, and thus should be
applied only during the scope of that function.

While it's questionable if a test process should be able to skip
some of its generated output (by means of an `avocado.utils.process`
function parameter), the fact is that the utility libraries offer
that at a lower level.
Signed-off-by: NCleber Rosa <crosa@redhat.com>

In commit 4514a8ad, the default value from the command line option
`--output-check-record` was being used to set the mode of the
`avocado.utils.process` APIs with regards to output record mode.

The problem with that is it makes it impossible for the API to
distinguish between "no explicit behavior requested" and "explicitly
being asked to disable recording".
Signed-off-by: NCleber Rosa <crosa@redhat.com>

It looks like Travis does some different handling of process, and
we need to manually force the writes to the stdout.
Signed-off-by: NCleber Rosa <crosa@redhat.com>

And the support for combined mode.  So far, there was no need to
determine how processes would be run, and how their output would be
collected: the stdout and stderr would always be collcted separately.

Now, with the need to support the "combined" mode, the
`--output-check-record` command line switch not only has to control
which files are populated (stdout.expected, stderr.expected) but also
it needs to control the execution of process, because it's what will
generate the "raw material" for a possible "output.expected" file.
Signed-off-by: NCleber Rosa <crosa@redhat.com>

The 4235b329 changed the
stdout/stderr/output logging handlers to not to add tailing '\n's, but
these are processed by LoggingFile, which is designed to remove the
'\n's and to logged them per line. As a result the unittest was
incorrectly changed as the first outputs actually add the '\n' which was
removed in the expected output.

This patch keeps the raw approach as it is necessary for the last
buffered line and instead of buffering puts all lines including the
tailing '\n' and not '\n' for the remaining last line (if it's
available). Consequently this should improve the situation where output
is being produced by stderr and stdout concurrently as no buffering of
the non-tailing-newline output is done on LoggingFile level.
Signed-off-by: NLukáš Doktor <ldoktor@redhat.com>

Avocado process module has a routine to drain the std* buffer with a
per-line basis loop. This loop makes the new-line character to be
dropped.

On the other hand, the Python logging system is hard-coded to add a new-line
character at the end of the msg string.

The problem takes place when `process` writes a line which originally
had new-line character at the end but the logging system still adds the
new-line character to that line.

In this patch:
- avocado.utils.process to include the new-line character in the lines
  that originally had one.
- avocado.core.test to use for raw outputs a custom logging FileHandler
  that does not add a new-line character at the end of each line.

Verified with (checking `.../test-results/1-.../[stdout|stderr|output]`):

    $ avocado run --external-runner '/bin/echo' 'foo'

    $ avocado run --external-runner '/bin/echo -n' 'foo'

    $ avocado run --external-runner '/bin/ls' '.'

    $ avocado run --external-runner '/bin/ls' 'nonexistingfile

Reference: https://trello.com/c/dH7FLT75Signed-off-by: NAmador Pahim <apahim@redhat.com>

The html plugin uses entry-point "avocado_result_html", but the pkg name
is "avocado-framework-plugin-result-html" and that is what
"pkg_resources.require" accepts.

This required few tweaks related to recent changes to html plugin which
produces a single result file instead of multiple files.
Signed-off-by: NLukáš Doktor <ldoktor@redhat.com>

The basedir was mainly set correctly, but sometimes not used in some of
the tests. Let's move the `os.chdir` to `setUp` phase where it makes
sense and add new ones where they were not present.

Related to basedir the test_mux was using hardcoded path based on the
method used to execute the test, let's change it to `__file__` which
gives better results.
Signed-off-by: NLukáš Doktor <ldoktor@redhat.com>

Some tests that rely on output check may also rely on the combined
generated output (stdout+stderr).  The first step is to record that
in the same order it was generated.
Signed-off-by: NCleber Rosa <crosa@redhat.com>

Writing to sys.stdout|stderr from inside test should be treated as any
other output in test and should be prefixed by [std* ] to emphasize it's
not comming directly from logging, but from direct output. This prefix
should  and is only be prefixed in job.log and not in stdout|stderr
files.
Signed-off-by: NLukáš Doktor <ldoktor@redhat.com>

Attempting to close a journal that wasn't initialized is certainly a
bug.  This will check for its initialization first.
Signed-off-by: NCleber Rosa <crosa@redhat.com>

The UI was showing information about the job, no matter if the job got
to run tests.

Even though the formal available statuses of a job are not too evolved
and documented, Avocado already sets a job status as "PASS" when it
finishes running the tests.  This involves most of the job phases.

Let's use that to check if the UI should print information about the
test results and job execution time.
Signed-off-by: NCleber Rosa <crosa@redhat.com>

When the tests references cannot be resolved to a test_suite, the
results plugins should not output anything.

Reference: https://trello.com/c/oMQsQHC6Signed-off-by: NAmador Pahim <apahim@redhat.com>

Currently the `selftests/run_coverage` only reports unit coverage. This
patch allows specifying custom `avocado` command in selftests and uses
it to run coverage to also include the functional tests to results.
Signed-off-by: NLukáš Doktor <ldoktor@redhat.com>

This is a simple collection of syntatic fixes that work across
Python 2.7 and 3.x.
Signed-off-by: NCleber Rosa <crosa@redhat.com>

It is possible to set relative paths in settings, but once we evaluate
them we should turn them into absolute paths to avoid incorrect
representation on `os.chdir` (or so)
Signed-off-by: NLukáš Doktor <ldoktor@redhat.com>

The job has the number of tests, but not the total of test executions.
This information comes from the result attribute 'tests_total', updated
by the runner.

This patch changes the tap plugin so it can use the correct information.

Reference: https://trello.com/c/tiusKNNjSigned-off-by: NAmador Pahim <apahim@redhat.com>

The unittest standard library on Python 2.7 an later has everything
that the unittest2 backport is supposed to have.  Let's then drop all
the conditional imports of unittest2 and stick with unittest.
Signed-off-by: NCleber Rosa <crosa@redhat.com>

To provide support for passing arguments for simple tests in command
line, we were creating some cases where the test reference could not be
properly handled. Also, for test references with white-spaces, we have
an inconsistent behavior. See reference.

This patch removes the support for arguments in test references,
restoring the proper behavior, which is expected to be shell-like.

As consequence, we now handle correctly white-spaces in tests references
as well as Unicode characters.

Reference: https://www.redhat.com/archives/avocado-devel/2016-November/msg00011.html
Reference: https://trello.com/c/bfE9NBblSigned-off-by: NAmador Pahim <apahim@redhat.com>

Default bootstrap library contains a custom font we don't really use.
Let's generate a version without it from:

    http://getbootstrap.com/customize/

Note I only removed the font, it still can be minimized...
Signed-off-by: NLukáš Doktor <ldoktor@redhat.com>

We do produce `results.xml` and `results.json`, let's also include
`results.tap` by default and allow disabling this by `--tap-job-result`.

Note it's impossible to initialize the default result.tap in `__init__`
as we only get `args` and not the job itself, therefor this patch adds
it as first thing inside the `pre_tests` step.
Signed-off-by: NLukáš Doktor <ldoktor@redhat.com>

Over the time some non-optimal "avocado run" executions accumulated.
This fixes (hopefully) all of them by adding `--sysinfo` and
`--job-results-dir`.
Signed-off-by: NLukáš Doktor <ldoktor@redhat.com>

When no C compiler is available, these tests will produce errors.
Let's skip them when the requirements cannot be met.
Signed-off-by: NCleber Rosa <crosa@redhat.com>

Currently one of the test_output tests depends on a correct English
message, but the test runs under the system locales. Let's force the
LC_ALL for that execution.
Signed-off-by: NLukáš Doktor <ldoktor@redhat.com>

This step makes the HTML result a proper Python plugin, by living
outside the Avocado package namespace.

The RPM package is also adjusted: it now builds ands install both
Avocado and the HTML result plugin while keeping the two separate
installable packages.

Developers can use the usual `make link` and `make clean` to enable
and disable all plugins (including this one).
Signed-off-by: NCleber Rosa <crosa@redhat.com>

This change replaces the in-job logic to deal with the standard output
with the FileOrStdoutAction that does it at the command line parsing
time.  Then, it deals with a possible command line option conflict (the
first two players that attempt to use stdout) at the application level,
exiting the application in such case.
Signed-off-by: NCleber Rosa <crosa@redhat.com>

By ignoring the exception, and preventing Avocado from crashing.
Since the actions inside the block that may trigger an IOError are the
writes to stdout and stderr, we can be pretty confident that there
will be no adverse effects.

Reference: https://trello.com/c/sztjbAXySigned-off-by: NCleber Rosa <crosa@redhat.com>

One of the most important points of having TAP support is allowing
other tools to parse the results of the Avocado tests.  IMHO the best
way to test that we're producing sound TAP support is to use stable
implementations for that.

AFAIK, Perl is where TAP was born, and its TAP library is kind of a
reference implementation.  Let's use that, if available, to check if
our tap generation is sound.
Signed-off-by: NCleber Rosa <crosa@redhat.com>
Signed-off-by: NLukáš Doktor <ldoktor@redhat.com>

This patch fixes and improves resolving of the test aliases (paths
relative to `data_dir.get_tests_dir`).

1. It removes the magic `.py` suffix, which prevented non-python files
   from being resolved (simple.sh was resolved to $tests/simple.sh.py)
2. It adds the feature to filter the class/method from the test
   reference by parsing $test_reference:$class.$method for implicit
   resolver. (previously this worked only for absolute path resolver)

Visible changes are:

* `avocado run passtest` => fails (used to work)
* `avocado run passtest.py` => works (did not work)
* `avocado run env_variables.sh` => works (did not work)
* `avocado run passtest.py:PassTest.test` => works (did not work)
Signed-off-by: NLukáš Doktor <ldoktor@redhat.com>

Xunit/XML CDATA/attributes must not contain non-printable characters.
This patch escapes such characters using hex value.
Signed-off-by: NLukáš Doktor <ldoktor@redhat.com>

While working on the avocado logging system we thought it'd be
consistent and better to log certain streams to sys.stderr rather than
sys.stdout. It turned out to be a bit confusing and unexpected, so after
a discussion this commit adjusts the streams to:

 * avocado.app - loglevel <= info goes to stdout
               - loglevel > info goes to stderr
 * other streams - go always into stdout
 * sys.stdout - goes to sys.stdout when enabled
 * sys.stderr - goes to sys.stderr when enabled
              - goes to sys.stdout when enabled as test output
Signed-off-by: NLukáš Doktor <ldoktor@redhat.com>

The `--silent` argument is valid for all commands, let's add it to
the core avocado args. For backward compatibility keep supporting it in
"run" subcommand too.
Signed-off-by: NLukáš Doktor <ldoktor@redhat.com>

This patch changes the way logging is initialized in avocado app:

1. stdout/stderr/logging is replaced with in-memory handlers
2. when arg parsing is done, logging is reconfigured accordingly to
   setting and re-logs stored messages (when streams are enabled)
3. when job starts and "test" stream is enabled, "" and "avocado.test"
   streams are enabled
4. when job finishes and "test" stream enabled the additional outputs,
   the additional outputs are disabled again.

The 3 and 4 is necessarily as the "" logger contains all logs including
avocado initialization (stevedore, PIL, ...). This way the "test" stream
still outputs everything, but only during the job-execution and not
during avocado initialization and cleanup. (one can use "early" log to
see those).

Last but not least it keeps "avocado.app" output in stdout (and stderr),
but it moves all other outputs to stderr instead. This is IMO better
handling as one can distinguish between debug and "additional" output.
Signed-off-by: NLukáš Doktor <ldoktor@redhat.com>

It's more common in our code (and pretty much around all Python
tests) to use the TestCase utility methods such as `assertEqual`
instead of doing `assert foo == bar`. Let's remove the few
occurrences of the `assert` idiom for cohesiveness.
Signed-off-by: NCleber Rosa <crosa@redhat.com>

Avocado defines avocado.core.result.TestResult, so that alternative
test result classes can inherit from it. When a user writes a new
test result class, he's left wondering how to activate that.

First, for consistency, Avocado should define proper interfaces
for "test result plugins". This is on sight, and will ultimately
allow users to register new test result implementations simply by
registering the setuptools entry points, just as it's currently
done (on versions >= 0.31.0) for command line related plugins.

Until then, let's at least improve the current state of things.
These changes believe that the following improvements are given here:

* avocado.core.job.Job._set_output_plugins now won't look at every
  possible key in the application arguments, and won't silence
  possible errors while trying to instantiate test result classes.
* test result writers can now use a documented method for registering
  their test result classes.

The same approach was not chosen to be done with the test runner
classes because there can only by one test runner per Avocado job.
Signed-off-by: NCleber Rosa <crosa@redhat.com>

Signed-off-by: NCleber Rosa <crosa@redhat.com>

Signed-off-by: NCleber Rosa <crosa@redhat.com>

Signed-off-by: NCleber Rosa <crosa@redhat.com>