docs: add logging capabilities docs to test writers [v2]

Let's give a real example on how to use logging streams when writing
tests, and use the runner options that allow for logging streams to
be captured into log files.
Signed-off-by: NCleber Rosa <crosa@redhat.com>

External Runner: deal with empty test URL

A test base on the external runner feature depends on both the
external runner and the test URL. These two are always combined to
generate the complete command that will be executed by Avocado.

But, the Avocado command line application does not require URLs to be
given on the command line, because these may come from other sources
on loaders other than the basic FileLoader.

So, let's also explicitly require a URL to be given to to the External
Runner loader to return a test factory (class and parameters).
Signed-off-by: NCleber Rosa <crosa@redhat.com>

Command line argument parser: disable abbreviations [v2]

To avoid ambiguity, and allow for precise argument names and this
command line examples, let's disable abbreviations. One example where
this led to confusion:

 $ avocado run --show test passtest

Instead of

 $ avocado --show test run passtest

With this change, the first command line would result in:

 avocado run: error: unrecognized arguments: --show

Which is much more precise than:

 Unable to discover url(s) 'test' with loader plugins(s) 'file',
 'external', try running 'avocado list -V test' to see the details.

Disabling the arguments abbreviation without depending on the fix
applied to Python itself is suggested here:

 http://bugs.python.org/msg204678

Reference: https://trello.com/c/Z3lWiJp0Signed-off-by: NCleber Rosa <crosa@redhat.com>

And deprecate the --multiplex-files option, which was actually
the complete name of the option, but was not being used formally
by our documentation.
Signed-off-by: NCleber Rosa <crosa@redhat.com>

examples.testplans: Add tty cleanup logging selftest

Handling tty in travis is quite problematic, this adds a manual selftest
for paginator issue where tty was left in a different state than it was
before avocado execution.
Signed-off-by: NLukáš Doktor <ldoktor@redhat.com>

This patch adds "Logging system" chapter to the documentation describing
the existing and recent features related to logging.
Signed-off-by: NLukáš Doktor <ldoktor@redhat.com>

The `--store-logging-stream` logically belongs to output an results
instead of the generic section.
Signed-off-by: NLukáš Doktor <ldoktor@redhat.com>

This gives the streams behaviors that were discussed and defined a
formal set of tests.
Signed-off-by: NCleber Rosa <crosa@redhat.com>
Signed-off-by: NLukáš Doktor <ldoktor@redhat.com>

There was a hack to enable default streams when avocado failed to parse
arguments. This was necessary as --silent used to disable all output and
the user would have no idea of what went wrong. This behavior changed
and we always enable stderr leaving the user at least something to start
with. So instead of trying to be smart let's not surprise our users and
be consistent all the time.
Signed-off-by: NLukáš Doktor <ldoktor@redhat.com>

Also in the command line arguments help messages.
Signed-off-by: NCleber Rosa <crosa@redhat.com>

When a user enables the early stream, it also enables the "test"
stream, so let's tell the user that.
Signed-off-by: NCleber Rosa <crosa@redhat.com>

Currently the custom argparse.error writes everything directly to
stderr. This patch replaces the direct write to suitable logging stream.

The bigger change is it splits the message and outputs the "--help"
output to stdout and puts only the error message into the stderr.

This way the output is more consistent and on colored term more
readable.
Signed-off-by: NLukáš Doktor <ldoktor@redhat.com>

The built-in "all" and "none" streams are not handled by the parser
anymore, add the handling into the "output.reconfigure".
Signed-off-by: NLukáš Doktor <ldoktor@redhat.com>

PEP8 recommends to import global variables from other modules rather
than re-using them. This patch replaces the `output.STD_OUTPUT` one and
removes the unnecessary wrapper around `STD_OUTPUT.close()`.
Signed-off-by: NLukáš Doktor <ldoktor@redhat.com>

While working on the avocado logging system we thought it'd be
consistent and better to log certain streams to sys.stderr rather than
sys.stdout. It turned out to be a bit confusing and unexpected, so after
a discussion this commit adjusts the streams to:

 * avocado.app - loglevel <= info goes to stdout
               - loglevel > info goes to stderr
 * other streams - go always into stdout
 * sys.stdout - goes to sys.stdout when enabled
 * sys.stderr - goes to sys.stderr when enabled
              - goes to sys.stdout when enabled as test output
Signed-off-by: NLukáš Doktor <ldoktor@redhat.com>

We decided, that the "--silent" mode should keep the stderr enabled.
This commit adjusts the code accordingly.
Signed-off-by: NLukáš Doktor <ldoktor@redhat.com>

Currently we use global variables and store combined stdout/stderr
streams into one in-memory object. This suits for ordering purposes, but
does not keep the information of what stream was the message written to.
This commit creates a wrapping class, which uses the shared storage
(list) to store messages altogether with the stream they were used with.

Additionally this class provides some basic handling of streams and we
can avoid using global variables.
Signed-off-by: NLukáš Doktor <ldoktor@redhat.com>

No actual changes, just couple of docstring fixes.
Signed-off-by: NLukáš Doktor <ldoktor@redhat.com>

We must reflect the environment variables later in code, therefor we
need to reorder the way we discover what streams are enabled.
Additionally don't return early on `--silent` as env-variable might
already enable some streams.
Signed-off-by: NLukáš Doktor <ldoktor@redhat.com>

The advised names for top level variables is UPPER_CASE. Let's respect
that. Also the TERM_SUPPORT could be useful for logging initialization,
therefor this commit moves it above it.
Signed-off-by: NLukáš Doktor <ldoktor@redhat.com>

Fabric automatically echoes all text typed into the terminal back
out to the user. This makes avocado treat the echoed strings as
fabric return.

This patch disables terminal echo in fabric.

Reference: https://trello.com/c/xqN7bq6NSigned-off-by: NAmador Pahim <apahim@redhat.com>

Test: increment to tagged name not applied on "unsafe" names

Depending on the name of the test, it may need a "safe" tagged name,
that is, one that can be turned into a filesystem path. Most, if not all,
cases apply to SIMPLE tests, that have slashes in the test names, such
as "/bin/true".

The test class code needs to check if the "safe" test name directory
exists, and apply the incremental numeric tag to it.

Also, an explicit check that raises a test setup failure was added, so
that Avocado won't silently reuse an existing test log dir.

Reference: https://trello.com/c/nPrrRv5NSigned-off-by: NCleber Rosa <crosa@redhat.com>

selftests/functional/test_basic.py: extend timeout on test_kill_stopp…

Given the last CI jobs, RunnerSimpleTest:test_kill_stopped_sleep() has
been failing often. The obvious reason, given the nature of the test
and the heavily shared and loaded computing resources they run on, is
the lack of resources.

One example of such as failure:

   ======================================================================
   FAIL: test_kill_stopped_sleep (selftests.functional.test_basic.RunnerSimpleTest)
   ----------------------------------------------------------------------
   Traceback (most recent call last):
     File "/home/travis/build/avocado-framework/avocado/selftests/functional/test_basic.py", line 549, in test_kill_stopped_sleep
       % proc.get_output())
   AssertionError: Avocado process still alive 1s after job-timeout:
   JOB ID     : 5c65a06b4ab15e69d6c4a1fc762368e0e99cbc1f
   JOB LOG    : /tmp/avocado_selftests.functional.test_basicHIyhV4/job-2016-03-09T15.08-5c65a06/job.log
   TESTS      : 1
    (1/1) /bin/sleep 60:
   ctrl+z pressed, stopping test (5046)
   ERROR

So, let's give 4 extra seconds for the process to be
terminated. Hopefully failures in the timeout handling will still be
revealed given the relatively small buffer.
Signed-off-by: NCleber Rosa <crosa@redhat.com>

Trying to follow the docs to install avocado from source,
make requirements command fails doe to the missing
dependencies.

This patch adds the two missing dependencies I had to install
in addition to the listed in docs.

Reference: https://trello.com/c/1ryTxsXwSigned-off-by: NAmador Pahim <apahim@redhat.com>

Record the args and reload loader options on job replay.

Reference: https://trello.com/c/N3UCnR05Signed-off-by: NAmador Pahim <apahim@redhat.com>

data_dir: make settings dynamic

The configurations related to the "data_dir" are currently read at
module load time. This means that changes in the configuration between
module load time and changes to the settings (such as parsing an
additional config file) is never seen.

Since the avocado command line application parses extra configuration
files (given with `--config`) after the module is loaded, these extra
configuration files are never applied to the data_dir configuration.

There's one more issue with regards to the the settings usage: because
once references to the `settings` instance singleton are grabbed, they
will always point to the same settings objects. The data_dir unittests
exercise changes to the settings objects by replacing that
instance. So, let's refer to the settings using the full location
(module.attribute), which will always give back the "current" (be it
original or replaced) settings instance.
Signed-off-by: NCleber Rosa <crosa@redhat.com>

The current implementation of _get_bogus_settings() solves a partial need
of the setUp() method. Let's combine both into a single (private) utility
method (_get_temporary_dirs_mapping_and_config()) that can be re-used.

The major goal is to re-use it when testing how data_dir reacts to changes
in the settings at run time.
Signed-off-by: NCleber Rosa <crosa@redhat.com>

We do use default to None everywhere. external-runner-chdir seems to be the only
exception, being defaults to 'off'. This was harmless until we need to record
external-runner-chedir to further retrieve it in replay jobs.

This patch changes external-runner-chdir default from 'off' to None.
Signed-off-by: NAmador Pahim <apahim@redhat.com>