This patch fixes and improves resolving of the test aliases (paths
relative to `data_dir.get_tests_dir`).

1. It removes the magic `.py` suffix, which prevented non-python files
   from being resolved (simple.sh was resolved to $tests/simple.sh.py)
2. It adds the feature to filter the class/method from the test
   reference by parsing $test_reference:$class.$method for implicit
   resolver. (previously this worked only for absolute path resolver)

Visible changes are:

* `avocado run passtest` => fails (used to work)
* `avocado run passtest.py` => works (did not work)
* `avocado run env_variables.sh` => works (did not work)
* `avocado run passtest.py:PassTest.test` => works (did not work)
Signed-off-by: NLukáš Doktor <ldoktor@redhat.com>

This release introduces a new versioning scheme, so the "big bump"
from 0.34.0 to 35.0 is not such a big bump.

Some additional changes with regards to versioning were added, such as
the way the remote runner checks for Avocado on the remote host.
Signed-off-by: NCleber Rosa <crosa@redhat.com>

Not all contrib content is created equal, so let's just move scripts
to a properly named directory, in anticipation for different types of
content.
Signed-off-by: NCleber Rosa <crosa@redhat.com>

This script introduces a pair of directories that will be searched
for executable files, to be run before and after the actual execution
of a job.
Signed-off-by: NCleber Rosa <crosa@redhat.com>

This commit implements the serialized test ids described in the
Introduce proper test IDs RFC.

https://www.redhat.com/archives/avocado-devel/2016-March/msg00024.html

It implements `TestName` class, which contains the test uid, test name
and the variant uid and allows querying for the file-system-friendly
name.

The workflow is:

1. tests are discovered, name is translated to "Test Name" by loader
2. test_suite is passed to the runner (new) along with the number of
   tests+variants to be executed (used to get number of digits)
3. the Mux (params generator) yields the template + (new) variant id
4. the runner replaces template['name'] to TestName(uid, test_name,
   variant_id); where uid is currently no executed tests, test_name is
   the original name from Loader and variant_id is either None or the
   variant index.

This commit makes the `tag` argument unused. To avoid problems a warning
is issued on it's usage so we can remove it in the upcoming releases.
Signed-off-by: NLukáš Doktor <ldoktor@redhat.com>

Add support for identity files as valid SSH options
to our remote plugin. The use case for this is connecting
to Amazon EC2 instances using the provided keypair (.pem
files).

avocado run --remote-hostname my-machine.com --remote-username myuser --remote-key-file /path/to/myfile.pem passtest
Signed-off-by: NLucas Meneghel Rodrigues <lookkas@gmail.com>

When Avocado runs tests on remote machines it always accepts the
host SSH key fingerprint. This could allow credentials to be stolen
if the remote host name or address is spoofed.

This patch adds to Avocado the configuration option to enable/disable
the reject_unknown_hosts option, so user can choose between safe or
flexible.

Notice there is an issue in Paramiko where it does not recognizes
ecdsa keys in known_hosts file. So for reject_unknown_hosts to work,
we currently have to accept the remote host key fingerprint in RSA format.
To do so, the SSH command shouled be like below:

  $ ssh -oHostKeyAlgorithms='ssh-rsa' <host>

Reference: https://trello.com/c/oqyAvZBqSigned-off-by: NAmador Pahim <apahim@redhat.com>

When the total number of tests is lower than number of executed tests,
avocado reports incorrect number of tests. Let's increase the number of
tests in such case.
Signed-off-by: NLukáš Doktor <ldoktor@redhat.com>

This selftest makes sure test statuses are assessed correctly on the
test end inside jsonresult (which also verifies TestResult handling).

Additionally it makes sure jsonresult handles unknown statuses and
missing values in the status correctly.
Signed-off-by: NLukáš Doktor <ldoktor@redhat.com>

Signed-off-by: NCleber Rosa <crosa@redhat.com>

We're substituting `os.path` with flexmock, let's not forget to clean-up
properly.
Signed-off-by: NLukáš Doktor <ldoktor@redhat.com>

only the basename of the `self.filename` has to be shorter than
maxsize-5, not the full path.

Also the IOError in selftest is rather misleading as it relates to
missing source file. Let's just check the path is correctly set and
additionally verify `_record_reference_*` is not crashing when filename
not associated.
Signed-off-by: NLukáš Doktor <ldoktor@redhat.com>

Currently, exit codes cover all situations with one single value.

This patch makes exit codes to be ORable, so we can better represent
the variety of things that happens during an Avocado execution.

Reference: https://trello.com/c/SU5fixgHSigned-off-by: NAmador Pahim <apahim@redhat.com>

The xunit plugin claims to be compatible with jenkins-ci junit-4.xsd,
but it does not pass on the validation dur to the wrong key name 'skip'.

This patch changes the key name to 'skipped' and adds a unittest to validate
the xunit output against the junit-4.xsd schema.

The junit-4.xsd file is coming from:
https://svn.jenkins-ci.org/trunk/hudson/dtkit/dtkit-format/dtkit-junit-model/src/main/resources/com/thalesgroup/dtkit/junit/model/xsd/junit-4.xsd

Reference: https://trello.com/c/ZdEb0TblSigned-off-by: NAmador Pahim <apahim@redhat.com>

When an invalid python code is present in a test class, avocado crashes
badly. This patch adds a TestLoaderError class to be used in those cases,
so we use it to fake the invalid test class, fail the test with ERROR estatus
and report the exception in job log.

Reference: https://trello.com/c/3zDIjTuYSigned-off-by: NAmador Pahim <apahim@redhat.com>

The leading "." makes files invisible on unix systems, which might be
confusing to users. Let's replace with "_".
Signed-off-by: NLukáš Doktor <ldoktor@redhat.com>

This adds selftests regarding the long filenames fixes.
Signed-off-by: NLukáš Doktor <ldoktor@redhat.com>

When avocado fails early in the process (for example in
_setup_environment_variables) it crashes, because the test status is
None. Other example could be user-defined status, which is not inside
user-facing-statuses.

This patch modifies the "status" to "ERROR" when test reports
unsupported status and sets related values to provide details regarding
this failure.
Signed-off-by: NLukáš Doktor <ldoktor@redhat.com>

When a job is timed out during a test execution, we fail the test
and put status ERROR in the test. The job then exits with the rc
AVOCADO_TESTS_FAIL. This patch fixes this, making the test status
INTERRUPTED and the job to exit with AVOCADO_JOB_INTERRUPTED.

Also, when a job is timed out before a test, the test is skipped
and the job exits with rc AVOCADO_ALL_OK. For that case, this patch
makes the job to exit with AVOCADO_JOB_INTERRUPTED instead, keeping
the test status as SKIP.

Given this change, now we have the following combinations of test status
and job return code:

Case1:
    - Test1: PASS
    - Test2: SKIP (TestTimeoutSkip)

    Job RC: AVOCADO_JOB_INTERRUPTED

Case2:
    - Test1: PASS
    - Test2: INTERRUPTED (TestTimeoutInterrupted)
    - Test3: SKIP (TestTimeoutSkip)

    Job RC: AVOCADO_JOB_INTERRUPTED

Case3:
    - Test1: PASS
    - Test2: FAIL
    - Test3: INTERRUPTED (TestTimeoutInterrupted)
    - Test4: SKIP (TestTimeoutSkip)

    Job RC: AVOCADO_JOB_INTERRUPTED
Signed-off-by: NAmador Pahim <apahim@redhat.com>

People tend to mix unicode, strings and encoded strings. This patch adds
"avocado.utils.astring.string_safe_encode" function, which tries to
allow all of the listed inputs and it assumes "utf-8" is in use for
encoded strings.

This function is then used in the tabular_output to avoid
"UnicodeDecodeErrors" for all listed input types.
Signed-off-by: NLukáš Doktor <ldoktor@redhat.com>

In case no tests are found we advice users to run `avocado list -V`
followed by list of urls, which does not really work with no-urls given.
This patch adds second message in case the user does not provide urls
(eg. is using the `--vt-config` option).
Signed-off-by: NLukáš Doktor <ldoktor@redhat.com>

The --show stream:level is actually broken as the "name" variable is not
tuple, but only the stream name. The variable was renamed to
"stream_level" instead.
Signed-off-by: NLukáš Doktor <ldoktor@redhat.com>

Xunit/XML CDATA/attributes must not contain non-printable characters.
This patch escapes such characters using hex value.
Signed-off-by: NLukáš Doktor <ldoktor@redhat.com>

A test base on the external runner feature depends on both the
external runner and the test URL. These two are always combined to
generate the complete command that will be executed by Avocado.

But, the Avocado command line application does not require URLs to be
given on the command line, because these may come from other sources
on loaders other than the basic FileLoader.

So, let's also explicitly require a URL to be given to to the External
Runner loader to return a test factory (class and parameters).
Signed-off-by: NCleber Rosa <crosa@redhat.com>

And deprecate the --multiplex-files option, which was actually
the complete name of the option, but was not being used formally
by our documentation.
Signed-off-by: NCleber Rosa <crosa@redhat.com>

This gives the streams behaviors that were discussed and defined a
formal set of tests.
Signed-off-by: NCleber Rosa <crosa@redhat.com>
Signed-off-by: NLukáš Doktor <ldoktor@redhat.com>

While working on the avocado logging system we thought it'd be
consistent and better to log certain streams to sys.stderr rather than
sys.stdout. It turned out to be a bit confusing and unexpected, so after
a discussion this commit adjusts the streams to:

 * avocado.app - loglevel <= info goes to stdout
               - loglevel > info goes to stderr
 * other streams - go always into stdout
 * sys.stdout - goes to sys.stdout when enabled
 * sys.stderr - goes to sys.stderr when enabled
              - goes to sys.stdout when enabled as test output
Signed-off-by: NLukáš Doktor <ldoktor@redhat.com>

Given the last CI jobs, RunnerSimpleTest:test_kill_stopped_sleep() has
been failing often. The obvious reason, given the nature of the test
and the heavily shared and loaded computing resources they run on, is
the lack of resources.

One example of such as failure:

   ======================================================================
   FAIL: test_kill_stopped_sleep (selftests.functional.test_basic.RunnerSimpleTest)
   ----------------------------------------------------------------------
   Traceback (most recent call last):
     File "/home/travis/build/avocado-framework/avocado/selftests/functional/test_basic.py", line 549, in test_kill_stopped_sleep
       % proc.get_output())
   AssertionError: Avocado process still alive 1s after job-timeout:
   JOB ID     : 5c65a06b4ab15e69d6c4a1fc762368e0e99cbc1f
   JOB LOG    : /tmp/avocado_selftests.functional.test_basicHIyhV4/job-2016-03-09T15.08-5c65a06/job.log
   TESTS      : 1
    (1/1) /bin/sleep 60:
   ctrl+z pressed, stopping test (5046)
   ERROR

So, let's give 4 extra seconds for the process to be
terminated. Hopefully failures in the timeout handling will still be
revealed given the relatively small buffer.
Signed-off-by: NCleber Rosa <crosa@redhat.com>

Record the args and reload loader options on job replay.

Reference: https://trello.com/c/N3UCnR05Signed-off-by: NAmador Pahim <apahim@redhat.com>

The configurations related to the "data_dir" are currently read at
module load time. This means that changes in the configuration between
module load time and changes to the settings (such as parsing an
additional config file) is never seen.

Since the avocado command line application parses extra configuration
files (given with `--config`) after the module is loaded, these extra
configuration files are never applied to the data_dir configuration.

There's one more issue with regards to the the settings usage: because
once references to the `settings` instance singleton are grabbed, they
will always point to the same settings objects. The data_dir unittests
exercise changes to the settings objects by replacing that
instance. So, let's refer to the settings using the full location
(module.attribute), which will always give back the "current" (be it
original or replaced) settings instance.
Signed-off-by: NCleber Rosa <crosa@redhat.com>

The current implementation of _get_bogus_settings() solves a partial need
of the setUp() method. Let's combine both into a single (private) utility
method (_get_temporary_dirs_mapping_and_config()) that can be re-used.

The major goal is to re-use it when testing how data_dir reacts to changes
in the settings at run time.
Signed-off-by: NCleber Rosa <crosa@redhat.com>

This check "avocado -v" produces correct output and returns 0.
Signed-off-by: NLukáš Doktor <ldoktor@redhat.com>

The `--silent` argument is valid for all commands, let's add it to
the core avocado args. For backward compatibility keep supporting it in
"run" subcommand too.
Signed-off-by: NLukáš Doktor <ldoktor@redhat.com>

The "View" concept was developed to abstract the messages and events.
The problem is it was only abused to pass the status to other places, to
colorize the messages and to allow creating paginated view.

There already is a nice class hooked to all important events, the
"TestResult".

For (not only human readable) messages this commit uses the standard
python logging as it's pretty well known, widely used and very scalable.
The colored output is handled by already existing
"ProgressStreamHandler", which maps: DEBUG,INFO,WARNING and >=ERROR
messages to previously existing: minor, message, warning and error event
types.

The paginator was unified and is initialized during logging reconfigure.
During reconfigure all previously logged messages are re-logged into the
output so one does not lose those messages. Another great difference is
that the Paginator is cleaned at exit by avocado and does not require
complex handling to avoid broken console. To use paginator one just
enables it in "args" and writes to any available stream/stdout/stderr
Signed-off-by: NLukáš Doktor <ldoktor@redhat.com>

Both "stream" and "args" are present in the job. Let's only pass
the "job" object and let the Result class to take the needed objects.

Additionally allow overriding the output for json/xunit/html in order
to allow users to specify different value than the one from args.
Signed-off-by: NLukáš Doktor <ldoktor@redhat.com>

Find for files in multiple locations, caching it when successfully fetched.
Available as fetch_asset() method in avocado.Test().

Reference: https://trello.com/c/KTeMIx0uSigned-off-by: NAmador Pahim <apahim@redhat.com>

All *SkipTests are now inherited from "SkipTest" baseclass. This
baseclass contains improved __init__ which removes any unsupported
parameters from the Test.__init__ super call.

This is because those classes are used to substitute other avocado test
classes, for example "avocado-vt" tests which contain "vt_params"
argument unsupported by Test.__init__.
Signed-off-by: NLukáš Doktor <ldoktor@redhat.com>

The tree/multiplex modules have clear conditionals when the system
is not multiplex capable, namely because of the lack of YAML libraries.

Let's reflect that on their tests.
Signed-off-by: NCleber Rosa <crosa@redhat.com>

This commit changes the idiom on (hopefully all) except statements,
from:
   except foo, details:

to:
   except foo as details:

Which is compatible with both Python 2 and 3.
Signed-off-by: NCleber Rosa <crosa@redhat.com>

This change introduces symbolic names for various filesystem
permission modes, instead of the literal octals with which
Python 3 shows no compassion.
Signed-off-by: NCleber Rosa <crosa@redhat.com>