avocado.core.exceptions: Update fail_on_error and rename to fail_on

This patch renames the fail_on_error decorator to fail_on and adds
support to supply list of exceptions. This way the decorator can be used
not only to make tests fail on generic exceptions but also to specify
parts of code with expected failures of given type.
Signed-off-by: NLukáš Doktor <ldoktor@redhat.com>

Signed-off-by: NLucas Meneghel Rodrigues <lmr@redhat.com>

This fixes #628.

Add a couple of simple shell scripts that can be listed
and executed with avocado. Let's update our documentation
to use this new dir and avoid people confused with our
wrappers.
Signed-off-by: NLucas Meneghel Rodrigues <lmr@redhat.com>

Include our newest plugin into the list of functional tests.
Signed-off-by: NLucas Meneghel Rodrigues <lmr@redhat.com>

Give to test writers the fail_on_error decorator, which
should turn any generic exception into an avocado
TestFail.

This is important for some tests that can raise
any exception during a complex operation, and any of those
exceptions can be considered a test failure.
Signed-off-by: NLucas Meneghel Rodrigues <lmr@redhat.com>

Explain that skip() can only be called from inside setUp(),
and tell the expected avocado behavior when writers don't
follow that rule.
Signed-off-by: NLucas Meneghel Rodrigues <lmr@redhat.com>

Make sure avocado complains loudly when people try to call
skip() from outside setUp() - this shouldn't be allowed in
avocado tests. Let the user know of the problem and ask them
to fix their tests. Also, update the docstring for skip(),
which will reduce the chances of test writers using it in
inappropriate places.
Signed-off-by: NLucas Meneghel Rodrigues <lmr@redhat.com>

Have the clear definition of each test status in the
ReferenceGuide.
Signed-off-by: NLucas Meneghel Rodrigues <lmr@redhat.com>

In order to keep things simple and clear, enforce the
following test results rules:

 * `PASS`: the test passed, which means all conditions
   being tested have passed.
 * `FAIL`: the test failed, which means at least one
   condition being tested has failed. Ideally, it should
   mean a problem in the software being tested has been found.
 * `ERROR`: an error happened during the test execution.
   This can happen, for example, if there's a bug in the test
   runner, in its libraries or if a resource breaks unexpectedly.
   Uncaught exceptions in the test code will also result in this
   status.
 * `SKIP`: the test runner decided a requested test should not
   be executed. This can happen, for example, due to missing
   requirements in the test environment or when there's a job
   timeout.

In order to do that, let's not make the behavior on exceptions
other than TestFail to be configurable. Therefore, the config
value uncaught_exception_result must be removed and the status
of the test always be 'ERROR'.
Signed-off-by: NLucas Meneghel Rodrigues <lmr@redhat.com>

Signed-off-by: NLucas Meneghel Rodrigues <lmr@redhat.com>

This was added to the Travis CI job, so it doesn't need
to be explicitly executed. Let's drop it.
Signed-off-by: NLucas Meneghel Rodrigues <lmr@redhat.com>

Signed-off-by: NLucas Meneghel Rodrigues <lmr@redhat.com>

subprocess.check_output is a new API starting in python 2.7.
So let's use our in house subprocess lib here to make it
compatible with 2.6.
Signed-off-by: NLucas Meneghel Rodrigues <lmr@redhat.com>

So that we can do cyclical deps checking in the avocado source.
Signed-off-by: NLucas Meneghel Rodrigues <lmr@redhat.com>

Add a couple new tests to release.json:

1) Test that avocado-virt can assign values to the
   command line correctly.
2) Run the avocado migration test, as it's a fairly
   complex test that exercises a lot of functionality
   in avocado-virt.
Signed-off-by: NLucas Meneghel Rodrigues <lmr@redhat.com>

Handle Ctrl+C while executing the test plan more cleanly.
Also, make each code path executed by app.run() return
either 0 on success or 1 on failure, so that we get
proper exit codes for the script.
Signed-off-by: NLucas Meneghel Rodrigues <lmr@redhat.com>

Execution of tests remotely copies tests unnecessarily

avocado.core.plugins: Rename --env and improve --mux-path desc [v2]

--env injects node/key/values into the multiplexed tree. It's 100%
related to mux therefor --mux-inject would better reflect the
functionality.

The --mux-path description was rather misleading, hopefully this will
help users to get better notion (although reading the documentation is
still preferred).
Signed-off-by: NLukáš Doktor <ldoktor@redhat.com>

run_suite() actually happens earlier, and is only called once, so it seems
to be a better place for the check of Avocado on the remote system. If this
condition holds true, then copy the requested tests to the remote system.
Signed-off-by: NCleber Rosa <crosa@redhat.com>

The result.setup() method is now only used to setup the communication
channel, and the copy of the tests will be made conditional, based on
the presence of Avocado on the remote machine.

On the unittest side, I'm quite adverse to heavy mocking, so I favored
removing all these painful, and IMHO of little value checks.
Signed-off-by: NCleber Rosa <crosa@redhat.com>

Instead of copying the entire test directory, let's only copy the
selected test and it's data directory.
Signed-off-by: NCleber Rosa <crosa@redhat.com>

This method should really return earlier if wasn't asked to copy file
to remote side, that is, there's no need to create that directory in
those cases.
Signed-off-by: NCleber Rosa <crosa@redhat.com>

Usability fixes on "$ avocado run --help"

This is a collection of little small touches that, IMHO, add a little
bit more consistency to the plugins involved in the Avocado `run`
command.
Signed-off-by: NCleber Rosa <crosa@redhat.com>

The command line options and documentations refer to "process",
"applications", etc. It's more natural to think and describe them as
"executables" instead.
Signed-off-by: NCleber Rosa <crosa@redhat.com>

The command line options and documentations refer to "binaries" a
lot, with options descriptions such as BINARY_PATH. It's more
natural to think and describe them as "executables" instead.
Signed-off-by: NCleber Rosa <crosa@redhat.com>

API docs: generate separate sections for the 3 modules boundaries

Avocado currently has 3 very distinct namespaces boundaries: test
API, utils, and core. Let's create 3 separate sections so that
users know which one(s) they could use for a given task.
Signed-off-by: NCleber Rosa <crosa@redhat.com>

The introduction was meant to be kept shorter, without the bullet
point items, because proper descriptions are in the inner sections.
Signed-off-by: NCleber Rosa <crosa@redhat.com>

Documentation Review (4th round, v2)

Whenever we use remoter.Remote(), our wrapper to fabric
inside tests, we end up losing information that could
be used for debugging purposes. Let's introduce an
remote related log to the avocado test results dir
(remote.log), that will be populated if
remoter.Remote() is used.

With this patch, tests that use remoter.Remote()
(avocado-virt) will have better debugging information
available.
Signed-off-by: NLucas Meneghel Rodrigues <lmr@redhat.com>

In order to troubleshoot more effectively problems
happening in remote/vm operations, make all output
coming from fabric and paramiko to a 'remote.log'
file in the top level job directory.
Signed-off-by: NLucas Meneghel Rodrigues <lmr@redhat.com>

In order to let multiple loggers to get the same information,
let the LoggingFile object to accept a list of loggers, rather
than a single one.
Signed-off-by: NLucas Meneghel Rodrigues <lmr@redhat.com>