Signed-off-by: NLukáš Doktor <ldoktor@redhat.com>

Let's give a real example on how to use logging streams when writing
tests, and use the runner options that allow for logging streams to
be captured into log files.
Signed-off-by: NCleber Rosa <crosa@redhat.com>

Asset fetcher changed the fallback directory to be the data_dir/cache
isntead of a test temporary directory. This patch reflects in docs
that change.
Signed-off-by: NAmador Pahim <apahim@redhat.com>

Find for files in multiple locations, caching it when successfully fetched.
Available as fetch_asset() method in avocado.Test().

Reference: https://trello.com/c/KTeMIx0uSigned-off-by: NAmador Pahim <apahim@redhat.com>

The get_data_path utility API of the Test class is a convenience,
it simply joins the name of a given file to the datadir location.

IMHO the value is low when compared to the pollution of the namespace
and yet another compatibility/stability promise.
Signed-off-by: NCleber Rosa <crosa@redhat.com>

These two attributes are simply utility logger instances that test
writers can use to write "directly" to the test process' STDOUT and
STDERR.

I agree that having those ready to use inside a test is a nice to
have, but I don't think it justifies the polution to the test class
namespace.

Let's remove those and document how the user can use Python's standard
logging API to retrieve the logger instances that Avocado provides to
tests. This will continue to exist and be supported.
Signed-off-by: NCleber Rosa <crosa@redhat.com>

While going through the dependency list (requirements*.txt files) and
performing our self tests out of virtual environments, I noticed that
some tests are run outside the virtual environments.

The reason is that, even though the virtual environment is activated
for the test session (and say, `which python` gives `/venv/bin/python`),
we have hard coded `/usr/bin/python` in most places.

According to the some discussions on the virtualenv project itself[1],
a quick solution is to revert to the also common `/usr/bin/env python`
way of pointing to the Python interpreter.

[1] - https://github.com/pypa/virtualenv/issues/124Signed-off-by: NCleber Rosa <crosa@redhat.com>

Add a note about safe test loading and the docstring
tags functionality.
Signed-off-by: NLucas Meneghel Rodrigues <lmr@redhat.com>

Signed-off-by: NLucas Meneghel Rodrigues <lmr@redhat.com>

Currently the yaml contents in this page is missing '!mux'
keyword, run as-is will show users errors. Fixing by adding
keyword '!mux' to the yaml content.
Signed-off-by: NSrikanth Aithal <sraithal@linux.vnet.ibm.com>

Change docs to show an output that reflects the current
state of avocado.
Signed-off-by: NLucas Meneghel Rodrigues <lmr@redhat.com>

Instead of putting outputs on separate lines, condensate
test results summary into a single line. The new output
looks like:

$ avocado run passtest
JOB ID     : f2f5060440bd57cba646c1f223ec8c40d03f539b
JOB LOG    : /home/user/avocado/job-results/job-2015-07-27T17.13-f2f5060/job.log
JOB HTML   : /home/user/avocado/job-results/job-2015-07-27T17.13-f2f5060/html/results.html
TESTS      : 1
(1/1) passtest.py:PassTest.test: PASS (0.00 s)
RESULTS    : PASS 1 | ERROR 0 | FAIL 0 | SKIP 0 | WARN 0 | INTERRUPT 0
TIME       : 0.00 s

We updated a few unittests in order to not depend on the
looks of the human output anymore, since unless we are
specifically testing for human output behavior, the unittests
should use machine readable output. Also, the documentation
was updated to reflect the new output layout.
Signed-off-by: NLucas Meneghel Rodrigues <lmr@redhat.com>

Explain that skip() can only be called from inside setUp(),
and tell the expected avocado behavior when writers don't
follow that rule.
Signed-off-by: NLucas Meneghel Rodrigues <lmr@redhat.com>

Avocado currently has 3 very distinct namespaces boundaries: test
API, utils, and core. Let's create 3 separate sections so that
users know which one(s) they could use for a given task.
Signed-off-by: NCleber Rosa <crosa@redhat.com>

This doesn't seem to be the best place for this piece of information.
Let's remove it for now, and it may (or may not) have a place
somewhere else.
Signed-off-by: NCleber Rosa <crosa@redhat.com>

At its core, what's really important to get accross is that an Avocado test
is compatible with unittest.TestCase, and it can be run with compatible
test runners.
Signed-off-by: NCleber Rosa <crosa@redhat.com>

We don't implement all of unittest, but rather inherit from the TestCase class.
Let's also fix the references to the assert* methods.
Signed-off-by: NCleber Rosa <crosa@redhat.com>

The given example was using the old test module/class. Also, for clarity
sake, let's add the "new" import style for the Avocado Test class.
Signed-off-by: NCleber Rosa <crosa@redhat.com>

--mux-entry was a bit confusing as what it does it splits the result
params into multiple slices and while quering for keys they are
evaluated in the specified order, thus behaving the same as Linux PATH
bash variable.
Signed-off-by: NLukáš Doktor <ldoktor@redhat.com>

Pretty obvious typo with an extra "d".
Signed-off-by: NCleber Rosa <crosa@redhat.com>

After reading the text and shuffling the order a bit, I think this
has an easier to follow structure. A few highlights:

* An Avocado test is really a method, that is, for many methods we
  have many tests.
* Suppress the attempt to briefly explain the Multiplexer here,
  giving instead a link to a more complete documentation.

Changes from v1:
* Mention the function naming conventions for the test methods and
  link to PEP 8 relevant section for functions and classes.
* Break into finer grained sections.
Signed-off-by: NCleber Rosa <crosa@redhat.com>

[fixup] docs: simplify Writing Avocado Test section text
Signed-off-by: NCleber Rosa <crosa@redhat.com>

The idea is to remove everything that is not critical to the user's
understanding of the core concepts.

Also, to sync to the new multi-test in a single class capabilities
of Avocado, let's name the main test method `test`. This paves the
way for other (multi test) examples that name test methods with
the exact same prefix (test_foo, test_bar and so on).
Signed-off-by: NCleber Rosa <crosa@redhat.com>

This includes moving the reference text and diagram on test order
resolution to the Reference Guide. Since this also included
information on what are simple and instrumented tests, let's
merge and reuse that.

Also, let's rename "Simple example" to "Basic example" so that it
does not confuse users with the concept of a Simple Test, since this
example is actually an Instrumented Test.
Signed-off-by: NCleber Rosa <crosa@redhat.com>

Since Avocado in our context is a proper name, and not really just
some fruit, let's refer to it using a capitalized word.
Signed-off-by: NCleber Rosa <crosa@redhat.com>

Only avocado.Test is of general interest to test writers. For that reason
the entire test module has been moved to the avocado.core namespace.

The recommended way to go about writing instrumented tests is something like:

   from avocado import Test
   class MyTest(Test):
      def test_foo(self):
         do_stuff()

Changes from v2:
* Fixed references in docstrings
* Fixed refrences of avocado.test in Writing Test Guide docs

Changes from v1:
* Fixed comment typo on commit message
* Fixed old reference of avocado.test (then test.Test) instead of
  avocado then Test in GDB docs.
Signed-off-by: NCleber Rosa <crosa@redhat.com>

This patch changes the default mux-entry to `/run` as it tends to be
more convenient.

The biggest change is that this location is used as default location
for added YAML files. It's also used as the base when using relative
path YAML file injection. The only way to include file into the exact
location is to use absolute path (`/your/location`).

The benefit is that single simple YAML file doesn't require any additional
location and is in the default mux-entry location. Check the documentation
for details.
Signed-off-by: NLukáš Doktor <ldoktor@redhat.com>

We forgot to update all places where the params API
was referenced, leaving us with old constructs in code
samples, such as referring to a params key by
self.params.key.
Signed-off-by: NLucas Meneghel Rodrigues <lmr@redhat.com>

This patch adds documentation of the "new params API" and removes the
compatility layer. From now on only the API with paths is valid:

    self.params.get(key, path='*', default=None)

where the default path '*' matches anything from --mux-entry path.
Default path is `/test`.
Signed-off-by: NLukáš Doktor <ldoktor@redhat.com>

This patch modifies all example tests to use the new API. It uses the
default path, which is `*`, which matches the mux-entry paths ONLY.
In order to get the values it's necessarily to inject the YAML files
into the mux-entry path, by default `/test`, eg:

    avocado run sleeptest -m /test:examples/tests/sleeptest.py.data/
    sleeptest.yaml

It might seen as a bit more complicated to execute, but it's definitely
easier for maintaining and combining existing YAML files. To execute
matrix of tests you simply inject them into 2 different `/test` children,
eg: `-m /test/by_length:lengths.yaml /test/by_method:methods.yaml`.
Signed-off-by: NLukáš Doktor <ldoktor@redhat.com>

Export avocado.core.job.main as avocado.main, so that make it look
like an API for test developers to use.
Signed-off-by: NRudá Moura <rmoura@redhat.com>

Move avocado.job to avocado.core.job, in conformance to our
purpose of keeping things not related to test developers,
inside avocado.core, isolated from users.
Signed-off-by: NRudá Moura <rmoura@redhat.com>

The default UI has changed a bit, so let's update the documentation to
match it.
Signed-off-by: NCleber Rosa <crosa@redhat.com>

Remove words for setup, cleanup and action and update to the new slang:
setUp, tearDown and runTest.
Signed-off-by: NRudá Moura <rmoura@redhat.com>

This patch is initial support for people using custom bash scripts
with avocado. Tests should use:

    PATH=$(avocado "exec-path"):$PATH

and then they can utilize the helpers.

This version contain functions to write to Test.log the same way it's
possible from python including failing the test with TestWarn in case
avocado_warn was used.
Signed-off-by: NLukáš Doktor <ldoktor@redhat.com>

From the current command line `--disable-output-check` replace it
with the new command line `--output-check {on,off}`, defaults to `on`.

Use `--output-check=off` to behave like ``--disable-output-check`.

Also, validate the value from command line `--output-check-record`.
Signed-off-by: NRudá Moura <rmoura@redhat.com>

This patch simplifies the concept of WARN test. In case user uses the
self.log.warn or self.log.warning methods, test is marked as dirty and
in case everything else worked fine it finishes with TestWarn exception.
Signed-off-by: NLukáš Doktor <ldoktor@redhat.com>

Add support to inspect and find tests inside a directory, recursively,
by using the method `discover_url`. It takes a url (path) and
returns the tests parameters it discovers.

The loader object has the .discover() method, that can be used
to inspect the location, and the .validate() method, that will
discover any problems on the user's input.

We are also changing the behavior of avocado to error out on
invalid inputs (missing paths or files that are not avocado
tests). Therefore, the concept of MISSING or NOT_A_TEST both
disappear from the UI. Unittests were updated to reflect the
new status quo.
Signed-off-by: NRudá Moura <rmoura@redhat.com>
Signed-off-by: NLucas Meneghel Rodrigues <lmr@redhat.com>

Explain to the user that the class name for their avocado test
is arbitrary (that's actually a recent change, see 6674048d).
Also advise them to stick with CamelCase, just for style
purposes.
Signed-off-by: NLucas Meneghel Rodrigues <lmr@redhat.com>

Explain to the user how avocado resolves a path and decides
whether it is a test, and what kind of test it is. Use a diagram
to clarify the decision process used.
Signed-off-by: NLucas Meneghel Rodrigues <lmr@redhat.com>

Currently avocado uses /var/tmp/avocado as the default
location for the avocado tmp dir. This is a bug from the
security standpoint, since a malicious user could create
a /var/tmp/avocado symlink pointing to an important directory
of the system.

So, instead of using /var/tmp/avocado, remove the ability
to configure the base tmp directory (other than the traditional
$TMPDIR environment variable), and make the data_dir.get_tmp_dir()
API to return a safe, non predictable temporary directory.

Recommended reading:

https://kurt.seifried.org/2012/03/14/creating-temporary-files-securely/
http://www.linuxsecurity.com/content/view/115462/151/Signed-off-by: NLucas Meneghel Rodrigues <lmr@redhat.com>