There's a better maintained trinity test at:

   https://github.com/avocado-framework-tests/avocado-misc-tests/blob/master/fuzz/trinity.py

Let's drop this outdated example.

Reference: https://github.com/avocado-framework/avocado/issues/2285
Reference: https://trello.com/c/uCcng8dN/1168-drop-trinity-example-testSigned-off-by: NCleber Rosa <crosa@redhat.com>

The suite order means the order in which the variants will be applied to
tests. Right now we run all variants of first test, then the same for
the second, third, ... tests. The new option "--execution-order" allows
to alternatively specify "tests-per-variant" mode where we run first
variant of all tests, then second variant, third, ...
Signed-off-by: NLukáš Doktor <ldoktor@redhat.com>

Just like `avocado list` presents the statistics for the test types,
it may be useful to also list the number of tests that have a
particular tag.
Signed-off-by: NCleber Rosa <crosa@redhat.com>

Test tags (via docstring directives) have been supported in Avocado
for a number of releases, but up until now user would only be able to
see the tags on their tests by resorting to the source code.

This was already difficult when the tags where set only at the class
level, now that they can be set both on the class and method
docstring, users would have to do the "math" on their minds.

Let's display the test tags when the verbose switch is given to the
list command to make the life of users easier.
Signed-off-by: NCleber Rosa <crosa@redhat.com>

This is a UI tweak, as suggested during review, to remove the tests
time.

It's kept separately, because it can be easily dropped if any reason
to keep it arises.
Signed-off-by: NCleber Rosa <crosa@redhat.com>

A long standing task is to show the job execution time, instead of the
sum of tests execution time.  After some major work, this is now
possible.

This change adds the job execution time to the human interface UI, in
addition to the test execution time.  Some users may prefer the job
execution time alone, while others may prefer all the time statistics
on a single line.

Since we cannot please everyone, this version just adds it.  I believe
UI tweaks can follow.
Signed-off-by: NCleber Rosa <crosa@redhat.com>

With the resultdb optional plugin new arguments were added which are
missing in our man pages.
Signed-off-by: NLukáš Doktor <ldoktor@redhat.com>

man is quite outdated. This patch updates the man to reflect the
current options and UI.

Reference: https://trello.com/c/UB6hG2mDSigned-off-by: NAmador Pahim <apahim@redhat.com>

This change removes the knowledge, and dependency, that the current
Avocado core has of the HTML rendering code.

The only user visible change is that the UI message about the
generated HTML file now appears one line after it used to appear.  The
reason is not because of the HTML plugin itself, but because the
results are now generated outside the context of the job, and that is,
chronologically a bit later than it used to be.
Signed-off-by: NCleber Rosa <crosa@redhat.com>

Avocado has until now shown very simplified time information for jobs,
and some could even say unprecise or just plain incorrect.

The fact is that the time displayed in the UI (HumanResult) and also
on the various other supported formats is the sum of the run time of
individual tests.

The problem with that is tha a job may include other actions that can
take time, from little to substancial amounts.  This makes it clear
that a job that say, took 30 seconds to run (although not displayed at
this point), spent only 20 seconds actually running tests.
Signed-off-by: NCleber Rosa <crosa@redhat.com>

And deprecate the --multiplex-files option, which was actually
the complete name of the option, but was not being used formally
by our documentation.
Signed-off-by: NCleber Rosa <crosa@redhat.com>

In preparation for the large plugin refactor, let's simplify the code ASAP
and drop related (custom) functionality that is going to be replaced.

Standard setuptools entrypoints is going to provide better capability then
our custom code that supports external 'avocado_*.py'module loading.
Signed-off-by: NCleber Rosa <crosa@redhat.com>

This patch doesn't change any functionality, it only renames the
inner_runner to external_runner (the plugin is called simply external)
Signed-off-by: NLukáš Doktor <ldoktor@redhat.com>

The BUGGY test concept relies on loading/executing the Python file
containing the tests and failing to do so. Since Avocado won't
load/execute Python test files anymore, it's not possible or desirable
to keep the BUGGY tests around.
Signed-off-by: NCleber Rosa <crosa@redhat.com>

In some external test suites, what is defined as a "test" can not be
run by itself and need a specific runner script/tool.

This introduces the concept of an "inner runner", that is, a custom
test runner or tool that can deal with custom tests that do follow
the Avocado SIMPLE test definition.

This implements the Trello card:

 https://trello.com/c/TpXecE2n/486-introduce-inner-runner-to-avocado-run

More information can be found in the man page section about it.

Changes from v1:
 * removed "level" from "inner level runner" mentions
 * replicated docs from man page to "main" docs
Signed-off-by: NCleber Rosa <crosa@redhat.com>

Instead of putting outputs on separate lines, condensate
test results summary into a single line. The new output
looks like:

$ avocado run passtest
JOB ID     : f2f5060440bd57cba646c1f223ec8c40d03f539b
JOB LOG    : /home/user/avocado/job-results/job-2015-07-27T17.13-f2f5060/job.log
JOB HTML   : /home/user/avocado/job-results/job-2015-07-27T17.13-f2f5060/html/results.html
TESTS      : 1
(1/1) passtest.py:PassTest.test: PASS (0.00 s)
RESULTS    : PASS 1 | ERROR 0 | FAIL 0 | SKIP 0 | WARN 0 | INTERRUPT 0
TIME       : 0.00 s

We updated a few unittests in order to not depend on the
looks of the human output anymore, since unless we are
specifically testing for human output behavior, the unittests
should use machine readable output. Also, the documentation
was updated to reflect the new output layout.
Signed-off-by: NLucas Meneghel Rodrigues <lmr@redhat.com>

The command line options and documentations refer to "process",
"applications", etc. It's more natural to think and describe them as
"executables" instead.
Signed-off-by: NCleber Rosa <crosa@redhat.com>

The command line options and documentations refer to "binaries" a
lot, with options descriptions such as BINARY_PATH. It's more
natural to think and describe them as "executables" instead.
Signed-off-by: NCleber Rosa <crosa@redhat.com>

Update the manual page regarding wrapping process, we're using
the shell-like style to select the process to wrap on scripts.

The documentation explains the previous behavior, before
the commit f6bf2c02Signed-off-by: NRudá Moura <rmoura@redhat.com>

Update the man page with new options on the multiplexer.
Signed-off-by: NLucas Meneghel Rodrigues <lmr@redhat.com>

Use command line --config to read a custom configuration from a file.
Note this option is for the avocado application, so it has
effect for all subcommands.

Example:

$ avocado --config /var/tmp/avocado.conf config
Config files read (in order):
/home/rmoura/Work/avocado/etc/avocado/avocado.conf
/home/rmoura/Work/avocado/etc/avocado/conf.d/gdb.conf
/var/tmp/avocado.conf
...

Also, update man-page about the new option --config.
Signed-off-by: NRudá Moura <rmoura@redhat.com>

The default UI has changed a bit, so let's update the documentation to
match it.
Signed-off-by: NCleber Rosa <crosa@redhat.com>

Those are not necessary in the avocado options and can
safely stay as a config file only option.
Signed-off-by: NLucas Meneghel Rodrigues <lmr@redhat.com>

Remove another seldomly used option, --job-log-level, and
move the opportunity to set the value to the config file.
Signed-off-by: NLucas Meneghel Rodrigues <lmr@redhat.com>

Add command line `--job-results-dir` to the Avocado runner, so we can
store the job results in a different directory than the one
configured.

Remove `--logdir` from the main avocado application,
this option is obsolete and actually it was not working.

Update the manual page regarding this change.
Signed-off-by: NRudá Moura <rmoura@redhat.com>

From the current command line `--disable-output-check` replace it
with the new command line `--output-check {on,off}`, defaults to `on`.

Use `--output-check=off` to behave like ``--disable-output-check`.

Also, validate the value from command line `--output-check-record`.
Signed-off-by: NRudá Moura <rmoura@redhat.com>

Some typos found during automated inspection of the code.
Signed-off-by: NLucas Meneghel Rodrigues <lmr@redhat.com>

Signed-off-by: NLucas Meneghel Rodrigues <lmr@redhat.com>

Add support to inspect and find tests inside a directory, recursively,
by using the method `discover_url`. It takes a url (path) and
returns the tests parameters it discovers.

The loader object has the .discover() method, that can be used
to inspect the location, and the .validate() method, that will
discover any problems on the user's input.

We are also changing the behavior of avocado to error out on
invalid inputs (missing paths or files that are not avocado
tests). Therefore, the concept of MISSING or NOT_A_TEST both
disappear from the UI. Unittests were updated to reflect the
new status quo.
Signed-off-by: NRudá Moura <rmoura@redhat.com>
Signed-off-by: NLucas Meneghel Rodrigues <lmr@redhat.com>

Implement runner command line option `--sysinfo` to turn on or off
any reports created and related to the execution of sysinfo, like
profilers, command lines utilities and other hardware logs. Default: on.

Example: avocado run --sysinfo=off passtest

There is a new configuration options inside [sysinfo.collect]
named "enabled" which will turn the feature on and off too.
It accepts a boolean value and the default is to enable sysinfo.

The command line option overwrites the configuration value.
Signed-off-by: NRudá Moura <rmoura@redhat.com>

Minor improvement to the initial description in the man page
and update links to point to the (new) project page everywhere.
Signed-off-by: NAdemar de Souza Reis Jr <areis@redhat.com>

Avocado itself is released under GPLv2 (explicit version), but most of the code
is currently licensed under GPLv2+, so make that clear in the documentation.
Signed-off-by: NAdemar de Souza Reis Jr <areis@redhat.com>

As a subcommand (option, really) of the plugin command. The generated
distro file is supposed to be portable and be named canonically based
on the name, version, release and arch.

Changes from v0:
 * Return proper exit status when required command line arguments are
   missing
Signed-off-by: NCleber Rosa <crosa@redhat.com>

At this stage, the distro plugin is very simplistic and prints the
detected distribution of the current machine. One could algue that
this is rather useless, but, with the component isolation work this
command is going to be helpful for troubleshooting purposes.

Also, this plugin is a placeholder where other functionality will
live, including (but not limited to):

 * the ability to create "distro definition" files, akin to
   autotest's autotest-distro-def-create.
 * the ability to view "distro definition" files, akin to
   autotest's autotest-distro-def-view.
 * the ability to download "distro definition" files from a
   repository
 * (maybe) the ability to push a distro definition file to
   avocado-server. This may be delegated to a full blown REST
   client though.
Signed-off-by: NCleber Rosa <crosa@redhat.com>

Instead of leaving our own users in doc references, replace
/home/$USER to simply $HOME.
Signed-off-by: NLucas Meneghel Rodrigues <lmr@redhat.com>

As discussed in the Trello card

https://trello.com/c/WsDMg7RP/161-initial-support-for-avocado-config-files-server-and-client

Move settings.ini to avocado.conf. The basic idea is
that the name is more intuitive.
Signed-off-by: NLucas Meneghel Rodrigues <lmr@redhat.com>

Due the concept of "drop-in" tests didn't match the actual
purpose of running any script or binary as a test in Avocado, we're
going to use the term "simple test" (defined in SimpleTest class)
to contrast with "native test", which are Python modules that
uses Avocado's API.
Signed-off-by: NRudá Moura <rmoura@redhat.com>

This patch adds support for multiple yaml files by merging the values
together. Merge overrides the nodes value and appends missing children
iteratively. It's deterministic AND order dependant (yaml1 yaml2 migh
not provide the same results as yaml2 yaml1!).
Signed-off-by: NLukáš Doktor <ldoktor@redhat.com>

Add documentation manual page part for the command line
feature `--wrapper`.
Signed-off-by: NRudá Moura <rmoura@redhat.com>

Developers using Avocado's GDB support may also need to use their custom
build of gdbserver.

This implements the same logic and options as the previous commit does
for the GDB binary, including API level changes.

Changes from v1:
   * use `avocado.utils.process.find_command` as the first default
     location for the gdbserver binary, falling back to the hard coded
     default of `/usr/bin/gdbserver` if that fails.
Signed-off-by: NCleber Rosa <crosa@redhat.com>