I think the setUp section is a better place for the "SKIP working only
in setUp" than the "Running third party test suites" section. Let's move
it there.
Signed-off-by: NLukáš Doktor <ldoktor@redhat.com>

We emphasize the `whiteboard` extensively, but we should not forget to
mention the `outputdir` which is reserved for custom test-produced files
to be attached.
Signed-off-by: NLukáš Doktor <ldoktor@redhat.com>

The html output plugin is optional and not installed by default. Let's
increase the visibility of the ways to make it working.
Signed-off-by: NLukáš Doktor <ldoktor@redhat.com>

* https://github.com/avocado-framework/avocado:
  Fixup: format strings for tap output

Before this patch tap output to a file fails with below error

 File "/usr/lib/python2.7/site-packages/avocado_framework-42.0-py2.7.egg/avocado/core/result.py", line 178, in check_test
    self.end_test(state)

  File "/usr/lib/python2.7/site-packages/avocado_framework-42.0-py2.7.egg/avocado/plugins/tap.py", line 69, in end_test
    self.__write("#   " + line)

  File "/usr/lib/python2.7/site-packages/avocado_framework-42.0-py2.7.egg/avocado/plugins/tap.py", line 35, in writeln
    return self.output.write(msg % args + "\n")

TypeError: not enough arguments for format string

This patch addresses above issue.
Signed-off-by: NSatheesh Rajendran <sathnaga@linux.vnet.ibm.com>
Signed-off-by: NLukáš Doktor <ldoktor@redhat.com>

Occasionally the 2s wait is not enough, let's wait 2.5s to be sure we
don't hit the 2s ignore window.
Signed-off-by: NLukáš Doktor <ldoktor@redhat.com>

Signed-off-by: NLukáš Doktor <ldoktor@redhat.com>

* https://github.com/avocado-framework/avocado:
  Pylint fixes

Recent pylint update is more pedantic about module level spacing. There
are no changes to code, only couple of extra spaces to make it happy.
Signed-off-by: NLukáš Doktor <ldoktor@redhat.com>

When running tests remotely, html plugin might be already present. Let's
remove the previously existing files to avoid failure and to always get
the latest results.

NOTE: We might want to re-consider whether to skip the html update, or
whether to always update results, but the current status is not optimal.
Signed-off-by: NLukáš Doktor <ldoktor@redhat.com>

The joblogdir was added several years ago but it was not defined in
remote.test. Let's add it and use the local_job_logdir because this
variable is used during start_tests/start_test/end_test/... which could
make the local avocado to process files which does not exists or files
which does not belong to this job.
Signed-off-by: NLukáš Doktor <ldoktor@redhat.com>

Signed-off-by: NCleber Rosa <crosa@redhat.com>

Signed-off-by: NCleber Rosa <crosa@redhat.com>

Besides the port of the TAP plugin, this makes the Result only
consider a test to have run after it finishes

The way the Result class has been accounting tests run is a
consequence of the implementation detail.  Before any tests are
actually run, that is, in `start_tests()`, the counter is incremented.
This is obviously wrong.

The counter should only be incremented at `end_test()`, which is
already done.  AFAICT, the current approach was working because the
subclasses of Result, such as HumanResult, would print the current
status on their own `end_test()`, and then only after that they would
call the parent (Result) `end_test()`.  This, together with the
reconcile method, seems to have kept the counters accurate to end
users, even when they were not accurate internally.
Signed-off-by: NCleber Rosa <crosa@redhat.com>

Signed-off-by: NCleber Rosa <crosa@redhat.com>

Even though Avocado already publishes an interface for result plugins
(in the avocado.core.plugin_interfaces module), that one is suitable
for plugins that will generate one output after the job as a whole
finishes.

There is another type of use case, already present in Avocado, but
still not implemented using the newer plugin mechanism.  The use
case is result plugins that must respond to events, some that come
from the job and some that come from tests.

The EventResult interface is a collection of Job interfaces that
would happen before tests start running (pre_tests and post_tests)
and events that are closer to the test themselves (start_test,
test_progress and end_test).

The new dispatcher calls plugins that implement the ResultEvents
interface.
Signed-off-by: NCleber Rosa <crosa@redhat.com>

This is the right place to criticize the test suite.  Even though I
don't feel it's right to abort the application from within the job
class, this prevents the current user experience to break.

In the near future, I'd like to see removed, and avocado runs without
a test suite to run and give these exact results: no tests were
available to be run, so none were executed.
Signed-off-by: NCleber Rosa <crosa@redhat.com>

The html plugin can only cope with finished results. Currently this is
not problem, because `render` is not called, but following patches adds
the possibility to postprocess results in such cases so let's prepare
for it.
Signed-off-by: NLukáš Doktor <ldoktor@redhat.com>

A result has a 1:1 relation to a Job. To match one to the other the
job `unique_id` is saved to the result as `job_unique_id`.

The same way that a job alwayas was a unique_id, the result should
never miss the job_unique_id.  So, let's not try to hide or postpone
errors from such as scenario.
Signed-off-by: NCleber Rosa <crosa@redhat.com>

This change introduces what is going to be the only one result
instance to a job.  Instead of keeping track of the same information
on multiple plugins, plugins will refer to the unique job result
instance.

This changes does not remove the result proxy just yet, because by
doing so, the plugins that depend on it would cease to run.  Once
plugins are ported to use the unique result instance, the result proxy
will be removed.
Signed-off-by: NCleber Rosa <crosa@redhat.com>

Some tests might finish before producing any output, then the runner
should reflect that instead of IOError followed by hang because of
missing test_state.
Signed-off-by: NLukáš Doktor <ldoktor@redhat.com>

* https://github.com/avocado-framework/avocado:
  update pydot version
  call read_dot() from the corresponding module

Let's update pydot to work with the latest networkx version.
Signed-off-by: NAmador Pahim <apahim@redhat.com>

graphviz drawing tools are no longer imported into the top
level namespace of networkx (networkx >= 1.11). This patch
changes the read_dot(), using it from the actual module.
Signed-off-by: NAmador Pahim <apahim@redhat.com>

Let's include in the `Releasing avocado` guide the steps required to
configure Read The Docs every version release.
Signed-off-by: NAmador Pahim <apahim@redhat.com>

We have docs available on Red The Docs for all versions from 36.0
onwards. Let's include in each release notes the link for the version
documentation so users can easily see the docs for that version.
Signed-off-by: NAmador Pahim <apahim@redhat.com>

* https://github.com/avocado-framework/avocado:
  Loader: use symbolic names for discovery test type selection

Let's make these symbols available in the documentation, and use them
where they're not yet being used.
Signed-off-by: NCleber Rosa <crosa@redhat.com>

Let's give used 1 year os so to prepare for this deprecation.

Reference: https://trello.com/c/9scZuNOm/585-get-rid-of-default-paramsSigned-off-by: NAmador Pahim <apahim@redhat.com>

This API is being depracated. Let's remove the examples from our docs.

Reference: https://trello.com/c/9scZuNOm/585-get-rid-of-default-paramsSigned-off-by: NAmador Pahim <apahim@redhat.com>

According to the test name RFC, once a test references goes though
the resolver, it becomes a test name.  When it's set to be run by
the runner, and to be recorded in the result, it becomes a test ID.

Since the JSON records the result, the `url` key is really misleading,
and it should be named `id` (as in test ID).

Compatibility has been kept, that is, the key named `url` still exists,
and can be removed in a future version.
Signed-off-by: NCleber Rosa <crosa@redhat.com>

Since we're renaming all of the mentions from URL to Test References, it also
makes sense to rename the `jobdata/urls` file to `jobdata/test_references`.

As compatibility measures, this tries to read the `urls` file when
`test_references` is not available (newer avocado and older jobdata) and
also adds a symlink from `test_referneces` to a link called `urls`, when
older avocado finds a newer jobdata.
Signed-off-by: NCleber Rosa <crosa@redhat.com>

It's been a while since the "Test ID RFC" came out:

https://www.redhat.com/archives/avocado-devel/2016-March/msg00024.html

Still, we've been referring to test URLs all around our code
and also on user visible strings.  This is an attempt to rename
the mentions of "URLs" that really should be "test references"
or simply "references" at this point.
Signed-off-by: NCleber Rosa <crosa@redhat.com>