Introduce --rawstats which prints all statistics fields from the new API
similarly to how the virsh event handler prints them.
Signed-off-by: NPeter Krempa <pkrempa@redhat.com>
Reviewed-by: NEric Blake <eblake@redhat.com>

Introduce the --anystats flag which does not skip the printing of the
stats if the job was unsuccessful.
Signed-off-by: NPeter Krempa <pkrempa@redhat.com>
Reviewed-by: NEric Blake <eblake@redhat.com>

Printing that a job failed is rather unhelpful. Print at least the
operation which failed.

Achieve this by moving the check whether to print stats later but
replace it with a check which will skip printing of the operation if
there's no job.
Signed-off-by: NPeter Krempa <pkrempa@redhat.com>
Reviewed-by: NEric Blake <eblake@redhat.com>

Signed-off-by: NPeter Krempa <pkrempa@redhat.com>
Reviewed-by: NEric Blake <eblake@redhat.com>

Signed-off-by: NPeter Krempa <pkrempa@redhat.com>
Reviewed-by: NEric Blake <eblake@redhat.com>

virDomainGetJobStats destroys the completed statistics on the first
read. Give the user possibility to keep them around if they wish so.

Add a flag VIR_DOMAIN_JOB_STATS_KEEP_COMPLETED which will read the stats
without destroying them.
Signed-off-by: NPeter Krempa <pkrempa@redhat.com>
Reviewed-by: NEric Blake <eblake@redhat.com>

To simplify the stats printer code we convert the new statistics from
the typed parameter list into the old stats structure.

Extract this code since it takes a lot of space.
Signed-off-by: NPeter Krempa <pkrempa@redhat.com>
Reviewed-by: NEric Blake <eblake@redhat.com>

The 32-bit x86 binary is called qemu-system-i386, not
qemu-system-i686. This mistake across many test XML files was
not noticed because the mistake was also made in testutilsqemu.c
when mocking the capabilities.
Reviewed-by: NErik Skultety <eskultet@redhat.com>
Signed-off-by: NDaniel P. Berrangé <berrange@redhat.com>

For managed save we can choose between various compression
methods. I randomly tested the 'xz' program on a 8 GB guest
and was surprised to have to wait > 50 minutes for it to
finish compressing, with 'xz' burning 100% cpu for the
entire time. Despite the impressive compression, this is
completely useless in the real world as it is far too long
to wait to save the VM.

The 'xz' binary defaults to '-6' optimization level which
aims for high compression, with moderate memory usage,
at the expense of speed.

This change switches it to use the '-3' optimization level
which is documented as being the one that optimizes speed
at expense of compression. Even with this, it will still
outperform all the other options in terms of compression
level. It is a little less than x4 faster than '-6' which
means it starts to be a viable choice to use 'xz' for
people who really want best compression.

The test results on a 1 GB, fairly freshly booted VM are
as follows

  format | save  | restore  size
  =======+=======+=============
  raw    |   05s |    1s  | 428 MB
  lzop   |   05s |    3s  | 160 MB
  gzip   |   29s |    5s  | 118 MB
  bz2    |   54s |   22s  | 114 MB
  xz     | 4m37s |   13s  |  86 MB
  xz -3  | 1m20s |   12s  |  95 MB

Based on this we can say

 * For moderate compression with no noticable loss in speed

       => use lzop

 * For high compression with moderate loss in speed

       => use gzip

 * For best compression with significant loss in speed

       => use xz
Reviewed-by: NMichal Privoznik <mprivozn@redhat.com>
Signed-off-by: NDaniel P. Berrangé <berrange@redhat.com>

My commit 3bbe1020 forgot
to update the news.
Signed-off-by: NJán Tomko <jtomko@redhat.com>

Commit v5.7.0-248-g03449e25 removed "cd tests" without updating the
patch to test-suite.log.
Signed-off-by: NJiri Denemark <jdenemar@redhat.com>
Reviewed-by: NPavel Hrdina <phrdina@redhat.com>

Signed-off-by: NJán Tomko <jtomko@redhat.com>

This is a very simple and straightforward implementation of the opposite
what buildPool does for the disk backend.
The background for this change comes from an existing test case in TCK
which does use the delete method for a pool of type disk, but it
truly could not have ever worked since the implementation simply
wasn't there for the pool of type disk.
Signed-off-by: NErik Skultety <eskultet@redhat.com>
Reviewed-by: NMichal Privoznik <mprivozn@redhat.com>

When SELinux support was first introduced the libselinux library
wasn't that advanced and setfilecon_raw() or fsetfilecon_raw()
could fail even when the target context was set. Looking at the
current code [1][2] this is no longer the case. We can drop our
workarounds.

1: https://github.com/SELinuxProject/selinux/blob/master/libselinux/src/setfilecon.c#L10
2: https://github.com/SELinuxProject/selinux/blob/master/libselinux/src/fsetfilecon.c#L10Signed-off-by: NMichal Privoznik <mprivozn@redhat.com>
Reviewed-by: NDaniel P. Berrangé <berrange@redhat.com>

Signed-off-by: NJim Fehlig <jfehlig@suse.com>

The news schema requires two digits for both month and day in the date
attribute. s/2/02/ in the day value of date to fix the following
'make check' failure

2165) Checking ../docs/news.xml against ../news.rng                     ...
libvirt: XML Util error : XML document failed to validate against schema:
Unable to validate doc against /home/jfehlig/virt/upstream/libvirt/build/../docs/schemas/../news.rng
Element release failed to validate attributes
Signed-off-by: NJim Fehlig <jfehlig@suse.com>

* docs/news.xml: updated for the release
Signed-off-by: NDaniel Veillard <veillard@redhat.com>

Commit 4b58fdf2 which enabled block copy also for network
destinations needed to limit when the 'mirror' storage source is
initialized in cases when we e.g. don't have an appropriate backend.

Limiting it just to virStorageFileSupportsCreate is too restrictive as
for example we can't precreate block devices and thus wouldn't
initialize the 'mirror' but since it's a local source we'd try to
examine it. This would fail since it wouldn't be initialized.

Fix it by introducing a more granular check whether certain operations
are supported and fix the check interlocks.

https://bugzilla.redhat.com/show_bug.cgi?id=1778058Signed-off-by: NPeter Krempa <pkrempa@redhat.com>
Reviewed-by: NJán Tomko <jtomko@redhat.com>

We tolerate image format detection during block copy in very specific
circumstances, but the code didn't error out on failure of the format
detection.
Signed-off-by: NPeter Krempa <pkrempa@redhat.com>
Reviewed-by: NJán Tomko <jtomko@redhat.com>

The API XML files are generated files, so live in the build dir not the
source dir.
Reviewed-by: NPavel Hrdina <phrdina@redhat.com>
Signed-off-by: NDaniel P. Berrangé <berrange@redhat.com>

Gluster 6.0 is not built on i686 for RHEL-8, which prevents libvirt from
building. Let's just disable gluster there as all we need are client
libraries anyway.
Signed-off-by: NJiri Denemark <jdenemar@redhat.com>
Reviewed-by: NFabiano Fidêncio <fidencio@redhat.com>

In v5.9.0-273-g8ecab214 I've tried to fix a lock ordering
problem, but introduced a crasher. Problem is that because the
client lock is unlocked (in order to honour lock ordering) the
stream we are currently checking in daemonStreamFilter() might be
freed and thus stream->priv might not even exist when the control
get to virMutexLock() call.

To resolve this, grab an extra reference to the stream and handle
its cleanup should the refcounter reach zero after the deref.
If that's the case and we are the only ones holding a reference
to the stream, we MUST return a positive value to make
virNetServerClientDispatchRead() break its loop where it iterates
over filters. The problem is, if we did not do so, then
"filter = filter->next" line will read from a memory that was
just freed (freeing a stream also unregisters its filter).
Signed-off-by: NMichal Privoznik <mprivozn@redhat.com>
Reviewed-by: NJán Tomko <jtomko@redhat.com>

In commit 2ccb5335 I've refactored how we fill the typed parameters
for domain statistics. The commit introduced a regression in the
formating of stats for IOthreads by using the array index to label the
entries as it's common for all other types of statistics rather than
the iothread IDs used for iothreads.

Since only the design of iothread deviates from the common approach used
in all other statistic types this was not caught.

https://bugzilla.redhat.com/show_bug.cgi?id=1778014Signed-off-by: NPeter Krempa <pkrempa@redhat.com>
Reviewed-by: NJán Tomko <jtomko@redhat.com>

The original implementation used QEMU_ADD_COUNT_PARAM which added the
'count' suffix, but 'cnt' was documented. Fix the documentation to
conform with the original implementation.
Signed-off-by: NPeter Krempa <pkrempa@redhat.com>
Reviewed-by: NJán Tomko <jtomko@redhat.com>

Before we rewrote nss plugin so that it doesn't use libvirt's
internal functions it used virLeaseReadCustomLeaseFile() to parse
.status files. After the rewrite it's using read() + yajl_parse()
+ yajl_complete_parse(). There's one catch though,
virLeaseReadCustomLeaseFile() skipped over empty files.

An empty .status file is created when a network is started. This
is because we configure dnsmasq to use our leasehelper. So the
first thing it does it calls it as follows:

  DNSMASQ_INTERFACE=virbr0 /usr/libexec/libvirt_leaseshelper init

which causes the leasehelper to create empty virbr0.status file.
If there is only one libvirt network then that is no problem -
there are no other .status files to parse anyway. But if there
are two or more networks then the first empty .status file causes
whole parsing process and subsequently the whole name lookup
process to fail.

Fixes: v5.7.0-rc1~343
Reported-by: NPavel Hrdina <phrdina@redhat.com>
Signed-off-by: NMichal Privoznik <mprivozn@redhat.com>
Reviewed-by: NDaniel P. Berrangé <berrange@redhat.com>

After generating the API HTML files we run xmllint in docs/html/*.html
to validate the correctness. Since

  commit 0aa8536f
  Author: Daniel P. Berrangé <berrange@redhat.com>
  Date:   Wed Nov 20 14:49:26 2019 +0000

    docs: generate API reference pages for admin, qemu & lxc libraries

we have many rules generating files into docs/html/. The xmllint
calls for each rule are picking up files which are part-generated by
other parallel build rules resulting in transient errors like:

  GEN      html/index.html
  GEN      html/index-admin.html
  GEN      html/index-qemu.html
  GEN      html/index-lxc.html
  GEN      hvsupport.html.in
html/index-lxc.html:1: parser error : Document is empty

^
make[4]: *** [Makefile:2407: html/index-qemu.html] Error 1

The easiest solution is to move the xmllint rules to the 'make check'
phase of the build.
Reviewed-by: NMichal Privoznik <mprivozn@redhat.com>
Signed-off-by: NDaniel P. Berrangé <berrange@redhat.com>

On Fedora 31 with GCC 9.2.1, compiling qemuxml2argvtest takes
about 36 seconds since

  commit 30c6d992
  Author: Jiri Denemark <jdenemar@redhat.com>
  Date:   Thu Oct 24 17:51:42 2019 +0200

    qemuxml2argvtest: Update host arch for DO_TEST*ARCH* tests

The optimizer is hitting some pathological performance behaviour due to
the high number of branches in the mymain() method.

Pushing the branch tests down into the testCompareXMLToArgv method
brings the compile time down to 3 seconds.

This likely related to this GCC bug:

  https://gcc.gnu.org/bugzilla/show_bug.cgi?id=58479Reviewed-by: NDaniel Henrique Barboza <danielhb413@gmail.com>
Signed-off-by: NDaniel P. Berrangé <berrange@redhat.com>

The virTypedParamsFilter function doesn't mind params == NULL if nparams
is zero. And there's no need to check for params == NULL && nparams > 0
because this is checked higher in the stack.

In fact all the virCheckNonNull* checks in virTypedParamsFilter are
useless.

https://bugzilla.redhat.com/show_bug.cgi?id=1777094Signed-off-by: NJiri Denemark <jdenemar@redhat.com>
Reviewed-by: NDaniel Henrique Barboza <danielhb413@gmail.com>
Reviewed-by: NErik Skultety <eskultet@redhat.com>

The default macOS image in travis is broken, throwing python
exceptions when trying to install glib. Explicitly ask for the
newer 10.3 image which works correctly.

We now need to also point to the homebrew installed libxml2
rather than the OS distro provided one, since the OS distro
one has a pkg-config file present, but no actual header
files.
Reviewed-by: NPavel Hrdina <phrdina@redhat.com>
Signed-off-by: NDaniel P. Berrangé <berrange@redhat.com>

Signed-off-by: NMichal Privoznik <mprivozn@redhat.com>
Reviewed-by: NJán Tomko <jtomko@redhat.com>

Now that we have a separate job type which will not trigger normal code
paths for terminating job we can remove the ad-hoc handling.

This possibly fixes the issue of a broken job inheriting the disk and
then finishing in which case we'd not detach the backing chain.
Signed-off-by: NPeter Krempa <pkrempa@redhat.com>
Reviewed-by: NCole Robinson <crobinso@redhat.com>

To better track jobs we couldn't parse let's introduce a new job type
which will clarify semantics internally in few places.
Signed-off-by: NPeter Krempa <pkrempa@redhat.com>
Reviewed-by: NCole Robinson <crobinso@redhat.com>

We will need to clear per-job type data when we will be marking a
blockjob as broken in the new way. Extract the code for future reuse.
Signed-off-by: NPeter Krempa <pkrempa@redhat.com>
Reviewed-by: NCole Robinson <crobinso@redhat.com>

Both failure to refresh and to dismiss the job are very unlikely but if
they happen there's not much we can do about the blockjob.

The concluded job handlers treat it as if the job failed if we don't
update the state to 'QEMU_BLOCKJOB_STATE_COMPLETED' which is probably
the safest thing to do here.
Signed-off-by: NPeter Krempa <pkrempa@redhat.com>
Reviewed-by: NCole Robinson <crobinso@redhat.com>

Otherwise it would get dropped later on as untracked despite us knowing
about it. Additionally since we cancelled it we must wait to dismiss it
which would not be possible if we unregister it. This also opened a
window for a race condition since the job state change event of the
just-cancelled job might be delivered prior to us unregistering the job
in which case everything would work properly.
Signed-off-by: NPeter Krempa <pkrempa@redhat.com>
Reviewed-by: NCole Robinson <crobinso@redhat.com>

Since we don't know what happened to the job we can't do much about it
but we can at least log that this happened.
Signed-off-by: NPeter Krempa <pkrempa@redhat.com>
Reviewed-by: NCole Robinson <crobinso@redhat.com>

We must exit the monitor prior to refusing other work, otherwise the VM
object will become unusable.

This bug was introduced in commit v5.5.0-244-gc4123837 but thankfully
the code path was not excercised without QEMU_CAPS_BLOCKDEV.
Signed-off-by: NPeter Krempa <pkrempa@redhat.com>
Reviewed-by: NCole Robinson <crobinso@redhat.com>

Block jobs may be members of async jobs so it makes more sense to
refresh block job state after we do steps for async job recovery.
Signed-off-by: NPeter Krempa <pkrempa@redhat.com>
Reviewed-by: NCole Robinson <crobinso@redhat.com>

qemu returns an error message in the job statistics even if the job was
cancelled to emphasize it was not successful. Libvirt didn't properly
transform it into QEMU_BLOCKJOB_STATE_CANCELLED though.
Signed-off-by: NPeter Krempa <pkrempa@redhat.com>
Reviewed-by: NCole Robinson <crobinso@redhat.com>

Commit ed56851f didn't wire up fetching of the statistics for the
job which are reported by 'query-jobs'.
Signed-off-by: NPeter Krempa <pkrempa@redhat.com>
Reviewed-by: NCole Robinson <crobinso@redhat.com>