CI flagged a failing mingw build, due to:
In file included from ../../src/conf/checkpoint_conf.c:24:
../gnulib/lib/configmake.h:8:17: error: expected identifier or '(' before string constant
    8 | #define DATADIR "/usr/i686-w64-mingw32/sys-root/mingw/share"
      |                 ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

As previously learned in commits bd205a90 and 976abdf6, gnulib's
configmake.h header does #define DATADIR "string...", while mingw's
<winsock2.h> expects to declare a type named DATADIR. As long as the
mingw system header is included first before configmake.h, the two
uses do not conflict, but until gnulib is patched to make configmake.h
automatically work around the issue, our immediate fix is the
workaround of rearranging our include order to insure no conflict.
Copy the paradigm used in domain_conf.c of using <unistd.h> to trigger
the indirect inclusion of <winsock2.h> on mingw.

Fixes: 1a4df34aSigned-off-by: NEric Blake <eblake@redhat.com>

Signed-off-by: NAndrea Bolognani <abologna@redhat.com>

Turning a NULL URI instead the empty string is very misleading when
reading the debug logs as the distinction between the two is
functionally important.
Reviewed-by: NAndrea Bolognani <abologna@redhat.com>
Signed-off-by: NDaniel P. Berrangé <berrange@redhat.com>

The recently added test27 spawns commandhelper. This is fine,
except, one of the things that commandhelper does is it records
arguments it was spawn with into commandhelper.log. Other test
cases then use checkoutput() to compare the arguments against the
expected ones and also unlink() the log file. However, test27()
is not doing that and thus it leaves the file behind. This
breaks distcheck.
Signed-off-by: NMichal Privoznik <mprivozn@redhat.com>
Reviewed-by: NDaniel P. Berrangé <berrange@redhat.com>

Signed-off-by: NIlias Stamatis <stamatis.iliass@gmail.com>
Reviewed-by: NErik Skultety <eskultet@redhat.com>

These test cases will start failing once the test driver provides
implementation for the virDomainGetCPUStats API.
Signed-off-by: NIlias Stamatis <stamatis.iliass@gmail.com>
Reviewed-by: NErik Skultety <eskultet@redhat.com>

The returned array of qemuMonitorJobInfo structs must be freed.

164 (16 direct, 148 indirect) bytes in 1 blocks are definitely lost in loss record 64 of 84
   at 0x4A3568B: realloc (vg_replace_malloc.c:826)
   by 0x4D888BD: virReallocN (viralloc.c:244)
   by 0x4D889B3: virExpandN (viralloc.c:293)
   by 0x4D88C87: virInsertElementsN (viralloc.c:435)
   by 0x214004: qemuMonitorJSONGetJobInfo (qemu_monitor_json.c:9185)
   by 0x148B3F: testQueryJobs (qemumonitorjsontest.c:2979)
   by 0x14C192: virTestRun (testutils.c:174)
   by 0x14BF36: mymain (qemumonitorjsontest.c:3286)
   by 0x14E256: virTestMain (testutils.c:1096)
   by 0x14BFD9: main (qemumonitorjsontest.c:3298)
Signed-off-by: NMichal Privoznik <mprivozn@redhat.com>
ACKed-by: NPeter Krempa <pkrempa@redhat.com>

Originally the names of the KVM CPU features were only used internally
for looking up their CPUID bits. So we used "__kvm_" prefix for them to
make sure the names do not collide with normal CPU features stored in
our CPU map.

But with QEMU 4.1 we check which features were enabled or disabled by a
freshly started QEMU process using their names rather than their CPUID
bits (mostly because of MSR features). Thus we need to change our made
up internal names into the actual names used by QEMU.
Signed-off-by: NJiri Denemark <jdenemar@redhat.com>
Tested-by: NVitaly Kuznetsov <vkuznets@redhat.com>
Reviewed-by: NJán Tomko <jtomko@redhat.com>

Most of the internally defined KVM CPUID features are not actually used
by libvirt. The QEMU driver may enable or disable them on the command
line, but we don't check for the associated CPU properties or CPUID
bits. They would be useless with QEMU 4.1 anyway since their names were
only remotely similar to the actual feature names.
Signed-off-by: NJiri Denemark <jdenemar@redhat.com>
Tested-by: NVitaly Kuznetsov <vkuznets@redhat.com>
Reviewed-by: NJán Tomko <jtomko@redhat.com>

All the features are hyperv features even though they are provided by
KVM with QEMU. The "KVM" part in the macro names does not make a lot of
sense.
Signed-off-by: NJiri Denemark <jdenemar@redhat.com>
Tested-by: NVitaly Kuznetsov <vkuznets@redhat.com>
Reviewed-by: NJán Tomko <jtomko@redhat.com>

Starting with QEMU 4.1, we're using the canonical feature names on the
command line and avoid aliases to prepare for possible deprecation of
all aliases in QEMU. But we do so only for features from our CPU map,
hyperv features defined in the code were unchanged and this patch fixes
it. Some features use "hv-" prefix unconditionally because they were
introduced recently enough to always support spelling with a dash.
Signed-off-by: NJiri Denemark <jdenemar@redhat.com>
Tested-by: NVitaly Kuznetsov <vkuznets@redhat.com>
Reviewed-by: NJán Tomko <jtomko@redhat.com>

Originally the names of the hyperv CPU features were only used
internally for looking up their CPUID bits. So we used "__kvm_hv_"
prefix for them to make sure the names do not collide with normal CPU
features stored in our CPU map.

But with QEMU 4.1 we check which features were enabled or disabled by a
freshly started QEMU process using their names rather than their CPUID
bits (mostly because of MSR features). Thus we need to change our made
up internal names into the actual names used by QEMU. Most of the names
are only used with QEMU 4.1 and newer and the reset was introduced with
QEMU recently enough to already support spelling with "-". Thus we don't
need to define them as "hv_*" with a translation to "hv-*" for new QEMU.

Without this patch libvirt would mistakenly report all hyperv features
as unavailable and refuse to start any domain using them with QEMU 4.1.
Reported-by: NVitaly Kuznetsov <vkuznets@redhat.com>
Signed-off-by: NJiri Denemark <jdenemar@redhat.com>
Tested-by: NVitaly Kuznetsov <vkuznets@redhat.com>
Reviewed-by: NJán Tomko <jtomko@redhat.com>

Earlier patches mentioned that the initial implementation will prevent
snapshots and checkpoints from being used on the same domain at once.
However, the actual restriction is done in this separate patch to make
it easier to lift that restriction via a revert, when we are finally
ready to tackle that integration in the future.
Signed-off-by: NEric Blake <eblake@redhat.com>
Reviewed-by: NDaniel P. Berrangé <berrange@redhat.com>

Time to actually issue the QMP transactions that create and delete
persistent checkpoints, resolving TODOs intentionally left earlier in
the series.  For create, we only need one transaction: inside, we
visit all disks affected by the checkpoint, and create a new enabled
bitmap, as well as disabling the bitmap of the first ancestor
checkpoint (if any) that also had a bitmap.  For deletion, we need
multiple QMP calls: for each disk, if there is an ancestor checkpoint
with a bitmap, then the bitmap must be merged (including activating
the ancestor bitmap if the leaf node changes), all before deleting the
bitmap from the checkpoint being removed.
Signed-off-by: NEric Blake <eblake@redhat.com>

Qemu bitmap operations require knowing the node name associated with
the format layer (the qcow2 file); as upcoming patches will be
grabbing that information frequently, make a helper function to access
it.

Another potential benefit of this function is that we have a single
place where we could insert a QMP node-name scraping call if we don't
currently know the node name, when -blockdev is not supported;
however, the goal is that we hopefully don't ever have to do that
because we instead scrape node names only at the point where they
change.
Signed-off-by: NEric Blake <eblake@redhat.com>
Reviewed-by: NDaniel P. Berrangé <berrange@redhat.com>

A lot of this work heavily copies from the existing snapshot APIs.
What's more, this patch is (intentionally) very similar to the
checkpoint code just added in the test driver, to the point that qemu
checkpoints are not fully usable in this patch, but it at least
bisects and builds cleanly.  The separation between patches is done
because the grunt work of saving and restoring XML and tracking
relations between checkpoints is common to the test driver, while the
later patch adding integration with QMP is specific to qemu.

Also note that the interlocking to prevent checkpoints and snapshots
from existing at the same time will be a separate patch, to make it
easier to revert that restriction when we finally round out the design
for supporting interaction between the two concepts.
Signed-off-by: NEric Blake <eblake@redhat.com>

This is similar to the existing directory for snapshots; the domain
will save one xml file per checkpoint, for reloading on the next
libvirtd restart.  Fortunately, since checkpoints mandate RNG
validation, we are assured that the checkpoint name will be usable as
a file name (no abuse of '../escape' as a checkpoint name, for
example).
Signed-off-by: NEric Blake <eblake@redhat.com>
Reviewed-by: NDaniel P. Berrangé <berrange@redhat.com>

Introduce the handler for finalizing a block commit and active bloc
commit job which will allow to use it with blockdev.
Signed-off-by: NPeter Krempa <pkrempa@redhat.com>
Reviewed-by: NJán Tomko <jtomko@redhat.com>

Introduce the handler for finalizing a block pull job which will allow
to use it with blockdev.

This patch also contains some additional machinery which is required to
store all the relevant job data in the status XML which will also be
reused with other block job types.
Signed-off-by: NPeter Krempa <pkrempa@redhat.com>
Reviewed-by: NJán Tomko <jtomko@redhat.com>

The @creation variable wasn't used - caused a Travis build failure.
Signed-off-by: NJohn Ferlan <jferlan@redhat.com>

In case of an incoming migration we do not need to run swtpm_setup
with all the parameters but only want to get the benefit of it
creating a TPM state file for us that we can then label with an
SELinux label. The actual state will be overwritten by the in-
coming state. So we have to pass an indicator for incomingMigration
all the way to the command line parameter generation for swtpm_setup.
Signed-off-by: NStefan Berger <stefanb@linux.ibm.com>
Reviewed-by: NMarc-André Lureau <marcandre.lureau@redhat.com>

Similar to virsh-snapshots. Provides decent coverage of the checkpoint
API, the test driver implementation, and the virsh access to the API.
A later patch will worry about testing that snapshots and checkpoints
are mutually exclusive (in part so it is easier to revert that when we
finally implement the interaction and lift that restriction).
Signed-off-by: NEric Blake <eblake@redhat.com>
Reviewed-by: NDaniel P. Berrangé <berrange@redhat.com>

A lot of this work heavily copies from the existing snapshot APIs.
The test driver doesn't really have to do anything more than just
expose an interface into libvirt metadata, making it possible to test
saving and restoring XML, and tracking relations between multiple
checkpoints.
Signed-off-by: NEric Blake <eblake@redhat.com>
Reviewed-by: NDaniel P. Berrangé <berrange@redhat.com>

Introduce a bunch of new virsh commands for managing checkpoints in
isolation. More commands are needed for performing incremental
backups, but these commands were easy to implement by modeling heavily
after virsh-snapshot.c. There is no need for checkpoint-revert or
checkpoint-current since those snapshot APIs have no checkpoint
counterpart.  Similarly, it is not necessary to change which
checkpoint is current when redefining from XML, since until we
integrate checkpoints with snapshots, there is only a linear chain
(and you can deduce the current checkpoint by instead using
'checkpoint-list --leaves').  Other aspects of checkpoint-list are
also a bit simpler than the snapshot counterpart, in part because we
don't have to cater to back-compat to older API.

Upcoming patches will test these interfaces once the test driver
supports checkpoints.
Signed-off-by: NEric Blake <eblake@redhat.com>
Reviewed-by: NDaniel P. Berrangé <berrange@redhat.com>

The remote code generator had to be taught about the new
virDomainCheckpointPtr type, at which point the remote driver code for
checkpoints can be generated.
Signed-off-by: NEric Blake <eblake@redhat.com>
Reviewed-by: NDaniel P. Berrangé <berrange@redhat.com>

Creating a checkpoint does not modify guest-visible state,
but does modify host resources.  Rather than reuse existing
domain:write, domain:block_write, or domain:snapshot access
controls, it seems better to introduce a new access control
specific to tasks related to checkpoints and incremental
backups of guest disk state.
Signed-off-by: NEric Blake <eblake@redhat.com>
Reviewed-by: NDaniel P. Berrangé <berrange@redhat.com>

Wire up the use of a checkpoint list into each domain, similar to the
existing snapshot list.  This includes adding a function for checking
that a redefine operation fits in with the existing list, as well as
various filtering capabilities over the list contents.
Signed-off-by: NEric Blake <eblake@redhat.com>
Reviewed-by: NDaniel P. Berrangé <berrange@redhat.com>

Create a new file for managing a list of checkpoint objects, borrowing
heavily from existing virDomainSnapshotObjList paradigms.

Note that while snapshots definitely have a use case for multiple
children to a single parent (create a base snapshot, create a child
snapshot, revert to the base, then create another child snapshot),
it's harder to predict how checkpoints will play out with reverting to
prior points in time. Thus, in initial use, given a list of
checkpoints, you never have more than one child, and we can treat the
most-recent leaf node as the parent of the next node creation, without
having to expose a notion of a current node in XML or public API.
However, as the snapshot machinery is already generic, it is easier to
reuse the generic machinery that tracks relations between domain
moments than it is to open-code a new list-management scheme just for
checkpoints (hence, we still have internal functions related to a
current checkpoint, even though that has no observable effect
externally, as well as the addition of a function to easily find the
lone leaf in the list to use as the current checkpoint).
Signed-off-by: NEric Blake <eblake@redhat.com>
Reviewed-by: NDaniel P. Berrangé <berrange@redhat.com>

Add a new file checkpoint_conf.c that performs the translation to and
from new XML describing a checkpoint. The code shares a common base
class with snapshots, since a checkpoint similarly represents the
domain state at a moment in time. Add some basic testing of round trip
XML handling through the new code.

Of note - this code intentionally differs from snapshots in that XML
schema validation is unconditional, rather than based on a public API
flag.  We have many existing interfaces that still need to add a flag
for opt-in schema validation, but those interfaces have existing
clients that may not have been producing strictly-compliant XML, or we
may still uncover bugs where our RNG grammar is inconsistent with our
code (where omitting the opt-in flag allows existing apps to keep
working while waiting for an RNG patch).  But since checkpoints are
brand-new, it's easier to ensure the code matches the schema by always
using the schema.  If needed, a later patch could extend the API and
add a flag to turn on to request schema validation, rather than having
it forced (possibly just the validation of the <domain> sub-element
during REDEFINE) - but if a user encounters XML that looks like it
should be good but fails to validate with our RNG schema, they would
either have to upgrade to a new libvirt that adds the new flag, or
upgrade to a new libvirt that fixes the RNG schema, which implies
adding such a flag won't help much.

Also, the redefine flag requires the <domain> sub-element to be
present, rather than catering to historical back-compat to older
versions.
Signed-off-by: NEric Blake <eblake@redhat.com>

Now that various new API have been added or are coming soon, it is
worth a landing page that gives an overview of capturing various
pieces of guest state, and which APIs are best suited to which tasks.
Signed-off-by: NEric Blake <eblake@redhat.com>
Reviewed-by: NJohn Ferlan <jferlan@redhat.com>
Reviewed-by: NDaniel P. Berrangé <berrange@redhat.com>

Introduce a bunch of new public APIs related to backup checkpoints.
Checkpoints are modeled heavily after virDomainSnapshotPtr (both
represent a point in time of the guest), although a snapshot exists
with the intent of rolling back to that state, while a checkpoint
exists to make it possible to create an incremental backup at a later
time.  We may have a future hypervisor that can completely manage
checkpoints without libvirt metadata, but the first two planned
hypervisors (qemu and test) both always use libvirt for tracking
metadata relations between checkpoints, so for now, I've deferred
the counterpart of virDomainSnapshotHasMetadata for a separate
API addition at a later date if there is ever a need for it.

Note that until we allow snapshots and checkpoints to exist
simultaneously on the same domain (although the actual prevention of
this will be in a separate patch for the sake of an easier revert down
the road), that it is not possible to branch out to create more than
one checkpoint child to a given parent, although it may become
possible later when we revert to a snapshot that coincides with a
checkpoint.  This also means that for now, the decision of which
checkpoint becomes the parent of a newly created one is the only
checkpoint with no child (so while there are APIs for dealing with a
current snapshot, we do not need those for checkpoints).  We may end
up exposing a notion of a current checkpoint later, but it's easier to
add stuff when proven needed than to blindly support it now and wish
we hadn't exposed it.

The following map shows the API relations to snapshots, with new APIs
on the right:

Operate on a domain object to create/redefine a child:
virDomainSnapshotCreateXML          virDomainCheckpointCreateXML

Operate on a child object for lifetime management:
virDomainSnapshotDelete             virDomainCheckpointDelete
virDomainSnapshotFree               virDomainCheckpointFree
virDomainSnapshotRef                virDomainCheckpointRef

Operate on a child object to learn more about it:
virDomainSnapshotGetXMLDesc         virDomainCheckpointGetXMLDesc
virDomainSnapshotGetConnect         virDomainCheckpointGetConnect
virDomainSnapshotGetDomain          virDomainCheckpointGetDomain
virDomainSnapshotGetName            virDomainCheckpiontGetName
virDomainSnapshotGetParent          virDomainCheckpiontGetParent
virDomainSnapshotHasMetadata        (deferred for later)
virDomainSnapshotIsCurrent          (no counterpart, see note above)

Operate on a domain object to list all children:
virDomainSnapshotNum                (no counterparts, these are the old
virDomainSnapshotListNames           racy interfaces)
virDomainSnapshotListAllSnapshots   virDomainListAllCheckpoints

Operate on a child object to list descendents:
virDomainSnapshotNumChildren        (no counterparts, these are the old
virDomainSnapshotListChildrenNames   racy interfaces)
virDomainSnapshotListAllChildren    virDomainCheckpointListAllChildren

Operate on a domain to locate a particular child:
virDomainSnapshotLookupByName       virDomainCheckpointLookupByName
virDomainSnapshotCurrent            (no counterpart, see note above)
virDomainHasCurrentSnapshot         (no counterpart, old racy interface)

Operate on a snapshot to roll back to earlier state:
virDomainSnapshotRevert             (no counterpart, instead checkpoints
                                     are used in incremental backups via
				     XML to virDomainBackupBegin)
Signed-off-by: NEric Blake <eblake@redhat.com>
Reviewed-by: NDaniel P. Berrangé <berrange@redhat.com>

Prepare for new checkpoint APIs by describing the XML that will
represent a checkpoint.  The checkpoint XML is modeled heavily after
virDomainSnapshotPtr. See the docs for more details.

Add testsuite coverage for some minimal uses of the XML (bare minimum,
the sample from html, and a full dumpxml, and some counter-examples
that should fail schema validation). Although use of the REDEFINE flag
will require the <domain> subelement to be present, it is easier for
most of the tests to provide counterpart output produced with the
NO_DOMAIN flag (particularly since synthesizing a valid <domain>
during testing is not trivial).
Signed-off-by: NEric Blake <eblake@redhat.com>
Reviewed-by: NDaniel P. Berrangé <berrange@redhat.com>

If we are using -blockdev, then node names are always available
(because we set them).  But when not using it, we have to scrape node
names from QMP, and want to do so as infrequently as possible.  We
were scraping node names after reconnecting a new libvirtd to an
existing guest (see qemuProcessReconnect), and after any block job
that may have changed the set of node names we care about (legacy
block jobs), but forgot to scrape the names when first starting a
guest.  Do so now in order to allow the checkpoint code to always have
access to a node name without having to repeat a node name scrape
itself.

Future patches may need to clean up qemuDomainSetBlockThreshold (if
node names are always available, then it doesn't need to repeat a
scrape) and/or hotplug and media changes (if the addition of new nodes
can result in a null node name, then scraping at that point in time
would be appropriate).  But for now, this patch addresses only the
most common instance of a missing node name.
Signed-off-by: NEric Blake <eblake@redhat.com>
Reviewed-by: NDaniel P. Berrangé <berrange@redhat.com>

Since we are checking the 2nd parameter in the function for NULL,
we need to remove ATTRIBUTE_NONNULL(2) from the prototype.
Signed-off-by: NStefan Berger <stefanb@linux.ibm.com>
Reviewed-by: NJohn Ferlan <jferlan@redhat.com>
Message-Id: <20190726205633.2041912-5-stefanb@linux.vnet.ibm.com>

Remove the ATTRIBUTE_NONNULL(1) from virCommandSetSendBuffer()
prototype since we are checking for '!cmd' and move the initialization
if 'i' after the test for '!cmd'.
Signed-off-by: NStefan Berger <stefanb@linux.ibm.com>
Reviewed-by: NJohn Ferlan <jferlan@redhat.com>
Message-Id: <20190726205633.2041912-4-stefanb@linux.vnet.ibm.com>

Fix a potential memory leak by calling virCommandFree() in the cleanup
section.
Signed-off-by: NStefan Berger <stefanb@linux.ibm.com>
Reviewed-by: NJohn Ferlan <jferlan@redhat.com>
Message-Id: <20190726205633.2041912-3-stefanb@linux.vnet.ibm.com>

Use the existing variables rather then calling virTPMSwtpmXYZ().
Signed-off-by: NStefan Berger <stefanb@linux.ibm.com>
Reviewed-by: NJohn Ferlan <jferlan@redhat.com>
Message-Id: <20190726205633.2041912-2-stefanb@linux.vnet.ibm.com>

Create an empty log file if the log file was removed, otherwise the
transaction to set the security labels on the file will fail.
Signed-off-by: NStefan Berger <stefanb@linux.ibm.com>
Reviewed-by: NDaniel P. Berrangé <berrange@redhat.com>
Message-Id: <20190726210706.24440-3-stefanb@linux.ibm.com>

Set the transactionStarted to false if the commit failed. If this is not
done, then the failure path will report 'no transaction is set' and hide
more useful error reports.
Signed-off-by: NStefan Berger <stefanb@linux.ibm.com>
Reviewed-by: NDaniel P. Berrangé <berrange@redhat.com>
Message-Id: <20190726210706.24440-2-stefanb@linux.ibm.com>

Creating an 'exp' output file, but never comparing it against the
actual output, does not actually constitute testing the output. :)

Fixes: 280a2b41Signed-off-by: NEric Blake <eblake@redhat.com>