Minor semantic change - allow domain xml to be generated in place
within a larger buffer, rather than having to go through a
temporary string.

* src/conf/domain_conf.c (virDomainDefFormatInternal): Add
parameter.
(virDomainDefFormat, virDomainObjFormat): Update callers.

Migration is another case of stranding metadata.  And since
snapshot metadata is arbitrarily large, there's no way to
shoehorn it into the migration cookie of migration v3.

This patch consolidates two existing locations for migration
validation into one helper function, then enhances that function
to also do the new checks.  If we could always trust the source
to validate migration, then the destination would not have to
do anything; but since older servers that did not do checking
can migrate to newer destinations, we have to repeat some of
the same checks on the destination; meanwhile, we want to
detect failures as soon as possible.  With migration v2, this
means that validation will reject things at Prepare on the
destination if the XML exposes the problem, otherwise at Perform
on the source; with migration v3, this means that validation
will reject things at Begin on the source, or if the source
is old and the XML exposes the problem, then at Prepare on the
destination.

This patch is necessarily over-strict.  Once a later patch
properly handles auto-cleanup of snapshot metadata on the
death of a transient domain, then the only time we actually
need snapshots to prevent migration is when using the
--undefinesource flag on a persistent source domain.

It is possible to recreate snapshot metadata on the destination
with VIR_DOMAIN_SNAPSHOT_CREATE_REDEFINE and
VIR_DOMAIN_SNAPSHOT_CREATE_CURRENT.  But for now, that is limited,
since if we delete the snapshot metadata prior to migration,
then we won't know the name of the current snapshot to pass
along; and if we delete the snapshot metadata after migration
and use the v3 migration cookie to pass along the name of the
current snapshot, then we need a way to bypass the fact that
this patch refuses migration with snapshot metadata present.

So eventually, we may have to introduce migration protocol v4
that allows feature negotiation and an arbitrary number of
handshake exchanges, so as to pass as many rpc calls as needed
to transfer all the snapshot xml hierarchy.

But all of that is thoughts for the future; for now, the best
course of action is to quit early, rather than get into a
funky state of stale metadata; then relax restrictions later.

* src/qemu/qemu_migration.h (qemuMigrationIsAllowed): Make static.
* src/qemu/qemu_migration.c (qemuMigrationIsAllowed): Alter
signature, and allow checks for both outgoing and incoming.
(qemuMigrationBegin, qemuMigrationPrepareAny)
(qemuMigrationPerformJob): Update callers.

A nice benefit of deleting all snapshots at undefine time is that
you don't have to do any reparenting or subtree identification - since
everything goes, this is an O(n) process, whereas using multiple
virDomainSnapshotDelete calls would be O(n^2) or worse.  But it is
only doable for snapshot metadata, where we are in control of the
data being deleted; for the actual snapshots, there's too much
likelihood of something going wrong, and requiring even more API
calls to figure out what failed in the meantime, so callers are
better off deleting the snapshot data themselves one snapshot at
a time where they can deal with failures as they happen.

* src/qemu/qemu_driver.c (qemuDomainUndefineFlags): Honor new flags.

As more clients start to want to know this information, doing
a PATH stat walk and malloc for every client adds up.

We are only caching the location, not the capabilities, so even
if qemu-img is updated in the meantime, it will still probably
live in the same location.  So there is no need to worry about
clearing this particular cache.

* src/qemu/qemu_conf.h (qemud_driver): Add member.
* src/qemu/qemu_driver.c (qemudShutdown): Cleanup.
(qemuFindQemuImgBinary): Add an argument, and cache result.
(qemuDomainSnapshotForEachQcow2, qemuDomainSnapshotDiscard)
(qemuDomainSnapshotCreateInactive, qemuDomainSnapshotRevertInactive)
(qemuDomainSnapshotCreateXML, qemuDomainRevertToSnapshot): Update
callers.

Prepare for code sharing.  No semantic change.

* src/qemu/qemu_driver.c (qemuFindQemuImgBinary)
(qemuDomainSnapshotWriteMetadata)
(qemuDomainSnapshotDiscard)
(qemuDomainSnapshotForEachQcow2): Float up.
(qemuDomainSnapshotDiscardDescendant): Likewise, and rename...
(qemuDomainSnapshotDiscardAll): ...for generic use.
(qemuDomainSnapshotDelete): Update caller.

Just as leaving managed save metadata behind can cause problems
when creating a new domain that happens to collide with the name
of the just-deleted domain, the same is true of leaving any
snapshot metadata behind.  For safety sake, extend the semantic
change of commit b26a9fa9 to also cover snapshot metadata as a
reason to reject undefining an inactive domain.  A future patch
will make sure that shutdown of a transient domain automatically
deletes snapshot metadata (whether by destroy, shutdown, or
guest-initiated action).  Management apps of transient domains
should take care to capture xml of snapshots, if it is necessary
to recreate the snapshot metadata on a later transient domain
with the same name and uuid.

This also documents a new flag that hypervisors can choose to
support as a shortcut for taking care of the metadata as part of
the undefine process; however, nontrivial driver support for these
flags will be deferred to future patches.

Note that ESX and VBox can never be transient; therefore, they
do not have to worry about automatic cleanup after shutdown
(the persistent domain still remains); likewise they never
store snapshot metadata, so the undefine flag is trivial.
The nontrivial work remaining is thus in the qemu driver.

* include/libvirt/libvirt.h.in
(VIR_DOMAIN_UNDEFINE_SNAPSHOTS_METADATA): New flag.
* src/libvirt.c (virDomainUndefine, virDomainUndefineFlags):
Document new limitations and flag.
* src/esx/esx_driver.c (esxDomainUndefineFlags): Trivial
implementation.
* src/vbox/vbox_tmpl.c (vboxDomainUndefineFlags): Likewise.
* src/qemu/qemu_driver.c (qemuDomainUndefineFlags): Enforce
the limitations.

Redefining a qemu snapshot requires a bit of a tweak to the common
snapshot parsing code, but the end result is quite nice.

Be careful that redefinitions do not introduce circular parent
chains.  Also, we don't want to allow conversion between online
and offline existing snapshots.  We could probably do some more
validation for snapshots that don't already exist to make sure
they are even feasible, by parsing qemu-img output, but that
can come later.

* src/conf/domain_conf.h (virDomainSnapshotParseFlags): New
internal flags.
* src/conf/domain_conf.c (virDomainSnapshotDefParseString): Alter
signature to take internal flags.
* src/esx/esx_driver.c (esxDomainSnapshotCreateXML): Update caller.
* src/vbox/vbox_tmpl.c (vboxDomainSnapshotCreateXML): Likewise.
* src/qemu/qemu_driver.c (qemuDomainSnapshotCreateXML): Support
new public flags.

Supporting NO_METADATA on snapshot creation is interesting - we must
still return a valid opaque snapshot object, but the user can't get
anything out of it (unless we add a virDomainSnapshotGetName()),
since it is no longer registered with the domain.

Also, virsh now tries to query for secure xml, in anticipation of
when we store <domain> xml inside <domainsnapshot>; for now, we
can trivially support it, since we have nothing secure.

* src/qemu/qemu_driver.c (qemuDomainSnapshotCreateXML): Support
new flag.
(qemuDomainSnapshotGetXMLDesc): Trivially support VIR_DOMAIN_XML_SECURE.

The first two flags are essential for being able to replicate
snapshot hierarchies across multiple hosts, which will come in
handy for supervised migrations.  It also allows a management app
to take a snapshot of a transient domain, save the metadata, stop
the domain, recreate a new transient domain by the same name,
redefine the snapshot, then revert to it.

This is not quite as convenient as leaving the metadata behind
after a domain is no longer around, but doing that has a few
problems: 1. the libvirt API can only delete snapshot metadata
if there is a valid domain handle to use to get to that snapshot
object - if stale data is left behind without a domain, there is
no way to request that the data be cleaned up. 2. creating a new
domain with the same name but different uuid than the older
domain where a snapshot existed cannot use the older snapshot
data; this risks confusing libvirt, and forbidding the stale
data is similar to the recent patch to forbid stale managed save.

The first two flags might be useful on hypervisors with no metadata,
but only for modifying the notion of the current snapshot;
however, I don't know how to do that for ESX or VBox.

The third flag is a convenience option, to combine a creation with
a delete metadata into one step.  It is trivial for hypervisors
with no metadata.

The qemu changes will be involved enough to warrant a separate patch.

* include/libvirt/libvirt.h.in
(VIR_DOMAIN_SNAPSHOT_CREATE_REDEFINE)
(VIR_DOMAIN_SNAPSHOT_CREATE_CURRENT)
(VIR_DOMAIN_SNAPSHOT_CREATE_NO_METADATA): New flags.
* src/libvirt.c (virDomainSnapshotCreateXML): Document them, and
enforce mutual exclusion.
* src/esx/esx_driver.c (esxDomainSnapshotCreateXML): Trivial
implementation.
* src/vbox/vbox_tmpl.c (vboxDomainSnapshotCreateXML): Likewise.
* docs/formatsnapshot.html.in: Document re-creation.

Filtering for roots is pretty easy to do.

* src/conf/domain_conf.h (virDomainSnapshotObjListGetNames)
(virDomainSnapshotObjListNum): Update prototype.
* src/conf/domain_conf.c (virDomainSnapshotObjListCopyNames)
(virDomainSnapshotObjListGetNames, virDomainSnapshotObjListCount)
(virDomainSnapshotObjListNum): Support filtering.
* src/qemu/qemu_driver.c (qemuDomainSnapshotNum)
(qemuDomainSnapshotListNames): Update callers.

To make it easier to know when undefine will fail because of existing
snapshot metadata, we need to know how many snapshots have metadata.

Also, it is handy to filter the list of snapshots to just those that
have no parents; document that flag now, but implement it in later patches.

* include/libvirt/libvirt.h.in (VIR_DOMAIN_SNAPSHOT_LIST_ROOTS)
(VIR_DOMAIN_SNAPSHOT_LIST_METADATA): New flags.
* src/libvirt.c (virDomainSnapshotNum)
(virDomainSnapshotListNames): Document them.
* src/esx/esx_driver.c (esxDomainSnapshotNum)
(esxDomainSnapshotListNames): Implement trivial flag.
* src/vbox/vbox_tmpl.c (vboxDomainSnapshotNum)
(vboxDomainSnapshotListNames): Likewise.
* src/qemu/qemu_driver.c (qemuDomainSnapshotNum)
(qemuDomainSnapshotListNames): Likewise.

Adding this was trivial compared to the previous patch for fixing
qemu snapshot deletion in the first place.

* src/qemu/qemu_driver.c (qemuDomainSnapshotDiscard): Add
parameter.
(qemuDomainSnapshotDiscardDescendant, qemuDomainSnapshotDelete):
Update callers.

A future patch will make it impossible to remove a domain if it
would leave behind any libvirt-tracked metadata about snapshots,
since stale metadata interferes with a new domain by the same name.
But requiring snaphot contents to be deleted before removing a
domain is harsh; with qemu, qemu-img can still make use of the
contents after the libvirt domain is gone.  Therefore, we need
an option to get rid of libvirt tracking information, but not
the actual contents.  For hypervisors that do not track any
metadata in libvirt, the implementation is trivial; all remaining
hypervisors (really, just qemu) will be dealt with separately.

* include/libvirt/libvirt.h.in
(VIR_DOMAIN_SNAPSHOT_DELETE_METADATA_ONLY): New flag.
* src/libvirt.c (virDomainSnapshotDelete): Document it.
* src/esx/esx_driver.c (esxDomainSnapshotDelete): Trivially
supported when there is no libvirt metadata.
* src/vbox/vbox_tmpl.c (vboxDomainSnapshotDelete): Likewise.

Similar to the last patch in isolating the filtering from the
client actions, so that clients don't have to reinvent the
filtering.

* src/conf/domain_conf.h (virDomainSnapshotForEachChild): New
prototype.
* src/libvirt_private.syms (domain_conf.h): Export it.
* src/conf/domain_conf.c (virDomainSnapshotActOnChild)
(virDomainSnapshotForEachChild): New functions.
(virDomainSnapshotCountChildren): Delete.
(virDomainSnapshotHasChildren): Simplify.
* src/qemu/qemu_driver.c (qemuDomainSnapshotReparentChildren)
(qemuDomainSnapshotDelete): Likewise.

Deleting a snapshot and all its descendants had problems with
tracking the current snapshot.  The deletion does not necessarily
proceed in depth-first order, so a parent could be deleted
before a child, wreaking havoc on passing the notion of the
current snapshot to the parent.  Furthermore, even if traversal
were depth-first, doing multiple file writes to pass current up
the chain one snapshot at a time is wasteful, comparing to a
single update to the current snapshot at the end of the algorithm.

* src/qemu/qemu_driver.c (snap_remove): Add field.
(qemuDomainSnapshotDiscard): Add parameter.
(qemuDomainSnapshotDiscardDescendant): Adjust accordingly.
(qemuDomainSnapshotDelete): Properly reset current.

This one's nasty.  Ever since we fixed virHashForEach to prevent
nested hash iterations for safety reasons (commit fba550f6),
virDomainSnapshotDelete with VIR_DOMAIN_SNAPSHOT_DELETE_CHILDREN
has been broken for qemu: it deletes children, while leaving
grandchildren intact but pointing to a no-longer-present parent.
But even before then, the code would often appear to succeed to
clean up grandchildren, but risked memory corruption if you have
a large and deep hierarchy of snapshots.

For acting on just children, a single virHashForEach is sufficient.
But for acting on an entire subtree, it requires iteration; and
since we declared recursion as invalid, we have to switch to a
while loop.  Doing this correctly requires quite a bit of overhaul,
so I added a new helper function to isolate the algorithm from the
actions, so that callers do not have to reinvent the iteration.

Note that this _still_ does not handle CHILDREN correctly if one
of the children is the current snapshot; that will be next.

* src/conf/domain_conf.h (_virDomainSnapshotDef): Add mark.
(virDomainSnapshotForEachDescendant): New prototype.
* src/libvirt_private.syms (domain_conf.h): Export it.
* src/conf/domain_conf.c (virDomainSnapshotMarkDescendant)
(virDomainSnapshotActOnDescendant)
(virDomainSnapshotForEachDescendant): New functions.
* src/qemu/qemu_driver.c (qemuDomainSnapshotDiscardChildren):
Replace...
(qemuDomainSnapshotDiscardDescenent): ...with callback that
doesn't nest hash traversal.
(qemuDomainSnapshotDelete): Use new function.

Each snapshot lookup was iterating over the entire hash table, O(n),
instead of honing in directly on the hash key, amortized O(1).

Besides, fixing this means that virDomainSnapshotFindByName can now
be used inside another virHashForeach iteration (without this patch,
attempts to lookup a snapshot by name during a hash iteration will
fail due to nested iteration).

* src/conf/domain_conf.c (virDomainSnapshotFindByName): Simplify.
(virDomainSnapshotObjListSearchName): Delete unused function.

For a system checkpoint of a running or paused domain, it's fairly
easy to honor new flags for altering which state to use after the
revert.  For an inactive snapshot, the revert has to be done while
there is no qemu process, so do back-to-back transitions; this also
lets us revert to inactive snapshots even for transient domains.

* src/qemu/qemu_driver.c (qemuDomainRevertToSnapshot): Support new
flags.

Commit 5e47785b broke reverts to offline system checkpoint snapshots
with older qemu, since there is no longer any code path to use
qemu -loadvm on next boot.  Meanwhile, reverts to offline system
checkpoints have been broken for newer qemu, both before and
after that commit, since -loadvm no longer works to revert to
disk state without accompanying vm state.  Fix both of these by
using qemu-img to revert disk state.

Meanwhile, consolidate the (now 3) clients of a qemu-img iteration
over all disks of a VM into one function, so that any future
algorithmic fixes to the FIXMEs in that function after partial
loop iterations are dealt with at once.  That does mean that this
patch doesn't handle partial reverts very well, but we're not
making the situation any worse in this patch.

* src/qemu/qemu_driver.c (qemuDomainRevertToSnapshot): Use
qemu-img rather than 'qemu -loadvm' to revert to offline snapshot.
(qemuDomainSnapshotRevertInactive): New helper.
(qemuDomainSnapshotCreateInactive): Factor guts...
(qemuDomainSnapshotForEachQcow2): ...into new helper.
(qemuDomainSnapshotDiscard): Use it.

If you take a checkpoint snapshot of a running domain, then pause
qemu, then restore the snapshot, the result should be a running
domain, but the code was leaving things paused.  Furthermore, if
you take a checkpoint of a paused domain, then run, then restore,
there was a brief but non-deterministic window of time where the
domain was running rather than paused.  Fix both of these
discrepancies by always pausing before restoring.

Also, check that the VM is active every time lock is dropped
between two monitor calls.

Finally, straighten out the events that get emitted on each
transition.

* src/qemu/qemu_driver.c (qemuDomainRevertToSnapshot): Always
pause before reversion, and improve events.

Implement the new running/paused overrides for saved state management.

Unfortunately, for virDomainSaveImageDefineXML, the saved state
updates are write-only - I don't know of any way to expose a way
to query the current run/pause setting of an existing save image
file to the user without adding a new API or modifying the domain
xml of virDomainSaveImageGetXMLDesc to include a new element to
reflect the state bit encoded into the save image.  However, I
don't think this is a show-stopper, since the API is designed to
leave the state bit alone unless an explicit flag is used to
change it.

* src/qemu/qemu_driver.c (qemuDomainSaveInternal)
(qemuDomainSaveImageOpen): Adjust signature.
(qemuDomainSaveFlags, qemuDomainManagedSave)
(qemuDomainRestoreFlags, qemuDomainSaveImageGetXMLDesc)
(qemuDomainSaveImageDefineXML, qemuDomainObjRestore): Adjust
callers.

While it is nice that snapshots and saved images remember whether
the domain was running or paused, sometimes the restoration phase
wants to guarantee a particular state (paused to allow hot-plugging,
or running without needing to call resume).  This introduces new
flags to allow the control, and a later patch will implement the
flags for qemu.

* include/libvirt/libvirt.h.in (VIR_DOMAIN_SAVE_RUNNING)
(VIR_DOMAIN_SAVE_PAUSED, VIR_DOMAIN_SNAPSHOT_REVERT_RUNNING)
(VIR_DOMAIN_SNAPSHOT_REVERT_PAUSED): New flags.
* src/libvirt.c (virDomainSaveFlags, virDomainRestoreFlags)
(virDomainManagedSave, virDomainSaveImageDefineXML)
(virDomainRevertToSnapshot): Document their use, and enforce
mutual exclusion.

There are two classes of management apps that track events - one
that only cares about on/off (and only needs to track EVENT_STARTED
and EVENT_STOPPED), and one that cares about paused/running (also
tracks EVENT_SUSPENDED/EVENT_RESUMED).  To keep both classes happy,
any transition that can go from inactive to paused must emit two
back-to-back events - one for started and one for suspended (since
later resuming of the domain will only send RESUMED, but the first
class isn't tracking that).

This also fixes a bug where virDomainCreateWithFlags with the
VIR_DOMAIN_START_PAUSED flag failed to start paused when restoring
from a managed save image.

* include/libvirt/libvirt.h.in (VIR_DOMAIN_EVENT_SUSPENDED_RESTORED)
(VIR_DOMAIN_EVENT_SUSPENDED_FROM_SNAPSHOT)
(VIR_DOMAIN_EVENT_RESUMED_FROM_SNAPSHOT): New sub-events.
* src/qemu/qemu_driver.c (qemuDomainRevertToSnapshot): Use them.
(qemuDomainSaveImageStartVM): Likewise, and add parameter.
(qemudDomainCreate, qemuDomainObjStart): Send suspended event when
starting paused.
(qemuDomainObjRestore): Add parameter.
(qemuDomainObjStart, qemuDomainRestoreFlags): Update callers.
* examples/domain-events/events-c/event-test.c
(eventDetailToString): Map new detail strings.

- create a new "redirdev" element for this purpose

QEMU uses USB bus name "usb.0" when using the legacy -usb argument.
If we want to allow USB devices to specify their addresses with legacy
-usb, we should either in case of legacy bus name drop the 0 from the
address bus, or just drop the 0 from device id. This patch does the
later.

Another solution would be to permit addressing on non-legacy USB
controllers only.

Applies only to piix3 and check if piix3 controller is on correct
address, or report error

So that devices can be attached to hubs. Example, to attach to first
port of a usb-hub on port 1.

      <hub type='usb'>
         <address type='usb' bus='0' port='1'/>
      </hub>

      <input type='mouse' type='usb'>
         <address type='usb' bus='0' port='1.1'/>
      </hub>

also add a test entry

domain parsing and serialization code, qemu driver backend and
a couple of test

Companion controllers take an extra 'master' attribute to associate
them.

Also add tests for this

Expand the domain and the QEmu driver code
Adds a couple of tests

The model by default is piix3-uchi.

Example:
<controller type='usb' index='0' model='ich9-ehci'/>

List is: piix3-uhci piix4-uhci ehci ich9-ehci1 ich9-uhci1 ich9-uhci2
ich9-uhci3 vt82c686b-uhci pci-ohci

Since we are about to add USB controller support let's remove the
ambiguity

Commit 6766ff10 introduced a corner case bug with snapshot creation:
if a snapshot is created, but then we hit OOM while trying to
create the return value of the function, then we have polluted the
internal directory with the snapshot metadata with no way to clean
it up from the running libvirtd.

* src/qemu/qemu_driver.c (qemuDomainSnapshotCreateXML): Don't
write metadata file on OOM condition.

Newer QEMU introduced cache=directsync for -drive, this patchset
is to expose it in libvirt layer.

  * Introduced a new QEMU capability flag ($prefix_CACHE_DIRECTSYNC),
    As even $prefix_CACHE_V2 is set, we can't known if directsync
    is supported.

This patch adds the ability to make the filesystem for a filesystem
pool during a pool build.

The patch adds two new flags, no overwrite and overwrite, to control
when mkfs gets executed.  By default, the patch preserves the
current behavior, i.e., if no flags are specified, pool build on a
filesystem pool only makes the directory on which the filesystem
will be mounted.

If the no overwrite flag is specified, the target device is checked
to determine if a filesystem of the type specified in the pool is
present.  If a filesystem of that type is already present, mkfs is
not executed and the build call returns an error.  Otherwise, mkfs
is executed and any data present on the device is overwritten.

If the overwrite flag is specified, mkfs is always executed, and any
existing data on the target device is overwritten unconditionally.

There is a goto before "conn" is initialized.

Several users have reported problems with 'virsh start' failing because
it was encountering a managed save situation where the managed save file
was incomplete.  Be more robust to this by using two different magic
numbers, so that newer libvirt can gracefully handle an incomplete file
differently than a complete one, while older libvirt will at least fail
up front rather than trying to load only to have qemu fail at the end.

Managed save is a convenience - it exists to preserve as much state
as possible; if the state was not preserved, it is reasonable to just
log that fact, then proceed with a fresh boot.  On the other hand,
user saves are under user control, so we must fail, but by making
the failure message distinct, the user can better decide how to handle
the situation of an incomplete save file.

* src/qemu/qemu_driver.c (QEMUD_SAVE_PARTIAL): New define.
(qemuDomainSaveInternal): Use it to mark incomplete images.
(qemuDomainSaveImageOpen, qemuDomainObjRestore): Add parameter
that controls what to do with partial images.
(qemuDomainRestoreFlags, qemuDomainSaveImageGetXMLDesc)
(qemuDomainSaveImageDefineXML, qemuDomainObjStart): Update callers.
Based on an initial idea by Osier Yang.