If a connection to destination host is lost during peer-to-peer
migration (because keepalive protocol timed out), we won't be able to
finish the migration and it doesn't make sense to wait for qemu to
transmit all data. This patch automatically cancels such migration
without waiting for virDomainAbortJob to be called.

Rename the macvtap.c file to virnetdevmacvlan.c to reflect its
functionality. Move the port profile association code out into
virnetdevvportprofile.c. Make the APIs available unconditionally
to callers

* src/util/macvtap.h: rename to src/util/virnetdevmacvlan.h,
* src/util/macvtap.c: rename to src/util/virnetdevmacvlan.c
* src/util/virnetdevvportprofile.c, src/util/virnetdevvportprofile.h:
  Pull in vport association code
* src/Makefile.am, src/conf/domain_conf.h, src/qemu/qemu_conf.c,
  src/qemu/qemu_conf.h, src/qemu/qemu_driver.c: Update include
  paths & remove conditional compilation

In preparation for code re-organization, rename the Macvtap
management APIs to have the following patterns

  virNetDevMacVLanXXXXX     - macvlan/macvtap interface management
  virNetDevVPortProfileXXXX - virtual port profile management

* src/util/macvtap.c, src/util/macvtap.h: Rename APIs
* src/conf/domain_conf.c, src/network/bridge_driver.c,
  src/qemu/qemu_command.c, src/qemu/qemu_command.h,
  src/qemu/qemu_driver.c, src/qemu/qemu_hotplug.c,
  src/qemu/qemu_migration.c, src/qemu/qemu_process.c,
  src/qemu/qemu_process.h: Update for renamed APIs

This reverts commit ef1065cf; see also this bug report:
https://bugzilla.redhat.com/show_bug.cgi?id=751900

In qemu 0.15.1 and earlier, during migration to file, the
qemu_savevm_state_begin and qemu_savevm_state_iterate methods
will both process as much migration data as possible until either

  1. The file descriptor returns EAGAIN
  2. The bandwidth rate limit is reached

If we set the rate limit to ULONG_MAX, test 2 never becomes true. We're
passing a plain file descriptor to QEMU and POSIX does not support EAGAIN on
regular files / block devices, so test 1 never becomes true either.

In the 'virsh save --bypass-cache' case, we pass a pipe instead of a
regular fd, but using a pipe adds I/O overhead, so always passing a
pipe just so qemu can see EAGAIN doesn't seem nice.

The ultimate fix needs to come from qemu - background migration must
respect asynchronous abort requests, or else periodically return
control to the main handling loop without an EAGAIN and without
waiting to hit an insanely large amount of data.  But until a
version of qemu is fixed to support "unlimited" data rates while
still allowing cancellation, the best we can do is avoid the
automatic use of unlimited rates from within libvirt (users can
still explicitly change the migration rates, if they are aware that
they are giving up the ability to cancel a job).

Reverting the lone use of QEMU_DOMAIN_FILE_MIG_BANDWIDTH_MAX is
the simplest patch; this slows migration back down to a default
32M/sec cap, but also ensures that the main qemu processing loop
will still be responsive to cancellation requests.  Hopefully
upstream qemu will provide us a means of safely using unlimited
speed, including a runtime probe of that capability.

* src/qemu/qemu_migration.c (qemuMigrationToFile): Revert attempt
to use unlimited migration bandwidth when migrating to file.
Signed-off-by: NDaniel Veillard <veillard@redhat.com>
Signed-off-by: NEric Blake <eblake@redhat.com>

- changed some return 1's to return -1
- changed if (rc) error checks to if (rc < 0)
- fixed some other minor convention violations

I might have missed some. Can fix in another patch or can respin
Signed-off-by: NRoopa Prabhu <roprabhu@cisco.com>
Reported-by: NEric Blake <eblake@redhat.com>
Reported-by: NLaine Stump <laine@laine.org>
Signed-off-by: NEric Blake <eblake@redhat.com>

<domainsnapshot> is the first public instance of <domain> being
used as a sub-element, although we have two other private uses
(runtime state, and migration cookie).  Although indentation has
no effect on XML parsing, using it makes the output more consistent.

This uses virBuffer auto-indentation to obtain the effect, for all
but the portions of <domain> that are not generated a line at a
time into the same virBuffer.  Further patches will clean up the
remaining problems.

* src/conf/domain_conf.h (virDomainDefFormatInternal): New prototype.
* src/conf/domain_conf.c (virDomainDefFormatInternal): Export.
(virDomainObjFormat, virDomainSnapshotDefFormat): Update callers.
* src/libvirt_private.syms (domain_conf.h): Add new export.
* src/qemu/qemu_migration.c (qemuMigrationCookieXMLFormat): Use
new function.
(qemuMigrationCookieXMLFormatStr): Update caller.

Explicitly disallow conflicts between domain name from dxml and dname.

Detected by Coverity.  Leak introduced in commit 72de0d28.

* src/qemu/qemu_migration.c (qemuMigrationCookieGraphicsXMLParse):
Clean up on success.

Destination libvirtd remembers the original name in the prepare phase
and clears it in the finish phase. The original name is used when
comparing domain name in migration cookie.

* src/qemu/qemu_migration.c: if 'vmdef' is NULL, the function
  virDomainSaveConfig still dereferences it, it doesn't make
  sense, so should add return value check to make sure 'vmdef'
  is non-NULL before calling virDomainSaveConfig, in addition,
  in order to debug later, also should record error information
  into log.
Signed-off-by: NAlex Jia <ajia@redhat.com>

If migration failed in Prepare phase after virDomainAssignDef and before
a job was started, the domain object was not properly removed.

as they are not used with debugging turned off.

If a domain has inactive XML we want to transfer it to destination
when migrating with VIR_MIGRATE_PERSIST_DEST. In order to harm
the migration protocol as least as possible, a optional cookie was
chosen.

Commit 282fe1f0 documented that transient domains will auto-delete
any snapshot metadata when the last reference to the domain is
removed, and that management apps are in charge of grabbing any
snapshot metadata prior to that point.  However, this was not
actually implemented for qemu until now.

* src/qemu/qemu_driver.c (qemudDomainCreate)
(qemuDomainDestroyFlags, qemuDomainSaveInternal)
(qemudDomainCoreDump, qemuDomainRestoreFlags, qemudDomainDefine)
(qemuDomainUndefineFlags, qemuDomainMigrateConfirm3)
(qemuDomainRevertToSnapshot): Clean up snapshot metadata.
* src/qemu/qemu_migration.c (qemuMigrationPrepareAny)
(qemuMigrationPerformJob, qemuMigrationPerformPhase)
(qemuMigrationFinish): Likewise.
* src/qemu/qemu_process.c (qemuProcessHandleMonitorEOF)
(qemuProcessReconnect, qemuProcessReconnectHelper)
(qemuProcessAutoDestroyDom): Likewise.

Adjust qemuMigrationRun() to use migMaxBandwidth in qemuDomainObjPrivate
structure when setting qemu migration speed.  Caller-specified 'resource'
parameter overrides migMaxBandwidth.

The qemu migration speed default is 32MiB/s as defined in migration.c

/* Migration speed throttling */
static int64_t max_throttle = (32 << 20);

There's no need to throttle migration when targeting a file, so set migration
speed to unlimited prior to migration, and restore to libvirt default value
after migration.

Default units is MB for migrate_set_speed monitor command, so
(INT64_MAX / (1024 * 1024)) is used for unlimited migration speed.

Tested with both json and text monitors.

Commit 498d7833 cleans up some of virtual file names for parsing strings
in memory. This patch cleans up (hopefuly) the rest forgotten by the
first patch.

This patch also changes all of the previously modified "filenames" to
valid URI's replacing spaces for underscores.

Changes to v1:
- Replace all spaces for underscores, so that the strings form valid
  URI's
- Replace spaces in places changed by commit 498d7833

Migration is another case of stranding metadata.  And since
snapshot metadata is arbitrarily large, there's no way to
shoehorn it into the migration cookie of migration v3.

This patch consolidates two existing locations for migration
validation into one helper function, then enhances that function
to also do the new checks.  If we could always trust the source
to validate migration, then the destination would not have to
do anything; but since older servers that did not do checking
can migrate to newer destinations, we have to repeat some of
the same checks on the destination; meanwhile, we want to
detect failures as soon as possible.  With migration v2, this
means that validation will reject things at Prepare on the
destination if the XML exposes the problem, otherwise at Perform
on the source; with migration v3, this means that validation
will reject things at Begin on the source, or if the source
is old and the XML exposes the problem, then at Prepare on the
destination.

This patch is necessarily over-strict.  Once a later patch
properly handles auto-cleanup of snapshot metadata on the
death of a transient domain, then the only time we actually
need snapshots to prevent migration is when using the
--undefinesource flag on a persistent source domain.

It is possible to recreate snapshot metadata on the destination
with VIR_DOMAIN_SNAPSHOT_CREATE_REDEFINE and
VIR_DOMAIN_SNAPSHOT_CREATE_CURRENT.  But for now, that is limited,
since if we delete the snapshot metadata prior to migration,
then we won't know the name of the current snapshot to pass
along; and if we delete the snapshot metadata after migration
and use the v3 migration cookie to pass along the name of the
current snapshot, then we need a way to bypass the fact that
this patch refuses migration with snapshot metadata present.

So eventually, we may have to introduce migration protocol v4
that allows feature negotiation and an arbitrary number of
handshake exchanges, so as to pass as many rpc calls as needed
to transfer all the snapshot xml hierarchy.

But all of that is thoughts for the future; for now, the best
course of action is to quit early, rather than get into a
funky state of stale metadata; then relax restrictions later.

* src/qemu/qemu_migration.h (qemuMigrationIsAllowed): Make static.
* src/qemu/qemu_migration.c (qemuMigrationIsAllowed): Alter
signature, and allow checks for both outgoing and incoming.
(qemuMigrationBegin, qemuMigrationPrepareAny)
(qemuMigrationPerformJob): Update callers.

In a SELinux or root-squashing NFS environment, libvirt has to go
through some hoops to create a new file that qemu can then open()
by name.  Snapshots are a case where we want to guarantee an empty
file that qemu can open; also, reopening a save file to convert it
from being marked partial to complete requires a reopen to avoid
O_DIRECT headaches.  Refactor some existing code to make it easier
to reuse in later patches.

* src/qemu/qemu_migration.h (qemuMigrationToFile): Drop parameter.
* src/qemu/qemu_migration.c (qemuMigrationToFile): Let cgroup do
the stat, rather than asking caller to do it and pass info down.
* src/qemu/qemu_driver.c (qemuOpenFile): New function, pulled from...
(qemuDomainSaveInternal): ...here.
(doCoreDump, qemuDomainSaveImageOpen): Use it here as well.

Commit 32617617 made it possible to use pipes instead of sockets
for outgoing tunneled migration; however, it caused a regression
because the pipe was never given a SELinux label.

* src/qemu/qemu_migration.c (doTunnelMigrate): Label outgoing pipe.

When a user migrates a domain by command as

libvirt saves vm's domain XML config in destination host after migration.
But it saves vm->def. Then, the saved XML contains some garbage.

  <domain type='kvm' id='50'>
                     ^^^^^^^^
  ...
   <console type='pty' tty='/dev/pts/5'>
                       ^^^^^^^^^^^^^^^^^

Avoid saving unnecessary things by saving persistent vm definition.

Changing the current vm, and writing that change to the file
system, all before a new qemu starts, is risky; it's hard to
roll back if starting the new qemu fails for some reason.
Instead of abusing vm->current_snapshot and making the command
line generator decide whether the current snapshot warrants
using -loadvm, it is better to just directly pass a snapshot all
the way through the call chain if it is to be loaded.

This frees up the last use of snapshot->def->active for qemu's
use, so the next patch can repurpose that field for tracking
which snapshot is current.

* src/qemu/qemu_command.c (qemuBuildCommandLine): Don't use active
field of snapshot.
* src/qemu/qemu_process.c (qemuProcessStart): Add a parameter.
* src/qemu/qemu_process.h (qemuProcessStart): Update prototype.
* src/qemu/qemu_migration.c (qemuMigrationPrepareAny): Update
callers.
* src/qemu/qemu_driver.c (qemudDomainCreate)
(qemuDomainSaveImageStartVM, qemuDomainObjStart)
(qemuDomainRevertToSnapshot): Likewise.
(qemuDomainSnapshotSetCurrentActive)
(qemuDomainSnapshotSetCurrentInactive): Delete unused functions.

* src/qemu/qemu_migration.c: avoid dead 'ret' assignment and silence
  clang warning.

Detected by ccc-analyzer:

  CC     libvirt_driver_qemu_la-qemu_migration.lo
qemu/qemu_migration.c:2046:5: warning: Value stored to 'ret' is never read
    ret = qemuMigrationConfirm(driver, sconn, vm,
    ^     ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

Repetitive patterns should be factored.  The sign of a good
factorization is a change that kills 5x more lines than it adds :)

* src/conf/domain_conf.c (virDomainDeviceDefParse)
(virDomainSnapshotDefParseString): Use new convenience macros.
* src/conf/storage_conf.c (virStoragePoolDefParseSourceString):
Likewise.
* src/cpu/cpu.c (cpuCompareXML, cpuBaselineXML): Likewise.
* src/esx/esx_vi.c (esxVI_Context_Execute): Likewise.
* src/qemu/qemu_migration.c (qemuMigrationCookieXMLParseStr):
Likewise.
* src/security/virt-aa-helper.c (caps_mockup): Likewise.
* src/test/test_driver.c (testOpenFromFile): Likewise.
* tests/cputest.c (cpuTestLoadXML, cpuTestLoadMultiXML):
Likewise.
* tools/virsh.c (cmdFreecell, makeCloneXML, cmdVNCDisplay)
(cmdTTYConsole, cmdDetachInterface, cmdDetachDisk)
(cmdSnapshotCreate, cmdSnapshotCreateAs, cmdSnapshotCurrent)
(cmdSnapshotList, cmdSnapshotParent): Likewise.

Fix pointer for error message uri if domain migration fails.

BZ# 730244

By opening a connection to remote qemu process ourselves and passing the
socket to qemu we get much better errors than just "migration failed"
when the connection is opened by qemu.

The core of these two functions is very similar and most of it is even
exactly the same. Factor out the core functionality into a separate
function to remove code duplication and make further changes easier.

Warning detected by Coverity.  No need for the NULL check, and
removing it silences the warning without any semantic change.

* src/qemu/qemu_migration.c (qemuMigrationFinish): All entries to
endjob had non-NULL vm.

Currently, we attempt to run sync job and async job at the same time. It
means that the monitor commands for two jobs can be run in any order.

In the function qemuDomainObjEnterMonitorInternal():
    if (priv->job.active == QEMU_JOB_NONE && priv->job.asyncJob) {
        if (qemuDomainObjBeginNestedJob(driver, obj) < 0)
We check whether the caller is an async job by priv->job.active and
priv->job.asynJob. But when an async job is running, and a sync job is
also running at the time of the check, then priv->job.active is not
QEMU_JOB_NONE. So we cannot check whether the caller is an async job
in the function qemuDomainObjEnterMonitorInternal(), and must instead
put the burden on the caller to tell us when an async command wants
to do a nested job.

Once the burden is on the caller, then only async monitor enters need
to worry about whether the VM is still running; for sync monitor enter,
the internal return is always 0, so lots of ignore_value can be dropped.

* src/qemu/THREADS.txt: Reflect new rules.
* src/qemu/qemu_domain.h (qemuDomainObjEnterMonitorAsync): New
prototype.
* src/qemu/qemu_process.h (qemuProcessStartCPUs)
(qemuProcessStopCPUs): Add parameter.
* src/qemu/qemu_migration.h (qemuMigrationToFile): Likewise.
(qemuMigrationWaitForCompletion): Make static.
* src/qemu/qemu_domain.c (qemuDomainObjEnterMonitorInternal): Add
parameter.
(qemuDomainObjEnterMonitorAsync): New function.
(qemuDomainObjEnterMonitor, qemuDomainObjEnterMonitorWithDriver):
Update callers.
* src/qemu/qemu_driver.c (qemuDomainSaveInternal)
(qemudDomainCoreDump, doCoreDump, processWatchdogEvent)
(qemudDomainSuspend, qemudDomainResume, qemuDomainSaveImageStartVM)
(qemuDomainSnapshotCreateActive, qemuDomainRevertToSnapshot):
Likewise.
* src/qemu/qemu_process.c (qemuProcessStopCPUs)
(qemuProcessFakeReboot, qemuProcessRecoverMigration)
(qemuProcessRecoverJob, qemuProcessStart): Likewise.
* src/qemu/qemu_migration.c (qemuMigrationToFile)
(qemuMigrationWaitForCompletion, qemuMigrationUpdateJobStatus)
(qemuMigrationJobStart, qemuDomainMigrateGraphicsRelocate)
(doNativeMigrate, doTunnelMigrate, qemuMigrationPerformJob)
(qemuMigrationPerformPhase, qemuMigrationFinish)
(qemuMigrationConfirm): Likewise.
* src/qemu/qemu_hotplug.c: Drop unneeded ignore_value.

Once it's plugged in, the <listen> element will be an optional
replacement for the "listen" attribute that graphics elements already
have. If the <listen> element is type='address', it will have an
attribute called 'address' which will contain an IP address or dns
name that the guest's display server should listen on. If, however,
type='network', the <listen> element should have an attribute called
'network' that will be set to the name of a network configuration to
get the IP address from.

* docs/schemas/domain.rng: updated to allow the <listen> element

* docs/formatdomain.html.in: document the <listen> element and its
  attributes.

* src/conf/domain_conf.[hc]:

  1) The domain parser, formatter, and data structure are modified to
     support 0 or more <listen> subelements to each <graphics>
     element. The old style "legacy" listen attribute is also still
     accepted, and will be stored internally just as if it were a
     separate <listen> element. On output (i.e. format), the address
     attribute of the first <listen> element of type 'address' will be
     duplicated in the legacy "listen" attribute of the <graphic>
     element.

  2) The "listenAddr" attribute has been removed from the unions in
     virDomainGRaphicsDef for graphics types vnc, rdp, and spice.
     This attribute is now in the <listen> subelement (aka
     virDomainGraphicsListenDef)

  3) Helper functions were written to provide simple access
     (both Get and Set) to the listen elements and their attributes.

* src/libvirt_private.syms: export the listen helper functions

* src/qemu/qemu_command.c, src/qemu/qemu_hotplug.c,
  src/qemu/qemu_migration.c, src/vbox/vbox_tmpl.c,
  src/vmx/vmx.c, src/xenxs/xen_sxpr.c, src/xenxs/xen_xm.c

  Modify all these files to use the listen helper functions rather
  than directly referencing the (now missing) listenAddr
  attribute. There can be multiple <listen> elements to a single
  <graphics>, but the drivers all currently only support one, so all
  replacements of direct access with a helper function indicate index
  "0".

* tests/* - only 3 of these are new files added explicitly to test the
  new <listen> element. All the others have been modified to reflect
  the fact that any legacy "listen" attributes passed in to the domain
  parse will be saved in a <listen> element (i.e. one of the
  virDomainGraphicsListenDefs), and during the domain format function,
  both the <listen> element as well as the legacy attributes will be
  output.

This doesn't abort migration job in any phase, yet.

Call qemu monitor command directly within a special job that is only
allowed during outgoing migration.

Call qemu monitor command directly within a special job that is only
allowed during outgoing migration.

Like other query commands, this can now be called directly during
migration.

Like other query commands, this can now be called directly during
migration.