QEMU identified a race condition between the device state serialization
and the end of storage migration. Both QEMU and libvirt needs to be
updated to fix this.

Our migration work flow is modified so that after starting the migration
we to wait for QEMU to enter "pre-switchover", "postcopy-active", or
"completed" state. Once there, we cancel all block jobs as usual. But if
QEMU is in "pre-switchover", we need to resume the migration afterwards
and wait again for the real end (either "postcopy-active" or
"completed" state).

Old QEMU will just enter either "postcopy-active" or "completed"
directly, which is still correctly handled even by new libvirt. The
"pre-switchover" state will only be entered if QEMU supports it and the
pause-before-switchover capability was enabled. Thus all combinations of
libvirt and QEMU will work, but only new QEMU with new libvirt will
avoid the race condition.
Signed-off-by: NJiri Denemark <jdenemar@redhat.com>

This new capability enables a pause before device state serialization so
that we can finish all block jobs without racing with the end of the
migration. The pause is indicated by "pre-switchover" state. Once we're
done QEMU enters "device" migration state.

This patch just defines the new capability and QEMU migration states and
their mapping to our job states.
Signed-off-by: NJiri Denemark <jdenemar@redhat.com>

Instead of enumerating all states which need to be turned into
QEMU_DOMAIN_JOB_STATUS_FAILED (and failing to add all of them), it's
better to mention just the one which needs to be left alone.
Signed-off-by: NJiri Denemark <jdenemar@redhat.com>
Reviewed-by: NJohn Ferlan <jferlan@redhat.com>

Almost every failure in qemuMigrationRun while we are talking to QEMU
monitor results in a jump to exit_monitor label. The only exception is
removed by this patch.
Signed-off-by: NJiri Denemark <jdenemar@redhat.com>
Reviewed-by: NJohn Ferlan <jferlan@redhat.com>

The "ret" variable is used for storing the return value of a function
and should not be used as a temporary variable.
Signed-off-by: NJiri Denemark <jdenemar@redhat.com>
Reviewed-by: NJohn Ferlan <jferlan@redhat.com>

Merge cancel and cancelPostCopy sections with the generic error section,
where we can easily decide whether canceling the ongoing migration is
required.
Signed-off-by: NJiri Denemark <jdenemar@redhat.com>
Reviewed-by: NJohn Ferlan <jferlan@redhat.com>

Let cleanup only do things common to both failure and success paths and
move error handling code inside the new "error" section.
Signed-off-by: NJiri Denemark <jdenemar@redhat.com>
Reviewed-by: NJohn Ferlan <jferlan@redhat.com>

Some code which was supposed to be executed only when migration
succeeded was buried inside the cleanup code.
Signed-off-by: NJiri Denemark <jdenemar@redhat.com>
Reviewed-by: NJohn Ferlan <jferlan@redhat.com>

When adding a new job state it's useful to let the compiler complain
about places where we need to think about what to do with the new
state.
Signed-off-by: NJiri Denemark <jdenemar@redhat.com>
Reviewed-by: NJohn Ferlan <jferlan@redhat.com>

All calls to qemuMonitorGetMigrationCapability in QEMU driver are
replaced with qemuMigrationCapsGet.
Signed-off-by: NJiri Denemark <jdenemar@redhat.com>
Reviewed-by: NJohn Ferlan <jferlan@redhat.com>

We need to send allowReboot in the migration cookie to ensure the same
behavior of the virDomainSetLifecycleAction() API on the destination.

Consider this scenario:

    1. On the source the domain is started with:
        <on_poweroff>destroy</on_poweroff>
        <on_reboot>restart</on_reboot>
        <on_crash>destroy</on_crash>

    2. User calls an API to set "destroy" for <on_reboot>:
        <on_poweroff>destroy</on_poweroff>
        <on_reboot>destroy</on_reboot>
        <on_crash>destroy</on_crash>

    3. The guest is migrated to a different host

    4a. Without the allowReboot in the migration cookie the QEMU
        process on destination would be started with -no-reboot
        which would prevent using the virDomainSetLifecycleAction() API
        for the rest of the guest lifetime.

    4b. With the allowReboot in the migration cookie the QEMU process
        on destination is started without -no-reboot like it was started
        on the source host and the virDomainSetLifecycleAction() API
        continues to work.

The following patch adds a QEMU implementation of the
virDomainSetLifecycleAction() API and that implementation disallows
using the API if all actions are set to "destroy" because we add
"-no-reboot" on the QEMU command line.  Changing the lifecycle action
is in this case pointless because the QEMU process is always terminated.
Reviewed-by: NJohn Ferlan <jferlan@redhat.com>
Signed-off-by: NPavel Hrdina <phrdina@redhat.com>

When migration fails, QEMU may provide a description of the error in
the reply to query-migrate QMP command. We can fetch this error and use
it instead of the generic "unexpectedly failed" message.
Signed-off-by: NJiri Denemark <jdenemar@redhat.com>
Reviewed-by: NPavel Hrdina <phrdina@redhat.com>

Pass flags to the function rather than just whether we have incoming
migration. This also enforces correct startup policy for USB devices
when reverting from a snapshot.

qemuMigrationPrepareAny called multiple of the functions starting the
qemu process for incoming migration by adding the flags explicitly.
Extract them to a variable so that they can be easily used for other
calls or changed in the future.

Seeing a log message saying 'flags=93' is ambiguous & confusing unless
you happen to know that libvirt always prints flags as hex.  Change our
debug messages so that they always add a '0x' prefix when printing flags,
and '0' prefix when printing mode. A few other misc places gain a '0x'
prefix in error messages too.
Signed-off-by: NDaniel P. Berrange <berrange@redhat.com>

Drop all the monitor code necessary to do the downstream block jobs.
Reviewed-by: NEric Blake <eblake@redhat.com>

In case of real migration (not migrating to file on save, dump etc)
migration info is not complete at time qemu finishes migration
in normal (non postcopy) mode. We need to update disks stats,
downtime info etc. Thus let's not expose this job status as
completed.

To archive this let's set status to 'qemu completed' after
qemu reports migration is finished. It is not visible as complete
job to clients. Cookie code on confirm phase will finally turn
job into completed. As we don't need more things to do when
migrating to file status is set to 'completed' as before
in this case.
Signed-off-by: NJiri Denemark <jdenemar@redhat.com>

When getting job info in case mirror does not reach ready phase
fetch mirror stats from qemu. Otherwise mirror stats are already
saved in current job.
Signed-off-by: NJiri Denemark <jdenemar@redhat.com>

Instead of checking stat.status let's set status to migrating
as soon as migrate command is send (waiting for completion
is a good place too).
Signed-off-by: NJiri Denemark <jdenemar@redhat.com>

Setting status to none has little value - getting job status
will not return even elapsed time.

After this patch getting job stats stays correct in a sence
it will not fetch migration stats because it consults
stats.status before doing the fetch.
Signed-off-by: NJiri Denemark <jdenemar@redhat.com>

qemuMigrationFetchJobStatus is rather inconvinient. Some of its
callers don't need status to be updated, some don't need to update
elapsed time right away. So let's update status or elapsed time
in callers instead.

This patch drops updating job status on getting job stats by
client. This way we will not provide status 'completed' while
it is not yet updated by migration routine.
Signed-off-by: NJiri Denemark <jdenemar@redhat.com>

qemuMonitorGetMigrationStats will do it for us anyway.
Signed-off-by: NJiri Denemark <jdenemar@redhat.com>

This way we get stats only in one place. The former code waits for
complete/postcopy status basically and don't need to mess with stats.

The patch drops raising an error on stats updates failure. This
does not make much sense anyway.
Signed-off-by: NJiri Denemark <jdenemar@redhat.com>

Let's introduce QEMU_DOMAIN_JOB_STATUS_POSTCOPY state for job.current->status
instead of checking job.current->stats.status. The latter can be changed
when fetching migration statistics. Moving state function from the variable
and leave only store function seems more managable.

This patch removes all state checking usage of stats except for
qemuDomainGetJobStatsInternal. This place will be handled separately.
Signed-off-by: NJiri Denemark <jdenemar@redhat.com>

This patch simply switches code from using VIR_DOMAIN_JOB_* to
introduced QEMU_DOMAIN_JOB_STATUS_*. Later this gives us freedom
to introduce states for postcopy and mirroring phases.
Signed-off-by: NJiri Denemark <jdenemar@redhat.com>

At some places we either already have synchronous job or we just
released it. Also, some APIs might want to use this code without
having to release their job. Anyway, the job acquire code is
moved out to qemuDomainRemoveInactiveJob so that
qemuDomainRemoveInactive does just what it promises.
Signed-off-by: NMichal Privoznik <mprivozn@redhat.com>
Reviewed-by: NJohn Ferlan <jferlan@redhat.com>

Offline migration transfers only the domain definition.

Resolves: https://bugzilla.redhat.com/show_bug.cgi?id=1449715Signed-off-by: NPavel Hrdina <phrdina@redhat.com>

At present shared disks can be migrated with either readonly or cache=none. But
cache=directsync should be safe for migration, because both cache=directsync and cache=none
don't use the host page cache, and cache=direct write through qemu block layer cache.
Signed-off-by: NPeng Hao <peng.hao2@zte.com.cn>
Reviewed-by: NWang Yechao <wang.yechao255@zte.com.cn>

Should be followed with qemuDomainObjExitMonitor only if
qemuDomainObjEnterMonitorAsync returns 0.
Signed-off-by: NJiri Denemark <jdenemar@redhat.com>

While qemuProcessIncomingDefNew takes an fd argument and stores it in
qemuProcessIncomingDef structure, the caller is still responsible for
closing the file descriptor.

Introduced by commit v1.2.21-140-ge7c6f457.
Signed-off-by: NJiri Denemark <jdenemar@redhat.com>
Reviewed-by: NJohn Ferlan <jferlan@redhat.com>

Most places which want to check ABI stability for an active domain need
to call this API rather than the original
qemuDomainDefCheckABIStability. The only exception is in snapshots where
we need to decide what to do depending on the saved image data.

https://bugzilla.redhat.com/show_bug.cgi?id=1460952Signed-off-by: NJiri Denemark <jdenemar@redhat.com>
Reviewed-by: NPavel Hrdina <phrdina@redhat.com>

Use ATTRIBUTE_FALLTHROUGH, introduced by commit
5d84f596, instead of comments to
indicate that the fall through is an intentional behavior.
Signed-off-by: NMarc Hartmayer <mhartmay@linux.vnet.ibm.com>
Reviewed-by: NBoris Fiuczynski <fiuczy@linux.vnet.ibm.com>
Reviewed-by: NBjoern Walk <bwalk@linux.vnet.ibm.com>

If QEMU is new enough and we have the live updated CPU definition in
either save or migration cookie, we can use it to enforce ABI. The
original guest CPU from domain XML will be stored in private data.
Signed-off-by: NJiri Denemark <jdenemar@redhat.com>
Reviewed-by: NPavel Hrdina <phrdina@redhat.com>

Since the domain XML send during migration uses the original guest CPU
definition but we still want the destination to enforce ABI if it is new
enough, we send the live updated CPU definition in a migration cookie.
Signed-off-by: NJiri Denemark <jdenemar@redhat.com>
Reviewed-by: NPavel Hrdina <phrdina@redhat.com>

When persistent migration of a transient domain is requested but no
custom XML is passed to the migration API we would just let the
destination daemon make a persistent definition from the live definition
itself. This is not a problem now, but once the destination daemon
starts replacing the original CPU definition with the one from migration
cookie before starting a domain, it would need to add more ugly hacks to
reverse the operation. Let's just always send the persistent definition
in the cookie to make things a bit cleaner.
Signed-off-by: NJiri Denemark <jdenemar@redhat.com>
Reviewed-by: NPavel Hrdina <phrdina@redhat.com>

The destination host may not be able to start a domain using the live
updated CPU definition because either libvirt or QEMU may not be new
enough. Thus we need to send the original guest CPU definition.
Signed-off-by: NJiri Denemark <jdenemar@redhat.com>
Reviewed-by: NPavel Hrdina <phrdina@redhat.com>

While fixing a bug with incorrectly freed memory in commit
v3.1.0-399-g5498aa29, I accidentally broke persistent migration of
transient domains. Before adding qemuDomainDefCopy in the path, the code
just took NULL from vm->newDef and used it as the persistent def, which
resulted in no persistent XML being sent in the migration cookie. This
scenario is perfectly valid and the destination correctly handles it by
using the incoming live definition and storing it as the persistent one.

After the mentioned commit libvirtd would just segfault in the described
scenario.

https://bugzilla.redhat.com/show_bug.cgi?id=1446205Signed-off-by: NJiri Denemark <jdenemar@redhat.com>

When creating v3.2.0-77-g8be3ccd0 commit, I completely forgot that one
migration capability is very special. It's the "events" capability which
tells QEMU to report "MIGRATION" events. Since libvirt always wants the
events, it is enabled in qemuConnectMonitor and the rest of the code
should not touch it.

https://bugzilla.redhat.com/show_bug.cgi?id=1439841
https://bugzilla.redhat.com/show_bug.cgi?id=1441165Messed-up-by: NJiri Denemark <jdenemar@redhat.com>
Signed-off-by: NJiri Denemark <jdenemar@redhat.com>

Not all async jobs are visible via virDomainGetJobStats (either they are
too fast or getting the stats is not allowed during the job), but
forcing all of them to advertise the operation is easier than hunting
the jobs for which fetching statistics is allowed. And we won't need to
think about this when we add support for getting stats for more jobs.

https://bugzilla.redhat.com/show_bug.cgi?id=1441563Signed-off-by: NJiri Denemark <jdenemar@redhat.com>

0feebab2 adds calling qemuBlockNodeNamesDetect for completed job
on updating block jobs. This affects cancelling drive mirror logic as
this function drops vm lock. Now we have to recheck all disks
before the disk with the completed block job before going
to wait for block job events.
Signed-off-by: NJiri Denemark <jdenemar@redhat.com>