The mainly changes are:

1) Update qemuMonitorGetBlockStatsInfo and it's children (Text/JSON)
   functions to return the value of new latency fields.
2) Add new function qemuMonitorGetBlockStatsParamsNumber, which is
   to count how many parameters the underlying QEMU supports.
3) Update virDomainBlockStats in src/qemu/qemu_driver.c to be
   compatible with the changes by 1).

No one uses this yet, but it will be important once
virDomainSnapshotCreateXML learns a VIR_DOMAIN_SNAPSHOT_DISK_ONLY
flag, and the xml allows passing in the new file names.

* src/qemu/qemu_monitor.h (qemuMonitorDiskSnapshot): New prototype.
* src/qemu/qemu_monitor_text.h (qemuMonitorTextDiskSnapshot):
Likewise.
* src/qemu/qemu_monitor_json.h (qemuMonitorJSONDiskSnapshot):
Likewise.
* src/qemu/qemu_monitor.c (qemuMonitorDiskSnapshot): New
function.
* src/qemu/qemu_monitor_json.c (qemuMonitorJSONDiskSnapshot):
Likewise.

When an operation started by virDomainBlockPull completes (either with
success or with failure), raise an event to indicate the final status.
This API allow users to avoid polling on virDomainGetBlockJobInfo if
they would prefer to use an event mechanism.

* daemon/remote.c: Dispatch events to client
* include/libvirt/libvirt.h.in: Define event ID and callback signature
* src/conf/domain_event.c, src/conf/domain_event.h,
  src/libvirt_private.syms: Extend API to handle the new event
* src/qemu/qemu_driver.c: Connect to the QEMU monitor event
  for block_stream completion and emit a libvirt block pull event
* src/remote/remote_driver.c: Receive and dispatch events to application
* src/remote/remote_protocol.x: Wire protocol definition for the event
* src/remote_protocol-structs: structure definitions for protocol verification
* src/qemu/qemu_monitor.c, src/qemu/qemu_monitor.h,
  src/qemu/qemu_monitor_json.c: Watch for BLOCK_STREAM_COMPLETED event
  from QEMU monitor

The virDomainBlockPull* family of commands are enabled by the
following HMP/QMP commands: 'block_stream', 'block_job_cancel',
 'info block-jobs' / 'query-block-jobs', and 'block_job_set_speed'.

* src/qemu/qemu_driver.c src/qemu/qemu_monitor_text.[ch]: implement disk
  streaming by using the proper qemu monitor commands.
* src/qemu/qemu_monitor_json.[ch]: implement commands using the qmp monitor

qemu driver just accept xt_kbd codeset's keycode, so the lib virtkey
is used for translating keycodes from other codesets

When qemuMonitorCloseFileHandle is called in error path, we need to
preserve the original error since a possible further error when running
closefd monitor command is not very useful to users.

When attaching to an external QEMU process, it is neccessary
to check if the process is using KVM or not. This can be done
using a monitor command

* src/qemu/qemu_monitor.c, src/qemu/qemu_monitor.h,
  src/qemu/qemu_monitor_json.c, src/qemu/qemu_monitor_json.h,
  src/qemu/qemu_monitor_text.c, src/qemu/qemu_monitor_text.h: Add
  API for checking if KVM is enabled

This reverts commit 784ee08d.

This reverts commit 12cd77a0.

Conflicts:

	python/libvirt-override-virConnect.py
	python/libvirt-override.c
	src/remote/remote_protocol.x

For controlled shutdown we issue a 'system_powerdown' command
to the QEMU monitor. This triggers an ACPI event which (most)
guest OS wire up to a controlled shutdown. There is no equiv
ACPI event to trigger a controlled reboot. This patch attempts
to fake a reboot.

 - In qemuDomainObjPrivatePtr we have a bool fakeReboot
   flag.
 - The virDomainReboot method sets this flag and then
   triggers a normal 'system_powerdown'.
 - The QEMU process is started with '-no-shutdown'
   so that the guest CPUs pause when it powers off the
   guest
 - When we receive the 'POWEROFF' event from QEMU JSON
   monitor if fakeReboot is not set we invoke the
   qemuProcessKill command and shutdown continues
   normally
 - If fakeReboot was set, we spawn a background thread
   which issues 'system_reset' to perform a warm reboot
   of the guest hardware. Then it issues 'cont' to
   start the CPUs again

* src/qemu/qemu_command.c: Add -no-shutdown flag if
  we have JSON support
* src/qemu/qemu_domain.h: Add 'fakeReboot' flag to
  qemuDomainObjPrivate struct
* src/qemu/qemu_driver.c: Fake reboot using the
  system_powerdown command if JSON support is available
* src/qemu/qemu_monitor.c, src/qemu/qemu_monitor.h,
  src/qemu/qemu_monitor_json.c, src/qemu/qemu_monitor_json.h,
  src/qemu/qemu_monitor_text.c, src/qemu/qemu_monitor_text.h: Add
  binding for system_reset command
* src/qemu/qemu_process.c: Reset the guest & start CPUs if
  fakeReboot is set

When an operation started by virDomainBlockPullAll completes (either with
success or with failure), raise an event to indicate the final status.  This
allows an API user to avoid polling on virDomainBlockPullInfo if they would
prefer to use the event mechanism.

* daemon/remote.c: Dispatch events to client
* include/libvirt/libvirt.h.in: Define event ID and callback signature
* src/conf/domain_event.c, src/conf/domain_event.h,
  src/libvirt_private.syms: Extend API to handle the new event
* src/qemu/qemu_driver.c: Connect to the QEMU monitor event
  for block_stream completion and emit a libvirt block pull event
* src/remote/remote_driver.c: Receive and dispatch events to application
* src/remote/remote_protocol.x: Wire protocol definition for the event
* src/qemu/qemu_monitor.c, src/qemu/qemu_monitor.h,
  src/qemu/qemu_monitor_json.c: Watch for BLOCK_STREAM_COMPLETED event
  from QEMU monitor
Signed-off-by: NAdam Litke <agl@us.ibm.com>

The virDomainBlockPull* family of commands are enabled by the
'block_stream' and 'info block_stream' qemu monitor commands.

* src/qemu/qemu_driver.c src/qemu/qemu_monitor_text.[ch]: implement disk
  streaming by using the stream and info stream text monitor commands
* src/qemu/qemu_monitor_json.[ch]: implement commands using the qmp monitor
Signed-off-by: NAdam Litke <agl@us.ibm.com>
Acked-by: NDaniel P. Berrange <berrange@redhat.com>

Currently the QEMU monitor I/O handler code uses errno values
to report errors. This results in a sub-optimal error messages
on certain conditions, in particular when parsing JSON strings
malformed data simply results in 'EINVAL'.

This changes the code to use the standard libvirt error reporting
APIs. The virError is stored against the qemuMonitorPtr struct,
and when a monitor API is run, any existing stored error is copied
into that thread's error local

* src/qemu/qemu_monitor.c, src/qemu/qemu_monitor.h,
  src/qemu/qemu_monitor_json.c, src/qemu/qemu_monitor_text.c: Use
  virError APIs for all monitor I/O handling code

Currently whenever there is any failure with parsing the monitor,
this is treated in the same was as end-of-file (ie QEMU quit).
The domain is terminated, if not already dead.

With this change, failures in parsing the monitor stream do not
result in the death of QEMU. The guest continues running unchanged,
but all further use of the monitor will be disabled.

The VMM_FAILURE event will be emitted, and the mgmt application
can decide when to kill/restart the guest to re-gain control

* src/qemu/qemu_monitor.c, src/qemu/qemu_monitor.h: Run a
  different callback for monitor EOF vs error conditions.
* src/qemu/qemu_process.c: Emit VMM_FAILURE event when monitor
  fails

Use the graphics information from the QEMU migration cookie to
issue a 'client_migrate_info' monitor command to QEMU. This causes
the SPICE client to automatically reconnect to the target host
when migration completes

* src/qemu/qemu_migration.c: Set data for SPICE client relocation
  before starting migration on src
* src/qemu/qemu_monitor.c, src/qemu/qemu_monitor.h,
  src/qemu/qemu_monitor_json.c, src/qemu/qemu_monitor_json.h,
  src/qemu/qemu_monitor_text.c, src/qemu/qemu_monitor_text.h: Add
  new qemuMonitorGraphicsRelocate() command

A qemu domain can get paused when libvirtd is stopped (e.g., because of
I/O error) so we should check its current state when reconnecting to it.

* src/qemu/qemu_driver.c: new qemuDomainScreenshot() function
* src/qemu/qemu_monitor.c, src/qemu/qemu_monitor.h,
  src/qemu/qemu_monitor_json.c, src/qemu/qemu_monitor_json.h,
  src/qemu/qemu_monitor_text.c, src/qemu/qemu_monitor_text.h:
  Monitor command

Add the compiler attribute to ensure we don't introduce any more
ref bugs like were just patched in commit 9741f346, then explicitly
mark the remaining places in code that are safe.

* src/qemu/qemu_monitor.h (qemuMonitorUnref): Mark
ATTRIBUTE_RETURN_CHECK.
* src/conf/domain_conf.h (virDomainObjUnref): Likewise.
* src/conf/domain_conf.c (virDomainObjParseXML)
(virDomainLoadStatus): Fix offenders.
* src/openvz/openvz_conf.c (openvzLoadDomains): Likewise.
* src/vmware/vmware_conf.c (vmwareLoadDomains): Likewise.
* src/qemu/qemu_domain.c (qemuDomainObjBeginJob)
(qemuDomainObjBeginJobWithDriver)
(qemuDomainObjExitRemoteWithDriver): Likewise.
* src/qemu/qemu_monitor.c (QEMU_MONITOR_CALLBACK): Likewise.
Suggested by Daniel P. Berrange.

With only a single caller to these two monitor commands, I
didn't need to wrap a new WithFds version, but just change
the command itself.

* src/qemu/qemu_monitor.h (qemuMonitorAddNetdev)
(qemuMonitorAddHostNetwork): Add parameters.
* src/qemu/qemu_monitor.c (qemuMonitorAddNetdev)
(qemuMonitorAddHostNetwork): Add support for fd passing.
* src/qemu/qemu_hotplug.c (qemuDomainAttachNetDevice): Use it to
simplify code.

This is also a bug fix - on the error path, qemu_hotplug would
leave the configfd file leaked into qemu.  At least the next
attempt to hotplug a PCI device would reuse the same fdname,
and when the qemu getfd monitor command gets a new fd by the
same name as an earlier one, it closes the earlier one, so there
is no risk of qemu running out of fds.

* src/qemu/qemu_monitor.h (qemuMonitorAddDeviceWithFd): New
prototype.
* src/qemu/qemu_monitor.c (qemuMonitorAddDevice): Move guts...
(qemuMonitorAddDeviceWithFd): ...to new function, and add support
for fd passing.
* src/qemu/qemu_hotplug.c (qemuDomainAttachHostPciDevice): Use it
to simplify code.
Suggested by Daniel P. Berrange.

* src/qemu/qemu_monitor.h (qemuMonitorMigrateToFd): New
prototype.
* src/qemu/qemu_monitor.c (qemuMonitorMigrateToFd): New function.

So that it's obvious that they are supposed to be used with HMP commands.

JSON monitor command implementation can now just directly call text
monitor implementation and it will be automatically encapsulated into
QMP's human-monitor-command.

Currently users who want to use virDomainQemuMonitorCommand() API or
it's virsh equivalent has to use the same protocol as libvirt uses for
communication to qemu. Since the protocol is QMP with current qemu and
HMP much more usable for humans, one ends up typing something like the
following:

    virsh qemu-monitor-command DOM \
'{"execute":"human-monitor-command","arguments":{"command-line":"info kvm"}}'

which is not a very convenient way of debugging qemu.

This patch introduces --hmp option to qemu-monitor-command, which says
that the provided command is in HMP. If libvirt uses QMP to talk with
qemu, the command will automatically be converted into QMP. So the
example above is simplified to just

    virsh qemu-monitor-command --hmp DOM "info kvm"

Also the result is converted from

    {"return":"kvm support: enabled\r\n"}

to just plain HMP:

    kvm support: enabled

If libvirt talks to qemu in HMP, --hmp flag is obviously a noop.

* src/conf/domain_conf.h (virDomainChrDeviceType): Drop monitor.
* src/conf/domain_conf.c (virDomainChrDevice)
(virDomainChrDefParseTargetXML, virDomainChrDefFormat): Drop
monitor support.
* src/qemu/qemu_command.h (qemuBuildCommandLine): Alter signature.
* src/qemu/qemu_monitor.h (qemuMonitorOpen): Likewise.
* src/qemu/qemu_domain.h (_qemuDomainObjPrivate): Change type of
monConfig.
* src/qemu/qemu_domain.c (qemuDomainObjPrivateFree)
(qemuDomainObjPrivateXMLFormat, qemuDomainObjPrivateXMLParse):
Adjust to type change.
* src/qemu/qemu_command.c (qemuBuildCommandLine): Likewise.
* src/qemu/qemu_driver.c (qemuPrepareMonitorChr)
(qemudStartVMDaemon, qemuDomainXMLToNative, qemuConnectMonitor)
(qemudShutdownVMDaemon): Likewise.
* src/qemu/qemu_hotplug.c (qemuDomainAttachNetDevice): Likewise.
* src/qemu/qemu_monitor.c (qemuMonitorOpen): Likewise.
* tests/qemuxml2argvtest.c (testCompareXMLToArgvFiles): Likewise.

* daemon/Makefile.am: Avoid spurious space before tabs.
* src/Makefile.am: Likewise.
* examples/dominfo/Makefile.am: Likewise.
* examples/domsuspend/Makefile.am: Likewise.
* tools/Makefile.am: Likewise.
* src/datatypes.h (VIR_CONNECT_MAGIC): Likewise.
* src/internal.h (TODO): Likewise.
* src/qemu/qemu_monitor.h (QEMU_MONITOR_MIGRATE): Likewise.
* src/xen/xen_hypervisor.c (XEN_V2_OP_GETAVAILHEAP): Likewise.
* src/xen/xs_internal.h: Likewise.

Currently libvirt doesn't confirm whether the guest has responded to the
disk removal request.  In some cases this can leave the guest with
continued access to the device while the mgmt layer believes that it has
been removed.  With a recent qemu monitor command[1] we can
deterministically revoke a guests access to the disk (on the QEMU side)
to ensure no futher access is permitted.

This patch adds support for the drive_del() command and introduces it
in the disk removal paths.  If the guest is running in a QEMU without this
command we currently explicitly check for unknown command/CommandNotFound
and log the issue.

If QEMU supports the command we issue the drive_del command after we attempt
to remove the device.  The guest may respond and remove the block device
before we get to attempt to call drive_del.  In that case, we explicitly check
for 'Device not found' from the monitor indicating that the target drive
was auto-deleted upon guest responds to the device removal notification.

1. http://thread.gmane.org/gmane.comp.emulators.qemu/84745Signed-off-by: NRyan Harper <ryanh@us.ibm.com>

Currently libvirt doesn't confirm whether the guest has responded to the
disk removal request.  In some cases this can leave the guest with
continued access to the device while the mgmt layer believes that it has
been removed.  With a recent qemu monitor command[1] we can
deterministically revoke a guests access to the disk (on the QEMU side)
to ensure no futher access is permitted.

This patch adds support for the drive_unplug() command and introduces it
in the disk removal paths.  There is some discussion to be had about how
to handle the case where the guest is running in a QEMU without this
command (and the fact that we currently don't have a way of detecting
what monitor commands are available).

Changes since v2:
 - use VIR_ERROR to report when unplug command not found
Changes since v1:
 - return > 0 when command isn't present, < 0 on command failure
 - detect when drive_unplug command isn't present and log error
   instead of failing entire command
Signed-off-by: NRyan Harper <ryanh@us.ibm.com>

QEMU allows forcing a CDROM eject even if the guest has locked the device.
Expose this via a new UpdateDevice flag, VIR_DOMAIN_DEVICE_MODIFY_FORCE.

This has been requested for RHEV:

https://bugzilla.redhat.com/show_bug.cgi?id=626305

v2: Change flag name, bool cleanups

Older dash mistakenly truncates regular files when using <> redirection;
this kills our use of double dd to reduce storage overhead when
saving qemu images.  But qemu insists on running a command through
/bin/sh, so we work around it by having qemu run $sh -c 'real command'
when we have a replacement $sh in mind.

* configure.ac (VIR_WRAPPER_SHELL): Define to a replacement shell,
if /bin/sh is broken on <> redirection.
* src/qemu/qemu_monitor.h (VIR_WRAPPER_SHELL_PREFIX)
(VIR_WRAPPER_SHELL_SUFFIX): New macros.
* src/qemu/qemu_monitor_text.c (qemuMonitorTextMigrateToFile): Use
them.
* src/qemu/qemu_monitor_json.c (qemuMonitorJSONMigrateToFile):
Likewise.

Implement the qemu driver's virDomainQemuMonitorCommand
and hook it into the API entry point.

Changes since v1:
 - Rename the (external) qemuMonitorCommand to qemuDomainMonitorCommand
 - Add virCheckFlags to qemuDomainMonitorCommand

Changes since v2:
 - Drop ATTRIBUTE_UNUSED from the flags

Changes since v3:
 - Add a flag to priv so we only print out monitor command warning once.  Note
   that this has not been plumbed into qemuDomainObjPrivateXMLFormat or
   qemuDomainObjPrivateXMLParse, which means that if you run a monitor command,
   restart libvirtd, and then run another monitor command, you may get an
   an erroneous VIR_INFO.  It's a pretty minor matter, and I didn't think it
   warranted the additional code.
 - Add BeginJob/EndJob calls around EnterMonitor/ExitMonitor
Signed-off-by: NChris Lalancette <clalance@redhat.com>

The patches for shared storage migration were not correctly written
for json mode. Thus the 'blk' and 'inc' parameters were never being
set. In addition they didn't set the QEMU_MONITOR_MIGRATE_BACKGROUND
so migration was synchronous. Due to multiple bugs in QEMU's JSON
impl this wasn't noticed because it treated the sync migration requst
as asynchronous anyway. Finally 'background' parameter was converted
to take arbitrary flags but not renamed, and not all uses were changed
to unsigned int.

* src/qemu/qemu_driver.c: Set QEMU_MONITOR_MIGRATE_BACKGROUND in
  doNativeMigrate
* src/qemu/qemu_monitor_json.c: Process QEMU_MONITOR_MIGRATE_NON_SHARED_DISK
  and QEMU_MONITOR_MIGRATE_NON_SHARED_INC flags
* src/qemu/qemu_monitor.c, src/qemu/qemu_monitor.h,
  src/qemu/qemu_monitor_json.h, src/qemu/qemu_monitor_text.c,
  src/qemu/qemu_monitor_text.h: change 'int background' to
  'unsigned int flags' in migration APIs. Add logging of flags
  parameter

The current code pattern requires that callers of qemuMonitorClose
check for the return value == 0, and if so, set priv->mon = NULL
and release the reference held on the associated virDomainObjPtr

The change d84bb6d6 violated that
requirement, meaning that priv->mon never gets set to NULL, and
a reference count is leaked on virDomainObjPtr.

This design was a bad one, so remove the need to check the return
valueof qemuMonitorClose(). Instead allow registration of a
callback that's invoked just when the last reference on qemuMonitorPtr
is released.

Finally there was a potential reference leak in qemuConnectMonitor
in the failure path.

* src/qemu/qemu_monitor.c, src/qemu/qemu_monitor.h: Add a destroy
  callback invoked from qemuMonitorFree
* src/qemu/qemu_driver.c: Use the destroy callback to release the
  reference on virDomainObjPtr when the monitor is freed. Fix other
  potential reference count leak in connecting to monitor

Followup to https://bugzilla.redhat.com/show_bug.cgi?id=599091,
commit 20206a4b, to reduce disk waste in padding.

* src/qemu/qemu_monitor.h (QEMU_MONITOR_MIGRATE_TO_FILE_BS): Drop
back to 4k.
(QEMU_MONITOR_MIGRATE_TO_FILE_TRANSFER_SIZE): New macro.
* src/qemu/qemu_driver.c (qemudDomainSaveFlag): Update comment.
* src/qemu/qemu_monitor_text.c (qemuMonitorTextMigrateToFile): Use
two invocations of dd to output non-aligned large blocks.
* src/qemu/qemu_monitor_json.c (qemuMonitorJSONMigrateToFile):
Likewise.

See https://bugzilla.redhat.com/show_bug.cgi?id=599091

Saving a paused 512MB domain took 3m47s with the old block size of 512
bytes. Changing the block size to 1024*1024 decreased the time to 56
seconds. (Doubling again to 2048*1024 yielded 0 improvement; lowering
to 512k increased the save time to 1m10s, about 20%)

The virDomainGetBlockInfo API allows query physical block
extent and allocated block extent. These are normally the
same value unless storing a special format like qcow2
inside a block device. In this scenario we can query QEMU
to get the actual allocated extent.

Since last time:

 - Return fatal error in text monitor
 - Only invoke monitor command for block devices
 - Fix error handling JSON code

* src/qemu/qemu_driver.c: Fill in block aloction extent when VM
  is running
* src/qemu/qemu_monitor.c, src/qemu/qemu_monitor.h,
  src/qemu/qemu_monitor_json.c, src/qemu/qemu_monitor_json.h,
  src/qemu/qemu_monitor_text.c, src/qemu/qemu_monitor_text.h: Add
  API to query the highest block extent via info blockstats

QEMU is gaining a new monitor command netdev_add for hotplugging
NICs using the netdev backend code. We already support this on
the command this, though it is disabled. This adds support for
hotplug too, also to remain disabled until 0.13 QEMU is released

* src/qemu/qemu_driver.c: Support netdev hotplug for NICs
* src/qemu/qemu_monitor.c, src/qemu/qemu_monitor.h,
  src/qemu/qemu_monitor_json.c, src/qemu/qemu_monitor_json.h,
  src/qemu/qemu_monitor_text.c, src/qemu/qemu_monitor_text.h: Add
  support for netdev_add and netdev_remove commands