Using 'unsigned long' for memory values is risky on 32-bit platforms,
as a PAE guest can have more than 4GiB memory.  Our API is
(unfortunately) locked at 'unsigned long' and a scale of 1024, but
the rest of our system should consistently use 64-bit values,
especially since the previous patch centralized overflow checking.

* src/conf/domain_conf.h (_virDomainDef): Always use 64-bit values
for memory.  Change hugepage_backed to a bool.
* src/conf/domain_conf.c (virDomainDefParseXML)
(virDomainDefCheckABIStability, virDomainDefFormatInternal): Fix
clients.
* src/vmx/vmx.c (virVMXFormatConfig): Likewise.
* src/xenxs/xen_sxpr.c (xenParseSxpr, xenFormatSxpr): Likewise.
* src/xenxs/xen_xm.c (xenXMConfigGetULongLong): New function.
(xenXMConfigGetULong, xenXMConfigSetInt): Avoid truncation.
(xenParseXM, xenFormatXM): Fix clients.
* src/phyp/phyp_driver.c (phypBuildLpar): Likewise.
* src/openvz/openvz_driver.c (openvzDomainSetMemoryInternal):
Likewise.
* src/vbox/vbox_tmpl.c (vboxDomainDefineXML): Likewise.
* src/qemu/qemu_command.c (qemuBuildCommandLine): Likewise.
* src/qemu/qemu_process.c (qemuProcessStart): Likewise.
* src/qemu/qemu_monitor.h (qemuMonitorGetBalloonInfo): Likewise.
* src/qemu/qemu_monitor_text.h (qemuMonitorTextGetBalloonInfo):
Likewise.
* src/qemu/qemu_monitor_text.c (qemuMonitorTextGetBalloonInfo):
Likewise.
* src/qemu/qemu_monitor_json.h (qemuMonitorJSONGetBalloonInfo):
Likewise.
* src/qemu/qemu_monitor_json.c (qemuMonitorJSONGetBalloonInfo):
Likewise.
* src/qemu/qemu_driver.c (qemudDomainGetInfo)
(qemuDomainGetXMLDesc): Likewise.
* src/uml/uml_conf.c (umlBuildCommandLine): Likewise.

Qemu supports sizing by bytes; we shouldn't force the user to
round up if they really wanted an unaligned total size.

* include/libvirt/libvirt.h.in (VIR_DOMAIN_BLOCK_RESIZE_BYTES):
New flag.
* src/libvirt.c (virDomainBlockResize): Document it.
* src/qemu/qemu_monitor_json.c (qemuMonitorJSONBlockResize): Take
size in bytes.
* src/qemu/qemu_monitor_text.c (qemuMonitorTextBlockResize):
Likewise.  Pass bytes, not megabytes, to monitor.
* src/qemu/qemu_driver.c (qemuDomainBlockResize): Implement new
flag.

This actually wires up the new optional parameter to block_stream:
http://wiki.qemu.org/Features/LiveBlockMigration/ImageStreamingAPI

The error checking is still sparse, since libvirt must not use
qemu-img or header probing on a qcow2 file in use by qemu to
check if the backing file name is valid; so for now, libvirt is
relying on qemu to diagnose an incorrect backing name.  Fixing this
will require libvirt to track the entire backing file chain at the
time qemu is started and keeps it updated with snapshot and pull
operations.

* src/qemu/qemu_monitor_json.c (qemuMonitorJSONBlockJob): Add
parameter, and update callers.
* src/qemu/qemu_monitor_json.h (qemuMonitorJSONBlockJob): Update
signature.
* src/qemu/qemu_monitor.h (qemuMonitorBlockJob): Likewise.
* src/qemu/qemu_driver.c (qemuDomainBlockJobImpl): Update caller.
* src/qemu/qemu_monitor.c (qemuMonitorBlockJob): Likewise.

using 'system-wakeup' monitor command. It is supported only in JSON,
as we are enabling it if possible. Moreover, this command is available
in qemu-1.1+ which definitely has JSON.

In the future (my next patch in fact) we may want to make
decisions depending on qemu having a monitor command or not.
Therefore, we want to set qemuCaps flag instead of querying
on the monitor each time we are about to make that decision.

QEMU always sends details about all available block devices as an answer
for "info block"/"query-block" command. On the other hand, our
qemuMonitorGetBlockInfo was made for a single block devices queries
only. Thus, when asking for multiple devices, we asked qemu multiple
times to always get the same answer from which different parts were
filtered. This patch makes qemuMonitorGetBlockInfo return a hash table
of all block devices, which may later be used for getting details about
specific devices.

There is now a standard QEMU guest agent that can be installed
and given a virtio serial channel

    <channel type='unix'>
      <source mode='bind' path='/var/lib/libvirt/qemu/f16x86_64.agent'/>
      <target type='virtio' name='org.qemu.guest_agent.0'/>
    </channel>

The protocol that runs over the guest agent is JSON based and
very similar to the JSON monitor. We can't use exactly the same
code because there are some odd differences in the way messages
and errors are structured. The qemu_agent.c file is based on
a combination and simplification of qemu_monitor.c and
qemu_monitor_json.c

* src/qemu/qemu_agent.c, src/qemu/qemu_agent.h: Support for
  talking to the agent for shutdown
* src/qemu/qemu_domain.c, src/qemu/qemu_domain.h: Add thread
  helpers for talking to the agent
* src/qemu/qemu_process.c: Connect to agent whenever starting
  a guest
* src/qemu/qemu_monitor_json.c: Make variable static

When converting a linear enum to a string, we have checks in
place in the VIR_ENUM_IMPL macro to ensure that there is one
string for every value, which lets us quickly flag if a user
added a value but forgot to add a counterpart string.  However,
this only works if we use the _LAST marker.

* cfg.mk (sc_require_enum_last_marker): New syntax check.
* src/conf/domain_conf.h (virDomainSnapshotState): Add new marker.
* src/conf/domain_conf.c (virDomainSnapshotState): Fix offender.
* src/qemu/qemu_monitor_json.c (qemuMonitorWatchdogAction)
(qemuMonitorIOErrorAction, qemuMonitorGraphicsAddressFamily):
Likewise.
* src/util/virtypedparam.c (virTypedParameter): Likewise.

https://bugzilla.redhat.com/show_bug.cgi?id=770954

Implement the block I/O throttle setting and getting support to qemu
driver.
Signed-off-by: NLei Li <lilei@linux.vnet.ibm.com>
Signed-off-by: NZhi Yong Wu <wuzhy@linux.vnet.ibm.com>
Signed-off-by: NEric Blake <eblake@redhat.com>

Implements functions for both HMP and QMP mode.

For HMP mode, qemu uses "M" as the units by default, so the passed "sized"
is divided by 1024.

For QMP mode, qemu uses "Bytes" as the units by default, the passed "sized"
is multiplied by 1024.

All of the monitor functions return -1 on failure, 0 on success, or -2 if
not supported.

The QEMU monitor command 'add_client' can be used to connect to
a VNC or SPICE graphics display. This allows for implementation
of the virDomainOpenGraphics API

* src/qemu/qemu_driver.c: Implement virDomainOpenGraphics
* src/qemu/qemu_monitor.c, src/qemu/qemu_monitor.h,
  src/qemu/qemu_monitor_json.c, src/qemu/qemu_monitor_json.h,
  src/qemu/qemu_monitor_text.c, src/qemu/qemu_monitor_text.h:
  Add binding for 'add_client' command

Not all VNC/SPICE servers use a TCP socket for their connections.
It is possible to configure a UNIX socket server. The graphics
event must thus include a UNIX socket address type.

* include/libvirt/libvirt.h.in: Add UNIX socket address type
  for graphics event
* src/qemu/qemu_monitor_json.c: Add 'unix' string to address
  type enum

This change adds some systemtap/dtrace probes to the QEMU monitor
client code. In particular it allows watching of all operations
for a VM

* examples/systemtap/qemu-monitor.stp: Watch all monitor commands
* src/Makefile.am: Passing libdir/bindir/sbindir to dtrace2systemtap.pl
* src/dtrace2systemtap.pl: Accept libdir/bindir/sbindir as args
  and look for '# binary:' comment to mark probes against libvirtd
  vs libvirt.so
* src/qemu/qemu_monitor.c, src/qemu/qemu_monitor_json.c,
  src/qemu/qemu_monitor_text.c: Add probes for key functions

Detected by Coverity.  Introduced in commit b1b5b51a.

* src/qemu/qemu_monitor_json.c (qemuMonitorJSONGetBlockInfo):
Avoid null dereference.

To date, JSON disk snapshots worked by accident, as they were always
using hmp fallback due to a typo in commit e702b5ba not picking up
on the (intentional) difference in command names between the two
monitor protocols.

* src/qemu/qemu_monitor_json.c (qemuMonitorJSONDiskSnapshot):
Spell QMP command correctly.
Reported by Luiz Capitulino.

Coverity complained that 4 out of 5 callers to virJSONValueObjectGetBoolean
checked for errors.  But we documented that we don't care in this case.

* src/qemu/qemu_monitor_json.c (qemuMonitorJSONGetBlockInfo): Use
ignore_value.

If the daemon is restarted so we reconnect to monitor, cdrom media
can be ejected. In that case we don't want to show it in domain xml,
or require it on migration destination.

To check for disk status use 'info block' monitor command.

Current qemu is able to give us detailed domain status (not just if it
is running or not) which we can translate into a status reason.

/usr/lib/stdlib.h in Mac OS X and probably also in BSD's
exports this symbol :(

This patch adds handlers for modification of guest's interface
link state. Both HMP and QMP commands are supported, but as the
link state functionality is from the beginning supported in QMP
the HMP code will probably never be used.

The mainly changes are:

1) Update qemuMonitorGetBlockStatsInfo and it's children (Text/JSON)
   functions to return the value of new latency fields.
2) Add new function qemuMonitorGetBlockStatsParamsNumber, which is
   to count how many parameters the underlying QEMU supports.
3) Update virDomainBlockStats in src/qemu/qemu_driver.c to be
   compatible with the changes by 1).

No one uses this yet, but it will be important once
virDomainSnapshotCreateXML learns a VIR_DOMAIN_SNAPSHOT_DISK_ONLY
flag, and the xml allows passing in the new file names.

* src/qemu/qemu_monitor.h (qemuMonitorDiskSnapshot): New prototype.
* src/qemu/qemu_monitor_text.h (qemuMonitorTextDiskSnapshot):
Likewise.
* src/qemu/qemu_monitor_json.h (qemuMonitorJSONDiskSnapshot):
Likewise.
* src/qemu/qemu_monitor.c (qemuMonitorDiskSnapshot): New
function.
* src/qemu/qemu_monitor_json.c (qemuMonitorJSONDiskSnapshot):
Likewise.

The libvirt BlockPull API supports the use of an initial bandwidth limit but the
qemu block_stream API does not.  To get the desired behavior we use the two APIs
strung together: first BlockPull, then BlockJobSetSpeed.  We can do this at the
driver level to avoid duplicated code in each monitor path.
Signed-off-by: NAdam Litke <agl@us.ibm.com>

* src/qemu/qemu_monitor_json.c: Handle error "CommandNotFound" and
  report the error.

* src/qemu/qemu_monitor_text.c: If a sub info command is not found,
  it prints the output of "help info", for other commands,
  "unknown command" is printed.

Without this patch, libvirt always report:

  An error occurred, but the cause is unknown

This patch was adapted from a patch by Osier Yang <jyang@redhat.com> to
break out detection of unrecognized text monitor commands into a separate
function.
Signed-off-by: NAdam Litke <agl@us.ibm.com>

A virsh command like:

migrate --live --copy-storage-all Guest qemu+ssh://user@host/system
--persistent --verbose

shows

Migration: [  0 %]

during the storage copy and does not start counting
until the ram transfer starts

Fix this by scraping optional disk transfer status, and adding it
into the progress meter.

Coverity detected that 5 of 6 callers of virJSONValueArrayGet checked
for a NULL return; and that by not checking we risk a null deref
during an error.  The error is unlikely since the prior call to
virJSONValueArraySize would probably have already caught any botched
JSON array parse, but better safe than sorry.

* src/qemu/qemu_monitor_json.c (qemuMonitorJSONGetBlockJobInfo):
Check for NULL.
(qemuMonitorJSONExtractPtyPaths): Fix typo.

When an operation started by virDomainBlockPull completes (either with
success or with failure), raise an event to indicate the final status.
This API allow users to avoid polling on virDomainGetBlockJobInfo if
they would prefer to use an event mechanism.

* daemon/remote.c: Dispatch events to client
* include/libvirt/libvirt.h.in: Define event ID and callback signature
* src/conf/domain_event.c, src/conf/domain_event.h,
  src/libvirt_private.syms: Extend API to handle the new event
* src/qemu/qemu_driver.c: Connect to the QEMU monitor event
  for block_stream completion and emit a libvirt block pull event
* src/remote/remote_driver.c: Receive and dispatch events to application
* src/remote/remote_protocol.x: Wire protocol definition for the event
* src/remote_protocol-structs: structure definitions for protocol verification
* src/qemu/qemu_monitor.c, src/qemu/qemu_monitor.h,
  src/qemu/qemu_monitor_json.c: Watch for BLOCK_STREAM_COMPLETED event
  from QEMU monitor

The virDomainBlockPull* family of commands are enabled by the
following HMP/QMP commands: 'block_stream', 'block_job_cancel',
 'info block-jobs' / 'query-block-jobs', and 'block_job_set_speed'.

* src/qemu/qemu_driver.c src/qemu/qemu_monitor_text.[ch]: implement disk
  streaming by using the proper qemu monitor commands.
* src/qemu/qemu_monitor_json.[ch]: implement commands using the qmp monitor

qemu driver just accept xt_kbd codeset's keycode, so the lib virtkey
is used for translating keycodes from other codesets

When attaching to an external QEMU process, it is neccessary
to check if the process is using KVM or not. This can be done
using a monitor command

* src/qemu/qemu_monitor.c, src/qemu/qemu_monitor.h,
  src/qemu/qemu_monitor_json.c, src/qemu/qemu_monitor_json.h,
  src/qemu/qemu_monitor_text.c, src/qemu/qemu_monitor_text.h: Add
  API for checking if KVM is enabled

The event handler functions do not free the virJSONValuePtr
object. Every event received from a VM thus caused a memory
leak

* src/qemu/qemu_monitor_json.c: Fix leak of event object

This reverts commit 784ee08d.

This reverts commit 12cd77a0.

Conflicts:

	python/libvirt-override-virConnect.py
	python/libvirt-override.c
	src/remote/remote_protocol.x

For controlled shutdown we issue a 'system_powerdown' command
to the QEMU monitor. This triggers an ACPI event which (most)
guest OS wire up to a controlled shutdown. There is no equiv
ACPI event to trigger a controlled reboot. This patch attempts
to fake a reboot.

 - In qemuDomainObjPrivatePtr we have a bool fakeReboot
   flag.
 - The virDomainReboot method sets this flag and then
   triggers a normal 'system_powerdown'.
 - The QEMU process is started with '-no-shutdown'
   so that the guest CPUs pause when it powers off the
   guest
 - When we receive the 'POWEROFF' event from QEMU JSON
   monitor if fakeReboot is not set we invoke the
   qemuProcessKill command and shutdown continues
   normally
 - If fakeReboot was set, we spawn a background thread
   which issues 'system_reset' to perform a warm reboot
   of the guest hardware. Then it issues 'cont' to
   start the CPUs again

* src/qemu/qemu_command.c: Add -no-shutdown flag if
  we have JSON support
* src/qemu/qemu_domain.h: Add 'fakeReboot' flag to
  qemuDomainObjPrivate struct
* src/qemu/qemu_driver.c: Fake reboot using the
  system_powerdown command if JSON support is available
* src/qemu/qemu_monitor.c, src/qemu/qemu_monitor.h,
  src/qemu/qemu_monitor_json.c, src/qemu/qemu_monitor_json.h,
  src/qemu/qemu_monitor_text.c, src/qemu/qemu_monitor_text.h: Add
  binding for system_reset command
* src/qemu/qemu_process.c: Reset the guest & start CPUs if
  fakeReboot is set

When an operation started by virDomainBlockPullAll completes (either with
success or with failure), raise an event to indicate the final status.  This
allows an API user to avoid polling on virDomainBlockPullInfo if they would
prefer to use the event mechanism.

* daemon/remote.c: Dispatch events to client
* include/libvirt/libvirt.h.in: Define event ID and callback signature
* src/conf/domain_event.c, src/conf/domain_event.h,
  src/libvirt_private.syms: Extend API to handle the new event
* src/qemu/qemu_driver.c: Connect to the QEMU monitor event
  for block_stream completion and emit a libvirt block pull event
* src/remote/remote_driver.c: Receive and dispatch events to application
* src/remote/remote_protocol.x: Wire protocol definition for the event
* src/qemu/qemu_monitor.c, src/qemu/qemu_monitor.h,
  src/qemu/qemu_monitor_json.c: Watch for BLOCK_STREAM_COMPLETED event
  from QEMU monitor
Signed-off-by: NAdam Litke <agl@us.ibm.com>

The virDomainBlockPull* family of commands are enabled by the
'block_stream' and 'info block_stream' qemu monitor commands.

* src/qemu/qemu_driver.c src/qemu/qemu_monitor_text.[ch]: implement disk
  streaming by using the stream and info stream text monitor commands
* src/qemu/qemu_monitor_json.[ch]: implement commands using the qmp monitor
Signed-off-by: NAdam Litke <agl@us.ibm.com>
Acked-by: NDaniel P. Berrange <berrange@redhat.com>

Qemu once supported following memory stats which will returned by
"query_balloon":

    stat_put(dict, "actual", actual);
    stat_put(dict, "mem_swapped_in", dev->stats[VIRTIO_BALLOON_S_SWAP_IN]);
    stat_put(dict, "mem_swapped_out", dev->stats[VIRTIO_BALLOON_S_SWAP_OUT]);
    stat_put(dict, "major_page_faults", dev->stats[VIRTIO_BALLOON_S_MAJFLT]);
    stat_put(dict, "minor_page_faults", dev->stats[VIRTIO_BALLOON_S_MINFLT]);
    stat_put(dict, "free_mem", dev->stats[VIRTIO_BALLOON_S_MEMFREE]);
    stat_put(dict, "total_mem", dev->stats[VIRTIO_BALLOON_S_MEMTOT]);

But it later disabled all the stats except "actual" by commit
07b0403dfc2b2ac179ae5b48105096cc2d03375a.

libvirt doesn't parse "actual", so user will always see a empty result
with "virsh dommemstat $domain". Even qemu haven't disabled the stats,
we should support parsing "actual".

Currently the QEMU monitor I/O handler code uses errno values
to report errors. This results in a sub-optimal error messages
on certain conditions, in particular when parsing JSON strings
malformed data simply results in 'EINVAL'.

This changes the code to use the standard libvirt error reporting
APIs. The virError is stored against the qemuMonitorPtr struct,
and when a monitor API is run, any existing stored error is copied
into that thread's error local

* src/qemu/qemu_monitor.c, src/qemu/qemu_monitor.h,
  src/qemu/qemu_monitor_json.c, src/qemu/qemu_monitor_text.c: Use
  virError APIs for all monitor I/O handling code