domain parsing and serialization code, qemu driver backend and
a couple of test

Companion controllers take an extra 'master' attribute to associate
them.

Also add tests for this

Expand the domain and the QEmu driver code
Adds a couple of tests

The model by default is piix3-uchi.

Example:
<controller type='usb' index='0' model='ich9-ehci'/>

Newer QEMU introduced cache=directsync for -drive, this patchset
is to expose it in libvirt layer.

  * Introduced a new QEMU capability flag ($prefix_CACHE_DIRECTSYNC),
    As even $prefix_CACHE_V2 is set, we can't known if directsync
    is supported.

After supporting multi function pci device, we only reserve function 1 on slot 1.
The user can use the other function on slot 1 in the xml config file. We should
detect this wrong usage.

Changing the current vm, and writing that change to the file
system, all before a new qemu starts, is risky; it's hard to
roll back if starting the new qemu fails for some reason.
Instead of abusing vm->current_snapshot and making the command
line generator decide whether the current snapshot warrants
using -loadvm, it is better to just directly pass a snapshot all
the way through the call chain if it is to be loaded.

This frees up the last use of snapshot->def->active for qemu's
use, so the next patch can repurpose that field for tracking
which snapshot is current.

* src/qemu/qemu_command.c (qemuBuildCommandLine): Don't use active
field of snapshot.
* src/qemu/qemu_process.c (qemuProcessStart): Add a parameter.
* src/qemu/qemu_process.h (qemuProcessStart): Update prototype.
* src/qemu/qemu_migration.c (qemuMigrationPrepareAny): Update
callers.
* src/qemu/qemu_driver.c (qemudDomainCreate)
(qemuDomainSaveImageStartVM, qemuDomainObjStart)
(qemuDomainRevertToSnapshot): Likewise.
(qemuDomainSnapshotSetCurrentActive)
(qemuDomainSnapshotSetCurrentInactive): Delete unused functions.

Signed-off-by: NSteve Hodgson <shodgson@solarflare.com>

* src/qemu/qemu_command.c: s/VIR_ERR_NO_SUPPORT/VIR_ERR_CONFIG_UNSUPPORTED/

* src/qemu/qemu_driver.c: s/VIR_ERR_NO_SUPPORT/VIR_ERR_OPERATION_INVALID/

* src/qemu/qemu_process.c: s/VIR_ERR_NO_SUPPORT/VIR_ERR_OPERATION_INVALID/

In some versions of qemu, both virtio-blk-pci and virtio-net-pci
devices can have an event_idx setting that determines some details of
event processing. When it is enabled, it "reduces the number of
interrupts and exits for the guest". qemu will automatically enable
this feature when it is available, but there may be cases where this
new feature could actually make performance worse (NB: no such case
has been found so far).

As a safety switch in case such a situation is encountered in the
field, this patch adds a new attribute "event_idx" to the <driver>
element of both disk and interface devices. event_idx can be set to
"on" (to force event_idx on in case qemu has it disabled by default)
or "off" (for force event_idx off). In the case that event_idx support
isn't present in qemu, the attribute is ignored (this on the advice of
the qemu developer).

docs/formatdomain.html.in: document the new flag (marking it as
   "don't mess with this!"
docs/schemas/domain.rng: add event_idx in appropriate places
src/conf/domain_conf.[ch]: add event_idx to parser and formatter
src/libvirt_private.syms: export
   virDomainVirtioEventIdx(From|To)String
src/qemu/qemu_capabilities.[ch]: detect and report event_idx in
   disk/net
src/qemu/qemu_command.c: add event_idx parameter to qemu commandline
    when appropriate.
tests/qemuxml2argvdata/qemuxml2argv-event_idx.args,
tests/qemuxml2argvdata/qemuxml2argv-event_idx.xml,
tests/qemuxml2argvtest.c,
tests/qemuxml2xmltest.c: test cases for event_idx.

The following XML:

    <serial type='udp'>
      <source mode='connect' service='9999'/>
    </serial>

is accepted by domain_conf.c but maps to the qemu command line:

-chardev udp,host=127.0.0.1,port=2222,localaddr=(null),localport=(null)

qemu can cope with everything omitting except the connection port, which
seems to also be the intent of domain_conf validation, so let's not
generate bogus command lines for that case.
The defaults are empty strings for addresses and 0 for the localport

Additionally, tweak the qemu cli parsing to handle omitted host
parameters
for -serial udp

Quite a few leaks detected by coverity.  For chr, the leaks were
close enough to the allocations to plug in place; for disk, the
leaks were separated from the allocation by enough other lines with
intermediate failure cases that I refactored the cleanup instead.

* src/qemu/qemu_command.c (qemuParseCommandLine): Plug leaks.

Using a macro ensures that all the code is looking for the same
prefix.

* src/conf/domain_conf.h (VIR_NET_GENERATED_PREFIX): New macro.
* src/conf/domain_conf.c (virDomainNetDefParseXML): Use it.
* src/uml/uml_conf.c (umlConnectTapDevice): Likewise.
* src/qemu/qemu_command.c (qemuNetworkIfaceConnect): Likewise.
Suggested by Laine Stump.

The domain XML now understands the <listen> subelement of its
<graphics> element (including when listen type='network'), and the
network driver has an internal API that will turn a network name into
an IP address, so the final logical step is to put the glue into the
qemu driver so that when it is starting up a domain, if it finds
<listen type='network' network='xyz'/> in the XML, it will call the
network driver to get an IPv4 address associated with network xyz, and
tell qemu to listen for vnc (or spice) on that address rather than the
default address (localhost).

The motivation for this is that a large installation may want the
guests' VNC servers listening on physical interfaces rather than
localhost, so that users can connect directly from the outside; this
requires sending qemu the appropriate IP address to listen on. But
this address will of course be different for each host, and if a guest
might be migrated around from one host to another, it's important that
the guest's config not have any information embedded in it that is
specific to one particular host. <listen type='network.../> can solve
this problem in the following manner:

  1) on each host, define a libvirt network of the same name,
     associated with the interface on that host that should be used
     for listening (for example, a simple macvtap network: <forward
     mode='bridge' dev='eth0'/>, or host bridge network: <forward
     mode='bridge'/> <bridge name='br0'/>

  2) in the <graphics> element of each guest's domain xml, tell vnc to
     listen on the network name used in step 1:

     <graphics type='vnc' port='5922'>
       <listen type='network'network='example-net'/>
     </graphics>

(all the above also applies for graphics type='spice').

Once it's plugged in, the <listen> element will be an optional
replacement for the "listen" attribute that graphics elements already
have. If the <listen> element is type='address', it will have an
attribute called 'address' which will contain an IP address or dns
name that the guest's display server should listen on. If, however,
type='network', the <listen> element should have an attribute called
'network' that will be set to the name of a network configuration to
get the IP address from.

* docs/schemas/domain.rng: updated to allow the <listen> element

* docs/formatdomain.html.in: document the <listen> element and its
  attributes.

* src/conf/domain_conf.[hc]:

  1) The domain parser, formatter, and data structure are modified to
     support 0 or more <listen> subelements to each <graphics>
     element. The old style "legacy" listen attribute is also still
     accepted, and will be stored internally just as if it were a
     separate <listen> element. On output (i.e. format), the address
     attribute of the first <listen> element of type 'address' will be
     duplicated in the legacy "listen" attribute of the <graphic>
     element.

  2) The "listenAddr" attribute has been removed from the unions in
     virDomainGRaphicsDef for graphics types vnc, rdp, and spice.
     This attribute is now in the <listen> subelement (aka
     virDomainGraphicsListenDef)

  3) Helper functions were written to provide simple access
     (both Get and Set) to the listen elements and their attributes.

* src/libvirt_private.syms: export the listen helper functions

* src/qemu/qemu_command.c, src/qemu/qemu_hotplug.c,
  src/qemu/qemu_migration.c, src/vbox/vbox_tmpl.c,
  src/vmx/vmx.c, src/xenxs/xen_sxpr.c, src/xenxs/xen_xm.c

  Modify all these files to use the listen helper functions rather
  than directly referencing the (now missing) listenAddr
  attribute. There can be multiple <listen> elements to a single
  <graphics>, but the drivers all currently only support one, so all
  replacements of direct access with a helper function indicate index
  "0".

* tests/* - only 3 of these are new files added explicitly to test the
  new <listen> element. All the others have been modified to reflect
  the fact that any legacy "listen" attributes passed in to the domain
  parse will be saved in a <listen> element (i.e. one of the
  virDomainGraphicsListenDefs), and during the domain format function,
  both the <listen> element as well as the legacy attributes will be
  output.

Every DomainNetDef has a bandwidth, as does every portgroup.
Whenever a DomainNetDef of type NETWORK is about to be used, a call is
made to networkAllocateActualDevice(). This function chooses the "best"
bandwidth object and places it in the DomainActualNetDef.
From that point on, whenever some code needs to use the bandwidth data
for the interface, it's retrieved with virDomainNetGetActualBandwidth(),
which will always return the "best" info as determined in the
previous step.

These function executes 'tc' with appropriate arguments to set
desired QoS setting on interface or bridge during its creation.

The network driver needs to assign physical devices for use by modes
that use macvtap, keeping track of which physical devices are in use
(and how many instances, when the devices can be shared). Three calls
are added:

networkAllocateActualDevice - finds a physical device for use by the
domain, and sets up the virDomainActualNetDef accordingly.

networkNotifyActualDevice - assumes that the domain was already
running, but libvirtd was restarted, and needs to be notified by each
already-running domain about what interfaces they are using.

networkReleaseActualDevice - decrements the usage count of the
allocated physical device, and frees the virDomainActualNetDef to
avoid later accidentally using the device.

bridge_driver.[hc] - the new APIs. When WITH_NETWORK is false, these
functions are all #defined to be "0" in the .h file (effectively
becoming a NOP) to prevent link errors.

qemu_(command|driver|hotplug|process).c - add calls to the above APIs
    in the appropriate places.

tests/Makefile.am - we need to include libvirt_driver_network.la
    whenever libvirt_driver_qemu.la is linked, to avoid unreferenced
    symbols (in functions that are never called by the test
    programs...)

The qemu driver accesses fields in the virDomainNetDef directly, but
with the advent of the virDomainActualNetDef, some pieces of
information may be found in a different place (the ActualNetDef) if
the network connection is of type='network' and that network is of
forward type='bridge|private|vepa|passthrough'. The previous patch
added functions to mask this difference from callers - they hide the
decision making process and just pick the value from the proper place.

This patch uses those functions in the qemu driver as a first step in
making qemu work with the new network types. At this point, the
virDomainActualNetDef is guaranteed always NULL, so the GetActualX()
function will return exactly what the def->X that's being replaced
would have returned (ie bisecting is not compromised).

There is one place (in qemu_driver.c) where the internal details of
the NetDef are directly manipulated by the code, so the GetActual
functions cannot be used there without extra additional code; that
file will be treated in a separate patch.

The virtPortProfile in the domain interface struct is now a separately
allocated object *pointed to by* (rather than contained in) the main
virDomainNetDef object. This is done to make it easier to figure out
when a virtualPortProfile has/hasn't been specified in a particular
config.

In preparation for a future patch adding new virFile APIs.

* src/util/files.h, src/util/files.c: Move...
* src/util/virfile.h, src/util/virfile.c: ...here, and rename
functions to virFile prefix.  Macro names are intentionally
left alone.
* *.c: All '#include "files.h"' uses changed.
* src/Makefile.am (UTIL_SOURCES): Reflect rename.
* cfg.mk (exclude_file_name_regexp--sc_prohibit_close): Likewise.
* src/libvirt_private.syms: Likewise.
* docs/hacking.html.in: Likewise.
* HACKING: Regenerate.

Since libvirt is multi-threaded, we should use FD_CLOEXEC as much
as possible in the parent, and only relax fds to inherited after
forking, to avoid leaking an fd created in one thread to a fork
run in another thread.  This gets us closer to that ideal, by
making virCommand automatically clear FD_CLOEXEC on fds intended
for the child, as well as avoiding a window of time with non-cloexec
pipes created for capturing output.

* src/util/command.c (virExecWithHook): Use CLOEXEC in parent.  In
child, guarantee that all fds to pass to child are inheritable.
(getDevNull): Use CLOEXEC.
(prepareStdFd): New helper function.
(virCommandRun, virCommandRequireHandshake): Use pipe2.
* src/qemu/qemu_command.c (qemuBuildCommandLine): Simplify caller.

The LXC and UML drivers can both make use of auditing. Move
the qemu_audit.{c,h} files to src/conf/domain_audit.{c,h}

* src/conf/domain_audit.c: Rename from src/qemu/qemu_audit.c
* src/conf/domain_audit.h: Rename from src/qemu/qemu_audit.h
* src/Makefile.am: Remove qemu_audit.{c,h}, add domain_audit.{c,h}
* src/qemu/qemu_audit.h, src/qemu/qemu_cgroup.c,
  src/qemu/qemu_command.c, src/qemu/qemu_driver.c,
  src/qemu/qemu_hotplug.c, src/qemu/qemu_migration.c,
  src/qemu/qemu_process.c: Update for changed audit API names

Given a PID, the QEMU driver reads /proc/$PID/cmdline and
/proc/$PID/environ to get the configuration. This is fed
into the ARGV->XML convertor to build an XML configuration
for the process.

/proc/$PID/exe is resolved to identify the full command
binary path

After checking for name/uuid uniqueness, an attempt is
made to connect to the monitor socket. If successful
then 'info status' and 'info kvm' are issued to determine
whether the CPUs are running and if KVM is enabled.

* src/qemu/qemu_driver.c: Implement virDomainQemuAttach
* src/qemu/qemu_process.h, src/qemu/qemu_process.c: Add
  qemuProcessAttach to connect to the monitor of an
  existing QEMU process

To enable attaching to externally launched QEMU, we need
to be able to reverse engineer a guest XML config based
on the argv for a PID in /proc

* src/qemu/qemu_command.c, src/qemu/qemu_command.h: Add
  qemuParseCommandLinePid which extracts QEMU config from
  argv in /proc, given a PID number

When converting QEMU argv into a virDomainDefPtr, also extract
the pidfile, monitor character device config and the monitor
mode.

* src/qemu/qemu_command.c, src/qemu/qemu_command.h: Extract
  pidfile & monitor config from QEMU argv
* src/qemu/qemu_driver.c, tests/qemuargv2xmltest.c: Add extra
  params when calling qemuParseCommandLineString

This patch creates new <bios> element which, at this time has only the
attribute useserial='yes|no'. This attribute allow users to use
Serial Graphics Adapter and see BIOS messages from the very first moment
domain boots up. Therefore, users can choose boot medium, set PXE, etc.

cmd is not initialized to NULL, but we try to freed it if we
meet some error.

Coverity warns if the majority of callers check a function for
errors, but a few don't; but in qemu_audit and qemu_domain, the
choice to not check for failures was safe.  In qemu_command, the
failure to generate a uuid can only occur on a bad pointer.

* src/qemu/qemu_audit.c (qemuAuditCgroup): Ignore failure to get
  cgroup controller.
* src/qemu/qemu_domain.c (qemuDomainObjEnterMonitor)
  (qemuDomainObjEnterMonitorWithDriver): Ignore failure to get
  timestamp.
* src/qemu/qemu_command.c (qemuParseCommandLine): Check for error.

The 'function' field in the PCI address was not correctly
initialized, so it was building the wrong address address
string and so not removing all functions from the in use
list.

* src/qemu/qemu_command.c: Fix initialization of PCI function

For controlled shutdown we issue a 'system_powerdown' command
to the QEMU monitor. This triggers an ACPI event which (most)
guest OS wire up to a controlled shutdown. There is no equiv
ACPI event to trigger a controlled reboot. This patch attempts
to fake a reboot.

 - In qemuDomainObjPrivatePtr we have a bool fakeReboot
   flag.
 - The virDomainReboot method sets this flag and then
   triggers a normal 'system_powerdown'.
 - The QEMU process is started with '-no-shutdown'
   so that the guest CPUs pause when it powers off the
   guest
 - When we receive the 'POWEROFF' event from QEMU JSON
   monitor if fakeReboot is not set we invoke the
   qemuProcessKill command and shutdown continues
   normally
 - If fakeReboot was set, we spawn a background thread
   which issues 'system_reset' to perform a warm reboot
   of the guest hardware. Then it issues 'cont' to
   start the CPUs again

* src/qemu/qemu_command.c: Add -no-shutdown flag if
  we have JSON support
* src/qemu/qemu_domain.h: Add 'fakeReboot' flag to
  qemuDomainObjPrivate struct
* src/qemu/qemu_driver.c: Fake reboot using the
  system_powerdown command if JSON support is available
* src/qemu/qemu_monitor.c, src/qemu/qemu_monitor.h,
  src/qemu/qemu_monitor_json.c, src/qemu/qemu_monitor_json.h,
  src/qemu/qemu_monitor_text.c, src/qemu/qemu_monitor_text.h: Add
  binding for system_reset command
* src/qemu/qemu_process.c: Reset the guest & start CPUs if
  fakeReboot is set

For virtio disks and interfaces, qemu allows users to enable or disable
ioeventfd feature. This means, qemu can execute domain code, while
another thread waits for I/O event. Basically, in some cases it is win,
in some loss. This feature is available via 'ioeventfd' attribute in disk
and interface <driver> element. It accepts 'on' and 'off'. Leaving this
attribute out defaults to hypervisor decision.

The following patch addresses the problem that when a PASSTHROUGH
mode DIRECT NIC connection is made the MAC address of the NIC is
not automatically set and reset to the configured VM MAC and
back again.

The attached patch fixes this problem by setting and resetting the MAC
while remembering the previous setting while the VM is running.
This also works if libvirtd is restarted while the VM is running.

the patch passes make syntax-check

Prefer bootindex=N option for -device over the old way -boot ORDER
possibly accompanied with boot=on option for -drive. This gives us full
control over which device will actually be used for booting guest OS.
Moreover, if qemu doesn't support boot=on, this is the only way to boot
of certain disks in some configurations (such as virtio disks when used
together IDE disks) without transforming domain XML to use per device
boot elements.

From a security pov copy and paste between the guest and the client is not
always desirable. So we need to be able to enable/disable this. The best place
to do this from an administration pov is on the hypervisor, so the qemu cmdline
is getting a spice disable-copy-paste option, see bug 693645. Example qemu
invocation:
qemu -spice port=5932,disable-ticketing,disable-copy-paste

https://bugzilla.redhat.com/show_bug.cgi?id=693661

If qemu supports multi function PCI device, the format of the PCI address passed
to qemu is "bus=pci.0,multifunction=on,addr=slot.function".

If qemu does not support multi function PCI device, the format of the PCI address
passed to qemu is "bus=pci.0,addr=slot".

Hot pluging/unpluging multi PCI device is not supported now. So the function
of hotplugged PCI device must be 0. When we hot unplug it, we should set release
all functions in the slot.

If user does not specify the PCI address, we should auto assign an unused slot.

We will support multi function PCI device. So we should reserve all functions in
the slot if we want to reserve a slot.