The domain XML now understands the <listen> subelement of its
<graphics> element (including when listen type='network'), and the
network driver has an internal API that will turn a network name into
an IP address, so the final logical step is to put the glue into the
qemu driver so that when it is starting up a domain, if it finds
<listen type='network' network='xyz'/> in the XML, it will call the
network driver to get an IPv4 address associated with network xyz, and
tell qemu to listen for vnc (or spice) on that address rather than the
default address (localhost).

The motivation for this is that a large installation may want the
guests' VNC servers listening on physical interfaces rather than
localhost, so that users can connect directly from the outside; this
requires sending qemu the appropriate IP address to listen on. But
this address will of course be different for each host, and if a guest
might be migrated around from one host to another, it's important that
the guest's config not have any information embedded in it that is
specific to one particular host. <listen type='network.../> can solve
this problem in the following manner:

  1) on each host, define a libvirt network of the same name,
     associated with the interface on that host that should be used
     for listening (for example, a simple macvtap network: <forward
     mode='bridge' dev='eth0'/>, or host bridge network: <forward
     mode='bridge'/> <bridge name='br0'/>

  2) in the <graphics> element of each guest's domain xml, tell vnc to
     listen on the network name used in step 1:

     <graphics type='vnc' port='5922'>
       <listen type='network'network='example-net'/>
     </graphics>

(all the above also applies for graphics type='spice').

Once it's plugged in, the <listen> element will be an optional
replacement for the "listen" attribute that graphics elements already
have. If the <listen> element is type='address', it will have an
attribute called 'address' which will contain an IP address or dns
name that the guest's display server should listen on. If, however,
type='network', the <listen> element should have an attribute called
'network' that will be set to the name of a network configuration to
get the IP address from.

* docs/schemas/domain.rng: updated to allow the <listen> element

* docs/formatdomain.html.in: document the <listen> element and its
  attributes.

* src/conf/domain_conf.[hc]:

  1) The domain parser, formatter, and data structure are modified to
     support 0 or more <listen> subelements to each <graphics>
     element. The old style "legacy" listen attribute is also still
     accepted, and will be stored internally just as if it were a
     separate <listen> element. On output (i.e. format), the address
     attribute of the first <listen> element of type 'address' will be
     duplicated in the legacy "listen" attribute of the <graphic>
     element.

  2) The "listenAddr" attribute has been removed from the unions in
     virDomainGRaphicsDef for graphics types vnc, rdp, and spice.
     This attribute is now in the <listen> subelement (aka
     virDomainGraphicsListenDef)

  3) Helper functions were written to provide simple access
     (both Get and Set) to the listen elements and their attributes.

* src/libvirt_private.syms: export the listen helper functions

* src/qemu/qemu_command.c, src/qemu/qemu_hotplug.c,
  src/qemu/qemu_migration.c, src/vbox/vbox_tmpl.c,
  src/vmx/vmx.c, src/xenxs/xen_sxpr.c, src/xenxs/xen_xm.c

  Modify all these files to use the listen helper functions rather
  than directly referencing the (now missing) listenAddr
  attribute. There can be multiple <listen> elements to a single
  <graphics>, but the drivers all currently only support one, so all
  replacements of direct access with a helper function indicate index
  "0".

* tests/* - only 3 of these are new files added explicitly to test the
  new <listen> element. All the others have been modified to reflect
  the fact that any legacy "listen" attributes passed in to the domain
  parse will be saved in a <listen> element (i.e. one of the
  virDomainGraphicsListenDefs), and during the domain format function,
  both the <listen> element as well as the legacy attributes will be
  output.

The network driver needs to assign physical devices for use by modes
that use macvtap, keeping track of which physical devices are in use
(and how many instances, when the devices can be shared). Three calls
are added:

networkAllocateActualDevice - finds a physical device for use by the
domain, and sets up the virDomainActualNetDef accordingly.

networkNotifyActualDevice - assumes that the domain was already
running, but libvirtd was restarted, and needs to be notified by each
already-running domain about what interfaces they are using.

networkReleaseActualDevice - decrements the usage count of the
allocated physical device, and frees the virDomainActualNetDef to
avoid later accidentally using the device.

bridge_driver.[hc] - the new APIs. When WITH_NETWORK is false, these
functions are all #defined to be "0" in the .h file (effectively
becoming a NOP) to prevent link errors.

qemu_(command|driver|hotplug|process).c - add calls to the above APIs
    in the appropriate places.

tests/Makefile.am - we need to include libvirt_driver_network.la
    whenever libvirt_driver_qemu.la is linked, to avoid unreferenced
    symbols (in functions that are never called by the test
    programs...)

The qemu driver accesses fields in the virDomainNetDef directly, but
with the advent of the virDomainActualNetDef, some pieces of
information may be found in a different place (the ActualNetDef) if
the network connection is of type='network' and that network is of
forward type='bridge|private|vepa|passthrough'. The previous patch
added functions to mask this difference from callers - they hide the
decision making process and just pick the value from the proper place.

This patch uses those functions in the qemu driver as a first step in
making qemu work with the new network types. At this point, the
virDomainActualNetDef is guaranteed always NULL, so the GetActualX()
function will return exactly what the def->X that's being replaced
would have returned (ie bisecting is not compromised).

There is one place (in qemu_driver.c) where the internal details of
the NetDef are directly manipulated by the code, so the GetActual
functions cannot be used there without extra additional code; that
file will be treated in a separate patch.

The virtPortProfile in the domain interface struct is now a separately
allocated object *pointed to by* (rather than contained in) the main
virDomainNetDef object. This is done to make it easier to figure out
when a virtualPortProfile has/hasn't been specified in a particular
config.

In preparation for a future patch adding new virFile APIs.

* src/util/files.h, src/util/files.c: Move...
* src/util/virfile.h, src/util/virfile.c: ...here, and rename
functions to virFile prefix.  Macro names are intentionally
left alone.
* *.c: All '#include "files.h"' uses changed.
* src/Makefile.am (UTIL_SOURCES): Reflect rename.
* cfg.mk (exclude_file_name_regexp--sc_prohibit_close): Likewise.
* src/libvirt_private.syms: Likewise.
* docs/hacking.html.in: Likewise.
* HACKING: Regenerate.

When monitor is entered with qemuDomainObjEnterMonitorWithDriver, the
correct method for leaving and unlocking the monitor is
qemuDomainObjExitMonitorWithDriver.

If libvirtd is restarted when a job is running, the new libvirtd process
needs to know about that to be able to recover and rollback the
operation.

Query commands are safe to be called during long running jobs (such as
migration). This patch makes them all work without the need to
special-case every single one of them.

The patch introduces new job.asyncCond condition and associated
job.asyncJob which are dedicated to asynchronous (from qemu monitor
point of view) jobs that can take arbitrarily long time to finish while
qemu monitor is still usable for other commands.

The existing job.active (and job.cond condition) is used all other
synchronous jobs (including the commands run during async job).

Locking schema is changed to use these two conditions. While asyncJob is
active, only allowed set of synchronous jobs is allowed (the set can be
different according to a particular asyncJob) so any method that
communicates to qemu monitor needs to check if it is allowed to be
executed during current asyncJob (if any). Once the check passes, the
method needs to normally acquire job.cond to ensure no other command is
running. Since domain object lock is released during that time, asyncJob
could have been started in the meantime so the method needs to recheck
the first condition. Then, normal jobs set job.active and asynchronous
jobs set job.asyncJob and optionally change the list of allowed job
groups.

Since asynchronous jobs only set job.asyncJob, other allowed commands
can still be run when domain object is unlocked (when communicating to
remote libvirtd or sleeping). To protect its own internal synchronous
commands, the asynchronous job needs to start a special nested job
before entering qemu monitor. The nested job doesn't check asyncJob, it
only acquires job.cond and sets job.active to block other jobs.

The LXC and UML drivers can both make use of auditing. Move
the qemu_audit.{c,h} files to src/conf/domain_audit.{c,h}

* src/conf/domain_audit.c: Rename from src/qemu/qemu_audit.c
* src/conf/domain_audit.h: Rename from src/qemu/qemu_audit.h
* src/Makefile.am: Remove qemu_audit.{c,h}, add domain_audit.{c,h}
* src/qemu/qemu_audit.h, src/qemu/qemu_cgroup.c,
  src/qemu/qemu_command.c, src/qemu/qemu_driver.c,
  src/qemu/qemu_hotplug.c, src/qemu/qemu_migration.c,
  src/qemu/qemu_process.c: Update for changed audit API names

This option accepts 3 values:
-keep, to keep current client connected (Spice+VNC)
-disconnect, to disconnect client (Spice)
-fail, to fail setting password if there is a client connected (Spice)

The following patch addresses the problem that when a PASSTHROUGH
mode DIRECT NIC connection is made the MAC address of the NIC is
not automatically set and reset to the configured VM MAC and
back again.

The attached patch fixes this problem by setting and resetting the MAC
while remembering the previous setting while the VM is running.
This also works if libvirtd is restarted while the VM is running.

the patch passes make syntax-check

Prefer bootindex=N option for -device over the old way -boot ORDER
possibly accompanied with boot=on option for -drive. This gives us full
control over which device will actually be used for booting guest OS.
Moreover, if qemu doesn't support boot=on, this is the only way to boot
of certain disks in some configurations (such as virtio disks when used
together IDE disks) without transforming domain XML to use per device
boot elements.

Hot pluging/unpluging multi PCI device is not supported now. So the function
of hotplugged PCI device must be 0. When we hot unplug it, we should set release
all functions in the slot.

We do not support to hot unplug multi function PCI device now. If the device is
one function of multi function PCI device, we shoul not allow to hot unplugg
it.

As long as I was already touching the function...

* src/qemu/qemu_hotplug.c (qemuDomainChangeGraphics): Line wrap.

Detected by Coverity.  Bug introduced in commit 9d73efdb (v0.8.8).

* src/qemu/qemu_hotplug.c (qemuDomainChangeGraphics): Don't report
error on success.

The qemuAuditDisk calls in disk hotunplug operations were being
passed 'ret >= 0', but the code which sets ret to 0 was not yet
executed, and the error path had already jumped to the 'cleanup'
label. This meant hotunplug failures were never audited, and
hotunplug success was audited as a failure

* src/qemu/qemu_hotplug.c: Fix auditing of hotunplug

* src/conf/domain_conf.c, src/conf/domain_conf.h: APIs for
  inserting/finding/removing virDomainLeaseDefPtr instances
* src/qemu/qemu_driver.c: Wire up hotplug/unplug for leases
* src/qemu/qemu_hotplug.h, src/qemu/qemu_hotplug.c: Support
  for hotplug and unplug of leases

The QEMU integrates with the lock manager instructure in a number
of key places

 * During startup, a lock is acquired in between the fork & exec
 * During startup, the libvirtd process acquires a lock before
   setting file labelling
 * During shutdown, the libvirtd process acquires a lock
   before restoring file labelling
 * During hotplug, unplug & media change the libvirtd process
   holds a lock while setting/restoring labels

The main content lock is only ever held by the QEMU child process,
or libvirtd during VM shutdown. The rest of the operations only
require libvirtd to hold the metadata locks, relying on the active
QEMU still holding the content lock.

* src/qemu/qemu_conf.c, src/qemu/qemu_conf.h,
  src/qemu/libvirtd_qemu.aug, src/qemu/test_libvirtd_qemu.aug:
  Add config parameter for configuring lock managers
* src/qemu/qemu_driver.c: Add calls to the lock manager

These VIR_XXXX0 APIs make us confused, use the non-0-suffix APIs instead.

How do these coversions works? The magic is using the gcc extension of ##.
When __VA_ARGS__ is empty, "##" will swallow the "," in "fmt," to
avoid compile error.

example: origin				after CPP
	high_level_api("%d", a_int)	low_level_api("%d", a_int)
	high_level_api("a  string")	low_level_api("a  string")

About 400 conversions.

8 special conversions:
VIR_XXXX0("") -> VIR_XXXX("msg") (avoid empty format) 2 conversions
VIR_XXXX0(string_literal_with_%) -> VIR_XXXX(%->%%) 0 conversions
VIR_XXXX0(non_string_literal) -> VIR_XXXX("%s", non_string_literal)
  (for security) 6 conversions
Signed-off-by: NLai Jiangshan <laijs@cn.fujitsu.com>

* src/qemu/qemu_hotplug.h, src/qemu/qemu_hotplug.c: Remove
  qemuCaps parameters from all methods
* src/qemu/qemu_driver.c: Don't create & pass qemuCaps to
  hotplug methods

Steps to reproduce this bug:
1. # cat net.xml # 00:03.0 has been used
    <interface type='network'>
      <mac address='52:54:00:04:72:f3'/>
      <source network='default'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/>
    </interface>

2. # virsh attach-device vm1 net.xml
   error: Failed to attach device from net.xml
   error: internal error unable to reserve PCI address 0:0:3

3. # virsh attach-device vm1 net.xml
   error: Failed to attach device from net.xml
   error: internal error unable to execute QEMU command 'device_add': Device 'rtl8139' could not be initialized

The reason of this bug is that: we can not reserve PCI address 0:0:3 because it has
been used, but we release PCI address when we reserve it failed.

Even with -Wuninitialized (which is part of autobuild.sh
--enable-compile-warnings=error), gcc does NOT catch this
use of an uninitialized variable:

{
  if (cond)
    goto error;
  int a = 1;
error:
  printf("%d", a);
}

which prints 0 (supposing the stack started life wiped) if
cond was true.  Clang will catch it, but we don't use clang
as often.  Using gcc -Wjump-misses-init catches it, but also
gives false positives:

{
  if (cond)
    goto error;
  int a = 1;
  return a;
error:
  return 0;
}

Here, a was never used in the scope of the error block, so
declaring it after goto is technically fine (and clang agrees).
However, given that our HACKING already documents a preference
to C89 decl-before-statement, the false positive warning is
enough of a prod to comply with HACKING.

[Personally, I'd _really_ rather use C99 decl-after-statement
to minimize scope, but until gcc can efficiently and reliably
catch scoping and uninitialized usage bugs, I'll settle with
the compromise of enforcing a coding standard that happens to
reject false positives if it can also detect real bugs.]

* acinclude.m4 (LIBVIRT_COMPILE_WARNINGS): Add -Wjump-misses-init.
* src/util/util.c (__virExec): Adjust offenders.
* src/conf/domain_conf.c (virDomainTimerDefParseXML): Likewise.
* src/remote/remote_driver.c (doRemoteOpen): Likewise.
* src/phyp/phyp_driver.c (phypGetLparNAME, phypGetLparProfile)
(phypGetVIOSFreeSCSIAdapter, phypVolumeGetKey)
(phypGetStoragePoolDevice)
(phypVolumeGetPhysicalVolumeByStoragePool)
(phypVolumeGetPath): Likewise.
* src/vbox/vbox_tmpl.c (vboxNetworkUndefineDestroy)
(vboxNetworkCreate, vboxNetworkDumpXML)
(vboxNetworkDefineCreateXML): Likewise.
* src/xenapi/xenapi_driver.c (getCapsObject)
(xenapiDomainDumpXML): Likewise.
* src/xenapi/xenapi_utils.c (createVMRecordFromXml): Likewise.
* src/security/security_selinux.c (SELinuxGenNewContext):
Likewise.
* src/qemu/qemu_command.c (qemuBuildCommandLine): Likewise.
* src/qemu/qemu_hotplug.c (qemuDomainChangeEjectableMedia):
Likewise.
* src/qemu/qemu_process.c (qemuProcessWaitForMonitor): Likewise.
* src/qemu/qemu_monitor_text.c (qemuMonitorTextGetPtyPaths):
Likewise.
* src/qemu/qemu_driver.c (qemudDomainShutdown)
(qemudDomainBlockStats, qemudDomainMemoryPeek): Likewise.
* src/storage/storage_backend_iscsi.c
(virStorageBackendCreateIfaceIQN): Likewise.
* src/node_device/node_device_udev.c (udevProcessPCI): Likewise.

Steps to reproduce this bug:
# cat usb.xml
<hostdev mode='subsystem' type='usb'>
  <source>
    <address bus='0x001' device='0x003'/>
  </source>
</hostdev>
# virsh attach-device vm1 usb.xml
error: Failed to attach device from usb.xml
error: server closed connection:

The reason of this bug is that we set data.cgroup to NULL, and this will cause
libvirtd crashed.
Signed-off-by: NWen Congyang <wency@cn.fujitsu.com>

With only a single caller to these two monitor commands, I
didn't need to wrap a new WithFds version, but just change
the command itself.

* src/qemu/qemu_monitor.h (qemuMonitorAddNetdev)
(qemuMonitorAddHostNetwork): Add parameters.
* src/qemu/qemu_monitor.c (qemuMonitorAddNetdev)
(qemuMonitorAddHostNetwork): Add support for fd passing.
* src/qemu/qemu_hotplug.c (qemuDomainAttachNetDevice): Use it to
simplify code.

This is also a bug fix - on the error path, qemu_hotplug would
leave the configfd file leaked into qemu.  At least the next
attempt to hotplug a PCI device would reuse the same fdname,
and when the qemu getfd monitor command gets a new fd by the
same name as an earlier one, it closes the earlier one, so there
is no risk of qemu running out of fds.

* src/qemu/qemu_monitor.h (qemuMonitorAddDeviceWithFd): New
prototype.
* src/qemu/qemu_monitor.c (qemuMonitorAddDevice): Move guts...
(qemuMonitorAddDeviceWithFd): ...to new function, and add support
for fd passing.
* src/qemu/qemu_hotplug.c (qemuDomainAttachHostPciDevice): Use it
to simplify code.
Suggested by Daniel P. Berrange.

qemu_monitor was already returning -1 and setting errno to EINVAL
on any attempt to send an fd without a unix socket, but this was
a silent failure in the case of qemuDomainAttachHostPciDevice.
Meanwhile, qemuDomainAttachNetDevice was doing some sanity checking
for a better error message; it's better to consolidate that to a
central point in the API.

* src/qemu/qemu_hotplug.c (qemuDomainAttachNetDevice): Move sanity
checking...
* src/qemu/qemu_monitor.c (qemuMonitorSendFileHandle): ...into
central location.
Suggested by Chris Wright.

THREADS.txt states that the contents of vm should not be read or
modified while the vm lock is not held, but that the lock must not
be held while performing a monitor command.  This fixes all the
offenders that I could find.

* src/qemu/qemu_process.c (qemuProcessStartCPUs)
(qemuProcessInitPasswords, qemuProcessStart): Don't modify or
refer to vm state outside lock.
* src/qemu/qemu_driver.c (qemudDomainHotplugVcpus): Likewise.
* src/qemu/qemu_hotplug.c (qemuDomainChangeGraphicsPasswords):
Likewise.

Opening raw network devices with the intent of passing those fds to
qemu is worth an audit point.  This makes a multi-part audit: first,
we audit the device(s) that libvirt opens on behalf of the MAC address
of a to-be-created interface (which can independently succeed or
fail), then we audit whether qemu actually started the network device
with the same MAC (so searching backwards for successful audits with
the same MAC will show which fd(s) qemu is actually using).  Note that
it is possible for the fd to be successfully opened but no attempt
made to pass the fd to qemu (for example, because intermediate
nwfilter operations failed) - no interface start audit will occur in
that case; so the audit for a successful opened fd does not imply
rights given to qemu unless there is a followup audit about the
attempt to start a new interface.

Likewise, when a network device is hot-unplugged, there is only one
audit message about the MAC being discontinued; again, searching back
to the earlier device open audits will show which fds that qemu quits
using (and yes, I checked via /proc/<qemu-pid>/fd that qemu _does_
close out the fds associated with an interface on hot-unplug).  The
code would require much more refactoring to be able to definitively
state which device(s) were discontinued at that point, since we
currently don't record anywhere in the XML whether /dev/vhost-net was
opened for a given interface.

* src/qemu/qemu_audit.h (qemuAuditNetDevice): New prototype.
* src/qemu/qemu_audit.c (qemuAuditNetDevice): New function.
* src/qemu/qemu_command.h (qemuNetworkIfaceConnect)
(qemuPhysIfaceConnect, qemuOpenVhostNet): Adjust prototype.
* src/qemu/qemu_command.c (qemuNetworkIfaceConnect)
(qemuPhysIfaceConnect, qemuOpenVhostNet): Add audit points and
adjust parameters.
(qemuBuildCommandLine): Adjust caller.
* src/qemu/qemu_hotplug.c (qemuDomainAttachNetDevice): Likewise.

* src/qemu/qemu_hotplug.c (qemuDomainAttachNetDevice): Honor vhost
designations, similar to qemu_command code paths.
* src/qemu/qemu_command.h (qemuOpenVhostNet): New prototype.
* src/qemu/qemu_command.c (qemuOpenVhostNet): Export.

Also add ATTRIBUTE_NONNULL markers.

* src/qemu/qemu_audit.h: The pattern qemuDomainXXXAudit is
inconsistent; prefer qemuAuditXXX instead.
* src/qemu/qemu_audit.c: Reflect the renames.
* src/qemu/qemu_driver.c: Likewise.
* src/qemu/qemu_hotplug.c: Likewise.
* src/qemu/qemu_migration.c: Likewise.
* src/qemu/qemu_process.c: Likewise.

The way to detach a USB disk is the same as that to detach a SCSI
disk. Rename this function and we can use it to detach a USB disk.
Signed-off-by: NWen Congyang <wency@cn.fujitsu.com>

Currently it sounds like spice is completely unsupported, which is
confusing.

* src/qemu/qemu_audit.h (qemuDomainHostdevAudit): New prototype.
* src/qemu/qemu_audit.c (qemuDomainHostdevAudit): New function.
(qemuDomainStartAudit): Call as appropriate.
* src/qemu/qemu_hotplug.c (qemuDomainAttachHostPciDevice)
(qemuDomainAttachHostUsbDevice, qemuDomainDetachHostPciDevice)
(qemuDomainDetachHostUsbDevice): Likewise.

* src/qemu/qemu_cgroup.h (struct qemuCgroupData): New helper type.
(qemuSetupDiskPathAllow, qemuSetupChardevCgroup)
(qemuTeardownDiskPathDeny): Drop unneeded prototypes.
(qemuSetupDiskCgroup, qemuTeardownDiskCgroup): Adjust prototype.
* src/qemu/qemu_cgroup.c
(qemuSetupDiskPathAllow, qemuSetupChardevCgroup)
(qemuTeardownDiskPathDeny): Mark static and use new type.
(qemuSetupHostUsbDeviceCgroup): Use new type.
(qemuSetupDiskCgroup): Alter signature.
(qemuSetupCgroup): Adjust caller.
* src/qemu/qemu_hotplug.c (qemuDomainAttachHostUsbDevice)
(qemuDomainDetachPciDiskDevice, qemuDomainDetachSCSIDiskDevice):
Likewise.
* src/qemu/qemu_driver.c (qemudDomainAttachDevice)
(qemuDomainUpdateDeviceFlags): Likewise.

This is done for two reasons:
- we are getting very close to 64 flags which is the maximum we can use
  with unsigned long long
- by using LL constants in enum we already violates C99 constraint that
  enum values have to fit into int

The new name complies more with the fact that it contains a set of
qemuCapsFlags.

Three new functions (qemuCapsSet, qemuCapsClear, and qemuCapsGet) were
introduced replacing direct bit operations.

The new names comply more with the fact that they are all members of
enum qemuCapsFlags.