In a first step I am converting the netlink message construction in
macvtap code to use libnl. It's pretty much a 1:1 conversion except that
now the message needs to be allocated and deallocated.

When <uuid> is not in the XML, a virUUIDGenerate() ends up being called which
is unnecessary and can lead to crashes if /dev/urandom isn't available
because virRandomInitialize() is not called within virt-aa-helper. This patch
adds verify_xpath_context() and updates caps_mockup() to use it.

Bug-Ubuntu: https://launchpad.net/bugs/672943

If virDomainAttachDevice() was called with an image that was located
on a root-squashed NFS server, and in a directory that was unreadable
by root on the machine running libvirtd, the attach would fail due to
an attempt to change the selinux label of the image with EACCES (which
isn't covered as an ignore case in SELinuxSetFilecon())

NFS doesn't support SELinux labelling anyway, so we mimic the failure
handling of commit 93a18bba, which
just ignores the errors if the target is on an NFS filesystem (in
SELinuxSetSecurityAllLabel() only, though.)

This can be seen as a follow-on to commit
347d266c, which ignores file open
failures of files on NFS that occur directly in
virDomainDiskDefForeachPath() (also necessary), but does not ignore
failures in functions that are called from there (eg
SELinuxSetSecurityFileLabel()).

Introduce implementations of the virDomainOpenConsole() API
for LXC, Xen and UML drivers.

* src/lxc/lxc_driver.c, src/lxc/lxc_driver.c,
  src/xen/xen_driver.c: Wire up virDomainOpenConsole

The util/threads.c/h code already has APIs for mutexes,
condition variables and thread locals. This commit adds
in code for actually creating threads.

* src/libvirt_private.syms: Export new symbols
* src/util/threads.h: Define APIs virThreadCreate, virThreadSelf,
  virThreadIsSelf and virThreadJoin
* src/util/threads-win32.c, src/util/threads-win32.h: Win32
  impl of threads
* src/util/threads-pthread.c, src/util/threads-pthread.h: POSIX
  impl of threads

This provides an implementation of the virDomainOpenConsole
API with the QEMU driver. For the streams code, this reuses
most of the code previously added for the tunnelled migration
streams since it is generic.

* src/qemu/qemu_driver.c: Support virDomainOpenConsole

To avoid the need for duplicating implementations of virStream
drivers, provide a generic implementation that can handle any
FD based stream. This code is copied from the existing impl
in the QEMU driver, with the locking moved into the stream
impl, and addition of a read callback

The FD stream code will refuse to operate on regular files or
block devices, since those can't report EAGAIN properly when
they would block on I/O

* include/libvirt/virterror.h, include/libvirt/virterror.h: Add
  VIR_FROM_STREAM error domain
* src/qemu/qemu_driver.c: Remove code obsoleted by the new
  generic streams driver.
* src/fdstream.h, src/fdstream.c, src/fdstream.c,
  src/libvirt_private.syms: Generic reusable FD based streams

Now that bi-directional, non-blocking streams are supported
in the remote driver, some of the VIR_WARN statements need
to be reduced to VIR_DEBUG.

* src/remote/remote_driver.c: Lower logging level

This provides an implementation of the virDomainOpenConsole
API for the remote driver client and server.

* daemon/remote.c: Server side impl
* src/remote/remote_driver.c: Client impl
* src/remote/remote_protocol.x: Wire definition

To enable virsh console (or equivalent) to be used remotely
it is necessary to provide remote access to the /dev/pts/XXX
pseudo-TTY associated with the console/serial/parallel device
in the guest. The virStream API provide a bi-directional I/O
stream capability that can be used for this purpose. This
patch thus introduces a virDomainOpenConsole API that uses
the stream APIs.

* src/libvirt.c, src/libvirt_public.syms,
  include/libvirt/libvirt.h.in, src/driver.h: Define the
  new virDomainOpenConsole API
* src/esx/esx_driver.c, src/lxc/lxc_driver.c,
  src/opennebula/one_driver.c, src/openvz/openvz_driver.c,
  src/phyp/phyp_driver.c, src/qemu/qemu_driver.c,
  src/remote/remote_driver.c, src/test/test_driver.c,
  src/uml/uml_driver.c, src/vbox/vbox_tmpl.c,
  src/xen/xen_driver.c, src/xenapi/xenapi_driver.c: Stub
  API entry point

The current remote driver code for streams only supports
blocking I/O mode. This is fine for the usage with migration
but is a problem for more general use cases, in particular
bi-directional streams.

This adds supported for the stream callbacks and non-blocking
I/O. with the minor caveat is that it doesn't actually do
non-blocking I/O for sending stream data, only receiving it.
A future patch will try to do non-blocking sends, but this is
quite tricky to get right.

* src/remote/remote_driver.c: Allow non-blocking I/O for
  streams and support callbacks

The /dev/console device inside the container must NOT map
to the real /dev/console device node, since this allows the
container control over the current host console. A fun side
effect of this is that starting a container containing a
real Fedora OS will kill off your X server.

Remove the /dev/console node, and replace it with a symlink
to the primary console TTY

* src/lxc/lxc_container.c: Replace /dev/console with a
  symlink to /dev/pty/0
* src/lxc/lxc_controller.c: Remove /dev/console from cgroups
  ACL

QEMU allows forcing a CDROM eject even if the guest has locked the device.
Expose this via a new UpdateDevice flag, VIR_DOMAIN_DEVICE_MODIFY_FORCE.

This has been requested for RHEV:

https://bugzilla.redhat.com/show_bug.cgi?id=626305

v2: Change flag name, bool cleanups

I am trying to use a qcow image with libvirt where the backing 'file' is a
qemu-nbd server.  Unfortunately virDomainDiskDefForeachPath() assumes that
backingStore is always a real file so something like 'nbd:0:3333' is rejected
because a file with that name cannot be accessed.  Note that I am not worried
about directly using nbd images.  That would require a new disk type with XML
markup, etc.  I only want it to be permitted as a backingStore

The following patch implements danpb's suggestion:
> I think I'm inclined to push the logic for skipping NBD one stage higher.
> I'd rather expect virStorageFileGetMetadata() to return all backing
> stores, even if not files. The virDomainDiskDefForeachPath() method
> should definitely ignore non-file backing stores though.
>
> So what I'm thinking is to extend the virStorageFileMetadata struct and
> just add a 'bool isFile' field to it. Default this field to true, unless
> you see the prefix of nbd: in which case set it to false. The
> virDomainDiskDefForeachPath() method can then skip over any backing
> store with isFile == false
Signed-off-by: NAdam Litke <agl@us.ibm.com>
Cc: Daniel P. Berrange <berrange@redhat.com>

xencapstest calls xenHypervisorMakeCapabilitiesInternal with conn == NULL
which calls xenDaemonNodeGetTopology with conn == NULL when a recent
enough Xen was detected (sys_interface_version >= SYS_IFACE_MIN_VERS_NUMA).
But xenDaemonNodeGetTopology insists in having conn != NULL and fails,
because it expects to be able to talk to an actual xend.

We cannot do that in a 'make check' test. Therefore, only call the xend
subdriver function when conn isn't NULL.

Reported by Andy Howell and Jim Fehlig.

Using automated replacement with sed and editing I have now replaced all
occurrences of close() with VIR_(FORCE_)CLOSE() except for one, of
course. Some replacements were straight forward, others I needed to pay
attention. I hope I payed attention in all the right places... Please
have a look. This should have at least solved one more double-close
error.

Also add a test case for the VMX handling of it.

This extends the SPICE XML to allow channel security options

    <graphics type='spice' port='-1' tlsPort='-1' autoport='yes'>
      <channel name='main' mode='secure'/>
      <channel name='record' mode='insecure'/>
    </graphics>

Any non-specified channel uses the default, which allows both
secure & insecure usage

* src/conf/domain_conf.c, src/conf/domain_conf.h,
  src/libvirt_private.syms: Add XML syntax for specifying per
  channel security options for spice.
* src/qemu/qemu_conf.c: Configure channel security with spice

QEMU crashes & burns if you try multiple Cirrus video cards, but
QXL copes fine. Adapt QEMU config code to allow multiple QXL
video cards

* src/qemu/qemu_conf.c: Support multiple QXL video cards

This extends the XML syntax for <graphics> to allow a password
expiry time to be set

eg

  <graphics type='vnc' port='5900' autoport='yes' keymap='en-us' passwd='12345' passwdValidTo='2010-04-09T15:51:00'/>

The timestamp is in UTC.

* src/conf/domain_conf.h: Pull passwd out into separate struct
  virDomainGraphicsAuthDef to allow sharing between VNC & SPICE
* src/conf/domain_conf.c: Add parsing/formatting of new passwdValidTo
  argument
* src/opennebula/one_conf.c, src/qemu/qemu_conf.c, src/qemu/qemu_driver.c,
  src/xen/xend_internal.c, src/xen/xm_internal.c: Update for changed
  struct containing VNC password

In common with VNC, the QEMU driver configuration file is used
specify the host level TLS certificate location and a default
password / listen address

* src/qemu/qemu.conf: Add spice_listen, spice_tls,
  spice_tls_x509_cert_dir & spice_password config params
* src/qemu/qemu_conf.c, src/qemu/qemu_conf.h: Parsing of
  spice config parameters and updating -spice arg generation
  to use them
* tests/qemuxml2argvdata/qemuxml2argv-graphics-spice-rhel6.args,
  tests/qemuxml2argvtest.c: Expand test case to cover driver
  level configuration

* src/qemu/qemu_driver.c: Allocate the TCP ports for SPICE
  before starting guest

This supports the -spice argument posted for review against
the latest upstream QEMU/KVM. This supports the bare minimum
config with port, TLS port & listen address. The x509 bits are
added in a later patch.

* src/qemu_conf.c, src/qemu_conf.h: Add SPICE flag. Check for
  -spice availability. Format -spice arg for command line
* qemuhelptest.c: Add SPICE flag
* qemuxml2argvdata/qemuxml2argv-graphics-spice.args: Add <graphics>
  for spice
* qemuxml2argvdata/qemuxml2argv-graphics-spice.xml: Add -spice arg
* qemuxml2argvtest.c: Add SPICE flag

This supports the '-vga qxl' parameter in upstream QEMU/KVM
which has SPICE support added. This isn't particularly useful
until you get the next patch for -spice support. Also note that
while the libvirt XML supports multiple video devices, this
patch only supports a single one. A later patch can add support
for 2nd, 3rd, etc PCI devices for QXL

* src/qemu/qemu_conf.h: Flag for QXL support
* src/qemu/qemu_conf.c: Probe for '-vga qxl' support and implement it
* tests/qemuxml2argvtest.c, tests/qemuxml2xmltest.c,
  tests/qemuxml2argvdata/qemuxml2argv-graphics-spice.args,
  tests/qemuxml2argvdata/qemuxml2argv-graphics-spice.xml: Test
  case for generating spice args with RHEL6 kvm

This adds an element

 <graphics type='spice' port='5903' tlsPort='5904' autoport='yes' listen='127.0.0.1'/>

This is the bare minimum that should be exposed in the guest
config for SPICE. Other parameters are better handled as per
host level configuration tunables

* docs/schemas/domain.rng: Define the SPICE <graphics> schema
* src/domain_conf.h, src/domain_conf.c: Add parsing and formatting
  for SPICE graphics config
* src/qemu_conf.c: Complain about unsupported graphics types

* src/qemu_conf.c: Add dummy entry in enumeration
* docs/schemas/domain.rng: Add 'qxl' as a type for the <video> tag
* src/domain_conf.c, src/domain_conf.h: Add QXL to video type
  enumerations

Suggested by danpb, as it's not up-to-date anymore and
lacks many functions that were added to libvirtd.

There is no point in trying to fill params beyond the first error,
because when lxcDomainGetMemoryParameters returns -1 then the caller
cannot detect which values in params are valid.

Also fix a typo in the documentation of the function.

The patch is based on the possiblity in the QEmu command line to
add -smbios options allowing to override the default values picked
by QEmu. We need to detect this first from QEmu help output.
If the domain is defined with smbios to be inherited from host
then we pass the values coming from the Host own SMBIOS, but
if the domain is defined with smbios to come from sysinfo, we
use the ones coming from the domain definition.

* src/qemu/qemu_conf.h: add the QEMUD_CMD_FLAG_SMBIOS_TYPE enum
  value
* src/qemu/qemu_conf.c: scan the help output for the smbios support,
  and if available add support based on the domain definitions,
  and host data
* tests/qemuhelptest.c: add the new enum in the outputs

Read and store the data when initializing the driver.

Move existing routines about virSysinfoDef to an util module,
add a new entry point virSysinfoRead() to read the host values
with dmidecode

* src/conf/domain_conf.c src/conf/domain_conf.h src/util/sysinfo.c
  src/util/sysinfo.h: move to a new module, add virSysinfoRead()
* src/Makefile.am: handle the new module build
* src/libvirt_private.syms: new internal symbols
* include/libvirt/virterror.h src/util/virterror.c: defined a new
  error code for that module
* po/POTFILES.in: add new file for translations

the element has a mode attribute allowing only 3 values:
  - emulate: use the smbios emulation from the hypervisor
  - host: try to use the smbios values from the node
  - sysinfo: grab the values from the <sysinfo> fields

* docs/schemas/domain.rng: extend the schemas
* src/conf/domain_conf.h: add the flag to the domain config
* src/conf/domain_conf.h: parse and serialize the smbios if present

* src/conf/domain_conf.h: defines a new internal type added to the
  domain structure
* src/conf/domain_conf.c: parsing and serialization of that new type

During a shutdown/restart cycle libvirtd forgot the macvtap device name that it had created on behalf of a VM so that a stale macvtap device remained on the host when the VM terminated. Libvirtd has to actively tear down a macvtap device and it uses its name for identifying which device to tear down.

The solution is to not blank out the <target dev='...'/> completely, but only blank it out on VMs that are not active. So, if a VM is active, the device name makes it into the XML and is also being parsed. If a VM is not active, the device name is discarded.

virPipeReadUntilEOF is used to read the stdout of exec'ed
and this could fail to capture the full output and read only
1024 bytes.
  The problem is that this is based on a poll loop, and in the
loop we read at most 1024 bytes per file descriptor, but we also
note in the loop if poll indicates that the process won't output
more than that on that fd by setting finished[i] = 1.
  The simplest way is that if we read a full buffer make sure
finished[i] is still 0 because we will need another pass in the
loop.