I am trying to use a qcow image with libvirt where the backing 'file' is a
qemu-nbd server.  Unfortunately virDomainDiskDefForeachPath() assumes that
backingStore is always a real file so something like 'nbd:0:3333' is rejected
because a file with that name cannot be accessed.  Note that I am not worried
about directly using nbd images.  That would require a new disk type with XML
markup, etc.  I only want it to be permitted as a backingStore

The following patch implements danpb's suggestion:
> I think I'm inclined to push the logic for skipping NBD one stage higher.
> I'd rather expect virStorageFileGetMetadata() to return all backing
> stores, even if not files. The virDomainDiskDefForeachPath() method
> should definitely ignore non-file backing stores though.
>
> So what I'm thinking is to extend the virStorageFileMetadata struct and
> just add a 'bool isFile' field to it. Default this field to true, unless
> you see the prefix of nbd: in which case set it to false. The
> virDomainDiskDefForeachPath() method can then skip over any backing
> store with isFile == false
Signed-off-by: NAdam Litke <agl@us.ibm.com>
Cc: Daniel P. Berrange <berrange@redhat.com>

This extends the SPICE XML to allow channel security options

    <graphics type='spice' port='-1' tlsPort='-1' autoport='yes'>
      <channel name='main' mode='secure'/>
      <channel name='record' mode='insecure'/>
    </graphics>

Any non-specified channel uses the default, which allows both
secure & insecure usage

* src/conf/domain_conf.c, src/conf/domain_conf.h,
  src/libvirt_private.syms: Add XML syntax for specifying per
  channel security options for spice.
* src/qemu/qemu_conf.c: Configure channel security with spice

This extends the XML syntax for <graphics> to allow a password
expiry time to be set

eg

  <graphics type='vnc' port='5900' autoport='yes' keymap='en-us' passwd='12345' passwdValidTo='2010-04-09T15:51:00'/>

The timestamp is in UTC.

* src/conf/domain_conf.h: Pull passwd out into separate struct
  virDomainGraphicsAuthDef to allow sharing between VNC & SPICE
* src/conf/domain_conf.c: Add parsing/formatting of new passwdValidTo
  argument
* src/opennebula/one_conf.c, src/qemu/qemu_conf.c, src/qemu/qemu_driver.c,
  src/xen/xend_internal.c, src/xen/xm_internal.c: Update for changed
  struct containing VNC password

This adds an element

 <graphics type='spice' port='5903' tlsPort='5904' autoport='yes' listen='127.0.0.1'/>

This is the bare minimum that should be exposed in the guest
config for SPICE. Other parameters are better handled as per
host level configuration tunables

* docs/schemas/domain.rng: Define the SPICE <graphics> schema
* src/domain_conf.h, src/domain_conf.c: Add parsing and formatting
  for SPICE graphics config
* src/qemu_conf.c: Complain about unsupported graphics types

* src/qemu_conf.c: Add dummy entry in enumeration
* docs/schemas/domain.rng: Add 'qxl' as a type for the <video> tag
* src/domain_conf.c, src/domain_conf.h: Add QXL to video type
  enumerations

Suggested by danpb, as it's not up-to-date anymore and
lacks many functions that were added to libvirtd.

Move existing routines about virSysinfoDef to an util module,
add a new entry point virSysinfoRead() to read the host values
with dmidecode

* src/conf/domain_conf.c src/conf/domain_conf.h src/util/sysinfo.c
  src/util/sysinfo.h: move to a new module, add virSysinfoRead()
* src/Makefile.am: handle the new module build
* src/libvirt_private.syms: new internal symbols
* include/libvirt/virterror.h src/util/virterror.c: defined a new
  error code for that module
* po/POTFILES.in: add new file for translations

the element has a mode attribute allowing only 3 values:
  - emulate: use the smbios emulation from the hypervisor
  - host: try to use the smbios values from the node
  - sysinfo: grab the values from the <sysinfo> fields

* docs/schemas/domain.rng: extend the schemas
* src/conf/domain_conf.h: add the flag to the domain config
* src/conf/domain_conf.h: parse and serialize the smbios if present

* src/conf/domain_conf.h: defines a new internal type added to the
  domain structure
* src/conf/domain_conf.c: parsing and serialization of that new type

During a shutdown/restart cycle libvirtd forgot the macvtap device name that it had created on behalf of a VM so that a stale macvtap device remained on the host when the VM terminated. Libvirtd has to actively tear down a macvtap device and it uses its name for identifying which device to tear down.

The solution is to not blank out the <target dev='...'/> completely, but only blank it out on VMs that are not active. So, if a VM is active, the device name makes it into the XML and is also being parsed. If a VM is not active, the device name is discarded.

The network address was being set to 192.168.122.0 instead
of 192.168.122.0/24. Fix this by removing the unneccessary
'network' field from virNetworkDef and just pass the
network address and netmask into the iptables APIs directly.

* src/conf/network_conf.h, src/conf/network_conf.c: Remove
  the 'network' field from virNEtworkDef.
* src/network/bridge_driver.c: Update for iptables API changes
* src/util/iptables.c, src/util/iptables.h: Require the
  network address + netmask pair to be passed in

The boot server IP address is optional, so it needs to be
checked before attempting to parse it.

* src/conf/network_conf.c: Don't parse NULL ip address for
  boot server

Instead of storing the IP address string in virNetwork related
structs, store the parsed virSocketAddr. This will make it
easier to add IPv6 support in the future, by letting driver
code directly check what address family is present

* src/conf/network_conf.c, src/conf/network_conf.h,
  src/network/bridge_driver.c: Convert to use virSocketAddr
  in virNetwork, instead of char *.
* src/util/bridge.c, src/util/bridge.h,
  src/util/dnsmasq.c, src/util/dnsmasq.h,
  src/util/iptables.c, src/util/iptables.h: Convert to
  take a virSocketAddr instead of char * for any IP
  address parameters
* src/util/network.h: Add macros to determine if an address
  is set, and what address family is set.

The  inet_pton and inet_ntop functions are obsolete, replaced
by getaddrinfo+getnameinfo with the AI_NUMERICHOST flag set.
These can be accessed via the virSocket APIs.

The bridge.c code had methods for fetching the IP address of
a bridge which used inet_ntop. Aside from the use of inet_ntop
these methods are broken, because a NIC can have multiple
addresses and this only returns one address. Since the methods
are never used, just remove them.

* src/conf/network_conf.c, src/nwfilter/nwfilter_learnipaddr.c:
  Replace inet_pton and inet_ntop with virSocket APIs
* src/util/bridge.c, src/util/bridge.h: Remove unused methods
  which called inet_ntop.

The virSocketParse method was not doing any error reporting
which meant the true cause of the problem was lost. Remove
all error reporting from callers, and push it into virSocketParse

* src/util/network.c: Add error reporting to virSocketParse
* src/conf/domain_conf.c, src/conf/network_conf.c,
  src/network/bridge_driver.c: Remove error reporting in
  callers of virSocketParse

The getnameinfo() function is more flexible than inet_ntop()
avoiding the need to if/else the code based on socket family.
Also make it support UNIX socket addrs and allow inclusion
of a port (service) address. Finally do proper error reporting
via normal APIs.

* src/conf/domain_conf.c, src/nwfilter/nwfilter_ebiptables_driver.c,
  src/qemu/qemu_conf.c: Fix error handling with virSocketFormat
* src/util/network.c: Rewrite virSocketFormat to use getnameinfo
  and cope with UNIX socket addrs.

The nwIPAddress was simply a wrapper about virSocketAddr.
Just use the latter directly, removing all the extra field
de-references from code & helper APIs for parsing/formatting.

Also remove all the redundant casts from strong types to
void * and then immediately back to strong types.

* src/conf/nwfilter_conf.h: Remove nwIPAddress
* src/conf/nwfilter_conf.c, src/nwfilter/nwfilter_ebiptables_driver.c:
  Update to use virSocketAddr and remove void * casts.

The virSocketParseAddr function was accepting any AF_* constant
and using that to set the ai_flags field in struct addrinfo.
This is invalid, since address families must go in the ai_family
field of the struct.

* src/util/network.c: Fix handling of address family
* src/conf/network_conf.c, src/network/bridge_driver.c: Pass
  AF_UNSPEC instead of relying on it being 0.

Some operations on socket addresses need to know the length of
the sockaddr struct for the particular address family. This
info was being discarded when passing around virSocketAddr
instances. Turn it from a union into a struct containing
union+socklen_t fields, so length is always kept around.

* src/util/network.h: Add socklen_t field to virSocketAddr
* src/util/network.c, src/network/bridge_driver.c,
  src/conf/domain_conf.c: Update to take account of new
  struct definition.

Also output the min_guarantee element when set.

Although this patch adds a distinction between maximum vcpus and
current vcpus in the XML, the values should be identical for all
drivers at this point.  Only in subsequent per-driver patches will
a distinction be made.

In general, virDomainGetInfo should prefer the current vcpus.

* src/conf/domain_conf.h (_virDomainDef): Adjust vcpus to unsigned
short, to match virDomainGetInfo limit.  Add maxvcpus member.
* src/conf/domain_conf.c (virDomainDefParseXML)
(virDomainDefFormat): parse and print out vcpu details.
* src/xen/xend_internal.c (xenDaemonParseSxpr)
(xenDaemonFormatSxpr): Manage both vcpu numbers, and require them
to be equal for now.
* src/xen/xm_internal.c (xenXMDomainConfigParse)
(xenXMDomainConfigFormat): Likewise.
* src/phyp/phyp_driver.c (phypDomainDumpXML): Likewise.
* src/openvz/openvz_conf.c (openvzLoadDomains): Likewise.
* src/openvz/openvz_driver.c (openvzDomainDefineXML)
(openvzDomainCreateXML, openvzDomainSetVcpusInternal): Likewise.
* src/vbox/vbox_tmpl.c (vboxDomainDumpXML, vboxDomainDefineXML):
Likewise.
* src/xenapi/xenapi_driver.c (xenapiDomainDumpXML): Likewise.
* src/xenapi/xenapi_utils.c (createVMRecordFromXml): Likewise.
* src/esx/esx_vmx.c (esxVMX_ParseConfig, esxVMX_FormatConfig):
Likewise.
* src/qemu/qemu_conf.c (qemuBuildSmpArgStr)
(qemuParseCommandLineSmp, qemuParseCommandLine): Likewise.
* src/qemu/qemu_driver.c (qemudDomainHotplugVcpus): Likewise.
* src/opennebula/one_conf.c (xmlOneTemplate): Likewise.

Since bugs due to double-closed file descriptors are difficult to track down in a multi-threaded system, I am introducing the VIR_CLOSE(fd) macro to help avoid mistakes here.

There are lots of places where close() is being used. In this patch I am only cleaning up usage of close() in src/conf where the problems were.

I also dare to declare close() as being deprecated in libvirt code base (HACKING).

Patch to prevent multiple nwfilters with different name but same UUID.

This introduces new attribute to filesystem element
to support customizable access mode for mount type.
Valid accessmode are: passthrough, mapped and squash.

Usage:
        <filesystem type='mount' accessmode='passthrough'>
          <source dir='/export/to/guest'/>
          <target dir='mount_tag'/>
        </filesystem>

passthrough is the default model if not specified, that's
also the current behaviour.

 This is from a bug report and conversation on IRC where Soren reported that while a filter update is occurring on one or more VMs (due to a rule having been edited for example), a deadlock can occur when a VM referencing a filter is started.

The problem is caused by the two locking sequences of

qemu driver, qemu domain, filter             # for the VM start operation
filter, qemu_driver, qemu_domain            # for the filter update operation

that obviously don't lock in the same order. The problem is the 2nd lock sequence. Here the qemu_driver lock is being grabbed in qemu_driver:qemudVMFilterRebuild()

The following solution is based on the idea of trying to re-arrange the 2nd sequence of locks as follows:

qemu_driver, filter, qemu_driver, qemu_domain

and making the qemu driver recursively lockable so that a second lock can occur, this would then lead to the following net-locking sequence

qemu_driver, filter, qemu_domain

where the 2nd qemu_driver lock has been ( logically ) eliminated.

The 2nd part of the idea is that the sequence of locks (filter, qemu_domain) and (qemu_domain, filter) becomes interchangeable if all code paths where filter AND qemu_domain are locked have a preceding qemu_domain lock that basically blocks their concurrent execution

So, the following code paths exist towards qemu_driver:qemudVMFilterRebuild where we now want to put a qemu_driver lock in front of the filter lock.

-> nwfilterUndefine()   [ locks the filter ]
    -> virNWFilterTestUnassignDef()
        -> virNWFilterTriggerVMFilterRebuild()
            -> qemudVMFilterRebuild()

-> nwfilterDefine()
    -> virNWFilterPoolAssignDef() [ locks the filter ]
        -> virNWFilterTriggerVMFilterRebuild()
            -> qemudVMFilterRebuild()

-> nwfilterDriverReload()
    -> virNWFilterPoolLoadAllConfigs()
        ->virNWFilterPoolObjLoad()
            -> virNWFilterPoolAssignDef() [ locks the filter ]
                -> virNWFilterTriggerVMFilterRebuild()
                    -> qemudVMFilterRebuild()

-> nwfilterDriverStartup()
    -> virNWFilterPoolLoadAllConfigs()
        ->virNWFilterPoolObjLoad()
            -> virNWFilterPoolAssignDef() [ locks the filter ]
                -> virNWFilterTriggerVMFilterRebuild()
                    -> qemudVMFilterRebuild()

Qemu is not the only driver using the nwfilter driver, but also the UML driver calls into it. Therefore qemuVMFilterRebuild() can be exchanged with umlVMFilterRebuild() along with the driver lock of qemu_driver that can now be a uml_driver. Further, since UML and Qemu domains can be running on the same machine, the triggering of a rebuild of the filter can touch both types of drivers and their domains.

In the patch below I am now extending each nwfilter callback driver with functions for locking and unlocking the (VM) driver (UML, QEMU) and introduce new functions for locking all registered callback drivers and unlocking them. Then I am distributing the lock-all-cbdrivers/unlock-all-cbdrivers call into the above call paths. The last shown callpath starting with nwfilterDriverStart() is problematic since it is initialize before the Qemu and UML drives are and thus a lock in the path would result in a NULL pointer attempted to be locked -- the call to virNWFilterTriggerVMFilterRebuild() is never called, so we never lock either the qemu_driver or the uml_driver in that path. Therefore, only the first 3 paths now receive calls to lock and unlock all callback drivers. Now that the locks are distributed where it matters I can remove the qemu_driver and uml_driver lock from qemudVMFilterRebuild() and umlVMFilterRebuild() and not requiring the recursive locks.

For now I want to put this out as an RFC patch. I have tested it by 'stretching' the critical section after the define/undefine functions each lock the filter so I can (easily) concurrently execute another VM operation (suspend,start). That code is in this patch and if you want you can de-activate it. It seems to work ok and operations are being blocked while the update is being done.
I still also want to verify the other assumption above that locking filter and qemu_domain always has a preceding qemu_driver lock.

Adding parsing code for memory tunables in the domain xml file
also change the internal define structures used for domain memory
informations
Adds a new specific test

To enable the CPU XML from the capabilities to be pasted directly
into the guest XML with no editing, pick a sensible default for
match and feature policy. The CPU match will be exact and the
feature policy will be require. This should ensure safety for
migration and give DWIM semantics for users

* src/conf/cpu_conf.c: Default to exact match and require policy
* docs/formatdomain.html.in: Document new defaults

The patch below extends the XML parser and generator so that every l3 protocol
now can have a state attribute.

Since version 4.1 ESX(i) can expose virtual serial devices over TCP.

Add support in the VMX handling code for this, add test cases to cover
it and add links to some documentation.

ESX supports two additional protocols: TELNETS and TLS. Add them to
the list of serial-over-TCP protocols.

The <vcpu cpuset=...> attribute has been available since commit
e193b5dd, but without documentation or RNG validation.

* docs/schemas/domain.rng (vcpu): Further validate cpuset.
* docs/formatdomain.html.in: Document it.
* src/conf/domain_conf.c: Fix typos.

The patch below extends the XML parser and generator so that every protocol
now can have a comment node. Comments are limited to 256 characters.

This patch fixes a bug appearing on big endian machines where the returned XML is not the one that is expected (see test/nwfilterxml2xmltest). The problem is due to for example the casting of pointers to unsigned integers to void * and then back to 16 bit integers.

* src/conf/network_conf.c (virNetworkAllocateBridge): Avoid
limited buffer from snprintf.

Other drivers will need this same functionality, so move it to up to
conf/domain_conf.c and give it a more general name.
Signed-off-by: NSoren Hansen <soren@linux2go.dk>

Xen supports on_crash actions coredump-{destroy,restart}.  libvirt
cannot parse config returned by xend that contains either of these
actions

xen52 # xm li -l test | grep on_crash
    (on_crash coredump-restart)
xen52 # virsh dumpxml test
error: internal error unknown lifecycle type coredump-restart

This patch adds a new virDomainLifecycleCrash enum and appends
the new options to existing destroy, restart, preserve, and
rename-restart options.

I wrote a patch to add support for listing the Vendor and Model of a
storage pool in the storage pool XML.  This would allow vendor
extensions of specific devices.  The patch includes a test for the new
attributes as well.

Patrick Dignan

This patch fixes a couple of complaints from valgrind when tickling libvirtd with SIGHUP.

The first two files contain fixes for memory leaks. The 3rd one initializes an uninitialized variable. The 4th one is another memory leak.

  The balloon device is automatically added to qemu guests if supported,
but it may be useful to desactivate it. The simplest to not change the
existing behaviour is to allow
  <memballoon type="none"/>
as an extra option to desactivate it (it is automatically added if the
memballoon construct is missing for the domain).
The following simple patch just adds the extra option and does not
change the default behaviour but avoid creating a balloon device if
type="none" is used.

* docs/schemas/domain.rng: add the extra type attribute value
* src/conf/domain_conf.c src/conf/domain_conf.h: add the extra enum
  value
* src/qemu/qemu_conf.c: if enum is NONE, don't activate the device,
  i.e. don't pass the args to qemu/kvm