The QEMU 0.13 release is finally out and from testing in RHEL-6
we know that its JSON and netdev features are now good enough
for us to use by default.

* src/qemu/qemu_conf.c: Enable JSON + netdev for QEMU >= 0.13

There is no point in trying to fill params beyond the first error,
because when qemuDomainGetMemoryParameters returns -1 then the caller
cannot detect which values in params are valid.

To get them under the common VIR_DOMAIN_MEMORY_* prefix.

This sets the process name to the same value as the Windows title,
but since the name is limited to 16 chars only this is kept as a
configuration option and turned off by default
* src/qemu/qemu.conf src/qemu/qemu_conf.[ch]: hceck for support in the
  QEmu help output, add the option in qemu conf file and augment
  qemudBuildCommandLine to add it if switched on
* src/qemu/libvirtd_qemu.aug src/qemu/test_libvirtd_qemu.aug: augment
  the augeas lenses accordingly
* tests/qemuhelptest.c: cope with the extra flag being detected now

Most operations are audited at the libvirtd level; auditing in
src/libvirt.c would result in two audit entries per operation (one in
the client, one in libvirtd).

The only exception is a domain stopping of its own will (e.g. because
the user clicks on "shutdown" inside the interface).  There can often be
no client connected at the time the domain stops, so libvirtd does not
have any virConnectPtr object on which to attach an event watch.  This
patch therefore adds auditing directly inside the qemu driver (other
drivers are not supported).

* src/qemu/qemu_driver.c (qemudDomainSetVcpusFlags)
(qemudDomainGetVcpusFlags): Support all feasible flag
combinations.

* src/qemu/qemu_conf.c (qemuParseCommandLineSmp): Distinguish
between vcpus and maxvcpus, for new enough qemu.
* tests/qemuargv2xmltest.c (mymain): Add new test.
* tests/qemuxml2argvtest.c (mymain): Likewise.
* tests/qemuxml2xmltest.c (mymain): Likewise.
* tests/qemuxml2argvdata/qemuxml2argv-smp.args: New file.

Although this patch adds a distinction between maximum vcpus and
current vcpus in the XML, the values should be identical for all
drivers at this point.  Only in subsequent per-driver patches will
a distinction be made.

In general, virDomainGetInfo should prefer the current vcpus.

* src/conf/domain_conf.h (_virDomainDef): Adjust vcpus to unsigned
short, to match virDomainGetInfo limit.  Add maxvcpus member.
* src/conf/domain_conf.c (virDomainDefParseXML)
(virDomainDefFormat): parse and print out vcpu details.
* src/xen/xend_internal.c (xenDaemonParseSxpr)
(xenDaemonFormatSxpr): Manage both vcpu numbers, and require them
to be equal for now.
* src/xen/xm_internal.c (xenXMDomainConfigParse)
(xenXMDomainConfigFormat): Likewise.
* src/phyp/phyp_driver.c (phypDomainDumpXML): Likewise.
* src/openvz/openvz_conf.c (openvzLoadDomains): Likewise.
* src/openvz/openvz_driver.c (openvzDomainDefineXML)
(openvzDomainCreateXML, openvzDomainSetVcpusInternal): Likewise.
* src/vbox/vbox_tmpl.c (vboxDomainDumpXML, vboxDomainDefineXML):
Likewise.
* src/xenapi/xenapi_driver.c (xenapiDomainDumpXML): Likewise.
* src/xenapi/xenapi_utils.c (createVMRecordFromXml): Likewise.
* src/esx/esx_vmx.c (esxVMX_ParseConfig, esxVMX_FormatConfig):
Likewise.
* src/qemu/qemu_conf.c (qemuBuildSmpArgStr)
(qemuParseCommandLineSmp, qemuParseCommandLine): Likewise.
* src/qemu/qemu_driver.c (qemudDomainHotplugVcpus): Likewise.
* src/opennebula/one_conf.c (xmlOneTemplate): Likewise.

Note - this wrapping is completely mechanical; the old API will
function identically, since the new API validates that the exact
same flags are provided by the old API.  On a per-driver basis,
it may make sense to have the old API pass a different set of flags,
but that should be done in the per-driver patch that implements
the full range of flag support in the new API.

* src/esx/esx_driver.c (esxDomainSetVcpus, escDomainGetMaxVpcus):
Move guts...
(esxDomainSetVcpusFlags, esxDomainGetVcpusFlags): ...to new
functions.
(esxDriver): Trivially support the new API.
* src/openvz/openvz_driver.c (openvzDomainSetVcpus)
(openvzDomainSetVcpusFlags, openvzDomainGetMaxVcpus)
(openvzDomainGetVcpusFlags, openvzDriver): Likewise.
* src/phyp/phyp_driver.c (phypDomainSetCPU)
(phypDomainSetVcpusFlags, phypGetLparCPUMAX)
(phypDomainGetVcpusFlags, phypDriver): Likewise.
* src/qemu/qemu_driver.c (qemudDomainSetVcpus)
(qemudDomainSetVcpusFlags, qemudDomainGetMaxVcpus)
(qemudDomainGetVcpusFlags, qemuDriver): Likewise.
* src/test/test_driver.c (testSetVcpus, testDomainSetVcpusFlags)
(testDomainGetMaxVcpus, testDomainGetVcpusFlags, testDriver):
Likewise.
* src/vbox/vbox_tmpl.c (vboxDomainSetVcpus)
(vboxDomainSetVcpusFlags, virDomainGetMaxVcpus)
(virDomainGetVcpusFlags, virDriver): Likewise.
* src/xen/xen_driver.c (xenUnifiedDomainSetVcpus)
(xenUnifiedDomainSetVcpusFlags, xenUnifiedDomainGetMaxVcpus)
(xenUnifiedDomainGetVcpusFlags, xenUnifiedDriver): Likewise.
* src/xenapi/xenapi_driver.c (xenapiDomainSetVcpus)
(xenapiDomainSetVcpusFlags, xenapiDomainGetMaxVcpus)
(xenapiDomainGetVcpusFlags, xenapiDriver): Likewise.
(xenapiError): New helper macro.

* src/driver.h (virDrvDomainSetVcpusFlags)
(virDrvDomainGetVcpusFlags): New typedefs.
(_virDriver): New callback members.
* src/esx/esx_driver.c (esxDriver): Add stub for driver.
* src/lxc/lxc_driver.c (lxcDriver): Likewise.
* src/opennebula/one_driver.c (oneDriver): Likewise.
* src/openvz/openvz_driver.c (openvzDriver): Likewise.
* src/phyp/phyp_driver.c (phypDriver): Likewise.
* src/qemu/qemu_driver.c (qemuDriver): Likewise.
* src/remote/remote_driver.c (remote_driver): Likewise.
* src/test/test_driver.c (testDriver): Likewise.
* src/uml/uml_driver.c (umlDriver): Likewise.
* src/vbox/vbox_tmpl.c (Driver): Likewise.
* src/xen/xen_driver.c (xenUnifiedDriver): Likewise.
* src/xenapi/xenapi_driver.c (xenapiDriver): Likewise.

Over root-squashing nfs, when virFileOperation() is called as uid==0,
it may fail with EACCES, but also with EPERM, due to
virFileOperationNoFork()'s failed attemp to chown a writable file.

qemudDomainSaveFlag() should expect this case, too.

qemudOpenAsUID is intended to open a file with the credentials of a
specified uid. Current implementation fails if the file is accessible to
one of uid's groups but not owned by uid.

This patch replaces the supplementary group list that the child process
inherited from libvirtd with the default group list of uid.

This introduces new attribute to filesystem element
to support customizable access mode for mount type.
Valid accessmode are: passthrough, mapped and squash.

Usage:
        <filesystem type='mount' accessmode='passthrough'>
          <source dir='/export/to/guest'/>
          <target dir='mount_tag'/>
        </filesystem>

passthrough is the default model if not specified, that's
also the current behaviour.

Explicitly raising a nice error in the case user tries to migrate a
guest with assigned host devices is much better than waiting for a
mysterious error with no clue for the reason.

This enables support for nested SVM using the regular CPU
model/features block. If the CPU model or features include
'svm', then the '-enable-nesting' flag will be added to the
QEMU command line. Latest out of tree patches for nested
'vmx', no longer require the '-enable-nesting' flag. They
instead just look at the cpu features. Several of the models
already include svm support, but QEMU was just masking out
the svm bit silently. So this will enable SVM on such
models

* src/qemu/qemu_conf.h: flag for -enable-nesting
* src/qemu/qemu_conf.c: Use -enable-nesting if VMX or SVM are in
  the CPUID
* src/cpu/cpu.h, src/cpu/cpu.c: API to check for a named feature
* src/cpu/cpu_x86.c: x86 impl of feature check
* src/libvirt_private.syms: Add cpuHasFeature
* src/qemuhelptest.c: Add nesting flag where required

 This is from a bug report and conversation on IRC where Soren reported that while a filter update is occurring on one or more VMs (due to a rule having been edited for example), a deadlock can occur when a VM referencing a filter is started.

The problem is caused by the two locking sequences of

qemu driver, qemu domain, filter             # for the VM start operation
filter, qemu_driver, qemu_domain            # for the filter update operation

that obviously don't lock in the same order. The problem is the 2nd lock sequence. Here the qemu_driver lock is being grabbed in qemu_driver:qemudVMFilterRebuild()

The following solution is based on the idea of trying to re-arrange the 2nd sequence of locks as follows:

qemu_driver, filter, qemu_driver, qemu_domain

and making the qemu driver recursively lockable so that a second lock can occur, this would then lead to the following net-locking sequence

qemu_driver, filter, qemu_domain

where the 2nd qemu_driver lock has been ( logically ) eliminated.

The 2nd part of the idea is that the sequence of locks (filter, qemu_domain) and (qemu_domain, filter) becomes interchangeable if all code paths where filter AND qemu_domain are locked have a preceding qemu_domain lock that basically blocks their concurrent execution

So, the following code paths exist towards qemu_driver:qemudVMFilterRebuild where we now want to put a qemu_driver lock in front of the filter lock.

-> nwfilterUndefine()   [ locks the filter ]
    -> virNWFilterTestUnassignDef()
        -> virNWFilterTriggerVMFilterRebuild()
            -> qemudVMFilterRebuild()

-> nwfilterDefine()
    -> virNWFilterPoolAssignDef() [ locks the filter ]
        -> virNWFilterTriggerVMFilterRebuild()
            -> qemudVMFilterRebuild()

-> nwfilterDriverReload()
    -> virNWFilterPoolLoadAllConfigs()
        ->virNWFilterPoolObjLoad()
            -> virNWFilterPoolAssignDef() [ locks the filter ]
                -> virNWFilterTriggerVMFilterRebuild()
                    -> qemudVMFilterRebuild()

-> nwfilterDriverStartup()
    -> virNWFilterPoolLoadAllConfigs()
        ->virNWFilterPoolObjLoad()
            -> virNWFilterPoolAssignDef() [ locks the filter ]
                -> virNWFilterTriggerVMFilterRebuild()
                    -> qemudVMFilterRebuild()

Qemu is not the only driver using the nwfilter driver, but also the UML driver calls into it. Therefore qemuVMFilterRebuild() can be exchanged with umlVMFilterRebuild() along with the driver lock of qemu_driver that can now be a uml_driver. Further, since UML and Qemu domains can be running on the same machine, the triggering of a rebuild of the filter can touch both types of drivers and their domains.

In the patch below I am now extending each nwfilter callback driver with functions for locking and unlocking the (VM) driver (UML, QEMU) and introduce new functions for locking all registered callback drivers and unlocking them. Then I am distributing the lock-all-cbdrivers/unlock-all-cbdrivers call into the above call paths. The last shown callpath starting with nwfilterDriverStart() is problematic since it is initialize before the Qemu and UML drives are and thus a lock in the path would result in a NULL pointer attempted to be locked -- the call to virNWFilterTriggerVMFilterRebuild() is never called, so we never lock either the qemu_driver or the uml_driver in that path. Therefore, only the first 3 paths now receive calls to lock and unlock all callback drivers. Now that the locks are distributed where it matters I can remove the qemu_driver and uml_driver lock from qemudVMFilterRebuild() and umlVMFilterRebuild() and not requiring the recursive locks.

For now I want to put this out as an RFC patch. I have tested it by 'stretching' the critical section after the define/undefine functions each lock the filter so I can (easily) concurrently execute another VM operation (suspend,start). That code is in this patch and if you want you can de-activate it. It seems to work ok and operations are being blocked while the update is being done.
I still also want to verify the other assumption above that locking filter and qemu_domain always has a preceding qemu_driver lock.

Make use of the existing <filesystem> element to support plan9fs
filesystem passthrough in the QEMU driver

    <filesystem type='mount'>
      <source dir='/export/to/guest'/>
      <target dir='/import/from/host'/>
    </filesystem>

NB, the target is not actually a directory, it is merely a arbitrary
string tag that is exported to the guest as a hint for where to mount
it.

QEmu startup will pick up the memory tunables specified in the domain
configuration file

Driver interface for getting memory parameters, eg. hard_limit,
soft_limit and swap_hard_limit based on cgroup support

Driver interface for setting memory hard_limit, soft_limit and swap
hard_limit based on cgroup support

Adding parsing code for memory tunables in the domain xml file
also change the internal define structures used for domain memory
informations
Adds a new specific test

Public api to set/get memory tunables supported by the hypervisors.

dv:
* some cleanups in libvirt.c
* adding extra checks in libvirt.c new entry points

v4:
* Move exporting public API to this patch
* Add unsigned int flags to the public api for future extensions

v3:
* Add domainGetMemoryParamters and NULL in all the driver interface

v2:
* Initialize domainSetMemoryParameters to NULL in all the driver
  interface structure.

A QEMU guest can have upto VIR_DOMAIN_BOOT_LAST boot entries
defined. When building the QEMU arg, each entry takes a
single byte. This means the array must be declared to be
VIR_DOMAIN_BOOT_LAST+1 bytes in length to allow for the
trailing null

* src/qemu/qemu_conf.c: Fix off-by-1 boot arg array size

QMP in QEMU 0.13 has been fixed to enforce type correctness,
this means that boolean types must be true or false, not
integers.
Signed-off-by: NLuiz Capitulino <lcapitulino@redhat.com>

QMP in QEMU 0.13 has been fixed to enforce type correctness,
this means that boolean types must be true or false, not
integers.
Signed-off-by: NLuiz Capitulino <lcapitulino@redhat.com>

Other drivers will need this same functionality, so move it to up to
conf/domain_conf.c and give it a more general name.
Signed-off-by: NSoren Hansen <soren@linux2go.dk>

Previously QEMU enabled KQEMU by default and had -no-kqemu.
0.11.x switched to requiring -enable-kqemu. 0.12.x dropped
kqemu entirely. This patch adds support for -enable-kqemu
so 0.11.x works. It replaces a huge set of if() with a
switch() to make the code a bit more readable.

* src/qemu/qemu_conf.c, src/qemu/qemu_conf.h: Support
  -enable-kqemu

We already filled the PCI address structure when we checked whether it's
free or not, so let's just use the structure here instead of filling it
again.

* src/qemu/qemu_driver.c (qemudGetProcessInfo): Clean up.
* src/uml/uml_driver.c (umlGetProcessInfo): Likewise.
* src/xen/sexpr.c (_string2sexpr): Likewise.

The current version of the qemu managed save implementation
is subject to a race where the domain shuts down between
the time that we start the command and the time that we
actually try to do the save.  Close this race by making
qemuDomainSaveFlags() expect both the driver and the passed-in
vm object to be locked before executing.
Signed-off-by: NChris Lalancette <clalance@redhat.com>

This also makes the message consistent with the message used in error
path of qemudDomainAttachHostPciDevice.

When reconnecting to existing VMs, we re-reserved only those PCI
addresses which were explicitly mentioned in domain XML. Since some
addresses are always reserved (e.g., 0:0:0 and 0:0:1), we need to handle
those too.

Also all this should only be done if device flag is supported by qemu.

In this patch I am extending and fixing the nwfilter module's reload support to stop all ongoing threads (for learning IP addresses of interfaces) and rebuild the filtering rules of all interfaces of all VMs when libvirt is started. Now libvirtd rebuilds the filters upon the SIGHUP signal and libvirtd restart.

About the patch: The nwfilter functions require a virConnectPtr. Therefore I am opening a connection in qemudStartup, which later on needs to be closed outside where the driver lock is held since otherwise it ends up in a deadlock due to virConnectClose() trying to lock the driver as well.

I have tested this now for a while with several machines running and needing the IP address learner thread(s). The rebuilding of the firewall rules seems to work fine following libvirtd restart or a SIGHUP. Also the termination of libvirtd worked fine.

Since the qemu process is running as qemu:qemu, it can't actually
look at the unix socket in /var/run/libvirt/qemu which is owned by
root and has permission 700.  Move the unix socket to
/var/lib/libvirt/qemu, which is already owned by qemu:qemu.

Thanks to Justin Clift for test this out for me.
Signed-off-by: NChris Lalancette <clalance@redhat.com>

The problem is that on the source of the migration, libvirtd
is responsible for creating the unix socket over which the data
will flow.  Since libvirtd is running as root, this file will
be created as root.  When the qemu process running as qemu:qemu
goes to access the unix file to write data to it, it will get
permission denied and fail.  Make sure to change the owner
of the unix file to qemu:qemu.

Thanks to Justin Clift for testing this patch out for me.
Signed-off-by: NChris Lalancette <clalance@redhat.com>