To allow messages from different threads to be untangled,
include an integer thread identifier in log messages.

* src/util/logging.c: Include thread ID
* src/util/threads.h, src/util/threads.h, src/util/threads-pthread.c:
  Add new virThreadSelfID() function
* configure.ac: Check for sys/syscall.h

Do this by adding a helper function to get the persistent domain config. This
should be useful for other functions that may eventually want to alter
the persistent domain config (attach/detach device). Also make similar changes
to the test drivers setvcpus command.

A caveat is that the function will return the running config for a transient
domain, rather than error. This simplifies callers, as long as they use
other methods to ensure the guest is persistent.

This function sets the running domain definition as transient, by reparsing
the persistent config and assigning it to newDef. This ensures that any
changes made to the running definition and not the persistent config are
discarded when the VM is shutdown.

If we don't escape ' or \ xend can't parse the generated sexpr. This
might over apply the EscapeSexpr routine, but it shouldn't hurt.

* src/util/memory.h (VIR_RESIZE_N): New macro.
* src/util/memory.c (virResizeN): New function.
* src/libvirt_private.syms: Export new helper.
* docs/hacking.html.in: Document it.
* HACKING: Regenerate.

* src/util/memory.h (VIR_REALLOC_N): Update docs.
(VIR_EXPAND_N, VIR_SHRINK_N): New macros.
(virAlloc, virAllocN, virReallocN, virAllocVar, virFree): Add some
gcc attributes.
* src/util/memory.c (virExpandN, virShrinkN): New functions.
(virReallocN): Update docs.
* src/libvirt_private.syms: Export new helpers.
* docs/hacking.html.in: Prefer newer interfaces over
VIR_REALLOC_N, since uninitialized memory can bite us.
* HACKING: Regenerate.

Similarly to deprecating close(), I am now deprecating fclose() and
introduce VIR_FORCE_FCLOSE() and VIR_FCLOSE(). Also, fdopen() is replaced with
VIR_FDOPEN().

Most of the files are opened in read-only mode, so usage of
VIR_FORCE_CLOSE() seemed appropriate. Others that are opened in write
mode already had the fclose()<  0 check and I converted those to
VIR_FCLOSE()<  0.

I did not find occurrences of possible double-closed files on the way.

The util/threads.c/h code already has APIs for mutexes,
condition variables and thread locals. This commit adds
in code for actually creating threads.

* src/libvirt_private.syms: Export new symbols
* src/util/threads.h: Define APIs virThreadCreate, virThreadSelf,
  virThreadIsSelf and virThreadJoin
* src/util/threads-win32.c, src/util/threads-win32.h: Win32
  impl of threads
* src/util/threads-pthread.c, src/util/threads-pthread.h: POSIX
  impl of threads

To avoid the need for duplicating implementations of virStream
drivers, provide a generic implementation that can handle any
FD based stream. This code is copied from the existing impl
in the QEMU driver, with the locking moved into the stream
impl, and addition of a read callback

The FD stream code will refuse to operate on regular files or
block devices, since those can't report EAGAIN properly when
they would block on I/O

* include/libvirt/virterror.h, include/libvirt/virterror.h: Add
  VIR_FROM_STREAM error domain
* src/qemu/qemu_driver.c: Remove code obsoleted by the new
  generic streams driver.
* src/fdstream.h, src/fdstream.c, src/fdstream.c,
  src/libvirt_private.syms: Generic reusable FD based streams

This extends the SPICE XML to allow channel security options

    <graphics type='spice' port='-1' tlsPort='-1' autoport='yes'>
      <channel name='main' mode='secure'/>
      <channel name='record' mode='insecure'/>
    </graphics>

Any non-specified channel uses the default, which allows both
secure & insecure usage

* src/conf/domain_conf.c, src/conf/domain_conf.h,
  src/libvirt_private.syms: Add XML syntax for specifying per
  channel security options for spice.
* src/qemu/qemu_conf.c: Configure channel security with spice

Move existing routines about virSysinfoDef to an util module,
add a new entry point virSysinfoRead() to read the host values
with dmidecode

* src/conf/domain_conf.c src/conf/domain_conf.h src/util/sysinfo.c
  src/util/sysinfo.h: move to a new module, add virSysinfoRead()
* src/Makefile.am: handle the new module build
* src/libvirt_private.syms: new internal symbols
* include/libvirt/virterror.h src/util/virterror.c: defined a new
  error code for that module
* po/POTFILES.in: add new file for translations

NFS does not support file labelling, so ignore this error
for stdin_path when on NFS.

* src/security/security_selinux.c: Ignore failures on labelling
  stdin_path on NFS
* src/util/storage_file.c, src/util/storage_file.h: Refine
  virStorageFileIsSharedFS() to allow it to check for a
  specific FS type.

* src/util/util.h
* src/util/util.c
* src/libvirt_private.syms

When we mount any cgroup without "-o devices", we will fail to start vms:

error: Failed to start domain vm1
error: Unable to deny all devices for vm1: No such file or directory

When we mount any cgroup without "-o cpu", we will fail to get schedinfo:
Scheduler      : posix
error: unable to get cpu shares tunable: No such file or directory

We should only use the cgroup controllers which are mounted on host.
So I add virCgroupMounted() for qemuCgroupControllerActive()
Signed-off-by: NLai Jiangshan <laijs@cn.fujitsu.com>

Commit 9bd3cce0 added virFork and
virDriverLoadModule to libvirt_private.syms, but virFork didn't have
a body on Win32 and virDriverLoadModule was already correctly
exported conditional via libvirt_driver_modules.syms.

Add a helper API for ecscaping the value in audit log
messages

* src/util/virtaudit.h, src/util/virtaudit.c,
  src/libvirt_private.syms: Add virAuditEncode

The getnameinfo() function is more flexible than inet_ntop()
avoiding the need to if/else the code based on socket family.
Also make it support UNIX socket addrs and allow inclusion
of a port (service) address. Finally do proper error reporting
via normal APIs.

* src/conf/domain_conf.c, src/nwfilter/nwfilter_ebiptables_driver.c,
  src/qemu/qemu_conf.c: Fix error handling with virSocketFormat
* src/util/network.c: Rewrite virSocketFormat to use getnameinfo
  and cope with UNIX socket addrs.

* src/libvirt_private.syms: Sort by header name, then within
header, and drop duplicate virNetworkDefParseNode,
virFileLinkPointsTo and virXPathBoolean.

The libvirt_util.la library was mistakenly linked into libvirtd
directly. Since libvirt_util.la is already linked to libvirt.so,
this resulted in libvirtd getting two copies of the code and
more critically 2 copies of static global variables.

Testing in turn exposed a issue with loadable modules. The
gnulib replacement functions are not exported to loadable
modules. Rather than trying to figure out the name sof all
gnulib functions & export them, just linkage all loadable
modules against libgnu.la statically.

* daemon/Makefile.am: Remove linkage of libvirt_util.la
  and libvirt_driver.la
* src/Makefile.am: Link driver modules against libgnu.la
* src/libvirt.c: Don't try to load modules which were
  compiled out
* src/libvirt_private.syms: Export all other internal
  symbols that are required  by drivers

Since bugs due to double-closed file descriptors are difficult to track down in a multi-threaded system, I am introducing the VIR_CLOSE(fd) macro to help avoid mistakes here.

There are lots of places where close() is being used. In this patch I am only cleaning up usage of close() in src/conf where the problems were.

I also dare to declare close() as being deprecated in libvirt code base (HACKING).

This enables support for nested SVM using the regular CPU
model/features block. If the CPU model or features include
'svm', then the '-enable-nesting' flag will be added to the
QEMU command line. Latest out of tree patches for nested
'vmx', no longer require the '-enable-nesting' flag. They
instead just look at the cpu features. Several of the models
already include svm support, but QEMU was just masking out
the svm bit silently. So this will enable SVM on such
models

* src/qemu/qemu_conf.h: flag for -enable-nesting
* src/qemu/qemu_conf.c: Use -enable-nesting if VMX or SVM are in
  the CPUID
* src/cpu/cpu.h, src/cpu/cpu.c: API to check for a named feature
* src/cpu/cpu_x86.c: x86 impl of feature check
* src/libvirt_private.syms: Add cpuHasFeature
* src/qemuhelptest.c: Add nesting flag where required

 This is from a bug report and conversation on IRC where Soren reported that while a filter update is occurring on one or more VMs (due to a rule having been edited for example), a deadlock can occur when a VM referencing a filter is started.

The problem is caused by the two locking sequences of

qemu driver, qemu domain, filter             # for the VM start operation
filter, qemu_driver, qemu_domain            # for the filter update operation

that obviously don't lock in the same order. The problem is the 2nd lock sequence. Here the qemu_driver lock is being grabbed in qemu_driver:qemudVMFilterRebuild()

The following solution is based on the idea of trying to re-arrange the 2nd sequence of locks as follows:

qemu_driver, filter, qemu_driver, qemu_domain

and making the qemu driver recursively lockable so that a second lock can occur, this would then lead to the following net-locking sequence

qemu_driver, filter, qemu_domain

where the 2nd qemu_driver lock has been ( logically ) eliminated.

The 2nd part of the idea is that the sequence of locks (filter, qemu_domain) and (qemu_domain, filter) becomes interchangeable if all code paths where filter AND qemu_domain are locked have a preceding qemu_domain lock that basically blocks their concurrent execution

So, the following code paths exist towards qemu_driver:qemudVMFilterRebuild where we now want to put a qemu_driver lock in front of the filter lock.

-> nwfilterUndefine()   [ locks the filter ]
    -> virNWFilterTestUnassignDef()
        -> virNWFilterTriggerVMFilterRebuild()
            -> qemudVMFilterRebuild()

-> nwfilterDefine()
    -> virNWFilterPoolAssignDef() [ locks the filter ]
        -> virNWFilterTriggerVMFilterRebuild()
            -> qemudVMFilterRebuild()

-> nwfilterDriverReload()
    -> virNWFilterPoolLoadAllConfigs()
        ->virNWFilterPoolObjLoad()
            -> virNWFilterPoolAssignDef() [ locks the filter ]
                -> virNWFilterTriggerVMFilterRebuild()
                    -> qemudVMFilterRebuild()

-> nwfilterDriverStartup()
    -> virNWFilterPoolLoadAllConfigs()
        ->virNWFilterPoolObjLoad()
            -> virNWFilterPoolAssignDef() [ locks the filter ]
                -> virNWFilterTriggerVMFilterRebuild()
                    -> qemudVMFilterRebuild()

Qemu is not the only driver using the nwfilter driver, but also the UML driver calls into it. Therefore qemuVMFilterRebuild() can be exchanged with umlVMFilterRebuild() along with the driver lock of qemu_driver that can now be a uml_driver. Further, since UML and Qemu domains can be running on the same machine, the triggering of a rebuild of the filter can touch both types of drivers and their domains.

In the patch below I am now extending each nwfilter callback driver with functions for locking and unlocking the (VM) driver (UML, QEMU) and introduce new functions for locking all registered callback drivers and unlocking them. Then I am distributing the lock-all-cbdrivers/unlock-all-cbdrivers call into the above call paths. The last shown callpath starting with nwfilterDriverStart() is problematic since it is initialize before the Qemu and UML drives are and thus a lock in the path would result in a NULL pointer attempted to be locked -- the call to virNWFilterTriggerVMFilterRebuild() is never called, so we never lock either the qemu_driver or the uml_driver in that path. Therefore, only the first 3 paths now receive calls to lock and unlock all callback drivers. Now that the locks are distributed where it matters I can remove the qemu_driver and uml_driver lock from qemudVMFilterRebuild() and umlVMFilterRebuild() and not requiring the recursive locks.

For now I want to put this out as an RFC patch. I have tested it by 'stretching' the critical section after the define/undefine functions each lock the filter so I can (easily) concurrently execute another VM operation (suspend,start). That code is in this patch and if you want you can de-activate it. It seems to work ok and operations are being blocked while the update is being done.
I still also want to verify the other assumption above that locking filter and qemu_domain always has a preceding qemu_driver lock.

Provides interfaces for setting/getting memory tunables like hard_limit,
soft_limit and swap_hard_limit

Commit 1fe2927a forgot to export a symbol.

* src/libvirt_private.syms (virHexToBin): Add.
* src/.gitignore: Ignore temporary file.

The patch below extends the XML parser and generator so that every l3 protocol
now can have a state attribute.

Other drivers will need this same functionality, so move it to up to
conf/domain_conf.c and give it a more general name.
Signed-off-by: NSoren Hansen <soren@linux2go.dk>

This patch attempts to take advantage of a newly added netfilter
module to correct for a problem with some guest DHCP client
implementations when used in conjunction with a DHCP server run on the
host systems with packet checksum offloading enabled.

The problem is that, when the guest uses a RAW socket to read the DHCP
response packets, the checksum hasn't yet been fixed by the IP stack,
so it is incorrect.

The fix implemented here is to add a rule to the POSTROUTING chain of
the mangle table in iptables that fixes up the checksum for packets on
the virtual network's bridge that are destined for the bootpc port (ie
"dhcpc", ie port 68) port on the guest.

Only very new versions of iptables will have this support (it will be
in the next upstream release), so a failure to add this rule only
results in a warning message. The iptables patch is here:

  http://patchwork.ozlabs.org/patch/58525/

A corresponding kernel module patch is also required (the backend of
the iptables patch) and that will be in the next release of the
kernel.

Enable specifying a virtio console device with:

<console type='pty'>
  <target type='virtio'/>
</console>

Add the library entry point for the new virDomainQemuMonitorCommand()
entry point.  Because this is not part of the "normal" libvirt API,
it gets its own header file, library file, and will eventually
get its own over-the-wire protocol later in the series.

Changes since v1:
 - Go back to using the virDriver table for qemuDomainMonitorCommand, due to
   linking issues
 - Added versioning information to the libvirt-qemu.so

Changes since v2:
 - None

Changes since v3:
 - Add LGPL header to libvirt-qemu.c
 - Make virLibConnError and virLibDomainError macros instead of function calls

Changes since v4:
 - Move exported symbols to libvirt_qemu.syms
Signed-off-by: NChris Lalancette <clalance@redhat.com>

To allow compatibility with older QEMU PCI device slot assignment
it is necessary to explicitly track the balloon device in the
XML. This introduces a new device

   <memballoon model='virtio|xen'/>

It can also have a PCI address, auto-assigned if necessary.

The memballoon will be automatically added to all Xen and QEMU
guests by default.

* docs/schemas/domain.rng: Add <memballoon> element
* src/conf/domain_conf.c, src/conf/domain_conf.h: parsing
  and formatting for memballoon device. Always add a memory
  balloon device to Xen/QEMU if none exists in XML
* src/libvirt_private.syms: Export memballoon model APIs
* src/qemu/qemu_conf.c, src/qemu/qemu_conf.h: Honour the
  PCI device address in memory balloon device
* tests/*: Update to test new functionality

There is duplicated code which iterates over disk backing stores
performing some action. Provide a convenient helper for doing
this to eliminate duplication & risk of mistakes with disk format
probing

* src/conf/domain_conf.c, src/conf/domain_conf.h,
  src/libvirt_private.syms: Add virDomainDiskDefForeachPath()

The virStorageFileGetMetadataFromFD did two jobs in one. First
it probed for storage type, then it extracted metadata for the
type. It is desirable to be able to separate these jobs, allowing
probing without querying metadata, and querying metadata without
probing.

To prepare for this, split out probing code into a new pair of
methods

  virStorageFileProbeFormatFromFD
  virStorageFileProbeFormat

* src/util/storage_file.c, src/util/storage_file.h,
  src/libvirt_private.syms: Introduce virStorageFileProbeFormat
  and virStorageFileProbeFormatFromFD

The parallel, serial, console and channel devices are all just
character devices. A lot of code needs todo the same thing to
all these devices. This provides an convenient API for iterating
over all of them.

* src/conf/domain_conf.c, src/conf/domain_conf.c,
  src/libvirt_private.syms: Add virDomainChrDefForeach

Following Daniel Berrange's multiple helpful suggestions for improving
this patch and introducing another driver interface, I now wrote the
below patch where the nwfilter driver registers the functions to
instantiate and teardown the nwfilters with a function in
conf/domain_nwfilter.c called virDomainConfNWFilterRegister. Previous
helper functions that were called from qemu_driver.c and qemu_conf.c
were move into conf/domain_nwfilter.h with slight renaming done for
consistency. Those functions now call the function expored by
domain_nwfilter.c, which in turn call the functions of the new driver
interface, if available.

The network driver is not doing correct checking for
duplicate UUID/name values. This introduces a new method
virNetworkObjIsDuplicate, based on the previously
written virDomainObjIsDuplicate.

* src/conf/network_conf.c, src/conf/network_conf.c,
  src/libvirt_private.syms: Add virNetworkObjIsDuplicate,
* src/network/bridge_driver.c: Call virNetworkObjIsDuplicate
  for checking uniqueness of uuid/names

The storage pool driver is not doing correct checking for
duplicate UUID/name values. This introduces a new method
virStoragePoolObjIsDuplicate, based on the previously
written virDomainObjIsDuplicate.

* src/conf/storage_conf.c, src/conf/storage_conf.c,
  src/libvirt_private.syms: Add virStoragePoolObjIsDuplicate,
* src/storage/storage_driver.c: Call virStoragePoolObjIsDuplicate
  for checking uniqueness of uuid/names

This patch that adds support for configuring 802.1Qbg and 802.1Qbh
switches. The 802.1Qbh part has been successfully tested with real
hardware. The 802.1Qbg part has only been tested with a (dummy)
server that 'behaves' similarly to how we expect lldpad to 'behave'.

The following changes were made during the development of this patch:

 - Merging Scott's v13-pre1 patch
 - Fixing endptr related bug while using virStrToLong_ui() pointed out
   by Jim Meyering
 - Addressing Jim Meyering's comments to v11
 - requiring mac address to the vpDisassociateProfileId() function to
   pass it further to the 802.1Qbg disassociate part (802.1Qbh untouched)
 - determining pid of lldpad daemon by reading it from /var/run/libvirt.pid
   (hardcode as is hardcode alson in lldpad sources)
 - merging netlink send code for kernel target and user space target
   (lldpad) using one function nlComm() to send the messages
 - adding a select() after the sending and before the reading of the
   netlink response in case lldpad doesn't respond and so we don't hang
 - when reading the port status, in case of 802.1Qbg, no status may be
   received while things are 'in progress' and only at the end a status
   will be there.
 - when reading the port status, use the given instanceId and vf to pick
   the right IFLA_VF_PORT among those nested under IFLA_VF_PORTS.
 - never sending nor parsing IFLA_PORT_SELF type of messages in the
   802.1Qbg case
 - iterating over the elements in a IFLA_VF_PORTS to pick the right
   IFLA_VF_PORT by either IFLA_PORT_PROFILE and given profileId
   (802.1Qbh) or IFLA_PORT_INSTANCE_UUID and given instanceId (802.1Qbg)
   and reading the current status in IFLA_PORT_RESPONSE.
 - recycling a previous patch that adds functionality to interface.c to
   - get the vlan identifier on an interface
   - get the flags of an interface and some convenience function to
     check whether an interface is 'up' or not (not currently used here)
 - adding function to determine the root physical interface of an
   interface. For example if a macvtap is linked to eth0.100, it will
   find eth0. Also adding a function that finds the vlan on the 'way to
   the root physical interface'
 - conveying the root physical interface name and index in case of 802.1Qbg
 - conveying mac address of macvlan device and vlan identifier in
   IFLA_VFINFO_LIST[ IFLA_VF_INFO[ IFLA_VF_MAC(mac), IFLA_VF_VLAN(vlan) ] ]
   to (future) lldpad via netlink
  - To enable build with --without-macvtap rename the
    [dis|]associatePortProfileId functions, prepend 'vp' before their
    name and make them non-static functions.
  - Renaming variable multicast to nltarget_kernel and inverting
    the logic
  - Addressing Jim Meyering's comments; this also touches existing
    code for example for correcting indentation of break statements or
    simplification of switch statements.
  - Renamed occurrencvirVirtualPortProfileDef to virVirtualPortProfileParamses
  - 802.1Qbg part prepared for sending a RTM_SETLINK and getting
    processing status back plus a subsequent RTM_GETLINK to
    get IFLA_PORT_RESPONSE.
    Note: This interface for 802.1Qbg may still change
  - [David Allan] move getPhysfn inside IFLA_VF_PORT_MAX to avoid
compiler
    warning when latest if_link.h isn't available
  - move from Stefan's 802.1Qb{g|h} XML v8 to v9
  - move hostuuid and vf index calcs to inside doPortProfileOp8021Qbh
  - remove debug fprintfs
  - use virGetHostUUID (thanks Stefan!)
  - fix compile issue when latest if_link.h isn't available
  - change poll timeout to 10s, at 1/8 intervals
     - if polling times out, log msg and return -ETIMEDOUT
  - Add Stefan's code for getPortProfileStatus
  - Poll for up to 2 secs for port-profile status, at 1/8 sec intervals:
     - if status indicates error, abort openMacvtapTap
     - if status indicates success, exit polling
     - if status is "in-progress" after 2 secs of polling, exit
       polling loop silently, without error

My patch finishes out the 802.1Qbh parts, which Stefan had mostly complete.
I've tested using the recent kernel updates for VF_PORT netlink msgs and
enic for Cisco's 10G Ethernet NIC.  I tested many VMs, each with several
direct interfaces, each configured with a port-profile per the XML.  VM-to-VM,
and VM-to-external work as expected.  VM-to-VM on same host (using same NIC)
works same as VM-to-VM where VMs are on diff hosts.  I'm able to change
settings on the port-profile while the VM is running to change the virtual
port behaviour.  For example, adjusting a QoS setting like rate limit.  All
VMs with interfaces using that port-profile immediatly see the effect of the
change to the port-profile.

I don't have a SR-IOV device to test so source dev is a non-SR-IOV device,
but most of the code paths include support for specifing the source dev and
VF index.  We'll need to complete this by discovering the PF given the VF
linkdev.  Once we have the PF, we'll also have the VF index.  All this info-
mation is available from sysfs.

We've been running into a lot of situations where
virGetHostname() is returning "localhost", where a plain
gethostname() would have returned the correct thing.  This
is because virGetHostname() is *always* trying to canonicalize
the name returned from gethostname(), even when it doesn't
have to.

This patch changes virGetHostname so that if the value returned
from gethostname() is already FQDN or localhost, it returns
that string directly.  If the value returned from gethostname()
is a shortened hostname, then we try to canonicalize it.  If
that succeeds, we returned the canonicalized hostname.  If
that fails, and/or returns "localhost", then we just return
the original string we got from gethostname() and hope for
the best.

Note that after this patch it is up to clients to check whether
"localhost" is an allowed return value.  The only place
where it's currently not is in qemu migration.
Signed-off-by: NChris Lalancette <clalance@redhat.com>

Spurious / in a pool target path makes life difficult for apps using the
GetVolByPath, and doing other path based comparisons with pools. This
has caused a few issues for virt-manager users:

https://bugzilla.redhat.com/show_bug.cgi?id=494005
https://bugzilla.redhat.com/show_bug.cgi?id=593565

Add a new util API which removes spurious /, virFileSanitizePath. Sanitize
target paths when parsing pool XML, and for paths passed to GetVolByPath.

v2: Leading // must be preserved, properly sanitize path=/, sanitize
    away /./ -> /

v3: Properly handle starting ./ and ending /.

v4: Drop all '.' handling, just sanitize / for now.

Allow for a host UUID in the capabilities XML. Local drivers
will initialize this from the SMBIOS data. If a sanity check
shows SMBIOS uuid is invalid, allow an override from the
libvirtd.conf configuration file

* daemon/libvirtd.c, daemon/libvirtd.conf: Support a host_uuid
  configuration option
* docs/schemas/capability.rng: Add optional host uuid field
* src/conf/capabilities.c, src/conf/capabilities.h: Include
  host UUID in XML
* src/libvirt_private.syms: Export new uuid.h functions
* src/lxc/lxc_conf.c, src/qemu/qemu_driver.c,
  src/uml/uml_conf.c: Set host UUID in capabilities
* src/util/uuid.c, src/util/uuid.h: Support for host UUIDs
* src/node_device/node_device_udev.c: Use the host UUID functions
* tests/confdata/libvirtd.conf, tests/confdata/libvirtd.out: Add
  new host_uuid config option to test