* .gnulib: Update to latest, for at least a stdint.h fix
* src/storage/storage_driver.c (storageVolumeZeroSparseFile)
(storageWipeExtent): Use better type, although it still triggers
spurious -Wformat warning on MacOS's gcc.

Many operations are not valid on inactive storage pools. The
storage driver is currently returning VIR_ERR_INTERNAL_ERROR
in these cases, rather than the more suitable error code
VIR_ERR_OPERATION_INVALID

* src/storage/storage_driver.c: Fix error code when pool
  is not active

When libvirt starts up all storage pools default to the inactive
state, even if the underlying storage is already active on the
host. This introduces a new API into the internal storage backend
drivers that checks whether a storage pool is already active. If
the pool is active at libvirtd startup, the volume list will be
immediately populated.

* src/storage/storage_backend.h: New internal API for checking
  storage pool state
* src/storage/storage_driver.c: Check whether a pool is active
  upon driver startup
* src/storage/storage_backend_fs.c, src/storage/storage_backend_iscsi.c,
  src/storage/storage_backend_logical.c, src/storage/storage_backend_mpath.c,
  src/storage/storage_backend_scsi.c: Add checks for pool state

* bootstrap.conf (gnulib_modules): Add configmake.
* daemon/Makefile.am (libvirtd_CFLAGS): Drop defines provided by
gnulib.
* src/Makefile.am (INCLUDES): Likewise.
* tests/Makefile.am (INCLUDES): Likewise.
* tools/Makefile.am (virsh_CFLAGS): Likewise.
* daemon/libvirtd.c (qemudInitPaths, usage, main): Update
clients.
* src/cpu/cpu_map.c (CPUMAPFILE): Likewise.
* src/driver.c (DEFAULT_DRIVER_DIR): Likewise.
* src/internal.h (_): Likewise.
* src/libvirt.c (virInitialize): Likewise.
* src/lxc/lxc_conf.h (LXC_CONFIG_DIR, LXC_STATE_DIR, LXC_LOG_DIR):
Likewise.
* src/lxc/lxc_conf.c (lxcCapsInit, lxcLoadDriverConfig):
Likewise.
* src/network/bridge_driver.c (NETWORK_PID_DIR)
(NETWORK_STATE_DIR, DNSMASQ_STATE_DIR, networkStartup): Likewise.
* src/nwfilter/nwfilter_driver.c (nwfilterDriverStartup):
Likewise.
* src/qemu/qemu_conf.c (qemudLoadDriverConfig): Likewise.
* src/qemu/qemu_driver.c (qemudStartup): Likewise.
* src/remote/remote_driver.h (LIBVIRTD_PRIV_UNIX_SOCKET)
(LIBVIRTD_PRIV_UNIX_SOCKET_RO, LIBVIRTD_CONFIGURATION_FILE)
(LIBVIRT_PKI_DIR): Likewise.
* src/secret/secret_driver.c (secretDriverStartup): Likewise.
* src/security/security_apparmor.c (VIRT_AA_HELPER): Likewise.
* src/security/virt-aa-helper.c (main): Likewise.
* src/storage/storage_backend_disk.c (PARTHELPER): Likewise.
* src/storage/storage_driver.c (storageDriverStartup): Likewise.
* src/uml/uml_driver.c (TEMPDIR, umlStartup): Likewise.
* src/util/hooks.c (LIBVIRT_HOOK_DIR): Likewise.
* tools/virsh.c (main): Likewise.
* docs/hooks.html.in: Likewise.

Using automated replacement with sed and editing I have now replaced all
occurrences of close() with VIR_(FORCE_)CLOSE() except for one, of
course. Some replacements were straight forward, others I needed to pay
attention. I hope I payed attention in all the right places... Please
have a look. This should have at least solved one more double-close
error.

The storage pool driver is mistakenly using the error code
VIR_ERR_INVALID_STORAGE_POOL which is for diagnosing invalid
pointers. This patch switches it to use VIR_ERR_NO_STORAGE_POOL
which is the correct code for cases where the storage pool does
not exist

* src/storage/storage_driver.c: Replace VIR_ERR_INVALID_STORAGE_POOL
  with VIR_ERR_NO_STORAGE_POOL

The storage pool driver is not doing correct checking for
duplicate UUID/name values. This introduces a new method
virStoragePoolObjIsDuplicate, based on the previously
written virDomainObjIsDuplicate.

* src/conf/storage_conf.c, src/conf/storage_conf.c,
  src/libvirt_private.syms: Add virStoragePoolObjIsDuplicate,
* src/storage/storage_driver.c: Call virStoragePoolObjIsDuplicate
  for checking uniqueness of uuid/names

Spurious / in a pool target path makes life difficult for apps using the
GetVolByPath, and doing other path based comparisons with pools. This
has caused a few issues for virt-manager users:

https://bugzilla.redhat.com/show_bug.cgi?id=494005
https://bugzilla.redhat.com/show_bug.cgi?id=593565

Add a new util API which removes spurious /, virFileSanitizePath. Sanitize
target paths when parsing pool XML, and for paths passed to GetVolByPath.

v2: Leading // must be preserved, properly sanitize path=/, sanitize
    away /./ -> /

v3: Properly handle starting ./ and ending /.

v4: Drop all '.' handling, just sanitize / for now.

Run this:
  git grep -l 'VIR_ERROR\s*("'|xargs perl -pi -e \
    's/(VIR_ERROR)\s*\((".*?"),/$1(_($2),/'

* global: patch created by running:
for f in $(git ls-files '*.[ch]') ; do
    cppi $f > $f.t && mv $f.t $f
done

When trying to delete a pool the error message claimed the volume
could not be deleted.

* src/storage/storage_driver.c: Error message referred to
  volumes instead of pools

The virConnectPtr is no longer required for error reporting since
that is recorded in a thread local. Remove use of virConnectPtr
from all APIs in storage_conf.{h,c} and storage_encryption_conf.{h,c}
and update all callers to match

It was used for error reporting only.

In order to avoid problems trying to chown files that were created by
root on a root-squashing nfs server, fork a new process that setuid's
to the desired uid before creating the file. (It's only done this way
if the pool containing the new volume is of type 'netfs', otherwise
the old method of creating the file followed by chown() is used.)

This changes the semantics of the "create_func" slightly - previously
it was assumed that this function just created the file, then the
caller would chown it to the desired uid. Now, create_func does both
operations.

There are multiple functions that can take on the role of create_func:

createFileDir - previously called mkdir(), now calls virDirCreate().
virStorageBackendCreateRaw - previously called open(),
                             now calls virFileCreate().
virStorageBackendCreateQemuImg - use virRunWithHook() to setuid/gid.
virStorageBackendCreateQcowCreate - same.
virStorageBackendCreateBlockFrom - preserve old behavior (but attempt
                                   chown when necessary even if not root)

* src/storage/storage_backend.[ch] src/storage/storage_backend_disk.c
  src/storage/storage_backend_fs.c src/storage/storage_backend_logical.c
  src/storage/storage_driver.c: change the create_func implementations,
  also propagate the pool information to be able to detect NETFS ones.

* src/storage/storage_driver.c: Fix IsPersistent() and IsActivE()
  methods on storage pools to use 'storagePrivateData' instead
  of 'privateData'. Also fix naming convention of objects

Replace free(virBufferContentAndReset()) with virBufferFreeAndReset().
Update documentation and replace all remaining calls to free() with
calls to VIR_FREE(). Also add missing calls to virBufferFreeAndReset()
and virReportOOMError() in OOM error cases.

This implements the virConnectIsSecure, virConnectIsEncrypted,
virDomainIsPersistent, virDomainIsActive, virNetworkIsActive,
virNetworkIsPersistent, virStoragePoolIsActive,
virStoragePoolIsPersistent, virInterfaceIsActive APIs in
(nearly) all drivers. Exceptions are:

 phyp: missing domainIsActive/Persistent
 esx: missing domainIsPersistent
 opennebula: missing domainIsActive/Persistent

* src/remote/remote_protocol.x: Define remote wire ABI for newly
  added APIs.
* daemon/remote_dispatch*.h: Re-generated from remote_protocol.x
* src/esx/esx_driver.c, src/lxc/lxc_driver.c, src/network/bridge_driver.c,
  src/opennebula/one_driver.c, src/openvz/openvz_conf.c,
  src/openvz/openvz_driver.c, src/phyp/phyp_driver.c,
  src/remote/remote_driver.c, src/storage/storage_driver.c,
  src/test/test_driver.c, src/uml/uml_driver.c, src/vbox/vbox_tmpl.c,
  src/xen/xen_driver.c, src/xen/xen_driver.h, src/xen/xen_inotify.c,
  src/xen/xen_inotify.h: Implement all the new APIs where possible

* src/libvirt.c src/lxc/lxc_conf.c src/lxc/lxc_container.c
  src/lxc/lxc_controller.c src/node_device/node_device_hal.c
  src/openvz/openvz_conf.c src/qemu/qemu_driver.c
  src/qemu/qemu_monitor_text.c src/remote/remote_driver.c
  src/storage/storage_backend_disk.c src/storage/storage_driver.c
  src/util/logging.c src/xen/sexpr.c src/xen/xend_internal.c
  src/xen/xm_internal.c: Steve Grubb <sgrubb@redhat.com> sent a code
  review and those are the fixes correcting the problems

The LXC driver was mistakenly returning -1 for lxcStartup()
in scenarios that are not an error. This caused the libvirtd
to quit for unprivileged users. This fixes the return code
of LXC driver, and also adds a "name" field to the virStateDriver
struct and logging to make it easier to find these problems
in the future

* src/driver.h: Add a 'name' field to state driver to allow
  easy identification during failures
* src/libvirt.c: Log name of failed driver for virStateInit
  failures
* src/lxc/lxc_driver.c: Don't return a failure code for
  lxcStartup() if LXC is not available on this host, simply
  disable the driver.
* src/network/bridge_driver.c, src/node_device/node_device_devkit.c,
  src/node_device/node_device_hal.c, src/opennebula/one_driver.c,
  src/qemu/qemu_driver.c, src/remote/remote_driver.c,
  src/secret/secret_driver.c, src/storage/storage_driver.c,
  src/uml/uml_driver.c, src/xen/xen_driver.c: Fill in name
  field in virStateDriver struct

* daemon/qemud.c, src/Makefile.am: Adapt for changed paths
* src/storage*.c, src/storage/*.h, src/parthelpre.c: Move
  to src/storage/

Fix all thread locking bugs reported by object-locking test
case.

NB, some of the driver locking is getting too coarse. Driver
mutexes really need to be turned into RW locks instead to
significantly increase concurrency.

* src/lxc_driver.c: Fix useof driver when unlocked in the methods
  lxcDomainGetInfo, lxcSetSchedulerParameters, and
  lxcGetSchedulerParameters
* src/opennebula/one_driver.c: Fix missing unlock in oneDomainUndefine.
  Fix use of driver when unlocked in oneDomainGetInfo,
  oneGetOSType, oneDomainShutdown
* src/qemu_driver.c: Fix use of driver when unlocked in
  qemudDomainSavem, qemuGetSchedulerType, qemuSetSchedulerParameters
  and qemuGetSchedulerParameters
* src/storage_driver.c: Re-work storagePoolCreate to avoid bogus
  lock checking warning. Re-work storageVolumeCreateXMLFrom to
  remove a potential NULL de-reference & avoid bogus lock check
  warnings
* src/test.c: Remove testDomainAssignDef since it break lock chekc
  warnings.
* tests/object-locking.ml: Add oneDriverLock, oneDriverUnlock
  and one_driver_t methods/types to allow lock checking on the
   OpenNebula drivers

* src/storage_driver.c (storagePoolGetInfo, storagePoolDefine):
Remove variable, backend, since its value was never used.

We need to unlock the first pool before looking up the second, since the
search locks every pool it checks.

* src/network_driver.c src/node_device.c src/storage_driver.c:
  many places in the code reported 'No xxx with matching name" after
  a Lookup error without reporting the name used by the failed lookup

  by mistake, patch by Dave Allan.
daniel

The storage driver arranges its parsing routines in a way that make them
difficult to use in the test driver for non-default file parsing. This
refactoring moves things to be consistent with the way domain_conf and
network_conf do things.

CreateXMLFrom changes accidentally caused all raw volume creation to be
fully allocated (as though allocation == capacity). Fix this.

Also force CreateXMLFrom to maintain the previous behavior: sparseness
should still be maintained since we search for holes when copying, and the
clone behavior hasn't been tested with anything but the broken behavior.

  patch by Henrik Persson
Daniel

There is some funkiness here, since we are either dealing with 2 different
pools (which means validation x 2) or the same pool, where we have to be
careful not to deadlock.