When libvirt starts up all storage pools default to the inactive
state, even if the underlying storage is already active on the
host. This introduces a new API into the internal storage backend
drivers that checks whether a storage pool is already active. If
the pool is active at libvirtd startup, the volume list will be
immediately populated.

* src/storage/storage_backend.h: New internal API for checking
  storage pool state
* src/storage/storage_driver.c: Check whether a pool is active
  upon driver startup
* src/storage/storage_backend_fs.c, src/storage/storage_backend_iscsi.c,
  src/storage/storage_backend_logical.c, src/storage/storage_backend_mpath.c,
  src/storage/storage_backend_scsi.c: Add checks for pool state

* src/storage/storage_backend.h (VIR_STORAGE_VOL_OPEN_DEFAULT):
Adjust s/#define/# define/, and align continued lines.

If a directory pool contains pipes or sockets, a pool start can fail or hang:

https://bugzilla.redhat.com/show_bug.cgi?id=589577

We already try to avoid these special files, but only attempt after
opening the path, which is where the problems lie. Unify volume opening
into helper functions, which use the proper open() flags to avoid error,
followed by fstat to validate storage mode.

Previously, virStorageBackendUpdateVolTargetInfoFD attempted to enforce the
storage mode check, but allowed callers to detect this case and silently
continue. In practice, only the FS backend was using this feature, the rest
were treating unknown mode as an error condition. Unfortunately the InfoFD
function wasn't raising an error message here, so error reporting was
busted.

This patch adds 2 functions: virStorageBackendVolOpen, and
virStorageBackendVolOpenModeSkip. The latter retains the original opt out
semantics, the former now throws an explicit error.

This patch maintains the previous volume mode checks: allowing specific
modes for specific pool types requires a bit of surgery, since VolOpen
is called through several different helper functions.

v2: Use ATTRIBUTE_NONNULL. Drop stat check, just open with
    O_NONBLOCK|O_NOCTTY.

v3: Move mode check logic back to VolOpen. Use 2 VolOpen functions with
    different error semantics.

v4: Make second VolOpen function more extensible. Didn't opt to change
    FS backend defaults, this can just be to fix the original bug.

v5: Prefix default flags with VIR_, use ATTRIBUTE_RETURN_CHECK

Volume detection in the scsi backend was duplicating code already
present in storage_backend.c. Let's drop the duplicate code.

Also, change the shared function name to be less generic, and remove
some error squashing in the other call site.

* global: patch created by running:
for f in $(git ls-files '*.[ch]') ; do
    cppi $f > $f.t && mv $f.t $f
done

The virConnectPtr is no longer required for error reporting since
that is recorded in a thread local. Remove use of virConnectPtr
from all APIs in storage_conf.{h,c} and storage_encryption_conf.{h,c}
and update all callers to match

In order to avoid problems trying to chown files that were created by
root on a root-squashing nfs server, fork a new process that setuid's
to the desired uid before creating the file. (It's only done this way
if the pool containing the new volume is of type 'netfs', otherwise
the old method of creating the file followed by chown() is used.)

This changes the semantics of the "create_func" slightly - previously
it was assumed that this function just created the file, then the
caller would chown it to the desired uid. Now, create_func does both
operations.

There are multiple functions that can take on the role of create_func:

createFileDir - previously called mkdir(), now calls virDirCreate().
virStorageBackendCreateRaw - previously called open(),
                             now calls virFileCreate().
virStorageBackendCreateQemuImg - use virRunWithHook() to setuid/gid.
virStorageBackendCreateQcowCreate - same.
virStorageBackendCreateBlockFrom - preserve old behavior (but attempt
                                   chown when necessary even if not root)

* src/storage/storage_backend.[ch] src/storage/storage_backend_disk.c
  src/storage/storage_backend_fs.c src/storage/storage_backend_logical.c
  src/storage/storage_driver.c: change the create_func implementations,
  also propagate the pool information to be able to detect NETFS ones.

* src/storage/storage_backend.h: Include <stdint.h>.

* daemon/qemud.c, src/Makefile.am: Adapt for changed paths
* src/storage*.c, src/storage/*.h, src/parthelpre.c: Move
  to src/storage/

Remove the bogus dependancy between node_device.c & storage_backend.c
by moving the virWaitForDevices into util.h where it can be shared
safely

* src/storage_backend_disk.c, src/storage_backend_logical.c,
  src/storage_backend_mpath.c, src/storage_backend_scsi.c: Replace
  virStorageBackendWaitForDevices with virFileWaitForDevices
* src/storage_backend.c, src/storage_backend.h: Remove
  virStorageBackendWaitForDevices, virWaitForDevices
* src/util.h, src/util.c: Add virFileWaitForDevices
* configure.in: Move xmlrpc check further down after pkgconfig
  is detected
* src/Makefile.am: Add missing XMLRPC_CFLAGS/LIBS to opennebula
* src/libvirt_private.syms: Add many missing exports

* configure.in src/Makefile.am src/storage_backend.[ch]
  src/storage_conf.[ch] src/storage_backend_mpath.[ch] po/POTFILES.in:
  add a new module for storage multipath, it requires device-mapper

Have storage building functions be definitions of
virStorageBackendBuildVolFrom: we will need to do this in the future anyways
if we ever support the flags attribute.

These will be used by other pool cloning implementations.

* src/node_device.[ch] src/storage_backend.[ch]: fixes a
  module dependency problem, patch by Dave Allan
Daniel

There is some funkiness here, since we are either dealing with 2 different
pools (which means validation x 2) or the same pool, where we have to be
careful not to deadlock.

* src/libvirt_private.syms src/storage_backend.h
  src/storage_backend_fs.c src/storage_conf.h src/storage_driver.c:
  drop the pool lock when allocating fs volumes, patch by Cole Robinson
daniel

really should be synchronizing against udev.  This is generally done by a call
to udevsettle, which is exactly what this patch implements for the storage
backends that are likely to create new /dev nodes.  I believe I've read that
even after udevsettle, you are not guaranteed that devices are all the way
created, so we still need the polling in the rest of the sources, but this
should give us a much better chance of things existing as we expect.
Signed-off-by: NChris Lalancette <clalance@redhat.com>

ability to resolve any kind of volume path to the pool target volume
path.  For instance, if the pool was defined with a
<target><path>/dev/disk/by-id</path></target> section, and one of the
volumes is /dev/disk/by-id/scsi-S_beaf11, then you would be able to
call virStorageVolLookupByPath("/dev/sdc"), and get the correct volume
back.
Signed-off-by: NChris Lalancette <clalance@redhat.com>

cleanup patch, add a ".defaultFormat" member to .poolOptions.  In
storage_conf.c, if virXPathString(/pool/source/format/@type) returns NULL, then
set the pool type to .defaultFormat; otherwise, lookup the type via
formatFromString.
Signed-off-by: NChris Lalancette <clalance@redhat.com>

iSCSI volumes.  This is implemented in the
virStorageBackendUpdateVolInfoFD function, so all future callers will
automatically benefit.  This is a somewhat large patch because the
conversion of the virStorageBackendPartTableTypeToString necessitated
a change to the formatToString and formatFromString function pointers,
which caused fallout in other places in the storage stuff.  The good
news is that most of these callers are now converted over to the
VIR_ENUM_IMPL, which means a lot of redundant code is now gone.
Signed-off-by: NChris Lalancette <clalance@redhat.com>

* src/storage_backend.h src/storage_backend_logical.c
src/storage_conf.c src/storage_conf.h src/virsh.c:
Applied patches from David Lively to add storage source
elements needed for storage pool
* docs/formatstorage.html docs/formatstorage.html.in: associated
documentation
Daniel

*not* virStorageReportError() if the fork()/exec() process it spawned returned a
!= 0 exit code.  Rather, it returns the exitcode in this case, and it is up to
the higher level to determine whether this is a fatal error or not.  The use
case for this change is in the iSCSI stuff; older versions of iscsiadm tools
would return a failure when getting the session number, despite the command
succeeding.
Signed-off-by: NChris Lalancette <clalance@redhat.com>

Done with these commands:
git grep -l Local.variab|xargs \
  perl -0x3b -pi -e 's,\n+/\*\n \* vim:(.|\n)*,\n,'

git grep -l Local.variab|xargs \
  perl -0x3b -pi -e 's,\n+/\*\n \* Local variables:\n(.|\n)*,\n,'