Add APIs that will allow to use the storage driver to assist in
operations on files even for remote filesystems without native
representation as files in the host.

Include the name of the storage backend in the error message instead of
just the number.

When attempting to backport gluster pools to an older versoin
where there is no VIR_STRDUP, I got a crash from calling
strdup(,NULL). Rather than relying on the current else branch
safely doing nothing when there is no fd, it is easier to just
skip it. While at it, there's no need to explicitly set
perms.label to NULL after a VIR_FREE().

* src/storage/storage_backend.c
(virStorageBackendUpdateVolTargetInfoFD): Minor optimization.
Signed-off-by: NEric Blake <eblake@redhat.com>

Commit 348b4e25 introduced a potential problem (thankfully not
in any release): we are attempting to use virFileReadHeaderFD()
on a file that was opened with O_NONBLOCK.  While this
shouldn't be a problem in practice (because O_NONBLOCK
typically doesn't affect regular or block files, and fifos and
sockets cannot be storage volumes), it's better to play it safe
to avoid races from opening an unexpected file type while also
avoiding problems with having to handle EAGAIN while read()ing.

Based on a report by Dan Berrange.

* src/storage/storage_backend.c
(virStorageBackendVolOpenCheckMode): Fix up fd after avoiding race.
Signed-off-by: NEric Blake <eblake@redhat.com>

We already had code for handling allocation different than
capacity for sparse files; we just had to wire it up to be
used when inspecting gluster images.

* src/storage/storage_backend.c
(virStorageBackendUpdateVolTargetInfoFD): Handle no fd.
* src/storage/storage_backend_gluster.c
(virStorageBackendGlusterRefreshVol): Handle sparse files.
Signed-off-by: NEric Blake <eblake@redhat.com>

We support gluster volumes in domain XML, so we also ought to
support them as a storage pool.  Besides, a future patch will
want to take advantage of libgfapi to handle the case of a
gluster device holding qcow2 rather than raw storage, and for
that to work, we need a storage backend that can read gluster
storage volume contents.  This sets up the framework.

Note that the new pool is named 'gluster' to match a
<disk type='network'><source protocol='gluster'> image source
already supported in a <domain>; it does NOT match the
<pool type='netfs'><source><target type='glusterfs'>,
since that uses a FUSE mount to a local file name rather than
a network name.

This and subsequent patches have been tested against glusterfs
3.4.1 (available on Fedora 19); there are likely bugs in older
versions that may prevent decent use of gfapi, so this patch
enforces the minimum version tested.  A future patch may lower
the minimum.  On the other hand, I hit at least two bugs in
3.4.1 that will be fixed in 3.5/3.4.2, where it might be worth
raising the minimum: glfs_readdir is nicer to use than
glfs_readdir_r [1], and glfs_fini should only return failure on
an actual failure [2].

[1] http://lists.gnu.org/archive/html/gluster-devel/2013-10/msg00085.html
[2] http://lists.gnu.org/archive/html/gluster-devel/2013-10/msg00086.html

* configure.ac (WITH_STORAGE_GLUSTER): New conditional.
* m4/virt-gluster.m4: new file.
* libvirt.spec.in (BuildRequires): Support gluster in spec file.
* src/conf/storage_conf.h (VIR_STORAGE_POOL_GLUSTER): New pool
type.
* src/conf/storage_conf.c (poolTypeInfo): Treat similar to
sheepdog and rbd.
(virStoragePoolDefFormat): Don't output target for gluster.
* src/storage/storage_backend_gluster.h: New file.
* src/storage/storage_backend_gluster.c: Likewise.
* po/POTFILES.in: Add new file.
* src/storage/storage_backend.c (backends): Register new type.
* src/Makefile.am (STORAGE_DRIVER_GLUSTER_SOURCES): Build new files.
* src/storage/storage_backend.h (_virStorageBackend): Documet
assumption.
Signed-off-by: NEric Blake <eblake@redhat.com>

vol-clone reports out of memory error with disk type on ppc64.

Currently, wbytes is defined as size_t type (8 bytes), but
args's value in ioctl(fd, args..) in kernel is int (4 bytes).
This makes wbytes 2^32 times larger, causing an out of memory error.

This patch changes size_t to int to synchronize with kernel.

[1] https://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git/tree/block/ioctl.c?id=5e01dc7b#n363
[2] https://lkml.org/lkml/2013/11/1/620Signed-off-by: NLi Zhang <zhlcindy@linux.vnet.ibm.com>
Signed-off-by: NJán Tomko <jtomko@redhat.com>

We are calling fstat() at least twice per storage volume in
a directory storage pool; this is rather wasteful.  Refactoring
this is also a step towards making code reusable for gluster,
where gluster can provide struct stat but cannot use fstat().

* src/storage/storage_backend.h
(virStorageBackendVolOpenCheckMode)
(virStorageBackendUpdateVolTargetInfoFD): Update signature.
* src/storage/storage_backend.c
(virStorageBackendVolOpenCheckMode): Pass stat results back.
(virStorageBackendUpdateVolTargetInfoFD): Use existing stats.
(virStorageBackendVolOpen, virStorageBackendUpdateVolTargetInfo):
Update callers.
* src/storage/storage_backend_fs.c (virStorageBackendProbeTarget):
Likewise.
* src/storage/storage_backend_scsi.c
(virStorageBackendSCSIUpdateVolTargetInfo): Likewise.
* src/storage/storage_backend_mpath.c
(virStorageBackendMpathUpdateVolTargetInfo): Likewise.
Signed-off-by: NEric Blake <eblake@redhat.com>

Most of the usage of getuid()/getgid() is in cases where we are
considering what privileges we have. As such the code should be
using the effective IDs, not real IDs.
Signed-off-by: NDaniel P. Berrange <berrange@redhat.com>

Commit id '532fef36' added a call to fallocate() and some error
handling based on whether or not the function existed. This new
call resulted in libvirt-cim/cimtest failures when attempting to
create a volume with "0" (zero) allocation value. The failure is
logged as:

Oct  9 07:51:33 localhost libvirtd[8030]: cannot allocate 0 bytes in
file '/var/lib/libvirt/images/cimtest-vol.img': Invalid argument

This can also be seen with virsh vol-create-as:

error: Failed to create vol test
error: cannot allocate 0 bytes in file '/home/vm-images/test': Invalid
argument

error: Failed to create vol test
error: cannot allocate 0 bytes in file '/home/vm-images/test': Invalid
argument

It turns out fallocate() will return EINVAL when the incoming 'len'
(or allocation) value is 0 (or less).

Commit 532fef36 added two-state 'need_alloc' and exposed
'want_sparse' which also only has two states.

Change their type from int to bool.

On RHEL 5, compilation fails with:

storage/storage_backend.c: In function 'createRawFile':
storage/storage_backend.c:339: warning: implicit declaration of function 'fallocate'
storage/storage_backend.c:339: warning: nested extern declaration of 'fallocate' [-Wnested-externs]

But:

$ grep HAVE_FALLOCATE config.h
/* #undef HAVE_FALLOCATE */

Huh? It turns out that in kernels that old, fallocate() is not
implemented (config.h is correct), but <linux/fs.h> defines
HAVE_FALLOCATE as an empty witness macro for a completely
different purpose.  Since storage_backend.c is including
<linux/fs.h> on RHEL 5, we are hosed by the kernel definition.
Newer kernels no longer pollute the namespace, and it's fairly
easy to convert to an expression that works with both the old
kernel witness and the new-style config.h (undefined or 1).

Problem introduced in commit 532fef36.

* src/storage/storage_backend.c (createRawFile): Avoid namespace
pollution from kernel, by checking HAVE_FALLOCATE for a value.
Signed-off-by: NEric Blake <eblake@redhat.com>

Fixed the safezero call for allocating the rest of the file after cloning
an existing volume; it used to always use a zero offset, causing it to
only allocate the beginning of the file.

Also modified file creation to try to use fallocate(2) to pre-allocate
disk space before copying any data to make sure it fails early on if disk
is full and makes sure we can skip zero blocks when copying file contents.

If fallocate isn't available we will zero out the rest of the file after
cloning and only use sparse cloning if client requested a lower allocation
than the input volume's capacity.
Signed-off-by: NOskari Saarenmaa <os@ohmu.fi>

POSIX states that the preg parameter to regcomp() is undefined on
failure, so no need to call regfree() in these cases.

http://pubs.opengroup.org/onlinepubs/009695399/functions/regcomp.html

See also a discussion on the libvirt dev list

https://www.redhat.com/archives/libvir-list/2013-September/msg00262.html

The VIR_FREE() macro will cast away any const-ness. This masked a
number of places where we passed a 'const char *' string to
VIR_FREE. Fortunately in all of these cases, the variable was not
in fact const data, but a heap allocated string. Fix all the
variable declarations to reflect this.
Signed-off-by: NDaniel P. Berrange <berrange@redhat.com>

qemu-img is going to switch the default for QCOW2
to QCOW2v3 (compat=1.1)

Extend the probing for qemu-img command line options to check
if -o compat is supported. If the volume definition specifies
the qcow2 format but no compat level and -o compat is supported,
specify -o compat=0.10 to create a QCOW2v2 image.

https://bugzilla.redhat.com/show_bug.cgi?id=997977

The switch statement in 'virStorageBackendCreateQemuImgOpts' used the
for loop end condition 'VIR_STORAGE_FILE_FEATURE_LAST' as a possible value,
but since that cannot happen Coverity spits out a DEADCODE message. Adding
the Coverity tag just removes the Coverity message

Convert the type of loop iterators named 'i', 'j', k',
'ii', 'jj', 'kk', to be 'size_t' instead of 'int' or
'unsigned int', also santizing 'ii', 'jj', 'kk' to use
the normal 'i', 'j', 'k' naming
Signed-off-by: NDaniel P. Berrange <berrange@redhat.com>

Commit id '53d5967c' introduced the following:

TEST: storagevolxml2argvtest
      ..............                           14  OK
==25636== 358 (264 direct, 94 indirect) bytes in 1 blocks are definitely lost in loss record 67 of 75
==25636==    at 0x4A06B6F: calloc (vg_replace_malloc.c:593)
==25636==    by 0x4C95791: virAlloc (viralloc.c:124)
==25636==    by 0x4CA0BB4: virCommandNewArgs (vircommand.c:805)
==25636==    by 0x4CA0C88: virCommandNew (vircommand.c:789)
==25636==    by 0x408602: virStorageBackendCreateQemuImgCmd (storage_backend.c:849)
==25636==    by 0x405427: testCompareXMLToArgvHelper (storagevolxml2argvtest.c:61)
==25636==    by 0x4064DF: virtTestRun (testutils.c:158)
==25636==    by 0x40516F: mymain (storagevolxml2argvtest.c:195)
==25636==    by 0x406B1A: virtTestMain (testutils.c:722)
==25636==    by 0x37C1021A04: (below main) (libc-start.c:225)
==25636==
PASS: storagevolxml2argvtest

It's not used anywhere except for the switch in
virStorageBackendCreateQemuImgOpts, where leaving it in causes
a dead code coverity warning and omitting it breaks compilation
because of unhandled enum value.

Introduced by 6298f74d.

Add -o compat= and -o lazy_refcounts options for qemu-img.

Split out option string generation to make adding new options easier
and simplify the code.

This has been disabled for years

Found with 'git grep "= 1"'.

Setting of local variables in virStorageBackendCreateQemuImgCmd was
unnecessarily cluttered with ternary operators and repeated testing of
of conditions.

This patch refactors the function to use if statements and improves
error reporting in case inputvol is specified but does not contain
target path. Previously we would complain about "unknown storage vol
type 0" instead of the actual problem.

Instead of a unknown flag error report that metadata pre-allocation is
not supported with the requested volume creation method.

Resolves: https://bugzilla.redhat.com/show_bug.cgi?id=969982

Comply with the coding standard and save a few lines.

This resolves:

     https://bugzilla.redhat.com/show_bug.cgi?id=851411
     https://bugzilla.redhat.com/show_bug.cgi?id=955500

The first problem was that virFileOpenAs was returning fd (-1) in one
of the error cases rather than ret (-errno), so the caller thought
that the error was EPERM rather than ENOENT.

The second problem was that some log messages in the general purpose
qemuOpenFile() function would always say "Failed to create" even if
the caller hadn't included O_CREAT (i.e. they were trying to open an
existing file).

This fixes virFileOpenAs to jump down to the error return (which
returns ret instead of fd) in the previously mentioned incorrect
failure case of virFileOpenAs(), removes all error logging from
virFileOpenAs() (since the callers report it), and modifies
qemuOpenFile to appropriately use "open" or "create" in its log
messages.

NB: I seriously considered removing logging from all callers of
virFileOpenAs(), but there is at least one case where the caller
doesn't want virFileOpenAs() to log any errors, because it's just
going to try again (qemuOpenFile()). We can't simply make a silent
variation of virFileOpenAs() though, because qemuOpenFile() can't make
the decision about whether or not it wants to retry until after
virFileOpenAs() has already returned an error code.

Likewise, I also considered changing virFileOpenAs() to return -1 with
errno set on return, and may still do that, but only as a separate
patch, as it obscures the intent of this patch too much.

The source code base needs to be adapted as well. Some files
include virutil.h just for the string related functions (here,
the include is substituted to match the new file), some include
virutil.h without any need (here, the include is removed), and
some require both.

Ensure that the driver struct field names match the public
API names. For an API virXXXX we must have a driver struct
field xXXXX. ie strip the leading 'vir' and lowercase any
leading uppercase letters.
Signed-off-by: NDaniel P. Berrange <berrange@redhat.com>

When creating a logical volume with virStorageVolCreateXMLFrom,
"qemu-img convert" is called internally if clonevol is a file volume.
Then, vol->target.format is used as output_fmt parameter but the
target.format of logical volumes is always 0 because logical volumes
haven't the volume format type element.

Fortunately, 0 was treated as RAW file format before commit f772b3d9,
so there was no problem. But now, 0 is treated as the type of none,
qemu-img fails with "Unknown file format 'none'".

This patch fixes this issue by treating output block devices as RAW
file format like for input block devices.
Signed-off-by: NAtsushi Kumagai <kumagai-atsushi@mxc.nes.nec.co.jp>

uid_t and gid_t are opaque types, ranging from s32 to u32 to u64.

Explicitly cast the magic -1 to the appropriate type.
Signed-off-by: NPhilipp Hahn <hahn@univention.de>

uid_t and gid_t are opaque types, ranging from s32 to u32 to u64.

Explicitly cast them to unsigned int for printing.
Signed-off-by: NPhilipp Hahn <hahn@univention.de>

For really old qemu-img binaries which do not support specifying
the format of the backing file, display a DEBUG message instead of
INFO that this can't be done.

Modify the expected output of storagevolxml2argv tests as well.