In preparation for a future patch adding new virFile APIs.

* src/util/files.h, src/util/files.c: Move...
* src/util/virfile.h, src/util/virfile.c: ...here, and rename
functions to virFile prefix.  Macro names are intentionally
left alone.
* *.c: All '#include "files.h"' uses changed.
* src/Makefile.am (UTIL_SOURCES): Reflect rename.
* cfg.mk (exclude_file_name_regexp--sc_prohibit_close): Likewise.
* src/libvirt_private.syms: Likewise.
* docs/hacking.html.in: Likewise.
* HACKING: Regenerate.

* src/storage/storage_backend.c (virStorageBackendCreateBlockFrom)
(virStorageBackendCreateQemuImg)
(virStorageBackendCreateQcowCreate): Reject unknown flags.
* src/storage/storage_backend_disk.c (virStorageBackendDiskBuildPool)
(virStorageBackendDiskDeleteVol): Likewise.
* src/storage/storage_backend_fs.c
(virStorageBackendFileSystemNetFindPoolSources)
(virStorageBackendFileSystemBuild)
(virStorageBackendFileSystemDelete, createFileDir)
(virStorageBackendFileSystemVolBuildFrom)
(virStorageBackendFileSystemVolDelete): Likewise.
* src/storage/storage_backend_iscsi.c
(virStorageBackendISCSIFindPoolSources): Likewise.
* src/storage/storage_backend_logical.c
(virStorageBackendLogicalFindPoolSources)
(virStorageBackendLogicalBuildPool)
(virStorageBackendLogicalDeletePool)
(virStorageBackendLogicalDeleteVol): Likewise.
* src/storage/storage_driver.c (storageOpen, storagePoolCreate)
(storagePoolDefine, storagePoolRefresh, storagePoolGetXMLDesc)
(storageVolumeCreateXML, storageVolumeCreateXMLFrom)
(storageVolumeGetXMLDesc): Likewise.

No caller was using the flags argument, and this function is internal
only, so we might as well skip it.

* src/util/util.h (safezero): Update signature.
* src/util/util.c (safezero): Update function.
* src/locking/lock_driver_sanlock.c
(virLockManagerSanlockSetupLockspace)
(virLockManagerSanlockCreateLease): Update all callers.
* src/storage/storage_backend.c (createRawFile): Likewise.

virStorageBackendCreateRaw: createRawFile already reported the
exact error.

Before the fix:

error: Failed to create vol vol-create.img
error: cannot create path '/var/lib/libvirt/images/vol-create.img': Unknown error 18446744073709551597

After the fix:

error: Failed to create vol vol-create.img
error: cannot fill file '/var/lib/libvirt/images/vol-create.img': No space left on device

Most of the safezero() implementations return -1 on error,
setting errno. The safezero() impl using posix_fallocate()
though returned a positive errno value on error (due to
the unusual API contract of posix_fallocate() compared to
most syscall APIs).

* src/util/util.c: Ensure safezero() returns -1 and sets
  errno on error.
* src/storage/storage_backend.c: Change safezero != 0 to
  < 0 for detecting errors

Suggested here:
https://www.redhat.com/archives/libvir-list/2011-May/msg00594.html

* src/storage/storage_backend.c (virStorageBackendCreateQemuImg):
Generate size inline.

Since directories can be used for <filesystem> passthrough, they are
basically storage volumes.

v2:
    Skip ., .., lost+found dirs

v3:
    Use gnulib last_component

v4:
    Use gnulib "dirname.h", not system <dirname.h>
    Don't skip lost+found

Two additional places need initgroups call to properly work in an
environment where the UID is allowed to open/create stuff through its
supplementary groups.

Commit d7b26792 introduced
a return value picking of qemu-img on '-h', but without any comment.

qemu-img returns non-zero status on -h. Therefore we need to
provide virCommandRun() a non-NULL exit status pointer.

virRunWithHook is now unused, so we can drop it. Tested w/ raw + qcow2
volume creation and copying.

v2:
    Use opaque data to skip hook second time around
    Simply command building

v3:
    Drop explicit FindFileInPath

Some clients overwrite the error from the regex helper, or do half-baked
error reporting with the exitstatus.

v2:
    Simplify command creation
    Add a missing virCommandFree
    Use virCommand auto-cleanup for async process

Jim Meyering recently improved gnulib to catch various grammar
errors during 'make syntax-check'.

* .gnulib: Update to latest, for syntax-check improvements.
* include/libvirt/libvirt.h.in (virConnectAuthCallbackPtr): Use
cannot rather than two words.
* src/driver.c: Likewise.
* src/driver.h (VIR_SECRET_GET_VALUE_INTERNAL_CALL): Likewise.
* src/remote/remote_driver.c (initialize_gnutls): Likewise.
* src/util/pci.c (pciBindDeviceToStub): Likewise.
* src/storage/storage_backend.c (virStorageBackendCreateQemuImg):
Likewise.
(virStorageBackendUpdateVolTargetInfoFD): Avoid doubled word.
* docs/formatdomain.html.in: Likewise.
* src/qemu/qemu_process.c (qemuProcessStart): Likewise.
* cfg.mk (exclude_file_name_regexp--sc_prohibit_can_not)
(exclude_file_name_regexp--sc_prohibit_doubled_word): Exclude
existing translation problems.

This patch enables the relative backing file path support provided by
qemu-img create.

If a relative path is specified for the backing file, it is converted
to an absolute path using the storage pool path. The absolute path is
used to verify that the backing file exists. If the backing file exists,
the relative path is allowed and will be provided to qemu-img create.

* src/storage/storage_backend.c (createRawFileOpHook): Change
signature.
(struct createRawFileOpHookData): Delete unused struct.
(virStorageBackendCreateRaw): Adjust caller.
* src/qemu/qemu_driver.c (struct fileOpHookData): Delete unused
struct.
(qemudDomainSaveFileOpHook): Rename...
(qemuDomainSaveFileOpHook): ...and change signature.
(qemudDomainSaveFlag): Adjust caller.

This patch intentionally doesn't change indentation, in order to
make it easier to review the real changes.

* src/util/util.h (VIR_FILE_OP_RETURN_FD, virFileOperationHook):
Delete.
(virFileOperation): Rename...
(virFileOpenAs): ...and reduce parameters.
* src/util/util.c (virFileOperationNoFork, virFileOperation):
Rename and simplify.
* src/qemu/qemu_driver.c (qemudDomainSaveFlag): Adjust caller.
* src/storage/storage_backend.c (virStorageBackendCreateRaw):
Likewise.
* src/libvirt_private.syms: Reflect rename.

* src/storage/storage_backend.c (virStorageBackendCreateRaw): Use
new virFileOperation flag.

Child processes don't always reach _exit(); if they die from a
signal, then any messages should still be accurate.  Most users
either expect a 0 status (thankfully, if status==0, then
WIFEXITED(status) is true and WEXITSTATUS(status)==0 for all
known platforms) or were filtering on WIFEXITED before printing
a status, but a few were missing this check.  Additionally,
nwfilter_ebiptables_driver was making an assumption that works
on Linux (where WEXITSTATUS shifts and WTERMSIG just masks)
but fails on other platforms (where WEXITSTATUS just masks and
WTERMSIG shifts).

* src/util/command.h (virCommandTranslateStatus): New helper.
* src/libvirt_private.syms (command.h): Export it.
* src/util/command.c (virCommandTranslateStatus): New function.
(virCommandWait): Use it to also diagnose status from signals.
* src/security/security_apparmor.c (load_profile): Likewise.
* src/storage/storage_backend.c
(virStorageBackendQEMUImgBackingFormat): Likewise.
* src/util/util.c (virExecDaemonize, virRunWithHook)
(virFileOperation, virDirCreate): Likewise.
* daemon/remote.c (remoteDispatchAuthPolkit): Likewise.
* src/nwfilter/nwfilter_ebiptables_driver.c (ebiptablesExecCLI):
Likewise.

This patch changes zerobuf variable from array to VIR_ALLOC_N().
Signed-off-by: NMinoru Usui <usui@mxm.nes.nec.co.jp>

For newer qemu-img, the help string for "backing file format" is
"[-F backing_fmt]".

Fix the wrong logic error by commit e997c268.

* src/storage/storage_backend.c

qemu-img silently disable "-e", so we can't use it for volume
encryption anymore, change it into "-o encryption=on" if qemu
supports "-o" option.

virRun gives pretty useful error output, let's not overwrite it unless there
is a good reason. Some places were providing more information about what
the commands were _attempting_ to do, however that's usually less useful from
a debugging POV than what actually happened.

Use it in all places where a memory or storage request size is converted
to a larger granularity. This avoids requesting too small memory or storage
sizes that could result from the truncation done by a simple division.

This extends the round up fix in 6002e040
to the whole codebase.

Instead of reporting errors for odd values in the VMX code round them up.

Update the QEMU Argv tests accordingly as the original memory size 219200
isn't a even multiple of 1024 and is rounded up to 215 megabyte now. Change
it to 219100 and 219136. Use two different values intentionally to make
sure that rounding up works.

Update virsh.pod accordingly, as rounding down and rejecting are replaced
by rounding up.

VIR_ERR_INVALID_* is meant for invalid pointers only.

If there is a dangling symbolic link in filesystem pool, the pool
will fail to start or refresh, this patch is to fix it by ignoring
it with a warning log.

security_context_t happens to be a typedef for char*, and happens to
begin with a string usable as a raw context string.  But in reality,
it is an opaque type that may or may not have additional information
after the first NUL byte, where that additional information can
include pointers that can only be freed via freecon().

Proof is from this valgrind run of daemon/libvirtd:

==6028== 839,169 (40 direct, 839,129 indirect) bytes in 1 blocks are definitely lost in loss record 274 of 274
==6028==    at 0x4A0515D: malloc (vg_replace_malloc.c:195)
==6028==    by 0x3022E0D48C: selabel_open (label.c:165)
==6028==    by 0x3022E11646: matchpathcon_init_prefix (matchpathcon.c:296)
==6028==    by 0x3022E1190D: matchpathcon (matchpathcon.c:317)
==6028==    by 0x4F9D842: SELinuxRestoreSecurityFileLabel (security_selinux.c:382)

800k is a lot of memory to be leaking.

* src/storage/storage_backend.c
(virStorageBackendUpdateVolTargetInfoFD): Avoid leak on error.
* src/security/security_selinux.c
(SELinuxReserveSecurityLabel, SELinuxGetSecurityProcessLabel)
(SELinuxRestoreSecurityFileLabel): Use correct function to free
security_context_t.

Similarly to deprecating close(), I am now deprecating fclose() and
introduce VIR_FORCE_FCLOSE() and VIR_FCLOSE(). Also, fdopen() is replaced with
VIR_FDOPEN().

Most of the files are opened in read-only mode, so usage of
VIR_FORCE_CLOSE() seemed appropriate. Others that are opened in write
mode already had the fclose()<  0 check and I converted those to
VIR_FCLOSE()<  0.

I did not find occurrences of possible double-closed files on the way.

Using automated replacement with sed and editing I have now replaced all
occurrences of close() with VIR_(FORCE_)CLOSE() except for one, of
course. Some replacements were straight forward, others I needed to pay
attention. I hope I payed attention in all the right places... Please
have a look. This should have at least solved one more double-close
error.

* src/storage/storage_backend.c (virStorageBackendCreateQemuImg)
(virStorageBackendCreateQcowCreate): Use virAsprintf instead.
* src/storage/storage_backend_disk.c
(virStorageBackendDiskCreateVol, virStorageBackendDiskPartFormat):
Likewise.

Found by clang.  Clang complained that virStorageBackendProbeTarget
could dereference NULL if backingStoreFormat was NULL, but since all
callers passed a valid pointer, I added attributes instead of null
checks.

* src/storage/storage_backend.c
(virStorageBackendQEMUImgBackingFormat): Kill dead store.
* src/storage/storage_backend_fs.c (virStorageBackendProbeTarget):
Likewise.  Skip null checks, by adding attributes.

One error exit in virStorageBackendCreateBlockFrom was setting the
return value to errno. The convention for volume build functions is to
return 0 on success or -1 on failure. Not only was it not necessary to
set the return value (it defaults to -1, and is set to 0 when
everything has been successfully completed), in the case that some
caller were checking for < 0 rather than != 0, they would incorrectly
believe that it completed successfully.

Previously virStorageBackendCopyToFD would simply return -1 on
error. This made the error return from one of its callers inconsistent
(createRawFileOpHook is supposed to return -errno, but if
virStorageBackendCopyToFD failed, createRawFileOpHook would just
return -1). Since there is a useful errno in every case of error
return from virStorageBackendCopyToFD, and since the other uses of
that function ignore the return code (beyond simply checking to see if
it is < 0), this is a safe change.

virFileOperation previously returned 0 on success, or the value of
errno on failure. Although there are other functions in libvirt that
use this convention, the preferred (and more common) convention is to
return 0 on success and -errno (or simply -1 in some cases) on
failure. This way the check for failure is always (ret < 0).

* src/util/util.c - change virFileOperation and virFileOperationNoFork to
                    return -errno on failure.

* src/storage/storage_backend.c, src/qemu/qemu_driver.c
  - change the hook functions passed to virFileOperation to return
    -errno on failure.

Originally the storage volume files were opened with O_DSYNC to make
sure they were flushed to disk immediately. It turned out that this
was extremely slow in some cases, so the O_DSYNC was removed in favor
of just calling fsync() after all the data had been written. However,
this call to fsync was inside the block that is executed to zero-fill
the end of the volume file. In cases where the new volume is copied
from an old volume, and they are the same length, this fsync would
never take place.

Now the fsync is *always* done, unless there is an error (in which
case it isn't important, and is most likely inappropriate.

A missing set of braces around an error condition caused us to skip
zero'ing out the remainder of a new volume file if the new volume was
longer than the original (the goto was supposed to be taken only in
the case of error, but was always being taken).

When creating qcow2 files with a backing store, it is important
to set an explicit format to prevent QEMU probing. The storage
backend was only doing this if it found a 'kvm-img' binary. This
is wrong because plenty of kvm-img binaries don't support an
explicit format, and plenty of 'qemu-img' binaries do support
a format. The result was that most qcow2 files were not getting
a backing store format.

This patch runs 'qemu-img -h' to check for the two support
argument formats

  '-o backing_format=raw'
  '-F raw'

and use whichever option it finds

* src/storage/storage_backend.c: Query binary to determine
  how to set the backing store format

If a directory pool contains pipes or sockets, a pool start can fail or hang:

https://bugzilla.redhat.com/show_bug.cgi?id=589577

We already try to avoid these special files, but only attempt after
opening the path, which is where the problems lie. Unify volume opening
into helper functions, which use the proper open() flags to avoid error,
followed by fstat to validate storage mode.

Previously, virStorageBackendUpdateVolTargetInfoFD attempted to enforce the
storage mode check, but allowed callers to detect this case and silently
continue. In practice, only the FS backend was using this feature, the rest
were treating unknown mode as an error condition. Unfortunately the InfoFD
function wasn't raising an error message here, so error reporting was
busted.

This patch adds 2 functions: virStorageBackendVolOpen, and
virStorageBackendVolOpenModeSkip. The latter retains the original opt out
semantics, the former now throws an explicit error.

This patch maintains the previous volume mode checks: allowing specific
modes for specific pool types requires a bit of surgery, since VolOpen
is called through several different helper functions.

v2: Use ATTRIBUTE_NONNULL. Drop stat check, just open with
    O_NONBLOCK|O_NOCTTY.

v3: Move mode check logic back to VolOpen. Use 2 VolOpen functions with
    different error semantics.

v4: Make second VolOpen function more extensible. Didn't opt to change
    FS backend defaults, this can just be to fix the original bug.

v5: Prefix default flags with VIR_, use ATTRIBUTE_RETURN_CHECK

Volume detection in the scsi backend was duplicating code already
present in storage_backend.c. Let's drop the duplicate code.

Also, change the shared function name to be less generic, and remove
some error squashing in the other call site.