This adds several tests for remaining hash APIs (custom
hasher/comparator functions are not covered yet, though).

All tests pass both before and after the "Simplify hash implementation".

When buf->error is 1, we do not return buf->content in the function
virBufferContentAndReset(). So we should free buf->content when
vsnprintf() failed.

Suggested by Daniel P. Berrange

* .gnulib: Update to latest for passfd fixes.
* bootstrap.conf (gnulib_modules): Add passfd.
* src/util/util.c (virFileOpenAs): Simplify.

We can exploit the fact that gcc warns about int-to-pointer conversion
in ternary cond?(void*):(int) in order to prevent future mistakes of
calling VIR_FREE on a scalar lvalue.  For example, between commits
158ba873 and 802e2df9, we would have had this warning:

cc1: warnings being treated as errors
remote.c: In function 'remoteDispatchListNetworks':
remote.c:3684:70: error: pointer/integer type mismatch in conditional expression

There are still a number of places that malloc into a const char*;
while it would probably be worth scrubbing them to use char*
instead, that is a separate patch, so we have to cast away const
in VIR_FREE for now.

* src/util/memory.h (VIR_FREE): Make gcc warn about integers.
Iteratively developed from a patch by Christophe Fergeau.

mingw lacks the counterpart to PTHREAD_MUTEX_INITIALIZER, so the
best we can do is portably expose once-only runtime initialization.

* src/util/threads.h (virOnceControlPtr): New opaque type.
(virOnceFunc): New callback type.
(virOnce): New prototype.
* src/util/threads-pthread.h (virOnceControl): Declare.
(VIR_ONCE_CONTROL_INITIALIZER): Define.
* src/util/threads-win32.h (virOnceControl)
(VIR_ONCE_CONTROL_INITIALIZER): Likewise.
* src/util/threads-pthread.c (virOnce): Implement in pthreads.
* src/util/threads-win32.c (virOnce): Implement in WIN32.
* src/libvirt_private.syms: Export it.

Inspired by Eric Blake

And from all related macros and functions.

It costs quite a few processor cycles to go through printf parsing
just to determine that we only meant to append.

* src/xen/xend_internal.c (xend_op_ext): Consolidate multiple
printfs into one.
* src/qemu/qemu_command.c (qemuBuildWatchdogDevStr)
(qemuBuildUSBInputDevStr, qemuBuildSoundDevStr)
(qemuBuildSoundCodecStr, qemuBuildVideoDevStr): Likewise.
(qemuBuildCpuArgStr, qemuBuildCommandLine): Prefer virBufferAdd
over virBufferVsprintf for trivial appends.
* src/phyp/phyp_driver.c (phypExec, phypUUIDTable_Push)
(phypUUIDTable_Pull): Likewise.
* src/conf/nwfilter_conf.c (macProtocolIDFormatter)
(arpOpcodeFormatter, formatIPProtocolID, printStringItems)
(virNWFilterPrintStateMatchFlags, virNWIPAddressFormat)
(virNWFilterDefFormat): Likewise.
* src/security/virt-aa-helper.c (main): Likewise.
* src/util/sexpr.c (sexpr2string): Likewise.
* src/xenxs/xen_sxpr.c (xenFormatSxprChr): Likewise.
* src/xenxs/xen_xm.c (xenFormatXMDisk): Likewise.

gcc 4.6 warns when a variable is initialized but isn't used afterwards:

vmware/vmware_driver.c:449:18: warning: variable 'vmxPath' set but not used [-Wunused-but-set-variable]

This patch fixes these warnings. There are still 2 offending files:

- vbox_tmpl.c: the variable is used inside an #ifdef and is assigned several
  times outside of #ifdef. Fixing the warning would have required wrapping
  all the assignment inside #ifdef which hurts readability.

vbox/vbox_tmpl.c: In function 'vboxAttachDrives':
vbox/vbox_tmpl.c:3918:22: warning: variable 'accessMode' set but not used [-Wunused-but-set-variable]

- esx_vi_types.generated.c: the name implies it's generated code and I
  didn't want to dive into the code generator

esx/esx_vi_types.generated.c: In function 'esxVI_FileQueryFlags_Free':
esx/esx_vi_types.generated.c:1203:3: warning: variable 'item' set but not used [-Wunused-but-set-variable]

Commit 9677cd33 made it possible to
remove current entry when iterating through all hash entries. However,
it didn't properly handle a special case of removing first entry
assigned to a given key which contains several entries in its collision
list.

Jim Meyering recently improved gnulib to catch various grammar
errors during 'make syntax-check'.

* .gnulib: Update to latest, for syntax-check improvements.
* include/libvirt/libvirt.h.in (virConnectAuthCallbackPtr): Use
cannot rather than two words.
* src/driver.c: Likewise.
* src/driver.h (VIR_SECRET_GET_VALUE_INTERNAL_CALL): Likewise.
* src/remote/remote_driver.c (initialize_gnutls): Likewise.
* src/util/pci.c (pciBindDeviceToStub): Likewise.
* src/storage/storage_backend.c (virStorageBackendCreateQemuImg):
Likewise.
(virStorageBackendUpdateVolTargetInfoFD): Avoid doubled word.
* docs/formatdomain.html.in: Likewise.
* src/qemu/qemu_process.c (qemuProcessStart): Likewise.
* cfg.mk (exclude_file_name_regexp--sc_prohibit_can_not)
(exclude_file_name_regexp--sc_prohibit_doubled_word): Exclude
existing translation problems.

* .gnulib: Update to latest, for pipe2.
* bootstrap.conf (gnulib_modules): Add pipe2.
* src/util/event_poll.c (virEventPollInit): Use it, to avoid
problematic virSetCloseExec on mingw.

We should bind pci device to original driver when pciBindDeviceToStub() failed.
If the pci device is not bound to any driver before calling pciBindDeviceToStub(),
we should only unbind it from pci-stub. If it is bound to pci-stub, we should not
unbind it from pci-stub.

This patch do the following things:
1. rename the function as 'Unbind' is better than 'UnBind'.
2. pciUnbindDeviceFromStub() will be used in the function pciBindDeviceToStub() in
   next patch. Float it up, instead of having to have a forward declaration

This bug was introduce by commit 57162db8, and it will cause libvirtd crashed.

In file included from util/threads.c:31:
util/threads-pthread.c: In function 'virThreadSelfID':
util/threads-pthread.c:214: warning: cast from function call of type 'pthread_t' to non-matching type 'int' [-Wbad-function-cast]

* src/util/threads-pthread.c (virThreadSelfID) [!SYS_gettid]:
Add intermediate cast to silence gcc.

GCC is a little confused about the cast of beginthread/beginthreadex
from unsigned long -> void *. Go via an intermediate variable avoids
the bogus warning, and makes the code a little cleaner

* src/util/threads-win32.c: Avoid compiler warning in cast

Removes 4kb stack allocation in the XenD subdriver.

Use virAsprintf instead of snprintf.

Make virFileBuildPath operate on the heap instead of the stack. It
allocates a buffer instead of expecting a preexisting buffer.

Even with -Wuninitialized (which is part of autobuild.sh
--enable-compile-warnings=error), gcc does NOT catch this
use of an uninitialized variable:

{
  if (cond)
    goto error;
  int a = 1;
error:
  printf("%d", a);
}

which prints 0 (supposing the stack started life wiped) if
cond was true.  Clang will catch it, but we don't use clang
as often.  Using gcc -Wjump-misses-init catches it, but also
gives false positives:

{
  if (cond)
    goto error;
  int a = 1;
  return a;
error:
  return 0;
}

Here, a was never used in the scope of the error block, so
declaring it after goto is technically fine (and clang agrees).
However, given that our HACKING already documents a preference
to C89 decl-before-statement, the false positive warning is
enough of a prod to comply with HACKING.

[Personally, I'd _really_ rather use C99 decl-after-statement
to minimize scope, but until gcc can efficiently and reliably
catch scoping and uninitialized usage bugs, I'll settle with
the compromise of enforcing a coding standard that happens to
reject false positives if it can also detect real bugs.]

* acinclude.m4 (LIBVIRT_COMPILE_WARNINGS): Add -Wjump-misses-init.
* src/util/util.c (__virExec): Adjust offenders.
* src/conf/domain_conf.c (virDomainTimerDefParseXML): Likewise.
* src/remote/remote_driver.c (doRemoteOpen): Likewise.
* src/phyp/phyp_driver.c (phypGetLparNAME, phypGetLparProfile)
(phypGetVIOSFreeSCSIAdapter, phypVolumeGetKey)
(phypGetStoragePoolDevice)
(phypVolumeGetPhysicalVolumeByStoragePool)
(phypVolumeGetPath): Likewise.
* src/vbox/vbox_tmpl.c (vboxNetworkUndefineDestroy)
(vboxNetworkCreate, vboxNetworkDumpXML)
(vboxNetworkDefineCreateXML): Likewise.
* src/xenapi/xenapi_driver.c (getCapsObject)
(xenapiDomainDumpXML): Likewise.
* src/xenapi/xenapi_utils.c (createVMRecordFromXml): Likewise.
* src/security/security_selinux.c (SELinuxGenNewContext):
Likewise.
* src/qemu/qemu_command.c (qemuBuildCommandLine): Likewise.
* src/qemu/qemu_hotplug.c (qemuDomainChangeEjectableMedia):
Likewise.
* src/qemu/qemu_process.c (qemuProcessWaitForMonitor): Likewise.
* src/qemu/qemu_monitor_text.c (qemuMonitorTextGetPtyPaths):
Likewise.
* src/qemu/qemu_driver.c (qemudDomainShutdown)
(qemudDomainBlockStats, qemudDomainMemoryPeek): Likewise.
* src/storage/storage_backend_iscsi.c
(virStorageBackendCreateIfaceIQN): Likewise.
* src/node_device/node_device_udev.c (udevProcessPCI): Likewise.

* .gnulib: Update to latest, for nonblocking module.
* bootstrap.conf (gnulib_modules): Add nonblocking.
* src/util/util.c (virSetBlocking): Defer to gnulib.

qemu driver uses a 4K buffer for reading qemu log file. This is enough
when only qemu's output is present in the log file. However, when
debugging messages are turned on, intermediate libvirt process fills the
log with a bunch of debugging messages before it executes qemu binary.
In such a case the buffer may become too small. However, we are not
really interested in libvirt messages so they can be filtered out from
the buffer.

New APIs are added allowing streaming of content to/from
storage volumes.

* include/libvirt/libvirt.h.in: Add virStorageVolUpload and
  virStorageVolDownload APIs
* src/driver.h, src/libvirt.c, src/libvirt_public.syms: Stub
  code for new APIs
* src/storage/storage_driver.c, src/esx/esx_storage_driver.c:
  Add dummy entries in driver table for new APIs

The O_NONBLOCK flag doesn't work as desired on plain files
or block devices. Introduce an I/O helper program that does
the blocking I/O operations, communicating over a pipe that
can support O_NONBLOCK

* src/fdstream.c, src/fdstream.h: Add non-blocking I/O
  on plain files/block devices
* src/Makefile.am, src/util/iohelper.c: I/O helper program
* src/qemu/qemu_driver.c, src/lxc/lxc_driver.c,
  src/uml/uml_driver.c, src/xen/xen_driver.c: Update for
  streams API change

* src/util/command.c (virCommandAbort) [WIN32]: Provide stub.
Reported by Daniel P. Berrange's autobuilder.

Separating the indentation from the real patch made review easier.

* src/util/util.c (virFileOpenAs): Whitespace changes.

This patch intentionally doesn't change indentation, in order to
make it easier to review the real changes.

* src/util/util.h (VIR_FILE_OP_RETURN_FD, virFileOperationHook):
Delete.
(virFileOperation): Rename...
(virFileOpenAs): ...and reduce parameters.
* src/util/util.c (virFileOperationNoFork, virFileOperation):
Rename and simplify.
* src/qemu/qemu_driver.c (qemudDomainSaveFlag): Adjust caller.
* src/storage/storage_backend.c (virStorageBackendCreateRaw):
Likewise.
* src/libvirt_private.syms: Reflect rename.

Currently, the hook function in virFileOperation is extremely limited:
it must be async-signal-safe, and cannot modify any memory in the
parent process.  It is much handier to return a valid fd and operate
on it in the parent than to deal with hook restrictions.

* src/util/util.h (VIR_FILE_OP_RETURN_FD): New flag.
* src/util/util.c (virFileOperationNoFork, virFileOperation):
Honor new flag.

* src/util/util.h (virSetInherit): New prototype.
* src/util/util.c (virSetCloseExec): Move guts...
(virSetInherit): ...to new function, and allow clearing.
* src/libvirt_private.syms (util.h): Export it.

* src/util/logging.c (virLogStartup, virLogSetBufferSize):
Over-allocate, so that a debugger can just print the circular
buffer.  Suggested by Daniel Veillard.

* src/util/hooks.c (virHookCheck): Missing hooks should just be
debug, not warn.

Sometimes, an asynchronous helper is started (such as a compressor
or iohelper program), but a later error means that we want to
abort that child.  Make this easier.

Note that since daemons and virCommandRunAsync can't mix, the only
time virCommandFree can reap a process is if someone did
virCommandRunAsync for a non-daemon and didn't stash the pid.

* src/util/command.h (virCommandAbort): New prototype.
* src/util/command.c (_virCommand): Add new field.
(virCommandRunAsync, virCommandWait): Track whether pid was used.
(virCommandFree): Reap child if caller did not request pid.
(virCommandAbort): New function.
* src/libvirt_private.syms (command.h): Export it.
* tests/commandtest.c (test19): New test.

It doesn't make sense to run a daemon without synchronously
waiting for the child process to reply whether the daemon has
been kicked off and pidfile written yet.

* src/util/command.c (VIR_EXEC_RUN_SYNC): New constant.
(virCommandRun): Set temporary flag.
(virCommandRunAsync): Use it to prevent async runs of intermediate
child when spawning asynchronous daemon grandchild.

Child processes don't always reach _exit(); if they die from a
signal, then any messages should still be accurate.  Most users
either expect a 0 status (thankfully, if status==0, then
WIFEXITED(status) is true and WEXITSTATUS(status)==0 for all
known platforms) or were filtering on WIFEXITED before printing
a status, but a few were missing this check.  Additionally,
nwfilter_ebiptables_driver was making an assumption that works
on Linux (where WEXITSTATUS shifts and WTERMSIG just masks)
but fails on other platforms (where WEXITSTATUS just masks and
WTERMSIG shifts).

* src/util/command.h (virCommandTranslateStatus): New helper.
* src/libvirt_private.syms (command.h): Export it.
* src/util/command.c (virCommandTranslateStatus): New function.
(virCommandWait): Use it to also diagnose status from signals.
* src/security/security_apparmor.c (load_profile): Likewise.
* src/storage/storage_backend.c
(virStorageBackendQEMUImgBackingFormat): Likewise.
* src/util/util.c (virExecDaemonize, virRunWithHook)
(virFileOperation, virDirCreate): Likewise.
* daemon/remote.c (remoteDispatchAuthPolkit): Likewise.
* src/nwfilter/nwfilter_ebiptables_driver.c (ebiptablesExecCLI):
Likewise.

This simplifies several callers that were repeating checks already
guaranteed by util.c, and makes other callers more robust to now
reject directories.  remote_driver.c was over-strict - access(,R_OK)
is only needed to execute a script file; a binary only needs
access(,X_OK) (besides, it's unusual to see a file with x but not
r permissions, whether script or binary).

* cfg.mk (sc_prohibit_access_xok): New syntax-check rule.
(exclude_file_name_regexp--sc_prohibit_access_xok): Exempt one use.
* src/network/bridge_driver.c (networkStartRadvd): Fix offenders.
* src/qemu/qemu_capabilities.c (qemuCapsProbeMachineTypes)
(qemuCapsInitGuest, qemuCapsInit, qemuCapsExtractVersionInfo):
Likewise.
* src/remote/remote_driver.c (remoteFindDaemonPath): Likewise.
* src/uml/uml_driver.c (umlStartVMDaemon): Likewise.
* src/util/hooks.c (virHookCheck): Likewise.

* src/util/command.c (virCommandRunAsync): Since virExec only
creates pidfiles for daemon, enforce this in virCommand.