Wen Congyang reported that we have a double-close bug if we fail
virFDStreamOpenInternal, since childfd duplicated one of the fds[]
array contents.  In truth, since we always transfer both members
of fds to other variables, we should close the fds through those
other names, and just use fds[] for pipe().

Bug present since 0.9.0 (commit e886237a).

* src/fdstream.c (virFDStreamOpenFileInternal): Swap scope of
childfd and fds[], to avoid a double close.
(cherry picked from commit f3cfc7c8)

virCommandRunAsync() will set errfd if it succeed. We should
close it if virFDStreamOpenInternal() fails.
(cherry picked from commit 655cffa0)

This patch adds another callback to a FDstream object. The original
callback is used by the daemon stream driver to handle events.

This callback is called if and only if the stream is about to be closed.
This might be used to handle cleanup steps after a fdstream exits. This
will be used later on in ensuring mutually exclusive access to consoles.

* src/fdstream.c:
        - emit the callback, when stream is being closed
        - add data structures needed to handle the callback
        - add function to register callback
* src/fdstream.h:
        - define function prototypes for the callback
(cherry picked from commit 0c4bfdda)

This patch causes the fdstream driver to call the stream event callback
if virStreamAbort() is called on a stream using this driver.

A remote handler for a stream can only detect changes via stream events,
so this event callback is necessary in order to enable a daemon to abort
a stream in such a way that the client will see the change.

* src/fdstream.c:
        - modify close function to call stream event callback
(cherry picked from commit 95fdc1bc)

Revert 6a1f5f56.  Now that libvirt_iohelper takes fds by
inheritance rather than by open() (commit 1eb66479), there is
no longer a race where the parent can unlink() a file prior to
the iohelper open()ing the same file.  From there, it makes
more sense to have the callers both create and unlink, rather
than the caller create and the stream unlink, since the latter
was only needed when iohelper had to do the unlink.

* src/fdstream.h (virFDStreamOpenFile, virFDStreamCreateFile):
Callers are responsible for deletion.
* src/fdstream.c (virFDStreamOpenFileInternal): Don't leak created
file on failure.
(virFDStreamOpenFile, virFDStreamCreateFile): Drop parameter.
* src/lxc/lxc_driver.c (lxcDomainOpenConsole): Update callers.
* src/qemu/qemu_driver.c (qemuDomainScreenshot)
(qemuDomainOpenConsole): Likewise.
* src/storage/storage_driver.c (storageVolumeDownload)
(storageVolumeUpload): Likewise.
* src/uml/uml_driver.c (umlDomainOpenConsole): Likewise.
* src/vbox/vbox_tmpl.c (vboxDomainScreenshot): Likewise.
* src/xen/xen_driver.c (xenUnifiedDomainOpenConsole): Likewise.

Rather than making the iohelper subject to a race in reopening
the file, it is nicer to pass an already-open fd by inheritance.

The old synopsis form must continue to work - if someone updates
their libvirt package and installs a new libvirt_iohelper but
without restarting the old libvirtd daemon, then the daemon can
still make calls using the old syntax but the new iohelper.

* src/util/iohelper.c (runIO): Split code for open...
(prepare): ...to new function.
(usage): Update synopsis.
(main): Allow alternate calling form.
* src/fdstream.c (virFDStreamOpenFileInternal): Use alternate form.

In preparation for a future patch adding new virFile APIs.

* src/util/files.h, src/util/files.c: Move...
* src/util/virfile.h, src/util/virfile.c: ...here, and rename
functions to virFile prefix.  Macro names are intentionally
left alone.
* *.c: All '#include "files.h"' uses changed.
* src/Makefile.am (UTIL_SOURCES): Reflect rename.
* cfg.mk (exclude_file_name_regexp--sc_prohibit_close): Likewise.
* src/libvirt_private.syms: Likewise.
* docs/hacking.html.in: Likewise.
* HACKING: Regenerate.

Enforce the recent flags cleanups - we want to use 'unsigned int flags'
in any of our APIs (except where backwards compatibility is important,
in the public migration APIs), and that all flags are checked for
validity (except when there are stub functions that completely
ignore the flags argument).

There are a few minor tweaks done here to avoid false positives:
signed arguments passed to open() are renamed oflags, and flags
arguments that are legitimately ignored are renamed flags_unused.

* cfg.mk (sc_flags_usage): New rule.
(exclude_file_name_regexp--sc_flags_usage): And a few exemptions.
(sc_flags_debug): Tweak wording.
* src/util/iohelper.c (runIO, main): Rename variable.
* src/util/util.c (virSetInherit): Likewise.
* src/fdstream.h (virFDStreamOpenFile, virFDStreamCreateFile):
Likewise.
* src/fdstream.c (virFDStreamOpenFileInternal)
(virFDStreamOpenFile, virFDStreamCreateFile): Likewise.
* src/util/command.c (virExecWithHook) [WIN32]: Likewise.
* src/util/util.c (virFileOpenAs, virDirCreate) [WIN32]: Likewise.
* src/locking/lock_manager.c (virLockManagerPluginNew)
[!HAVE_DLFCN_H]: Likewise.
* src/locking/lock_driver_nop.c (virLockManagerNopNew)
(virLockManagerNopAddResource, virLockManagerNopAcquire)
(virLockManagerNopRelease, virLockManagerNopInquire): Likewise.

By requesting the pid in virCommandRunAsync, fdstream was claiming
that it would manually wait for the process.  But on the failure
path, the child process was being leaked.

* src/fdstream.c (virFDStreamOpenFileInternal): Auto-reap child.

Continuation of commit 313ac7fd, and enforce things with a syntax
check.

Technically, virNetServerClientCalculateHandleMode is not printing
a mode_t, but rather a collection of VIR_EVENT_HANDLE_* bits;
however, these bits are < 8, so there is no different in the
output, and that was the easiest way to silence the new syntax check.

* cfg.mk (sc_flags_debug): New syntax check.
(exclude_file_name_regexp--sc_flags_debug): Add exemptions.
* src/fdstream.c (virFDStreamOpenFileInternal): Print flags in
hex, mode_t in octal.
* src/libvirt-qemu.c (virDomainQemuMonitorCommand)
(virDomainQemuAttach): Likewise.
* src/locking/lock_driver_nop.c (virLockManagerNopInit): Likewise.
* src/locking/lock_driver_sanlock.c (virLockManagerSanlockInit):
Likewise.
* src/locking/lock_manager.c: Likewise.
* src/qemu/qemu_migration.c: Likewise.
* src/qemu/qemu_monitor.c: Likewise.
* src/rpc/virnetserverclient.c
(virNetServerClientCalculateHandleMode): Print mode with %o.

When adding a callback to an FD stream, we take an extra reference
on the virStreamPtr instance. We forgot to registered a free function
with the callback, so when the callback was removed, the extra
reference held on virStreamPtr was not released.

* src/fdstream.c: Use a free callback to release reference on
  virStreamPtr when removing callback

If a callback being invoked from a stream issues a virStreamAbort
operation, the stream data will be free'd but the callback will
then still try to use this. Delay free'ing of the stream data when
a callback is dispatching

* src/fdstream.c: Delay stream free when callback is active

Since we virEventRegisterDefaultImpl is now a public API, callers need
a way to invoke the default registered Handle and Timeout functions. We
already have general functions for these internally, so promote
them to the public API.

v2:
    Actually add APIs to libvirt.h

virFDStreamClose used a mutex after it was freed, and failed
to destroy that mutex on its last use.

* src/fdstream.c (virFDStreamFree): Inline into sole caller...
(virFDStreamClose): ...to avoid use-after-free and leak.
Reported by Matthias Bolte.

This is needed if we want to transfer a temporary file. If the
transfer is done with iohelper, we might run into a race condition,
where we unlink() file before iohelper is executed.

* src/fdstream.c, src/fdstream.h,
  src/util/iohelper.c: Add new option
* src/lxc/lxc_driver.c, src/qemu/qemu_driver.c,
  src/storage/storage_driver.c, src/uml/uml_driver.c,
  src/xen/xen_driver.c: Expand existing function calls

And from all related macros and functions.

The O_NONBLOCK flag doesn't work as desired on plain files
or block devices. Introduce an I/O helper program that does
the blocking I/O operations, communicating over a pipe that
can support O_NONBLOCK

* src/fdstream.c, src/fdstream.h: Add non-blocking I/O
  on plain files/block devices
* src/Makefile.am, src/util/iohelper.c: I/O helper program
* src/qemu/qemu_driver.c, src/lxc/lxc_driver.c,
  src/uml/uml_driver.c, src/xen/xen_driver.c: Update for
  streams API change

* src/fdstream.c (virFDStreamOpenFile, virFDStreamCreateFile):
Use VIR_FORCE_CLOSE instead of close.
* tests/commandtest.c (mymain): Likewise.
* tools/virsh.c (editFile): Use virCommand instead of system.
* src/util/util.c (__virExec): Special case preservation of std
file descriptors to child.

To avoid the need for duplicating implementations of virStream
drivers, provide a generic implementation that can handle any
FD based stream. This code is copied from the existing impl
in the QEMU driver, with the locking moved into the stream
impl, and addition of a read callback

The FD stream code will refuse to operate on regular files or
block devices, since those can't report EAGAIN properly when
they would block on I/O

* include/libvirt/virterror.h, include/libvirt/virterror.h: Add
  VIR_FROM_STREAM error domain
* src/qemu/qemu_driver.c: Remove code obsoleted by the new
  generic streams driver.
* src/fdstream.h, src/fdstream.c, src/fdstream.c,
  src/libvirt_private.syms: Generic reusable FD based streams