In a few places in libvirt we busy-wait for events, for example qemu
creating a monitor socket.  This is problematic because:

 - We need to choose a sufficiently small polling period so that
   libvirt doesn't add unnecessary delays.

 - We need to choose a sufficiently large polling period so that
   the effect of busy-waiting doesn't affect the system.

The solution to this conflict is to use an exponential backoff.

This patch adds two functions to hide the details, and modifies a few
places where we currently busy-wait.
Signed-off-by: NRichard W.M. Jones <rjones@redhat.com>
(cherry picked from commit beaa447a)

fdstream.c: In function 'virFDStreamWrite':
fdstream.c:390:29: error: logical 'or' of equal expressions [-Werror=logical-op]
        if (errno == EAGAIN || errno == EWOULDBLOCK) {
                            ^~

Fedora rawhide now uses gcc 6.0 and there is a bug with -Wlogical-op
producing false warnings.

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=69602

Use GCC pragma push/pop and ignore -Wlogical-op for GCC that supports
push/pop pragma and also has this bug.
Signed-off-by: NPavel Hrdina <phrdina@redhat.com>
(cherry picked from commit d713a6b1)

Some lines in this file are misaligned which fires up my OCD.
Signed-off-by: NMichal Privoznik <mprivozn@redhat.com>

According to the POSIX standard, off_t (returned by lseek) is defined as
signed integral type no shorter than int. Because our offset variable is defined
as unsigned long long, the original check was passed successfully if UINT64_MAX had
been used as offset value, due to implicit conversion.

Resolves: https://bugzilla.redhat.com/show_bug.cgi?id=1177219

Not all files we want to find using virFileFindResource{,Full} are
generated when libvirt is built, some of them (such as RNG schemas) are
distributed with sources. The current API was not able to find source
files if libvirt was built in VPATH.

Both RNG schemas and cpu_map.xml are distributed in source tarball.
Signed-off-by: NJiri Denemark <jdenemar@redhat.com>

Since virStreamFree will call virObjectUnref anyway, let's just use that
directly so as to avoid the possibility that we inadvertently clear out
a pending error message when using the public API.

The commit "f5b4c141" introduced new "force" parameter
for "virFDStreamOpenFileInternal" but forget to update
one call of that function.
Signed-off-by: NPavel Hrdina <phrdina@redhat.com>

virStorageBackendVolDownloadLocal and virStorageBackendVolUploadLocal
use virFDStreamOpenFile function to work with the volume fd.

virFDStreamOpenFile calls virFDStreamOpenFileInternal that implements
handling of the non-blocking I/O. If a file is not a character device and
not a fifo, it uses libvirt_iohelper.

On FreeBSD, it doesn't work as expected because disk devices (including
ZFS volumes) are exposed as character devices, and ZFS volumes do not
support open(2) with O_NONBLOCK.

To overcome this, introduce a forceIOHelper flag to
virFDStreamOpenFileInternal that forces using libvirt_iohelper. And
introduce virFDStreamOpenBlockDevice that calls
virFDStreamOpenFileInternal with the forceIOHelper set to true.

virFDStreamOpenInternal terminates if virSetNonBlock fails. As
virSetNonBlock uses gnulib's set_nonblocking_flag that sets errno,
call virReportSystemError() to let user know the reason of fail.

Instead of hardcoding LIBEXECDIR as the location of the libvirt_iohelper
binary, use virFileFindResource to optionally find it in the current
build directory.
Signed-off-by: NDaniel P. Berrange <berrange@redhat.com>

Add virFDStreamOpenPTY() function which is a wrapper around
virFDStreamOpenFileInternal() with putting the device it opens into a
raw mode.

Make virChrdevOpen() use virFDStreamOpenPTY() for
VIR_DOMAIN_CHR_TYPE_PTY devices.

This fixes mangled console output when libvirt runs on FreeBSD as it
requires device it opens to be placed into a raw mode explicitly.

Any source file which calls the logging APIs now needs
to have a VIR_LOG_INIT("source.name") declaration at
the start of the file. This provides a static variable
of the virLogSource type.
Signed-off-by: NDaniel P. Berrange <berrange@redhat.com>

Auditing all callers of virCommandRun and virCommandWait that
passed a non-NULL pointer for exit status turned up some
interesting observations.  Many callers were merely passing
a pointer to avoid the overall command dying, but without
caring what the exit status was - but these callers would
be better off treating a child death by signal as an abnormal
exit.  Other callers were actually acting on the status, but
not all of them remembered to filter by WIFEXITED and convert
with WEXITSTATUS; depending on the platform, this can result
in a status being reported as 256 times too big.  And among
those that correctly parse the output, it gets rather verbose.
Finally, there were the callers that explicitly checked that
the status was 0, and gave their own message, but with fewer
details than what virCommand gives for free.

So the best idea is to move the complexity out of callers and
into virCommand - by default, we return the actual exit status
already cleaned through WEXITSTATUS and treat signals as a
failed command; but the few callers that care can ask for raw
status and act on it themselves.

* src/util/vircommand.h (virCommandRawStatus): New prototype.
* src/libvirt_private.syms (util/command.h): Export it.
* docs/internals/command.html.in: Document it.
* src/util/vircommand.c (virCommandRawStatus): New function.
(virCommandWait): Adjust semantics.
* tests/commandtest.c (test1): Test it.
* daemon/remote.c (remoteDispatchAuthPolkit): Adjust callers.
* src/access/viraccessdriverpolkit.c (virAccessDriverPolkitCheck):
Likewise.
* src/fdstream.c (virFDStreamCloseInt): Likewise.
* src/lxc/lxc_process.c (virLXCProcessStart): Likewise.
* src/qemu/qemu_command.c (qemuCreateInBridgePortWithHelper):
Likewise.
* src/xen/xen_driver.c (xenUnifiedXendProbe): Simplify.
* tests/reconnect.c (mymain): Likewise.
* tests/statstest.c (mymain): Likewise.
* src/bhyve/bhyve_process.c (virBhyveProcessStart)
(virBhyveProcessStop): Don't overwrite virCommand error.
* src/libvirt.c (virConnectAuthGainPolkit): Likewise.
* src/openvz/openvz_driver.c (openvzDomainGetBarrierLimit)
(openvzDomainSetBarrierLimit): Likewise.
* src/util/virebtables.c (virEbTablesOnceInit): Likewise.
* src/util/viriptables.c (virIpTablesOnceInit): Likewise.
* src/util/virnetdevveth.c (virNetDevVethCreate): Fix debug
message.
* src/qemu/qemu_capabilities.c (virQEMUCapsInitQMP): Add comment.
* src/storage/storage_backend_iscsi.c
(virStorageBackendISCSINodeUpdate): Likewise.
Signed-off-by: NEric Blake <eblake@redhat.com>

On win32, using text mode for binary files might result in short
reads since ASCII character 0x1A is interpreted as EOF. Also, it
could lead to problems using the seek functions because of the \r
handling.
Signed-off-by: NClaudio Bley <cbley@av-test.de>

Merge the virCommandPreserveFD / virCommandTransferFD methods
into a single virCommandPasFD method, and use a new
VIR_COMMAND_PASS_FD_CLOSE_PARENT to indicate their difference
in behaviour
Signed-off-by: NDaniel P. Berrange <berrange@redhat.com>

Convert the type of loop iterators named 'i', 'j', k',
'ii', 'jj', 'kk', to be 'size_t' instead of 'int' or
'unsigned int', also santizing 'ii', 'jj', 'kk' to use
the normal 'i', 'j', 'k' naming
Signed-off-by: NDaniel P. Berrange <berrange@redhat.com>

Currently the fdstream function hardcodes the location
of the iohelper to LIBEXECDIR "/libvirt_iohelper". This
is not convenient when trying to write test cases which
use this code. Add a virFDStreamSetIOHelper method to
allow the test cases to point to the location of the
un-installed iohelper binary.
Signed-off-by: NDaniel P. Berrange <berrange@redhat.com>

In b2878ed8 we added the O_NOCTTY
flag when opening files in the stream code. Unfortunately a later
piece of code was comparing the flags == O_RDONLY, without masking
out the non-access mode flags. This broke the iohelper when used
with streams for read, since it caused us to attach the stream
output pipe to the stream input FD instead of output FD :-(
Signed-off-by: NDaniel P. Berrange <berrange@redhat.com>

The source code base needs to be adapted as well. Some files
include virutil.h just for the string related functions (here,
the include is substituted to match the new file), some include
virutil.h without any need (here, the include is removed), and
some require both.

Ensure that the driver struct field names match the public
API names. For an API virXXXX we must have a driver struct
field xXXXX. ie strip the leading 'vir' and lowercase any
leading uppercase letters.
Signed-off-by: NDaniel P. Berrange <berrange@redhat.com>

When opening a stream to a device which is a TTY, that device
may become the controlling TTY of libvirtd, if libvirtd was
daemonized. This in turn means when the other end of the stream
closes, libvirtd gets SIGHUP, causing it to reload its config.
Prevent this by forcing O_NOCTTY on all streams that are opened
Signed-off-by: NDaniel P. Berrange <berrange@redhat.com>

https://www.gnu.org/licenses/gpl-howto.html recommends that
the 'If not, see <url>.' phrase be a separate sentence.

* tests/securityselinuxhelper.c: Remove doubled line.
* tests/securityselinuxtest.c: Likewise.
* globally: s/;  If/.  If/

Per the FSF address could be changed from time to time, and GNU
recommends the following now: (http://www.gnu.org/licenses/gpl-howto.html)

  You should have received a copy of the GNU General Public License
  along with Foobar.  If not, see <http://www.gnu.org/licenses/>.

This patch removes the explicit FSF address, and uses above instead
(of course, with inserting 'Lesser' before 'General').

Except a bunch of files for security driver, all others are changed
automatically, the copyright for securify files are not complete,
that's why to do it manually:

  src/security/security_selinux.h
  src/security/security_driver.h
  src/security/security_selinux.c
  src/security/security_apparmor.h
  src/security/security_apparmor.c
  src/security/security_driver.c

Update the streams code to use virReportError instead of
the streamsReportError custom macro
Signed-off-by: NDaniel P. Berrange <berrange@redhat.com>

Wen Congyang reported that we have a double-close bug if we fail
virFDStreamOpenInternal, since childfd duplicated one of the fds[]
array contents.  In truth, since we always transfer both members
of fds to other variables, we should close the fds through those
other names, and just use fds[] for pipe().

Bug present since 0.9.0 (commit e886237a).

* src/fdstream.c (virFDStreamOpenFileInternal): Swap scope of
childfd and fds[], to avoid a double close.

virCommandRunAsync() will set errfd if it succeed. We should
close it if virFDStreamOpenInternal() fails.

This patch adds another callback to a FDstream object. The original
callback is used by the daemon stream driver to handle events.

This callback is called if and only if the stream is about to be closed.
This might be used to handle cleanup steps after a fdstream exits. This
will be used later on in ensuring mutually exclusive access to consoles.

* src/fdstream.c:
        - emit the callback, when stream is being closed
        - add data structures needed to handle the callback
        - add function to register callback
* src/fdstream.h:
        - define function prototypes for the callback

This patch causes the fdstream driver to call the stream event callback
if virStreamAbort() is called on a stream using this driver.

A remote handler for a stream can only detect changes via stream events,
so this event callback is necessary in order to enable a daemon to abort
a stream in such a way that the client will see the change.

* src/fdstream.c:
        - modify close function to call stream event callback

In virFDStreamOpenFileInternal(), a errfd pipe is opened by
virCommandRunAsync() and given to virFDStreamOpenInternal().

It seems virFDStream should close errfd, just like the other
fd it is given.

This fixes screenshots leaking FDs:
http://bugzilla.redhat.com/show_bug.cgi?id=745761

Revert 6a1f5f56.  Now that libvirt_iohelper takes fds by
inheritance rather than by open() (commit 1eb66479), there is
no longer a race where the parent can unlink() a file prior to
the iohelper open()ing the same file.  From there, it makes
more sense to have the callers both create and unlink, rather
than the caller create and the stream unlink, since the latter
was only needed when iohelper had to do the unlink.

* src/fdstream.h (virFDStreamOpenFile, virFDStreamCreateFile):
Callers are responsible for deletion.
* src/fdstream.c (virFDStreamOpenFileInternal): Don't leak created
file on failure.
(virFDStreamOpenFile, virFDStreamCreateFile): Drop parameter.
* src/lxc/lxc_driver.c (lxcDomainOpenConsole): Update callers.
* src/qemu/qemu_driver.c (qemuDomainScreenshot)
(qemuDomainOpenConsole): Likewise.
* src/storage/storage_driver.c (storageVolumeDownload)
(storageVolumeUpload): Likewise.
* src/uml/uml_driver.c (umlDomainOpenConsole): Likewise.
* src/vbox/vbox_tmpl.c (vboxDomainScreenshot): Likewise.
* src/xen/xen_driver.c (xenUnifiedDomainOpenConsole): Likewise.

Rather than making the iohelper subject to a race in reopening
the file, it is nicer to pass an already-open fd by inheritance.

The old synopsis form must continue to work - if someone updates
their libvirt package and installs a new libvirt_iohelper but
without restarting the old libvirtd daemon, then the daemon can
still make calls using the old syntax but the new iohelper.

* src/util/iohelper.c (runIO): Split code for open...
(prepare): ...to new function.
(usage): Update synopsis.
(main): Allow alternate calling form.
* src/fdstream.c (virFDStreamOpenFileInternal): Use alternate form.

In preparation for a future patch adding new virFile APIs.

* src/util/files.h, src/util/files.c: Move...
* src/util/virfile.h, src/util/virfile.c: ...here, and rename
functions to virFile prefix.  Macro names are intentionally
left alone.
* *.c: All '#include "files.h"' uses changed.
* src/Makefile.am (UTIL_SOURCES): Reflect rename.
* cfg.mk (exclude_file_name_regexp--sc_prohibit_close): Likewise.
* src/libvirt_private.syms: Likewise.
* docs/hacking.html.in: Likewise.
* HACKING: Regenerate.

Enforce the recent flags cleanups - we want to use 'unsigned int flags'
in any of our APIs (except where backwards compatibility is important,
in the public migration APIs), and that all flags are checked for
validity (except when there are stub functions that completely
ignore the flags argument).

There are a few minor tweaks done here to avoid false positives:
signed arguments passed to open() are renamed oflags, and flags
arguments that are legitimately ignored are renamed flags_unused.

* cfg.mk (sc_flags_usage): New rule.
(exclude_file_name_regexp--sc_flags_usage): And a few exemptions.
(sc_flags_debug): Tweak wording.
* src/util/iohelper.c (runIO, main): Rename variable.
* src/util/util.c (virSetInherit): Likewise.
* src/fdstream.h (virFDStreamOpenFile, virFDStreamCreateFile):
Likewise.
* src/fdstream.c (virFDStreamOpenFileInternal)
(virFDStreamOpenFile, virFDStreamCreateFile): Likewise.
* src/util/command.c (virExecWithHook) [WIN32]: Likewise.
* src/util/util.c (virFileOpenAs, virDirCreate) [WIN32]: Likewise.
* src/locking/lock_manager.c (virLockManagerPluginNew)
[!HAVE_DLFCN_H]: Likewise.
* src/locking/lock_driver_nop.c (virLockManagerNopNew)
(virLockManagerNopAddResource, virLockManagerNopAcquire)
(virLockManagerNopRelease, virLockManagerNopInquire): Likewise.

By requesting the pid in virCommandRunAsync, fdstream was claiming
that it would manually wait for the process.  But on the failure
path, the child process was being leaked.

* src/fdstream.c (virFDStreamOpenFileInternal): Auto-reap child.