e5a1bee0 introduced a regression in Boxes: when Boxes is left idle
(it's still doing some libvirt calls in the background), the
libvirt connection gets closed after a few minutes. What happens is
that this code in virNetClientIOHandleOutput gets triggered:

if (!thecall)
    return -1; /* Shouldn't happen, but you never know... */

and after the changes in e5a1bee0, this causes the libvirt connection
to be closed.

Upon further investigation, what happens is that
virNetClientIOHandleOutput is called from gvir_event_handle_dispatch
in libvirt-glib, which is triggered because the client fd became
writable. However, between the times gvir_event_handle_dispatch
is called, and the time the client lock is grabbed and
virNetClientIOHandleOutput is called, another thread runs and
completes the current call. 'thecall' is then NULL when the first
thread gets to run virNetClientIOHandleOutput.

After describing this situation on IRC, danpb suggested this:

11:37 < danpb> In that case I think the correct thing would be to change
               'return -1' above to 'return 0' since that's not actually an
               error - its a rare, but expected event

which is what this patch is doing. I've tested it against master
libvirt, and I didn't get disconnected in ~10 minutes while this
happens in less than 5 minutes without this patch.

src/test/test_driver.c: Implement poolListAllVolumes.

src/storage/storage_driver.c: Implement poolListAllVolumes.

The RPC generator doesn't returning support list of object, this
patch do the work manually.

  * daemon/remote.c:
    Implemente the server side handler remoteDispatchStoragePoolListAllVolumes

  * src/remote/remote_driver.c:
    Add remote driver handler remoteStoragePoolListAllVolumes

  * src/remote/remote_protocol.x:
    New RPC procedure REMOTE_PROC_STORAGE_POOL_LIST_ALL_VOLUMES and
    structs to represent the args and ret for it.

  * src/remote_protocol-structs: Likewise.

Simply returns the storage volume objects. No supported filter
flags.

include/libvirt/libvirt.h.in: Declare the API
python/generator.py: Skip the function for generating. virStoragePool.py
                     will be added in later patch.
src/driver.h: virDrvStoragePoolListVolumesFlags
src/libvirt.c: Implementation for the API.
src/libvirt_public.syms: Export the symbol to public

GNOME Boxes sometimes stops getting domain events from libvirtd, even
after restarting it. Further investigation in libvirtd shows that
events are properly queued with virDomainEventStateQueue, but the
timer virDomainEventTimer which flushes the events and sends them to
the clients never gets called. Looking at the event queue in gdb
shows that it's non-empty and that its size increases with each new
events.

virDomainEventTimer is set up in virDomainEventStateRegister[ID]
when going from 0 client connecte to 1 client connected, but is
initially disabled. The timer is removed in
virDomainEventStateRegister[ID] when the last client is disconnected
(going from 1 client connected to 0).

This timer (which handles sending the events to the clients) is
enabled in virDomainEventStateQueue when queueing an event on an
empty queue (queue containing 0 events). It's disabled in
virDomainEventStateFlush after flushing the queue (ie removing all
the elements from it). This way, no extra work is done when the queue
is empty, and when the next event comes up, the timer will get
reenabled because the queue will go from 0 event to 1 event, which
triggers enabling the timer.

However, with this Boxes bug, we have a client connected (Boxes), a
non-empty queue (there are events waiting to be sent), but a disabled
timer, so something went wrong.

When Boxes connects (it's the only client connecting to the libvirtd
instance I used for debugging), the event timer is not set as expected
(state->timer == -1 when virDomainEventStateRegisterID is called),
but at the same time the event queue is not empty. In other words,
we had no clients connected, but pending events. This also explains
why the timer never gets enabled as this is only done when an event
is queued on an empty queue.

I think this can happen if an event gets queued using
virDomainEventStateQueue and the client disconnection happens before
the event timer virDomainEventTimer gets a chance to run and flush
the event. In this situation, virDomainEventStateDeregister[ID] will
get called with a non-empty event queue, the timer will be destroyed
if this was the only client connected. Then, when other clients connect
at a later time, they will never get notified about domain events as
the event timer will never get enabled because the timer is only
enabled if the event queue is empty when virDomainEventStateRegister[ID]
gets called, which will is no longer the case.

To avoid this issue, this commit makes sure to remove all events from
the event queue when the last client in unregistered. As there is
no longer anyone interested in receiving these events, these events
are stale so there is no need to keep them around. A client connecting
later will have no interest in getting events that happened before it
got connected.

The introduction of /sys/fs/cgroup came in fairly recent kernels.
Prior to that time distros would pick a custom directory like
/cgroup or /dev/cgroup. We need to auto-detect where this is,
rather than hardcoding it
Signed-off-by: NDaniel P. Berrange <berrange@redhat.com>

Add some non-null annotations to qemuMonitorOpen and also
check that the error callback is set, since it is mandatory
Signed-off-by: NDaniel P. Berrange <berrange@redhat.com>

While PMSUSPENDED state was added a long time ago, we didn't have
corresponding life cycle event.

This patch adds a helper to deal with assigning values to
virTypedParameter structures from strings. The helper parses the value
from the string and assigns it to the corresponding union value.

The quota and period tunables for cpu scheduler accept only a certain
range of values. When changing the live configuration invalid values get
rejected. This check is not performed when changing persistent config.

This patch adds a separate range check, that improves error messages
when changing live config and adds the check for persistent config.
This check is done only when using the API. It is still possible to
specify invalid values in the XML.

This patch tries to clean the code up a little bit and shorten very long
lines.

The apparent semantic change from moving the condition before calling
the setter function is a non-issue here as the setter function is a
no-op when called with both arguments zero.

Commit 2a41bc95 dropped a dependency on gawk, but we can go one step
further and avoid awk altogether.

* src/nwfilter/nwfilter_ebiptables_driver.c
(iptablesLinkIPTablesBaseChain): Simplify command.
(ebiptablesDriverInit, ebiptablesDriverShutdown): Drop awk probe.

This patch removed the "--filterwin2k" dnsmasq command line
parameter which was unnecessary for domain specification,
possibly blocked some usage, and was command line clutter.

Gene Czarcinski <gene@czarc.net>

FreeBSD and OpenBSD have a <net/if.h> that is not self-contained;
and mingw lacks the header altogether.  But gnulib has just taken
care of that for us, so we might as well simplify our code.  In
the process, I got a syntax-check failure if we don't also take
the gnulib execinfo module.

* .gnulib: Update to latest, for execinfo and net_if.
* bootstrap.conf (gnulib_modules): Add execinfo and net_if modules.
* configure.ac: Let gnulib check for headers.  Simplify check for
'struct ifreq', while also including enough prereq headers.
* src/internal.h (IF_NAMESIZE): Drop, now that gnulib guarantees it.
* src/nwfilter/nwfilter_learnipaddr.h: Use correct header for
IF_NAMESIZE.
* src/util/virnetdev.c (includes): Assume <net/if.h> exists.
* src/util/virnetdevbridge.c (includes): Likewise.
* src/util/virnetdevtap.c (includes): Likewise.
* src/util/logging.c (includes): Assume <execinfo.h> exists.
(virLogStackTraceToFd): Handle gnulib's fallback implementation.

Introduced in commit f299ddd6.

* src/check-symfile.pl: Fix whitespace.
* .dir-locals.el (perl-mode): Prevent future occurrences.

When the event symbols were added to the public API, not all
of them were removed from the private exports list. Solaris
gets unhappy when there are duplicated symbols. Extend the
symfile check to test for this scenario

src/test/test_driver.c: Implement listAllStoragePools

src/storage/storage_driver.c: Implement listAllStoragePools.

The RPC generator doesn't support returning list of object, this patch does
the work manually.

  * daemon/remote.c:
    Implement the server side handler remoteDispatchConnectListAllStoragePools

  * src/remote/remote_driver.c:
    Add remote driver handler remoteConnectListAllStoragePools.

  * src/remote/remote_protocol.x:
    New RPC procedure REMOTE_PROC_CONNECT_LIST_ALL_STORAGE_POOLS and
    structs to represent the args and ret for it.

  * src/remote_protocol-structs: Likewise.

src/conf/storage_conf.c: Add virStoragePoolMatch to filter the
pools; Add virStoragePoolList to iterate over the pool objects
with filter.

src/conf/storage_conf.h: Declare virStoragePoolMatch,
virStoragePoolList, and the macros for filters.

src/libvirt_private.syms: Export helper virStoragePoolList.

This introduces a new API to list the storage pool objects,
4 groups of flags are provided to filter the returned pools:

  * Active or not

  * Autostarting or not

  * Persistent or not

  * And the pool type.

include/libvirt/libvirt.h.in: New enum virConnectListAllStoragePoolFlags;
                              Declare the API.
python/generator.py: Skip the generating
src/driver.h: (virDrvConnectListAllStoragePools)
src/libvirt.c: Implementation for the API.
src/libvirt_public.syms: Export the symbol.

ESX doesn't use the common virDomainObj implementation so this patch
adds a separate implementation.

This driver supports all currently defined filtering flags, but as with
other drivers some combinations yield a empty result list.

Hyperv doesn't use the common virDomainObj implementation so this patch
adds a separate implementation.

This driver supports all currently added flags for filtering although
some of those don't make sense with this driver (no support yet) and
thus produce no output when used.

I tested both OpenBSD and cygwin; both failed 'make check' with:

  GEN    check-symfile
Can't return outside a subroutine at ./check-symfile.pl line 13.

Perl requires 'exit 77' instead of 'return 77' in that context,
but even with that tweak, the build still fails, since the exit
code of 77 is only special to explicit TESTS=foo listings, and
not to make-only dependency rules where we are not going through
automake's test framework.

* src/check-symfile.pl: Kill bogus platform check...
* src/Makefile.am (check-symfile): ...and replace with an automake
conditional.

This fixes https://bugzilla.redhat.com/show_bug.cgi?id=852984

If a network or interface is configured to use Open vSwitch, but
ovs-vswitchd (the Open vSwitch database service) isn't running, the
ovs-vsctl add-port/del-port commands will hang indefinitely rather
than returning an error. There is a --nowait option, but that appears
to have no effect on add-port and del-port commands, so instead we add
a --timeout=5 to the commands - they will retry for up to 5 seconds,
then fail if there is no response.

OpenBSD ships with gcc 4.2.1, which annoyingly treats all format
strings as though they were also attribute((nonnull)).  The two
concepts are orthogonal, though, as evidenced by the number of
spurious warnings it generates on uses where we know that
virReportError specifically handles NULL instead of a format
string; worse, since we now force -Werror on git builds, it
prevents development builds on OpenBSD.

I hate to do this, as it disables ALL format checking on older
gcc, and therefore misses out on some useful checks (code that
happened to compile on Linux may still have type mismatches
when compiled on other platforms, as evidenced by the number
of times I have fixed formatting mismatches for uid_t as found
by warnings on Cygwin), but I don't see any other way to keep
-Werror alive and still compile on OpenBSD.

A more invasive change would be to make virReportError() mark
its format attribute as nonnull, and fix (a lot of) fallout;
we may end up doing that anyways as part of danpb's error
refactoring improvements, but not today.

* src/internal.h (ATTRIBUTE_FMT_PRINTF): Use preferred spellings.
* m4/virt-compile-warnings.m4 (-Wformat): Disable on older gcc.

This is another fix for the emulator-pin series. When going through
the cputune pinning settings, the current code is trying to pin all
the CPUs, even when not all of them are specified. This causes error
in the subsequent function which, of course, cannot find the cpu to
pin. Since it's enough to pass the correct VCPU ID to the function,
the fix is trivial.

Only VNC_{{DIS,}CONNECTED,INITIALIZED} and SPICE_INITIALIZED events are
documented to support server/auth field and even there it is marked as
optional. Emit "" auth scheme in case QEMU didn't send it.

As a request was raised for this, I added few lines in the "Notes"
part of the "security_driver" comments about allowed values.

On OpenBSD, clock_gettime() exists in libc rather than librt, and
blindly linking with -lrt made the build fail.  Gnulib already
did the work for determining which libraries to use, so we should
reuse that work rather than doing it ourselves.

* bootstrap.conf (gnulib_modules): Pull in clock-time.
* configure.ac (RT_LIBS): Drop.
* src/Makefile.am (libvirt_util_la_LIBADD): Use gnulib variable
instead.
* src/util/virtime.c (includes): Simplify.

After discussion with DB we decided to rename the new iolimit
element as it creates the impression it would be there to
limit (i.e. throttle) I/O instead of specifying immutable
characteristics of a block device.
This is also backed by the fact that the term I/O Limits has
vanished from newer storage admin documentation.
Signed-off-by: NViktor Mihajlovski <mihajlov@linux.vnet.ibm.com>

When reboot using qemu guest agent was requested, qemu driver kept
waiting for SHUTDOWN event from qemu. However, such event is never
emitted during guest reboot and qemu driver would keep waiting forever.

When the password change failed we updated the graphic definition
anyways, which is not desired.

This patch adds support for running qemu guests with the required
parameters to forcefully enable or disable BIOS advertising of S3 and
S4 states.  The support for this is added to capabilities and there is
also a qemu command parameter parsing implemented.

There is a new <pm/> element implemented that can control what ACPI
sleeping states will be advertised by BIOS and allowed to be switched
to by libvirt. The default keeps defaults on hypervisor, otherwise
forces chosen setting.
The documentation of the pm element is added as well.

Implementation of iolimits for the qemu driver with
capability probing for block size attribute and
command line generation for block sizes.
Including testcase for qemuxml2argvtest.
Signed-off-by: NViktor Mihajlovski <mihajlov@linux.vnet.ibm.com>