Open-coding one VIR_FREE in the test suite just doesn't seem right.

The future QEMU capabilities cache needs to be able to invalidate
itself if the libvirtd binary or any loadable modules are changed
on disk. Record the 'ctime' value for these binaries and provide
helper APIs to query it. This approach assumes that if libvirt.so
is changed, then libvirtd will also change, which should usually
be the case with libtool's wrapper scripts that cause libvirtd to
get re-linked
Signed-off-by: NDaniel P. Berrange <berrange@redhat.com>

GNULIB provides APIs for calculating md5 and sha256 hashes,
but these APIs only return you raw byte arrays. Most users
in libvirt want the hash in printable string format. Add
some helper APIs in util/vircrypto.{c,h} for doing this.
Signed-off-by: NDaniel P. Berrange <berrange@redhat.com>

Signed-off-by: NChunyan Liu <cyliu@suse.com>

Auditing all callers of virCommandRun and virCommandWait that
passed a non-NULL pointer for exit status turned up some
interesting observations.  Many callers were merely passing
a pointer to avoid the overall command dying, but without
caring what the exit status was - but these callers would
be better off treating a child death by signal as an abnormal
exit.  Other callers were actually acting on the status, but
not all of them remembered to filter by WIFEXITED and convert
with WEXITSTATUS; depending on the platform, this can result
in a status being reported as 256 times too big.  And among
those that correctly parse the output, it gets rather verbose.
Finally, there were the callers that explicitly checked that
the status was 0, and gave their own message, but with fewer
details than what virCommand gives for free.

So the best idea is to move the complexity out of callers and
into virCommand - by default, we return the actual exit status
already cleaned through WEXITSTATUS and treat signals as a
failed command; but the few callers that care can ask for raw
status and act on it themselves.

* src/util/vircommand.h (virCommandRawStatus): New prototype.
* src/libvirt_private.syms (util/command.h): Export it.
* docs/internals/command.html.in: Document it.
* src/util/vircommand.c (virCommandRawStatus): New function.
(virCommandWait): Adjust semantics.
* tests/commandtest.c (test1): Test it.
* daemon/remote.c (remoteDispatchAuthPolkit): Adjust callers.
* src/access/viraccessdriverpolkit.c (virAccessDriverPolkitCheck):
Likewise.
* src/fdstream.c (virFDStreamCloseInt): Likewise.
* src/lxc/lxc_process.c (virLXCProcessStart): Likewise.
* src/qemu/qemu_command.c (qemuCreateInBridgePortWithHelper):
Likewise.
* src/xen/xen_driver.c (xenUnifiedXendProbe): Simplify.
* tests/reconnect.c (mymain): Likewise.
* tests/statstest.c (mymain): Likewise.
* src/bhyve/bhyve_process.c (virBhyveProcessStart)
(virBhyveProcessStop): Don't overwrite virCommand error.
* src/libvirt.c (virConnectAuthGainPolkit): Likewise.
* src/openvz/openvz_driver.c (openvzDomainGetBarrierLimit)
(openvzDomainSetBarrierLimit): Likewise.
* src/util/virebtables.c (virEbTablesOnceInit): Likewise.
* src/util/viriptables.c (virIpTablesOnceInit): Likewise.
* src/util/virnetdevveth.c (virNetDevVethCreate): Fix debug
message.
* src/qemu/qemu_capabilities.c (virQEMUCapsInitQMP): Add comment.
* src/storage/storage_backend_iscsi.c
(virStorageBackendISCSINodeUpdate): Likewise.
Signed-off-by: NEric Blake <eblake@redhat.com>

Thanks to namespaces, we have a couple of places in the code
base that want to reflect a child exit status, including the
ability to detect death by a signal, back to a grandparent.
Best to make it a reusable function.

* src/util/virprocess.h (virProcessExitWithStatus): New prototype.
* src/libvirt_private.syms (util/virprocess.h): Export it.
* src/util/virprocess.c (virProcessExitWithStatus): New function.
* tests/commandtest.c (test23): Test it.
Signed-off-by: NEric Blake <eblake@redhat.com>

We will need to call virDomainNetDefFormat() from the network hook (in
the network driver).

This function is needed for user namespaces, where we need to chmod()
the cgroup to the initial uid/gid such that systemd is allowed to
use the cgroup.
Signed-off-by: NRichard Weinberger <richard@nod.at>
Signed-off-by: NDaniel P. Berrange <berrange@redhat.com>

Add a virStringReplace method to virstring.{h,c} to perform
substring matching and replacement
Signed-off-by: NDaniel P. Berrange <berrange@redhat.com>

Add a virStringSearch method to virstring.{c,h} which performs
a regex match against a string and returns the matching substrings.
Signed-off-by: NDaniel P. Berrange <berrange@redhat.com>

Systemd does not forget about the cases, where client service needs to
wait for daemon service to initialize and start accepting new clients.
Setting a dependency in client is not enough as systemd doesn't know
when the daemon has initialized itself and started accepting new
clients. However, it offers a mechanism to solve this. The daemon needs
to call a special systemd function by which the daemon tells "I'm ready
to accept new clients". This is exactly what we need with
libvirtd-guests (client) and libvirtd (daemon). So now, with this
change, libvirt-guests.service is invoked not any sooner than
libvirtd.service calls the systemd notify function.
Signed-off-by: NMichal Privoznik <mprivozn@redhat.com>

The virDomainGetRootFilesystem method can be generalized to allow
any filesystem path to be obtained.

While doing this, start a new test case for purpose of testing various
helper methods in the domain_conf.{c,h} files, such as this one.
Signed-off-by: NDaniel P. Berrange <berrange@redhat.com>

To reuse this from other drivers, like lxc.

This reads blkio stats from blkio.throttle.io_service_bytes and
blkio.throttle.io_serviced.

Use it for the default mouse.
Signed-off-by: NLi Zhang <zhlcindy@linux.vnet.ibm.com>
Signed-off-by: NJán Tomko <jtomko@redhat.com>

Basically, the idea is copied from domain code, where tainting
exists for a while. Currently, only one taint reason exists -
VIR_NETWORK_TAINT_HOOK to mark those networks which caused invoking
of hook script.
Signed-off-by: NMichal Privoznik <mprivozn@redhat.com>

In the next patch I'm going to need the network format function that
takes virBuffer as argument. However, slightly change of name is more
appropriate then: virNetworkDefFormatBuf to match the rest of functions
that format an object to buffer.
Signed-off-by: NMichal Privoznik <mprivozn@redhat.com>

Implement virProcessRunInMountNamespace, which runs callback of type
virProcessNamespaceCallback in a container namespace. This uses a
child process to run the callback, since you can't change the mount
namespace of a thread. This implies that callbacks have to be careful
about what code they run due to async safety rules.

Idea by Dan Berrange, based on an initial report by Reco
<recoverym4n@gmail.com> at
http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=732394Signed-off-by: NDaniel Berrange <berrange@redhat.com>
Signed-off-by: NEric Blake <eblake@redhat.com>

Add a helper function which takes a file path and ensures
that all directory components leading up to the file exist.
IOW, it strips the filename part of the path and passes
the result to virFileMakePath.
Signed-off-by: NDaniel P. Berrange <berrange@redhat.com>

All the data for getting the actual type is present in the snapshot
config. There is no need to have this function private to the qemu
driver and it will be re-used later in other parts of libvirt

All the data for getting the actual type is present in the domain
config. There is no need to have this function private to the qemu
driver and it will be re-used later in other parts of libvirt

virConf now honours a VIR_CONF_FLAG_LXC_FORMAT flag to handle LXC
configuration files. The differences are that property names can
contain '.' character and values are all strings without any bounding
quotes.

Provide a new virConfWalk function calling a handler on all non-comment
values. This function will be used by the LXC conversion code to loop
over LXC configuration lines.

This commit allows to attach/detach a <filesystem> device in qemu. For
this purpose I'm introducing two new functions: virDomainFSInsert() and
virDomainFSRemove() and adding necessary code in the qemu driver.  It
compares filesystems based on their "destination" folder. So if two
filesystems share the same destination, they are considered equal and
the qemu driver would reject the insertion.
Signed-off-by: NMatthieu Coudron <mattator@gmail.com>

virKModConfig()        - Return a buffer containing kernel module configuration
virKModLoad()          - Load a specific module into the kernel configuration
virKModUnload()        - Unload a specific module from the kernel configuration
virKModIsBlacklisted() - Determine whether a module is blacklisted within
                         the kernel configuration

The NWFilter code has as a deadlock race condition between
the virNWFilter{Define,Undefine} APIs and starting of guest
VMs due to mis-matched lock ordering.

In the virNWFilter{Define,Undefine} codepaths the lock ordering
is

  1. nwfilter driver lock
  2. virt driver lock
  3. nwfilter update lock
  4. domain object lock

In the VM guest startup paths the lock ordering is

  1. virt driver lock
  2. domain object lock
  3. nwfilter update lock

As can be seen the domain object and nwfilter update locks are
not acquired in a consistent order.

The fix used is to push the nwfilter update lock upto the top
level resulting in a lock ordering for virNWFilter{Define,Undefine}
of

  1. nwfilter driver lock
  2. nwfilter update lock
  3. virt driver lock
  4. domain object lock

and VM start using

  1. nwfilter update lock
  2. virt driver lock
  3. domain object lock

This has the effect of serializing VM startup once again, even if
no nwfilters are applied to the guest. There is also the possibility
of deadlock due to a call graph loop via virNWFilterInstantiate
and virNWFilterInstantiateFilterLate.

These two problems mean the lock must be turned into a read/write
lock instead of a plain mutex at the same time. The lock is used to
serialize changes to the "driver->nwfilters" hash, so the write lock
only needs to be held by the define/undefine methods. All other
methods can rely on a read lock which allows good concurrency.
Signed-off-by: NDaniel P. Berrange <berrange@redhat.com>

Add virRWLock backed up by a POSIX rwlock primitive
Signed-off-by: NDaniel P. Berrange <berrange@redhat.com>

It doesn't make sense to fail if the SCSI host device is specified
as "shareable" explicitly between domains (NB, it works if and only
if the device is specified as "shareable" for *all* domains,
otherwise it fails).

To fix the problem, this patch introduces an array for virSCSIDevice
struct, which records all the names of domain which are using the
device (note that the recorded domains must specify the device as
shareable).  And the change on the data struct brings on many
subsequent changes in the code.

Prior to this patch, the "shareable" tag didn't work as expected,
it actually work like "non-shareable".  So this patch also added notes
in formatdomain.html to declare the fact.

* src/util/virscsi.h:
  - Remove virSCSIDeviceGetUsedBy
  - Change definition of virSCSIDeviceGetUsedBy and virSCSIDeviceListDel
  - Add virSCSIDeviceIsAvailable

* src/util/virscsi.c:
  - struct virSCSIDevice: Change "used_by" to be an array; Add
    "n_used_by" as the array count
  - virSCSIDeviceGetUsedBy: Removed
  - virSCSIDeviceFree: frees the "used_by" array
  - virSCSIDeviceSetUsedBy: Copy the domain name to avoid potential
    memory corruption
  - virSCSIDeviceIsAvailable: New
  - virSCSIDeviceListDel: Change the logic, for device which is already
    in the list, just remove the corresponding entry in "used_by". And
    since it's only used in one place, we can safely removing the code
    to find out the dev in the list first.
  - Copyright updating

* src/libvirt_private.sys:
  - virSCSIDeviceGetUsedBy: Remove
  - virSCSIDeviceIsAvailable: New

* src/qemu/qemu_hostdev.c:
  - qemuUpdateActiveScsiHostdevs: Check if the device existing before
    adding it to the list;
  - qemuPrepareHostdevSCSIDevices: Error out if the not all domains
    use the device as "shareable"; Also don't try to add the device
    to the activeScsiHostdevs list if it already there; And make
    more sensible error w.r.t the current "shareable" value in
    driver->activeScsiHostdevs.
  - qemuDomainReAttachHostScsiDevices: Change the logic according
    to the changes on helpers.
Signed-off-by: NOsier Yang <jyang@redhat.com>

This reverts commit 2996e6be
and some parts of 2636dc8c.

The former one tried to implement QoS setting on bridgeless networks.
However, as discussed upstream [1], the patch is far away from being
useful in even a single case. The whole idea of network QoS is to have
aggregated limits over several interfaces. This patch is doing
completely the opposite when merging two QoS settings (from the network
and the domain interface) into one which is then set at the domain
interface itself, not the network.

The latter one is the test for the previous one. Now none of them makes
sense.

1: https://www.redhat.com/archives/libvir-list/2014-January/msg01441.html

Conflicts:
	tests/virnetdevbandwidthtest.c: New test has been introduced since
    then.

There are some units within libvirt that utilize virCommand API to run
some commands and deserve own unit testing. These units are, however,
not desired to be rewritten to dig virCommand API usage out. As a great
example virNetDevBandwidth could be used. The problem with the bandwidth
unit is: it uses virCommand API heavily. Therefore we need a mechanism
to not really run a command, but rather see its string representation
after which we can decide if the unit construct the correct sequence of
commands or not.
Signed-off-by: NMichal Privoznik <mprivozn@redhat.com>

https://bugzilla.redhat.com/show_bug.cgi?id=1055484

Currently, libvirt's XML schema of network allows QoS to be defined for
every network even though it has no bridge. For instance:

<network>
    <name>vdsm-no-bridge</name>
    <forward mode='passthrough'>
      <interface dev='em1.10'/>
    </forward>
    <bandwidth>
        <inbound average='1000' peak='5000' burst='1024'/>
        <outbound average='1000' burst='1024'/>
    </bandwidth>
</network>

The bandwidth limitations can be, however, applied even on such
networks. In fact, they are going to be applied on the interface that
will be connected to the network on a domain startup. This approach,
however, has one limitation. With bridged networks, there are two points
where QoS can be set: bridge and domain interface. The lower limit of
the two is enforced then. For instance, if the interface has 10Mbps
average, but the network only 1Mbps, there's no way for interface to
transmit packets faster than the 1Mbps limit. With two points this is
enforced by kernel.  With only one point, we must combine both QoS
settings into one which is set afterwards. Look at
virNetDevBandwidthMinimal() and you'll understand immediately what I
mean.
Signed-off-by: NMichal Privoznik <mprivozn@redhat.com>

Unlike the host devices of other types, SCSI host device XML supports
"shareable" tag. This patch introduces it for the virSCSIDevice struct
for a later patch use (to detect if the SCSI device is shareable when
preparing the SCSI host device in QEMU driver).

spice-server offers an API to disable file transfer messages
on the agent channel between the client and the guest.
This is supported in qemu through the disable-agent-file-xfer option.

This patch exposes this option to libvirt.
Adds a new element 'filetransfer', with one property,
'enable', which accepts a boolean.
Default is enabled, for backward compatibility.

Depends on the capability exported in the first patch of the series.
Signed-off-by: NFrancesco Romani <fromani@redhat.com>