Separate out the internals as they will become more complex soon.
Signed-off-by: NPeter Krempa <pkrempa@redhat.com>
Reviewed-by: NMichal Privoznik <mprivozn@redhat.com>

Every supported qemu is able to return the list of machine types it
supports so we can start validating it against that list. The advantage
is a better error message, and the change will also prevent having stale
test data.
Signed-off-by: NPeter Krempa <pkrempa@redhat.com>
Reviewed-by: NMichal Privoznik <mprivozn@redhat.com>

Enumerate all missing machine types for all missing architectures for
the fake capabilities used in many existing tests. This will allow
stricter validation whether qemu actually supports given machine type
since we already have some behaviour dependant on the actual machine
type.
Signed-off-by: NPeter Krempa <pkrempa@redhat.com>
Reviewed-by: NMichal Privoznik <mprivozn@redhat.com>

Use the 's390-ccw-virtio' machine type which is actually supported by
the qemu we gathered the test data from.
Signed-off-by: NPeter Krempa <pkrempa@redhat.com>
Reviewed-by: NMichal Privoznik <mprivozn@redhat.com>

This machine type comes from downstream ubuntu 15.10. Replace it with a
somewhat equivalent qemu-2.3 machine type as we do have test data for
that.

The change allows the CPU code to pick a proper default CPU in the
'-latest' cases.
Signed-off-by: NPeter Krempa <pkrempa@redhat.com>
Reviewed-by: NMichal Privoznik <mprivozn@redhat.com>

Use 'versatilepb' instead of a fake 'non-virt' machine type.
Signed-off-by: NPeter Krempa <pkrempa@redhat.com>
Reviewed-by: NMichal Privoznik <mprivozn@redhat.com>

Use 'pc' instead of ancient 'rhel-6.1'.
Signed-off-by: NPeter Krempa <pkrempa@redhat.com>
Reviewed-by: NMichal Privoznik <mprivozn@redhat.com>

Signed-off-by: NPeter Krempa <pkrempa@redhat.com>
Reviewed-by: NMichal Privoznik <mprivozn@redhat.com>

For testing with synthetic capabilities we pre-fill the qemu
capabilities with some machine types. Historically there were two arrays
for KVM and TCG but that's not necessary. Make both instances of x86_64
data share the same array as the other architectures do.

This will later on simplify filling in all the other machine types which
are required for the test suite.
Signed-off-by: NPeter Krempa <pkrempa@redhat.com>
Reviewed-by: NMichal Privoznik <mprivozn@redhat.com>

The tests prefixed default-video* were enabled only for the xml2xml
testing and used impossible configurations.

Enable them for xml2argv testing fix them:
1) aarch64: remove pointless cpu mode
2) s390x: remove pointless cpu and use existing machine type
3) riscv: remove pointless cpu
4) x86: remove pointless cpu and use existing machine type
5) ppc65: use correct machine type and enable USB
Signed-off-by: NPeter Krempa <pkrempa@redhat.com>
Reviewed-by: NMichal Privoznik <mprivozn@redhat.com>

XML->XML testing uses DO_TEST_CAPS_LATEST so use it also for the
XML->argv testing. Additionally use the same more modern machine type
in both tests.
Signed-off-by: NPeter Krempa <pkrempa@redhat.com>
Reviewed-by: NMichal Privoznik <mprivozn@redhat.com>

Mirror what's done in the xml2argv test and use recent capabilities.
Signed-off-by: NPeter Krempa <pkrempa@redhat.com>
Reviewed-by: NMichal Privoznik <mprivozn@redhat.com>

The data is tested against the latest qemu binaries so we should use the
proper architecture. Also the test is used against data from qemu 1.5.3
and thus we should use a machine type that qemu supported.
Signed-off-by: NPeter Krempa <pkrempa@redhat.com>
Reviewed-by: NMichal Privoznik <mprivozn@redhat.com>

Debian/Ubuntu linkers are more strict that other distros requiring glib
to be linked explicitly.

macOS needs -export-dynamic instead of -Wl,--export-dynamic
Signed-off-by: NDaniel P. Berrangé <berrange@redhat.com>

Signed-off-by: NDaniel P. Berrangé <berrange@redhat.com>

Similarly to 510d154a we need to prevent
doing too deeply nested backing chains and reject them with a sane error
message.

Add a loop to go through the snapshots prior to attempting actually
creating them to prevent some possible inconsistent scenarios.

We don't need to do it when reusing backing chains as we'll be
re-detecting the backing chain in that case anyways.
Signed-off-by: NPeter Krempa <pkrempa@redhat.com>
Reviewed-by: NEric Blake <eblake@redhat.com>

Don't adopt the backing store data when reusing images provided by the
user. This will force a backing chain re-probe as users might have
passed in something unexpected in the overlay where our view of the
backing chain would not correspond.

This is done only for inactive snapshots as there we have way less
verification.
Signed-off-by: NPeter Krempa <pkrempa@redhat.com>
Reviewed-by: NEric Blake <eblake@redhat.com>

Currently the libvirt Go modules are accessed by applications using
their github repository URLs. This is undesirable as we don't want
applications to have a direct dependancy on a specific source repo
location. We want to enable applications to use the Go packages via
the libvirt.org namespace.

When you do "go get libvirt.org/libvirt-go", the Go client will do an
HTTPS request to that URL, and parse the HTML content to look for a
<meta> tag which tells it where to the find the GIT repository.

   https://golang.org/cmd/go/#hdr-Remote_import_paths

This adds two pages to support this Go module resolution. They are
not linked from anywhere as we don't expect users to actually look
at them. If someone does happen upon them, there's some boilerplate
text to send them off to godoc.org for API documentation.

Since the pages we're adding have a .html extension, we will also
use a small apache config tweak on the server

    RewriteEngine on
    RewriteRule ^/libvirt-go$ /libvirt-go.html [L]
    RewriteRule ^/libvirt-go-xml$ /libvirt-go-xml.html [L]
Reviewed-by: NJiri Denemark <jdenemar@redhat.com>
Signed-off-by: NDaniel P. Berrangé <berrange@redhat.com>

The previous "QEMU shim" proof of concept was taking an approach of only
caring about initial spawning of the QEMU process. It was then
registered with the libvirtd daemon who took over management of it. The
intent was that later libvirtd would be refactored so that the shim
retained control over the QEMU monitor and libvirt just forwarded APIs
to each shim as needed. This forwarding of APIs would require quite alot
of significant refactoring of libvirtd to achieve.

This impl thus takes a quite different approach, explicitly deciding to
keep the VMs completely separate from those seen & managed by libvirtd.
Instead it uses the new "qemu:///embed" URI scheme to embed the entire
QEMU driver in the shim, running with a custom root directory.

Once the driver is initialization, the shim starts a VM and then waits
to shutdown automatically when QEMU shuts down, or should kill QEMU if
it is terminated itself. This ought to use the AUTO_DESTROY feature but
that is not yet available in embedded mode, so we rely on installing a
few signal handlers to gracefully kill QEMU. This isn't reliable if
we crash of course, but you can restart with the same root dir.

Note this program does not expose any way to manage the QEMU process,
since there's no RPC interface enabled. It merely starts the VM and
cleans up when the guest shuts down at the end. This program is
installed to /usr/bin/virt-qemu-run enabling direct use by end users.
Most use cases will probably want to integrate the concept directly
into their respective application codebases. This standalone binary
serves as a nice demo though, and also provides a way to measure
performance of the startup process quite simply.
Reviewed-by: NMichal Privoznik <mprivozn@redhat.com>
Signed-off-by: NDaniel P. Berrangé <berrange@redhat.com>

This enables support for running the secret driver embedded to the
calling application process using a URI:

   secret:///embed?root=/some/path

When using the embedded mode with a root=/var/tmp/embed, the
driver will use the following paths:

       configDir: /var/tmp/embed/etc/secrets
        stateDir: /var/tmp/embed/run/secrets

These are identical whether the embedded driver is privileged
or unprivileged.

This compares with the system instance which uses

       configDir: /etc/libvirt/secrets
        stateDir: /var/lib/libvirt/secrets

When an embedded instance of the secret driver is open, any other
embedded drivers will automatically use the embedded secret driver.
Reviewed-by: NMichal Privoznik <mprivozn@redhat.com>
Signed-off-by: NDaniel P. Berrangé <berrange@redhat.com>

This enables support for running QEMU embedded to the calling
application process using a URI:

   qemu:///embed?root=/some/path

Note that it is important to keep the path reasonably short to
avoid risk of hitting the limit on UNIX socket path names
which is 108 characters.

When using the embedded mode with a root=/var/tmp/embed, the
driver will use the following paths:

                logDir: /var/tmp/embed/log/qemu
           swtpmLogDir: /var/tmp/embed/log/swtpm
         configBaseDir: /var/tmp/embed/etc/qemu
              stateDir: /var/tmp/embed/run/qemu
         swtpmStateDir: /var/tmp/embed/run/swtpm
              cacheDir: /var/tmp/embed/cache/qemu
                libDir: /var/tmp/embed/lib/qemu
       swtpmStorageDir: /var/tmp/embed/lib/swtpm
 defaultTLSx509certdir: /var/tmp/embed/etc/pki/qemu

These are identical whether the embedded driver is privileged
or unprivileged.

This compares with the system instance which uses

                logDir: /var/log/libvirt/qemu
           swtpmLogDir: /var/log/swtpm/libvirt/qemu
         configBaseDir: /etc/libvirt/qemu
              stateDir: /run/libvirt/qemu
         swtpmStateDir: /run/libvirt/qemu/swtpm
              cacheDir: /var/cache/libvirt/qemu
                libDir: /var/lib/libvirt/qemu
       swtpmStorageDir: /var/lib/libvirt/swtpm
 defaultTLSx509certdir: /etc/pki/qemu

At this time all features present in the QEMU driver are available when
running in embedded mode, availability matching whether the embedded
driver is privileged or unprivileged.
Reviewed-by: NMichal Privoznik <mprivozn@redhat.com>
Signed-off-by: NDaniel P. Berrangé <berrange@redhat.com>

The driver URI scheme:

  "$drivername:///embed?root=/some/path"

enables a new way to use the drivers by embedding them directly in the
calling process. To use this the process must have a thread running the
libvirt event loop. This URI will then cause libvirt to dynamically load
the driver module and call its global initialization function. This
syntax is applicable to any driver, but only those will have been
modified to support a custom root directory and embed URI path will
successfully open.

The application can now make normal libvirt API calls which are all
serviced in-process with no RPC layer involved.

It is required to specify an explicit root directory, and locks will be
acquired on this directory to avoid conflicting with another app that
might accidentally pick the same directory.

Use of '/' is not explicitly forbidden, but note that the file layout
used underneath the embedded driver root does not match the file
layout used by system/session mode drivers. So this cannot be used as
a backdoor to interact with, or fake, the system/session mode drivers.

Libvirt will create arbitrary files underneath this root directory. The
root directory can be kept untouched across connection open attempts if
the application needs persistence. The application is responsible for
purging everything underneath this root directory when finally no longer
required.

Even when a virt driver is used in embedded mode, it is still possible
for it to in turn use functionality that calls out to other secondary
drivers in libvirtd. For example an embedded instance of QEMU can open
the network, secret or storage drivers in the system libvirtd.

That said, the application would typically want to at least open an
embedded secret driver ("secret:///embed?root=/some/path"). Note that
multiple different embedded drivers can use the same root prefix and
co-operate just as they would inside a normal libvirtd daemon.

A key thing to note is that for this to work, the application that links
to libvirt *MUST* be built with -Wl,--export-dynamic to ensure that
symbols from libvirt.so are exported & thus available to the dynamically
loaded driver module. If libvirt.so itself was dynamically loaded then
RTLD_GLOBAL must be passed to dlopen().
Reviewed-by: NMichal Privoznik <mprivozn@redhat.com>
Signed-off-by: NDaniel P. Berrangé <berrange@redhat.com>

The intent here is to allow the virt drivers to be run directly embedded
in an arbitrary process without interfering with libvirtd. To achieve
this they need to store all their configuration & state in a separate
directory tree from the main system or session libvirtd instances.

This can be useful for doing testing of the virt drivers in "make check"
without interfering with the user's own libvirtd instances.

It can also be used for applications using KVM/QEMU as a piece of
infrastructure to build an service, rather than for general purpose
OS hosting. A long standing example is libguestfs, which would prefer
if its temporary VMs did show up in the main libvirtd VM list, because
this confuses apps such as OpenStack Nova. A more recent example would
be Kata which is using KVM as a technology to build containers.
Reviewed-by: NMichal Privoznik <mprivozn@redhat.com>
Reviewed-by: NCole Robinson <crobinso@redhat.com>
Signed-off-by: NDaniel P. Berrangé <berrange@redhat.com>

Reviewed-by: NMichal Privoznik <mprivozn@redhat.com>
Signed-off-by: NDaniel P. Berrangé <berrange@redhat.com>

If a domain is configured to have an egl-headless display and a virtio
video device, virgl will be enabled automatically within the guest, even
if the video device is configured with accel3d='no'.

In this case we should explicitly pass 'virgl=off' to qemu.

See https://bugzilla.redhat.com/show_bug.cgi?id=1791236 for more
information.
Signed-off-by: NJonathon Jongsma <jjongsma@redhat.com>
Reviewed-by: NMichal Privoznik <mprivozn@redhat.com>

Signed-off-by: NHan Han <hhan@redhat.com>
Reviewed-by: NMichal Privoznik <mprivozn@redhat.com>

Since v4.2-rc0, QEMU introduced a builtin rng backend that uses
getrandom() syscall to generate random. Add it to libvirt with the
backend model 'builtin'.

https://bugzilla.redhat.com/show_bug.cgi?id=1785091Signed-off-by: NHan Han <hhan@redhat.com>
Reviewed-by: NMichal Privoznik <mprivozn@redhat.com>

The 'builtin' rng backend model can be used as following:
  <rng model='virtio'>
    <backend model='builtin'/>
  </rng>
Signed-off-by: NHan Han <hhan@redhat.com>
Reviewed-by: NMichal Privoznik <mprivozn@redhat.com>

For qemu object like rng-builtin, there are no properties after id
property. We should always set comma after object id. Otherwise it will
cause trailing comma on object:
    -object rng-builtin,id=ID,
Signed-off-by: NHan Han <hhan@redhat.com>
Reviewed-by: NMichal Privoznik <mprivozn@redhat.com>

It is used to check if qemu is capable of rng-builtin object.

This object is added since qemu-4.2.0-rc0, commit 6c4e9d48.
Signed-off-by: NHan Han <hhan@redhat.com>
Reviewed-by: NMichal Privoznik <mprivozn@redhat.com>

Since v5.6.0-48-g270583ed we try to cache domain capabilities,
i.e. store filled virDomainCaps in a hash table in virQEMUCaps
for future use. However, there's a race condition in the way it's
implemented. We use virQEMUCapsGetDomainCapsCache() to obtain the
pointer to the hash table, then we search the hash table for
cached data and if none is found the domcaps is constructed and
put into the table. Problem is that this is all done without any
locking, so if there are two threads trying to do the same, one
will succeed and the other will fail inserting the data into the
table.

Also, the API looks a bit fishy - obtaining pointer to the hash
table is dangerous.

The solution is to use a mutex that guards the whole operation
with the hash table. Then, the API can be changes to return
virDomainCapsPtr directly.

Resolves: https://bugzilla.redhat.com/show_bug.cgi?id=1791790Signed-off-by: NMichal Privoznik <mprivozn@redhat.com>
Reviewed-by: NPeter Krempa <pkrempa@redhat.com>

When fixing [1] I've ran attached reproducer and had it spawn
1024 threads and query capabilities XML in each one of them. This
lead libvirtd to hit the RLIMIT_NOFILE limit which was kind of
expected. What wasn't expected was a subsequent segfault. It
happened because virCPUProbeHost failed and returned NULL. We've
taken the NULL and passed it to virCapabilitiesHostNUMARef()
which dereferenced it. Code inspection showed the same flas in
virQEMUDriverGetHostNUMACaps(), so I'm fixing both places.

1: https://bugzilla.redhat.com/show_bug.cgi?id=1791790Signed-off-by: NMichal Privoznik <mprivozn@redhat.com>
Reviewed-by: NPeter Krempa <pkrempa@redhat.com>

The virCapabilitiesGetNodeInfo() function has the usual return
value semantics for integeres: a negative value means an error,
zero or a positive value means success. However, the function
call done in virCPUProbeHost() doesn't check for the return value
accordingly.
Signed-off-by: NMichal Privoznik <mprivozn@redhat.com>
Reviewed-by: NPeter Krempa <pkrempa@redhat.com>

Don't use ERANGE as it doesn't make much sense in the error message.
Also point out that the reply from qemu was too large which is not
obvious from the original error:

 error: No complete monitor response found in 10485760 bytes: Numerical result out of range

The new message will read:

 error: internal error: QEMU monitor reply exceeds buffer size (10485760 bytes)
Signed-off-by: NPeter Krempa <pkrempa@redhat.com>
Reviewed-by: NJiri Denemark <jdenemar@redhat.com>

libvirt treats 'luks' images as raw+encryption. The logic in
qemuBlockStorageSourceCreateFormat skipped the creation if the requested
image was raw but didn't take into account the encryption.

This manifested itself e.g. when attempting to do a virsh blockcopy with
the following XML:

    <disk type='file' device='disk'>
      <driver name='qemu' type='raw'/>
      <source file='/tmp/enccpy'>
        <encryption format='luks'>
          <secret type='passphrase' uuid='0a81f5b2-8403-7b23-c8d6-21ccc2f80d6f'/>
        </encryption>
      </source>
    </disk>

Where qemu would report the following error:

 unable to execute QEMU command 'blockdev-add': Volume is not in LUKS format

rather than actually formatting the image first.
Signed-off-by: NPeter Krempa <pkrempa@redhat.com>
Reviewed-by: NEric Blake <eblake@redhat.com>

Test that adding a duplicate entry is rejected properly. This also
allows to see the error message of the duplicate key addition in verbose
mode.
Signed-off-by: NPeter Krempa <pkrempa@redhat.com>
Reviewed-by: NMichal Privoznik <mprivozn@redhat.com>

If we get a user reporting this error message being shown it's pretty
useless in terms of actually debugging it since we don't know which hash
and which key are actually subject to the error.

This patch adds a new hash table callback which formats the
user-readable version of the hash key and reports it in the new message
which will look like:

"Duplicate hash table key 'blah'"

That way we will at least have an anchor point where to start the
search.

There are two special implementations of keys which are numeric so we
add specific printer functions for them.
Signed-off-by: NPeter Krempa <pkrempa@redhat.com>
Reviewed-by: NMichal Privoznik <mprivozn@redhat.com>

Use the user-configured name of the bitmap when merging the appropriate
bitmaps for an incremental backup so that the user can see it as
configured. Additionally expose the default bitmap name if nothing is
configured.
Signed-off-by: NPeter Krempa <pkrempa@redhat.com>
Reviewed-by: NDaniel Henrique Barboza <danielhb413@gmail.com>

Pass the exportname as configured when exporting the image via NBD and
fill it with the default if it's not configured.
Signed-off-by: NPeter Krempa <pkrempa@redhat.com>
Reviewed-by: NDaniel Henrique Barboza <danielhb413@gmail.com>
Reviewed-by: NEric Blake <eblake@redhat.com>

If users wish to use different name for exported disks or bitmaps
the new fields allow to do so. Additionally they also document the
current settings.
Signed-off-by: NPeter Krempa <pkrempa@redhat.com>
Reviewed-by: NDaniel Henrique Barboza <danielhb413@gmail.com>
Reviewed-by: NEric Blake <eblake@redhat.com>