Signed-off-by: NKevin Wolf <kwolf@redhat.com>

Signed-off-by: NKevin Wolf <kwolf@redhat.com>

Test that preallocating metadata results in a somewhat larger qcow2
file, but preallocating data only affects the disk usage of the data
file and the qcow2 file stays small.
Signed-off-by: NKevin Wolf <kwolf@redhat.com>

Provide an option to force QEMU to always keep the external data file
consistent as a standalone read-only raw image.

At the moment, this means making sure that write_zeroes requests are
forwarded to the data file instead of just updating the metadata, and
checking that no backing file is used.
Signed-off-by: NKevin Wolf <kwolf@redhat.com>

Rather than requiring that the external data file node is passed
explicitly when creating the qcow2 node, store the filename in the
designated header extension during .bdrv_create and read it from there
as a default during .bdrv_open.
Signed-off-by: NKevin Wolf <kwolf@redhat.com>

qcow2_alloc_compressed_cluster_offset() used to return the cluster
offset for success and 0 for error. This doesn't only conflict with 0 as
a valid host offset, but also loses the error code.

Similar to the change made to qcow2_alloc_cluster_offset() for
uncompressed clusters in commit 148da7ea, make the function return
0/-errno and return the allocated cluster offset in a by-reference
parameter.
Signed-off-by: NKevin Wolf <kwolf@redhat.com>

This adds basic constants, struct fields and helper function for
external data file support to the implementation.

QCOW2_INCOMPAT_MASK and QCOW2_AUTOCLEAR_MASK are not updated yet so that
opening images with an external data file still fails (we don't handle
them correctly yet).
Signed-off-by: NKevin Wolf <kwolf@redhat.com>

Signed-off-by: NKevin Wolf <kwolf@redhat.com>
Reviewed-by: NEric Blake <eblake@redhat.com>

Various sed regexp from common.filter use sed GNU extensions.
Instead of spending time to write these regex to be POSIX compliant,
verify the GNU sed is available and use it.
Signed-off-by: NPhilippe Mathieu-Daudé <philmd@redhat.com>
Signed-off-by: NKevin Wolf <kwolf@redhat.com>

Bash is not always installed as /bin/bash. In particular on OpenBSD,
the package installs it in /usr/local/bin.
Use the 'env' shebang to search bash in the $PATH.

Patch created mechanically by running:

  $ git grep -lE '#! ?/bin/bash' -- tests/qemu-iotests \
    | while read f; do \
      sed -i 's|^#!.\?/bin/bash$|#!/usr/bin/env bash|' $f; \
    done
Reviewed-by: NEric Blake <eblake@redhat.com>
Signed-off-by: NPhilippe Mathieu-Daudé <philmd@redhat.com>
Signed-off-by: NKevin Wolf <kwolf@redhat.com>

Some test cases require specific formats. The method decorator
skip_if_unsupported() checks if requested formats are whitelisted.
The test #139 was selected for a sample output, after running
$ ./check -qcow2 131-140

137 3s ...
138 0s ...
139 2s ...
    [case not run] testBlkDebug (__main__.TestBlockdevDel): formats ['blkdebug'] are not whitelisted
    [case not run] testBlkVerify (__main__.TestBlockdevDel): formats ['blkverify'] are not whitelisted
    [case not run] testQuorum (__main__.TestBlockdevDel): formats ['quorum'] are not whitelisted
140 0s ...
Not run: 131 135 136
Some cases not run in: 139
Passed all 7 tests
Signed-off-by: NAndrey Shinkevich <andrey.shinkevich@virtuozzo.com>
Signed-off-by: NKevin Wolf <kwolf@redhat.com>

Supported formats listed by 'qemu' may differ from those listed by
'qemu-img' due to whitelists. Some test cases require specific formats
that may be used with qemu. They can be inquired directly by running
'qemu -drive format=help'. The response takes whitelists into account.
The method supported_formats() serves for that. The method decorator
skip_if_unsupported() checks if all requested formats are whitelisted.
If not, the test case will be skipped. That has been implemented in
the 'check' file in the way similar to the 'test notrun' mechanism.
Suggested-by: NRoman Kagan <rkagan@virtuozzo.com>
Suggested-by: NVladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
Suggested-by: NKevin Wolf <kwolf@redhat.com>
Signed-off-by: NAndrey Shinkevich <andrey.shinkevich@virtuozzo.com>
Signed-off-by: NKevin Wolf <kwolf@redhat.com>

Replace the binary mode with the default text one when *.notrun
files are opened for skipped tests. That change is made for the
compatibility with Python 3 which returns error otherwise.
Signed-off-by: NKevin Wolf <kwolf@redhat.com>
Signed-off-by: NAndrey Shinkevich <andrey.shinkevich@virtuozzo.com>
Signed-off-by: NKevin Wolf <kwolf@redhat.com>

Test 238 does not require the kvm accelerator.  Using the qtest
accelerator allows the test to run in both non-kvm and non-tcg
environments.

iotests.VM implicitly uses the qtest accelerator and is really the class
that this test should be using.  Switch to that instead of
qemu.QEMUMachine.
Suggested-by: NThomas Huth <thuth@redhat.com>
Signed-off-by: NStefan Hajnoczi <stefanha@redhat.com>
Signed-off-by: NKevin Wolf <kwolf@redhat.com>

When we run "certtool 2>&1 | head -1" the latter command is likely to
complete and exit before certtool has written everything it wants to
stderr. In at least the RHEL-7 gnutls 3.3.29 this causes certtool to
quit with broken pipe before it has finished writing the desired
output file to disk. This causes non-deterministic failures of the
iotest 233 because the certs are sometimes zero length files.
If certtool fails the "head -1" means we also lose any useful error
message it would have printed.

Thus this patch gets rid of the pipe and post-processes the output in a
more flexible & reliable manner.
Reported-by: NThomas Huth <thuth@redhat.com>
Signed-off-by: NDaniel P. Berrangé <berrange@redhat.com>
Message-Id: <20190220145819.30969-3-berrange@redhat.com>
Reviewed-by: NEric Blake <eblake@redhat.com>
Signed-off-by: NEric Blake <eblake@redhat.com>

If we abort the iotest early the server.log file might contain useful
information for diagnosing the problem. Ensure its contents are
displayed in this case.
Reviewed-by: NEric Blake <eblake@redhat.com>
Signed-off-by: NDaniel P. Berrangé <berrange@redhat.com>
Message-Id: <20190220145819.30969-2-berrange@redhat.com>
[eblake: fix shell quoting]
Signed-off-by: NEric Blake <eblake@redhat.com>

The data type for bytes in Python 3 differs from the one in Python 2.
The type cast that is compatible with both versions was applied.
Signed-off-by: NNir Soffer <nsoffer@redhat.com>
Signed-off-by: NAndrey Shinkevich <andrey.shinkevich@virtuozzo.com>
Reported-by: NKevin Wolf <kwolf@redhat.com>
Message-Id: <1551197495-24425-1-git-send-email-andrey.shinkevich@virtuozzo.com>
Reviewed-by: NEric Blake <eblake@redhat.com>
Signed-off-by: NEric Blake <eblake@redhat.com>

VDI keeps the whole bitmap in memory, and the maximum size (which is
tested here) is 2 GB.  This may not be available on all machines, and it
rarely is available when running a 32 bit build.

Fix this by making VM.run_job() return the error string if an error
occurred, and checking whether that contains "Could not allocate bmap"
in 211.  If so, the test is skipped.
Signed-off-by: NMax Reitz <mreitz@redhat.com>
Message-id: 20190218180646.30282-1-mreitz@redhat.com
Reviewed-by: NEric Blake <eblake@redhat.com>
Reviewed-by: NPhilippe Mathieu-Daudé <philmd@redhat.com>
Signed-off-by: NMax Reitz <mreitz@redhat.com>

The previous patch includes the LUKS payload overhead into the qemu-img
measure calculation for qcow2.  Update qemu-iotests 178 to exercise this
new code path.
Reviewed-by: NMax Reitz <mreitz@redhat.com>
Reviewed-by: NPhilippe Mathieu-Daudé <philmd@redhat.com>
Signed-off-by: NStefan Hajnoczi <stefanha@redhat.com>
Message-id: 20190218104525.23674-3-stefanha@redhat.com
Signed-off-by: NMax Reitz <mreitz@redhat.com>

This follows what qmp() does, so the output will correspond to the
actual QMP command.
Signed-off-by: NMax Reitz <mreitz@redhat.com>
Message-id: 20190210145736.1486-11-mreitz@redhat.com
Signed-off-by: NMax Reitz <mreitz@redhat.com>

Adding a telnet monitor for no real purpose on a fixed port is not so
great.  Just use a null monitor instead.
Signed-off-by: NMax Reitz <mreitz@redhat.com>
Reviewed-by: NJohn Snow <jsnow@redhat.com>
Message-id: 20190210145736.1486-10-mreitz@redhat.com
Signed-off-by: NMax Reitz <mreitz@redhat.com>

8908b253 has implemented filtering of
remote paths for NFS, but forgot SSH.  This patch takes care of that.
Signed-off-by: NMax Reitz <mreitz@redhat.com>
Reviewed-by: NJohn Snow <jsnow@redhat.com>
Message-id: 20190210145736.1486-9-mreitz@redhat.com
Signed-off-by: NMax Reitz <mreitz@redhat.com>

filter_qmp_testfiles() currently filters the filename only for specific
keys.  However, there are more keys that take filenames (such as
block-commit's @top and @base, or ssh's @path), and it does not make
sense to list them all here.  "$TEST_DIR/$PID-" should have enough
entropy not to appear anywhere randomly.
Signed-off-by: NMax Reitz <mreitz@redhat.com>
Reviewed-by: NJohn Snow <jsnow@redhat.com>
Message-id: 20190210145736.1486-8-mreitz@redhat.com
Signed-off-by: NMax Reitz <mreitz@redhat.com>

On Python 2.x, strings are not always unicode strings.  This function
checks whether a given value is a plain string, or a unicode string (if
there is a difference).
Signed-off-by: NMax Reitz <mreitz@redhat.com>
Reviewed-by: NJohn Snow <jsnow@redhat.com>
Message-id: 20190210145736.1486-7-mreitz@redhat.com
Signed-off-by: NMax Reitz <mreitz@redhat.com>

Fixes: 08fcd611Signed-off-by: NMax Reitz <mreitz@redhat.com>
Reviewed-by: NJohn Snow <jsnow@redhat.com>
Message-id: 20190210145736.1486-6-mreitz@redhat.com
Signed-off-by: NMax Reitz <mreitz@redhat.com>

With IMGOPTSSYNTAX, $TEST_IMG is useless for this test (it only tests
the file-posix protocol driver).  Therefore, if $TEST_IMG_FILE is set,
use that instead.

Because this test requires the file protocol, $TEST_IMG_FILE will always
be set if $IMGOPTSSYNTAX is true.
Signed-off-by: NMax Reitz <mreitz@redhat.com>
Reviewed-by: NJohn Snow <jsnow@redhat.com>
Message-id: 20190210145736.1486-5-mreitz@redhat.com
Signed-off-by: NMax Reitz <mreitz@redhat.com>

This test creates no such file.
Signed-off-by: NMax Reitz <mreitz@redhat.com>
Reviewed-by: NEric Blake <eblake@redhat.com>
Reviewed-by: NJohn Snow <jsnow@redhat.com>
Message-id: 20190210145736.1486-4-mreitz@redhat.com
Signed-off-by: NMax Reitz <mreitz@redhat.com>

math.ceil() returns an integer on Python 3.x, but a float on Python 2.x.
range() always needs integers, so we need an explicit conversion on 2.x
(which does not hurt on 3.x).

It is not quite clear whether we want to support Python 2.x for any
prolonged time, but this may as well be fixed along with the other
issues some iotests have right now.
Signed-off-by: NMax Reitz <mreitz@redhat.com>
Reviewed-by: NJohn Snow <jsnow@redhat.com>
Reviewed-by: NPhilippe Mathieu-Daudé <philmd@redhat.com>
Message-id: 20190210145736.1486-3-mreitz@redhat.com
Signed-off-by: NMax Reitz <mreitz@redhat.com>

A previous commit removed the default filters for qmp_log with the
intention to make them explicit; but this happened only for test 206.
There are more tests (for more exotic image formats than qcow2) which
require the filename filter, though.

Note that 237 is still broken for Python 2.x, which is fixed in the next
commit.

Fixes: f8ca8609Signed-off-by: NMax Reitz <mreitz@redhat.com>
Reviewed-by: NJohn Snow <jsnow@redhat.com>
Message-id: 20190210145736.1486-2-mreitz@redhat.com
Signed-off-by: NMax Reitz <mreitz@redhat.com>

Signed-off-by: NMax Reitz <mreitz@redhat.com>
Message-id: 20190201192935.18394-32-mreitz@redhat.com
Signed-off-by: NMax Reitz <mreitz@redhat.com>

Currently, BlockDriver.bdrv_refresh_filename() is supposed to both
refresh the filename (BDS.exact_filename) and set BDS.full_open_options.
Now that we have generic code in the central bdrv_refresh_filename() for
creating BDS.full_open_options, we can drop the latter part from all
BlockDriver.bdrv_refresh_filename() implementations.

This also means that we can drop all of the existing default code for
this from the global bdrv_refresh_filename() itself.

Furthermore, we now have to call BlockDriver.bdrv_refresh_filename()
after having set BDS.full_open_options, because the block driver's
implementation should now be allowed to depend on BDS.full_open_options
being set correctly.

Finally, with this patch we can drop the @options parameter from
BlockDriver.bdrv_refresh_filename(); also, add a comment on this
function's purpose in block/block_int.h while touching its interface.

This completely obsoletes blklogwrite's implementation of
.bdrv_refresh_filename().
Signed-off-by: NMax Reitz <mreitz@redhat.com>
Message-id: 20190201192935.18394-25-mreitz@redhat.com
Signed-off-by: NMax Reitz <mreitz@redhat.com>

Instead of having every block driver which implements
bdrv_refresh_filename() copy all of the strong runtime options over to
bs->full_open_options, implement this process generically in
bdrv_refresh_filename().

This patch only adds this new generic implementation, it does not remove
the old functionality. This is done in a follow-up patch.

With this patch, some superfluous information (that should never have
been there) may be removed from some JSON filenames, as can be seen in
the change to iotests 110's and 228's reference outputs.
Signed-off-by: NMax Reitz <mreitz@redhat.com>
Message-id: 20190201192935.18394-24-mreitz@redhat.com
Signed-off-by: NMax Reitz <mreitz@redhat.com>

Test 110 tests relative backing filenames for complex BDS trees.  Now
that the originally supposedly failing test passes, let us add a new
failing test: Quorum can never work automatically (without detecting
whether all child nodes have the same base directory, but that would be
rather inconsistent behavior).
Signed-off-by: NMax Reitz <mreitz@redhat.com>
Reviewed-by: NAlberto Garcia <berto@igalia.com>
Message-id: 20190201192935.18394-21-mreitz@redhat.com
Signed-off-by: NMax Reitz <mreitz@redhat.com>

bdrv_get_full_backing_filename_from_filename() breaks down when it comes
to JSON filenames. Using bdrv_dirname() as the basis is better because
since we have BDS, we can descend through the BDS tree to the protocol
layer, which gives us a greater probability of finding a non-JSON name;
also, bdrv_dirname() is more correct as it allows block drivers to
override the generation of that directory name in a protocol-specific
way.

We still need to keep bdrv_get_full_backing_filename_from_filename(),
though, because it has valid callers which need it during image creation
when no BDS is available yet.

This makes a test case in qemu-iotest 110, which was supposed to fail,
work. That is actually good, but we need to change the reference output
(and the comment in 110) accordingly.
Signed-off-by: NMax Reitz <mreitz@redhat.com>
Reviewed-by: NAlberto Garcia <berto@igalia.com>
Message-id: 20190201192935.18394-20-mreitz@redhat.com
Signed-off-by: NMax Reitz <mreitz@redhat.com>

Signed-off-by: NMax Reitz <mreitz@redhat.com>
Message-id: 20190201192935.18394-9-mreitz@redhat.com
Signed-off-by: NMax Reitz <mreitz@redhat.com>

This function queries a node; since we cannot do that right now, it
executes query-named-block-nodes and returns the matching node's object.
Signed-off-by: NMax Reitz <mreitz@redhat.com>
Reviewed-by: NJohn Snow <jsnow@redhat.com>
Reviewed-by: NAlberto Garcia <berto@igalia.com>
Reviewed-by: NEric Blake <eblake@redhat.com>
Message-id: 20190201192935.18394-8-mreitz@redhat.com
Signed-off-by: NMax Reitz <mreitz@redhat.com>

Signed-off-by: NMax Reitz <mreitz@redhat.com>
Message-id: 20190201192935.18394-7-mreitz@redhat.com
Reviewed-by: NEric Blake <eblake@redhat.com>
Signed-off-by: NMax Reitz <mreitz@redhat.com>

Basically, bdrv_refresh_filename() should respect all children of a
BlockDriverState. However, generally those children are driver-specific,
so this function cannot handle the general case. On the other hand,
there are only few drivers which use other children than @file and
@backing (that being vmdk, quorum, and blkverify).

Most block drivers only use @file and/or @backing (if they use any
children at all). Both can be implemented directly in
bdrv_refresh_filename.

The user overriding the file's filename is already handled, however, the
user overriding the backing file is not. If this is done, opening the
BDS with the plain filename of its file will not be correct, so we may
not set bs->exact_filename in that case.

iotest 051 contains test cases for overriding the backing file, and so
its output changes with this patch applied.
Signed-off-by: NMax Reitz <mreitz@redhat.com>
Reviewed-by: NAlberto Garcia <berto@igalia.com>
Message-id: 20190201192935.18394-6-mreitz@redhat.com
Signed-off-by: NMax Reitz <mreitz@redhat.com>

This is a simple move of Python code that wraps common QEMU
functionality, and are used by a number of different tests
and scripts.

By treating that code as a real Python module, we can more easily:
 * reuse code
 * have a proper place for the module's own unittests
 * apply a more consistent style
 * generate documentation
Signed-off-by: NCleber Rosa <crosa@redhat.com>
Reviewed-by: NCaio Carrara <ccarrara@redhat.com>
Reviewed-by: NStefan Hajnoczi <stefanha@redhat.com>
Message-Id: <20190206162901.19082-2-crosa@redhat.com>
Signed-off-by: NCleber Rosa <crosa@redhat.com>

Since qemu currently doesn't flush persistent bitmaps to disk until
shutdown (which might be MUCH later), it's useful if 'query-block'
at least shows WHICH bitmaps will (eventually) make it to persistent
storage.  Update affected iotests.
Signed-off-by: NEric Blake <eblake@redhat.com>
Reviewed-by: NVladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
Reviewed-by: NJohn Snow <jsnow@redhat.com>
Message-id: 20190204210512.27458-1-eblake@redhat.com
Signed-off-by: NJohn Snow <jsnow@redhat.com>