Whether a write cache is used or not is a decision that concerns the
user (e.g. the guest device) rather than the backend. It was already
logically part of the BB level as bdrv_move_feature_fields() always kept
it on top of the BDS tree; with this patch, the core of it (the actual
flag and the additional flushes) is also implemented there.

Direct callers of bdrv_open() must pass BDRV_O_CACHE_WB now if bs
doesn't have a BlockBackend attached.
Signed-off-by: NKevin Wolf <kwolf@redhat.com>
Reviewed-by: NMax Reitz <mreitz@redhat.com>

We don't want to silently ignore a flush error.

Also, there is little point in avoiding the flush for writethrough modes
and once WCE is moved to the BB layer, we definitely need the flush here
because bdrv_pwrite() won't involve one any more.
Signed-off-by: NKevin Wolf <kwolf@redhat.com>
Reviewed-by: NMax Reitz <mreitz@redhat.com>

All callers of blk_new_open() either don't rely on the WCE bit set after
blk_new_open() because they explicitly set it anyway, or they pass
BDRV_O_CACHE_WB unconditionally.

This patch changes blk_new_open() so that it always enables writeback
mode and asserts that BDRV_O_CACHE_WB is clear. For those callers that
used to pass BDRV_O_CACHE_WB unconditionally, the flag is removed now.
Signed-off-by: NKevin Wolf <kwolf@redhat.com>
Reviewed-by: NMax Reitz <mreitz@redhat.com>

Signed-off-by: NKevin Wolf <kwolf@redhat.com>
Reviewed-by: NMax Reitz <mreitz@redhat.com>

Signed-off-by: NKevin Wolf <kwolf@redhat.com>
Acked-by: NStefano Stabellini <stefano.stabellini@eu.citrix.com>
Reviewed-by: NMax Reitz <mreitz@redhat.com>

Signed-off-by: NKevin Wolf <kwolf@redhat.com>
Reviewed-by: NMax Reitz <mreitz@redhat.com>

It always only set the BDRV_O_CACHE_WB flag, which is going to go away.
In order to make the next changes more local for better reviewability
this patches expands the macro.
Signed-off-by: NKevin Wolf <kwolf@redhat.com>
Reviewed-by: NMax Reitz <mreitz@redhat.com>

Signed-off-by: NKevin Wolf <kwolf@redhat.com>
Reviewed-by: NMax Reitz <mreitz@redhat.com>

Signed-off-by: NKevin Wolf <kwolf@redhat.com>
Reviewed-by: NMax Reitz <mreitz@redhat.com>

It's like bdrv_parse_cache_flags(), except that writethrough mode isn't
included in the flags, but returned as a separate bool.
Signed-off-by: NKevin Wolf <kwolf@redhat.com>
Reviewed-by: NMax Reitz <mreitz@redhat.com>

This patch introduces block driver that implement recording
and replaying of block devices' operations.
All block completion operations are added to the queue.
Queue is flushed at checkpoints and information about processed requests
is recorded to the log. In replay phase the queue is matched with
events read from the log. Therefore block devices requests are processed
deterministically.
Signed-off-by: NPavel Dovgalyuk <pavel.dovgaluk@ispras.ru>
[ kwolf: Rebased onto modified and already applied part of the series ]
Signed-off-by: NKevin Wolf <kwolf@redhat.com>

This patch fixes error message in saving loop of the asynchronous events queue.
Signed-off-by: NPavel Dovgalyuk <pavel.dovgaluk@ispras.ru>
[ kwolf: Fixed format string to use PRId64 instead of %d ]
Signed-off-by: NKevin Wolf <kwolf@redhat.com>

This patch fixes scheduling of bottom halves when record/replay is enabled.
Now BH are not added to replay queue when asynchronous events are disabled.
This may happen in startup and loadvm/savevm phases of execution.
Signed-off-by: NPavel Dovgalyuk <pavel.dovgaluk@ispras.ru>
Signed-off-by: NKevin Wolf <kwolf@redhat.com>

This patch adds callback for flush request. This callback is responsible
for flushing whole block devices stack. bdrv_flush function does not
proceed to underlying devices. It should be performed by this callback
function, if needed.
Signed-off-by: NPavel Dovgalyuk <pavel.dovgaluk@ispras.ru>
Signed-off-by: NKevin Wolf <kwolf@redhat.com>

It is important that the QEMU luks implementation retains 100%
compatibility with the reference implementation provided by
the combination of the linux kernel dm-crypt module and cryptsetup
userspace tools.

There is a matrix of tests to be performed with different sets
of encryption settings. For each matrix entry, two tests will
be performed. One will create a LUKS image with the cryptsetup
tool and then do I/O with both cryptsetup & qemu-io. The other
will create the image with qemu-img and then again do I/O with
both cryptsetup and qemu-io.

The new I/O test 149 performs interoperability testing between
QEMU and the reference implementation. Such testing inherantly
requires elevated privileges, so to this this the user must have
configured passwordless sudo access. The test will automatically
skip if sudo is not available.

The test has to be run explicitly thus:

    cd tests/qemu-iotests
    ./check -luks 149
Reviewed-by: NEric Blake <eblake@redhat.com>
Signed-off-by: NDaniel P. Berrange <berrange@redhat.com>
Signed-off-by: NKevin Wolf <kwolf@redhat.com>

For a couple of releases we have been warning

  Encrypted images are deprecated
  Support for them will be removed in a future release.
  You can use 'qemu-img convert' to convert your image to an unencrypted one.

This warning was issued by system emulators, qemu-img, qemu-nbd
and qemu-io. Such a broad warning was issued because the original
intention was to rip out all the code for dealing with encryption
inside the QEMU block layer APIs.

The new block encryption framework used for the LUKS driver does
not rely on the unloved block layer API for encryption keys,
instead using the QOM 'secret' object type. It is thus no longer
appropriate to warn about encryption unconditionally.

When the qcow/qcow2 drivers are converted to use the new encryption
framework too, it will be practical to keep AES-CBC support present
for use in qemu-img, qemu-io & qemu-nbd to allow for interoperability
with older QEMU versions and liberation of data from existing encrypted
qcow2 files.

This change moves the warning out of the generic block code and
into the qcow/qcow2 drivers. Further, the warning is set to only
appear when running the system emulators, since qemu-img, qemu-io,
qemu-nbd are expected to support qcow2 encryption long term now that
the maint burden has been eliminated.
Signed-off-by: NDaniel P. Berrange <berrange@redhat.com>
Reviewed-by: NEric Blake <eblake@redhat.com>
Signed-off-by: NKevin Wolf <kwolf@redhat.com>

Add a block driver that is capable of supporting any full disk
encryption format. This utilizes the previously added block
encryption code, and at this time supports the LUKS format.

The driver code is capable of supporting any format supported
by the QCryptoBlock module, so it registers one block driver
for each format. This patch only registers the "luks" driver
since the "qcow" driver is there only for back-compatibility
with existing qcow built-in encryption.

New LUKS compatible volumes can be formatted using qemu-img
with defaults for all settings.

$ qemu-img create --object secret,data=123456,id=sec0 \
      -f luks -o key-secret=sec0 demo.luks 10G

Alternatively the cryptographic settings can be explicitly
set

$ qemu-img create --object secret,data=123456,id=sec0 \
      -f luks -o key-secret=sec0,cipher-alg=aes-256,\
                 cipher-mode=cbc,ivgen-alg=plain64,hash-alg=sha256 \
      demo.luks 10G

And query its size

$ qemu-img info demo.img
image: demo.img
file format: luks
virtual size: 10G (10737418240 bytes)
disk size: 132K
encrypted: yes

Note that it was not necessary to provide the password
when querying info for the volume. The password is only
required when performing I/O on the volume

All volumes created by this new 'luks' driver should be
capable of being opened by the kernel dm-crypt driver.

The only algorithms listed in the LUKS spec that are
not currently supported by this impl are sha512 and
ripemd160 hashes and cast6 cipher.
Reviewed-by: NEric Blake <eblake@redhat.com>
Signed-off-by: NDaniel P. Berrange <berrange@redhat.com>
[ kwolf - Added #include to resolve conflict with da34e65c ]
Signed-off-by: NKevin Wolf <kwolf@redhat.com>

Add a 'log' method to iotests.py which prints messages to
stdout, with optional filtering of data. Port over some
standard filters already present in the shell common.filter
code to be usable in python too.
Reviewed-by: NEric Blake <eblake@redhat.com>
Signed-off-by: NDaniel P. Berrange <berrange@redhat.com>
Signed-off-by: NKevin Wolf <kwolf@redhat.com>

The iotests.py helper provides a main() method for running
tests via the python unit test framework. Not all tests
will want to use this, so refactor it to split the testing
of compatible formats and platforms into separate helper
methods
Reviewed-by: NEric Blake <eblake@redhat.com>
Signed-off-by: NDaniel P. Berrange <berrange@redhat.com>
Signed-off-by: NKevin Wolf <kwolf@redhat.com>

The python I/O tests helper for running qemu-img/qemu-io
setup stdout to be captured to a pipe, but left stderr
untouched. As a result, if something failed in qemu-img/
qemu-io, data written to stderr would get output directly
and not line up with data on the test stdout due to
buffering.  If we explicitly redirect stderr to the same
pipe as stdout, things are much clearer when they go
wrong.
Reviewed-by: NEric Blake <eblake@redhat.com>
Signed-off-by: NDaniel P. Berrange <berrange@redhat.com>
Signed-off-by: NKevin Wolf <kwolf@redhat.com>

The qemu-img/qemu-io tools prompt for disk encryption passwords
regardless of whether any are actually required. Adding a check
on bdrv_key_required() avoids this prompt for disk formats which
have been converted to the QCryptoSecret APIs.

This is just a temporary hack to ensure the block I/O tests
continue to work after each patch, since the last patch will
completely delete all the password prompting code.
Reviewed-by: NEric Blake <eblake@redhat.com>
Signed-off-by: NDaniel P. Berrange <berrange@redhat.com>
Signed-off-by: NKevin Wolf <kwolf@redhat.com>

When opening an image it is useful to know whether the caller
intends to perform I/O on the image or not. In the case of
encrypted images this will allow the block driver to avoid
having to prompt for decryption keys when we merely want to
query header metadata about the image. eg qemu-img info

This flag is enforced at the top level only, since even if
we don't want todo I/O on the 'qcow2' file payload, the
underlying 'file' driver will still need todo I/O to read
the qcow2 header, for example.
Reviewed-by: NEric Blake <eblake@redhat.com>
Signed-off-by: NDaniel P. Berrange <berrange@redhat.com>
Signed-off-by: NKevin Wolf <kwolf@redhat.com>

bdrv_query_blk_stats() does not need access to all of BlockStats,
BlockDeviceStats is enough and is what this function is actually
supposed to fill.
Signed-off-by: NMax Reitz <mreitz@redhat.com>
Signed-off-by: NKevin Wolf <kwolf@redhat.com>

This is the only instance of bdrv_query_blk_stats() accessing anything
in the BlockStats structure other than s->stats, so let us move it to
its caller (where it makes just as much sense) allowing us to make
bdrv_query_blk_stats() take a pointer to the BlockDeviceStats instead of
BlockStats.
Signed-off-by: NMax Reitz <mreitz@redhat.com>
Signed-off-by: NKevin Wolf <kwolf@redhat.com>

Using heap instead of stack for better safety.
Signed-off-by: NPeter Xu <peterx@redhat.com>
Reviewed-by: NEric Blake <eblake@redhat.com>
Reviewed-by: NMarkus Armbruster <armbru@redhat.com>
Signed-off-by: NKevin Wolf <kwolf@redhat.com>

Fix two places to use literal printf format when possible.
Signed-off-by: NPeter Xu <peterx@redhat.com>
Reviewed-by: NEric Blake <eblake@redhat.com>
Reviewed-by: NMarkus Armbruster <armbru@redhat.com>
Signed-off-by: NKevin Wolf <kwolf@redhat.com>

The function is unused since commit f21d96d0 ('block: Use BdrvChild in
BlockBackend').
Signed-off-by: NKevin Wolf <kwolf@redhat.com>
Reviewed-by: NEric Blake <eblake@redhat.com>

Mac OS X can be picky when it comes to allowing the user
to use physical devices in QEMU. Most mounted volumes
appear to be off limits to QEMU. If an issue is detected,
a message is displayed showing the user how to unmount a
volume. Now QEMU uses both CD and DVD media.
Signed-off-by: NJohn Arbuckle <programmingkidx@gmail.com>
Signed-off-by: NKevin Wolf <kwolf@redhat.com>

Writethrough mode is going to become a BlockBackend feature rather than
a BDS one, so forbid it in places where we won't be able to support it
when the code finally matches the envisioned design.

We only allowed setting the cache mode of non-root nodes after the 2.5
release, so we're still free to make this change.

The target of block jobs is now always opened in a writeback mode
because it doesn't have a BlockBackend attached. This makes more sense
anyway because block jobs know when to flush. If the graph is modified
on job completion, the original cache mode moves to the new root, so
for the guest device writethough always stays enabled if it was
configured this way.
Signed-off-by: NKevin Wolf <kwolf@redhat.com>
Reviewed-by: NEric Blake <eblake@redhat.com>

First of all, we're generally not writing to backing files, but when we
do, it's in the context of block jobs which know very well when to flush
the image.
Signed-off-by: NKevin Wolf <kwolf@redhat.com>
Reviewed-by: NEric Blake <eblake@redhat.com>

The WCE bit is a frontend property and should not be part of the backend
configuration. This is especially important because the same BDS can be
used by different users with different WCE requirements.
Signed-off-by: NKevin Wolf <kwolf@redhat.com>

This patch changes dirty bitmaps from following a BlockBackend in graph
changes to sticking with the node they were created at. For the full
discussion, read the following mailing list thread:

  [Qemu-block] block: Dirty bitmaps and COR in bdrv_move_feature_fields()
  https://lists.nongnu.org/archive/html/qemu-block/2016-02/msg00745.html

In summary, the justification for this change is:

* When moving the dirty bitmap to the top of the tree was introduced in
  bdrv_append() in commit a9fc4408, it didn't actually have any effect
  because there could never be a bitmap in use when bdrv_append() was
  called (op blockers would prevent this). This is still true today for
  all internal uses of dirty bitmaps.

* Support for user-defined dirty bitmaps was introduced in 2.4, but we
  discouraged users from using it because we didn't consider it ready
  yet.

  Moreover, in 2.5, the bdrv_swap() removal introduced a bug that left
  dangling pointers if a dirty bitmap was present (the anchors of the
  dirty bitmap were swapped, but the back link in the first element
  wasn't updated), so it didn't even work correctly.

* block-dirty-bitmap-add takes an arbitrary node name, even if no
  BlockBackend is attached. This suggests that it is a node level
  operation and not a BlockBackend one. Consequently, there is no reason
  for dirty bitmaps to stay with a BlockBackend that was attached to the
  node they were created for.

* It was suggested that block-dirty-bitmap-add could track the node if a
  node name was specified, and track the BlockBackend if the device name
  was specified. This would however be inconsistent with other QMP
  commands. Commands that accept both device and node names currently
  interpret the device name just as an alias for the current root node
  of that BlockBackend.

* Dirty bitmaps have a name that is only unique amongst the bitmaps in a
  specific node. Moving bitmaps could lead to name clashes. Automatic
  renaming would involve too much magic.

* Persistent bitmaps are stored in a specific node. Moving them around
  automatically might be at least surprising, but it would probably also
  become a real problem because that would have to happen atomically
  without the management tool knowing of the operation.

At the end of the day it seems to be very clear that it was a mistake to
include dirty bitmaps in bdrv_move_feature_fields(). The functionality
of moving bitmaps and/or attaching them to a BlockBackend instead will
probably be needed, but it should be done with a new explicit QMP
command or option.
Signed-off-by: NKevin Wolf <kwolf@redhat.com>
Reviewed-by: NEric Blake <eblake@redhat.com>

Ever since we first introduced bdrv_append() in commit 8802d1fd ('qapi:
Introduce blockdev-group-snapshot-sync command'), the copy-on-read flag
was moved to the new top layer when taking a snapshot. The only problem
is that it doesn't make a whole lot of sense.

The use case for manually enabled CoR is to avoid reading data twice
from a slow remote image, so we want to save it to a local overlay, say
an ISO image accessed via HTTP to a local qcow2 overlay. When taking a
snapshot, we end up with a backing chain like this:

    http <- local.qcow2 <- snap_overlay.qcow2

There is no point in doing CoR from local.qcow2 into snap_overlay.qcow2,
we just want to keep copying data from the remote source into
local.qcow2.

The other use case of CoR is in the context of streaming, which isn't
very interesting for bdrv_move_feature_fields() because op blockers
prevent this combination.

This patch makes the copy-on-read flag stay on the image for which it
was originally set and prevents it from being propagated to the new
overlay. It is no longer intended to move CoR to the BlockBackend level.
In order for this to make sense, we also need to keep the respective
image read-write.

As a side effect of these changes, creating a live snapshot image (as
opposed to using an existing externally created one) on top of a COR
block device works now. It used to fail because it tried to open its
backing file both read-only and with COR.
Signed-off-by: NKevin Wolf <kwolf@redhat.com>
Reviewed-by: NEric Blake <eblake@redhat.com>

The call in hmp_drive_del() is dead code because blk_remove_bs() is
called a few lines above. The only other remaining user is
bdrv_delete(), which only abuses bdrv_make_anon() to remove it from the
named nodes list. This path inlines the list entry removal into
bdrv_delete() and removes bdrv_make_anon().
Signed-off-by: NKevin Wolf <kwolf@redhat.com>
Reviewed-by: NEric Blake <eblake@redhat.com>
Reviewed-by: NMax Reitz <mreitz@redhat.com>

# gpg: Signature made Tue 29 Mar 2016 01:48:09 BST using RSA key ID C0DE3057
# gpg: Good signature from "Jeffrey Cody <jcody@redhat.com>"
# gpg:                 aka "Jeffrey Cody <jeff@codyprime.org>"
# gpg:                 aka "Jeffrey Cody <codyprime@gmail.com>"

* remotes/cody/tags/block-pull-request:
  qemu-iotests: add no-op streaming test
  qemu-iotests: fix test_stream_partial()
  block: never cancel a streaming job without running stream_complete()
Signed-off-by: NPeter Maydell <peter.maydell@linaro.org>

slirp updates

# gpg: Signature made Tue 29 Mar 2016 00:16:05 BST using RSA key ID FB6B2F1D
# gpg: Good signature from "Samuel Thibault <samuel.thibault@gnu.org>"
# gpg:                 aka "Samuel Thibault <sthibault@debian.org>"
# gpg:                 aka "Samuel Thibault <samuel.thibault@inria.fr>"
# gpg:                 aka "Samuel Thibault <samuel.thibault@labri.fr>"
# gpg:                 aka "Samuel Thibault <samuel.thibault@ens-lyon.org>"
# gpg: WARNING: This key is not certified with sufficiently trusted signatures!
# gpg:          It is not certain that the signature belongs to the owner.
# Primary key fingerprint: 900C B024 B679 31D4 0F82  304B D017 8C76 7D06 9EE6
#      Subkey fingerprint: F632 74CD C630 0873 CB3D  29D9 E3E5 1CE8 FB6B 2F1D

* remotes/thibault/tags/samuel-thibault:
  Rework ipv6 options
  Use C99 flexible array instead of 1-byte trailing array
  Avoid embedding struct mbuf in other structures
  slirp: send icmp6 errors when UDP send failed
  slirp: Fix memory leak on small incoming ipv4 packet
Signed-off-by: NPeter Maydell <peter.maydell@linaro.org>

VFIO updates 2016-03-28

 - Use 128bit math to avoid asserts with IOMMU regions (Bandan Das)

# gpg: Signature made Mon 28 Mar 2016 23:16:52 BST using RSA key ID 3BB08B22
# gpg: Good signature from "Alex Williamson <alex.williamson@redhat.com>"
# gpg:                 aka "Alex Williamson <alex@shazbot.org>"
# gpg:                 aka "Alex Williamson <alwillia@redhat.com>"
# gpg:                 aka "Alex Williamson <alex.l.williamson@gmail.com>"

* remotes/awilliam/tags/vfio-update-20160328.0:
  vfio: convert to 128 bit arithmetic calculations when adding mem regions
Signed-off-by: NPeter Maydell <peter.maydell@linaro.org>

Rename the recently-added ip6-foo options into ipv6-foo options, to make
them coherent with other ipv6 options.

Also rework the documentation.
Signed-off-by: NSamuel Thibault <samuel.thibault@ens-lyon.org>
Reviewed-by: NEric Blake <eblake@redhat.com>

Signed-off-by: NPeter Maydell <peter.maydell@linaro.org>
Signed-off-by: NSamuel Thibault <samuel.thibault@ens-lyon.org>

vfio_listener_region_add for a iommu mr results in
an overflow assert since iommu memory region is initialized
with UINT64_MAX. Convert calculations to 128 bit arithmetic
for iommu memory regions and let int128_get64 assert for non iommu
regions if there's an overflow.
Suggested-by: NAlex Williamson <alex.williamson@redhat.com>
Signed-off-by: NBandan Das <bsd@redhat.com>
[missed (end - 1) on 2nd trace call, move llsize closer to use]
Signed-off-by: NAlex Williamson <alex.williamson@redhat.com>