Block patches

# gpg: Signature made Thu 23 Oct 2014 18:56:05 BST using RSA key ID C88F2FD6
# gpg: Good signature from "Kevin Wolf <kwolf@redhat.com>"

* remotes/kevin/tags/for-upstream: (32 commits)
  qemu-img: Print error if check failed
  block: char devices on FreeBSD are not behind a pager
  iotests: Add test for qcow2 L1 table update
  qcow2: Do not overflow when writing an L1 sector
  iotests: Add test for map commands
  qemu-io: Respect early image end for map
  block: Respect underlying file's EOF
  docs/qcow2: Limit refcount_order to [0, 6]
  docs/qcow2: Correct refcount_block_entries
  qcow2: Drop REFCOUNT_SHIFT
  iotests: Add test for potentially damaging repairs
  iotests: Fix test outputs
  qcow2: Clean up after refcount rebuild
  qcow2: Rebuild refcount structure during check
  qcow2: Do not perform potentially damaging repairs
  qcow2: Fix refcount blocks beyond image end
  qcow2: Reuse refcount table in calculate_refcounts()
  qcow2: Let inc_refcounts() resize the reftable
  qcow2: Let inc_refcounts() return -errno
  qcow2: Split fail code in L1 and L2 checks
  ...
Signed-off-by: NPeter Maydell <peter.maydell@linaro.org>

* mreitz/block:
  qemu-img: Print error if check failed
  block: char devices on FreeBSD are not behind a pager

Currently, if bdrv_check() fails either by returning -errno or having
check_errors set, qemu-img check just exits with 1 after having told the
user that there were no errors on the image. This is bad.

Instead of printing the check result if there were internal errors which
were so bad that bdrv_check() could not even complete with 0 as a return
value, qemu-img check should inform the user about the error.
Signed-off-by: NMax Reitz <mreitz@redhat.com>
Reviewed-by: NEric Blake <eblake@redhat.com>
Reviewed-by: NMarkus Armbruster <armbru@redhat.com>

QMP patches

# gpg: Signature made Thu 23 Oct 2014 16:05:52 BST using RSA key ID E24ED5A7
# gpg: Good signature from "Luiz Capitulino <lcapitulino@gmail.com>"

* remotes/qmp-unstable/tags/for-upstream:
  monitor: delete device_del_bus_completion
  monitor: add del completion for peripheral device
  qdev: add qdev_build_hotpluggable_device_list helper
  MAINTAINERS: add entry for qobject files
  dump: Turn some functions to void to make code cleaner
  dump: Propagate errors into qmp_dump_guest_memory()
  virtio-balloon: Tweak recent fix for integer overflow
Signed-off-by: NPeter Maydell <peter.maydell@linaro.org>

Introduce a new flag to mark devices that require requests to be aligned and
replace the usage of BDRV_O_NOCACHE and O_DIRECT with this flag when
appropriate.

If a character device is used as a backend on a FreeBSD host set this flag
unconditionally.
Signed-off-by: NRoger Pau Monné <roger.pau@citrix.com>
Reviewed-by: NMax Reitz <mreitz@redhat.com>
Cc: Kevin Wolf <kwolf@redhat.com>
Cc: Stefan Hajnoczi <stefanha@redhat.com>
Signed-off-by: NMax Reitz <mreitz@redhat.com>

Updating the L1 table should not result in random data being written.
This adds a test for that.
Signed-off-by: NMax Reitz <mreitz@redhat.com>
Reviewed-by: NEric Blake <eblake@redhat.com>
Signed-off-by: NKevin Wolf <kwolf@redhat.com>

While writing an L1 table sector, qcow2_write_l1_entry() copies the
respective range from s->l1_table to the local "buf" array. The size of
s->l1_table does not have to be a multiple of L1_ENTRIES_PER_SECTOR;
thus, limit the index which is used for copying all entries to the L1
size.

Cc: qemu-stable@nongnu.org
Signed-off-by: NMax Reitz <mreitz@redhat.com>
Reviewed-by: NPeter Lieven <pl@kamp.de>
Reviewed-by: NEric Blake <eblake@redhat.com>
Signed-off-by: NKevin Wolf <kwolf@redhat.com>

Add a test for qemu-img map and qemu-io -c map on truncated files.
Signed-off-by: NMax Reitz <mreitz@redhat.com>
Signed-off-by: NKevin Wolf <kwolf@redhat.com>

bdrv_is_allocated() may report zero clusters which most probably means
the image (file) is shorter than expected. Respect this case in order to
avoid an infinite loop.
Signed-off-by: NMax Reitz <mreitz@redhat.com>
Reviewed-by: NEric Blake <eblake@redhat.com>
Reviewed-by: NBenoît Canet <benoit.canet@nodalink.com>
Reviewed-by: NKevin Wolf <kwolf@redhat.com>
Signed-off-by: NKevin Wolf <kwolf@redhat.com>

When falling through to the underlying file in
bdrv_co_get_block_status(), if it returns that the query offset is
beyond the file end (by setting *pnum to 0), return the range to be
zero and do not let the number of sectors for which information could be
obtained be overwritten.
Signed-off-by: NMax Reitz <mreitz@redhat.com>
Signed-off-by: NKevin Wolf <kwolf@redhat.com>

Specify the upper limit of refcount_order to be 6 (that is,
refcount_bits = 64). Any larger value does not make much sense when all
offsets, sizes, cluster counts etc. "only" have a width of 64 bit as
well, and very large values would be very difficult to support.
Therefore, just cap it at the largest reasonable value.
Suggested-by: NEric Blake <eblake@redhat.com>
Signed-off-by: NMax Reitz <mreitz@redhat.com>
Signed-off-by: NKevin Wolf <kwolf@redhat.com>

A refblock entry may have a different size than 16 bits, it may even be
smaller than a byte. Correct the refcount_block_entries calculation
accordingly.
Signed-off-by: NMax Reitz <mreitz@redhat.com>
Signed-off-by: NKevin Wolf <kwolf@redhat.com>

With BDRVQcowState.refcount_block_bits, we don't need REFCOUNT_SHIFT
anymore.
Signed-off-by: NMax Reitz <mreitz@redhat.com>
Reviewed-by: NEric Blake <eblake@redhat.com>
Signed-off-by: NKevin Wolf <kwolf@redhat.com>

There are certain cases where repairing a qcow2 image might actually
damage it further (or rather, where repairing it has in fact damaged it
further with the old qcow2 check implementation). This should not
happen, so add a test for these cases.

Furthermore, the repair function now repairs refblocks beyond the image
end by resizing the image accordingly. Add several tests for this as
well.
Signed-off-by: NMax Reitz <mreitz@redhat.com>
Signed-off-by: NKevin Wolf <kwolf@redhat.com>

039, 060 and 061 all create images with referenced clusters having a
refcount of 0. Because previous commits changed handling of such errors,
these tests now have a different output. Fix it.

Furthermore, 060 created a refblock with a refcount greater than one
which now results in having to rebuild the refcount structure as well.
Signed-off-by: NMax Reitz <mreitz@redhat.com>
Reviewed-by: NEric Blake <eblake@redhat.com>
Signed-off-by: NKevin Wolf <kwolf@redhat.com>

Because the old refcount structure will be leaked after having rebuilt
it, we need to recalculate the refcounts and run a leak-fixing operation
afterwards (if leaks should be fixed at all).
Signed-off-by: NMax Reitz <mreitz@redhat.com>
Signed-off-by: NKevin Wolf <kwolf@redhat.com>

The previous commit introduced the "rebuild" variable to qcow2's
implementation of the image consistency check. Now make use of this by
adding a function which creates a completely new refcount structure
based solely on the in-memory information gathered before.

The old refcount structure will be leaked, however. This leak will be
dealt with in a follow-up commit.
Signed-off-by: NMax Reitz <mreitz@redhat.com>
Signed-off-by: NKevin Wolf <kwolf@redhat.com>

If a referenced cluster has a refcount of 0, increasing its refcount may
result in clusters being allocated for the refcount structures. This may
overwrite the referenced cluster, therefore we cannot simply increase
the refcount then.

In such cases, we can either try to replicate all the refcount
operations solely for the check operation, basing the allocations on the
in-memory refcount table; or we can simply rebuild the whole refcount
structure based on the in-memory refcount table. Since the latter will
be much easier, do that.

To prepare for this, introduce a "rebuild" boolean which should be set
to true whenever a fix is rather dangerous or too complicated using the
current refcount structures. Another example for this is refcount blocks
being referenced more than once.
Signed-off-by: NMax Reitz <mreitz@redhat.com>
Signed-off-by: NKevin Wolf <kwolf@redhat.com>

If the qcow2 check function detects a refcount block located beyond the
image end, grow the image appropriately. This cannot break anything and
is the logical fix for such a case.
Signed-off-by: NMax Reitz <mreitz@redhat.com>
Signed-off-by: NKevin Wolf <kwolf@redhat.com>

We will later call calculate_refcounts multiple times, so reuse the
refcount table if possible.
Signed-off-by: NMax Reitz <mreitz@redhat.com>
Reviewed-by: NBenoît Canet <benoit.canet@nodalink.com>
Reviewed-by: NKevin Wolf <kwolf@redhat.com>
Signed-off-by: NKevin Wolf <kwolf@redhat.com>

Now that the refcount table can be passed around by reference, do that
for inc_refcounts() (and subsequently check_refcounts_l1() and
check_refcounts_l2()) and use it for resizing it when a cluster after
the image end is encountered.
Signed-off-by: NMax Reitz <mreitz@redhat.com>
Signed-off-by: NKevin Wolf <kwolf@redhat.com>

As of a future patch, inc_refcounts() will have to throw errors which
are generally signaled by returning -errno. Therefore, let it return an
integer which is either 0 for success or -errno and handle the -errno
case in all callers.
Signed-off-by: NMax Reitz <mreitz@redhat.com>
Signed-off-by: NKevin Wolf <kwolf@redhat.com>

Instead of printing out an error message, incrementing check_errors and
returning a fixed -errno, just do cleanups and return -ret, with ret set
by the code which threw the exception (jumped to the fail label).

Also, increment check_errors on error in check_refcounts_l2().
Signed-off-by: NMax Reitz <mreitz@redhat.com>
Signed-off-by: NKevin Wolf <kwolf@redhat.com>

Use int64_t for the entry count of the in-memory refcount table
throughout the check functions.
Signed-off-by: NMax Reitz <mreitz@redhat.com>
Signed-off-by: NKevin Wolf <kwolf@redhat.com>

Pull check_refblocks() before calculate_refcounts() so we can drop its
static declaration.
Signed-off-by: NMax Reitz <mreitz@redhat.com>
Reviewed-by: NBenoît Canet <benoit.canet@nodalink.com>
Reviewed-by: NKevin Wolf <kwolf@redhat.com>
Signed-off-by: NKevin Wolf <kwolf@redhat.com>

When implementing variable refcounts, we want to be able to easily find
all the places in qemu which are tied to a certain refcount order.
Replace sizeof(uint16_t) in the check code by sizeof(**refcount_table)
so we can later find it more easily.
Signed-off-by: NMax Reitz <mreitz@redhat.com>
Reviewed-by: NEric Blake <eblake@redhat.com>
Signed-off-by: NKevin Wolf <kwolf@redhat.com>

Put the code for calculating the reference counts and comparing them
during qemu-img check into own functions.
Signed-off-by: NMax Reitz <mreitz@redhat.com>
Reviewed-by: NBenoît Canet <benoit.canet@nodalink.com>
Reviewed-by: NKevin Wolf <kwolf@redhat.com>
Signed-off-by: NKevin Wolf <kwolf@redhat.com>

When opening dirty images, qcow2's repair function should not only
repair errors but leaks as well.
Signed-off-by: NMax Reitz <mreitz@redhat.com>
Reviewed-by: NEric Blake <eblake@redhat.com>
Reviewed-by: NBenoît Canet <benoit.canet@nodalink.com>
Reviewed-by: NKevin Wolf <kwolf@redhat.com>
Signed-off-by: NKevin Wolf <kwolf@redhat.com>

The size of a refblock entry is (in theory) variable; calculate
therefore the number of entries per refblock and the according bit shift
(1 << x == entry count) when opening an image.
Signed-off-by: NMax Reitz <mreitz@redhat.com>
Reviewed-by: NEric Blake <eblake@redhat.com>
Signed-off-by: NKevin Wolf <kwolf@redhat.com>

These functions call their non-0-counterparts and then fill the
allocated buffer with 0 (if the allocation has been successful).
Signed-off-by: NMax Reitz <mreitz@redhat.com>
Reviewed-by: NEric Blake <eblake@redhat.com>
Signed-off-by: NKevin Wolf <kwolf@redhat.com>

In preparation to possible automatic regression and performance
testing for the block layer I found that the iotests don't work
for all protocols anymore.

In commit 1f7bf7d0 I started to change supported protocols from
generic to file for various tests. Unfortunately, some tests
added in the meantime again carry generic protocol altough they
can only work with file because they require local file access.

The other way around for some tests that only support file I added
NFS protocol after confirming they work.
Signed-off-by: NPeter Lieven <pl@kamp.de>
Reviewed-by: NBenoît Canet <benoit.canet@nodalink.com>
Signed-off-by: NKevin Wolf <kwolf@redhat.com>

There are macros for these operations, so make use of them.
Signed-off-by: NMax Reitz <mreitz@redhat.com>
Reviewed-by: NEric Blake <eblake@redhat.com>
Signed-off-by: NKevin Wolf <kwolf@redhat.com>

More work for the block device maintainers!
Signed-off-by: NPaolo Bonzini <pbonzini@redhat.com>
Signed-off-by: NKevin Wolf <kwolf@redhat.com>

Signed-off-by: NKevin Wolf <kwolf@redhat.com>

AioContext falls under the block layer, mark it as such.
Signed-off-by: NPaolo Bonzini <pbonzini@redhat.com>
Signed-off-by: NKevin Wolf <kwolf@redhat.com>

device_del_bus_completion() that gathers devices from buses is unused; delete
it.
Signed-off-by: NZhu Guihua <zhugh.fnst@cn.fujitsu.com>
Reviewed-by: NMarcel Apfelbaum <marcel.a@redhat.com>
Reviewed-by: NIgor Mammedov <imammedo@redhat.com>
Signed-off-by: NLuiz Capitulino <lcapitulino@redhat.com>

Add peripheral_device_del_completion() to let peripheral device del completion
be possible.
Signed-off-by: NZhu Guihua <zhugh.fnst@cn.fujitsu.com>
Reviewed-by: NMarcel Apfelbaum <marcel.a@redhat.com>
Reviewed-by: NIgor Mammedov <imammedo@redhat.com>
Signed-off-by: NLuiz Capitulino <lcapitulino@redhat.com>

For peripheral device del completion, add a function to build a list for
hotpluggable devices.
Signed-off-by: NZhu Guihua <zhugh.fnst@cn.fujitsu.com>
Reviewed-by: NMarcel Apfelbaum <marcel.a@redhat.com>
Reviewed-by: NIgor Mammedov <imammedo@redhat.com>
Signed-off-by: NLuiz Capitulino <lcapitulino@redhat.com>

Signed-off-by: NLuiz Capitulino <lcapitulino@redhat.com>

Functions shouldn't return an error code and an Error object at the same time.
Turn all these functions that returning Error object to void.
We also judge if a function success or fail by reference to the local_err.
Signed-off-by: Nzhanghailiang <zhang.zhanghailiang@huawei.com>
Reviewed-by: NEric Blake <eblake@redhat.com>
Signed-off-by: NLuiz Capitulino <lcapitulino@redhat.com>