this converts read, write and flush functions from aio to coroutines
eliminating almost 200 lines of code.

The requirement for libiscsi is bumped to version 1.4.0 which was
released in may 2012.
Signed-off-by: NPeter Lieven <pl@kamp.de>
Signed-off-by: NPaolo Bonzini <pbonzini@redhat.com>

VERIFY emulation was completely botched (and remained botched through
all the refactorings).  The command must be emulated both in check-medium
mode (BYTCHK=00, which we implement by doing nothing) and in check-bytes
mode (which we do not implement yet).  Unlike WRITE AND VERIFY (which we
treat simply as WRITE with FUA bit set), VERIFY cannot be handled like
READ.  In fact the device is _receiving_ data for VERIFY, not _sending_
it like READ.

Cc: qemu-stable@nongnu.org
Tested-by: NHervé Poussineau <hpoussin@reactos.org>
Signed-off-by: NPaolo Bonzini <pbonzini@redhat.com>

The amount of bytes to transfer depends on the BYTCHK field.
If any data is transferred, it is sent to the device.

Cc: qemu-stable@nongnu.org
Tested-by: NHervé Poussineau <hpoussin@reactos.org>
Signed-off-by: NPaolo Bonzini <pbonzini@redhat.com>

SMTP does not preserve newlines.  This is normally not a problem if the
email body uses DOS or UNIX newlines consistently.  In 051.out we mix
UNIX newlines with DOS newlines (since QEMU monitor output uses \r\n).

This patch filters the QEMU monitor output so the golden master file
uses UNIX newlines exclusively.

The result is that patches touching 051.out will apply cleanly without
mangling newlines after this commit.
Signed-off-by: NStefan Hajnoczi <stefanha@redhat.com>

This patch drops a special case where aio_poll(ctx, true) returns false
instead of blocking if no file descriptors are waiting on I/O.  Now it
is possible to block in aio_poll() to wait for aio_notify().

This change eliminates busy waiting.  bdrv_drain_all() used to rely on
busy waiting to completed throttled I/O requests but this is no longer
required so we can simplify aio_poll().

Note that aio_poll() still returns false when aio_notify() was used.  In
other words, stopping a blocking aio_poll() wait is not considered
making progress.

Adjust test-aio /aio/bh/callback-delete/one which assumed aio_poll(ctx,
true) would immediately return false instead of blocking.
Reviewed-by: NAlex Bligh <alex@alex.org.uk>
Signed-off-by: NStefan Hajnoczi <stefanha@redhat.com>

Since cc0681c4 ("block: Enable the new
throttling code in the block layer.") bdrv_drain_all() no longer spins.
The code used to look as follows:

  do {
      busy = qemu_aio_wait();

      /* FIXME: We do not have timer support here, so this is effectively
       * a busy wait.
       */
      QTAILQ_FOREACH(bs, &bdrv_states, list) {
          while (qemu_co_enter_next(&bs->throttled_reqs)) {
              busy = true;
          }
      }
  } while (busy);

Note that throttle requests are kicked but I/O throttling limits are
still in effect.  The loop spins until the vm_clock time allows the
request to make progress and complete.

The new throttling code introduced bdrv_start_throttled_reqs().  This
function not only kicks throttled requests but also temporarily disables
throttling so requests can run.

The outdated FIXME comment can be removed.  Also drop the busy = true
assignment since we overwrite it immediately afterwards.
Reviewed-by: NAlex Bligh <alex@alex.org.uk>
Signed-off-by: NStefan Hajnoczi <stefanha@redhat.com>

Signed-off-by: NHu Tao <hutao@cn.fujitsu.com>
Reviewed-by: NFam Zheng <famz@redhat.com>
Signed-off-by: NStefan Hajnoczi <stefanha@redhat.com>

when doing very large jobs updating the progress only every 2%
is too rare.
Signed-off-by: NPeter Lieven <pl@kamp.de>
Signed-off-by: NStefan Hajnoczi <stefanha@redhat.com>

this patch shortens requests to end at an aligned sector so that
the next request starts aligned.

[Squashed Peter's fix for bdrv_get_info() failure discussed on the
mailing list.
--Stefan]
Reviewed-by: NPaolo Bonzini <pbonzini@redhat.com>
Signed-off-by: NPeter Lieven <pl@kamp.de>
Signed-off-by: NStefan Hajnoczi <stefanha@redhat.com>

since the convert process is basically a sync operation it might
be benificial in some case to change the hardcoded I/O buffer
size to a greater value.

This patch increases the I/O buffer size if the output
driver advertises an optimal transfer length or discard alignment
that is greater than the default buffer size of 2M.
Reviewed-by: NPaolo Bonzini <pbonzini@redhat.com>
Signed-off-by: NPeter Lieven <pl@kamp.de>
Signed-off-by: NStefan Hajnoczi <stefanha@redhat.com>

Reviewed-by: NPaolo Bonzini <pbonzini@redhat.com>
Signed-off-by: NPeter Lieven <pl@kamp.de>
Signed-off-by: NStefan Hajnoczi <stefanha@redhat.com>

Reviewed-by: NPaolo Bonzini <pbonzini@redhat.com>
Signed-off-by: NPeter Lieven <pl@kamp.de>
Signed-off-by: NStefan Hajnoczi <stefanha@redhat.com>

this patch aims to set bdi->cluster_size to the internal page size
of the iscsi target so that enabled callers can align requests
properly.
Reviewed-by: NPaolo Bonzini <pbonzini@redhat.com>
Signed-off-by: NPeter Lieven <pl@kamp.de>
Signed-off-by: NStefan Hajnoczi <stefanha@redhat.com>

Reviewed-by: NEric Blake <eblake@redhat.com>
Reviewed-by: NPaolo Bonzini <pbonzini@redhat.com>
Signed-off-by: NPeter Lieven <pl@kamp.de>
Signed-off-by: NStefan Hajnoczi <stefanha@redhat.com>

we currently do not check if a sector is allocated during convert.
This means if a sector is unallocated that we allocate a bounce
buffer of zeroes, find out its zero later and do not write it
in the best case. In the worst case this can lead to reading
blocks from a raw device (like iSCSI) altough we could easily
know via get_block_status that they are zero and simply skip them.

This patch also fixes the progress output not being at 100% after
a successful conversion.
Signed-off-by: NPeter Lieven <pl@kamp.de>
Reviewed-by: NPaolo Bonzini <pbonzini@redhat.com>
Signed-off-by: NStefan Hajnoczi <stefanha@redhat.com>

Signed-off-by: NWenchao Xia <xiawenc@linux.vnet.ibm.com>
Signed-off-by: NStefan Hajnoczi <stefanha@redhat.com>

Signed-off-by: NWenchao Xia <xiawenc@linux.vnet.ibm.com>
Signed-off-by: NStefan Hajnoczi <stefanha@redhat.com>

Now qemu-img convert have similar options as qemu-nbd for internal
snapshot.
Signed-off-by: NWenchao Xia <xiawenc@linux.vnet.ibm.com>
Signed-off-by: NStefan Hajnoczi <stefanha@redhat.com>

This case can't run when IMGPROTO=nbd, since it needs to create some
internal snapshot which would fail for EOF write request, even when
TEST_IMG is exported with "-f raw" in common.rc, so set _supported_proto
to file.

_require_command() is changed to tip what util is missing, instead
of printing a blank.
Signed-off-by: NWenchao Xia <xiawenc@linux.vnet.ibm.com>
Signed-off-by: NStefan Hajnoczi <stefanha@redhat.com>

Now it is possible to directly export an internal snapshot, which
can be used to probe the snapshot's contents without qemu-img
convert.
Signed-off-by: NWenchao Xia <xiawenc@linux.vnet.ibm.com>
Signed-off-by: NStefan Hajnoczi <stefanha@redhat.com>

Since later this function will be used so improve it. The only caller of it
now is qemu-img, and it is not impacted by introduce function
bdrv_snapshot_load_tmp_by_id_or_name() that call bdrv_snapshot_load_tmp()
twice to keep old search logic. bdrv_snapshot_load_tmp_by_id_or_name() return
int to let caller know the errno, and errno will be used later.
Also fix a typo in comments of bdrv_snapshot_delete().
Signed-off-by: NWenchao Xia <xiawenc@linux.vnet.ibm.com>
Signed-off-by: NStefan Hajnoczi <stefanha@redhat.com>

Format "raw" doesn't always work on certain file systems (e.g. tmpfs).
Use qcow2 to make the allocation status explicit and split into a new
case.

[Resolved merge conflict due to "qemu-io> " prompt filter, added 074 to
group file, and fixed up s/048/074/ copy-paste mistake.
--Stefan]
Signed-off-by: NFam Zheng <famz@redhat.com>
Reviewed-by: NWenchao Xia <xiawenc@linux.vnet.ibm.com>
Signed-off-by: NStefan Hajnoczi <stefanha@redhat.com>

Whitespace changes to align columns.
Signed-off-by: NFam Zheng <famz@redhat.com>
Reviewed-by: NBenoit Canet <benoit@irqsave.net>
Signed-off-by: NStefan Hajnoczi <stefanha@redhat.com>

So that the tests can run faster.
Signed-off-by: NFam Zheng <famz@redhat.com>
Reviewed-by: NWenchao Xia <xiawenc@linux.vnet.ibm.com>
Signed-off-by: NStefan Hajnoczi <stefanha@redhat.com>

This replaces _unsupported_qemu_io_options and check for support of
current cache mode, and allow to provide a default if user didn't
specify.
Signed-off-by: NFam Zheng <famz@redhat.com>
Reviewed-by: NWenchao Xia <xiawenc@linux.vnet.ibm.com>
Signed-off-by: NStefan Hajnoczi <stefanha@redhat.com>

This will allow overriding cache mode from the "-c mode" option.
Signed-off-by: NFam Zheng <famz@redhat.com>
Reviewed-by: NWenchao Xia <xiawenc@linux.vnet.ibm.com>
Signed-off-by: NStefan Hajnoczi <stefanha@redhat.com>

The option sets cache mode used in the tests. "-nocache" is changed to
an alias to "-c none", and internally passes "-t none" to qemu-io.

Python scripts will make use of option this in the next commit.
Signed-off-by: NFam Zheng <famz@redhat.com>
Reviewed-by: NWenchao Xia <xiawenc@linux.vnet.ibm.com>
Signed-off-by: NStefan Hajnoczi <stefanha@redhat.com>

Strictly speaking, this is only required for has_zero_init() == false,
but it's easy enough to just do a cluster-aligned write that is padded
with zeros after the header.

This fixes that after 'qemu-img create' header extensions are attempted
to be parsed that are really just random leftover data.

Cc: qemu-stable@nongnu.org
Signed-off-by: NKevin Wolf <kwolf@redhat.com>
Reviewed-by: NFam Zheng <famz@redhat.com>
Reviewed-by: NPaolo Bonzini <pbonzini@redhat.com>
Signed-off-by: NStefan Hajnoczi <stefanha@redhat.com>

Leaving the backing file open although it is not needed anymore can
cause problems if it is opened through a block driver which allows
exclusive access only and if the create function of the block driver
used for the top image (the one being created) tries to close and reopen
the image file (which will include opening the backing file a second
time).

In particular, this will happen with a backing file opened through
qemu-nbd and using qcow2 as the top image file format (which reopens the
image to flush it to disk).

In addition, the BlockDriverState in bdrv_img_create() is used for the
backing file only; it should therefore be made local to the respective
block.
Signed-off-by: NMax Reitz <mreitz@redhat.com>
Reviewed-by: NKevin Wolf <kwolf@redhat.com>
Reviewed-by: NWenchao Xia <xiawenc@linux.vnet.ibm.com>
Signed-off-by: NStefan Hajnoczi <stefanha@redhat.com>

Fetch the data to be written from the input buffer.  If it is all zeroes,
we can use the write_zeroes call (possibly with the new MAY_UNMAP flag).
Otherwise, do as many write cycles as needed, writing 512k at a time.

Strictly speaking, this is still incorrect because a zero cluster should
only be written if the MAY_UNMAP flag is set.  But this is a bug in qcow2
and the other formats, not in the SCSI code.
Signed-off-by: NPaolo Bonzini <pbonzini@redhat.com>
Signed-off-by: NStefan Hajnoczi <stefanha@redhat.com>

Since we report ANC_SUP==0 in VPD page B2h, we need to return
an error (ILLEGAL REQUEST/INVALID FIELD IN CDB) for all WRITE SAME
requests with ANCHOR==1.

Inspired by a similar patch to the LIO in-kernel target.
Signed-off-by: NPaolo Bonzini <pbonzini@redhat.com>
Signed-off-by: NStefan Hajnoczi <stefanha@redhat.com>

This is the same that is already done for WRITE SAME.
Signed-off-by: NPaolo Bonzini <pbonzini@redhat.com>
Signed-off-by: NStefan Hajnoczi <stefanha@redhat.com>

Signed-off-by: NPaolo Bonzini <pbonzini@redhat.com>
Signed-off-by: NStefan Hajnoczi <stefanha@redhat.com>

The code is similar to the implementation of discard and write_zeroes
with UNMAP.  However, failure must be propagated up to block.c.

The stale page cache problem can be reproduced as follows:

    # modprobe scsi-debug lbpws=1 lbprz=1
    # ./qemu-io /dev/sdXX
    qemu-io> write -P 0xcc 0 2M
    qemu-io> write -z 0 1M
    qemu-io> read -P 0x00 0 512
    Pattern verification failed at offset 0, 512 bytes
    qemu-io> read -v 0 512
    00000000:  cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc  ................
    ...

    # ./qemu-io --cache=none /dev/sdXX
    qemu-io> write -P 0xcc 0 2M
    qemu-io> write -z 0 1M
    qemu-io> read -P 0x00 0 512
    qemu-io> read -v 0 512
    00000000:  00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
    ...

And similarly with discard instead of "write -z".
Signed-off-by: NPaolo Bonzini <pbonzini@redhat.com>
Signed-off-by: NStefan Hajnoczi <stefanha@redhat.com>

See the next commit for the description of the Linux kernel problem
that is worked around in raw_open_common.
Signed-off-by: NPaolo Bonzini <pbonzini@redhat.com>
Signed-off-by: NStefan Hajnoczi <stefanha@redhat.com>

Writing zeroes to a file can be done by punching a hole if
MAY_UNMAP is set.

Note that in this case ENOTSUP is not ignored, but makes
the block layer fall back to the generic implementation.
Signed-off-by: NPaolo Bonzini <pbonzini@redhat.com>
Signed-off-by: NStefan Hajnoczi <stefanha@redhat.com>

The current check is right for MAY_UNMAP=1.  For MAY_UNMAP=0, just
try and fall back to regular writes as soon as a WRITE SAME command
fails.
Signed-off-by: NPaolo Bonzini <pbonzini@redhat.com>
Reviewed-by: NPeter Lieven <pl@kamp.de>
Signed-off-by: NStefan Hajnoczi <stefanha@redhat.com>

added myself to reflect recent work on the iscsi block driver.
Signed-off-by: NPeter Lieven <pl@kamp.de>
Signed-off-by: NPaolo Bonzini <pbonzini@redhat.com>
Signed-off-by: NStefan Hajnoczi <stefanha@redhat.com>

since commit 3ac21627 the default value changed to 0.
Signed-off-by: NPeter Lieven <pl@kamp.de>
Signed-off-by: NPaolo Bonzini <pbonzini@redhat.com>
Signed-off-by: NStefan Hajnoczi <stefanha@redhat.com>

This will let misaligned but large requests use zero clusters.  This
is important because the cluster size is not guest visible.
Signed-off-by: NPaolo Bonzini <pbonzini@redhat.com>
Reviewed-by: NPeter Lieven <pl@kamp.de>
Signed-off-by: NStefan Hajnoczi <stefanha@redhat.com>