Mostly bugfixes and cleanups from qemu-devel.  Yet another small patch from
the record/replay series, and a few SCSI and i386 patches as well.

# gpg: Signature made Wed 14 Jan 2015 09:39:14 GMT using RSA key ID 78C7AE83
# gpg: Good signature from "Paolo Bonzini <bonzini@gnu.org>"
# gpg:                 aka "Paolo Bonzini <pbonzini@redhat.com>"
# gpg: WARNING: This key is not certified with sufficiently trusted signatures!
# gpg:          It is not certain that the signature belongs to the owner.
# Primary key fingerprint: 46F5 9FBD 57D6 12E7 BFD4  E2F7 7E15 100C CD36 69B1
#      Subkey fingerprint: F133 3857 4B66 2389 866C  7682 BFFB D25F 78C7 AE83

* remotes/bonzini/tags/for-upstream:
  cpus: consistently use QEMU_CLOCK_VIRTUAL_RT for icount_warp_rt timer
  qemu-timer: rename timer_init to timer_init_tl
  scsi: fix cancellation when I/O was completed but DMA was not.
  rules.mak: Fix module build
  hw/scsi/lsi53c895a: add support for additional diag / debug registers
  qemu-common.h: optimise muldiv64 if int128 is available
  target-i386: do not memcpy in and out of xmm_regs
  target-i386: fix movntsd on big-endian hosts
  vl.c: fix regression when reading memory size from config file
  vl: Don't silently change topology when all -smp options were set
  vl: fix max_cpus check
  vl: Avoid unnecessary 'if' nesting
  9pfs: changed to use event_notifier instead of qemu_pipe
  vl.c: fix regression when reading machine type from config file
  char: restore stdio echo on resume from suspend.
Signed-off-by: NPeter Maydell <peter.maydell@linaro.org>

Fix mismatch between timer_new_ms and timer_mod.
Signed-off-by: NPavel Dovgalyuk <Pavel.Dovgaluk@ispras.ru>
Signed-off-by: NPaolo Bonzini <pbonzini@redhat.com>

timer_init is not called that often.  Free the name for an equivalent
of timer_new.
Signed-off-by: NPaolo Bonzini <pbonzini@redhat.com>

Commit d5776465 (scsi: Introduce scsi_req_cancel_complete, 2014-09-25)
was supposed to have no semantic change, but it missed a case.  When
r->aiocb has already been NULLed, but DMA was not complete and the
SCSI layer was waiting for scsi_req_continue, after the patch the
SCSI layer will not call the .cancel callback of SCSIBusInfo.

Fixes: d5776465
Cc: qemu-stable@nongnu.org
Reported-by: NDr. David Alan Gilbert <dgilbert@redhat.com>
Tested-by: NDr. David Alan Gilbert <dgilbert@redhat.com>
Reviewed-by: NFam Zheng <famz@redhat.com>
Signed-off-by: NPaolo Bonzini <pbonzini@redhat.com>

Module build is broken since commit c261d774 ( rules.mak: Fix DSO
build by pulling in archive symbols). That commit added .mo placeholders
of DSO to -y variables, in order to pull stub symbols to executable. But
the placeholders are unintentionally expanded in -y, rather than
filtered out while linking.

Fix it by moving the -objs expanding to before inserting .mo
placeholders.  Note that passing -cflags and -libs to member objects are
also moved to keep it happening before object expanding.
Reported-by: NBharata B Rao <bharata.rao@gmail.com>
Tested-by: NBharata B Rao <bharata.rao@gmail.com>
Signed-off-by: NFam Zheng <famz@redhat.com>
Signed-off-by: NPaolo Bonzini <pbonzini@redhat.com>

Some ancient Linux kernels read from registers 0x09 and 0x3c-3f during
boot. According to the spec these registers are for diag and debug
purposes only. If they are absend qemu aborts on read.
Signed-off-by: NPeter Lieven <pl@kamp.de>
Signed-off-by: NPaolo Bonzini <pbonzini@redhat.com>

Let compiler do the job to optimise the function.
Signed-off-by: NFrediano Ziglio <frediano.ziglio@huawei.com>
Signed-off-by: NPaolo Bonzini <pbonzini@redhat.com>
Signed-off-by: NFrediano Ziglio <freddy77@gmail.com>

After the next patch, we will move the high parts of AVX and AVX512 registers
in the same array as the SSE registers.  This will make it impossible to
memcpy an array of 128-bit values in and out of xmm_regs in one swoop.
Use a for loop instead.

Similarly, always use XMM_Q in translate.c.  This avoids introducing bugs
such as the one fixed in the previous patch.
Reviewed-by: NEduardo Habkost <ehabkost@redhat.com>
Signed-off-by: NPaolo Bonzini <pbonzini@redhat.com>

This was accessing an XMM register's low half without going through XMM_Q.

Cc: qemu-stable@nongnu.org
Reviewed-by: NEduardo Habkost <ehabkost@redhat.com>
Signed-off-by: NPaolo Bonzini <pbonzini@redhat.com>

This is happening because an actual logic is performed on the memory
arguments inside the main's switch, disregarding the config file content.

Solved by extracting the logic on a separate function and calling it
after the switch.
Signed-off-by: NMarcel Apfelbaum <marcel@redhat.com>
Signed-off-by: NPaolo Bonzini <pbonzini@redhat.com>

* remotes/sstabellini/xen-2015-01-13:
  xen-hvm: increase maxmem before calling xc_domain_populate_physmap
  xen-pt: Fix PCI devices re-attach failed
Signed-off-by: NPeter Maydell <peter.maydell@linaro.org>

Increase maxmem before calling xc_domain_populate_physmap_exact to
avoid the risk of running out of guest memory. This way we can also
avoid complex memory calculations in libxl at domain construction
time.

This patch fixes an abort() when assigning more than 4 NICs to a VM.
Signed-off-by: NStefano Stabellini <stefano.stabellini@eu.citrix.com>
Signed-off-by: NDon Slutz <dslutz@verizon.com>

# gpg: Signature made Tue 13 Jan 2015 13:48:06 GMT using RSA key ID 81AB73C8
# gpg: Good signature from "Stefan Hajnoczi <stefanha@redhat.com>"
# gpg:                 aka "Stefan Hajnoczi <stefanha@gmail.com>"

* remotes/stefanha/tags/block-pull-request: (38 commits)
  NVMe: Set correct VS Value for 1.1 Compliant Controllers
  MAINTAINERS: Add migration/block* to block subsystem
  MAINTAINERS: Update email addresses for Chrysostomos Nanakos
  nvme: Fix get/set number of queues feature
  ide: Implement VPD response for ATAPI
  block: Split BLOCK_OP_TYPE_COMMIT to BLOCK_OP_TYPE_COMMIT_{SOURCE, TARGET}
  block: limited request size in write zeroes unsupported path
  coroutine: try harder not to delete coroutines
  coroutine: drop qemu_coroutine_adjust_pool_size
  coroutine: rewrite pool to avoid mutex
  QSLIST: add lock-free operations
  test-coroutine: avoid overflow on 32-bit systems
  qemu-thread: add per-thread atexit functions
  coroutine-ucontext: use __thread
  qemu-iotests: Add supported os parameter for python tests
  qemu-iotests: Add "_supported_os Linux" to 058
  qemu-iotests: Replace "/bin/true" with "true"
  .gitignore: Ignore generated "common.env"
  libqos: Convert malloc-pc allocator to a generic allocator
  migration/block: fix pending() return value
  ...
Signed-off-by: NPeter Maydell <peter.maydell@linaro.org>

According to NVMe specifications Bits 15:08 represent Minor Version number.
Signed-off-by: NAnubhav Rakshit <anubhav.rakshit@gmail.com>
Signed-off-by: NStefan Hajnoczi <stefanha@redhat.com>

We are moving block-migration.c to the separated migration directory,
keep this file watched by block maintainers is a good idea.
Signed-off-by: NFam Zheng <famz@redhat.com>
Signed-off-by: NStefan Hajnoczi <stefanha@redhat.com>

Remove first email address and let the one from which I am contributing.
Signed-off-by: NChrysostomos Nanakos <chris@include.gr>
Signed-off-by: NStefan Hajnoczi <stefanha@redhat.com>

According to the specification, the low 16 bits should contain the number of
I/O submission queues, and the high 16 bits should contain the number of
I/O completion queues.
Signed-off-by: NAlex Friedman <alex@e8storage.com>
Acked-by: NKeith Busch <keith.busch@intel.com>
Signed-off-by: NStefan Hajnoczi <stefanha@redhat.com>

SCSI devices have multiple kinds of queries they need to respond
to, as defined in the "cmd inquiry" section in MMC-6 and SPC-3.

Relevent sections:
MMC-6 revision 2g:
      Non-VPD response data and pointer to SPC-3;
      Section 6.8 "Inquiry Command"
SPC-3 revision 23:
      Inquiry command and error handling:
      Section 6.4 "INQUIRY command"
      VPD data pages format:
      Section 7.6 "Vital product data parameters"

We implement these Vital Product Data queries for SCSI, but not for
ATAPI through IDE. The result is that if you are looking for the WWN
identifier via tools such as sg3_utils, you will be unable to query
our CD/DVD rom device to obtain it.

This patch adds the minimum number of mandatory responses as defined
by SPC-3, which include the "supported pages" response (page 0x00)
and the "Device Identification" response (page 0x83). It also correctly
responds when it receives a request for an illegal page to improve
error output from related tools.

The Device ID page contains an arbitrary list of identification
strings of various formats; the ID strings included in this patch
were chosen to mimic those provided by the libata driver when
emulating this SCSI query (model, serial, and wwn when present.)

Example:

# libata emulated response
[root@localhost ~]# sg_inq --id /dev/sda
VPD INQUIRY: Device Identification page
  Designation descriptor number 1, descriptor length: 24
    designator_type: vendor specific [0x0],  code_set: ASCII
    associated with the addressed logical unit
      vendor specific: QM00001
  Designation descriptor number 2, descriptor length: 72
    designator_type: T10 vendor identification,  code_set: ASCII
    associated with the addressed logical unit
      vendor id: ATA
      vendor specific: QEMU HARDDISK                           QM00001

# QEMU generated ATAPI response, with WWN
[root@localhost ~]# sg_inq --id /dev/sr0
VPD INQUIRY: Device Identification page
  Designation descriptor number 1, descriptor length: 24
    designator_type: vendor specific [0x0],  code_set: ASCII
    associated with the addressed logical unit
      vendor specific: QM00005
  Designation descriptor number 2, descriptor length: 72
    designator_type: T10 vendor identification,  code_set: ASCII
    associated with the addressed logical unit
      vendor id: ATA
      vendor specific: QEMU DVD-ROM                            QM00005
  Designation descriptor number 3, descriptor length: 12
    designator_type: NAA,  code_set: Binary
    associated with the addressed logical unit
      NAA 5, IEEE Company_id: 0xc50
      Vendor Specific Identifier: 0x15ea71bb
      [0x5000c50015ea71bb]

See also: hw/scsi/scsi-disk.c, scsi_disk_emulate_inquiry()
Signed-off-by: NJohn Snow <jsnow@redhat.com>
Signed-off-by: NStefan Hajnoczi <stefanha@redhat.com>

Like BLOCK_OP_TYPE_BACKUP_SOURCE and BLOCK_OP_TYPE_BACKUP_TARGET,
block-commit involves two asymmetric devices.

This change is not user-visible (yet), because commit only works with
device names.

But once we enable backing reference in blockdev-add, or specifying
node-name in block-commit command, we don't want the user to start two
commit jobs on the same backing chain, which will corrupt things because
of the final bdrv_swap.

Before we have per category blockers, splitting this type is still
better.

[Resolved virtio-blk dataplane conflict by replacing
BLOCK_OP_TYPE_COMMIT with both BLOCK_OP_TYPE_COMMIT_{SOURCE, TARGET}.
They are safe since the block job runs in the same AioContext as the
dataplane IOThread.
--Stefan]
Signed-off-by: NFam Zheng <famz@redhat.com>
Signed-off-by: NStefan Hajnoczi <stefanha@redhat.com>

If bs->bl.max_write_zeroes is large and we end up in the unsupported
path we might allocate a lot of memory for the iovector and/or even
generate an oversized requests.

Fix this by limiting the request by the minimum of the reported
maximum transfer size or 16MB (32768 sectors).
Reported-by: NDenis V. Lunev <den@openvz.org>
Signed-off-by: NPeter Lieven <pl@kamp.de>
Reviewed-by: NDenis V. Lunev <den@openvz.org>
Message-id: 1420457389-16332-1-git-send-email-pl@kamp.de
Signed-off-by: NStefan Hajnoczi <stefanha@redhat.com>

Placing coroutines on the global pool should be preferrable, because it
can help all threads.  But if the global pool is full, we can still
try to save some allocations by stashing completed coroutines on the
local pool.  This is quite cheap too, because it does not require
atomic operations, and provides a gain of 15% in the best case.
Signed-off-by: NPeter Lieven <pl@kamp.de>
Signed-off-by: NPaolo Bonzini <pbonzini@redhat.com>
Reviewed-by: NFam Zheng <famz@redhat.com>
Message-id: 1417518350-6167-8-git-send-email-pbonzini@redhat.com
Signed-off-by: NStefan Hajnoczi <stefanha@redhat.com>

This is not needed anymore.  The new TLS-based algorithm is adaptive.
Signed-off-by: NPaolo Bonzini <pbonzini@redhat.com>
Reviewed-by: NFam Zheng <famz@redhat.com>
Message-id: 1417518350-6167-7-git-send-email-pbonzini@redhat.com
Signed-off-by: NStefan Hajnoczi <stefanha@redhat.com>

This patch removes the mutex by using fancy lock-free manipulation of
the pool.  Lock-free stacks and queues are not hard, but they can suffer
from the ABA problem so they are better avoided unless you have some
deferred reclamation scheme like RCU.  Otherwise you have to stick
with adding to a list, and emptying it completely.  This is what this
patch does, by coupling a lock-free global list of available coroutines
with per-CPU lists that are actually used on coroutine creation.

Whenever the destruction pool is big enough, the next thread that runs
out of coroutines will steal the whole destruction pool.  This is positive
in two ways:

1) the allocation does not have to do any atomic operation in the fast
path, it's entirely using thread-local storage.  Once every POOL_BATCH_SIZE
allocations it will do a single atomic_xchg.  Release does an atomic_cmpxchg
loop, that hopefully doesn't cause any starvation, and an atomic_inc.

A later patch will also remove atomic operations from the release path,
and try to avoid the atomic_xchg altogether---succeeding in doing so if
all devices either use ioeventfd or are not submitting requests actively.

2) in theory this should be completely adaptive.  The number of coroutines
around should be a little more than POOL_BATCH_SIZE * number of allocating
threads; so this also empties qemu_coroutine_adjust_pool_size.  (The previous
pool size was POOL_BATCH_SIZE * number of block backends, so it was a bit
more generous.  But if you actually have many high-iodepth disks, it's better
to put them in different iothreads, which will also use separate thread
pools and aio=native file descriptors).

This speeds up perf/cost (in tests/test-coroutine) by a factor of ~1.33.
No matter if we end with some kind of coroutine bypass scheme or not,
it cannot hurt to optimize hot code.
Signed-off-by: NPaolo Bonzini <pbonzini@redhat.com>
Reviewed-by: NFam Zheng <famz@redhat.com>
Message-id: 1417518350-6167-6-git-send-email-pbonzini@redhat.com
Signed-off-by: NStefan Hajnoczi <stefanha@redhat.com>

These operations are trivial to implement and do not have ABA problems.
They are enough to implement simple multiple-producer, single consumer
lock-free lists or, as in the next patch, the multiple consumers can
steal a whole batch of elements and process them at their leisure.
Signed-off-by: NPaolo Bonzini <pbonzini@redhat.com>
Reviewed-by: NFam Zheng <famz@redhat.com>
Message-id: 1417518350-6167-5-git-send-email-pbonzini@redhat.com
Signed-off-by: NStefan Hajnoczi <stefanha@redhat.com>

unsigned long is not large enough to represent 1000000000 * duration there.
Just use floating point.
Signed-off-by: NPaolo Bonzini <pbonzini@redhat.com>
Reviewed-by: NFam Zheng <famz@redhat.com>
Message-id: 1417518350-6167-4-git-send-email-pbonzini@redhat.com
Signed-off-by: NStefan Hajnoczi <stefanha@redhat.com>

Destructors are the main additional feature of pthread TLS compared
to __thread.  If we were using C++ (hint, hint!) we could have used
thread-local objects with a destructor.  Since we are not, instead,
we add a simple Notifier-based API.

Note that the notifier must be per-thread as well.  We can add a
global list as well later, perhaps.

The Win32 implementation has some complications because a) detached
threads used not to have a QemuThreadData; b) the main thread does
not go through win32_start_routine, so we have to use atexit too.
Signed-off-by: NPaolo Bonzini <pbonzini@redhat.com>
Reviewed-by: NFam Zheng <famz@redhat.com>
Message-id: 1417518350-6167-3-git-send-email-pbonzini@redhat.com
Signed-off-by: NStefan Hajnoczi <stefanha@redhat.com>

ELF thread local storage is about 10% faster on tests/test-coroutine's
perf/cost test.  The timing on my machine is 190ns per iteration with
pthread TLS, 170 with ELF TLS.

Based on a patch by Kevin Wolf and Peter Lieven, but redone to follow
the model of coroutine-win32.c (including the important "noinline"
attribute!).

Platforms without thread-local storage (OpenBSD probably?) will need
a new-enough GCC for this to compile, in order to use the same emutls
support that Windows already relies on.
Signed-off-by: NPaolo Bonzini <pbonzini@redhat.com>
Reviewed-by: NFam Zheng <famz@redhat.com>
Message-id: 1417518350-6167-2-git-send-email-pbonzini@redhat.com
Signed-off-by: NStefan Hajnoczi <stefanha@redhat.com>

If I understand correctly, qemu-iotests never meant to be portable. We
only support Linux for all the shell cases, but didn't specify it for
python tests. Now add this and default all the python tests as Linux
only. If we cares enough later, we can override the parameter in
individual cases.
Signed-off-by: NFam Zheng <famz@redhat.com>
Signed-off-by: NStefan Hajnoczi <stefanha@redhat.com>

Other cases have this, and this test is not portable as well, as we want
to add "make check-block" to "make check", it shouldn't fail on Mac OS
X.
Reported-by: NPeter Maydell <peter.maydell@linaro.org>
Signed-off-by: NFam Zheng <famz@redhat.com>
Signed-off-by: NStefan Hajnoczi <stefanha@redhat.com>

The former is not portable because on Mac OSX it is /usr/bin/true.
Signed-off-by: NFam Zheng <famz@redhat.com>
Signed-off-by: NStefan Hajnoczi <stefanha@redhat.com>

Signed-off-by: NFam Zheng <famz@redhat.com>
Signed-off-by: NStefan Hajnoczi <stefanha@redhat.com>

Use the 'xl pci-attach $DomU $BDF' command to attach more than
one PCI devices to the guest, then detach the devices with
'xl pci-detach $DomU $BDF', after that, re-attach these PCI
devices again, an error message will be reported like following:

    libxl: error: libxl_qmp.c:287:qmp_handle_error_response: receive
    an error message from QMP server: Duplicate ID 'pci-pt-03_10.1'
    for device.

If using the 'address_space_memory' as the parameter of
'memory_listener_register', 'xen_pt_region_del' will not be called
if the memory region's name is not 'xen-pci-pt-*' when the devices
is detached. This will cause the device's related QemuOpts object
not be released properly.

Using the device's address space can avoid such issue, because the
calling count of 'xen_pt_region_add' when attaching and the calling
count of 'xen_pt_region_del' when detaching is the same, so all the
memory region ref and unref by the 'xen_pt_region_add' and
'xen_pt_region_del' can be released properly.
Signed-off-by: NLiang Li <liang.z.li@intel.com>
Reviewed-by: NPaolo Bonzini <pbonzini@redhat.com>
Reported-by: NLongtao Pang <longtaox.pang@intel.com>

The allocator in malloc-pc has been extracted, so it can be used in every arch.
This operation showed that both the alloc and free functions can be also
generic.
Because of this, the QGuestAllocator has been removed from is function to wrap
the alloc and free function, and now just contains the allocator parameters.
As a result, only the allocator initalizer and unitializer are arch dependent.
Signed-off-by: NMarc Marí <marc.mari.barcelo@gmail.com>
Reviewed-by: NStefan Hajnoczi <stefanha@redhat.com>
Reviewed-by: NJohn Snow <jsnow@redhat.com>
Signed-off-by: NStefan Hajnoczi <stefanha@redhat.com>

Because of wrong return value of .save_live_pending() in
migration/block.c, migration finishes before the whole disk is
transferred. Such situation occurs when the migration process is fast
enough, for example when source and dest are on the same host.

If in the bulk phase we return something < max_size, we will skip
transferring the tail of the device. Currently we have "set pending to
BLOCK_SIZE if it is zero" for bulk phase, but there no guarantee, that
it will be < max_size.

True approach is to return, for example, max_size+1 when we are in the
bulk phase.
Signed-off-by: NVladimir Sementsov-Ogievskiy <vsementsov@parallels.com>
Message-id: 1419933856-4018-2-git-send-email-vsementsov@parallels.com
Signed-off-by: NStefan Hajnoczi <stefanha@redhat.com>

Filter out the "main loop: WARNING: I/O thread spun for..." warning from
qemu output (it hardly matters for code specifically testing I/O).

Furthermore, use _filter_qemu in all the custom functions which run
qemu.
Signed-off-by: NMax Reitz <mreitz@redhat.com>
Signed-off-by: NKevin Wolf <kwolf@redhat.com>

This applies cases on drive-backup on blockdev-backup, except cases with
target format and mode.

Also add a case to check source == target.
Signed-off-by: NFam Zheng <famz@redhat.com>
Reviewed-by: NMax Reitz <mreitz@redhat.com>
Reviewed-by: NMarkus Armbruster <armbru@redhat.com>
Message-id: 1418899027-8445-5-git-send-email-famz@redhat.com
Signed-off-by: NMax Reitz <mreitz@redhat.com>

Signed-off-by: NFam Zheng <famz@redhat.com>
Reviewed-by: NJohn Snow <jsnow@redhat.com>
Reviewed-by: NMax Reitz <mreitz@redhat.com>
Reviewed-by: NMarkus Armbruster <armbru@redhat.com>
Message-id: 1418899027-8445-4-git-send-email-famz@redhat.com
Signed-off-by: NMax Reitz <mreitz@redhat.com>

Similar to drive-backup, but this command uses a device id as target
instead of creating/opening an image file.

Also add blocker on target bs, since the target is also a named device
now.

Add check and report error for bs == target which became possible but is
an illegal case with introduction of blockdev-backup.
Signed-off-by: NFam Zheng <famz@redhat.com>
Reviewed-by: NMax Reitz <mreitz@redhat.com>
Reviewed-by: NJohn Snow <jsnow@redhat.com>
Reviewed-by: NMarkus Armbruster <armbru@redhat.com>
Message-id: 1418899027-8445-3-git-send-email-famz@redhat.com
Signed-off-by: NMax Reitz <mreitz@redhat.com>

Signed-off-by: NFam Zheng <famz@redhat.com>
Reviewed-by: NEric Blake <eblake@redhat.com>
Reviewed-by: NMax Reitz <mreitz@redhat.com>
Reviewed-by: NMarkus Armbruster <armbru@redhat.com>
Message-id: 1418899027-8445-2-git-send-email-famz@redhat.com
Signed-off-by: NMax Reitz <mreitz@redhat.com>

Mirror and migration use dirty bitmaps for their purposes, and since
commit [block: per caller dirty bitmap] they use their own bitmaps, not
the global one. But they use old functions bdrv_set_dirty and
bdrv_reset_dirty, which change all dirty bitmaps.

Named dirty bitmaps series by Fam and Snow are affected: mirroring and
migration will spoil all (not related to this mirroring or migration)
named dirty bitmaps.

This patch fixes this by adding bdrv_set_dirty_bitmap and
bdrv_reset_dirty_bitmap, which change concrete bitmap. Also, to prevent
such mistakes in future, old functions bdrv_(set,reset)_dirty are made
static, for internal block usage.
Signed-off-by: NVladimir Sementsov-Ogievskiy <vsementsov@parallels.com>
CC: John Snow <jsnow@redhat.com>
CC: Fam Zheng <famz@redhat.com>
CC: Denis V. Lunev <den@openvz.org>
CC: Stefan Hajnoczi <stefanha@redhat.com>
CC: Kevin Wolf <kwolf@redhat.com>
Reviewed-by: NJohn Snow <jsnow@redhat.com>
Reviewed-by: NFam Zheng <famz@redhat.com>
Message-id: 1417081246-3593-1-git-send-email-vsementsov@parallels.com
Signed-off-by: NMax Reitz <mreitz@redhat.com>