Fixes: 5da7c35e("bios-tables-test: Add Q35/TPM-TIS test")
Signed-off-by: NLi Qiang <liq3ea@163.com>
Message-Id: <20200714153536.66060-1-liq3ea@163.com>
Reviewed-by: NEric Auger <eric.auger@redhat.com>
Reviewed-by: NIgor Mammedov <imammedo@redhat.com>
Signed-off-by: NThomas Huth <thuth@redhat.com>

fixes: usb storage regression, vfio display ramfb bug

# gpg: Signature made Thu 16 Jul 2020 10:30:58 BST
# gpg:                using RSA key 4CB6D8EED3E87138
# gpg: Good signature from "Gerd Hoffmann (work) <kraxel@redhat.com>" [full]
# gpg:                 aka "Gerd Hoffmann <gerd@kraxel.org>" [full]
# gpg:                 aka "Gerd Hoffmann (private) <kraxel@gmail.com>" [full]
# Primary key fingerprint: A032 8CFF B93A 17A7 9901  FE7D 4CB6 D8EE D3E8 7138

* remotes/kraxel/tags/fixes-20200716-pull-request:
  usb: fix storage regression
  vfio: fix use-after-free in display
Signed-off-by: NPeter Maydell <peter.maydell@linaro.org>

Merge tpm 2020/07/15 v1

# gpg: Signature made Wed 15 Jul 2020 20:16:21 BST
# gpg:                using RSA key B818B9CADF9089C2D5CEC66B75AD65802A0B4211
# gpg: Good signature from "Stefan Berger <stefanb@linux.vnet.ibm.com>" [unknown]
# gpg: WARNING: This key is not certified with a trusted signature!
# gpg:          There is no indication that the signature belongs to the owner.
# Primary key fingerprint: B818 B9CA DF90 89C2 D5CE  C66B 75AD 6580 2A0B 4211

* remotes/stefanberger/tags/pull-tpm-2020-07-15-1:
  tests: tpm: Skip over pcrUpdateCounter byte in result comparison
  tpm: tpm_spapr: Exit on TPM backend failures
Signed-off-by: NPeter Maydell <peter.maydell@linaro.org>

# gpg: Signature made Wed 15 Jul 2020 14:49:07 BST
# gpg:                using RSA key EF04965B398D6211
# gpg: Good signature from "Jason Wang (Jason Wang on RedHat) <jasowang@redhat.com>" [marginal]
# gpg: WARNING: This key is not certified with sufficiently trusted signatures!
# gpg:          It is not certain that the signature belongs to the owner.
# Primary key fingerprint: 215D 46F4 8246 689E C77F  3562 EF04 965B 398D 6211

* remotes/jasowang/tags/net-pull-request:
  ftgmac100: fix dblac write test
  net: detect errors from probing vnet hdr flag for TAP devices
  net: check if the file descriptor is valid before using it
  qemu-options.hx: Clean up and fix typo for colo-compare
  net/colo-compare.c: Expose compare "max_queue_size" to users
  hw/net: Added CSO for IPv6
  virtio-net: fix removal of failover device
Signed-off-by: NPeter Maydell <peter.maydell@linaro.org>

Fix the contition to figure whenever we need to wait for more data or
not.  Simply check the mode, if we are not in DATAIN state any more we
are done already and don't need to go ASYNC.

Fixes: 7ad3d51e ("usb: add short-packet handling to usb-storage driver")
Reported-by: NSai Pavan Boddu <saipava@xilinx.com>
Tested-by: NPaul Zimmerman <pauldzim@gmail.com>
Signed-off-by: NGerd Hoffmann <kraxel@redhat.com>
Message-id: 20200713062712.1476-1-kraxel@redhat.com

Calling ramfb_display_update() might replace the DisplaySurface with the
boot display, which in turn will free the currently active
DisplaySurface.

So clear our DisplaySurface pinter (dpy->region.surface pointer) to (a)
avoid use-after-free and (b) force replacing the boot display with the
real display when switching back.
Signed-off-by: NGerd Hoffmann <kraxel@redhat.com>
Reviewed-by: NAlex Williamson <alex.williamson@redhat.com>
Acked-by: NAlex Williamson <alex.williamson@redhat.com>
Message-id: 20200713124520.23266-1-kraxel@redhat.com

The TPM 2 code in libtpms was fixed to handle the PCR 'TCB group' according
to the PCClient profile. The change of the PCRs belonging to the 'TCB group'
now affects the pcrUpdateCounter in the TPM2_PCRRead() responses where its
value is now different (typically lower by '1') than what it was before. To
not fail the tests, we skip the comparison of the 14th byte, which
represents the pcrUpdateCounter.
Signed-off-by: NStefan Berger <stefanb@linux.ibm.com>
Reviewed-by: NMarc-André Lureau <marcandre.lureau@redhat.com>
Message-id: 20200707201625.4177419-3-stefanb@linux.vnet.ibm.com

Exit on TPM backend failures in the same way as the TPM CRB and TIS device
models do. With this change we now get an error report when the backend
did not start up properly:

error: internal error: qemu unexpectedly closed the monitor:
2020-07-07T12:49:28.333928Z qemu-system-ppc64: tpm-emulator: \
  TPM result for CMD_INIT: 0x101 operation failed
Signed-off-by: NStefan Berger <stefanb@linux.ibm.com>
Reviewed-by: NMarc-André Lureau <marcandre.lureau@redhat.com>
Message-id: 20200707201625.4177419-2-stefanb@linux.vnet.ibm.com

Signed-off-by: NPeter Maydell <peter.maydell@linaro.org>

Final fixes for 5.1-rc0

  - minor documentation nit
  - docker.py bootstrap fixes
  - tweak containers.yml wildcards
  - fix float16 nan detection
  - conditional use of -Wpsabi
  - fix missing iotlb data for plugins
  - proper locking for helper based bb count
  - drop ppc64abi32 from the plugin check-tcg test

# gpg: Signature made Wed 15 Jul 2020 11:59:08 BST
# gpg:                using RSA key 6685AE99E75167BCAFC8DF35FBD0DB095A9E2A44
# gpg: Good signature from "Alex Bennée (Master Work Key) <alex.bennee@linaro.org>" [full]
# Primary key fingerprint: 6685 AE99 E751 67BC AFC8  DF35 FBD0 DB09 5A9E 2A44

* remotes/stsquad/tags/pull-misc-for-rc0-150720-3:
  .travis.yml: skip ppc64abi32-linux-user with plugins
  plugins: expand the bb plugin to be thread safe and track per-cpu
  cputlb: ensure we save the IOTLB data in case of reset
  tests/plugins: don't unconditionally add -Wpsabi
  fpu/softfloat: fix up float16 nan recognition
  gitlab-ci/containers: Add missing wildcard where we should look for changes
  docker.py: fix fetching of FROM layers
  tests/docker: Remove the libssh workaround from the ubuntu 20.04 image
  docs/devel: fix grammar in multi-thread-tcg
Signed-off-by: NPeter Maydell <peter.maydell@linaro.org>

The test of the write of the dblac register was testing the old value
instead of the new value. This would accept the write of an invalid value
but subsequently refuse any following valid writes.
Signed-off-by: Nerik-smit <erik.lucas.smit@gmail.com>
Reviewed-by: NCédric Le Goater <clg@kaod.org>
Signed-off-by: NJason Wang <jasowang@redhat.com>

When QEMU sets up a tap based network device backend, it mostly ignores errors
reported from various ioctl() calls it makes, assuming the TAP file descriptor
is valid. This assumption can easily be violated when the user is passing in a
pre-opened file descriptor. At best, the ioctls may fail with a -EBADF, but if
the user passes in a bogus FD number that happens to clash with a FD number that
QEMU has opened internally for another reason, a wide variety of errnos may
result, as the TUNGETIFF ioctl number may map to a completely different command
on a different type of file.

By ignoring all these errors, QEMU sets up a zombie network backend that will
never pass any data. Even worse, when QEMU shuts down, or that network backend
is hot-removed, it will close this bogus file descriptor, which could belong to
another QEMU device backend.

There's no obvious guaranteed reliable way to detect that a FD genuinely is a
TAP device, as opposed to a UNIX socket, or pipe, or something else. Checking
the errno from probing vnet hdr flag though, does catch the big common cases.
ie calling TUNGETIFF will return EBADF for an invalid FD, and ENOTTY when FD is
a UNIX socket, or pipe which catches accidental collisions with FDs used for
stdio, or monitor socket.

Previously the example below where bogus fd 9 collides with the FD used for the
chardev saw:

$ ./x86_64-softmmu/qemu-system-x86_64 -netdev tap,id=hostnet0,fd=9 \
  -chardev socket,id=charchannel0,path=/tmp/qga,server,nowait \
  -monitor stdio -vnc :0
qemu-system-x86_64: -netdev tap,id=hostnet0,fd=9: TUNGETIFF ioctl() failed: Inappropriate ioctl for device
TUNSETOFFLOAD ioctl() failed: Bad address
QEMU 2.9.1 monitor - type 'help' for more information
(qemu) Warning: netdev hostnet0 has no peer

which gives a running QEMU with a zombie network backend.

With this change applied we get an error message and QEMU immediately exits
before carrying on and making a bigger disaster:

$ ./x86_64-softmmu/qemu-system-x86_64 -netdev tap,id=hostnet0,fd=9 \
  -chardev socket,id=charchannel0,path=/tmp/qga,server,nowait \
  -monitor stdio -vnc :0
qemu-system-x86_64: -netdev tap,id=hostnet0,vhost=on,fd=9: Unable to query TUNGETIFF on FD 9: Inappropriate ioctl for device
Reported-by: NDr. David Alan Gilbert <dgilbert@redhat.com>
Signed-off-by: NDaniel P. Berrange <berrange@redhat.com>
Tested-by: NDr. David Alan Gilbert <dgilbert@redhat.com>
Message-id: 20171027085548.3472-1-berrange@redhat.com
[lv: to simplify, don't check on EINVAL with TUNGETIFF as it exists since v2.6.27]
Signed-off-by: NLaurent Vivier <lvivier@redhat.com>
Signed-off-by: NJason Wang <jasowang@redhat.com>

qemu_set_nonblock() checks that the file descriptor can be used and, if
not, crashes QEMU. An assert() is used for that. The use of assert() is
used to detect programming error and the coredump will allow to debug
the problem.

But in the case of the tap device, this assert() can be triggered by
a misconfiguration by the user. At startup, it's not a real problem, but it
can also happen during the hot-plug of a new device, and here it's a
problem because we can crash a perfectly healthy system.

For instance:
 # ip link add link virbr0 name macvtap0 type macvtap mode bridge
 # ip link set macvtap0 up
 # TAP=/dev/tap$(ip -o link show macvtap0 | cut -d: -f1)
 # qemu-system-x86_64 -machine q35 -device pcie-root-port,id=pcie-root-port-0 -monitor stdio 9<> $TAP
 (qemu) netdev_add type=tap,id=hostnet0,vhost=on,fd=9
 (qemu) device_add driver=virtio-net-pci,netdev=hostnet0,id=net0,bus=pcie-root-port-0
 (qemu) device_del net0
 (qemu) netdev_del hostnet0
 (qemu) netdev_add type=tap,id=hostnet1,vhost=on,fd=9
 qemu-system-x86_64: .../util/oslib-posix.c:247: qemu_set_nonblock: Assertion `f != -1' failed.
 Aborted (core dumped)

To avoid that, add a function, qemu_try_set_nonblock(), that allows to report the
problem without crashing.

In the same way, we also update the function for vhostfd in net_init_tap_one() and
for fd in net_init_socket() (both descriptors are provided by the user and can
be wrong).
Signed-off-by: NLaurent Vivier <lvivier@redhat.com>
Reviewed-by: NPhilippe Mathieu-Daudé <philmd@redhat.com>
Signed-off-by: NJason Wang <jasowang@redhat.com>

Fix some typo and optimized some descriptions.
Signed-off-by: NZhang Chen <chen.zhang@intel.com>
Signed-off-by: NJason Wang <jasowang@redhat.com>

This patch allow users to set the "max_queue_size" according
to their environment.
Signed-off-by: NZhang Chen <chen.zhang@intel.com>
Signed-off-by: NJason Wang <jasowang@redhat.com>

Added fix for checksum offload for IPv6 if a backend doesn't
have a virtual header.
This patch is a part of IPv6 fragmentation.
Signed-off-by: NAndrew Melnychenko <andrew@daynix.com>
Signed-off-by: NJason Wang <jasowang@redhat.com>

If you have a networking device and its virtio failover device, and
you remove them in this order:
- virtio device
- the real device

You get qemu crash.
See bugzilla: https://bugzilla.redhat.com/show_bug.cgi?id=1820120

Bug exist on qemu 4.2 and 5.0.
But in 5.0 don't shows because commit
77b06bba

somehow papers over it.

CC: Jason Wang <jasowang@redhat.com>
CC: Michael S. Tsirkin <mst@redhat.com>
Reviewed-by: NMichael S. Tsirkin <mst@redhat.com>
Acked-by: NLaurent Vivier <lvivier@redhat.com>
Signed-off-by: NJuan Quintela <quintela@redhat.com>
Signed-off-by: NJason Wang <jasowang@redhat.com>

MIPS patches for 5.1

- A pair of fixes,
- Add Huacai Chen as MIPS KVM maintainer,
- Add Jiaxun Yang as designated MIPS TCG reviewer.

CI jobs results:
. https://travis-ci.org/github/philmd/qemu/builds/708079271
. https://gitlab.com/philmd/qemu/-/pipelines/166528104
. https://cirrus-ci.com/build/6483996878045184

# gpg: Signature made Tue 14 Jul 2020 20:59:58 BST
# gpg:                using RSA key FAABE75E12917221DCFD6BB2E3E32C2CDEADC0DE
# gpg: Good signature from "Philippe Mathieu-Daudé (F4BUG) <f4bug@amsat.org>" [full]
# Primary key fingerprint: FAAB E75E 1291 7221 DCFD  6BB2 E3E3 2C2C DEAD C0DE

* remotes/philmd-gitlab/tags/mips-next-20200714:
  MAINTAINERS: Adjust MIPS maintainership (add Huacai Chen & Jiaxun Yang)
  target/mips: Fix ADD.S FPU instruction
  target/mips: Remove identical if/else branches
Signed-off-by: NPeter Maydell <peter.maydell@linaro.org>

Python patches for 5.1

- Reduce race conditions on QEMUMachine::shutdown()

 1. Remove the "bare except" pattern in the existing shutdown code,
    which can mask problems and make debugging difficult.
 2. Ensure that post-shutdown cleanup is always performed, even when
    graceful termination fails.
 3. Unify cleanup paths such that no matter how the VM is terminated,
    the same functions and steps are always taken to reset the object
    state.
 4. Rewrite shutdown() such that any error encountered when attempting
    a graceful shutdown will be raised as an AbnormalShutdown exception.
    The pythonic idiom is to allow the caller to decide if this is a
    problem or not.

- Modify part of the python/qemu library to comply with:

  . mypy --strict
  . pylint
  . flake8

- Script for the TCG Continuous Benchmarking project that uses
  callgrind to dissect QEMU execution into three main phases:

  . code generation
  . JIT execution
  . helpers execution

CI jobs results:
. https://cirrus-ci.com/build/5421349961203712
. https://gitlab.com/philmd/qemu/-/pipelines/166556001
. https://travis-ci.org/github/philmd/qemu/builds/708102347

# gpg: Signature made Tue 14 Jul 2020 21:40:05 BST
# gpg:                using RSA key FAABE75E12917221DCFD6BB2E3E32C2CDEADC0DE
# gpg: Good signature from "Philippe Mathieu-Daudé (F4BUG) <f4bug@amsat.org>" [full]
# Primary key fingerprint: FAAB E75E 1291 7221 DCFD  6BB2 E3E3 2C2C DEAD C0DE

* remotes/philmd-gitlab/tags/python-next-20200714:
  python/qmp.py: add QMPProtocolError
  python/qmp.py: add casts to JSON deserialization
  python/qmp.py: Do not return None from cmd_obj
  python/qmp.py: re-absorb MonitorResponseError
  iotests.py: use qemu.qmp type aliases
  python/qmp.py: Define common types
  python/machine.py: change default wait timeout to 3 seconds
  python/machine.py: re-add sigkill warning suppression
  python/machine.py: split shutdown into hard and soft flavors
  tests/acceptance: Don't test reboot on cubieboard
  tests/acceptance: wait() instead of shutdown() where appropriate
  python/machine.py: Make wait() call shutdown()
  python/machine.py: Add a configurable timeout to shutdown()
  python/machine.py: Prohibit multiple shutdown() calls
  python/machine.py: Perform early cleanup for wait() calls, too
  python/machine.py: Add _early_cleanup hook
  python/machine.py: Close QMP socket in cleanup
  python/machine.py: consolidate _post_shutdown()
  scripts/performance: Add dissect.py script
Signed-off-by: NPeter Maydell <peter.maydell@linaro.org>

We actually see failures on threadcount running without plugins:

  retry.py -n 1000 -c -- \
    ./ppc64abi32-linux-user/qemu-ppc64abi32 \
    ./tests/tcg/ppc64abi32-linux-user/threadcount

which reports:

  0: 978 times (97.80%), avg time 0.270 (0.01 varience/0.08 deviation)
  -6: 21 times (2.10%), avg time 0.336 (0.01 varience/0.12 deviation)
  -11: 1 times (0.10%), avg time 0.502 (0.00 varience/0.00 deviation)
  Ran command 1000 times, 978 passes

But when running with plugins we hit the failure a lot more often:

  0: 91 times (91.00%), avg time 0.302 (0.04 varience/0.19 deviation)
  -11: 9 times (9.00%), avg time 0.558 (0.01 varience/0.11 deviation)
  Ran command 100 times, 91 passes

The crash occurs in guest code which is the same in both pass and fail
cases. However we see various messages reported on the console about
corrupted memory lists which seems to imply the guest memory allocation
is corrupted. This lines up with the seg fault being in the guest
__libc_free function. So we think this is a guest bug which is
exacerbated by various modes of translation. If anyone has access to
real hardware to soak test the test case we could prove this properly.
Signed-off-by: NAlex Bennée <alex.bennee@linaro.org>
Acked-by: NDavid Gibson <david@gibson.dropbear.id.au>
Reviewed-by: NPhilippe Mathieu-Daudé <f4bug@amsat.org>
Tested-by: NPhilippe Mathieu-Daudé <f4bug@amsat.org>
Message-Id: <20200714175516.5475-1-alex.bennee@linaro.org>

While there isn't any easy way to make the inline counts thread safe
we can ensure the callback based ones are. While we are at it we can
reduce introduce a new option ("idle") to dump a report of the current
bb and insn count each time a vCPU enters the idle state.
Signed-off-by: NAlex Bennée <alex.bennee@linaro.org>
Reviewed-by: NRobert Foley <robert.foley@linaro.org>
Cc: Dave Bort <dbort@dbort.com>

Message-Id: <20200713200415.26214-8-alex.bennee@linaro.org>

Any write to a device might cause a re-arrangement of memory
triggering a TLB flush and potential re-size of the TLB invalidating
previous entries. This would cause users of qemu_plugin_get_hwaddr()
to see the warning:

  invalid use of qemu_plugin_get_hwaddr

because of the failed tlb_lookup which should always succeed. To
prevent this we save the IOTLB data in case it is later needed by a
plugin doing a lookup.
Signed-off-by: NAlex Bennée <alex.bennee@linaro.org>
Reviewed-by: NRichard Henderson <richard.henderson@linaro.org>
Message-Id: <20200713200415.26214-7-alex.bennee@linaro.org>

Not all compilers support the -Wpsabi (clang-9 in my case). To handle
this gracefully we pare back the shared build machinery so the
Makefile is relatively "standalone". We still take advantage of
config-host.mak as configure has done a bunch of probing for us but
that is it.

Fixes: bac8d222Signed-off-by: NAlex Bennée <alex.bennee@linaro.org>
Reviewed-by: NThomas Huth <thuth@redhat.com>
Message-Id: <20200713200415.26214-6-alex.bennee@linaro.org>

Signed-off-by: NLIU Zhiwei <zhiwei_liu@c-sky.com>
Signed-off-by: NAlex Bennée <alex.bennee@linaro.org>
Reviewed-by: NRichard Henderson <richard.henderson@linaro.org>
Message-Id: <20200712234521.3972-2-zhiwei_liu@c-sky.com>
Message-Id: <20200713200415.26214-5-alex.bennee@linaro.org>

The tests/docker/* wildcard seems to only match the files that are directly
in the tests/docker folder - but changes to the files in the directory
tests/docker/dockerfiles are currently ignored. Seems like we need a
separate entry to match the files in that folder. With this wildcard added,
the stages now get re-run successfully when something in the dockerfiles
has been changed.
Signed-off-by: NThomas Huth <thuth@redhat.com>
Reviewed-by: NPhilippe Mathieu-Daudé <f4bug@amsat.org>
Signed-off-by: NAlex Bennée <alex.bennee@linaro.org>
Message-Id: <20200713182235.30379-1-thuth@redhat.com>

This worked on a system that was already bootstrapped because the
stage 2 images already existed even if they wouldn't be used. What we
should have pulled down was the FROM line containers first because
building on gitlab doesn't have the advantage of using our build
system to build the pre-requisite bits.

We still pull the image we want to build just in case we can use the
cached data.
Signed-off-by: NAlex Bennée <alex.bennee@linaro.org>
Reviewed-by: NDaniel P. Berrangé <berrange@redhat.com>
Tested-by: NPhilippe Mathieu-Daudé <f4bug@amsat.org>
Message-Id: <20200713200415.26214-4-alex.bennee@linaro.org>

The libssh problem only exists in Ubuntu 18.04 - we can enable it
in 20.04 again.
Signed-off-by: NThomas Huth <thuth@redhat.com>
Signed-off-by: NAlex Bennée <alex.bennee@linaro.org>
Reviewed-by: NPhilippe Mathieu-Daudé <f4bug@amsat.org>
Message-Id: <20200713185237.9419-1-thuth@redhat.com>
Message-Id: <20200713200415.26214-3-alex.bennee@linaro.org>

Review comment came just too late ;-)
Signed-off-by: NAlex Bennée <alex.bennee@linaro.org>
Reviewed-by: NRichard Henderson <richard.henderson@linaro.org>
Reviewed-by: NThomas Huth <thuth@redhat.com>
Reviewed-by: NPhilippe Mathieu-Daudé <f4bug@amsat.org>
Message-Id: <20200713200415.26214-9-alex.bennee@linaro.org>

Merge remote-tracking branch 'remotes/philmd-gitlab/tags/sdcard-CVE-2020-13253-pull-request' into staging

Fix CVE-2020-13253

By using invalidated address, guest can do out-of-bounds accesses.
These patches fix the issue by only allowing SD card image sizes
power of 2, and not switching to SEND_DATA state when the address
is invalid (out of range).

This issue was found using QEMU fuzzing mode (using --enable-fuzzing,
see docs/devel/fuzzing.txt) and reported by Alexander Bulekov.

Reproducer:
  https://bugs.launchpad.net/qemu/+bug/1880822/comments/1

CI jobs results:
. https://cirrus-ci.com/build/5157142548185088
. https://gitlab.com/philmd/qemu/-/pipelines/166381731
. https://travis-ci.org/github/philmd/qemu/builds/707956535

# gpg: Signature made Tue 14 Jul 2020 14:54:44 BST
# gpg:                using RSA key FAABE75E12917221DCFD6BB2E3E32C2CDEADC0DE
# gpg: Good signature from "Philippe Mathieu-Daudé (F4BUG) <f4bug@amsat.org>" [full]
# Primary key fingerprint: FAAB E75E 1291 7221 DCFD  6BB2 E3E3 2C2C DEAD C0DE

* remotes/philmd-gitlab/tags/sdcard-CVE-2020-13253-pull-request:
  hw/sd/sdcard: Do not switch to ReceivingData if address is invalid
  hw/sd/sdcard: Update coding style to make checkpatch.pl happy
  hw/sd/sdcard: Do not allow invalid SD card sizes
  hw/sd/sdcard: Simplify realize() a bit
  hw/sd/sdcard: Restrict Class 6 commands to SCSD cards
  tests/acceptance/boot_linux: Expand SD card image to power of 2
  tests/acceptance/boot_linux: Tag tests using a SD card with 'device:sd'
  docs/orangepi: Add instructions for resizing SD image to power of two
  MAINTAINERS: Cc qemu-block mailing list
Signed-off-by: NPeter Maydell <peter.maydell@linaro.org>

In the case that we receive a reply but are unable to understand it,
use this exception name to indicate that case.
Signed-off-by: NJohn Snow <jsnow@redhat.com>
Reviewed-by: NPhilippe Mathieu-Daudé <philmd@redhat.com>
Reviewed-by: NKevin Wolf <kwolf@redhat.com>
Message-Id: <20200710052220.3306-7-jsnow@redhat.com>
Signed-off-by: NPhilippe Mathieu-Daudé <philmd@redhat.com>

mypy and python type hints are not powerful enough to properly describe
JSON messages in Python 3.6. The best we can do, generally, is describe
them as Dict[str, Any].

Add casts to coerce this type for static analysis; but do NOT enforce
this type at runtime in any way.

Note: Python 3.8 adds a TypedDict construct which allows for the
description of more arbitrary Dictionary shapes. There is a third-party
module, "Pydantic", which is compatible with 3.6 that can be used
instead of the JSON library that parses JSON messages to fully-typed
Python objects, and may be preferable in some cases.

(That is well beyond the scope of this commit or series.)
Signed-off-by: NJohn Snow <jsnow@redhat.com>
Reviewed-by: NKevin Wolf <kwolf@redhat.com>
Message-Id: <20200710052220.3306-6-jsnow@redhat.com>
Signed-off-by: NPhilippe Mathieu-Daudé <philmd@redhat.com>

This makes typing the qmp library difficult, as it necessitates wrapping
Optional[] around the type for every return type up the stack. At some
point, it becomes difficult to discern or remember why it's None instead
of the expected object.

Use the python exception system to tell us exactly why we didn't get an
object. Remove this special-cased return.
Signed-off-by: NJohn Snow <jsnow@redhat.com>
Reviewed-by: NKevin Wolf <kwolf@redhat.com>
Message-Id: <20200710052220.3306-5-jsnow@redhat.com>
Signed-off-by: NPhilippe Mathieu-Daudé <philmd@redhat.com>

When I initially split this out, I considered this more of a machine
error than a QMP protocol error, but I think that's misguided.

Move this back to qmp.py and name it QMPResponseError. Convert
qmp.command() to use this exception type.
Signed-off-by: NJohn Snow <jsnow@redhat.com>
Reviewed-by: NPhilippe Mathieu-Daudé <philmd@redhat.com>
Reviewed-by: NKevin Wolf <kwolf@redhat.com>
Message-Id: <20200710052220.3306-4-jsnow@redhat.com>
Signed-off-by: NPhilippe Mathieu-Daudé <philmd@redhat.com>

iotests.py should use the type definitions from qmp.py instead of its
own.
Signed-off-by: NJohn Snow <jsnow@redhat.com>
Reviewed-by: NKevin Wolf <kwolf@redhat.com>
Message-Id: <20200710052220.3306-3-jsnow@redhat.com>
Signed-off-by: NPhilippe Mathieu-Daudé <philmd@redhat.com>

Define some common types that we'll need to annotate a lot of other
functions going forward.
Signed-off-by: NJohn Snow <jsnow@redhat.com>
Reviewed-by: NKevin Wolf <kwolf@redhat.com>
Message-Id: <20200710052220.3306-2-jsnow@redhat.com>
Signed-off-by: NPhilippe Mathieu-Daudé <philmd@redhat.com>

Machine.wait() does not appear to be used except in the acceptance tests,
and an infinite timeout by default in a test suite is not the most helpful.

Change it to 3 seconds, like the default shutdown timeout.
Signed-off-by: NJohn Snow <jsnow@redhat.com>
Reviewed-by: NPhilippe Mathieu-Daudé <philmd@redhat.com>
Reviewed-by: NCleber Rosa <crosa@redhat.com>
Tested-by: NCleber Rosa <crosa@redhat.com>
Message-Id: <20200710050649.32434-13-jsnow@redhat.com>
Signed-off-by: NPhilippe Mathieu-Daudé <philmd@redhat.com>

If the user kills QEMU on purpose, we don't need to warn
them about that having happened: they know already.
Signed-off-by: NJohn Snow <jsnow@redhat.com>
Reviewed-by: NPhilippe Mathieu-Daudé <philmd@redhat.com>
Reviewed-by: NCleber Rosa <crosa@redhat.com>
Message-Id: <20200710050649.32434-12-jsnow@redhat.com>
Signed-off-by: NPhilippe Mathieu-Daudé <philmd@redhat.com>

This is done primarily to avoid the 'bare except' pattern, which
suppresses all exceptions during shutdown and can obscure errors.

Replace this with a pattern that isolates the different kind of shutdown
paradigms (_hard_shutdown and _soft_shutdown), and a new fallback shutdown
handler (_do_shutdown) that gracefully attempts one before the other.

This split now also ensures that no matter what happens,
_post_shutdown() is always invoked.

shutdown() changes in behavior such that if it attempts to do a graceful
shutdown and is unable to, it will now always raise an exception to
indicate this. This can be avoided by the test writer in three ways:

1. If the VM is expected to have already exited or is in the process of
exiting, wait() can be used instead of shutdown() to clean up resources
instead. This helps avoid race conditions in shutdown.

2. If a test writer is expecting graceful shutdown to fail, shutdown
should be called in a try...except block.

3. If the test writer has no interest in performing a graceful shutdown
at all, kill() can be used instead.

Handling shutdown in this way makes it much more explicit which type of
shutdown we want and allows the library to report problems with this
process.
Signed-off-by: NJohn Snow <jsnow@redhat.com>
Reviewed-by: NPhilippe Mathieu-Daudé <philmd@redhat.com>
Reviewed-by: NCleber Rosa <crosa@redhat.com>
Tested-by: NCleber Rosa <crosa@redhat.com>
Message-Id: <20200710050649.32434-11-jsnow@redhat.com>
Signed-off-by: NPhilippe Mathieu-Daudé <philmd@redhat.com>

cubieboard does not have a functioning reboot, it halts and QEMU does
not exit.

vm.shutdown() is modified in a forthcoming patch that makes it less tolerant
of race conditions on shutdown; tests should consciously decide to WAIT
or to SHUTDOWN qemu.

So long as this test is attempting to reboot, the correct choice would
be to WAIT for the VM to exit. However, since that's broken, we should
SHUTDOWN instead.

SHUTDOWN is indeed what already happens when the test performs teardown,
however, if anyone fixes cubieboard reboot in the future, this test will
develop a new race condition that might be hard to debug.

Therefore: remove the reboot test and make it obvious that the VM is
still running when the test concludes, where the test teardown will do
the right thing.
Signed-off-by: NJohn Snow <jsnow@redhat.com>
Reviewed-by: NPhilippe Mathieu-Daudé <philmd@redhat.com>
Reviewed-by: NCleber Rosa <crosa@redhat.com>
Tested-by: NCleber Rosa <crosa@redhat.com>
Message-Id: <20200710050649.32434-10-jsnow@redhat.com>
Signed-off-by: NPhilippe Mathieu-Daudé <philmd@redhat.com>

When issuing 'reboot' to a VM with the no-reboot option, that VM will
exit. When then issuing a shutdown command, the cleanup may race.

Add calls to vm.wait() which will gracefully mark the VM as having
exited. Subsequent vm.shutdown() calls in generic tearDown code will not
race when called after completion of the call.
Signed-off-by: NJohn Snow <jsnow@redhat.com>
Reviewed-by: NPhilippe Mathieu-Daudé <philmd@redhat.com>
Reviewed-by: NCleber Rosa <crosa@redhat.com>
Tested-by: NCleber Rosa <crosa@redhat.com>
Message-Id: <20200710050649.32434-9-jsnow@redhat.com>
Signed-off-by: NPhilippe Mathieu-Daudé <philmd@redhat.com>