Python queue, 2018-02-05

# gpg: Signature made Mon 05 Feb 2018 23:07:57 GMT
# gpg:                using RSA key 2807936F984DC5A6
# gpg: Good signature from "Eduardo Habkost <ehabkost@redhat.com>"
# Primary key fingerprint: 5A32 2FD5 ABC4 D3DB ACCF  D1AA 2807 936F 984D C5A6

* remotes/ehabkost/tags/python-next-pull-request: (21 commits)
  docker: change Fedora images to run with python3
  travis: improve python version test coverage
  ui: update keycodemapdb to get py3 fixes
  input: add missing JIS keys to virtio input
  qemu.py: don't launch again before shutdown()
  qemu.py: cleanup redundant calls in launch()
  qemu.py: use poll() instead of 'returncode'
  qemu.py: always cleanup on shutdown()
  qemu.py: refactor launch()
  qemu.py: better control of created files
  qemu.py: remove unused import
  configure: allow use of python 3
  scripts: ensure signrom treats data as bytes
  qapi: force a UTF-8 locale for running Python
  qapi: ensure stable sort ordering when checking QAPI entities
  qapi: remove '-q' arg to diff when comparing QAPI output
  qapi: Adapt to moved location of 'maketrans' function in py3
  qapi: adapt to moved location of StringIO module in py3
  qapi: Use OrderedDict from standard library if available
  qapi: use items()/values() intead of iteritems()/itervalues()
  ...
Signed-off-by: NPeter Maydell <peter.maydell@linaro.org>

hppa-softmmu update

# gpg: Signature made Sun 04 Feb 2018 22:20:40 GMT
# gpg:                using RSA key 64DF38E8AF7E215F
# gpg: Good signature from "Richard Henderson <richard.henderson@linaro.org>"
# Primary key fingerprint: 7A48 1E78 868B 4DB6 A85A  05C0 64DF 38E8 AF7E 215F

* remotes/rth/tags/pull-hppa-20180204:
  roms/seabios-hppa: Update submodule and image
  tests: Enable boot-serial-test for hppa
  hw/hppa: Use qemu_log_mask instead of fprintf to stderr
Signed-off-by: NPeter Maydell <peter.maydell@linaro.org>

Fedora has switched to Python 3 by default, so it makes sense to use that
for testing QEMU builds, so we get testing of Python 3 compatibility.
Signed-off-by: NDaniel P. Berrange <berrange@redhat.com>
Message-Id: <20180116134217.8725-15-berrange@redhat.com>
Signed-off-by: NEduardo Habkost <ehabkost@redhat.com>

Currently travis declares ancient python 2.4 is desired. Update that to
2.6 which is the oldest version any targetted distros still needs. If we
just list a python 3 version at the top level this will double the
number of travis jobs we run which is unreasonable.

So arbitrarily pick the clang test matrix entries to build with python
3.0 and 3.6, to extend coverage of python versions, without increasing
job count or build time.
Signed-off-by: NDaniel P. Berrange <berrange@redhat.com>
Message-Id: <20180116134217.8725-14-berrange@redhat.com>
Signed-off-by: NEduardo Habkost <ehabkost@redhat.com>

Signed-off-by: NDaniel P. Berrange <berrange@redhat.com>
Message-Id: <20180116134217.8725-13-berrange@redhat.com>
Signed-off-by: NEduardo Habkost <ehabkost@redhat.com>

keycodemapdb updated to add the QKeyCodes muhenkan and katakanahiragana
Signed-off-by: NMiika S <miika9764@gmail.com>
Message-Id: <20180116134217.8725-12-berrange@redhat.com>
Reviewed-by: NEric Blake <eblake@redhat.com>
Signed-off-by: NEduardo Habkost <ehabkost@redhat.com>

If a VM is launched, files are created and a cleanup is required before
a new launch. This cleanup is executed by shutdown(), so shutdown() must
be called even if the VM is manually terminated (i.e. using kill).

This patch creates a control to make sure launch() will not be executed
again if shutdown() is not called after the previous launch().
Signed-off-by: NAmador Pahim <apahim@redhat.com>
Message-Id: <20180122205033.24893-7-apahim@redhat.com>
Signed-off-by: NEduardo Habkost <ehabkost@redhat.com>

Now that shutdown() is guaranteed to always execute self._load_io_log()
and self._post_shutdown(), their calls in 'except' became redundant and
we can safely replace it by a call to shutdown().
Reviewed-by: NFam Zheng <famz@redhat.com>
Reviewed-by: NEduardo Habkost <ehabkost@redhat.com>
Signed-off-by: NAmador Pahim <apahim@redhat.com>
Message-Id: <20180122205033.24893-6-apahim@redhat.com>
Signed-off-by: NEduardo Habkost <ehabkost@redhat.com>

The 'returncode' Popen attribute is not guaranteed to be updated. It
actually depends on a call to either poll(), wait() or communicate().

On the other hand, poll() will: "Check if child process has terminated.
Set and return returncode attribute."

Let's use the poll() to check whether the process is running and to get
the updated process exit code, when the process is finished.
Reviewed-by: NFam Zheng <famz@redhat.com>
eviewed-by: NEduardo Habkost <ehabkost@redhat.com>
Signed-off-by: NAmador Pahim <apahim@redhat.com>
Message-Id: <20180122205033.24893-5-apahim@redhat.com>
Signed-off-by: NEduardo Habkost <ehabkost@redhat.com>

Currently we only cleanup on shutdown() if the VM is running.

To make sure we will always cleanup, this patch makes the
self._load_io_log() and the self._post_shutdown() to
always be called on shutdown(), regardless the VM running state.
Reviewed-by: NFam Zheng <famz@redhat.com>
Reviewed-by: NEduardo Habkost <ehabkost@redhat.com>
Signed-off-by: NAmador Pahim <apahim@redhat.com>
Message-Id: <20180122205033.24893-4-apahim@redhat.com>
Signed-off-by: NEduardo Habkost <ehabkost@redhat.com>

This is just a refactor to separate the exception handler from the
actual launch procedure, improving the readability and making future
maintenances in this piece of code easier.
Reviewed-by: NFam Zheng <famz@redhat.com>
Reviewed-by: NEduardo Habkost <ehabkost@redhat.com>
Signed-off-by: NAmador Pahim <apahim@redhat.com>
Message-Id: <20180122205033.24893-3-apahim@redhat.com>
Signed-off-by: NEduardo Habkost <ehabkost@redhat.com>

To launch a VM, we need to create basically two files: the monitor
socket (if it's a UNIX socket) and the qemu log file.

For the qemu log file, we currently just open the path, which will
create the file if it does not exist or overwrite the file if it does
exist.

For the monitor socket, if it already exists, we are currently removing
it, even if it's not created by us.

This patch moves to _pre_launch() the responsibility to create a
temporary directory to host the files so we can remove the whole
directory on _post_shutdown().
Signed-off-by: NAmador Pahim <apahim@redhat.com>
Message-Id: <20180122205033.24893-2-apahim@redhat.com>
Reviewed-by: NEduardo Habkost <ehabkost@redhat.com>
Signed-off-by: NEduardo Habkost <ehabkost@redhat.com>

Removing 'import sys' as it's not used anywhere.
Signed-off-by: NAmador Pahim <apahim@redhat.com>
Message-Id: <20171114102246.22221-2-apahim@redhat.com>
Signed-off-by: NEduardo Habkost <ehabkost@redhat.com>

Signed-off-by: NDaniel P. Berrange <berrange@redhat.com>
Message-Id: <20180116134217.8725-11-berrange@redhat.com>
Reviewed-by: NEric Blake <eblake@redhat.com>
Signed-off-by: NEduardo Habkost <ehabkost@redhat.com>

Reviewed-by: NPhilippe Mathieu-Daudé <f4bug@amsat.org>
Signed-off-by: NDaniel P. Berrange <berrange@redhat.com>
Message-Id: <20180116134217.8725-10-berrange@redhat.com>
Signed-off-by: NEduardo Habkost <ehabkost@redhat.com>

Python2 did not validate locale correctness when reading input data, so
would happily read UTF-8 data in non-UTF-8 locales. Python3 is strict so
if you try to read UTF-8 data in the C locale, it will raise an error
for any UTF-8 bytes that aren't representable in 7-bit ascii encoding.
e.g.

UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 54: ordinal not in range(128)
Traceback (most recent call last):
  File "/tmp/qemu-test/src/scripts/qapi-commands.py", line 317, in <module>
    schema = QAPISchema(input_file)
  File "/tmp/qemu-test/src/scripts/qapi.py", line 1468, in __init__
    parser = QAPISchemaParser(open(fname, 'r'))
  File "/tmp/qemu-test/src/scripts/qapi.py", line 301, in __init__
    previously_included)
  File "/tmp/qemu-test/src/scripts/qapi.py", line 348, in _include
    exprs_include = QAPISchemaParser(fobj, previously_included, info)
  File "/tmp/qemu-test/src/scripts/qapi.py", line 271, in __init__
    self.src = fp.read()
  File "/usr/lib64/python3.5/encodings/ascii.py", line 26, in decode
    return codecs.ascii_decode(input, self.errors)[0]

More background on this can be seen in

  https://www.python.org/dev/peps/pep-0538/

Many distros support a new C.UTF-8 locale that is like the C locale,
but with UTF-8 instead of 7-bit ASCII. That is not entirely portable
though. This patch thus sets the LANG to "C", but overrides LC_CTYPE
to be en_US.UTF-8 locale. This gets us pretty close to C.UTF-8, but
in a way that should be portable to everywhere QEMU builds.

This patch only forces UTF-8 for QAPI scripts, since that is the one
showing the immediate error under Python3 with C locale, but potentially
we ought to force this for all python scripts used in the build process.
Signed-off-by: NDaniel P. Berrange <berrange@redhat.com>
Message-Id: <20180116134217.8725-9-berrange@redhat.com>
Reviewed-by: NEric Blake <eblake@redhat.com>
Signed-off-by: NEduardo Habkost <ehabkost@redhat.com>

Some early python 3.x versions will have different default
ordering when calling the 'values()' method on a dict, compared
to python 2.x and later 3.x versions. Explicitly sort the items
to get a stable ordering.
Reviewed-by: NPhilippe Mathieu-Daudé <f4bug@amsat.org>
Signed-off-by: NDaniel P. Berrange <berrange@redhat.com>
Message-Id: <20180116134217.8725-8-berrange@redhat.com>
Signed-off-by: NEduardo Habkost <ehabkost@redhat.com>

When the qapi schema tests fail they merely print that the expected
output didn't match the actual output. This is largely useless when
trying diagnose what went wrong. Removing the '-q' arg to diff
means that it is still silent on successful tests, but when it
fails we'll see details of the incorrect output.
Reviewed-by: NEric Blake <eblake@redhat.com>
Signed-off-by: NDaniel P. Berrange <berrange@redhat.com>
Message-Id: <20180116134217.8725-7-berrange@redhat.com>
Signed-off-by: NEduardo Habkost <ehabkost@redhat.com>

Reviewed-by: NEric Blake <eblake@redhat.com>
Signed-off-by: NDaniel P. Berrange <berrange@redhat.com>
Message-Id: <20180116134217.8725-6-berrange@redhat.com>
Signed-off-by: NEduardo Habkost <ehabkost@redhat.com>

Reviewed-by: NPhilippe Mathieu-Daudé <f4bug@amsat.org>
Signed-off-by: NDaniel P. Berrange <berrange@redhat.com>
Message-Id: <20180116134217.8725-5-berrange@redhat.com>
Signed-off-by: NEduardo Habkost <ehabkost@redhat.com>

The OrderedDict class appeared in the 'collections' module
from python 2.7 onwards, so use that in preference to our
local backport if available.
Reviewed-by: NPhilippe Mathieu-Daudé <f4bug@amsat.org>
Signed-off-by: NDaniel P. Berrange <berrange@redhat.com>
Message-Id: <20180116134217.8725-4-berrange@redhat.com>
Signed-off-by: NEduardo Habkost <ehabkost@redhat.com>

The iteritems()/itervalues() methods are gone in py3, but the
items()/values() methods are still around. The latter are less
efficient than the former in py2, but this has unmeasurably
small impact on QEMU build time, so taking portability over
efficiency is a net win.
Reviewed-by: NPhilippe Mathieu-Daudé <f4bug@amsat.org>
Signed-off-by: NDaniel P. Berrange <berrange@redhat.com>
Message-Id: <20180116134217.8725-3-berrange@redhat.com>
Signed-off-by: NEduardo Habkost <ehabkost@redhat.com>

Python 3 no longer supports the bare "print" statement, it must be
called as a normal function with round brackets. It is possible to
opt-in to this new syntax with Python 2.6 onwards by importing the
"print_function" from the "__future__" module, making it easy to
support Python 2 and 3 in parallel.
Reviewed-by: NPhilippe Mathieu-Daudé <f4bug@amsat.org>
Signed-off-by: NDaniel P. Berrange <berrange@redhat.com>
Message-Id: <20180116134217.8725-2-berrange@redhat.com>
Signed-off-by: NEduardo Habkost <ehabkost@redhat.com>

Merge tpm 2018/02/03 v1

# gpg: Signature made Sat 03 Feb 2018 14:02:35 GMT
# gpg:                using RSA key 75AD65802A0B4211
# gpg: Good signature from "Stefan Berger <stefanb@linux.vnet.ibm.com>"
# gpg: WARNING: This key is not certified with a trusted signature!
# gpg:          There is no indication that the signature belongs to the owner.
# Primary key fingerprint: B818 B9CA DF90 89C2 D5CE  C66B 75AD 6580 2A0B 4211

* remotes/stefanberger/tags/pull-tpm-2018-02-03-1:
  tpm: tis: move one-line function into caller
  MAINTAINERS: add pointer to tpm-next repository
  tpm: wrap stX_be_p in tpm_cmd_set_XYZ functions
  tpm: Split off tpm_crb_reset function
Signed-off-by: NPeter Maydell <peter.maydell@linaro.org>

Signed-off-by: NRichard Henderson <richard.henderson@linaro.org>

Reviewed-by: NThomas Huth <thuth@redhat.com>
Signed-off-by: NRichard Henderson <richard.henderson@linaro.org>

Reviewed-by: NAlex Bennée <alex.bennee@linaro.org>
Reported-by: NThomas Huth <thuth@redhat.com>
Signed-off-by: NRichard Henderson <richard.henderson@linaro.org>

Signed-off-by: NStefan Berger <stefanb@linux.vnet.ibm.com>
Reviewed-by: NMarc-André Lureau <marcandre.lureau@redhat.com>

Signed-off-by: NStefan Berger <stefanb@linux.vnet.ibm.com>
Reviewed-by: NEric Blake <eblake@redhat.com>
Reviewed-by: NMarc-André Lureau <marcandre.lureau@redhat.com>

Wrap the calls to stl_be_p and stw_be_p in tpm_cmd_set_XYZ functions
that are similar to existing getters.
Signed-off-by: NStefan Berger <stefanb@linux.vnet.ibm.com>
Reviewed-by: NMarc-André Lureau <marcandre.lureau@redhat.com>

Split off the tpm_crb_reset function part from tpm_crb_realize
that we need to run every time the machine resets.

Also register our reset function with the system since TYPE_DEVICE
seems to not get a reset otherwise.
Signed-off-by: NStefan Berger <stefanb@linux.vnet.ibm.com>
Reviewed-by: NMarc-André Lureau <marcandre.lureau@redhat.com>

ui: use QIONetListener in vnc, bugfixes for sdl1 and vnc.

# gpg: Signature made Fri 02 Feb 2018 07:17:36 GMT
# gpg:                using RSA key 4CB6D8EED3E87138
# gpg: Good signature from "Gerd Hoffmann (work) <kraxel@redhat.com>"
# gpg:                 aka "Gerd Hoffmann <gerd@kraxel.org>"
# gpg:                 aka "Gerd Hoffmann (private) <kraxel@gmail.com>"
# Primary key fingerprint: A032 8CFF B93A 17A7 9901  FE7D 4CB6 D8EE D3E8 7138

* remotes/kraxel/tags/ui-20180202-pull-request:
  ui: correctly advance output buffer when writing SASL data
  ui: convert VNC server to QIONetListener
  ui: fix mixup between qnum and qcode in SDL1 key handling
Signed-off-by: NPeter Maydell <peter.maydell@linaro.org>

virtio-gpu: disallow vIOMMU

# gpg: Signature made Fri 02 Feb 2018 08:31:52 GMT
# gpg:                using RSA key 4CB6D8EED3E87138
# gpg: Good signature from "Gerd Hoffmann (work) <kraxel@redhat.com>"
# gpg:                 aka "Gerd Hoffmann <gerd@kraxel.org>"
# gpg:                 aka "Gerd Hoffmann (private) <kraxel@gmail.com>"
# Primary key fingerprint: A032 8CFF B93A 17A7 9901  FE7D 4CB6 D8EE D3E8 7138

* remotes/kraxel/tags/vga-20180202-pull-request:
  virtio-gpu: disallow vIOMMU
Signed-off-by: NPeter Maydell <peter.maydell@linaro.org>

This series is mostly about 9p request cancellation. It fixes a
long standing bug (read "specification violation") where the server
would send an invalid response when the client has cancelled an
in-flight request. This was causing annoying spurious EINTR returns
in linux. The fix comes with some related testing in QTEST.

Other patches are code cleanup and improvements.

# gpg: Signature made Fri 02 Feb 2018 10:16:03 GMT
# gpg:                using RSA key 71D4D5E5822F73D6
# gpg: Good signature from "Greg Kurz <groug@kaod.org>"
# gpg:                 aka "Gregory Kurz <gregory.kurz@free.fr>"
# gpg:                 aka "[jpeg image of size 3330]"
# Primary key fingerprint: B482 8BAF 9431 40CE F2A3  4910 71D4 D5E5 822F 73D6

* remotes/gkurz/tags/for-upstream:
  tests/virtio-9p: explicitly handle potential integer overflows
  tests: virtio-9p: add FLUSH operation test
  libqos/virtio: return length written into used descriptor
  tests: virtio-9p: add WRITE operation test
  tests: virtio-9p: add LOPEN operation test
  tests: virtio-9p: use the synth backend
  tests: virtio-9p: wait for completion in the test code
  tests: virtio-9p: move request tag to the test functions
  9pfs: Correctly handle cancelled requests
  9pfs: drop v9fs_register_transport()
Signed-off-by: NPeter Maydell <peter.maydell@linaro.org>

audio: two small fixes.

# gpg: Signature made Fri 02 Feb 2018 07:49:20 GMT
# gpg:                using RSA key 4CB6D8EED3E87138
# gpg: Good signature from "Gerd Hoffmann (work) <kraxel@redhat.com>"
# gpg:                 aka "Gerd Hoffmann <gerd@kraxel.org>"
# gpg:                 aka "Gerd Hoffmann (private) <kraxel@gmail.com>"
# Primary key fingerprint: A032 8CFF B93A 17A7 9901  FE7D 4CB6 D8EE D3E8 7138

* remotes/kraxel/tags/audio-20180202-pull-request:
  hw/audio/sb16.c: change dolog() to qemu_log_mask()
  hw/audio/wm8750: move WM8750 declarations from i2c/i2c.h to audio/wm8750.h
Signed-off-by: NPeter Maydell <peter.maydell@linaro.org>

Lots of litte miscellaneous fixes for the IPMI code, plus
add me as the IPMI maintainer.

# gpg: Signature made Thu 01 Feb 2018 18:44:55 GMT
# gpg:                using RSA key 61F38C90919BFF81
# gpg: Good signature from "Corey Minyard <cminyard@mvista.com>"
# gpg:                 aka "Corey Minyard <minyard@acm.org>"
# gpg:                 aka "Corey Minyard <corey@minyard.net>"
# gpg:                 aka "Corey Minyard <minyard@mvista.com>"
# gpg: WARNING: This key is not certified with a trusted signature!
# gpg:          There is no indication that the signature belongs to the owner.
# Primary key fingerprint: FD0D 5CE6 7CE0 F59A 6688  2686 61F3 8C90 919B FF81

* remotes/cminyard/tags/for-release-20180201:
  ipmi: Allow BMC device properties to be set
  ipmi: disable IRQ and ATN on an external disconnect
  ipmi: Fix macro issues
  ipmi: Add the platform event message command
  ipmi: Don't set the timestamp on add events that don't have it
  ipmi: Fix SEL get/set time commands
  Add maintainer for the IPMI code
Signed-off-by: NPeter Maydell <peter.maydell@linaro.org>

# gpg: Signature made Thu 01 Feb 2018 11:15:42 GMT
# gpg:                using RSA key DAE8E10975969CE5
# gpg: Good signature from "Marc-André Lureau <marcandre.lureau@redhat.com>"
# gpg:                 aka "Marc-André Lureau <marcandre.lureau@gmail.com>"
# Primary key fingerprint: 87A9 BD93 3F87 C606 D276  F62D DAE8 E109 7596 9CE5

* remotes/elmarco/tags/dump-pull-request:
  dump-guest-memory.py: skip vmcoreinfo section if not available
Signed-off-by: NPeter Maydell <peter.maydell@linaro.org>

Signed-off-by: NGreg Kurz <groug@kaod.org>
Reviewed-by: NEric Blake <eblake@redhat.com>
Reviewed-by: NStefan Hajnoczi <stefanha@redhat.com>

The idea is to send a victim request that will possibly block in the
server and to send a flush request to cancel the victim request.

This patch adds two test to verifiy that:
- the server does not reply to a victim request that was actually
  cancelled
- the server replies to the flush request after replying to the
  victim request if it could not cancel it

9p request cancellation reference:

http://man.cat-v.org/plan_9/5/flushSigned-off-by: NGreg Kurz <groug@kaod.org>
Reviewed-by: NStefan Hajnoczi <stefanha@redhat.com>
(groug, change the test to only write a single byte to avoid
        any alignment or endianess consideration)

When a 9p request is flushed (ie, cancelled) by the guest, the device
is expected to simply mark the request as used, without sending a 9p
reply (ie, without writing anything into the used buffer).

To be able to test this, we need access to the length written by the
device into the used descriptor. This patch adds a uint32_t * argument
to qvirtqueue_get_buf() and qvirtio_wait_used_elem() for this purpose.

All existing users are updated accordingly.
Signed-off-by: NGreg Kurz <groug@kaod.org>
Reviewed-by: NStefan Hajnoczi <stefanha@redhat.com>