Commit fb0bc835 moved code from scripts/qapi-*.py to
scripts/qapi/*.py.  It neglected to update MAINTAINERS: scripts/qapi*
matches only the former, not the latter.  Do that now.
Signed-off-by: NMarkus Armbruster <armbru@redhat.com>
Message-Id: <20180620124827.17106-1-armbru@redhat.com>
Reviewed-by: NEric Blake <eblake@redhat.com>

Commit 1a9a507b "qapi-introspect: Hide type names" added local
variable @jsons to improve sorting, but also removed the sorting.  It
was part of a big series that went to v8, and it made sense until v2
or so...

Commit 7d0f982b replaced @jsons by @qlits, preserving the
uselessness.

Get rid of it.
Signed-off-by: NMarkus Armbruster <armbru@redhat.com>
Message-Id: <20180620124742.16979-1-armbru@redhat.com>
Reviewed-by: NMarc-André Lureau <marcandre.lureau@redhat.com>

This commit removes the PYTHON_UTF8 workaround. The problem with setting

  LC_ALL= LANG=C LC_CTYPE=en_US.UTF-8

is that the en_US.UTF-8 locale might not be available. In this case
setting above locales results in build errors even though another UTF-8
locale was originally set [1]. The only stable way of fixing the
encoding problem is by specifying the encoding in Python, like the
previous commit does.

[1] https://bugs.gentoo.org/657766Signed-off-by: NArfrever Frehtes Taifersar Arahesis <arfrever.fta@gmail.com>
Signed-off-by: NMatthias Maier <tamiko@43-1.org>
Message-Id: <20180618175958.29073-3-armbru@redhat.com>
Reviewed-by: NEduardo Habkost <ehabkost@redhat.com>
Reviewed-by: NEric Blake <eblake@redhat.com>
[Commit message tweaked]
Signed-off-by: NMarkus Armbruster <armbru@redhat.com>

Python 2 happily reads UTF-8 files in text mode, but Python 3 requires
either UTF-8 locale or an explicit encoding passed to open().  Commit
d4e5ec87 fixed this by setting the en_US.UTF-8 locale.  Falls apart
when the locale isn't be available.

Matthias Maier and Arfrever Frehtes Taifersar Arahesis proposed to use
binary mode instead, with manual conversion from bytes to str.  Works,
but opening with an explicit encoding is simpler, so do that.

Since Python 2's open() doesn't support the encoding parameter, we
need to suppress it with a version check.
Reported-by: NArfrever Frehtes Taifersar Arahesis <arfrever.fta@gmail.com>
Reported-by: NMatthias Maier <tamiko@43-1.org>
Signed-off-by: NMarkus Armbruster <armbru@redhat.com>
Message-Id: <20180618175958.29073-2-armbru@redhat.com>
Reviewed-by: NEduardo Habkost <ehabkost@redhat.com>
Reviewed-by: NEric Blake <eblake@redhat.com>

Flat unions may now have uncovered branches, so it is possible to get rid
of empty types defined for that purpose only.
Signed-off-by: NAnton Nefedov <anton.nefedov@virtuozzo.com>
Reviewed-by: NMarkus Armbruster <armbru@redhat.com>
Message-Id: <1529311206-76847-3-git-send-email-anton.nefedov@virtuozzo.com>
Signed-off-by: NMarkus Armbruster <armbru@redhat.com>

It often happens that just a few discriminator values imply extra data in
a flat union. Existing checks did not make possible to leave other values
uncovered. Such cases had to be worked around by either stating a dummy
(empty) type or introducing another (subset) discriminator enumeration.

Both options create redundant entities in qapi files for little profit.

With this patch it is not necessary anymore to add designated union
fields for every possible value of a discriminator enumeration.
Signed-off-by: NAnton Nefedov <anton.nefedov@virtuozzo.com>
Message-Id: <1529311206-76847-2-git-send-email-anton.nefedov@virtuozzo.com>
Reviewed-by: NMarkus Armbruster <armbru@redhat.com>
Signed-off-by: NMarkus Armbruster <armbru@redhat.com>

This new test verifies that qdict_flatten() does not modify a shallow
clone of the given QDict.
Reviewed-by: NEric Blake <eblake@redhat.com>
Signed-off-by: NMax Reitz <mreitz@redhat.com>
Message-Id: <20180611205203.2624-8-mreitz@redhat.com>
Reviewed-by: NMarkus Armbruster <armbru@redhat.com>
Signed-off-by: NMarkus Armbruster <armbru@redhat.com>

In its current form, qdict_flatten() removes all entries from nested
QDicts that are moved to the root QDict.  It is completely sufficient to
remove all old entries from the root QDict, however.  If the nested
dicts have a refcount of 1, this will automatically delete them, too.
And if they have a greater refcount, we probably do not want to modify
them in the first place.

The latter observation means that it was currently (in general)
impossible to qdict_flatten() a shallowly cloned dict because that would
empty nested QDicts in the original dict as well.  This patch changes
this, so you can now use qdict_flatten(qdict_shallow_clone(dict)) to get
a flattened copy without disturbing the original.
Signed-off-by: NMax Reitz <mreitz@redhat.com>
Message-Id: <20180611205203.2624-7-mreitz@redhat.com>
Reviewed-by: NMarkus Armbruster <armbru@redhat.com>
Signed-off-by: NMarkus Armbruster <armbru@redhat.com>

The event generator produces an enum, and put it in the last visited
module. It fits better in the main module, since it's the set of all
visited events, from all modules.
Signed-off-by: NMarc-André Lureau <marcandre.lureau@redhat.com>
Message-Id: <20180321115211.17937-3-marcandre.lureau@redhat.com>
Reviewed-by: NMarkus Armbruster <armbru@redhat.com>
Signed-off-by: NMarkus Armbruster <armbru@redhat.com>

Signed-off-by: NMarc-André Lureau <marcandre.lureau@redhat.com>
Message-Id: <20180321115211.17937-2-marcandre.lureau@redhat.com>
Reviewed-by: NMarkus Armbruster <armbru@redhat.com>
Signed-off-by: NMarkus Armbruster <armbru@redhat.com>

The code has a while() loop that always initialized 'end', and
the loop always executes at least once (as evidenced by the assert()
just prior to the loop).  But some versions of gcc still complain
that 'end' is used uninitialized, so silence them.
Signed-off-by: NEric Blake <eblake@redhat.com>
Reviewed-by: NPeter Maydell <peter.maydell@linaro.org>
Message-id: 20180622125814.345274-1-eblake@redhat.com
Signed-off-by: NPeter Maydell <peter.maydell@linaro.org>

HMP pull 2018-06-21

Minor fixes and reenable preconfig

# gpg: Signature made Thu 21 Jun 2018 17:43:09 BST
# gpg:                using RSA key 0516331EBC5BFDE7
# gpg: Good signature from "Dr. David Alan Gilbert (RH2) <dgilbert@redhat.com>"
# Primary key fingerprint: 45F5 C71B 4A0C B7FB 977A  9FA9 0516 331E BC5B FDE7

* remotes/dgilbert/tags/pull-hmp-20180621:
  hmp: Allow HMP in preconfig state again
  hmp: add exit_preconfig
  hmp: Add commands for preconfig
  qmp: Enable a few commands in preconfig state
  hmp: Restrict auto-complete in preconfig
  hmp: Allow help on preconfig commands
  hmp: Add flag for preconfig commands
  hmp-commands: use long for begin and length in dump-guest-memory
  monitor: report entirety of hmp command on error
Signed-off-by: NPeter Maydell <peter.maydell@linaro.org>

Add check-tcg machinary

This restores the ability to run TCG smoke tests by using our docker
infrastructure to support cross building simple tests. It represents
the first step to making better cross-architecture testing available
straight from the source tree ;-)

v2
  - fix quoting of target_compiler
  - make docker.py Py3 safe
  - tweak .travis.yml recipe
  - don't probe docker when HAVE_USER_DOCKER not set

# gpg: Signature made Thu 21 Jun 2018 07:23:45 BST
# gpg:                using RSA key FBD0DB095A9E2A44
# gpg: Good signature from "Alex Bennée (Master Work Key) <alex.bennee@linaro.org>"
# Primary key fingerprint: 6685 AE99 E751 67BC AFC8  DF35 FBD0 DB09 5A9E 2A44

* remotes/stsquad/tags/pull-tcg-testing-revivial-210618-2: (57 commits)
  .travis.yml: add check-tcg test
  tests/docker/Makefile.include: only force SID to NOCACHE if old
  docker: docker.py adding age check command
  tests/Makefile: call sub-makes with SKIP_DOCKER_BUILD=1
  docker: docker.py add check sub-command
  docker: docker.py don't conflate checksums for extra_files
  docker: docker.py use "version" to probe usage
  tests: add top-level make dependency for docker builds
  tests/tcg/i386: extend timeout for runcom test
  tests/tcg: override runners for broken tests
  tests/tcg: add run, diff, and skip helper macros
  tests/Makefile.include: add [build|clean|check]-tcg targets
  Makefile.target: add (clean-/build-)guest-tests targets
  tests/tcg/Makefile: update to be called from Makefile.target
  tests/tcg: enable building for PowerPC
  docker: move debian-powerpc-cross to sid based build
  tests/tcg: enable building for RISCV64
  tests/tcg: enable building for mips64
  tests/tcg: enable building for sparc64
  tests/tcg: enable building for sh4
  ...
Signed-off-by: NPeter Maydell <peter.maydell@linaro.org>

nbd patches for 2018-06-20

Add experimental x-nbd-server-add-bitmap to expose a disabled
bitmap over NBD, in preparation for a pull model incremental
backup scheme. Also fix a corner case protocol issue with
NBD_CMD_BLOCK_STATUS, and add new NBD_CMD_CACHE.

- Eric Blake: tests: Simplify .gitignore
- Eric Blake: nbd/server: Reject 0-length block status request
- Vladimir Sementsov-Ogievskiy: 0/6 NBD export bitmaps
- Vladimir Sementsov-Ogievskiy: nbd/server: introduce NBD_CMD_CACHE

# gpg: Signature made Thu 21 Jun 2018 15:53:55 BST
# gpg:                using RSA key A7A16B4A2527436A
# gpg: Good signature from "Eric Blake <eblake@redhat.com>"
# gpg:                 aka "Eric Blake (Free Software Programmer) <ebb9@byu.net>"
# gpg:                 aka "[jpeg image of size 6874]"
# Primary key fingerprint: 71C2 CC22 B1C4 6029 27D2  F3AA A7A1 6B4A 2527 436A

* remotes/ericb/tags/pull-nbd-2018-06-20-v2:
  nbd/server: introduce NBD_CMD_CACHE
  docs/interop: add nbd.txt
  qapi: new qmp command nbd-server-add-bitmap
  nbd/server: implement dirty bitmap export
  nbd/server: add nbd_meta_empty_or_pattern helper
  nbd/server: refactor NBDExportMetaContexts
  nbd/server: fix trace
  nbd/server: Reject 0-length block status request
  tests: Simplify .gitignore
Signed-off-by: NPeter Maydell <peter.maydell@linaro.org>

TCG patch queue:

Workaround macos assembler lossage.
Eliminate tb_lock.
Fix TB code generation overflow.

# gpg: Signature made Fri 15 Jun 2018 20:40:56 BST
# gpg:                using RSA key 64DF38E8AF7E215F
# gpg: Good signature from "Richard Henderson <richard.henderson@linaro.org>"
# Primary key fingerprint: 7A48 1E78 868B 4DB6 A85A  05C0 64DF 38E8 AF7E 215F

* remotes/rth/tags/pull-tcg-20180615:
  tcg: Reduce max TB opcode count
  tcg: remove tb_lock
  translate-all: remove tb_lock mention from cpu_restore_state_from_tb
  cputlb: remove tb_lock from tlb_flush functions
  translate-all: protect TB jumps with a per-destination-TB lock
  translate-all: discard TB when tb_link_page returns an existing matching TB
  translate-all: introduce assert_no_pages_locked
  translate-all: add page_locked assertions
  translate-all: use per-page locking in !user-mode
  translate-all: move tb_invalidate_phys_page_range up in the file
  translate-all: work page-by-page in tb_invalidate_phys_range_1
  translate-all: remove hole in PageDesc
  translate-all: make l1_map lockless
  translate-all: iterate over TBs in a page with PAGE_FOR_EACH_TB
  tcg: move tb_ctx.tb_phys_invalidate_count to tcg_ctx
  tcg: track TBs with per-region BST's
  qht: return existing entry when qht_insert fails
  qht: require a default comparison function
  tcg/i386: Use byte form of xgetbv instruction
Signed-off-by: NPeter Maydell <peter.maydell@linaro.org>

Handle nbd CACHE command. Just do read, without sending read data back.
Cache mechanism should be done by exported node driver chain.
Signed-off-by: NVladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
Message-Id: <20180413143156.11409-1-vsementsov@virtuozzo.com>
Reviewed-by: NEric Blake <eblake@redhat.com>
[eblake: fix two missing case labels in switch statements]
Signed-off-by: NEric Blake <eblake@redhat.com>

Describe new metadata namespace: "qemu".
Signed-off-by: NVladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
Message-Id: <20180609151758.17343-7-vsementsov@virtuozzo.com>
Reviewed-by: NEric Blake <eblake@redhat.com>
[eblake: grammar tweaks]
Signed-off-by: NEric Blake <eblake@redhat.com>

For now, the actual command ix x-nbd-server-add-bitmap, reflecting
the fact that we are still working on libvirt code that proves the
command works as needed, and also the fact that we may remove
bitmap-export-name (and just require that the exported name be the
bitmap name).
Signed-off-by: NVladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
Message-Id: <20180609151758.17343-6-vsementsov@virtuozzo.com>
Reviewed-by: NEric Blake <eblake@redhat.com>
[eblake: make the command experimental by adding x- prefix]
Signed-off-by: NEric Blake <eblake@redhat.com>

Handle a new NBD meta namespace: "qemu", and corresponding queries:
"qemu:dirty-bitmap:<export bitmap name>".

With the new metadata context negotiated, BLOCK_STATUS query will reply
with dirty-bitmap data, converted to extents. The new public function
nbd_export_bitmap selects which bitmap to export. For now, only one bitmap
may be exported.
Signed-off-by: NVladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
Message-Id: <20180609151758.17343-5-vsementsov@virtuozzo.com>
Reviewed-by: NEric Blake <eblake@redhat.com>
[eblake: wording tweaks, minor cleanups, additional tracing]
Signed-off-by: NEric Blake <eblake@redhat.com>

Add nbd_meta_pattern() and nbd_meta_empty_or_pattern() helpers for
metadata query parsing. nbd_meta_pattern() will be reused for the
"qemu" namespace in following patches.
Signed-off-by: NVladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
Message-Id: <20180609151758.17343-4-vsementsov@virtuozzo.com>
Reviewed-by: NEric Blake <eblake@redhat.com>
[eblake: comment tweaks]
Signed-off-by: NEric Blake <eblake@redhat.com>

Use NBDExport pointer instead of just export name: there is no need to
store a duplicated name in the struct; moreover, NBDExport will be used
further.
Signed-off-by: NVladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
Message-Id: <20180609151758.17343-3-vsementsov@virtuozzo.com>
Reviewed-by: NEric Blake <eblake@redhat.com>
[eblake: commit message grammar tweak]
Signed-off-by: NEric Blake <eblake@redhat.com>

Return code = 1 doesn't mean that we parsed base:allocation. Use
correct traces in both -parsed and -skipped cases.
Signed-off-by: NVladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
Message-Id: <20180609151758.17343-2-vsementsov@virtuozzo.com>
Reviewed-by: NEric Blake <eblake@redhat.com>
[eblake: comment tweaks]
Signed-off-by: NEric Blake <eblake@redhat.com>

The NBD spec says that behavior is unspecified if the client
requests 0 length for block status; but since the structured
reply is documenting as returning a non-zero length, it's
easier to just diagnose this with an EINVAL error than to
figure out what to return.

CC: qemu-stable@nongnu.org
Signed-off-by: NEric Blake <eblake@redhat.com>
Message-Id: <20180621124937.166549-1-eblake@redhat.com>
Reviewed-by: NVladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>

Commit 0bcc8e5b was yet another instance of 'git status' reporting
dirty files after an in-tree build, thanks to the new binary
tests/check-block-qdict.

Instead of piecemeal exemptions of each new binary as they are
added, let's use git's negative globbing feature to exempt ALL
files that have a 'test-' or 'check-' prefix, except for the ones
ending in '.c' or '.sh'.  We still have a couple of generated
files that then need (re-)exclusion, but the overall list is a
LOT shorter, and less prone to needing future edits.
Signed-off-by: NEric Blake <eblake@redhat.com>
Message-Id: <20180619203918.65450-1-eblake@redhat.com>
Reviewed-by: NPhilippe Mathieu-Daudé <f4bug@amsat.org>

Now we can cope with preconfig in HMP, reenable by reverting
commit 71dc578e.
Signed-off-by: NDr. David Alan Gilbert <dgilbert@redhat.com>
Reviewed-by: NPeter Xu <peterx@redhat.com>
Reviewed-by: NIgor Mammedov <imammedo@redhat.com>
Message-Id: <20180620153947.30834-8-dgilbert@redhat.com>
Signed-off-by: NDr. David Alan Gilbert <dgilbert@redhat.com>

Add the exit_preconfig command to return to normality.
Signed-off-by: NDr. David Alan Gilbert <dgilbert@redhat.com>
Reviewed-by: NPeter Xu <peterx@redhat.com>
Reviewed-by: NIgor Mammedov <imammedo@redhat.com>
Message-Id: <20180620153947.30834-7-dgilbert@redhat.com>
Signed-off-by: NDr. David Alan Gilbert <dgilbert@redhat.com>

Allow a bunch of the info commands to be used in preconfig.

version, chardev, name, uuid,memdev, iothreads
  Were enabled in QMP in the previous patch from Igor

status, hotpluggable_cpus
  Was enabled in the original allow-preconfig series

history
  is HMP specific

qom-tree
  Don't have a QMP equivalent

Also enable the qom commands qom-list and qom-set.
Signed-off-by: NDr. David Alan Gilbert <dgilbert@redhat.com>
Message-Id: <20180620153947.30834-6-dgilbert@redhat.com>
Reviewed-by: NIgor Mammedov <imammedo@redhat.com>
Signed-off-by: NDr. David Alan Gilbert <dgilbert@redhat.com>
  Dropped info numa as per Igor's 2018-06-21 review

Commands query-chardev, query-version, query-name, query-uuid,
query-iothreads, query-memdev are informational and do not depend on
the machine being initialized.  Make them available in preconfig
runstate to make the latter a little bit more useful.
The generic qom commands don't depend on the machine being initialized
either; so enabled qom-list, qom-get, qom-set, qom-list-types,
qom-list-properties.
Signed-off-by: NIgor Mammedov <imammedo@redhat.com>
Signed-off-by: NDr. David Alan Gilbert <dgilbert@redhat.com>
Reviewed-by: NEric Blake <eblake@redhat.com>
Message-Id: <20180620153947.30834-5-dgilbert@redhat.com>
Signed-off-by: NDr. David Alan Gilbert <dgilbert@redhat.com>

Don't show the commands that aren't available.
Signed-off-by: NDr. David Alan Gilbert <dgilbert@redhat.com>
Reviewed-by: NPeter Xu <peterx@redhat.com>
Reviewed-by: NIgor Mammedov <imammedo@redhat.com>
Message-Id: <20180620153947.30834-4-dgilbert@redhat.com>
Signed-off-by: NDr. David Alan Gilbert <dgilbert@redhat.com>

Allow the 'help' command in preconfig state but
make it only list the preconfig commands.
Signed-off-by: NDr. David Alan Gilbert <dgilbert@redhat.com>
Reviewed-by: NPeter Xu <peterx@redhat.com>
Reviewed-by: NIgor Mammedov <imammedo@redhat.com>
Message-Id: <20180620153947.30834-3-dgilbert@redhat.com>
Signed-off-by: NDr. David Alan Gilbert <dgilbert@redhat.com>

Add a flag to command definitions to allow them to be used in preconfig
and check it.
If users try to use commands that aren't available, tell them to use
the exit_preconfig comand we're adding in a few patches.
Signed-off-by: NDr. David Alan Gilbert <dgilbert@redhat.com>
Reviewed-by: NMarkus Armbruster <armbru@redhat.com>
Reviewed-by: NIgor Mammedov <imammedo@redhat.com>
Message-Id: <20180620153947.30834-2-dgilbert@redhat.com>
Signed-off-by: NDr. David Alan Gilbert <dgilbert@redhat.com>

The dump-guest-memory command is used to dump an area of guest memory
to a file, the piece of memory is specified by a begin address and
a length. These parameters are specified as ints and thus have a maximum
value of 4GB. This means you can't dump the guest memory past the first
4GB and instead get:
(qemu) dump-guest-memory tmp 0x100000000 0x100000000
'dump-guest-memory' has failed: integer is for 32-bit values
Try "help dump-guest-memory" for more information

This limitation is imposed in monitor_parse_arguments() since they are
both ints. hmp_dump_guest_memory() uses 64 bit quantities to store both
the begin and length values. Thus specify begin and length as long so
that the entire guest memory space can be dumped.
Signed-off-by: NSuraj Jitindar Singh <sjitindarsingh@gmail.com>
Message-Id: <20180620003202.10546-1-sjitindarsingh@gmail.com>
Reviewed-by: NDr. David Alan Gilbert <dgilbert@redhat.com>
Signed-off-by: NDr. David Alan Gilbert <dgilbert@redhat.com>

When a user incorrectly provides an hmp command, an error response will be
printed that prompts the user to try "help <command name>". However, when
the command contains multiple parts e.g. "info uuid xyz", only the last
whitespace delimited string will be reported (in this example "info" will
be dropped and the message will read "Try "help uuid" for more information",
which is incorrect).

Let's correct this by capturing the entirety of the command from the command
line -- excluding any extraneous characters.
Reported-by: NMikhail Fokin <fokin@de.ibm.com>
Signed-off-by: NCollin Walling <walling@linux.ibm.com>
Message-Id: <ee680f5e-ac9a-479d-f65e-9f8ae9cfe5d4@linux.ibm.com>
Reviewed-by: NEric Blake <eblake@redhat.com>
Signed-off-by: NDr. David Alan Gilbert <dgilbert@redhat.com>

Signed-off-by: NAlex Bennée <alex.bennee@linaro.org>
Reviewed-by: NPhilippe Mathieu-Daudé <f4bug@amsat.org>
Tested-by: NPhilippe Mathieu-Daudé <f4bug@amsat.org>

Now we can check the age of a docker image we can be a little more
intelligent about re-building Sid images and only force NOCACHE if
it is "old".
Signed-off-by: NAlex Bennée <alex.bennee@linaro.org>

This is useful for querying if an image is too old.
Signed-off-by: NAlex Bennée <alex.bennee@linaro.org>
Reviewed-by: NPhilippe Mathieu-Daudé <f4bug@amsat.org>

As we now ensure all the images we are going to use are built in the
top level make file lets not over complicate things by running the
full script again. We do run the check script just in case someone
deletes the docker image while we are running.
Signed-off-by: NAlex Bennée <alex.bennee@linaro.org>

This command allows you to check if we need to re-build a docker
image. If the image isn't in the repository or the checksums don't
match then we return false and some text (for processing in
makefiles).
Signed-off-by: NAlex Bennée <alex.bennee@linaro.org>

This just gets confusing especially as the helper function doesn't
even take into account any extra files (or the executable). Currently
the actual check just ignores them and also passes the result through
_dockerfile_preprocess so we fix that too.
Signed-off-by: NAlex Bennée <alex.bennee@linaro.org>
Reviewed-by: NPhilippe Mathieu-Daudé <f4bug@amsat.org>

The "images" command is a fairly heavyweight command to run as it
involves searching the whole docker file-system inventory. On a
machine with a lot of images this makes start-up fairly expensive.
Signed-off-by: NAlex Bennée <alex.bennee@linaro.org>