In its current form, qdict_flatten() removes all entries from nested
QDicts that are moved to the root QDict.  It is completely sufficient to
remove all old entries from the root QDict, however.  If the nested
dicts have a refcount of 1, this will automatically delete them, too.
And if they have a greater refcount, we probably do not want to modify
them in the first place.

The latter observation means that it was currently (in general)
impossible to qdict_flatten() a shallowly cloned dict because that would
empty nested QDicts in the original dict as well.  This patch changes
this, so you can now use qdict_flatten(qdict_shallow_clone(dict)) to get
a flattened copy without disturbing the original.
Signed-off-by: NMax Reitz <mreitz@redhat.com>
Message-Id: <20180611205203.2624-7-mreitz@redhat.com>
Reviewed-by: NMarkus Armbruster <armbru@redhat.com>
Signed-off-by: NMarkus Armbruster <armbru@redhat.com>

The event generator produces an enum, and put it in the last visited
module. It fits better in the main module, since it's the set of all
visited events, from all modules.
Signed-off-by: NMarc-André Lureau <marcandre.lureau@redhat.com>
Message-Id: <20180321115211.17937-3-marcandre.lureau@redhat.com>
Reviewed-by: NMarkus Armbruster <armbru@redhat.com>
Signed-off-by: NMarkus Armbruster <armbru@redhat.com>

Signed-off-by: NMarc-André Lureau <marcandre.lureau@redhat.com>
Message-Id: <20180321115211.17937-2-marcandre.lureau@redhat.com>
Reviewed-by: NMarkus Armbruster <armbru@redhat.com>
Signed-off-by: NMarkus Armbruster <armbru@redhat.com>

The code has a while() loop that always initialized 'end', and
the loop always executes at least once (as evidenced by the assert()
just prior to the loop).  But some versions of gcc still complain
that 'end' is used uninitialized, so silence them.
Signed-off-by: NEric Blake <eblake@redhat.com>
Reviewed-by: NPeter Maydell <peter.maydell@linaro.org>
Message-id: 20180622125814.345274-1-eblake@redhat.com
Signed-off-by: NPeter Maydell <peter.maydell@linaro.org>

HMP pull 2018-06-21

Minor fixes and reenable preconfig

# gpg: Signature made Thu 21 Jun 2018 17:43:09 BST
# gpg:                using RSA key 0516331EBC5BFDE7
# gpg: Good signature from "Dr. David Alan Gilbert (RH2) <dgilbert@redhat.com>"
# Primary key fingerprint: 45F5 C71B 4A0C B7FB 977A  9FA9 0516 331E BC5B FDE7

* remotes/dgilbert/tags/pull-hmp-20180621:
  hmp: Allow HMP in preconfig state again
  hmp: add exit_preconfig
  hmp: Add commands for preconfig
  qmp: Enable a few commands in preconfig state
  hmp: Restrict auto-complete in preconfig
  hmp: Allow help on preconfig commands
  hmp: Add flag for preconfig commands
  hmp-commands: use long for begin and length in dump-guest-memory
  monitor: report entirety of hmp command on error
Signed-off-by: NPeter Maydell <peter.maydell@linaro.org>

Add check-tcg machinary

This restores the ability to run TCG smoke tests by using our docker
infrastructure to support cross building simple tests. It represents
the first step to making better cross-architecture testing available
straight from the source tree ;-)

v2
  - fix quoting of target_compiler
  - make docker.py Py3 safe
  - tweak .travis.yml recipe
  - don't probe docker when HAVE_USER_DOCKER not set

# gpg: Signature made Thu 21 Jun 2018 07:23:45 BST
# gpg:                using RSA key FBD0DB095A9E2A44
# gpg: Good signature from "Alex Bennée (Master Work Key) <alex.bennee@linaro.org>"
# Primary key fingerprint: 6685 AE99 E751 67BC AFC8  DF35 FBD0 DB09 5A9E 2A44

* remotes/stsquad/tags/pull-tcg-testing-revivial-210618-2: (57 commits)
  .travis.yml: add check-tcg test
  tests/docker/Makefile.include: only force SID to NOCACHE if old
  docker: docker.py adding age check command
  tests/Makefile: call sub-makes with SKIP_DOCKER_BUILD=1
  docker: docker.py add check sub-command
  docker: docker.py don't conflate checksums for extra_files
  docker: docker.py use "version" to probe usage
  tests: add top-level make dependency for docker builds
  tests/tcg/i386: extend timeout for runcom test
  tests/tcg: override runners for broken tests
  tests/tcg: add run, diff, and skip helper macros
  tests/Makefile.include: add [build|clean|check]-tcg targets
  Makefile.target: add (clean-/build-)guest-tests targets
  tests/tcg/Makefile: update to be called from Makefile.target
  tests/tcg: enable building for PowerPC
  docker: move debian-powerpc-cross to sid based build
  tests/tcg: enable building for RISCV64
  tests/tcg: enable building for mips64
  tests/tcg: enable building for sparc64
  tests/tcg: enable building for sh4
  ...
Signed-off-by: NPeter Maydell <peter.maydell@linaro.org>

nbd patches for 2018-06-20

Add experimental x-nbd-server-add-bitmap to expose a disabled
bitmap over NBD, in preparation for a pull model incremental
backup scheme. Also fix a corner case protocol issue with
NBD_CMD_BLOCK_STATUS, and add new NBD_CMD_CACHE.

- Eric Blake: tests: Simplify .gitignore
- Eric Blake: nbd/server: Reject 0-length block status request
- Vladimir Sementsov-Ogievskiy: 0/6 NBD export bitmaps
- Vladimir Sementsov-Ogievskiy: nbd/server: introduce NBD_CMD_CACHE

# gpg: Signature made Thu 21 Jun 2018 15:53:55 BST
# gpg:                using RSA key A7A16B4A2527436A
# gpg: Good signature from "Eric Blake <eblake@redhat.com>"
# gpg:                 aka "Eric Blake (Free Software Programmer) <ebb9@byu.net>"
# gpg:                 aka "[jpeg image of size 6874]"
# Primary key fingerprint: 71C2 CC22 B1C4 6029 27D2  F3AA A7A1 6B4A 2527 436A

* remotes/ericb/tags/pull-nbd-2018-06-20-v2:
  nbd/server: introduce NBD_CMD_CACHE
  docs/interop: add nbd.txt
  qapi: new qmp command nbd-server-add-bitmap
  nbd/server: implement dirty bitmap export
  nbd/server: add nbd_meta_empty_or_pattern helper
  nbd/server: refactor NBDExportMetaContexts
  nbd/server: fix trace
  nbd/server: Reject 0-length block status request
  tests: Simplify .gitignore
Signed-off-by: NPeter Maydell <peter.maydell@linaro.org>

TCG patch queue:

Workaround macos assembler lossage.
Eliminate tb_lock.
Fix TB code generation overflow.

# gpg: Signature made Fri 15 Jun 2018 20:40:56 BST
# gpg:                using RSA key 64DF38E8AF7E215F
# gpg: Good signature from "Richard Henderson <richard.henderson@linaro.org>"
# Primary key fingerprint: 7A48 1E78 868B 4DB6 A85A  05C0 64DF 38E8 AF7E 215F

* remotes/rth/tags/pull-tcg-20180615:
  tcg: Reduce max TB opcode count
  tcg: remove tb_lock
  translate-all: remove tb_lock mention from cpu_restore_state_from_tb
  cputlb: remove tb_lock from tlb_flush functions
  translate-all: protect TB jumps with a per-destination-TB lock
  translate-all: discard TB when tb_link_page returns an existing matching TB
  translate-all: introduce assert_no_pages_locked
  translate-all: add page_locked assertions
  translate-all: use per-page locking in !user-mode
  translate-all: move tb_invalidate_phys_page_range up in the file
  translate-all: work page-by-page in tb_invalidate_phys_range_1
  translate-all: remove hole in PageDesc
  translate-all: make l1_map lockless
  translate-all: iterate over TBs in a page with PAGE_FOR_EACH_TB
  tcg: move tb_ctx.tb_phys_invalidate_count to tcg_ctx
  tcg: track TBs with per-region BST's
  qht: return existing entry when qht_insert fails
  qht: require a default comparison function
  tcg/i386: Use byte form of xgetbv instruction
Signed-off-by: NPeter Maydell <peter.maydell@linaro.org>

Handle nbd CACHE command. Just do read, without sending read data back.
Cache mechanism should be done by exported node driver chain.
Signed-off-by: NVladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
Message-Id: <20180413143156.11409-1-vsementsov@virtuozzo.com>
Reviewed-by: NEric Blake <eblake@redhat.com>
[eblake: fix two missing case labels in switch statements]
Signed-off-by: NEric Blake <eblake@redhat.com>

Describe new metadata namespace: "qemu".
Signed-off-by: NVladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
Message-Id: <20180609151758.17343-7-vsementsov@virtuozzo.com>
Reviewed-by: NEric Blake <eblake@redhat.com>
[eblake: grammar tweaks]
Signed-off-by: NEric Blake <eblake@redhat.com>

For now, the actual command ix x-nbd-server-add-bitmap, reflecting
the fact that we are still working on libvirt code that proves the
command works as needed, and also the fact that we may remove
bitmap-export-name (and just require that the exported name be the
bitmap name).
Signed-off-by: NVladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
Message-Id: <20180609151758.17343-6-vsementsov@virtuozzo.com>
Reviewed-by: NEric Blake <eblake@redhat.com>
[eblake: make the command experimental by adding x- prefix]
Signed-off-by: NEric Blake <eblake@redhat.com>

Handle a new NBD meta namespace: "qemu", and corresponding queries:
"qemu:dirty-bitmap:<export bitmap name>".

With the new metadata context negotiated, BLOCK_STATUS query will reply
with dirty-bitmap data, converted to extents. The new public function
nbd_export_bitmap selects which bitmap to export. For now, only one bitmap
may be exported.
Signed-off-by: NVladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
Message-Id: <20180609151758.17343-5-vsementsov@virtuozzo.com>
Reviewed-by: NEric Blake <eblake@redhat.com>
[eblake: wording tweaks, minor cleanups, additional tracing]
Signed-off-by: NEric Blake <eblake@redhat.com>

Add nbd_meta_pattern() and nbd_meta_empty_or_pattern() helpers for
metadata query parsing. nbd_meta_pattern() will be reused for the
"qemu" namespace in following patches.
Signed-off-by: NVladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
Message-Id: <20180609151758.17343-4-vsementsov@virtuozzo.com>
Reviewed-by: NEric Blake <eblake@redhat.com>
[eblake: comment tweaks]
Signed-off-by: NEric Blake <eblake@redhat.com>

Use NBDExport pointer instead of just export name: there is no need to
store a duplicated name in the struct; moreover, NBDExport will be used
further.
Signed-off-by: NVladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
Message-Id: <20180609151758.17343-3-vsementsov@virtuozzo.com>
Reviewed-by: NEric Blake <eblake@redhat.com>
[eblake: commit message grammar tweak]
Signed-off-by: NEric Blake <eblake@redhat.com>

Return code = 1 doesn't mean that we parsed base:allocation. Use
correct traces in both -parsed and -skipped cases.
Signed-off-by: NVladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
Message-Id: <20180609151758.17343-2-vsementsov@virtuozzo.com>
Reviewed-by: NEric Blake <eblake@redhat.com>
[eblake: comment tweaks]
Signed-off-by: NEric Blake <eblake@redhat.com>

The NBD spec says that behavior is unspecified if the client
requests 0 length for block status; but since the structured
reply is documenting as returning a non-zero length, it's
easier to just diagnose this with an EINVAL error than to
figure out what to return.

CC: qemu-stable@nongnu.org
Signed-off-by: NEric Blake <eblake@redhat.com>
Message-Id: <20180621124937.166549-1-eblake@redhat.com>
Reviewed-by: NVladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>

Commit 0bcc8e5b was yet another instance of 'git status' reporting
dirty files after an in-tree build, thanks to the new binary
tests/check-block-qdict.

Instead of piecemeal exemptions of each new binary as they are
added, let's use git's negative globbing feature to exempt ALL
files that have a 'test-' or 'check-' prefix, except for the ones
ending in '.c' or '.sh'.  We still have a couple of generated
files that then need (re-)exclusion, but the overall list is a
LOT shorter, and less prone to needing future edits.
Signed-off-by: NEric Blake <eblake@redhat.com>
Message-Id: <20180619203918.65450-1-eblake@redhat.com>
Reviewed-by: NPhilippe Mathieu-Daudé <f4bug@amsat.org>

Now we can cope with preconfig in HMP, reenable by reverting
commit 71dc578e.
Signed-off-by: NDr. David Alan Gilbert <dgilbert@redhat.com>
Reviewed-by: NPeter Xu <peterx@redhat.com>
Reviewed-by: NIgor Mammedov <imammedo@redhat.com>
Message-Id: <20180620153947.30834-8-dgilbert@redhat.com>
Signed-off-by: NDr. David Alan Gilbert <dgilbert@redhat.com>

Add the exit_preconfig command to return to normality.
Signed-off-by: NDr. David Alan Gilbert <dgilbert@redhat.com>
Reviewed-by: NPeter Xu <peterx@redhat.com>
Reviewed-by: NIgor Mammedov <imammedo@redhat.com>
Message-Id: <20180620153947.30834-7-dgilbert@redhat.com>
Signed-off-by: NDr. David Alan Gilbert <dgilbert@redhat.com>

Allow a bunch of the info commands to be used in preconfig.

version, chardev, name, uuid,memdev, iothreads
  Were enabled in QMP in the previous patch from Igor

status, hotpluggable_cpus
  Was enabled in the original allow-preconfig series

history
  is HMP specific

qom-tree
  Don't have a QMP equivalent

Also enable the qom commands qom-list and qom-set.
Signed-off-by: NDr. David Alan Gilbert <dgilbert@redhat.com>
Message-Id: <20180620153947.30834-6-dgilbert@redhat.com>
Reviewed-by: NIgor Mammedov <imammedo@redhat.com>
Signed-off-by: NDr. David Alan Gilbert <dgilbert@redhat.com>
  Dropped info numa as per Igor's 2018-06-21 review

Commands query-chardev, query-version, query-name, query-uuid,
query-iothreads, query-memdev are informational and do not depend on
the machine being initialized.  Make them available in preconfig
runstate to make the latter a little bit more useful.
The generic qom commands don't depend on the machine being initialized
either; so enabled qom-list, qom-get, qom-set, qom-list-types,
qom-list-properties.
Signed-off-by: NIgor Mammedov <imammedo@redhat.com>
Signed-off-by: NDr. David Alan Gilbert <dgilbert@redhat.com>
Reviewed-by: NEric Blake <eblake@redhat.com>
Message-Id: <20180620153947.30834-5-dgilbert@redhat.com>
Signed-off-by: NDr. David Alan Gilbert <dgilbert@redhat.com>

Don't show the commands that aren't available.
Signed-off-by: NDr. David Alan Gilbert <dgilbert@redhat.com>
Reviewed-by: NPeter Xu <peterx@redhat.com>
Reviewed-by: NIgor Mammedov <imammedo@redhat.com>
Message-Id: <20180620153947.30834-4-dgilbert@redhat.com>
Signed-off-by: NDr. David Alan Gilbert <dgilbert@redhat.com>

Allow the 'help' command in preconfig state but
make it only list the preconfig commands.
Signed-off-by: NDr. David Alan Gilbert <dgilbert@redhat.com>
Reviewed-by: NPeter Xu <peterx@redhat.com>
Reviewed-by: NIgor Mammedov <imammedo@redhat.com>
Message-Id: <20180620153947.30834-3-dgilbert@redhat.com>
Signed-off-by: NDr. David Alan Gilbert <dgilbert@redhat.com>

Add a flag to command definitions to allow them to be used in preconfig
and check it.
If users try to use commands that aren't available, tell them to use
the exit_preconfig comand we're adding in a few patches.
Signed-off-by: NDr. David Alan Gilbert <dgilbert@redhat.com>
Reviewed-by: NMarkus Armbruster <armbru@redhat.com>
Reviewed-by: NIgor Mammedov <imammedo@redhat.com>
Message-Id: <20180620153947.30834-2-dgilbert@redhat.com>
Signed-off-by: NDr. David Alan Gilbert <dgilbert@redhat.com>

The dump-guest-memory command is used to dump an area of guest memory
to a file, the piece of memory is specified by a begin address and
a length. These parameters are specified as ints and thus have a maximum
value of 4GB. This means you can't dump the guest memory past the first
4GB and instead get:
(qemu) dump-guest-memory tmp 0x100000000 0x100000000
'dump-guest-memory' has failed: integer is for 32-bit values
Try "help dump-guest-memory" for more information

This limitation is imposed in monitor_parse_arguments() since they are
both ints. hmp_dump_guest_memory() uses 64 bit quantities to store both
the begin and length values. Thus specify begin and length as long so
that the entire guest memory space can be dumped.
Signed-off-by: NSuraj Jitindar Singh <sjitindarsingh@gmail.com>
Message-Id: <20180620003202.10546-1-sjitindarsingh@gmail.com>
Reviewed-by: NDr. David Alan Gilbert <dgilbert@redhat.com>
Signed-off-by: NDr. David Alan Gilbert <dgilbert@redhat.com>

When a user incorrectly provides an hmp command, an error response will be
printed that prompts the user to try "help <command name>". However, when
the command contains multiple parts e.g. "info uuid xyz", only the last
whitespace delimited string will be reported (in this example "info" will
be dropped and the message will read "Try "help uuid" for more information",
which is incorrect).

Let's correct this by capturing the entirety of the command from the command
line -- excluding any extraneous characters.
Reported-by: NMikhail Fokin <fokin@de.ibm.com>
Signed-off-by: NCollin Walling <walling@linux.ibm.com>
Message-Id: <ee680f5e-ac9a-479d-f65e-9f8ae9cfe5d4@linux.ibm.com>
Reviewed-by: NEric Blake <eblake@redhat.com>
Signed-off-by: NDr. David Alan Gilbert <dgilbert@redhat.com>

Signed-off-by: NAlex Bennée <alex.bennee@linaro.org>
Reviewed-by: NPhilippe Mathieu-Daudé <f4bug@amsat.org>
Tested-by: NPhilippe Mathieu-Daudé <f4bug@amsat.org>

Now we can check the age of a docker image we can be a little more
intelligent about re-building Sid images and only force NOCACHE if
it is "old".
Signed-off-by: NAlex Bennée <alex.bennee@linaro.org>

This is useful for querying if an image is too old.
Signed-off-by: NAlex Bennée <alex.bennee@linaro.org>
Reviewed-by: NPhilippe Mathieu-Daudé <f4bug@amsat.org>

As we now ensure all the images we are going to use are built in the
top level make file lets not over complicate things by running the
full script again. We do run the check script just in case someone
deletes the docker image while we are running.
Signed-off-by: NAlex Bennée <alex.bennee@linaro.org>

This command allows you to check if we need to re-build a docker
image. If the image isn't in the repository or the checksums don't
match then we return false and some text (for processing in
makefiles).
Signed-off-by: NAlex Bennée <alex.bennee@linaro.org>

This just gets confusing especially as the helper function doesn't
even take into account any extra files (or the executable). Currently
the actual check just ignores them and also passes the result through
_dockerfile_preprocess so we fix that too.
Signed-off-by: NAlex Bennée <alex.bennee@linaro.org>
Reviewed-by: NPhilippe Mathieu-Daudé <f4bug@amsat.org>

The "images" command is a fairly heavyweight command to run as it
involves searching the whole docker file-system inventory. On a
machine with a lot of images this makes start-up fairly expensive.
Signed-off-by: NAlex Bennée <alex.bennee@linaro.org>

One problem with satisfying your docker dependencies in a sub-make it
you might end up trying to satisfy the dependency multiple times. This
is especially a problem with debian-sid based cross compilers and CI
setups. We solve this by doing a docker build pass at the top level
before any sub-makes are called.

We still need to satisfy dependencies in the Makefile.target call so
people can run tests from individual target directories. We introduce
a new Makefile.probe which gets called for each PROBE_TARGET and
allows us to build up the list. It does require multiply including
config-target.mak which shouldn't cause any issues as it shouldn't
define anything that clashes with config-host.mak. However we undefine
a few key variables each time around.
Signed-off-by: NAlex Bennée <alex.bennee@linaro.org>

The Travis hardware can be a little slow and the runcom test is fairly
heavy in calculating pi. Lets double the timeout so we don't trip up
during CI by mistake.
Signed-off-by: NAlex Bennée <alex.bennee@linaro.org>

To get a clean run of check-tcg these tests are currently skipped:

  - hello-mips for mips
  - linux-test for sparc
Signed-off-by: NAlex Bennée <alex.bennee@linaro.org>
Tested-by: NPhilippe Mathieu-Daudé <f4bug@amsat.org>
Reviewed-by: NPhilippe Mathieu-Daudé <f4bug@amsat.org>

As we aren't using the default runners for all the test cases it is
easy to miss out things like timeouts. To help with this we add some
helpers and use them so we only need to make core changes in one
place.
Signed-off-by: NAlex Bennée <alex.bennee@linaro.org>
Reviewed-by: NPhilippe Mathieu-Daudé <f4bug@amsat.org>
Tested-by: NPhilippe Mathieu-Daudé <f4bug@amsat.org>

This will ensure all linux-user targets build their guest test
programs and ensure check-tcg will run the respective tests.
Signed-off-by: NAlex Bennée <alex.bennee@linaro.org>
Reviewed-by: NPhilippe Mathieu-Daudé <f4bug@amsat.org>
Tested-by: NPhilippe Mathieu-Daudé <f4bug@amsat.org>

Now all the build infrastructure is in place we can build tests for
each guest that we support. That support mainly depends on having
cross compilers installed or docker setup. To keep all the logic for
that together we put the rules in tests/tcg/Makefile.include and
include it from the main Makefile.target.
Signed-off-by: NAlex Bennée <alex.bennee@linaro.org>
Reviewed-by: NPhilippe Mathieu-Daudé <f4bug@amsat.org>
Tested-by: NPhilippe Mathieu-Daudé <f4bug@amsat.org>

This make is now invoked from each individual target make with the
appropriate CC and EXTRA_CFLAGS set for each guest. It then includes
additional Makefile.targets from:

  - tests/tcg/multiarch (always)
  - tests/tcg/$(TARGET_BASE_ARCH) (if available)
  - tests/tcg/$(TARGET_NAME)

The order is important as the later Makefile's may want to suppress
TESTS from its base arch profile. Each included Makefile.target is
responsible for adding TESTS as well as defining any special build
instructions for individual tests.
Signed-off-by: NAlex Bennée <alex.bennee@linaro.org>
Reviewed-by: NRichard Henderson <richard.henderson@linaro.org>
Reviewed-by: NPhilippe Mathieu-Daudé <f4bug@amsat.org>
Tested-by: NPhilippe Mathieu-Daudé <f4bug@amsat.org>