Create a SiFive Unleashed U54 SoC and use that in the sifive_u machine.

We leave the SoC, RAM, device tree and reset/fdt loading as part of the
machine. All the other device creation has been moved to the SoC.
Signed-off-by: NAlistair Francis <alistair.francis@wdc.com>
Reviewed-by: NMichael Clark <mjc@sifive.com>

Merge remote-tracking branch 'remotes/stsquad/tags/pull-code-coverage-and-build-tweaks-050718-3' into staging

Code coverage and other build tweaks

  - revert 208ecb3e (and drop filter for mingw, tweak for check-tcg)
  - some travis speed-ups
  - modernise code coverage support
  - docker image cleanups
  - clean-up binfmt_misc docker infrastructure
  - add debian-powerpc-user-cross image for ppc32 build

# gpg: Signature made Thu 05 Jul 2018 17:00:02 BST
# gpg:                using RSA key FBD0DB095A9E2A44
# gpg: Good signature from "Alex Bennée (Master Work Key) <alex.bennee@linaro.org>"
# Primary key fingerprint: 6685 AE99 E751 67BC AFC8  DF35 FBD0 DB09 5A9E 2A44

* remotes/stsquad/tags/pull-code-coverage-and-build-tweaks-050718-3:
  docker: add linux-user powered cross builder for QEMU
  docker: add special rule for deboostrapped images
  docker: add special handling for FROM:debian-%-user targets
  docker: debian-bootstrap.pre allow customising of variant/url
  docker: drop QEMU build-dep from bootstrap
  docker: Do not run tests in 'intermediate' images
  docker: Clean the MXE base image
  docker: ubuntu: Use SDL2
  docker: ubuntu: Update the package list before installing new ones
  linux-user: add gcov support to preexit_cleanup
  linux-user: introduce preexit_cleanup
  build-system: add coverage-report target
  build-system: add clean-coverage target
  travis: add gcovr summary for GCOV build
  docker: add gcovr to travis image
  .gitignore: add .gcov files
  build-system: remove per-test GCOV reporting
  travis: test out-of-tree builds
  travis: do not waste time cloning unused submodules
  Revert "Makefile: Rename TARGET_DIRS to TARGET_LIST"
Signed-off-by: NPeter Maydell <peter.maydell@linaro.org>

We can't use cross compilers in the current Debian stable and Debian
sid is sketchy as hell. So for powerpc fall back to dog-fooding our
own linux-user to do the build.

As we can only build the base image with a suitably configured
source tree we fall back to checking for its existence when we can't
build it from scratch. However this does mean you don't have to keep
a static powerpc-linux-user in your active configuration just to
update the cross build image.
Signed-off-by: NAlex Bennée <alex.bennee@linaro.org>
Reviewed-by: NPhilippe Mathieu-Daudé <f4bug@amsat.org>
Tested-by: NPhilippe Mathieu-Daudé <f4bug@amsat.org>

Coverity model patches for 2018-07-05

# gpg: Signature made Thu 05 Jul 2018 14:11:03 BST
# gpg:                using RSA key 3870B400EB918653
# gpg: Good signature from "Markus Armbruster <armbru@redhat.com>"
# gpg:                 aka "Markus Armbruster <armbru@pond.sub.org>"
# Primary key fingerprint: 354B C8B3 D7EB 2A6B 6867  4E5F 3870 B400 EB91 8653

* remotes/armbru/tags/pull-cov-model-2018-07-05:
  coverity-model: Fix replay_get_byte()
Signed-off-by: NPeter Maydell <peter.maydell@linaro.org>

We might as well have a custom rule for this. For one thing the
dependencies are different. As the primary dependency for
docker-image-% could never be docker-image-debian-bootstrap we can
drop that test in the main rule as well.

Missing EXECUTABLE, DEB_ARCH and DEB_TYPE are treated as hard faults
now. We also error out if the EXECUTABLE file isn't there. We should
really do this with a dependency on any source rules but currently
subdir-FOO-linux-user isn't enough on a clean build.
Signed-off-by: NAlex Bennée <alex.bennee@linaro.org>
Reviewed-by: NPhilippe Mathieu-Daudé <f4bug@amsat.org>
Tested-by: NPhilippe Mathieu-Daudé <f4bug@amsat.org>

These will have been build with debootstrap so we need to check
against the debian-bootstrap dockerfile. This does mean sticking to
debian-FOO-user as the naming conventions for boot-strapped images.
The actual cross image is built on top.
Signed-off-by: NAlex Bennée <alex.bennee@linaro.org>
Reviewed-by: NPhilippe Mathieu-Daudé <f4bug@amsat.org>
Tested-by: NPhilippe Mathieu-Daudé <f4bug@amsat.org>

We default to the buildd variant as most of our images are for
building. However lets give the user the ability to specify "minbase"
if they want to create a simple base image for experimentation.

Allowing the tweaking of DEB_URL means we can also bootstrap other
Debian based OS's. For example:

  make docker-binfmt-image-debian-ubuntu-bionic-arm64 \
       DEB_ARCH=arm64 DEB_TYPE=bionic \
       DEB_VARIANT=minbase DEB_URL=http://ports.ubuntu.com/ \
       EXECUTABLE=./aarch64-linux-user/qemu-aarch64
Signed-off-by: NAlex Bennée <alex.bennee@linaro.org>
Reviewed-by: NPhilippe Mathieu-Daudé <f4bug@amsat.org>
Tested-by: NPhilippe Mathieu-Daudé <f4bug@amsat.org>

This is best done with any child images that actually need it.
Signed-off-by: NAlex Bennée <alex.bennee@linaro.org>
Reviewed-by: NPhilippe Mathieu-Daudé <f4bug@amsat.org>
Tested-by: NPhilippe Mathieu-Daudé <f4bug@amsat.org>

We can still build the DOCKER_INTERMEDIATE_IMAGES images,
but they won't appear in 'make test*@$IMAGE'.
Signed-off-by: NPhilippe Mathieu-Daudé <f4bug@amsat.org>
Signed-off-by: NAlex Bennée <alex.bennee@linaro.org>

Using the duplicated same package is confusing.
Reported-by: NPeter Maydell <peter.maydell@linaro.org>
Signed-off-by: NPhilippe Mathieu-Daudé <f4bug@amsat.org>
Signed-off-by: NAlex Bennée <alex.bennee@linaro.org>

Do not test the deprecated API versions (see cabd3584).
Debian MXE MinGW cross images are already using SDL2.
Signed-off-by: NPhilippe Mathieu-Daudé <f4bug@amsat.org>
Signed-off-by: NAlex Bennée <alex.bennee@linaro.org>

Since docker caches the different layers, updating the package
list does not invalidate the previous "apt-get update" layer,
and it is likely "apt-get install" hits an outdated repository.

See https://docs.docker.com/develop/develop-images/dockerfile_best-practices/#apt-get

This fixes:

  $ make docker-image-ubuntu V=1
  ./tests/docker/docker.py build qemu:ubuntu tests/docker/dockerfiles/ubuntu.docker   --add-current-user
  Sending build context to Docker daemon  3.072kB
  [...]
  E: Failed to fetch http://archive.ubuntu.com/ubuntu/pool/main/m/mesa/libgles2-mesa_17.0.7-0ubuntu0.16.04.2_amd64.deb  404  Not Found
  E: Failed to fetch http://archive.ubuntu.com/ubuntu/pool/main/m/mesa/libgles2-mesa-dev_17.0.7-0ubuntu0.16.04.2_amd64.deb  404  Not Found
  E: Unable to fetch some archives, maybe run apt-get update or try with --fix-missing?
  The command '/bin/sh -c apt-get -y install $PACKAGES' returned a non-zero code: 100
  tests/docker/Makefile.include:40: recipe for target 'docker-image-ubuntu' failed
  make: *** [docker-image-ubuntu] Error 1
Signed-off-by: NPhilippe Mathieu-Daudé <f4bug@amsat.org>
Signed-off-by: NAlex Bennée <alex.bennee@linaro.org>

As we don't always take the normal exit path when running a guest we
can skip the normal exit destructors where gcov normally dumps it's
info. The GCC manual suggests long running programs use __gcov_dump()
to flush out the coverage state periodically so we use that here.
Signed-off-by: NAlex Bennée <alex.bennee@linaro.org>
Reviewed-by: NPhilippe Mathieu-Daudé <f4bug@amsat.org>

To avoid repeating ourselves move our preexit clean-up code into a
helper function. I figured the continuing effort to split of the
syscalls made it worthwhile creating a new file for it now.
Signed-off-by: NAlex Bennée <alex.bennee@linaro.org>
Reviewed-by: NPhilippe Mathieu-Daudé <f4bug@amsat.org>
Reviewed-by: NDaniel P. Berrangé <berrange@redhat.com>
Reviewed-by: NLaurent Vivier <laurent@vivier.eu>

This will build a coverage report under the current directory in
reports/coverage. At the users option a report can be generated by
directly invoking something like:

  make foo/bar/coverage-report.html
Signed-off-by: NAlex Bennée <alex.bennee@linaro.org>
Reviewed-by: NPhilippe Mathieu-Daudé <f4bug@amsat.org>
Tested-by: NPhilippe Mathieu-Daudé <f4bug@amsat.org>
Reviewed-by: NDaniel P. Berrangé <berrange@redhat.com>

This can be used to remove any stale coverage data before any
particular test run. This is useful for analysing individual tests.
Signed-off-by: NAlex Bennée <alex.bennee@linaro.org>
Reviewed-by: NDaniel P. Berrangé <berrange@redhat.com>
Reviewed-by: NPhilippe Mathieu-Daudé <f4bug@amsat.org>
Tested-by: Philippe Mathieu-Daudé <f4bug@amsat.org>---

This gives a more useful summary, sorted by descending % coverage,
after the tests have run. The final numbers will give an idea if our
coverage is getting better or worse.

To keep the width sane we need to post process the file that the old
gcovr tool generates. This is done with a mix of sed, awk and column
in the scripts/coverage-summary.sh script.

As quite a lot of lines don't get covered at all we filter out all the
0% lines. If the file doesn't appear it is not being exercised.
Signed-off-by: NAlex Bennée <alex.bennee@linaro.org>
Reviewed-by: NDaniel P. Berrangé <berrange@redhat.com>

Useful for debugging if nothing else as the gcovr on the Travis images
are a little old.
Signed-off-by: NAlex Bennée <alex.bennee@linaro.org>
Reviewed-by: NPhilippe Mathieu-Daudé <f4bug@amsat.org>
Reviewed-by: NDaniel P. Berrangé <berrange@redhat.com>

These are temporary files generated on gcov runs and shouldn't be
included in the source tree.
Signed-off-by: NAlex Bennée <alex.bennee@linaro.org>
Reviewed-by: NPhilippe Mathieu-Daudé <f4bug@amsat.org>
Tested-by: NPhilippe Mathieu-Daudé <f4bug@amsat.org>
Reviewed-by: NDaniel P. Berrangé <berrange@redhat.com>

I'm not entirely sure who's using this information and certainly in a
CI environment it just washes over as additional noise. Later patches
will provide new reporting options so a user who wants to analyse
individual tests will be able to use that to get the information.
Signed-off-by: NAlex Bennée <alex.bennee@linaro.org>
Reviewed-by: NPhilippe Mathieu-Daudé <f4bug@amsat.org>
Tested-by: NPhilippe Mathieu-Daudé <f4bug@amsat.org>
Reviewed-by: NDaniel P. Berrangé <berrange@redhat.com>

Force one config to build 'out-of-tree' (object files and executables
are created in a tree outside the project source code).
Signed-off-by: NPhilippe Mathieu-Daudé <f4bug@amsat.org>
Signed-off-by: NAlex Bennée <alex.bennee@linaro.org>
Reviewed-by: NDaniel P. Berrangé <berrange@redhat.com>

Builds only require:
- dtc
- keycodemapdb
- capstone
Signed-off-by: NPhilippe Mathieu-Daudé <f4bug@amsat.org>
[AJB: drop wget cache]
Signed-off-by: NAlex Bennée <alex.bennee@linaro.org>
Reviewed-by: NDaniel P. Berrangé <berrange@redhat.com>

This reverts commit 208ecb3e. This was
causing problems by making DEF_TARGET_LIST pointless and having to
jump through hoops to build on mingw with a dully enabled config.
This includes a change to fix the per-guest TCG test probe which was
added after 208ecb3e and used TARGET_LIST.
Signed-off-by: NAlex Bennée <alex.bennee@linaro.org>
Cc: Paolo Bonzini <pbonzini@redhat.com>

Block layer patches:

- qcow2: Use worker threads for compression to improve performance of
  'qemu-img convert -W' and compressed backup jobs
- blklogwrites: New filter driver to log write requests to an image in
  the dm-log-writes format
- file-posix: Fix image locking during image creation
- crypto: Fix memory leak in error path
- Error out instead of silently truncating node names

# gpg: Signature made Thu 05 Jul 2018 11:24:33 BST
# gpg:                using RSA key 7F09B272C88F2FD6
# gpg: Good signature from "Kevin Wolf <kwolf@redhat.com>"
# Primary key fingerprint: DC3D EB15 9A9A F95D 3D74  56FE 7F09 B272 C88F 2FD6

* remotes/kevin/tags/for-upstream:
  file-posix: Unlock FD after creation
  file-posix: Fix creation locking
  block/blklogwrites: Add an option for the update interval of the log superblock
  block/blklogwrites: Add an option for appending to an old log
  block/blklogwrites: Change log_sector_size from int64_t to uint64_t
  block/crypto: Fix memory leak in create error path
  block: Don't silently truncate node names
  block: Add blklogwrites
  block: Move two block permission constants to the relevant enum
  qcow2: add compress threads
  qcow2: refactor data compression
  qemu-img: allow compressed not-in-order writes
Signed-off-by: NPeter Maydell <peter.maydell@linaro.org>

Coverity 2018.06 chokes on replay_get_byte():

    $ cov-make-library -of scripts/coverity-model.xmldb scripts/coverity-model.c
    output file: scripts/coverity-model.xmldb
    Compiling scripts/coverity-model.c with command /opt/cov-sa-2018.06/bin/cov-emit --dir /tmp/cov-armbru/930a6fb31e5f464fc1a53354b2deb66b/cov-make-library-emit -w --no_error_recovery --emit_header_functions --no_implicit_decl --preinclude /opt/cov-sa-2018.06/library/decls.h --c scripts/coverity-model.c
    "scripts/coverity-model.c", line 110: error #20: identifier "replay_file" is
              undefined
           if (replay_file) {
               ^

    Emit for file '/work/armbru/qemu/scripts/coverity-model.c' complete.
    [ERROR] 1 error detected in the compilation of "scripts/coverity-model.c".
    ERROR: cov-emit returned with code 1

Broken in commit 04a0afe5.  Fix by dumbing down.
Signed-off-by: NMarkus Armbruster <armbru@redhat.com>
Message-Id: <20180626085642.4973-1-armbru@redhat.com>
Reviewed-by: NPaolo Bonzini <pbonzini@redhat.com>

Pull request

# gpg: Signature made Wed 04 Jul 2018 07:33:42 BST
# gpg:                using RSA key 7DEF8106AAFC390E
# gpg: Good signature from "John Snow (John Huston) <jsnow@redhat.com>"
# Primary key fingerprint: FAEB 9711 A12C F475 812F  18F2 88A9 064D 1835 61EB
#      Subkey fingerprint: F9B7 ABDB BCAC DF95 BE76  CBD0 7DEF 8106 AAFC 390E

* remotes/jnsnow/tags/bitmaps-pull-request:
  dirty-bitmap: fix double lock on bitmap enabling
  block/dirty-bitmap: add bdrv_enable_dirty_bitmap_locked
Signed-off-by: NPeter Maydell <peter.maydell@linaro.org>

Monitor patches for 2018-07-03

# gpg: Signature made Tue 03 Jul 2018 22:20:13 BST
# gpg:                using RSA key 3870B400EB918653
# gpg: Good signature from "Markus Armbruster <armbru@redhat.com>"
# gpg:                 aka "Markus Armbruster <armbru@pond.sub.org>"
# Primary key fingerprint: 354B C8B3 D7EB 2A6B 6867  4E5F 3870 B400 EB91 8653

* remotes/armbru/tags/pull-monitor-2018-07-03-v2: (32 commits)
  qapi: Polish command flags documentation in qapi-code-gen.txt
  monitor: Improve some comments
  qmp: Clean up capability negotiation after commit 02130314
  qobject: Let qobject_from_jsonf() fail instead of abort
  qmp: Switch timestamp_put() to qdict_from_jsonf_nofail()
  qmp: Add some comments around null responses
  qmp: Simplify monitor_qmp_respond()
  qmp: Replace get_qmp_greeting() by qmp_greeting()
  qmp: Replace monitor_json_emitter{,raw}() by qmp_{queue,send}_response()
  qmp: Use QDict * instead of QObject * for response objects
  qmp: De-duplicate error response building
  qobject: New qdict_from_jsonf_nofail()
  monitor: Peel off @mon_global wrapper
  monitor: Rename use_io_thr to use_io_thread
  qmp: Don't let JSON errors jump the queue
  qmp: Don't let malformed in-band commands jump the queue
  tests/qmp-test: Demonstrate QMP errors jumping the queue
  qmp: Simplify code around monitor_qmp_dispatch_one()
  qmp: Always free QMPRequest with qmp_request_free()
  qmp: Revert change to handle_qmp_command tracepoint
  ...
Signed-off-by: NPeter Maydell <peter.maydell@linaro.org>

QAPI patches for 2018-07-03

# gpg: Signature made Tue 03 Jul 2018 21:52:55 BST
# gpg:                using RSA key 3870B400EB918653
# gpg: Good signature from "Markus Armbruster <armbru@redhat.com>"
# gpg:                 aka "Markus Armbruster <armbru@pond.sub.org>"
# Primary key fingerprint: 354B C8B3 D7EB 2A6B 6867  4E5F 3870 B400 EB91 8653

* remotes/armbru/tags/pull-qapi-2018-07-03:
  qapi: add conditions to SPICE type/commands/events on the schema
  qapi: add conditions to VNC type/commands/events on the schema
  qapi: add 'If:' section to generated documentation
  qapi-types: add #if conditions to types & visitors
  qapi/events: add #if conditions to events
  qapi/commands: add #if conditions to commands
  qapi-introspect: add preprocessor conditions to generated QLit
  qapi-introspect: modify to_qlit() to append ',' on level > 0
  qapi: add #if/#endif helpers
  qapi: mcgen() shouldn't indent # lines
  qapi: add 'ifcond' to visitor methods
  qapi: leave the ifcond attribute undefined until check()
  qapi: pass 'if' condition into QAPISchemaEntity objects
  qapi: add 'if' to top-level expressions
Signed-off-by: NPeter Maydell <peter.maydell@linaro.org>

Closing the FD does not necessarily mean that it is unlocked.  Fix this
by relinquishing all permission locks before qemu_close().
Reported-by: NKevin Wolf <kwolf@redhat.com>
Signed-off-by: NMax Reitz <mreitz@redhat.com>
Signed-off-by: NKevin Wolf <kwolf@redhat.com>

raw_apply_lock_bytes() takes a bit mask of "permissions that are NOT
shared".

Also, make the "perm" and "shared" variables uint64_t, because I do not
particularly like using ~ on signed integers (and other permission masks
are usually uint64_t, too).
Reported-by: NKevin Wolf <kwolf@redhat.com>
Signed-off-by: NMax Reitz <mreitz@redhat.com>
Signed-off-by: NKevin Wolf <kwolf@redhat.com>

This is a way to ensure that the log superblock is periodically
updated. Before, this was only done on flush requests, which may
not be enough if the VM exits abnormally, omitting the final flush.

The default interval is 4096 write requests.
Signed-off-by: NAri Sundholm <ari@tuxera.com>
Signed-off-by: NKevin Wolf <kwolf@redhat.com>

Suggested by Kevin Wolf. May be useful when testing multiple batches
of writes or doing long-term testing involving restarts of the VM.
Signed-off-by: NAri Sundholm <ari@tuxera.com>
Signed-off-by: NKevin Wolf <kwolf@redhat.com>

This was a simple oversight when working on intermediate versions
of the original patch which introduced blklogwrites.
Signed-off-by: NAri Sundholm <ari@tuxera.com>
Signed-off-by: NKevin Wolf <kwolf@redhat.com>

Fixes: Coverity CID 1393782
Signed-off-by: NKevin Wolf <kwolf@redhat.com>
Reviewed-by: NDaniel P. Berrangé <berrange@redhat.com>

If the user passes a too long node name string, we silently truncate it
to fit into BlockDriverState.node_name, i.e. to 31 characters. Apart
from surprising the user when the node has a different name than
requested, this also bypasses the check for duplicate names, so that the
same name can be assigned to multiple nodes.

Fix this by just making too long node names an error.
Reported-by: NPeter Krempa <pkrempa@redhat.com>
Signed-off-by: NKevin Wolf <kwolf@redhat.com>

Implements a block device write logging system, similar to Linux kernel
device mapper dm-log-writes. The write operations that are performed
on a block device are logged to a file or another block device. The
write log format is identical to the dm-log-writes format. Currently,
log markers are not supported.

This functionality can be used for crash consistency and fs consistency
testing. By implementing it in qemu, tests utilizing write logs can be
be used to test non-Linux drivers and older kernels.

The driver accepts an optional parameter to set the sector size used
for logging. This makes the driver require all requests to be aligned
to this sector size and also makes offsets and sizes of writes in the
log metadata to be expressed in terms of this value (the log format has
a granularity of one sector for offsets and sizes). This allows
accurate logging of writes to guest block devices that have unusual
sector sizes.

The implementation is based on the blkverify and blkdebug block
drivers.
Signed-off-by: NAapo Vienamo <aapo@tuxera.com>
Signed-off-by: NAri Sundholm <ari@tuxera.com>
Signed-off-by: NKevin Wolf <kwolf@redhat.com>

This allows using the two constants outside of block.c, which will
happen in a subsequent patch.
Signed-off-by: NAri Sundholm <ari@tuxera.com>
Signed-off-by: NKevin Wolf <kwolf@redhat.com>

Do data compression in separate threads. This significantly improve
performance for qemu-img convert with -W (allow async writes) and -c
(compressed) options.
Signed-off-by: NVladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
Signed-off-by: NKevin Wolf <kwolf@redhat.com>

Make a separate function for compression to be parallelized later.
 - use .avail_out field instead of .next_out to calculate size of
   compressed data. It looks more natural and it allows to keep dest to
   be void pointer
 - set avail_out to be at least one byte less than input, to be sure
   avoid inefficient compression earlier
Signed-off-by: NVladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
Signed-off-by: NKevin Wolf <kwolf@redhat.com>

No reason to forbid them, and they are needed to improve performance
with compress-threads in further patches.
Signed-off-by: NVladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
Signed-off-by: NKevin Wolf <kwolf@redhat.com>