Python queue, 2017-09-22

* MAINTAINERS update
* Fix logging issue on test scripts using qemu.py

# gpg: Signature made Fri 22 Sep 2017 15:41:43 BST
# gpg:                using RSA key 0x2807936F984DC5A6
# gpg: Good signature from "Eduardo Habkost <ehabkost@redhat.com>"
# Primary key fingerprint: 5A32 2FD5 ABC4 D3DB ACCF  D1AA 2807 936F 984D C5A6

* remotes/ehabkost/tags/python-next-pull-request:
  MAINTAINERS: Add Python scripts
  qemu.py: Call logging.basicConfig() automatically
Signed-off-by: NPeter Maydell <peter.maydell@linaro.org>

Cleber and I are volunteering to review and queue patches for the
Python scripts and modules in scripts/.

I'm setting "M: Odd fixes" because not all scripts are actively
maintained.
Signed-off-by: NEduardo Habkost <ehabkost@redhat.com>
Message-Id: <20170915230744.22942-1-ehabkost@redhat.com>
[ehabkost: add tests/*.py too]
Signed-off-by: NEduardo Habkost <ehabkost@redhat.com>

Not all scripts using qemu.py configure the Python logging
module, and end up generating a "No handlers could be found for
logger" message instead of actual log messages.

To avoid requiring every script using qemu.py to configure
logging manually, call basicConfig() when creating a QEMUMachine
object.  This won't affect scripts that already set up logging,
but will ensure that scripts that don't configure logging keep
working.
Reported-by: NKevin Wolf <kwolf@redhat.com>
Fixes: 4738b0a8Signed-off-by: NEduardo Habkost <ehabkost@redhat.com>
Message-Id: <20170921162234.847-1-ehabkost@redhat.com>
Reviewed-by: NCleber Rosa <crosa@redhat.com>
Acked-by: NLukáš Doktor <ldoktor@redhat.com>
Signed-off-by: NEduardo Habkost <ehabkost@redhat.com>

migration/next for 20170922

# gpg: Signature made Fri 22 Sep 2017 13:15:06 BST
# gpg:                using RSA key 0xF487EF185872D723
# gpg: Good signature from "Juan Quintela <quintela@redhat.com>"
# gpg:                 aka "Juan Quintela <quintela@trasno.org>"
# Primary key fingerprint: 1899 FF8E DEBF 58CC EE03  4B82 F487 EF18 5872 D723

* remotes/juanquintela/tags/migration/20170922-1:
  migration: split ufd_version_check onto receive/request features part
  migration: fix hardcoded function name in error report
  migration: pass MigrationIncomingState* into migration check functions
  migration: split common postcopy out of ram postcopy
  migration: fix ram_save_pending
  migration: add has_postcopy savevm handler
  bitmap: provide to_le/from_le helpers
  bitmap: introduce bitmap_count_one()
  bitmap: remove BITOP_WORD()
  migration: Split migration_fd_process_incoming
  migration: Create multifd migration threads
  migration: Create x-multifd-page-count parameter
  migration: Create x-multifd-channels parameter
  migration: Add multifd capability
  migration: Create migration_has_all_channels
  migration: Add comments to channel functions
  migration: Teach it about G_SOURCE_REMOVE
  migration: Create migration_ioc_process_incoming()
Signed-off-by: NPeter Maydell <peter.maydell@linaro.org>

Apparently GCC gets bent over comparing enum values against zero.
Replace the conditional with something less readable.
Tested-by: NMark Cave-Ayland <mark.cave-ayland@ilande.co.uk>
Signed-off-by: NJohn Snow <jsnow@redhat.com>
Reviewed-by: NStefan Hajnoczi <stefanha@redhat.com>
Message-id: 20170921013821.1673-1-jsnow@redhat.com
Signed-off-by: NPeter Maydell <peter.maydell@linaro.org>

This modification is necessary for userfault fd features which are
required to be requested from userspace.
UFFD_FEATURE_THREAD_ID is a one of such "on demand" feature, which will
be introduced in the next patch.

QEMU have to use separate userfault file descriptor, due to
userfault context has internal state, and after first call of
ioctl UFFD_API it changes its state to UFFD_STATE_RUNNING (in case of
success), but kernel while handling ioctl UFFD_API expects UFFD_STATE_WAIT_API.
So only one ioctl with UFFD_API is possible per ufd.
Reviewed-by: NDr. David Alan Gilbert <dgilbert@redhat.com>
Signed-off-by: NAlexey Perevalov <a.perevalov@samsung.com>
Signed-off-by: NJuan Quintela <quintela@redhat.com>

Reviewed-by: NJuan Quintela <quintela@redhat.com>
Reviewed-by: NDr. David Alan Gilbert <dgilbert@redhat.com>
Signed-off-by: NAlexey Perevalov <a.perevalov@samsung.com>
Signed-off-by: NJuan Quintela <quintela@redhat.com>

That tiny refactoring is necessary to be able to set
UFFD_FEATURE_THREAD_ID while requesting features, and then
to create downtime context in case when kernel supports it.
Signed-off-by: NAlexey Perevalov <a.perevalov@samsung.com>
Reviewed-by: NJuan Quintela <quintela@redhat.com>
Signed-off-by: NJuan Quintela <quintela@redhat.com>

Split common postcopy staff from ram postcopy staff.
Signed-off-by: NVladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
Reviewed-by: NDr. David Alan Gilbert <dgilbert@redhat.com>
Reviewed-by: NJuan Quintela <quintela@redhat.com>
Signed-off-by: NJuan Quintela <quintela@redhat.com>

Fill postcopy-able pending only if ram postcopy is enabled.
It is necessary because of there will be other postcopy-able states and
when ram postcopy is disabled, it should not spoil common postcopy
related pending.
Signed-off-by: NVladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
Reviewed-by: NDr. David Alan Gilbert <dgilbert@redhat.com>
Reviewed-by: NJuan Quintela <quintela@redhat.com>
Signed-off-by: NJuan Quintela <quintela@redhat.com>

Now postcopy-able states are recognized by not NULL
save_live_complete_postcopy handler. But when we have several different
postcopy-able states, it is not convenient. Ram postcopy may be
disabled, while some other postcopy enabled, in this case Ram state
should behave as it is not postcopy-able.

This patch add separate has_postcopy handler to specify behaviour of
savevm state.
Signed-off-by: NVladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
Reviewed-by: NJuan Quintela <quintela@redhat.com>
Signed-off-by: NJuan Quintela <quintela@redhat.com>

Provide helpers to convert bitmaps to little endian format. It can be
used when we want to send one bitmap via network to some other hosts.

One thing to mention is that, these helpers only solve the problem of
endianess, but it does not solve the problem of different word size on
machines (the bitmaps managing same count of bits may contains different
size when malloced). So we need to take care of the size alignment issue
on the callers for now.
Signed-off-by: NPeter Xu <peterx@redhat.com>
Reviewed-by: NJuan Quintela <quintela@redhat.com>
Signed-off-by: NJuan Quintela <quintela@redhat.com>

Count how many bits set in the bitmap.
Reviewed-by: NDr. David Alan Gilbert <dgilbert@redhat.com>
Signed-off-by: NPeter Xu <peterx@redhat.com>
Reviewed-by: NJuan Quintela <quintela@redhat.com>
Signed-off-by: NJuan Quintela <quintela@redhat.com>

We have BIT_WORD(). It's the same.
Signed-off-by: NPeter Xu <peterx@redhat.com>
Reviewed-by: NJuan Quintela <quintela@redhat.com>
Signed-off-by: NJuan Quintela <quintela@redhat.com>

We need that on later patches.
Signed-off-by: NJuan Quintela <quintela@redhat.com>
Reviewed-by: NDr. David Alan Gilbert <dgilbert@redhat.com>
Reviewed-by: NPeter Xu <peterx@redhat.com>
Reviewed-by: NDaniel P. Berrange <berrange@redhat.com>

Creation of the threads, nothing inside yet.
Signed-off-by: NJuan Quintela <quintela@redhat.com>
Reviewed-by: NDr. David Alan Gilbert <dgilbert@redhat.com>

--

Use pointers instead of long array names
Move to use semaphores instead of conditions as paolo suggestion

Put all the state inside one struct.
Use a counter for the number of threads created.  Needed during cancellation.

Add error return to thread creation

Add id field

Rename functions to multifd_save/load_setup/cleanup
Change recv parameters to a pointer to struct
Change back to a struct
Use Error * for _cleanup

Indicates how many pages we are going to send in each batch to a multifd
thread.
Signed-off-by: NJuan Quintela <quintela@redhat.com>
Reviewed-by: NDr. David Alan Gilbert <dgilbert@redhat.com>
Reviewed-by: NPeter Xu <peterx@redhat.com>

--

Be consistent with defaults and documentation
Use new DEFINE_PROP_*
Rename x-multifd-group to x-multifd-page-count

Indicates the number of channels that we will create.  By default we
create 2 channels.
Signed-off-by: NJuan Quintela <quintela@redhat.com>
Reviewed-by: NDr. David Alan Gilbert <dgilbert@redhat.com>
Reviewed-by: NPeter Xu <peterx@redhat.com>

--

Catch inconsistent defaults (eric).
Improve comment stating that number of threads is the same than number
of sockets
Use new DEFIN_PROP_*
Rename x-multifd-threads to x-multifd-threads

Signed-off-by: NJuan Quintela <quintela@redhat.com>
Reviewed-by: NDr. David Alan Gilbert <dgilbert@redhat.com>
Reviewed-by: NPeter Xu <peterx@redhat.com>
Reviewed-by: NDaniel P. Berrange <berrange@redhat.com>

--

Use new DEFINE_PROP

This function allows us to decide when to close the listener socket.
For now, we only need one connection.
Signed-off-by: NJuan Quintela <quintela@redhat.com>
Reviewed-by: NDaniel P. Berrange <berrange@redhat.com>

Signed-off-by: NJuan Quintela <quintela@redhat.com>
Reviewed-by: NDaniel P. Berrange <berrange@redhat.com>

As this is defined on glib 2.32, add compatibility macros for older glibs.
Signed-off-by: NJuan Quintela <quintela@redhat.com>
Reviewed-by: NDaniel P. Berrange <berrange@redhat.com>
Reviewed-by: NPeter Xu <peterx@redhat.com>

We pass the ioc instead of the fd.  This will allow us to have more
than one channel open.  We also make sure that we set the
from_src_file sooner, so we don't need to pass it as a parameter.
Signed-off-by: NJuan Quintela <quintela@redhat.com>
Reviewed-by: NDaniel P. Berrange <berrange@redhat.com>

--

Do not assing mis->from_src_file (peterxu)

Merge remote-tracking branch 'remotes/famz/tags/build-and-test-automation-pull-request' into staging

# gpg: Signature made Fri 22 Sep 2017 08:28:38 BST
# gpg:                using RSA key 0xCA35624C6A9171C6
# gpg: Good signature from "Fam Zheng <famz@redhat.com>"
# gpg: WARNING: This key is not certified with sufficiently trusted signatures!
# gpg:          It is not certain that the signature belongs to the owner.
# Primary key fingerprint: 5003 7CB7 9706 0F76 F021  AD56 CA35 624C 6A91 71C6

* remotes/famz/tags/build-and-test-automation-pull-request: (36 commits)
  docker: Drop 'set -e' from run script
  docker: Use archive-source.py
  tests: Add README for vm tests
  MAINTAINERS: Add tests/vm entry
  Makefile: Add rules to run vm tests
  tests: Add OpenBSD image
  tests: Add NetBSD image
  tests: Add FreeBSD image
  tests: Add ubuntu.i386 image
  tests: Add vm test lib
  tests: Add a test key pair
  scripts: Add archive-source.sh
  qemu.py: Add "wait()" method
  gitignore: Ignore vm test images
  MAINTAINERS: Fix subsystem name for "Build and test automation"
  buildsys: Move rdma libs to per object
  buildsys: Move brlapi libs to per object
  buildsys: Move usb redir cflags/libs to per object
  buildsys: Move libusb cflags/libs to per object
  buildsys: Move libcacard cflags/libs to per object
  ...
Signed-off-by: NPeter Maydell <peter.maydell@linaro.org>

Fix an s390x migration breakage up for 2.10 stable.
This will be fixed properly for 2.11.

# gpg: Signature made Fri 22 Sep 2017 08:28:22 BST
# gpg:                using RSA key 0xDECF6B93C6F02FAF
# gpg: Good signature from "Cornelia Huck <conny@cornelia-huck.de>"
# gpg:                 aka "Cornelia Huck <huckc@linux.vnet.ibm.com>"
# gpg:                 aka "Cornelia Huck <cornelia.huck@de.ibm.com>"
# gpg:                 aka "Cornelia Huck <cohuck@kernel.org>"
# gpg:                 aka "Cornelia Huck <cohuck@redhat.com>"
# Primary key fingerprint: C3D0 D66D C362 4FF6 A8C0  18CE DECF 6B93 C6F0 2FAF

* remotes/cohuck/tags/s390x-20170922:
  s390x/ais: for 2.10 stable: disable ais facility
Signed-off-by: NPeter Maydell <peter.maydell@linaro.org>

The only prototype doesn't need anything from the lib header, and not
including it here allows files that include this header, for example
vl.c, to compile without the libseccomp cflags.

The breakage is since c3883e1f for environments where `pkg-config
--cflags libseccomp" is non-empty.
Reported-by: NJan Kiszka <jan.kiszka@siemens.com>
Signed-off-by: NFam Zheng <famz@redhat.com>
Acked-by: NEduardo Otubo <otubo@redhat.com>
Message-id: 20170920083647.14599-1-famz@redhat.com
Signed-off-by: NPeter Maydell <peter.maydell@linaro.org>

The migration interface for ais was introduced with kernel 4.13
but the capability itself had been active since 4.12. As migration
support is considered necessary lets disable ais in the 2.10
stable version. A proper fix and re-enablement will be done
for qemu 2.11.
Signed-off-by: NChristian Borntraeger <borntraeger@de.ibm.com>
Message-Id: <20170921140834.14233-2-borntraeger@de.ibm.com>
Signed-off-by: NCornelia Huck <cohuck@redhat.com>

Signed-off-by: NFam Zheng <famz@redhat.com>

Signed-off-by: NFam Zheng <famz@redhat.com>
Reviewed-by: NAlex Bennée <alex.bennee@linaro.org>

Signed-off-by: NFam Zheng <famz@redhat.com>

Signed-off-by: NFam Zheng <famz@redhat.com>
Reviewed-by: NStefan Hajnoczi <stefanha@redhat.com>

Reviewed-by: NAlex Bennée <alex.bennee@linaro.org>
Tested-by: NAlex Bennée <alex.bennee@linaro.org>
Signed-off-by: NFam Zheng <famz@redhat.com>

The image is prepared following instructions as in:

https://wiki.qemu.org/Hosts/BSDSigned-off-by: NFam Zheng <famz@redhat.com>

The image is prepared following instructions as in:

https://wiki.qemu.org/Hosts/BSDSigned-off-by: NFam Zheng <famz@redhat.com>
Reviewed-by: NKamil Rytarowski <n54@gmx.com>

The image is prepared following instructions as in:

https://wiki.qemu.org/Hosts/BSDSigned-off-by: NFam Zheng <famz@redhat.com>
Tested-by: NPhilippe Mathieu-Daudé <f4bug@amsat.org>

This adds a 32bit guest.

The official LTS cloud image is downloaded and initialized with
cloud-init.
Signed-off-by: NFam Zheng <famz@redhat.com>

This is the common code to implement a "VM test" to

  1) Download and initialize a pre-defined VM that has necessary
  dependencies to build QEMU and SSH access.

  2) Archive $SRC_PATH to a .tar file.

  3) Boot the VM, and pass the source tar file to the guest.

  4) SSH into the VM, untar the source tarball, build from the source.
Signed-off-by: NFam Zheng <famz@redhat.com>
Reviewed-by: NAlex Bennée <alex.bennee@linaro.org>

This will be used by setup test user ssh.
Signed-off-by: NFam Zheng <famz@redhat.com>
Reviewed-by: NEric Blake <eblake@redhat.com>

Signed-off-by: NFam Zheng <famz@redhat.com>
Reviewed-by: NEric Blake <eblake@redhat.com>

Signed-off-by: NFam Zheng <famz@redhat.com>
Reviewed-by: NStefan Hajnoczi <stefanha@redhat.com>
Reviewed-by: NAlex Bennée <alex.bennee@linaro.org>