Assorted s390x patches:
- updates for virtio-ccw and s390-virtio, making them more similar
  to virtio-pci
- improvements regarding per-vcpu interrupts and migration

# gpg: Signature made Fri May  8 09:45:09 2015 BST using RSA key ID C6F02FAF
# gpg: Good signature from "Cornelia Huck <huckc@linux.vnet.ibm.com>"
# gpg:                 aka "Cornelia Huck <cornelia.huck@de.ibm.com>"

* remotes/cohuck/tags/s390x-20150508:
  s390x/kvm: migrate vcpu interrupt state
  s390x: move fpu regs into a subsection of the vmstate
  s390x/kvm: use ioctl KVM_S390_IRQ for vcpu interrupts
  virtio-ccw: implement ->device_plugged
  virtio-ccw: change realization sequence
  s390-virtio: clear {used,avail}_event_idx on reset as well
  s390-virtio: use common features
  s390-virtio: Accommodate guests using virtqueues too early
Signed-off-by: NPeter Maydell <peter.maydell@linaro.org>

This patch adds support to migrate vcpu interrupts.
We use ioctl KVM_S390_GET_IRQ_STATE and _SET_IRQ_STATE
to get/set the complete interrupt state for a vcpu.
Reviewed-by: NDavid Hildenbrand <dahi@linux.vnet.ibm.com>
Signed-off-by: NJens Freimann <jfrei@linux.vnet.ibm.com>
Signed-off-by: NCornelia Huck <cornelia.huck@de.ibm.com>

Let's move the floating point registers into a seperate subsection and
bump up the version id. This cleans up the current vmstate and will
allow for a future extension with vector registers in a compatible way.

This patch is based on a patch from Eric Farman.
Reviewed-by: NEric Farman <farman@linux.vnet.ibm.com>
Signed-off-by: NDavid Hildenbrand <dahi@linux.vnet.ibm.com>
Signed-off-by: NCornelia Huck <cornelia.huck@de.ibm.com>

KVM_S390_INT uses only two parameter fields. This is not
enough to pass all required information for certain interrupts.

A new ioctl KVM_S390_IRQ is available which allows us to
inject all local interrupts as defined in the Principles of
Operation. It takes a struct kvm_s390_irq as a parameter
which can store interrupt payload data for all interrupts.

Let's use the new ioctl for injecting vcpu interrupts.
Tested-by: NThomas Huth <thuth@linux.vnet.ibm.com>
Reviewed-by: NThomas Huth <thuth@linux.vnet.ibm.com>
Reviewed-by: NDavid Hildenbrand <dahi@linux.vnet.ibm.com>
Signed-off-by: NJens Freimann <jfrei@linux.vnet.ibm.com>
Signed-off-by: NCornelia Huck <cornelia.huck@de.ibm.com>

Let's move operations that are only valid after the backend has been
realized to a ->device_plugged callback, just as virtio-pci does.
Also reorder setting up the host feature bits to the sequence used
by virtio-pci.

While we're at it, also add a ->device_unplugged callback to stop
ioeventfd, just to be on the safe side.
Reviewed-by: NShannon Zhao <shannon.zhao@linaro.org>
Signed-off-by: NCornelia Huck <cornelia.huck@de.ibm.com>
Message-Id: <1429627016-30656-3-git-send-email-cornelia.huck@de.ibm.com>

virtio-ccw has an odd sequence of realizing devices: first the
device-specific relization (net, block, ...), then the generic
realization. It feels less odd to have the generic realization
callback trigger the device-specific realization instead (and this
also matches what virtio-pci does).

One thing to note: We need to defer initializing the cu model in the
sense id data until after the device-specific realization has been
performed, as we need to refer to the virtio device's device_id.
Signed-off-by: NCornelia Huck <cornelia.huck@de.ibm.com>
Message-Id: <1429627016-30656-2-git-send-email-cornelia.huck@de.ibm.com>

The old s390-virtio transport clears the vring used/avail indices in
the shared area on reset. When we enabled event_idx for virtio-blk, we
noticed that this is not enough: We also need to clear the published
used/avail event indices, or reboot will fail.
Signed-off-by: NChristian Borntraeger <borntraeger@de.ibm.com>
Signed-off-by: NCornelia Huck <cornelia.huck@de.ibm.com>

We used to avoid enabling event_idx for virtio-blk devices via
s390-virtio, but we now have a workaround in place for guests trying
to use the device before setting DRIVER_OK. Therefore, let's add
DEFINE_VIRTIO_COMMON_FEATURES to the base device so all devices get
those common features - and make s390-virtio use the same mechanism
as the other transports do.
Acked-by: NChristian Borntraeger <borntraeger@de.ibm.com>
Reviewed-by: NShannon Zhao <shannon.zhao@linaro.org>
Signed-off-by: NCornelia Huck <cornelia.huck@de.ibm.com>

Feature updates are not a synchronuous operation for the legacy
s390-virtio transport. This transport syncs the guest feature bits
(those from finalize) on the set_status hypercall. Before that qemu
thinks that features are zero, which means QEMU will misbehave, e.g.
it will not write the event index, even if the guest asks for it.

Let's detect the case where a kick happens before the driver is ready
and force sync the features.
With this workaround, it is now safe to switch to the common feature
bit handling code as used by all other transports.
Signed-off-by: NChristian Borntraeger <borntraeger@de.ibm.com>
Reviewed-by: NCornelia Huck <cornelia.huck@de.ibm.com>
Signed-off-by: NCornelia Huck <cornelia.huck@de.ibm.com>

Reviewed-by: NPaolo Bonzini <pbonzini@redhat.com>
Signed-off-by: NEdgar E. Iglesias <edgar.iglesias@xilinx.com>

migration/next for 20150507

# gpg: Signature made Thu May  7 17:42:19 2015 BST using RSA key ID 5872D723
# gpg: Good signature from "Juan Quintela <quintela@redhat.com>"
# gpg:                 aka "Juan Quintela <quintela@trasno.org>"

* remotes/juanquintela/tags/migration/20150507-1:
  migration: Fix migration state update issue
  migration: avoid divide by zero in xbzrle cache miss rate
  migration: Add hmp interface to set and query parameters
  migration: Add qmp commands to set and query parameters
  migration: Use an array instead of 3 parameters
  migration: Add interface to control compression
  migration: Add the core code for decompression
  migration: Make compression co-work with xbzrle
  migration: Add the core code of multi-thread compression
  migration: Split save_zero_page from ram_save_page
  arch_init: Add and free data struct for decompression
  arch_init: Alloc and free data struct for compression
  qemu-file: Add compression functions to QEMUFile
  migration: Add the framework of multi-thread decompression
  migration: Add the framework of multi-thread compression
  docs: Add a doc about multiple thread compression
Signed-off-by: NPeter Maydell <peter.maydell@linaro.org>

If live migration is very fast and can be completed in 1 second,
the dirty_sync_count of MigrationState will not be updated.
Then you will see "dirty sync count: 0" in qemu monitor even if
the actual dirty sync count is not 0.
Signed-off-by: NLiang Li <liang.z.li@intel.com>
Reviewed-by: NJuan Quintela <quintela@redhat.com>
Reviewed-by: NDr.David Alan Gilbert <dgilbert@redhat.com>
Signed-off-by: NJuan Quintela <quintela@redhat.com>

This bug manifested itself as a VM that could not be resumed by libvirt
following a migration:

  # virsh resume example
  error: Failed to resume domain example
  error: internal error: cannot parse json {"return":
    {"xbzrle-cache":
      {..., "cache-miss-rate": -nan, ...},
      ...
    }
  }: lexical error: malformed number, a digit is required after the minus sign.

This patch also ensures xbzrle_cache_miss_prev and iterations_prev are
cleared at the start of the migration.
Signed-off-by: NMichael Chapman <mike@very.puzzling.org>
Reviewed-by: NAmit Shah <amit.shah@redhat.com>
Signed-off-by: NJuan Quintela <quintela@redhat.com>

Add the hmp interface to tune and query the parameters used in
live migration.
Signed-off-by: NLiang Li <liang.z.li@intel.com>
Signed-off-by: NYang Zhang <yang.z.zhang@intel.com>
Signed-off-by: NJuan Quintela <quintela@redhat.com>

Add the qmp commands to tune and query the parameters used in live
migration.
Signed-off-by: NLiang Li <liang.z.li@intel.com>
Signed-off-by: NYang Zhang <yang.z.zhang@intel.com>
Signed-off-by: NJuan Quintela <quintela@redhat.com>

Put the three parameters related to multiple thread (de)compression
into an int array, and use an enum type to index the parameter.
Signed-off-by: NLiang Li <liang.z.li@intel.com>
Signed-off-by: NYang Zhang <yang.z.zhang@intel.com>
Signed-off-by: NJuan Quintela <quintela@redhat.com>

The multiple compression threads can be turned on/off through
qmp and hmp interface before doing live migration.
Signed-off-by: NLiang Li <liang.z.li@intel.com>
Signed-off-by: NYang Zhang <yang.z.zhang@intel.com>
Reviewed-by: NDr.David Alan Gilbert <dgilbert@redhat.com>
Reviewed-by: NEric Blake <eblake@redhat.com>
Signed-off-by: NJuan Quintela <quintela@redhat.com>

Implement the core logic of multiple thread decompression,
the decompression can work now.
Signed-off-by: NLiang Li <liang.z.li@intel.com>
Signed-off-by: NYang Zhang <yang.z.zhang@intel.com>
Signed-off-by: NJuan Quintela <quintela@redhat.com>

Now, multiple thread compression can co-work with xbzrle. when
xbzrle is on, multiple thread compression will only work at the
first round of RAM data sync.
Signed-off-by: NLiang Li <liang.z.li@intel.com>
Signed-off-by: NYang Zhang <yang.z.zhang@intel.com>
Reviewed-by: NDr.David Alan Gilbert <dgilbert@redhat.com>
Signed-off-by: NJuan Quintela <quintela@redhat.com>

Implement the core logic of the multiple thread compression. At this
point, multiple thread compression can't co-work with xbzrle yet.
Signed-off-by: NLiang Li <liang.z.li@intel.com>
Signed-off-by: NYang Zhang <yang.z.zhang@intel.com>
Signed-off-by: NJuan Quintela <quintela@redhat.com>

Split the function save_zero_page from ram_save_page so that we can
reuse it later.
Signed-off-by: NLiang Li <liang.z.li@intel.com>
Signed-off-by: NYang Zhang <yang.z.zhang@intel.com>
Reviewed-by: NJuan Quintela <quintela@redhat.com>
Signed-off-by: NJuan Quintela <quintela@redhat.com>

Define the data structure and variables used to do multiple thread
decompression, and add the code to initialize and free them.
Signed-off-by: NLiang Li <liang.z.li@intel.com>
Signed-off-by: NYang Zhang <yang.z.zhang@intel.com>
Reviewed-by: NDr.David Alan Gilbert <dgilbert@redhat.com>
Reviewed-by: NJuan Quintela <quintela@redhat.com>
Signed-off-by: NJuan Quintela <quintela@redhat.com>

Define the data structure and variables used to do multiple thread
compression, and add the code to initialize and free them.
Signed-off-by: NLiang Li <liang.z.li@intel.com>
Signed-off-by: NYang Zhang <yang.z.zhang@intel.com>
Reviewed-by: NDr.David Alan Gilbert <dgilbert@redhat.com>
Reviewed-by: NJuan Quintela <quintela@redhat.com>
Signed-off-by: NJuan Quintela <quintela@redhat.com>

qemu_put_compression_data() compress the data and put it to QEMUFile.
qemu_put_qemu_file() put the data in the buffer of source QEMUFile to
destination QEMUFile.
Signed-off-by: NLiang Li <liang.z.li@intel.com>
Signed-off-by: NYang Zhang <yang.z.zhang@intel.com>
Reviewed-by: NJuan Quintela <quintela@redhat.com>
Signed-off-by: NJuan Quintela <quintela@redhat.com>

Add the code to create and destroy the multiple threads those will be
used to do data decompression. Left some functions empty just to keep
clearness, and the code will be added later.
Signed-off-by: NLiang Li <liang.z.li@intel.com>
Signed-off-by: NYang Zhang <yang.z.zhang@intel.com>
Reviewed-by: NDr.David Alan Gilbert <dgilbert@redhat.com>
Reviewed-by: NJuan Quintela <quintela@redhat.com>
Signed-off-by: NJuan Quintela <quintela@redhat.com>

Add the code to create and destroy the multiple threads those will
be used to do data compression. Left some functions empty to keep
clearness, and the code will be added later.
Signed-off-by: NLiang Li <liang.z.li@intel.com>
Signed-off-by: NYang Zhang <yang.z.zhang@intel.com>
Reviewed-by: NDr.David Alan Gilbert <dgilbert@redhat.com>
Reviewed-by: NJuan Quintela <quintela@redhat.com>
Signed-off-by: NJuan Quintela <quintela@redhat.com>

Give some details about the multiple thread (de)compression and
how to use it in live migration.
Signed-off-by: NLiang Li <liang.z.li@intel.com>
Signed-off-by: NYang Zhang <yang.z.zhang@intel.com>
Reviewed-by: NDr.David Alan Gilbert <dgilbert@redhat.com>
Reviewed-by: NJuan Quintela <quintela@redhat.com>
Signed-off-by: NJuan Quintela <quintela@redhat.com>

size reduction merge

# gpg: Signature made Wed May  6 00:21:43 2015 BST using RSA key ID 4DD0279B
# gpg: Good signature from "Richard Henderson <rth7680@gmail.com>"
# gpg:                 aka "Richard Henderson <rth@redhat.com>"
# gpg:                 aka "Richard Henderson <rth@twiddle.net>"

* remotes/rth/tags/tcg-next-20150505:
  tcg: optimise memory layout of TCGTemp
Signed-off-by: NPeter Maydell <peter.maydell@linaro.org>

drop qapi nested structs

# gpg: Signature made Tue May  5 17:40:40 2015 BST using RSA key ID EB918653
# gpg: Good signature from "Markus Armbruster <armbru@redhat.com>"
# gpg:                 aka "Markus Armbruster <armbru@pond.sub.org>"

* remotes/armbru/tags/pull-qmp-2015-05-05: (40 commits)
  qapi: Check for member name conflicts with a base class
  qapi: Support (subset of) \u escapes in strings
  qapi: Tweak doc references to QMP when QGA is also meant
  qapi: Drop dead visitor code related to nested structs
  qapi: Drop support for inline nested types
  qapi: Drop inline nested structs in query-pci
  qapi: Drop inline nested struct in query-version
  qapi: Drop tests for inline nested structs
  qapi: Merge UserDefTwo and UserDefNested in tests
  qapi: Forbid 'type' in schema
  qapi: Use 'struct' instead of 'type' in schema
  qapi: Document 'struct' metatype
  qapi: Prefer 'struct' over 'type' in generator
  qapi: More rigorous checking for type safety bypass
  qapi: Whitelist commands that don't return dictionary
  qapi: Require valid names
  qapi: More rigourous checking of types
  qapi: Add some type check tests
  qapi: Unify type bypass and add tests
  qapi: Allow true, false and null in schema json
  ...
Signed-off-by: NPeter Maydell <peter.maydell@linaro.org>

Our type inheritance for both 'struct' and for flat 'union' merges
key/value pairs from the base class with those from the type in
question.  Although the C code currently boxes things so that there
is a distinction between which member is referred to, the QMP wire
format does not allow passing a key more than once in a single
object.  Besides, if we ever change the generated C code to not be
quite so boxy, we'd want to avoid duplicate member names there,
too.

Fix a testsuite entry added in an earlier patch, as well as adding
a couple more tests to ensure we have appropriate coverage.  Ensure
that collisions are detected, regardless of whether there is a
difference in opinion on whether the member name is optional.
Signed-off-by: NEric Blake <eblake@redhat.com>
Reviewed-by: NMarkus Armbruster <armbru@redhat.com>
Signed-off-by: NMarkus Armbruster <armbru@redhat.com>

The handling of \ inside QAPI strings was less than ideal, and
really only worked JSON's \/, \\, \", and our extension of \'
(an obvious extension, when you realize we use '' instead of ""
for strings).  For other things, like '\n', it resulted in a
literal 'n' instead of a newline.

Of course, at the moment, we really have no use for escaped
characters, as QAPI has to map to C identifiers, and we currently
support ASCII only for that.  But down the road, we may add
support for default values for string parameters to a command
or struct; if that happens, it would be nice to correctly support
all JSON escape sequences, such as \n or \uXXXX.  This gets us
closer, by supporting Unicode escapes in the ASCII range.

Since JSON does not require \OCTAL or \xXX escapes, and our QMP
implementation does not understand them either, I intentionally
reject it here, but it would be an easy addition if we desired it.
Likewise, intentionally refusing the NUL byte means we don't have
to worry about C strings being shorter than the qapi input.
Signed-off-by: NEric Blake <eblake@redhat.com>
Reviewed-by: NMarkus Armbruster <armbru@redhat.com>
Signed-off-by: NMarkus Armbruster <armbru@redhat.com>

We have more than one qapi schema in use by more than one protocol.
Add a new term 'Client JSON Protocol' for use throughout the
document, to avoid confusion on whether something refers only to
QMP and not QGA.
Signed-off-by: NEric Blake <eblake@redhat.com>
Reviewed-by: NMarkus Armbruster <armbru@redhat.com>
Signed-off-by: NMarkus Armbruster <armbru@redhat.com>

Now that we no longer have nested structs to visit, the use of
prefix strings is no longer required.  Remove the code that is
no longer reachable.
Signed-off-by: NEric Blake <eblake@redhat.com>
Reviewed-by: NMarkus Armbruster <armbru@redhat.com>
Signed-off-by: NMarkus Armbruster <armbru@redhat.com>

A future patch will be using a 'name':{dictionary} entry in the
QAPI schema to specify a default value for an optional argument
(see previous commit messages for more details why); but existing
use of inline nested structs conflicts with that goal. Now that
all commands have been changed to avoid inline nested structs,
nuke support for them, and turn it into a hard error. Update the
testsuite to reflect tighter parsing rules.
Signed-off-by: NEric Blake <eblake@redhat.com>
Reviewed-by: NMarkus Armbruster <armbru@redhat.com>
Signed-off-by: NMarkus Armbruster <armbru@redhat.com>

A future patch will be using a 'name':{dictionary} entry in the
QAPI schema to specify a default value for an optional argument
(see previous commit message for more details why); but existing
use of inline nested structs conflicts with that goal. This patch
fixes one of only two commands relying on nested types, by
breaking the nesting into an explicit type; it means that the
type is now boxed instead of unboxed in C code, but the QMP wire
format is unaffected by this change.

Prefer the safer g_new0() while making the conversion, and reduce
some long lines.
Signed-off-by: NEric Blake <eblake@redhat.com>
Reviewed-by: NMarkus Armbruster <armbru@redhat.com>
Signed-off-by: NMarkus Armbruster <armbru@redhat.com>

A future patch will be using a 'name':{dictionary} entry in the
QAPI schema to specify a default value for an optional argument
(see previous commit message for more details why); but existing
use of inline nested structs conflicts with that goal. This patch
fixes one of only two commands relying on nested types, by
breaking the nesting into an explicit type; it means that the
type is now boxed instead of unboxed in C code, but the QMP wire
format is unaffected by this change.

Prefer the safer g_new0() while making the conversion.
Signed-off-by: NEric Blake <eblake@redhat.com>
Reviewed-by: NMarkus Armbruster <armbru@redhat.com>
Signed-off-by: NMarkus Armbruster <armbru@redhat.com>

A future patch will be using a 'name':{dictionary} entry in the
QAPI schema to specify a default value for an optional argument;
but existing use of inline nested structs conflicts with that goal.

More precisely, a definition in the QAPI schema associates a name
with a set of properties:

Example 1: { 'struct': 'Foo', 'data': { MEMBERS... } }
associates the global name 'Foo' with properties (meta-type struct)
and MEMBERS...

Example 2: 'mumble': TYPE
within MEMBERS... above associates 'mumble' with properties (type
TYPE) and (optional false) within type Foo

The syntax of example 1 is extensible; if we need another property,
we add another name/value pair to the dictionary (such as
'base':TYPE).  The syntax of example 2 is not extensible, because
the right hand side can only be a type.

We have used name encoding to add a property: "'*mumble': 'int'"
associates 'mumble' with (type int) and (optional true).  Nice,
but doesn't scale.  So the solution is to change our existing uses
to be syntactic sugar to an extensible form:

   NAME: TYPE   --> NAME:  { 'type': TYPE, 'optional': false }
   *ONAME: TYPE --> ONAME: { 'type': TYPE, 'optional': true }

This patch fixes the testsuite to avoid inline nested types, by
breaking the nesting into explicit types; it means that the type
is now boxed instead of unboxed in C code, but makes no difference
on the wire (and if desired, a later patch could change the
generator to not do so much boxing in C).  When touching code to
add new allocations, also convert existing allocations to
consistently prefer typesafe g_new0 over g_malloc0 when a type
name is involved.
Signed-off-by: NEric Blake <eblake@redhat.com>
Reviewed-by: NMarkus Armbruster <armbru@redhat.com>
Signed-off-by: NMarkus Armbruster <armbru@redhat.com>

In the testsuite, UserDefTwo and UserDefNested were identical
structs other than the member names.  Reduce code duplication by
having just one type, and choose names that also favor reuse.
This will also make it easier for a later patch to get rid of
inline nested types in QAPI.  When touching code related to
allocations, convert g_malloc0(sizeof(Type)) to the more typesafe
g_new0(Type, 1).

Ensure that 'make check-qapi-schema check-unit' still passes.
Signed-off-by: NEric Blake <eblake@redhat.com>
Reviewed-by: NMarkus Armbruster <armbru@redhat.com>
Signed-off-by: NMarkus Armbruster <armbru@redhat.com>

Referring to "type" as both a meta-type (built-in, enum, union,
alternate, or struct) and a specific type (the name that the
schema uses for declaring structs) is confusing.  Finish up the
conversion to using "struct" in qapi schema by removing the hack
in the generator that allowed 'type'.
Signed-off-by: NEric Blake <eblake@redhat.com>
Reviewed-by: NMarkus Armbruster <armbru@redhat.com>
Signed-off-by: NMarkus Armbruster <armbru@redhat.com>

Referring to "type" as both a meta-type (built-in, enum, union,
alternate, or struct) and a specific type (the name that the
schema uses for declaring structs) is confusing.  Do the bulk of
the conversion to "struct" in qapi schema, with a fairly
mechanical:

for f in `find -name '*.json'; do sed -i "s/'type'/'struct'/"; done

followed by manually filtering out the places where we have a
'type' embedded in 'data'.  Then tweak a couple of tests whose
output changes slightly due to longer lines.

I also verified that the generated files for QMP and QGA (such
as qmp-commands.h) are the same before and after, as assurance
that I didn't leave in any accidental member name changes.
Signed-off-by: NEric Blake <eblake@redhat.com>
Reviewed-by: NMarkus Armbruster <armbru@redhat.com>
Signed-off-by: NMarkus Armbruster <armbru@redhat.com>