qapi: Another round of fixes and cleanups

# gpg: Signature made Fri 04 Sep 2015 14:48:54 BST using RSA key ID EB918653
# gpg: Good signature from "Markus Armbruster <armbru@redhat.com>"
# gpg:                 aka "Markus Armbruster <armbru@pond.sub.org>"

* remotes/armbru/tags/pull-qapi-2015-09-04: (33 commits)
  qapi: Generators crash when --output-dir isn't given, fix
  docs/qapi-code-gen.txt: Fix QAPI schema examples
  qapi: Simplify error reporting for array types
  qapi: Fix errors for non-string, non-dictionary members
  tests/qapi-schema: Cover non-string, non-dictionary members
  tests/qapi-schema: Cover two more syntax errors
  qapi: Drop one of two "simple union must not have base" checks
  qapi: Generated code cleanup
  qapi-commands: Drop useless initialization
  qapi-commands: Don't feed output of mcgen() to mcgen() again
  qapi-commands: Inline gen_marshal_output_call()
  qapi-commands: Fix gen_err_check(e) for e and e != 'local_err'
  qapi: Command returning anonymous type doesn't work, outlaw
  qapi: Fix to reject union command and event arguments
  qapi-tests: New tests for union, alternate command arguments
  tests/qapi-schema: Rename tests from data- to args-
  tests/qapi-schema: Restore test case for flat union base bug
  qapi: Document flaws in checking of names
  qapi: Document shortcoming with union 'data' branch
  qapi: Document that input visitor semantics are prone to leaks
  ...
Signed-off-by: NPeter Maydell <peter.maydell@linaro.org>

Signed-off-by: NMarkus Armbruster <armbru@redhat.com>
Reviewed-by: NEric Blake <eblake@redhat.com>

Signed-off-by: NMarkus Armbruster <armbru@redhat.com>
Reviewed-by: NEric Blake <eblake@redhat.com>

check_type() first checks and peels off the array type, then checks
the element type.  For two out of four error messages, it takes pains
to report errors for "array of T" instead of just T.  Odd.  Let's
examine the errors.

* Unknown element type, e.g.
  tests/qapi-schema/args-array-unknown.json:

      Member 'array' of 'data' for command 'oops' uses unknown type
      'array of NoSuchType'

  To make sense of this, you need to know that 'array of NoSuchType'
  refers to '[NoSuchType]'.  Easy enough.  However, simply reporting

      Member 'array' of 'data' for command 'oops' uses unknown type
      'NoSuchType'

  is at least as easy to understand.

* Element type's meta-type is inadmissible, e.g.
  tests/qapi-schema/returns-whitelist.json:

      'returns' for command 'no-way-this-will-get-whitelisted' cannot
      use built-in type 'array of int'

  'array of int' is technically not a built-in type, but that's
  pedantry.  However, simply reporting

      'returns' for command 'no-way-this-will-get-whitelisted' cannot
      use built-in type 'int'

  avoids the issue, and is at least as easy to understand.

* The remaining two errors are unreachable, because the array checking
  ensures that value is a string.

Thus, reporting some errors for "array of T" instead of just T works,
but doesn't really improve things.  Drop it.
Signed-off-by: NMarkus Armbruster <armbru@redhat.com>
Reviewed-by: NEric Blake <eblake@redhat.com>

Fixes the errors demonstrated by the previous commit.
Signed-off-by: NMarkus Armbruster <armbru@redhat.com>
Reviewed-by: NEric Blake <eblake@redhat.com>

We always report "should be a dictionary" then.  This is misleading:
when allow_dict, it can be a dictionary or a type name string, else it
can only be a type name.
Signed-off-by: NMarkus Armbruster <armbru@redhat.com>
Reviewed-by: NEric Blake <eblake@redhat.com>

Syntax error coverage should now be complete.
Signed-off-by: NMarkus Armbruster <armbru@redhat.com>
Reviewed-by: NEric Blake <eblake@redhat.com>

The first check ensures the second one can't trigger.  Drop the first
one, because the second one is in a more logical place, and emits a
nicer error message.
Signed-off-by: NMarkus Armbruster <armbru@redhat.com>
Reviewed-by: NEric Blake <eblake@redhat.com>

Clean up white-space, brace placement, and superfluous #ifdef
QAPI_TYPES_BUILTIN_CLEANUP_DEF.
Signed-off-by: NMarkus Armbruster <armbru@redhat.com>
Reviewed-by: NEric Blake <eblake@redhat.com>

In generated command handlers, the assignment to retval dominates its
only use.  Therefore, its initialization is useless.  Drop it.
Suggested-by: NEric Blake <eblake@redhat.com>
Signed-off-by: NMarkus Armbruster <armbru@redhat.com>
Reviewed-by: NEric Blake <eblake@redhat.com>

Multiple passes through mcgen() is prone to produce unwanted blank
lines, which we then combat by sprinkling .rstrip() on top.  Just
don't do it.
Signed-off-by: NMarkus Armbruster <armbru@redhat.com>
Reviewed-by: NEric Blake <eblake@redhat.com>

Signed-off-by: NMarkus Armbruster <armbru@redhat.com>
Reviewed-by: NEric Blake <eblake@redhat.com>

gen_err_check() hard-codes 'local_err' instead of substituting the
argument.  Currently harmless, since all callers pass either None or
'local_err'.
Signed-off-by: NMarkus Armbruster <armbru@redhat.com>
Reviewed-by: NEric Blake <eblake@redhat.com>

Reproducer: with

    { 'command': 'user_def_cmd4', 'returns': { 'a': 'int' } }

added to qapi-schema-test.json, qapi-commands.py dies when it tries to
generate the command handler function

    Traceback (most recent call last):
      File "/work/armbru/qemu/scripts/qapi-commands.py", line 359, in <module>
        ret = generate_command_decl(cmd['command'], arglist, ret_type) + "\n"
      File "/work/armbru/qemu/scripts/qapi-commands.py", line 29, in generate_command_decl
        ret_type=c_type(ret_type), name=c_name(name),
      File "/work/armbru/qemu/scripts/qapi.py", line 927, in c_type
        assert isinstance(value, str) and value != ""
    AssertionError

because the return type doesn't exist.

Simply outlaw this usage, and drop or dumb down test cases accordingly.
Signed-off-by: NMarkus Armbruster <armbru@redhat.com>
Reviewed-by: NEric Blake <eblake@redhat.com>

A command's or event's 'data' must be a struct type, given either as a
dictionary, or as struct type name.

Commit dd883c6f tightened the checking there, but not enough: we still
accept 'union'.  Fix to reject it.

We may want to support union types there, but we'll have to extend
qapi-commands.py and qapi-events.py for it.
Signed-off-by: NMarkus Armbruster <armbru@redhat.com>
Reviewed-by: NEric Blake <eblake@redhat.com>

A command's 'data' must be a struct type, given either as a
dictionary, or as struct type name.

Existing test case data-int.json covers simple type 'int'.  Add test
cases for type names referring to union and alternate types.

The latter is caught (good), but the former is not (bug).

Events have the same problem, but since they get checked by the same
code, we don't bother to duplicate the tests.
Signed-off-by: NMarkus Armbruster <armbru@redhat.com>
Reviewed-by: NEric Blake <eblake@redhat.com>

Since every schema entity has 'data', the data- prefix conveys no
information.  These tests actually exercise commands.  Only commands
have arguments, so change the prefix to to args-.
Signed-off-by: NMarkus Armbruster <armbru@redhat.com>
Reviewed-by: NEric Blake <eblake@redhat.com>

Test case added in commit 2fc00432, and messed up in commit 5223070c.
Signed-off-by: NMarkus Armbruster <armbru@redhat.com>
Reviewed-by: NEric Blake <eblake@redhat.com>

We don't actually enforce our "other than downstream extensions [...],
all names should begin with a letter" rule.  Add a FIXME.

We should reject names that differ only in '_' vs. '.'  vs. '-',
because they're liable to clash in generated C.  Add a FIXME.
Signed-off-by: NMarkus Armbruster <armbru@redhat.com>
Reviewed-by: NEric Blake <eblake@redhat.com>

Add a FIXME to remind us to fully audit whether removing the
'void *data' branch of each qapi union type can be done safely.
Signed-off-by: NEric Blake <eblake@redhat.com>
Message-Id: <1438297637-26789-1-git-send-email-eblake@redhat.com>
Signed-off-by: NMarkus Armbruster <armbru@redhat.com>

Most functions that can return a pointer or set an Error ** value
are decent enough to guarantee a NULL return when reporting an error.
Not so with our generated qapi visitor functions.  If the caller
is not careful to clean up partially-allocated objects on error,
then the caller suffers a memory leak.

Properly fixing it is probably complex enough to save for a later
day, so merely document it for now.
Signed-off-by: NEric Blake <eblake@redhat.com>
Message-Id: <1438295587-19069-1-git-send-email-eblake@redhat.com>
Signed-off-by: NMarkus Armbruster <armbru@redhat.com>

When event FOO's 'data' is a struct with a base, we consider only the
struct's direct members, and ignore its base.  The generated
qapi_event_send_foo() doesn't take arguments for base members.

No such events currently exist in the QMP schema.
Signed-off-by: NMarkus Armbruster <armbru@redhat.com>
Reviewed-by: NEric Blake <eblake@redhat.com>

We generate a declaration, but no definition.

The QMP schema has two: Qcow2OverlapChecks and BlockdevRef.  Neither
visit_type_Qcow2OverlapChecksKind() nor visit_type_BlockdevRefKind()
is actually used.
Signed-off-by: NMarkus Armbruster <armbru@redhat.com>
Reviewed-by: NEric Blake <eblake@redhat.com>

The generated code passes mangled schema names to visit_type_enum()
and union's visit_start_struct().  Fix it to pass the names
unadulterated, like we do everywhere else.

Only qapi-schema-test.json actually has names where this makes a
difference: enum __org.qemu_x-Enum, flat union __org.qemu_x-Union2,
simple union __org.qemu_x-Union1 and its implicit enum
__org.qemu_x-Union1Kind.
Signed-off-by: NMarkus Armbruster <armbru@redhat.com>
Reviewed-by: NEric Blake <eblake@redhat.com>

Use set because that's what it is.  While there, rename to
implicit_structs_seen.
Signed-off-by: NMarkus Armbruster <armbru@redhat.com>
Reviewed-by: NEric Blake <eblake@redhat.com>

The visit_type_implicit_FOO() are generated on demand, right before
their first use.  Used by visit_type_STRUCT_fields() when STRUCT has
base FOO, and by visit_type_UNION() when flat UNION has member a FOO.

If the schema defines FOO after its first use as struct base or flat
union member, visit_type_implicit_FOO() calls
visit_type_implicit_FOO() before its definition, which doesn't
compile.

Rearrange qapi-schema-test.json to demonstrate the bug.

Fix by generating the necessary forward declaration.
Signed-off-by: NMarkus Armbruster <armbru@redhat.com>
Reviewed-by: NEric Blake <eblake@redhat.com>

The struct generated for a flat union is weird: the members of its
base are at the end, except for the union tag, which is at the
beginning.

Example: qapi-schema-test.json has

    { 'struct': 'UserDefUnionBase',
      'data': { 'string': 'str', 'enum1': 'EnumOne' } }

    { 'union': 'UserDefFlatUnion',
      'base': 'UserDefUnionBase',
      'discriminator': 'enum1',
      'data': { 'value1' : 'UserDefA',
                'value2' : 'UserDefB',
                'value3' : 'UserDefB' } }

We generate:

    struct UserDefFlatUnion
    {
        EnumOne enum1;
        union {
            void *data;
            UserDefA *value1;
            UserDefB *value2;
            UserDefB *value3;
        };
        char *string;
    };

Change to put all base members at the beginning, unadulterated.  Not
only is this easier to understand, it also permits casting the flat
union to its base, if that should become useful.

We now generate:

    struct UserDefFlatUnion
    {
        /* Members inherited from UserDefUnionBase: */
        char *string;
        EnumOne enum1;
        /* Own members: */
        union { /* union tag is @enum1 */
            void *data;
            UserDefA *value1;
            UserDefB *value2;
            UserDefB *value3;
        };
    };
Signed-off-by: NMarkus Armbruster <armbru@redhat.com>
Reviewed-by: NEric Blake <eblake@redhat.com>

A flat union's tag member gets renamed to 'kind' in the generated
code.  Breaks when another member named 'kind' exists.

Example, adapted from qapi-schema-test.json:

    { 'struct': 'UserDefUnionBase',
      'data': { 'kind': 'str', 'enum1': 'EnumOne' } }

We generate:

    struct UserDefFlatUnion
    {
        EnumOne kind;
        union {
            void *data;
            UserDefA *value1;
            UserDefB *value2;
            UserDefB *value3;
        };
        char *kind;
    };

Kill the silly rename.
Reported-by: NEric Blake <eblake@redhat.com>
Signed-off-by: NMarkus Armbruster <armbru@redhat.com>
Reviewed-by: NEric Blake <eblake@redhat.com>

gen_sync_call()'s parameter indent is useless: gen_sync_call() uses it
only as optional argument for push_indent() and pop_indent(), their
default is four, and gen_sync_call()'s only caller passes four.  Drop
the parameter.

gen_visitor_input_containers_decl()'s parameter obj is always
"QOBJECT(args)".  Use that, and drop the parameter.

Drop unused parameters of gen_marshal_output(),
gen_marshal_input_decl(), generate_visit_struct_body(),
generate_visit_list(), generate_visit_enum(), generate_declaration(),
generate_enum_declaration(), generate_decl_enum().

Drop unused variables in generate_event_enum_lookup(),
generate_enum_lookup(), generate_visit_struct_fields(), check_event().
Signed-off-by: NMarkus Armbruster <armbru@redhat.com>
Reviewed-by: NEric Blake <eblake@redhat.com>

qapi-event.py breaks when you ask for a funny prefix like '@'.
Protect it.
Signed-off-by: NMarkus Armbruster <armbru@redhat.com>
Reviewed-by: NEric Blake <eblake@redhat.com>

Use c_name() instead of ad hoc code.  Doesn't upcase the -p prefix,
which is an improvement in my book.  Unbreaks prefix containing '.',
but other funny characters remain broken.  To be fixed next.
Signed-off-by: NMarkus Armbruster <armbru@redhat.com>
Reviewed-by: NEric Blake <eblake@redhat.com>

The guards around built-in declarations lose their _H.  It never made
much sense anyway.
Signed-off-by: NMarkus Armbruster <armbru@redhat.com>
Reviewed-by: NEric Blake <eblake@redhat.com>

Commit 05dfb26c added eatspace stripping to mcgen().  Move it to
cgen(), just in case somebody gets tempted to use cgen() directly
instead of via mcgen().

cgen() indents blank lines.  No such lines get generated right now,
but fix it anyway.

We use triple-quoted strings for program text, like this:

    '''
    Program text
    any number of lines
    '''

Keeps the program text relatively readable, but puts an extra newline
at either end.  mcgen() "fixes" that by dropping the first and last
line outright.  Drop only the newlines.

This unmasks a bug in qapi-commands.py: four quotes instead of three.
Fix it up.

Output doesn't change
Signed-off-by: NMarkus Armbruster <armbru@redhat.com>
Reviewed-by: NEric Blake <eblake@redhat.com>

It's idempotent.

While there, update examples to current code.
Signed-off-by: NMarkus Armbruster <armbru@redhat.com>
Reviewed-by: NEric Blake <eblake@redhat.com>

# gpg: Signature made Thu 03 Sep 2015 15:46:52 BST using RSA key ID 81AB73C8
# gpg: Good signature from "Stefan Hajnoczi <stefanha@redhat.com>"
# gpg:                 aka "Stefan Hajnoczi <stefanha@gmail.com>"

* remotes/stefanha/tags/tracing-pull-request:
  trace-events: Add hmp completion
Signed-off-by: NPeter Maydell <peter.maydell@linaro.org>

Add completion for the trace event names in the hmp trace-event
command.
Signed-off-by: NDr. David Alan Gilbert <dgilbert@redhat.com>
Message-id: 1439548063-18410-1-git-send-email-dgilbert@redhat.com
Signed-off-by: NStefan Hajnoczi <stefanha@redhat.com>

First batch of s390x patches for 2.5:
- introduce 2.5 compat machine
- support for migration of storage keys

# gpg: Signature made Thu 03 Sep 2015 11:28:06 BST using RSA key ID C6F02FAF
# gpg: Good signature from "Cornelia Huck <huckc@linux.vnet.ibm.com>"
# gpg:                 aka "Cornelia Huck <cornelia.huck@de.ibm.com>"

* remotes/cohuck/tags/s390x-20150903:
  s390x: Disable storage key migration on old machine type
  s390x: Migrate guest storage keys (initial memory only)
  s390x: Info skeys sub-command
  s390x: Dump-skeys hmp support
  s390x: Dump storage keys qmp command
  s390x: Enable new s390-storage-keys device
  s390x: Create QOM device for s390 storage keys
  s390x: add 2.5 compat s390-ccw-virtio machine
Signed-off-by: NPeter Maydell <peter.maydell@linaro.org>

queued tcg patches

# gpg: Signature made Wed 02 Sep 2015 22:35:37 BST using RSA key ID 4DD0279B
# gpg: Good signature from "Richard Henderson <rth7680@gmail.com>"
# gpg:                 aka "Richard Henderson <rth@redhat.com>"
# gpg:                 aka "Richard Henderson <rth@twiddle.net>"

* remotes/rth/tags/pull-tcg-20150902:
  tcg/i386: omit a few REXW prefixes in softmmu code
  tcg/aarch64: Fix tcg_out_qemu_{ld, st} for guest_base == 0
Signed-off-by: NPeter Maydell <peter.maydell@linaro.org>

cmpbge emulation improvements

# gpg: Signature made Wed 02 Sep 2015 20:25:10 BST using RSA key ID 4DD0279B
# gpg: Good signature from "Richard Henderson <rth7680@gmail.com>"
# gpg:                 aka "Richard Henderson <rth@redhat.com>"
# gpg:                 aka "Richard Henderson <rth@twiddle.net>"

* remotes/rth/tags/pull-axp-20150902:
  target-alpha: Special case cmpbge with zero
  target-alpha: Rewrite helper_cmpbge using bit tests
Signed-off-by: NPeter Maydell <peter.maydell@linaro.org>

This code disables storage key migration when an older machine type is
specified.
Reviewed-by: NDavid Hildenbrand <dahi@linux.vnet.ibm.com>
Signed-off-by: NJason J. Herne <jjherne@linux.vnet.ibm.com>
Signed-off-by: NCornelia Huck <cornelia.huck@de.ibm.com>