New methods c_name(), c_type(), c_null(), json_type(),
alternate_qtype().
Signed-off-by: NMarkus Armbruster <armbru@redhat.com>
Message-Id: <1442401589-24189-4-git-send-email-armbru@redhat.com>
Reviewed-by: NDaniel P. Berrange <berrange@redhat.com>
Reviewed-by: NEric Blake <eblake@redhat.com>

The QAPI code generators work with a syntax tree (nested dictionaries)
plus a few symbol tables (also dictionaries) on the side.

They have clearly outgrown these simple data structures.  There's lots
of rummaging around in dictionaries, and information is recomputed on
the fly.  For the work I'm going to do, I want more clearly defined
and more convenient interfaces.

Going forward, I also want less coupling between the back-ends and the
syntax tree, to make messing with the syntax easier.

Create a bunch of classes to represent QAPI schemata.

Have the QAPISchema initializer call the parser, then walk the syntax
tree to create the new internal representation, and finally perform
semantic analysis.

Shortcut: the semantic analysis still relies on existing check_exprs()
to do the actual semantic checking.  All this code needs to move into
the classes.  Mark as TODO.

Simple unions are lowered to flat unions.  Flat unions and structs are
represented as a more general object type.

Catching name collisions in generated code would be nice.  Mark as
TODO.

We generate array types eagerly, even though most of them aren't used.
Mark as TODO.

Nothing uses the new intermediate representation just yet, thus no
change to generated files.
Signed-off-by: NMarkus Armbruster <armbru@redhat.com>
Reviewed-by: NDaniel P. Berrange <berrange@redhat.com>
Reviewed-by: NEric Blake <eblake@redhat.com>

I want to name a new class QAPISchema.

While there, make it a new-style class.
Signed-off-by: NMarkus Armbruster <armbru@redhat.com>
Reviewed-by: NEric Blake <eblake@redhat.com>
Reviewed-by: NDaniel P. Berrange <berrange@redhat.com>
Message-Id: <1442401589-24189-2-git-send-email-armbru@redhat.com>

The camel_to_upper() method applies some heuristics to turn
a mixed case type name into an all-uppercase name. This is
used for example, to generate enum constant name prefixes.

The heuristics don't also generate a satisfactory name
though. eg

  { 'enum': 'QCryptoTLSCredsEndpoint',
    'data': ['client', 'server']}

Results in Q_CRYPTOTLS_CREDS_ENDPOINT_CLIENT. This has
an undesirable _ after the initial Q and is missing an
_ between the CRYPTO & TLS strings.

Rather than try to add more and more heuristics to try
to cope with this, simply allow the QAPI schema to
specify the desired enum constant prefix explicitly.

eg

  { 'enum': 'QCryptoTLSCredsEndpoint',
    'prefix': 'QCRYPTO_TLS_CREDS_ENDPOINT',
    'data': ['client', 'server']}

Now gives the QCRYPTO_TLS_CREDS_ENDPOINT_CLIENT name.
Signed-off-by: NDaniel P. Berrange <berrange@redhat.com>

A feature new in Python 2.7 crept into commit 77e703b8: re.subn()'s
fifth argument.  Avoid that, use re.compile().
Reported-by: NLaurent Desnogues <laurent.desnogues@gmail.com>
Signed-off-by: NMarkus Armbruster <armbru@redhat.com>
Reviewed-by: NEric Blake <eblake@redhat.com>
Tested-by: NLaurent Desnogues <laurent.desnogues@gmail.com>
Message-id: 1441640755-23902-1-git-send-email-armbru@redhat.com
Signed-off-by: NPeter Maydell <peter.maydell@linaro.org>

Signed-off-by: NMarkus Armbruster <armbru@redhat.com>
Reviewed-by: NEric Blake <eblake@redhat.com>

check_type() first checks and peels off the array type, then checks
the element type.  For two out of four error messages, it takes pains
to report errors for "array of T" instead of just T.  Odd.  Let's
examine the errors.

* Unknown element type, e.g.
  tests/qapi-schema/args-array-unknown.json:

      Member 'array' of 'data' for command 'oops' uses unknown type
      'array of NoSuchType'

  To make sense of this, you need to know that 'array of NoSuchType'
  refers to '[NoSuchType]'.  Easy enough.  However, simply reporting

      Member 'array' of 'data' for command 'oops' uses unknown type
      'NoSuchType'

  is at least as easy to understand.

* Element type's meta-type is inadmissible, e.g.
  tests/qapi-schema/returns-whitelist.json:

      'returns' for command 'no-way-this-will-get-whitelisted' cannot
      use built-in type 'array of int'

  'array of int' is technically not a built-in type, but that's
  pedantry.  However, simply reporting

      'returns' for command 'no-way-this-will-get-whitelisted' cannot
      use built-in type 'int'

  avoids the issue, and is at least as easy to understand.

* The remaining two errors are unreachable, because the array checking
  ensures that value is a string.

Thus, reporting some errors for "array of T" instead of just T works,
but doesn't really improve things.  Drop it.
Signed-off-by: NMarkus Armbruster <armbru@redhat.com>
Reviewed-by: NEric Blake <eblake@redhat.com>

Fixes the errors demonstrated by the previous commit.
Signed-off-by: NMarkus Armbruster <armbru@redhat.com>
Reviewed-by: NEric Blake <eblake@redhat.com>

The first check ensures the second one can't trigger.  Drop the first
one, because the second one is in a more logical place, and emits a
nicer error message.
Signed-off-by: NMarkus Armbruster <armbru@redhat.com>
Reviewed-by: NEric Blake <eblake@redhat.com>

Reproducer: with

    { 'command': 'user_def_cmd4', 'returns': { 'a': 'int' } }

added to qapi-schema-test.json, qapi-commands.py dies when it tries to
generate the command handler function

    Traceback (most recent call last):
      File "/work/armbru/qemu/scripts/qapi-commands.py", line 359, in <module>
        ret = generate_command_decl(cmd['command'], arglist, ret_type) + "\n"
      File "/work/armbru/qemu/scripts/qapi-commands.py", line 29, in generate_command_decl
        ret_type=c_type(ret_type), name=c_name(name),
      File "/work/armbru/qemu/scripts/qapi.py", line 927, in c_type
        assert isinstance(value, str) and value != ""
    AssertionError

because the return type doesn't exist.

Simply outlaw this usage, and drop or dumb down test cases accordingly.
Signed-off-by: NMarkus Armbruster <armbru@redhat.com>
Reviewed-by: NEric Blake <eblake@redhat.com>

A command's or event's 'data' must be a struct type, given either as a
dictionary, or as struct type name.

Commit dd883c6f tightened the checking there, but not enough: we still
accept 'union'.  Fix to reject it.

We may want to support union types there, but we'll have to extend
qapi-commands.py and qapi-events.py for it.
Signed-off-by: NMarkus Armbruster <armbru@redhat.com>
Reviewed-by: NEric Blake <eblake@redhat.com>

We don't actually enforce our "other than downstream extensions [...],
all names should begin with a letter" rule.  Add a FIXME.

We should reject names that differ only in '_' vs. '.'  vs. '-',
because they're liable to clash in generated C.  Add a FIXME.
Signed-off-by: NMarkus Armbruster <armbru@redhat.com>
Reviewed-by: NEric Blake <eblake@redhat.com>

gen_sync_call()'s parameter indent is useless: gen_sync_call() uses it
only as optional argument for push_indent() and pop_indent(), their
default is four, and gen_sync_call()'s only caller passes four.  Drop
the parameter.

gen_visitor_input_containers_decl()'s parameter obj is always
"QOBJECT(args)".  Use that, and drop the parameter.

Drop unused parameters of gen_marshal_output(),
gen_marshal_input_decl(), generate_visit_struct_body(),
generate_visit_list(), generate_visit_enum(), generate_declaration(),
generate_enum_declaration(), generate_decl_enum().

Drop unused variables in generate_event_enum_lookup(),
generate_enum_lookup(), generate_visit_struct_fields(), check_event().
Signed-off-by: NMarkus Armbruster <armbru@redhat.com>
Reviewed-by: NEric Blake <eblake@redhat.com>

qapi-event.py breaks when you ask for a funny prefix like '@'.
Protect it.
Signed-off-by: NMarkus Armbruster <armbru@redhat.com>
Reviewed-by: NEric Blake <eblake@redhat.com>

The guards around built-in declarations lose their _H.  It never made
much sense anyway.
Signed-off-by: NMarkus Armbruster <armbru@redhat.com>
Reviewed-by: NEric Blake <eblake@redhat.com>

Commit 05dfb26c added eatspace stripping to mcgen().  Move it to
cgen(), just in case somebody gets tempted to use cgen() directly
instead of via mcgen().

cgen() indents blank lines.  No such lines get generated right now,
but fix it anyway.

We use triple-quoted strings for program text, like this:

    '''
    Program text
    any number of lines
    '''

Keeps the program text relatively readable, but puts an extra newline
at either end.  mcgen() "fixes" that by dropping the first and last
line outright.  Drop only the newlines.

This unmasks a bug in qapi-commands.py: four quotes instead of three.
Fix it up.

Output doesn't change
Signed-off-by: NMarkus Armbruster <armbru@redhat.com>
Reviewed-by: NEric Blake <eblake@redhat.com>

Signed-off-by: NMarkus Armbruster <armbru@redhat.com>
Reviewed-by: NEric Blake <eblake@redhat.com>

Insert comments to separate sections dealing with parsing, semantic
analysis, code generation, and so forth.

Move helpers to their proper section.
Signed-off-by: NMarkus Armbruster <armbru@redhat.com>
Reviewed-by: NEric Blake <eblake@redhat.com>

To have expression semantic analysis in one place rather than two.
Signed-off-by: NMarkus Armbruster <armbru@redhat.com>
Reviewed-by: NEric Blake <eblake@redhat.com>

Screwed up in commit e53188ad.
Signed-off-by: NMarkus Armbruster <armbru@redhat.com>
Reviewed-by: NEric Blake <eblake@redhat.com>

We maintain a stack of filenames in include_hist for convenient cycle
detection.

As error_path() demonstrates, the same information is readily
available in the expr_info, so just use that, and drop include_hist.
Signed-off-by: NMarkus Armbruster <armbru@redhat.com>
Reviewed-by: NEric Blake <eblake@redhat.com>

We print the name as it appears in the include expression.  Tools
processing error messages want it relative to the working directory.
Make it so.
Signed-off-by: NMarkus Armbruster <armbru@redhat.com>
Reviewed-by: NEric Blake <eblake@redhat.com>

old name      new name
----------------------------
input_file    fname
input_relname fname
input_fname   abs_fname
include_path  incl_abs_fname
parent_info   incl_info
Signed-off-by: NMarkus Armbruster <armbru@redhat.com>
Reviewed-by: NEric Blake <eblake@redhat.com>

Signed-off-by: NMarkus Armbruster <armbru@redhat.com>
Reviewed-by: NEric Blake <eblake@redhat.com>

Signed-off-by: NMarkus Armbruster <armbru@redhat.com>
Reviewed-by: NEric Blake <eblake@redhat.com>

Signed-off-by: NMarkus Armbruster <armbru@redhat.com>
Reviewed-by: NEric Blake <eblake@redhat.com>

Mandatory option is silly, and the error handling is missing: the
programs crash when -i isn't supplied.  Make it an argument, and check
it properly.
Signed-off-by: NMarkus Armbruster <armbru@redhat.com>
Reviewed-by: NEric Blake <eblake@redhat.com>

Report to stderr, prefix with the program name.  Also reject
extra arguments.
Signed-off-by: NMarkus Armbruster <armbru@redhat.com>
Reviewed-by: NEric Blake <eblake@redhat.com>

Signed-off-by: NMarkus Armbruster <armbru@redhat.com>
Reviewed-by: NEric Blake <eblake@redhat.com>

Continuing the string of cleanups for supporting downstream names
containing '.', this patch focuses on ensuring c_type() can
handle a downstream name.  This patch alone does not fix the
places where generator output should be calling this function
but was open-coding things instead, but it gets us a step closer.

In particular, the changes to c_list_type() and type_name() mean
that type_name(FOO) now handles the case when FOO contains '.',
'-', or is a ticklish identifier other than a builtin (builtins
are exempted because ['int'] must remain mapped to 'intList' and
not 'q_intList').  Meanwhile, ['unix'] now maps to 'q_unixList'
rather than 'unixList', to match the fact that 'unix' is ticklish;
however, our naming conventions state that complex types should
start with a capital, so no type name following conventions will
ever have the 'q_' prepended.

Likewise, changes to c_type() mean that c_type(FOO) properly
handles an enum or complex type FOO with '.' or '-' in the
name, or is a ticklish identifier (again, a ticklish identifier
as a type name violates conventions).
Signed-off-by: NEric Blake <eblake@redhat.com>
Signed-off-by: NMarkus Armbruster <armbru@redhat.com>

c_type() is designed to be called on both string names and on
array designations, so 'name' is a bit misleading because it
operates on more than strings.  Also, no caller ever passes
an empty string.  Finally, + notation is a bit nicer to read
than '%s' % value for string concatenation.
Signed-off-by: NEric Blake <eblake@redhat.com>
Signed-off-by: NMarkus Armbruster <armbru@redhat.com>

Signed-off-by: NMarkus Armbruster <armbru@redhat.com>
Signed-off-by: NEric Blake <eblake@redhat.com>

Missed in commit b0b58195.
Signed-off-by: NMarkus Armbruster <armbru@redhat.com>
Signed-off-by: NEric Blake <eblake@redhat.com>

Signed-off-by: NMarkus Armbruster <armbru@redhat.com>
Signed-off-by: NEric Blake <eblake@redhat.com>

Signed-off-by: NMarkus Armbruster <armbru@redhat.com>
Signed-off-by: NEric Blake <eblake@redhat.com>

Signed-off-by: NMarkus Armbruster <armbru@redhat.com>
Signed-off-by: NEric Blake <eblake@redhat.com>

Now that the two functions are identical, we only need one of them,
and we might as well give it a more descriptive name.  Basically,
the function serves as the translation from a QAPI name into a
(portion of a) C identifier, without regards to whether it is a
variable or function name.
Signed-off-by: NEric Blake <eblake@redhat.com>
Signed-off-by: NMarkus Armbruster <armbru@redhat.com>

c_fun() maps '.' to '_', c_var() doesn't.  Nothing prevents '.' in
QAPI names that get passed to c_var().

Which QAPI names get passed to c_fun(), to c_var(), or to both is not
obvious.  Names of command parameters and struct type members get
passed to c_var().

c_var() strips a leading '*', but this cannot happen.  c_fun()
doesn't.

Fix c_var() to work exactly like c_fun().

Perhaps they should be replaced by a single mapping function.
Signed-off-by: NMarkus Armbruster <armbru@redhat.com>
[add 'import string']
Signed-off-by: NEric Blake <eblake@redhat.com>
Reviewed-by: NAlberto Garcia <berto@igalia.com>

Our type inheritance for both 'struct' and for flat 'union' merges
key/value pairs from the base class with those from the type in
question.  Although the C code currently boxes things so that there
is a distinction between which member is referred to, the QMP wire
format does not allow passing a key more than once in a single
object.  Besides, if we ever change the generated C code to not be
quite so boxy, we'd want to avoid duplicate member names there,
too.

Fix a testsuite entry added in an earlier patch, as well as adding
a couple more tests to ensure we have appropriate coverage.  Ensure
that collisions are detected, regardless of whether there is a
difference in opinion on whether the member name is optional.
Signed-off-by: NEric Blake <eblake@redhat.com>
Reviewed-by: NMarkus Armbruster <armbru@redhat.com>
Signed-off-by: NMarkus Armbruster <armbru@redhat.com>

The handling of \ inside QAPI strings was less than ideal, and
really only worked JSON's \/, \\, \", and our extension of \'
(an obvious extension, when you realize we use '' instead of ""
for strings).  For other things, like '\n', it resulted in a
literal 'n' instead of a newline.

Of course, at the moment, we really have no use for escaped
characters, as QAPI has to map to C identifiers, and we currently
support ASCII only for that.  But down the road, we may add
support for default values for string parameters to a command
or struct; if that happens, it would be nice to correctly support
all JSON escape sequences, such as \n or \uXXXX.  This gets us
closer, by supporting Unicode escapes in the ASCII range.

Since JSON does not require \OCTAL or \xXX escapes, and our QMP
implementation does not understand them either, I intentionally
reject it here, but it would be an easy addition if we desired it.
Likewise, intentionally refusing the NUL byte means we don't have
to worry about C strings being shorter than the qapi input.
Signed-off-by: NEric Blake <eblake@redhat.com>
Reviewed-by: NMarkus Armbruster <armbru@redhat.com>
Signed-off-by: NMarkus Armbruster <armbru@redhat.com>