As the name suggests, the qapi2texi script converts JSON QAPI
description into a texi file suitable for different target
formats (info/man/txt/pdf/html...).

It parses the following kind of blocks:

Free-form:

  ##
  # = Section
  # == Subsection
  #
  # Some text foo with *emphasis*
  # 1. with a list
  # 2. like that
  #
  # And some code:
  # | $ echo foo
  # | -> do this
  # | <- get that
  #
  ##

Symbol description:

  ##
  # @symbol:
  #
  # Symbol body ditto ergo sum. Foo bar
  # baz ding.
  #
  # @param1: the frob to frobnicate
  # @param2: #optional how hard to frobnicate
  #
  # Returns: the frobnicated frob.
  #          If frob isn't frobnicatable, GenericError.
  #
  # Since: version
  # Notes: notes, comments can have
  #        - itemized list
  #        - like this
  #
  # Example:
  #
  # -> { "execute": "quit" }
  # <- { "return": {} }
  #
  ##

That's roughly following the following EBNF grammar:

api_comment = "##\n" comment "##\n"
comment = freeform_comment | symbol_comment
freeform_comment = { "# " text "\n" | "#\n" }
symbol_comment = "# @" name ":\n" { member | tag_section | freeform_comment }
member = "# @" name ':' [ text ] "\n" freeform_comment
tag_section = "# " ( "Returns:", "Since:", "Note:", "Notes:", "Example:", "Examples:" ) [ text ]  "\n" freeform_comment
text = free text with markup

Note that the grammar is ambiguous: a line "# @foo:\n" can be parsed
both as freeform_comment and as symbol_comment.  The actual parser
recognizes symbol_comment.

See docs/qapi-code-gen.txt for more details.

Deficiencies and limitations:
- the generated QMP documentation includes internal types
- union type support is lacking
- type information is lacking in generated documentation
- doc comment error message positions are imprecise, they point
  to the beginning of the comment.
- a few minor issues, all marked TODO/FIXME in the code
Signed-off-by: NMarc-André Lureau <marcandre.lureau@redhat.com>
Message-Id: <20170113144135.5150-16-marcandre.lureau@redhat.com>
Reviewed-by: NMarkus Armbruster <armbru@redhat.com>
[test-qapi.py tweaked to avoid trailing empty lines in .out]
Signed-off-by: NMarkus Armbruster <armbru@redhat.com>

Referring to "type" as both a meta-type (built-in, enum, union,
alternate, or struct) and a specific type (the name that the
schema uses for declaring structs) is confusing.  The confusion
is only made worse by the fact that the generator mostly already
refers to struct even when dealing with expr['type'].  This
commit changes the generator to consistently refer to it as
struct everywhere, plus a single back-compat tweak that allows
accepting the existing .json files as-is, so that the meat of
this change is separate from the mindless churn of that change.

Fix the testsuite fallout for error messages that change, and
in some cases, become more legible.  Improve comments to better
match our intentions where a struct (rather than any complex
type) is required.  Note that in some cases, an error message
now refers to 'struct' while the schema still refers to 'type';
that will be cleaned up in the later commit to the schema.
Signed-off-by: NEric Blake <eblake@redhat.com>
Reviewed-by: NMarkus Armbruster <armbru@redhat.com>
Signed-off-by: NMarkus Armbruster <armbru@redhat.com>

The previous commit demonstrated that the generator overlooked some
fairly basic broken expressions:
- missing metataype
- metatype key has a non-string value
- unknown key in relation to the metatype
- conflicting metatype (this patch treats the second metatype as an
unknown key of the first key visited, which is not necessarily the
first key the user typed)

Add check_keys to cover these situations, and update testcases to
match.  A couple other tests (enum-missing-data, indented-expr) had
to change since the validation added here occurs so early.
Conversely, changes to ident-with-escape results show that we still
have problems where our handling of escape sequences differs from
true JSON, which will matter down the road if we allow arbitrary
default string values for optional parameters (but for now is not
too bad, as we currently can avoid unicode escaping as we don't
need to represent anything beyond C identifier material).

While valid .json files won't trigger any of these cases, we might
as well be nicer to developers that make a typo while trying to add
new QAPI code.
Signed-off-by: NEric Blake <eblake@redhat.com>
Reviewed-by: NMarkus Armbruster <armbru@redhat.com>
Signed-off-by: NMarkus Armbruster <armbru@redhat.com>

Demonstrate that the qapi generator doesn't deal well with
expressions that aren't up to par. Later patches will improve
the expected results as the generator is made stricter.  Only
a few of the the added tests actually behave sanely at
rejecting obvious problems or demonstrating success.

Note that in some cases, we reject bad QAPI merely because our
pseudo-JSON parser does not yet know how to parse numbers.  This
series does not address that, but when a later series adds support
for numeric defaults of integer fields, the testsuite will ensure
that we don't lose the error (and hopefully that the error
message quality is improved).
Signed-off-by: NEric Blake <eblake@redhat.com>
Reviewed-by: NMarkus Armbruster <armbru@redhat.com>
Signed-off-by: NMarkus Armbruster <armbru@redhat.com>

Add tcg and cpu model initialization.
Add gen_intermediate_code function.
Activate target in configure and add softmmu config.
Signed-off-by: NBastian Koppelmann <kbastian@mail.uni-paderborn.de>
Message-id: 1409572800-4116-5-git-send-email-kbastian@mail.uni-paderborn.de
Signed-off-by: NPeter Maydell <peter.maydell@linaro.org>

Changes v1 -> v2:
 - userconfig variable is now bool, not int
Signed-off-by: NEduardo Habkost <ehabkost@redhat.com>
Signed-off-by: NAnthony Liguori <aliguori@us.ibm.com>

This patch addes a Bulldozer-based Opteron_G4 CPU model.

This version has the ffxsr bit actually disabled, to match what was
documented below. Thanks to Andre Przywara for spotting the bug.

I am trying to be conservative with the new model, so I am enabling only
features known to be useful to guests, and not enabling anything that
was not tested or found to be useful to a guest.

List of missing flags in comparison to real hardware:

- vme: host-specific feature.
- osxsave: it is not set here because it is set by the guest OS, not by KVM
- monitor: this is filtered out by the KVM module, so no point in
  enabling it.
- mmxext: untested, so not enabled.
- Perf*, Topology*, lwp, ibs: not emulated by KVM.
- wdt, skinit, osvw, altmovcr8, extapicspace, cmplegacy: untested,
  so not enabled.

List of new flags, in comparison to the Opteron_G3 model:

- xsave: xsave feature, already implemented by Qemu
- avx, aes, sse4.x, ssse3, pclmulqdq: all new state the new instructions
  could use is handled by the xsave state loading/saving code on Qemu.
- pdpe1gb: 1GB pages, supported by the KVM kernel module.
- ffxsr: untested, so not enabled
- fma4, xop: all new state the new instructions could use is handled by
  the xsave loading/saving code on Qemu.
- 3dnowprefetch: safe to pass through, though the flag is not used by
  Linux guests, at least.

Below is the comparison between the current Opteron_G3 model
and the new model being added.

- The "full" line contains the flags found on actual hardware.
- The "missing" line shows the flags that are present on actual
  hardware, but not on the added Opteron_G4 model.
- The "new" line shows the flags that were not on the Opteron_G3 model
  but are on Opteron_G4.

feature_edx:
  Opteron_G3: sse2 sse fxsr mmx clflush pse36 pat cmov mca pge mtrr sep apic cx8 mce pae msr tsc pse de     fpu
  full:       sse2 sse fxsr mmx clflush pse36 pat cmov mca pge mtrr sep apic cx8 mce pae msr tsc pse de vme fpu
  Opteron_G4: sse2 sse fxsr mmx clflush pse36 pat cmov mca pge mtrr sep apic cx8 mce pae msr tsc pse de     fpu
  missing:                                                                                              vme

feature_ecx:
  Opteron_G3:                       popcnt               cx16       monitor           sse3
  full:       avx osxsave xsave aes popcnt sse4.2 sse4.1 cx16 ssse3 monitor pclmulqdq sse3
  Opteron_G4: avx         xsave aes popcnt sse4.2 sse4.1 cx16 ssse3         pclmulqdq sse3
  missing:        osxsave                                           monitor
  new:        avx         xsave aes        sse4.2 sse4.1      ssse3         pclmulqdq

extfeature_edx:
  Opteron_G3: lm rdtscp               fxsr mmx        nx pse36 pat cmov mca pge mtrr syscall apic cx8 mce pae msr tsc pse de     fpu
  full:       lm rdtscp pdpe1gb ffxsr fxsr mmx mmxext nx pse36 pat cmov mca pge mtrr syscall apic cx8 mce pae msr tsc pse de vme fpu
  Opteron_G4: lm rdtscp pdpe1gb       fxsr mmx        nx pse36 pat cmov mca pge mtrr syscall apic cx8 mce pae msr tsc pse de     fpu
  missing:                                     mmxext                                                                        vme
  new:                  pdpe1gb

extfeature_ecx:
  Opteron_G3:                                                                misalignsse sse4a abm                        svm           lahf_lm
  full:       Perf* Topology* fma4 lwp wdt skinit xop ibs osvw 3dnowprefetch misalignsse sse4a abm altmovcr8 extapicspace svm cmplegacy lahf_lm
  Opteron_G4:                 fma4                xop          3dnowprefetch misalignsse sse4a abm                        svm           lahf_lm
  new:                        fma4                xop          3dnowprefetch
  missing:    Perf* Topology*      lwp wdt skinit     ibs osvw                                     altmovcr8 extapicspace     cmplegacy

Changes v1 -> v2:
 - Actually disable ffxsr bit

Cc: Andre Przywara <andre.przywara@amd.com>
Signed-off-by: NEduardo Habkost <ehabkost@redhat.com>
Signed-off-by: NAnthony Liguori <aliguori@us.ibm.com>

This patches add the definition of a SandyBridge CPU model.

Summary of differences:

Flags present on actual hardware, but not on the added model definition:

- pbe, tm, ht, ss, acpi, vme, xTPR, tm2, eist, smx: host-specific
  features, not exposed to guest.
- ds, ds-cpl, dtes64, pdcm: emulation not supported by KVM (although it
  may be added in the future if implementing PMU virtualization)
- pcid, vmx, monitor: not emulated by Qemu/KVM right now.
- osxsave: set by the guest OS, not by Qemu.

Flags added, that were not present on Westmere model:

- xsave: already supported by Qemu
- avx, pclmulqdq: all new state the new instructions could use is
  handled by xsave state loading/saving code.
- tsc-deadline, x2apic, rdtscp: already supported by Qemu/KVM.

Below there's a comparison of the features on the current Westmere CPU
model, and the SandyBridge CPU model.

- The "full" line contains the flags found on actual hardware.
- The "missing" line shows the flags that are present on actual
  hardware, but not on the added SandyBridge model.
- The "new" line shows the flags that were not on the Westmere model,
  but are on SandyBridge.

feature_edx:
  Westmere:                 sse2 sse fxsr mmx         clflush pse36 pat cmov mca pge mtrr sep apic cx8 mce pae msr tsc pse de     fpu
  full:        pbe tm ht ss sse2 sse fxsr mmx ds acpi clflush pse36 pat cmov mca pge mtrr sep apic cx8 mce pge msr tsc pse de vme fpu
  SandyBridge:              sse2 sse fxsr mmx         clflush pse36 pat cmov mca pge mtrr sep apic cx8 mce pae msr tsc pse de     fpu
  missing:     pbe tm ht ss                   ds acpi                                                                         vme

feature_ecx:
  Westmere:                      aes              popcnt        sse4.2 sse4.1                cx16 ssse3                                                  sse3
  full:        avx osxsave xsave aes tsc-deadline popcnt x2apic sse4.2 sse4.1 pcid pdcm xTPR cx16 ssse3 tm2 eist smx vmx ds-cpl monitor dtes64 pclmulqdq sse3
  SandyBridge: avx         xsave aes tsc-deadline popcnt x2apic sse4.2 sse4.1                cx16 ssse3                                        pclmulqdq sse3
  missing:         osxsave                                                    pcid pdcm xTPR            tm2 eist smx vmx ds-cpl monitor dtes64
  new:         avx         xsave     tsc-deadline        x2apic                                                                                pclmulqdq

extfeature_edx:
  Westmere:    i64        nx syscall
  full:        i64 rdtscp nx syscall
  SandyBridge: i64 rdtscp nx syscall
  new:             rdtscp

extfeature_ecx:
  Westmere:    lahf_lm
  full:        lahf_lm
  SandyBridge: lahf_lm

Cc: "Dugger, Donald D" <donald.d.dugger@intel.com>
Cc: "Zhang, Xiantao" <xiantao.zhang@intel.com>
Acked-by: NXiantao Zhang <xiantao.zhang@intel.com>
Signed-off-by: NEduardo Habkost <ehabkost@redhat.com>
Signed-off-by: NAnthony Liguori <aliguori@us.ibm.com>

This should have no visible effect, but it should just clean up the
config file a bit.

This is based on a previous patch from John Cooper where this was introduced
with many other changes at the same time. Original John's patch submission is
at Message-ID: <4DDAD5E7.2020002@redhat.com>, <http://marc.info/?l=qemu-devel&m=130618871926030>.

Changes v1 -> v2:
 - Rebase against latest Qemu git tree
Signed-off-by: NEduardo Habkost <ehabkost@redhat.com>
Signed-off-by: NAnthony Liguori <aliguori@us.ibm.com>

Version 1 of this patch was:

Message-Id: <1307041990-26194-11-git-send-email-ehabkost@redhat.com
http://marc.info/?l=qemu-devel&m=130704415919346

This version doesn't have the duplicate feature bits on extfeature_edx, though,
as they are being removed from the Intel models (as they are reserved bits on
Intel CPUs).

Version 1 patch description:

    This patch adds Westmere as a qemu cpu model.  The only
    additional guest visible feature of a Westmere relative
    to Nehalem is the inclusion of AES instructions.  However
    as other non-ABI visible modifications exist along with
    fabrication changes, the CPUID data of the corresponding
    deployed silicon was altered slightly to reflect this.

    We've seen isolated cases where apparently unrelated yet
    slightly incoherent CPUID data has caused problems, most
    notably during guest boot.  Providing Westmere as a
    model separate fro Nehalem allows us to more easily address
    such quirks.

    [ehabkost: edited commit message to have a better Subject line]
Signed-off-by: Njohn cooper <john.cooper@redhat.com>
Signed-off-by: NEduardo Habkost <ehabkost@redhat.com>

Changes version 1 -> version 2:
 - Remove the duplicate feature bits on extfeature_edx, that are
   reserved on Intel CPUs
 - Reorder feature flags
 - Remove x2apic from the definition because x2apic requires some fixes
   that have to be resubmitted
Signed-off-by: NEduardo Habkost <ehabkost@redhat.com>
Signed-off-by: NAnthony Liguori <aliguori@us.ibm.com>

This patch removes the replicated feature flags from cpuid 8000_0001:edx
(extfeature_edx) from Intel models, as the duplicated feature flags are present
only on AMD CPUs. On Intel models, only the i64, syscall, and xd flags are kept
on extfeature_edx.

This is based on a previous patch from John Cooper where this was introduced
with many other changes at the same time. Original John's patch submission is
at Message-ID: <4DDAD5E7.2020002@redhat.com>, <http://marc.info/?l=qemu-devel&m=130618871926030>.

Original John's patch description was:

    cpu model bug fixes and definition corrections

    This patch was intended to address the replicated feature
    flags in cpuid 8000_0001:edx from cpuid 0000_0001:edx.
    This is due to AMD's definition where these flags are
    mostly cloned in the 8000_0001:edx cpuid function.
    qemu64 attempted to glue together the respective Intel
    and AMD nearly disjoint features and this propagated to
    the new Intel models as doing so was believed conservative
    at the time.  However after further soak and test lugging
    around this cruft doesn't provide any value, could
    conceivably confuse a guest, and has confused users trying
    to maintain/add cpu definitions.  This also caused issues
    for libvirt attempting to track this mis-encoding.

    So we've here tossed out the AMD replicated definitions
    from the Intel models, added a few replications into AMD
    definitions which were missing according to AMD's latest
    CPUID document, and reordered the config file flags to
    follow intuitive sequential bit ordering.  Also two flag
    name aliases were added for clarity to Intel models.  The
    end result being the models definitions now conform to
    their respective cpuid specifications sans x2apic which is
    emulated by kvm.

    This was tested with the following combinations:

        [Conroe, Penryn, Nehalem] x [F12-64, win64, win32] -- Intel host
        [Opteron_G1, Opteron_G2, Opteron_G3] x [F12-64, win64, win32] -- AMD host

    Yielding successful boots in all cases.
Signed-off-by: Njohn cooper <john.cooper@redhat.com>

Changes v1 -> v2:
 - Rebase against latest Qemu git tree
Signed-off-by: NEduardo Habkost <ehabkost@redhat.com>
Signed-off-by: NAnthony Liguori <aliguori@us.ibm.com>

This patch adds some missing flags to extfeature_edx, that were missing
according to AMD's latest CPUID document.

This is based on a previous patch from John Cooper where this was introduced
with many other changes at the same time. Original John's patch submission is
at Message-ID: <4DDAD5E7.2020002@redhat.com>, <http://marc.info/?l=qemu-devel&m=130618871926030>.

Original John's patch description was:

    cpu model bug fixes and definition corrections

    This patch was intended to address the replicated feature
    flags in cpuid 8000_0001:edx from cpuid 0000_0001:edx.
    This is due to AMD's definition where these flags are
    mostly cloned in the 8000_0001:edx cpuid function.
    qemu64 attempted to glue together the respective Intel
    and AMD nearly disjoint features and this propagated to
    the new Intel models as doing so was believed conservative
    at the time.  However after further soak and test lugging
    around this cruft doesn't provide any value, could
    conceivably confuse a guest, and has confused users trying
    to maintain/add cpu definitions.  This also caused issues
    for libvirt attempting to track this mis-encoding.

    So we've here tossed out the AMD replicated definitions
    from the Intel models, added a few replications into AMD
    definitions which were missing according to AMD's latest
    CPUID document, and reordered the config file flags to
    follow intuitive sequential bit ordering.  Also two flag
    name aliases were added for clarity to Intel models.  The
    end result being the models definitions now conform to
    their respective cpuid specifications sans x2apic which is
    emulated by kvm.

    This was tested with the following combinations:

        [Conroe, Penryn, Nehalem] x [F12-64, win64, win32] -- Intel host
        [Opteron_G1, Opteron_G2, Opteron_G3] x [F12-64, win64, win32] -- AMD host

    Yielding successful boots in all cases.
Signed-off-by: Njohn cooper <john.cooper@redhat.com>

Changes v1 -> v2:
 - Rebase against latest Qemu git tree
Signed-off-by: NEduardo Habkost <ehabkost@redhat.com>
Signed-off-by: NAnthony Liguori <aliguori@us.ibm.com>

Use 'i64' instead of 'lm' and 'xd' instead of 'nx' on Intel models.

The flags have different names on Intel docs, so use those names for clarity.

This is based on a previous patch from John Cooper where this was introduced
with many other changes at the same time. Original John's patch submission is
at Message-ID: <4DDAD5E7.2020002@redhat.com>, <http://marc.info/?l=qemu-devel&m=130618871926030>.

Changes v1 -> v2:
 - Rebase patch against latest Qemu git tree
Signed-off-by: NEduardo Habkost <ehabkost@redhat.com>
Signed-off-by: NAnthony Liguori <aliguori@us.ibm.com>

This will make it easier to review and change the flag list in the future.

No behaviour change should be introduced by this, as it is just changing
the flag order on the config file.

To make sure the flag sets are really not changed by this patch, I have
used the following stupid script to compare the flag values in the
config files:
https://gist.github.com/1004885Signed-off-by: NEduardo Habkost <ehabkost@redhat.com>
Signed-off-by: NAnthony Liguori <aliguori@us.ibm.com>

This is a reimplementation of prior versions which adds
the ability to define cpu models for contemporary processors.
The added models are likewise selected via -cpu <name>,
and are intended to displace the existing convention
of "-cpu qemu64" augmented with a series of feature flags.

A primary motivation was determination of a least common
denominator within a given processor class to simplify guest
migration.  It is still possible to modify an arbitrary model
via additional feature flags however the goal here was to
make doing so unnecessary in typical usage.  The other
consideration was providing models names reflective of
current processors.  Both AMD and Intel have reviewed the
models in terms of balancing generality of migration vs.
excessive feature downgrade relative to released silicon.

This version of the patch replaces the prior hard wired
definitions with a configuration file approach for new
models.  Existing models are thus far left as-is but may
easily be transitioned to (or may be overridden by) the
configuration file representation.

Proposed new model definitions are provided here for current
AMD and Intel processors.  Each model consists of a name
used to select it on the command line (-cpu <name>), and a
model_id which corresponds to a least common denominator
commercial instance of the processor class.

A table of names/model_ids may be queried via "-cpu ?model":

        :
    x86       Opteron_G3  AMD Opteron 23xx (Gen 3 Class Opteron)
    x86       Opteron_G2  AMD Opteron 22xx (Gen 2 Class Opteron)
    x86       Opteron_G1  AMD Opteron 240 (Gen 1 Class Opteron)
    x86          Nehalem  Intel Core i7 9xx (Nehalem Class Core i7)
    x86           Penryn  Intel Core 2 Duo P9xxx (Penryn Class Core 2)
    x86           Conroe  Intel Celeron_4x0 (Conroe/Merom Class Core 2)
        :

Also added is "-cpu ?dump" which exhaustively outputs all config
data for all defined models, and "-cpu ?cpuid" which enumerates
all qemu recognized CPUID feature flags.

The pseudo cpuid flag 'check' when added to the feature flag list
will warn when feature flags (either implicit in a cpu model or
explicit on the command line) would have otherwise been quietly
unavailable to a guest:

    # qemu-system-x86_64 ... -cpu Nehalem,check
    warning: host cpuid 0000_0001 lacks requested flag 'sse4.2|sse4_2' [0x00100000]
    warning: host cpuid 0000_0001 lacks requested flag 'popcnt' [0x00800000]

A similar 'enforce' pseudo flag exists which in addition
to the above causes qemu to error exit if requested flags are
unavailable.

Configuration data for a cpu model resides in the target config
file which by default will be installed as:

    /usr/local/etc/qemu/target-<arch>.conf

The format of this file should be self explanatory given the
definitions for the above six models and essentially mimics
the structure of the static x86_def_t x86_defs.

Encoding of cpuid flags names now allows aliases for both the
configuration file and the command line which reconciles some
Intel/AMD/Linux/Qemu naming differences.

This patch was tested relative to qemu.git.
Signed-off-by: Njohn cooper <john.cooper@redhat.com>
Signed-off-by: NAnthony Liguori <aliguori@us.ibm.com>