This reuses the XML format that was introduced for Gluster.
Signed-off-by: NPaolo Bonzini <pbonzini@redhat.com>
Signed-off-by: NEric Blake <eblake@redhat.com>

These are supported by nbd-server and by the NBD server that QEMU
embeds for live image access.
Signed-off-by: NPaolo Bonzini <pbonzini@redhat.com>
Signed-off-by: NEric Blake <eblake@redhat.com>

Enable more testing of NBD parsing, to ensure rewrites work.
Signed-off-by: NPaolo Bonzini <pbonzini@redhat.com>
Signed-off-by: NEric Blake <eblake@redhat.com>

The virCaps structure gathered a ton of irrelevant data over time that.
The original reason is that it was propagated to the XML parser
functions.

This patch aims to create a new data structure virDomainXMLConf that
will contain immutable data that are used by the XML parser. This will
allow two things we need:

1) Get rid of the stuff from virCaps

2) Allow us to add callbacks to check and add driver specific stuff
after domain XML is parsed.

This first attempt removes pointers to private data allocation functions
to this new structure and update all callers and function that require
them.

Adds XML parsing and qemu commandline tests for the VirtIO RNG device
support.

To enable virCapabilities instances to be reference counted,
turn it into a virObject. All cases of virCapabilitiesFree
turn into virObjectUnref
Signed-off-by: NDaniel P. Berrange <berrange@redhat.com>

Like "rawio", "sgio" is only allowed for block disk of device
type "lun".

It doesn't default disk->sgio to "filtered" when parsing, as
it won't be able to distinguish explicitly requested "filtered"
and a default "filtered" in driver then. We have to error out for
explicit request when the kernel doesn't support the new sysfs
knob "unpriv_sgio", however, for defaulted "filtered", we can
just ignore it if the kernel doesn't support "unpriv_sgio".

QEMU supports setting vendor and product strings for disk since
1.2.0 (only scsi-disk, scsi-hd, scsi-cd support it), this patch
exposes it with new XML elements <vendor> and <product> of disk
device.

Remove the obsolete 'qemud' naming prefix and underscore
based type name. Introduce virQEMUDriverPtr as the replacement,
in common with LXC driver naming style

This patch adds QEMU support for the "relaxed" feature implemented by
previous patch.

This patch adds full support for EOI setting for domains. Because this
is CPU feature (flag), the model needs to be added even when it's not
specified. Fortunately this problem was already solved with kvmclock,
so this patch simply abuses that.

And due to the size of the patch (17 lines) I dared to include the tests.

Few tests were added which are checking whether the parsing of the xml
and command-line arguments is working and compatible with each other.

The following config elements now support a <vlan> subelements:

within a domain: <interface>, and the <actual> subelement of <interface>
within a network: the toplevel, as well as any <portgroup>

Each vlan element must have one or more <tag id='n'/> subelements.  If
there is more than one tag, it is assumed that vlan trunking is being
requested. If trunking is required with only a single tag, the
attribute "trunk='yes'" should be added to the toplevel <vlan>
element.

Some examples:

  <interface type='hostdev'/>
    <vlan>
      <tag id='42'/>
    </vlan>
    <mac address='52:54:00:12:34:56'/>
    ...
  </interface>

  <network>
    <name>vlan-net</name>
    <vlan trunk='yes'>
      <tag id='30'/>
    </vlan>
    <virtualport type='openvswitch'/>
  </network>

  <interface type='network'/>
    <source network='vlan-net'/>
    ...
  </interface>

  <network>
    <name>trunk-vlan</name>
    <vlan>
      <tag id='42'/>
      <tag id='43'/>
    </vlan>
    ...
  </network>

  <network>
    <name>multi</name>
    ...
    <portgroup name='production'/>
      <vlan>
        <tag id='42'/>
      </vlan>
    </portgroup>
    <portgroup name='test'/>
      <vlan>
        <tag id='666'/>
      </vlan>
    </portgroup>
  </network>

  <interface type='network'/>
    <source network='multi' portgroup='test'/>
    ...
  </interface>

IMPORTANT NOTE: As of this patch there is no backend support for the
vlan element for *any* network device type. When support is added in
later patches, it will only be for those select network types that
support setting up a vlan on the host side, without the guest's
involvement. (For example, it will be possible to configure a vlan for
a guest connected to an openvswitch bridge, but it won't be possible
to do that for one that is connected to a standard Linux host bridge.)

Test new codec type element.

Currently each USB2 companion controller gets put on a separate
PCI slot. Not only is this wasteful of PCI slots, but it is not
in compliance with the spec for USB2 controllers. The master
echi1 and all companion controllers should be in the same slot,
with echi1 in function 7, and uhci1-3 in functions 0-2 respectively.

* src/qemu/qemu_command.c: Special case handling of USB2 controllers
  to apply correct pci slot assignment
* tests/qemuxml2argvdata/qemuxml2argv-usb-ich9-ehci-addr.args,
  tests/qemuxml2argvdata/qemuxml2argv-usb-ich9-ehci-addr.xml: Expand
  test to cover automatic slot assignment

With this patch, one can also fully drive numad by:

  <vcpu>2</vcpu>
  <numatune>
    <memory placement='auto'/>
  </numatune>

New tests are added.

Though numad will manage the memory allocation of task dynamically,
it wants management application (libvirt) to pre-set the memory
policy according to the advisory nodeset returned from querying numad,
(just like pre-bind CPU nodeset for domain process), and thus the
performance could benefit much more from it.

This patch introduces new XML tag 'placement', value 'auto' indicates
whether to set the memory policy with the advisory nodeset from numad,
and its value defaults to the value of <vcpu> placement, or 'static'
if 'nodeset' is specified. Example of the new XML tag's usage:

  <numatune>
    <memory placement='auto' mode='interleave'/>
  </numatune>

Just like what current "numatune" does, the 'auto' numa memory policy
setting uses libnuma's API too.

If <vcpu> "placement" is "auto", and <numatune> is not specified
explicitly, a default <numatume> will be added with "placement"
set as "auto", and "mode" set as "strict".

The following XML can now fully drive numad:

1) <vcpu> placement is 'auto', no <numatune> is specified.

   <vcpu placement='auto'>10</vcpu>

2) <vcpu> placement is 'auto', no 'placement' is specified for
   <numatune>.

   <vcpu placement='auto'>10</vcpu>
   <numatune>
     <memory mode='interleave'/>
   </numatune>

And it's also able to control the CPU placement and memory policy
independently. e.g.

1) <vcpu> placement is 'auto', and <numatune> placement is 'static'

   <vcpu placement='auto'>10</vcpu>
   <numatune>
     <memory mode='strict' nodeset='0-10,^7'/>
   </numatune>

2) <vcpu> placement is 'static', and <numatune> placement is 'auto'

   <vcpu placement='static' cpuset='0-24,^12'>10</vcpu>
   <numatune>
     <memory mode='interleave' placement='auto'/>
   </numatume>

A follow up patch will change the XML formatting codes to always output
'placement' for <vcpu>, even it's 'static'.

In order to track a block copy job across libvirtd restarts, we
need to save internal XML that tracks the name of the file
holding the mirror.  Displaying this name in dumpxml might also
be useful to the user, even if we don't yet have a way to (re-)
start a domain with mirroring enabled up front.  This is done
with a new <mirror> sub-element to <disk>, as in:

    <disk type='file' device='disk'>
      <driver name='qemu' type='raw'/>
      <source file='/var/lib/libvirt/images/original.img'/>
      <mirror file='/var/lib/libvirt/images/copy.img' format='qcow2' ready='yes'/>
      ...
    </disk>

For now, the element is output-only, in live domains; it is ignored
when defining a domain or hot-plugging a disk (since those contexts
use VIR_DOMAIN_XML_INACTIVE in parsing).  The 'ready' attribute appears
when libvirt knows that the job has changed from the initial pulling
phase over to the mirroring phase, although absence of the attribute
is not a sure indicator of the current phase.  If we come up with a way
to make qemu start with mirroring enabled, we can relax the xml
restriction, and allow <mirror> (but not attribute 'ready') on input.
Testing active-only XML meant tweaking the testsuite slightly, but it
was worth it.

* docs/schemas/domaincommon.rng (diskspec): Add diskMirror.
* docs/formatdomain.html.in (elementsDisks): Document it.
* src/conf/domain_conf.h (_virDomainDiskDef): New members.
* src/conf/domain_conf.c (virDomainDiskDefFree): Clean them.
(virDomainDiskDefParseXML): Parse them, but only internally.
(virDomainDiskDefFormat): Output them.
* tests/qemuxml2argvdata/qemuxml2argv-disk-mirror.xml: New test file.
* tests/qemuxml2xmloutdata/qemuxml2xmlout-disk-mirror.xml: Likewise.
* tests/qemuxml2xmltest.c (testInfo): Alter members.
(testCompareXMLToXMLHelper): Allow more test control.
(mymain): Run new test.

Return statements with parameter enclosed in parentheses were modified
and parentheses were removed. The whole change was scripted, here is how:

List of files was obtained using this command:
git grep -l -e '\<return\s*([^()]*\(([^()]*)[^()]*\)*)\s*;' |             \
grep -e '\.[ch]$' -e '\.py$'

Found files were modified with this command:
sed -i -e                                                                 \
's_^\(.*\<return\)\s*(\(\([^()]*([^()]*)[^()]*\)*\))\s*\(;.*$\)_\1 \2\4_' \
-e 's_^\(.*\<return\)\s*(\([^()]*\))\s*\(;.*$\)_\1 \2\3_'

Then checked for nonsense.

The whole command looks like this:
git grep -l -e '\<return\s*([^()]*\(([^()]*)[^()]*\)*)\s*;' |             \
grep -e '\.[ch]$' -e '\.py$' | xargs sed -i -e                            \
's_^\(.*\<return\)\s*(\(\([^()]*([^()]*)[^()]*\)*\))\s*\(;.*$\)_\1 \2\4_' \
-e 's_^\(.*\<return\)\s*(\([^()]*\))\s*\(;.*$\)_\1 \2\3_'

Output is still in kibibytes, but input can now be in different
scales for ease of typing.

* src/conf/domain_conf.c (virDomainParseMemory): New helper.
(virDomainDefParseXML): Use it when parsing.
* docs/schemas/domaincommon.rng: Expand XML; rename memoryKBElement
to memoryElement and update callers.
* docs/formatdomain.html.in (elementsMemoryAllocation): Document
scaling.
* tests/qemuxml2argvdata/qemuxml2argv-memtune.xml: Adjust test.
* tests/qemuxml2xmltest.c: Likewise.
* tests/qemuxml2xmloutdata/qemuxml2xmlout-memtune.xml: New file.

This is the new interface type that sets up an SR-IOV PCI network
device to be assigned to the guest with PCI passthrough after
initializing some network device-specific things from the config
(e.g. MAC address, virtualport profile parameters). Here is an example
of the syntax:

  <interface type='hostdev' managed='yes'>
    <source>
      <address type='pci' domain='0' bus='0' slot='4' function='3'/>
    </source>
    <mac address='00:11:22:33:44:55'/>
    <address type='pci' domain='0' bus='0' slot='7' function='0'/>
  </interface>

This would assign the PCI card from bus 0 slot 4 function 3 on the
host, to bus 0 slot 7 function 0 on the guest, but would first set the
MAC address of the card to 00:11:22:33:44:55.

NB: The parser and formatter don't care if the PCI card being
specified is a standard single function network adapter, or a virtual
function (VF) of an SR-IOV capable network adapter, but the upcoming
code that implements the back end of this config will work *only* with
SR-IOV VFs. This is because modifying the mac address of a standard
network adapter prior to assigning it to a guest is pointless - part
of the device reset that occurs during that process will reset the MAC
address to the value programmed into the card's firmware.

Although it's not supported by any of libvirt's hypervisor drivers,
usb network hostdevs are also supported in the parser and formatter
for completeness and consistency. <source> syntax is identical to that
for plain <hostdev> devices, except that the <address> element should
have "type='usb'" added if bus/device are specified:

  <interface type='hostdev'>
    <source>
      <address type='usb' bus='0' device='4'/>
    </source>
    <mac address='00:11:22:33:44:55'/>
  </interface>

If the vendor/product form of usb specification is used, type='usb'
is implied:

  <interface type='hostdev'>
    <source>
      <vendor id='0x0012'/>
      <product id='0x24dd'/>
    </source>
    <mac address='00:11:22:33:44:55'/>
  </interface>

Again, the upcoming patch to fill in the backend of this functionality
will log an error and fail with "Unsupported Config" if you actually
try to assign a USB network adapter to a guest using <interface
type='hostdev'> - just use a standard <hostdev> entry in that case
(and also for single-port PCI adapters).

This reverts commit 7b345b69.

Conflicts:

	tests/qemuxml2argvdata/qemuxml2argv-disk-scsi-vscsi.xml

This reverts commit c9abfadf.

Conflicts:

	tests/qemuxml2argvdata/qemuxml2argv-disk-scsi-virtio-scsi.xml

Commit b170eb99 introduced a bug: domains that had an explicit
<seclabel type='none'/> when started would not be reparsed if
libvirtd restarted.  It turns out that our testsuite was not
exercising this because it never tried anything but inactive
parsing.  Additionally, the live XML for such a domain failed
to re-validate.  Applying just the tests/ portion of this patch
will expose the bugs that are fixed by the other two files.

* docs/schemas/domaincommon.rng (seclabel): Allow relabel under
type='none'.
* src/conf/domain_conf.c (virSecurityLabelDefParseXML): Per RNG,
presence of <seclabel> with no type implies dynamic.  Don't
require sub-elements for type='none'.
* tests/qemuxml2xmltest.c (mymain): Add test.
* tests/qemuxml2argvtest.c (mymain): Likewise.
* tests/qemuxml2argvdata/qemuxml2argv-seclabel-none.xml: Add file.
* tests/qemuxml2argvdata/qemuxml2argv-seclabel-none.args: Add file.
Reported by Ansis Atteka.

The bulk of this patch was done with:

sed -i 's/\(\bfree *(/VIR_FREE(/g' tests/*.c

followed by fixing the few compile errors that resulted.

* cfg.mk (exclude_file_name_regexp--sc_prohibit_raw_allocation):
Remove tests from exemption.
* tests/testutils.h: Add common header.
* tests/commandhelper.c: Fix offenders.
* tests/cputest.c: Likewise.
* tests/domainsnapshotxml2xmltest.c: Likewise.
* tests/interfacexml2xmltest.c: Likewise.
* tests/networkxml2argvtest.c: Likewise.
* tests/networkxml2xmltest.c: Likewise.
* tests/nodedevxml2xmltest.c: Likewise.
* tests/nodeinfotest.c: Likewise.
* tests/nwfilterxml2xmltest.c: Likewise.
* tests/qemuargv2xmltest.c: Likewise.
* tests/qemuxml2argvtest.c: Likewise.
* tests/qemuxml2xmltest.c: Likewise.
* tests/qemuxmlnstest.c: Likewise.
* tests/qparamtest.c: Likewise.
* tests/sexpr2xmltest.c: Likewise.
* tests/storagepoolxml2xmltest.c: Likewise.
* tests/storagevolxml2xmltest.c: Likewise.
* tests/testutils.c: Likewise.
* tests/virshtest.c: Likewise.
* tests/xencapstest.c: Likewise.
* tests/xmconfigtest.c: Likewise.
* tests/xml2sexprtest.c: Likewise.

When support for the rombar option was added, it was only added for
PCI passthrough devices, configured with <hostdev>. The same option is
available for any network device that is attached to the guest's PCI
bus. This patch allows setting rombar for any PCI network device type.

After adding cases to test this to qemuxml2argv-hostdev-pci-rombar.*,
I decided to rename those files (to qemuxml2argv-pci-rom.*) to more
accurately reflect the additional tests, and also noticed that up to
now we've only been performing a domainschematest for that case, so I
added the "pci-rom" test to both qemuxml2argv and qemuxml2xml (and in
the process found some bugs whose fixes I squashed into previous
commits of this series).

QEMU supports a bunch of CPUID features that are tied to the kvm CPUID
nodes rather than the processor's.  They are "kvmclock",
"kvm_nopiodelay", "kvm_mmu", "kvm_asyncpf".  These are not known to
libvirt and their CPUID leaf might move if (for example) the Hyper-V
extensions are enabled. Hence their handling would anyway require some
special-casing.

However, among these the most useful is kvmclock; an additional
"property" of this feature is that a <timer> element is a better model
than a CPUID feature.  Although, creating part of the -cpu command-line
from something other than the <cpu> XML element introduces some
ugliness.
Reviewed-by: NJiri Denemark <jdenemar@redhat.com>
Signed-off-by: NPaolo Bonzini <pbonzini@redhat.com>

Applications can now insert custom nodes and hierarchies into domain
configuration XML. Although currently not enforced, applications are
required to use their own namespaces on every custom node they insert,
with only one top-level element per namespace.

In case a hypervisor doesn't support the exact CPU model requested by a
domain XML, we automatically fallback to a closest CPU model the
hypervisor supports (and make sure we add/remove any additional features
if needed). This patch adds 'fallback' attribute to model element, which
can be used to disable this automatic fallback.

Adding a new model for virtio-scsi roughly follows the same scheme
as the previous patch.
Signed-off-by: NPaolo Bonzini <pbonzini@redhat.com>

KVM will be able to use a PCI SCSI controller even on POWER.  Let
the user specify the vSCSI controller by other means than a default.

After this patch, the QEMU driver will actually look at the model
and reject anything but auto, lsilogic and ibmvscsi.
Signed-off-by: NPaolo Bonzini <pbonzini@redhat.com>

In the past, generic SCSI commands issued from a guest to a virtio
disk were always passed through to the underlying disk by qemu, and
the kernel would also pass them on.

As a result of CVE-2011-4127 (see:
http://seclists.org/oss-sec/2011/q4/536), qemu now honors its
scsi=on|off device option for virtio-blk-pci (which enables/disables
passthrough of generic SCSI commands), and the kernel will only allow
the commands for physical devices (not for partitions or logical
volumes). The default behavior of qemu is still to allow sending
generic SCSI commands to physical disks that are presented to a guest
as virtio-blk-pci devices, but libvirt prefers to disable those
commands in the standard virtio block devices, enabling it only when
specifically requested (hopefully indicating that the requester
understands what they're asking for). For this purpose, a new libvirt
disk device type (device='lun') has been created.

device='lun' is identical to the default device='disk', except that:

1) It is only allowed if bus='virtio', type='block', and the qemu
   version is "new enough" to support it ("new enough" == qemu 0.11 or
   better), otherwise the domain will fail to start and a
   CONFIG_UNSUPPORTED error will be logged).

2) The option "scsi=on" will be added to the -device arg to allow
   SG_IO commands (if device !='lun', "scsi=off" will be added to the
   -device arg so that SG_IO commands are specifically forbidden).

Guests which continue to use disk device='disk' (the default) will no
longer be able to use SG_IO commands on the disk; those that have
their disk device changed to device='lun' will still be able to use SG_IO
commands.

*docs/formatdomain.html.in - document the new device attribute value.
*docs/schemas/domaincommon.rng - allow it in the RNG
*tests/* - update the args of several existing tests to add scsi=off, and
 add one new test that will test scsi=on.
*src/conf/domain_conf.c - update domain XML parser and formatter

*src/qemu/qemu_(command|driver|hotplug).c - treat
 VIR_DOMAIN_DISK_DEVICE_LUN *almost* identically to
 VIR_DOMAIN_DISK_DEVICE_DISK, except as indicated above.

Note that no support for this new device value was added to any
hypervisor drivers other than qemu, because it's unclear what it might
mean (if anything) to those drivers.

Implement the parsing and formatting of the XML addition of
the previous commit.  The new XML doesn't affect qemu command
line, so we can now test round-trip XML->memory->XML handling.

I chose to reuse the existing structure, even though per-device
override doesn't use all of those fields, rather than create a
new structure, in order to reuse more code.

* src/conf/domain_conf.h (_virDomainDiskDef): Add seclabel member.
* src/conf/domain_conf.c (virDomainDiskDefFree): Free it.
(virSecurityLabelDefFree): New function.
(virDomainDiskDefFormat): Print it.
(virSecurityLabelDefFormat): Reduce output if model not present.
(virDomainDiskDefParseXML): Alter signature, and parse seclabel.
(virSecurityLabelDefParseXML): Split...
(virSecurityLabelDefParseXMLHelper): ...into new helper.
(virDomainDeviceDefParse, virDomainDefParseXML): Update callers.
* tests/qemuxml2argvdata/qemuxml2argv-seclabel-dynamic-override.args:
New file.
* tests/qemuxml2xmltest.c (mymain): Enhance test.
* tests/qemuxml2argvtest.c (mymain): Likewise.

The RNG for <seclabel> was too strict - if it was present, then it
had to have sub-elements, even if those didn't make sense for the
given attributes.  Also, we didn't have any tests of <seclabel>
parsing or XML output.

In this patch, I added more parsing tests than output tests (since
the output populates and/or reorders fields not present in certain
inputs).  Making the RNG reliable is a precursor to using <seclabel>
variants in more places in the XML in later patches.

See also:
http://berrange.com/posts/2011/09/29/two-small-improvements-to-svirt-guest-configuration-flexibility-with-kvmlibvirt/

* docs/schemas/domaincommon.rng (seclabel): Tighten rules.
* tests/qemuxml2argvtest.c (mymain): New tests.
* tests/qemuxml2xmltest.c (mymain): Likewise.
* tests/qemuxml2argvdata/qemuxml2argv-seclabel-*.*: New files.

I got this failure on FreeBSD:

shunloadtest.c: In function 'main':
shunloadtest.c:150: error: 'EXIT_AM_SKIP' undeclared (first use in this function)

but inspection showed several other problems, all fixed here.

* tests/domainsnapshotxml2xmltest.c [!WITH_QEMU]: Ensure
EXIT_AM_SKIP is defined.
* tests/esxutilstest.c [!WITH_ESX]: Likewise.
* tests/openvzutilstest.c [!WITH_OPENVZ]: Likewise.
* tests/qemuargv2xmltest.c [!WITH_QEMU]: Likewise.
* tests/qemuhelptest.c [!WITH_QEMU]: Likewise.
* tests/qemuxml2argvtest.c [!WITH_QEMU]: Likewise.
* tests/qemuxml2xmltest.c [!WITH_QEMU]: Likewise.
* tests/qemuxmlnstest.c [!WITH_QEMU]: Likewise.
* tests/shunloadtest.c [!linux]: Likewise.
* tests/vmx2xmltest.c [!WITH_VMX]: Likewise.
* tests/xml2vmxtest.c [!WITH_VMX]: Likewise.

Signed-off-by: NLei Li <lilei@linux.vnet.ibm.com>
Signed-off-by: NZhi Yong Wu <wuzhy@linux.vnet.ibm.com>
Signed-off-by: NEric Blake <eblake@redhat.com>

Implement setting/getting per-device blkio weights in qemu,
using the cgroups blkio.weight_device tunable.