CPU numa topology implicitly allows memory specification in 'KiB'.

Enabling this to accept the 'unit' in which memory needs to be specified.
This now allows users to specify memory in units of choice, and
lists the same in 'KiB' -- just like other 'memory' elements in XML.

    <numa>
      <cell cpus='0-3' memory='1024' unit='MiB' />
      <cell cpus='4-7' memory='1024' unit='MiB' />
    </numa>

Also augment test cases to correctly model NUMA memory specification.
This adds the tag 'unit="KiB"' for memory attribute in NUMA cells.
Signed-off-by: NPrerna Saxena <prerna@linux.vnet.ibm.com>
Signed-off-by: NMichal Privoznik <mprivozn@redhat.com>

This new attribute will control whether or not libvirt will pay
attention to guest notifications about changes to network device mac
addresses and receive filters. The default for this is 'no' (for
security reasons). If it is set to 'yes' *and* the specified device
model and connection support it (currently only macvtap+virtio) then
libvirt will watch for NIC_RX_FILTER_CHANGED events, and when it
receives one, it will issue a query-rx-filter command, retrieve the
result, and modify the host-side macvtap interface's mac address and
unicast/multicast filters accordingly.

The functionality behind this attribute will be in a later patch. This
patch merely adds the attribute to the top-level of a domain's
<interface> as well as to <network> and <portgroup>, and adds
documentation and schema/xml2xml tests. Rather than adding even more
test files, I've just added the net attribute in various applicable
places of existing test files.

This patch adds parsing/formatting code as well as documentation for
shared memory devices.  This will currently be only accessible in QEMU
using it's ivshmem device, but is designed as generic as possible to
allow future expansion for other hypervisors.

In the devices section in the domain XML users may specify:

- For shmem device using a server:

 <shmem name='shmem0'>
   <server path='/tmp/socket-ivshmem0'/>
   <size unit='M'>32</size>
   <msi vectors='32' ioeventfd='on'/>
 </shmem>

- For ivshmem device not using an ivshmem server:

 <shmem name='shmem1'>
   <size unit='M'>32</size>
 </shmem>

Most of the configuration is made optional so it also allows
specifications like:

 <shmem name='shmem1/>
 <shmem name='shmem2'>
   <server/>
 </shmem>
Signed-off-by: NMaxime Leroy <maxime.leroy@6wind.com>
Signed-off-by: NMartin Kletzander <mkletzan@redhat.com>

Add options for tuning segment offloading:
<driver>
  <host csum='off' gso='off' tso4='off' tso6='off'
        ecn='off' ufo='off'/>
  <guest csum='off' tso4='off' tso6='off' ecn='off' ufo='off'/>
</driver>
which control the respective host_ and guest_ properties
of the virtio-net device.

Each attribute is optional, commit af8b4a2e forgot to mention that.
Signed-off-by: NMartin Kletzander <mkletzan@redhat.com>

Add the 'rawio' attribute to match _virDomainDiskDef and process the
hostdev XML similarly to the disk XML for a lun which supports/requires rawio

Signed-off-by: NMartin Kletzander <mkletzan@redhat.com>

Signed-off-by: NMartin Kletzander <mkletzan@redhat.com>

For tuning the network, alternative devices
for creating tap and vhost devices can be specified via:
<backend tap='/dev/net/tun' vhost='/dev/net-vhost'/>

I noticed this with the recent iothread pinning code, but the
problem existed longer than that. The XML validation required
users to supply <cputune> children in a strict order, even though
there was no conceptual reason why they can't occur in any order.

docs/ changes best viewed with -w

* docs/schemas/domaincommon.rng (cputune): Add interleave.
* tests/qemuxml2argvdata/qemuxml2argv-cputune-iothreads.xml: Swap
up order, copying canonical form...
* tests/qemuxml2xmloutdata/qemuxml2xmlout-cputune-iothreads.xml:
...here.
* tests/qemuxml2xmltest.c (mymain): Mark the difference.
Signed-off-by: NEric Blake <eblake@redhat.com>

https://bugzilla.redhat.com/show_bug.cgi?id=1101574

Add an option 'iothreadpin' to the <cpuset> to allow for setting the
CPU affinity for each IOThread.

The iothreadspin will mimic the vcpupin with respect to being able to
assign each iothread to a specific CPU, although iothreads ids start
at 1 while vcpu ids start at 0. This matches the iothread naming scheme.

When using split UEFI image, it may come handy if libvirt manages per
domain _VARS file automatically. While the _CODE file is RO and can be
shared among multiple domains, you certainly don't want to do that on
the _VARS file. This latter one needs to be per domain. So at the
domain startup process, if it's determined that domain needs _VARS
file it's copied from this master _VARS file. The location of the
master file is configurable in qemu.conf.

Temporary, on per domain basis the location of master NVRAM file can
be overridden by this @template attribute I'm inventing to the
<nvram/> element. All it does is holding path to the master NVRAM file
from which local copy is created. If that's the case, the map in
qemu.conf is not consulted.
Signed-off-by: NMichal Privoznik <mprivozn@redhat.com>
Acked-by: NLaszlo Ersek <lersek@redhat.com>

Up to now, users can configure BIOS via the <loader/> element. With
the upcoming implementation of UEFI this is not enough as BIOS and
UEFI are conceptually different. For instance, while BIOS is ROM, UEFI
is programmable flash (although all writes to code section are
denied). Therefore we need new attribute @type which will
differentiate the two. Then, new attribute @readonly is introduced to
reflect the fact that some images are RO.

Moreover, the OVMF (which is going to be used mostly), works in two
modes:
1) Code and UEFI variable store is mixed in one file.
2) Code and UEFI variable store is separated in two files

The latter has advantage of updating the UEFI code without losing the
configuration. However, in order to represent the latter case we need
yet another XML element: <nvram/>. Currently, it has no additional
attributes, it's just a bare element containing path to the variable
store file.
Signed-off-by: NMichal Privoznik <mprivozn@redhat.com>
Acked-by: NLaszlo Ersek <lersek@redhat.com>
Signed-off-by: NMichal Privoznik <mprivozn@redhat.com>

Add a new disk "driver" attribute "iothread" to be parsed as the thread
number for the disk to use. In order to more easily facilitate the usage
and configuration of the iothread, a "zero" for the attribute indicates
iothreads are not supported for the device and a positive value indicates
the specific thread to try and use.

Introduce XML to allowing adding iothreads to the domain. These can be
used by virtio-blk-pci devices in order to assign a specific thread to
handle the workload for the device.  The iothreads are the official
implementation of the virtio-blk Data Plane that's been in tech preview
for QEMU.

QEMU 2.1 added support for the kvm=off option to the -cpu command,
allowing the KVM hypervisor signature to be hidden from the guest.
This enables disabling of some paravirualization features in the
guest as well as allowing certain drivers which test for the
hypervisor to load.  Domain XML syntax is as follows:

<domain type='kvm>
  ...
  <features>
    ...
    <kvm>
      <hidden state='on'/>
    </kvm>
  </features>
  ...
Signed-off-by: NAlex Williamson <alex.williamson@redhat.com>

Signed-off-by: NMartin Kletzander <mkletzan@redhat.com>

Introduce a new structure to handle an iSCSI host device based on the
existing virDomainHostdevSubsysSCSI by adding a "protocol='iscsi'" to
the <source/> element. The existing scsi_host subsystem RNG was modified
to read an optional "protocol='adapter'", although it won't be written
out nor is it documented as an option (by choice).

The new hostdev structure mimics the existing <disk/> element for an
iSCSI device (network) device. New XML is:

  <hostdev mode='subsystem' type='scsi' managed='yes'>
    <source protocol='iscsi' name='iqn.1992-01.com.example'>
      <host name='example.org' port='3260'/>
      <auth username='myname'>
        <secret type='iscsi' usage='mycluster_myname'/>
      </auth>
    </source>
    <address type='drive' controller='0' bus='0' target='2' unit='5'/>
  </hostdev>

The controller element will mimic the existing scsi_host code insomuch
as when 'lsi' and 'virtio-scsi' are used.

A future patch is going to wire up qemu active block commit jobs;
but as they have similar events and are canceled/pivoted in the
same way as block copy jobs, it is easiest to track all bookkeeping
for the commit job by reusing the <mirror> element.  This patch
adds domain XML to track which job was responsible for creating a
mirroring situation, and adds a job='copy' attribute to all
existing uses of <mirror>.  Along the way, it also massages the
qemu monitor backend to read the new field in order to generate
the correct type of libvirt job (even though it requires a
future patch to actually cause a qemu event that can be reported
as an active commit).  It also prepares to update persistent XML
to match changes made to live XML when a copy completes.

* docs/schemas/domaincommon.rng: Enhance schema.
* docs/formatdomain.html.in: Document it.
* src/conf/domain_conf.h (_virDomainDiskDef): Add a field.
* src/conf/domain_conf.c (virDomainBlockJobType): String conversion.
(virDomainDiskDefParseXML): Parse job type.
(virDomainDiskDefFormat): Output job type.
* src/qemu/qemu_process.c (qemuProcessHandleBlockJob): Distinguish
active from regular commit.
* src/qemu/qemu_driver.c (qemuDomainBlockCopy): Set job type.
(qemuDomainBlockPivot, qemuDomainBlockJobImpl): Clean up job type
on completion.
* tests/qemuxml2xmloutdata/qemuxml2xmlout-disk-mirror-old.xml:
Update tests.
* tests/qemuxml2argvdata/qemuxml2argv-disk-mirror.xml: Likewise.
* tests/qemuxml2argvdata/qemuxml2argv-disk-active-commit.xml: New
file.
* tests/qemuxml2xmltest.c (mymain): Drive new test.
Signed-off-by: NEric Blake <eblake@redhat.com>

Doing a blockcopy operation across a libvirtd restart is not very
robust at the moment.  In particular, we are clearing the <mirror>
element prior to telling qemu to finish the job.  Also, thanks to the
ability to request async completion, the user can easily regain
control prior to qemu actually finishing the effort, and they should
be able to poll the domain XML to see if the job is still going.

A future patch will fix things to actually wait until qemu is done
before modifying the XML to reflect the job completion.  But since
qemu issues identical BLOCK_JOB_COMPLETE events regardless of whether
the job was cancelled (kept the original disk) or completed (pivoted
to the new disk), we have to track which of the two operations were
used to end the job.  Furthermore, we'd like to avoid attempts to
end a job where we are already waiting on an earlier request to qemu
to end the job.  Likewise, if we miss the qemu event (perhaps because
it arrived during a libvirtd restart), we still need enough state
recorded to be able to determine how to modify the domain XML once
we reconnect to qemu and manually learn whether the job still exists.

Although this patch doesn't actually fix the problem, it is a
preliminary step that makes it possible to track whether a job
has already begun steps towards completion.

* src/conf/domain_conf.h (virDomainDiskMirrorState): New enum.
(_virDomainDiskDef): Convert bool mirroring to new enum.
* src/conf/domain_conf.c (virDomainDiskDefParseXML)
(virDomainDiskDefFormat): Handle new values.
* src/qemu/qemu_process.c (qemuProcessHandleBlockJob): Adjust
client.
* src/qemu/qemu_driver.c (qemuDomainBlockPivot)
(qemuDomainBlockJobImpl): Likewise.
* docs/schemas/domaincommon.rng (diskMirror): Expose new values.
* docs/formatdomain.html.in (elementsDisks): Document it.
* tests/qemuxml2argvdata/qemuxml2argv-disk-mirror.xml: Test it.
Signed-off-by: NEric Blake <eblake@redhat.com>

  <memoryBacking>
    <hugepages>
      <page size="1" unit="G" nodeset="0-3,5"/>
      <page size="2" unit="M" nodeset="4"/>
    </hugepages>
  </memoryBacking>
Signed-off-by: NMichal Privoznik <mprivozn@redhat.com>

* docs/schemas/domaincommon.rng: Add bhyve domain type, nmdm
  serial type and master and slave optional attributes for
  serial that are used by nmdm
* tests/domainschematest: Add bhyvexml2argvdata directory
  to validate bhyve XMLs

Added <capabilities> in the <features> section of LXC domains
configuration. This section can contain elements named after the
capabilities like:

  <mknod state="on"/>, keep CAP_MKNOD capability
  <sys_chroot state="off"/> drop CAP_SYS_CHROOT capability

Users can restrict or give more capabilities than the default using
this mechanism.

LXC network devices can now be assigned a custom NIC device name on the
container side. For example, this is configured with:

    <interface type='network'>
      <source network='default'/>
      <guest dev="eth1"/>
    </interface>

In this example the network card will appear as eth1 in the guest.

The previous commit 09d4d261 put the interleave at the wrong point;
it didn't allow interleaving with <memory>.

* docs/schema/domaincommon.rng (numatune): Fix interleave location.
* tests/qemuxml2argvdata/qemuxml2argv-numatune-memnode.xml: Adjust test.
Signed-off-by: NEric Blake <eblake@redhat.com>

Signed-off-by: NMartin Kletzander <mkletzan@redhat.com>

Signed-off-by: NMartin Kletzander <mkletzan@redhat.com>

In XML format, by definition, order of fields should not matter, so
order of parsing the elements doesn't affect the end result.  When
specifying guest NUMA cells, we depend only on the order of the 'cell'
elements.  With this patch all older domain XMLs are parsed as before,
but with the 'id' attribute they are parsed and formatted according to
that field.  This will be useful when we have tuning settings for
particular guest NUMA node.
Signed-off-by: NMartin Kletzander <mkletzan@redhat.com>

This patch adds support for the QEMU vhost-user feature to libvirt.
vhost-user enables the communication between a QEMU virtual machine
and other userspace process using the Virtio transport protocol.
It uses a char dev (e.g. Unix socket) for the control plane,
while the data plane based on shared memory.

The XML looks like:

<interface type='vhostuser'>
    <mac address='52:54:00:3b:83:1a'/>
    <source type='unix' path='/tmp/vhost.sock' mode='server'/>
    <model type='virtio'/>
</interface>
Signed-off-by: NMichele Paolino <m.paolino@virtualopensystems.com>
Signed-off-by: NMichal Privoznik <mprivozn@redhat.com>

This introduces two new attributes "cmd_per_lun" and "max_sectors" same
with the names QEMU uses for virtio-scsi. An example of the XML:

<controller type='scsi' index='0' model='virtio-scsi' cmd_per_lun='50'
max_sectors='512'/>

The corresponding QEMU command line:

-device virtio-scsi-pci,id=scsi0,cmd_per_lun=50,max_sectors=512,
bus=pci.0,addr=0x3
Signed-off-by: NMike Perez <thingee@gmail.com>
Signed-off-by: NJán Tomko <jtomko@redhat.com>

Commit 7c6fc394 introduced a regression in the XML produced for older
clients.  The argument at the time was that clients shouldn't be
depending on output-only data for something that is only going to
be triggered for a transient guest; but John Ferlan reported that
the automated testsuite was such a client.  It's better to be safe
than sorry by guaranteeing back-compat cruft.  Note that later
patches will be using <mirror> for active block commit, but there
we don't have to worry about back-compat.

* src/conf/domain_conf.c (virDomainDiskDefFormat): Restore old
style output when necessary.
* docs/schemas/domaincommon.rng: Validate back-compat style.
* docs/formatdomain.html.in: Update the documentation.
* tests/qemuxml2xmloutdata/qemuxml2xmlout-disk-mirror-old.xml:
Update tests.
* tests/qemuxml2argvdata/qemuxml2argv-disk-mirror.xml: Likewise.
Signed-off-by: NEric Blake <eblake@redhat.com>

Now that we track a disk mirror as a virStorageSource, we might
as well update the XML to theoretically allow any type of
mirroring destination (not just a local file).  A later patch
will also be reusing <mirror> to track the block commit of the
top layer of a chain, which is another case where libvirt needs
to update the backing chain after the job is finally pivoted,
and since backing chains can have network backing files as the
destination to commit into, it makes more sense to display that
in the XML.

This patch changes output-only XML; it was already documented
that <mirror> does not affect a domain definition at this point
(because qemu doesn't provide persistent bitmaps yet).  Any
application that was starting a block copy job with older libvirt
and then relying on the domain XML to determine if it was
complete will no longer be able to access the file= and format=
attributes of mirror that were previously used.  However, this is
not going to be a problem in practice: the only time a block copy
job works is on a transient domain, and any app that is managing
a transient domain probably already does enough of its own
bookkeeping to know which file it is mirroring into without
having to re-read it from the libvirt XML.  The one thing that
was likely to be used in a mirroring job was the ready=
attribute, which is unchanged.  Meanwhile, I made sure the schema
and parser still accept the old format, even if we no longer
output it, so that upgrading from an older version of libvirt is
seamless.

* docs/schemas/domaincommon.rng (diskMirror): Alter definition.
* src/conf/domain_conf.c (virDomainDiskDefParseXML): Parse two
styles of mirror elements.
(virDomainDiskDefFormat): Output new style.
* tests/qemuxml2argvdata/qemuxml2argv-disk-mirror-old.xml: New
file, copied from...
* tests/qemuxml2argvdata/qemuxml2argv-disk-mirror.xml: ...here
before modernizing.
* tests/qemuxml2xmloutdata/qemuxml2xmlout-disk-mirror-old*: New
files.
* tests/qemuxml2xmltest.c (mymain): Test both styles.
Signed-off-by: NEric Blake <eblake@redhat.com>

ppce500v2 is not machine supported by official release of QEMU.
It should be replaced by ppce500.
Signed-off-by: NJán Tomko <jtomko@redhat.com>

The XML for quite a longish backing chain is shown below:

  <disk type='network' device='disk'>
    <driver name='qemu' type='qcow2'/>
    <source protocol='nbd' name='bar'>
      <host transport='unix' socket='/var/run/nbdsock'/>
    </source>
    <backingStore type='block' index='1'>
      <format type='qcow2'/>
      <source dev='/dev/HostVG/QEMUGuest1'/>
      <backingStore type='file' index='2'>
        <format type='qcow2'/>
        <source file='/tmp/image2.qcow'/>
        <backingStore type='file' index='3'>
          <format type='qcow2'/>
          <source file='/tmp/image3.qcow'/>
          <backingStore type='file' index='4'>
            <format type='qcow2'/>
            <source file='/tmp/image4.qcow'/>
            <backingStore type='file' index='5'>
              <format type='qcow2'/>
              <source file='/tmp/image5.qcow'/>
              <backingStore type='file' index='6'>
                <format type='raw'/>
                <source file='/tmp/Fedora-17-x86_64-Live-KDE.iso'/>
                <backingStore/>
              </backingStore>
            </backingStore>
          </backingStore>
        </backingStore>
      </backingStore>
    </backingStore>
    <target dev='vdb' bus='virtio'/>
  </disk>

Various disk types and formats can be mixed in one chain. The
<backingStore/> empty element marks the end of the backing chain and it
is there mostly for future support of parsing the chain provided by a
user. If it's missing, we are supposed to probe for the rest of the
chain ourselves, otherwise complete chain was provided by the user. The
index attributes of backingStore elements can be used to unambiguously
identify a specific part of the image chain.
Signed-off-by: NJiri Denemark <jdenemar@redhat.com>

I noticed that depending on the <driver> attributes the user passed
in, the output may omit the <driver> element altogether.  For example,
the rerror_policy has had this problem since commit 4bb4109f in Oct
2011.  But in adding testsuite coverage to expose it, I found another
problem: the C code is just fine without a driver name, but the
XML validator required either a name or a cache mode.

* src/conf/domain_conf.c (virDomainDiskDefFormat): Update
conditional.
* docs/schemas/domaincommon.rng (diskDriver): Simplify.
* tests/qemuxml2argvdata/qemuxml2argv-disk-drive-copy-on-read.xml:
* tests/qemuxml2argvdata/qemuxml2argv-disk-drive-copy-on-read.args:
New files.
* tests/qemuxml2argvdata/qemuxml2argv-disk-drive-discard.xml:
Enhance test.
* tests/qemuxml2xmloutdata/qemuxml2xmlout-disk-drive-discard.xml:
Likewise.
* tests/qemuxml2argvtest.c (mymain): New test.
* tests/qemuxml2xmltest.c (mymain): Likewise.
Signed-off-by: NEric Blake <eblake@redhat.com>
Signed-off-by: NJiri Denemark <jdenemar@redhat.com>

To make <disk> schema more maintainable and to allow for moving the
pieces to a common file in the future. It relies on the ability to
override definitions as part of an include, set up in the previous
patch.

The diff is a bit hard to read, because it mixes reindentation
with refactoring; 'git diff -b --patience' may help.

* docs/schemas/domaincommon.rng (disk): Refactor into pieces.
(diskSource, diskSourceFile, diskSourceBlock, diskSourceDir)
(diskSourceVolume: New defines.
(diskSourceNetwork): Revise scope.
* docs/schemas/domainsnapshot.rng (disksnapshot): Adjust.
* tests/domainsnapshotxml2xmlin/disk-seclabel-invalid.xml,
tests/domainsnapshotxml2xmlin/disk-network-seclabel-invalid.xml: New
tests to check seclabel is forbidden in domain snapshot by schema.
Signed-off-by: NEric Blake <eblake@redhat.com>
Signed-off-by: NJiri Denemark <jdenemar@redhat.com>

This patch is my first experience playing with nested grammars,
as documented in http://relaxng.org/tutorial-20011203.html#IDA3PZR.
I plan on doing more overrides in order to make the RelaxNG
grammar mirror the C code refactoring into a common
virStorageSource, but where different clients of that source do
not support the same subset of functionality.  By starting with
something fairly easy to validate, I can make sure my later
patches will be possible.

This patch adds a use of the no-op <ref
name='sourceStartupPolicy'/> to the disksnapshot definition, so
that the snapshot version of a type='file' <source> more closely
resembles the version in domaincommon.  A future patch will merge
the two files into using a common define, but this patch is
sufficient for testing that adding <source
startupPolicy='optional'/> in any of the
tests/domainsnapshotxml2xmlin/*.xml files still gets rejected
unless it occurs within the <domain> subelement, because the
definition of startupPolicy is empty outside of domain.rng.

* docs/schemas/storagecommon.rng (storageStartupPolicy)
(storageSourceExtra): Create no-op defaults.
* docs/schemas/domainsnapshot.rng (domain): Use nested grammar
to avoid restricting <domain>.
(storageSourceExtra): Create new override.
(disksnapshot): Access overrides through common names.
* docs/schemas/domaincommon.rng (disk): Access overrides through
common names.
* docs/schemas/domain.rng (storageStartupPolicy)
(storageSourceExtra): Create new overrides.
Signed-off-by: NEric Blake <eblake@redhat.com>
Signed-off-by: NJiri Denemark <jdenemar@redhat.com>

We had incomplete RelaxNG support for storage formats listed
in virstoragefile.h: commit 027bf2ea added 'vdi' but forgot
to update the <volume> and <domain> xml lists; the <volume>
list was also missing 'fat' and 'vhd'.  Maintaining two lists
is a recipe for them getting out of sync, so make the list
common so that both contexts benefit the next time we add a
format in a single location.

* docs/schemas/domaincommon.rng (storageFormat): Move...
* docs/schemas/storagecommon.rng: ...here, and add vdi.
* docs/schemas/storagevol.rng (formatfile): Use common list.
Signed-off-by: NEric Blake <eblake@redhat.com>

In general, we try to make virt-xml-validate tolerant of input
elements in any order when possible.  However, as written, the
RNG grammar did not permit <source> unless there was an explicit
type= attribute (even though the C code manages just fine by
defaulting to type='file').  After making the attribute optional
on the 'file' branch, I noticed that the use of diskspec was now
redundant with the branch when no <source> was supplied.

View this patch with 'git diff -b' for a better picture of the
schema change.

* docs/schemas/domaincommon.rng (disk): Hoist 'diskspec' out of
choice, make type='file' default, and still preserve interleave.
* tests/qemuxml2xmloutdata/qemuxml2xmlout-disk-source-pool.xml:
* tests/qemuxml2xmloutdata/qemuxml2xmlout-disk-drive-discard.xml:
New files.
* tests/qemuxml2argvdata/qemuxml2argv-disk-source-pool.xml:
* tests/qemuxml2argvdata/qemuxml2argv-disk-drive-discard.xml:
Reorder XML.
* tests/qemuxml2xmltest.c (mymain): Cover new files.
Signed-off-by: NEric Blake <eblake@redhat.com>