This patch introduces the new forward mode='hostdev' along with
attribute managed. Includes updates to the network RNG and new xml
parser/formatter code.
Signed-off-by: NShradha Shah <sshah@solarflare.com>

The following config elements now support a <vlan> subelements:

within a domain: <interface>, and the <actual> subelement of <interface>
within a network: the toplevel, as well as any <portgroup>

Each vlan element must have one or more <tag id='n'/> subelements.  If
there is more than one tag, it is assumed that vlan trunking is being
requested. If trunking is required with only a single tag, the
attribute "trunk='yes'" should be added to the toplevel <vlan>
element.

Some examples:

  <interface type='hostdev'/>
    <vlan>
      <tag id='42'/>
    </vlan>
    <mac address='52:54:00:12:34:56'/>
    ...
  </interface>

  <network>
    <name>vlan-net</name>
    <vlan trunk='yes'>
      <tag id='30'/>
    </vlan>
    <virtualport type='openvswitch'/>
  </network>

  <interface type='network'/>
    <source network='vlan-net'/>
    ...
  </interface>

  <network>
    <name>trunk-vlan</name>
    <vlan>
      <tag id='42'/>
      <tag id='43'/>
    </vlan>
    ...
  </network>

  <network>
    <name>multi</name>
    ...
    <portgroup name='production'/>
      <vlan>
        <tag id='42'/>
      </vlan>
    </portgroup>
    <portgroup name='test'/>
      <vlan>
        <tag id='666'/>
      </vlan>
    </portgroup>
  </network>

  <interface type='network'/>
    <source network='multi' portgroup='test'/>
    ...
  </interface>

IMPORTANT NOTE: As of this patch there is no backend support for the
vlan element for *any* network device type. When support is added in
later patches, it will only be for those select network types that
support setting up a vlan on the host side, without the guest's
involvement. (For example, it will be possible to configure a vlan for
a guest connected to an openvswitch bridge, but it won't be possible
to do that for one that is connected to a standard Linux host bridge.)

<portgroup> allows a <bandwidth> element, but the schema didn't have
this. Since this makes for multiple elements in portgroup, they must
be interleaved.

<interface type='bridge'> needs to allow <virtualport> elements
for openvswitch, but the schema didn't allow this.

Signed-off-by: NDmitry Guryanov <dguryanov@parallels.com>

Just as each physical device used by a network has a connections
counter, now each network has a connections counter which is
incremented once for each guest interface that connects using this
network.

The count is output in the live network XML, like this:

   <network connections='20'>
   ...
   </network>

It is read-only, and for informational purposes only - it isn't used
internally anywhere by libvirt.

Until now, all attributes in a <virtualport> parameter list that were
acceptable for a particular type, were also required. There were no
optional attributes.

One of the aims of supporting <virtualport> in libvirt's virtual
networks and portgroups is to allow specifying the group-wide
parameters in the network's virtualport, and merge that with the
interface's virtualport, which will have the instance-specific info
(i.e. the interfaceid or instanceid).

Additionally, the guest's interface XML shouldn't need to know what
type of network connection will be used prior to runtime - it could be
openvswitch, 802.1Qbh, 802.1Qbg, or none of the above - but should
still be able to specify instance-specific info just in case it turns
out to be applicable.

Finally, up to now, the parser for virtualport has always generated a
random instanceid/interfaceid when appropriate, making it impossible
to leave it blank (which is what's required for virtualports within a
network/portprofile definition).

This patch modifies the parser and formatter of the <virtualport>
element in the following ways:

* because most of the attributes in a virNetDevVPortProfile are fixed
  size binary data with no reserved values, there is no way to embed a
  "this value wasn't specified" sentinel into the existing data. To
  solve this problem, the new *_specified fields in the
  virNetDevVPortProfile object that were added in a previous patch of
  this series are now set when the corresponding attribute is present
  during the parse.

* allow parsing/formatting a <virtualport> that has no type set. In
  this case, all fields are settable, but all are also optional.

* add a GENERATE_MISSING_DEFAULTS flag to the parser - if this flag is
  set and an instanceid/interfaceid is expected but not provided, a
  random one will be generated. This was previously the default
  behavior, but is now done only for virtualports inside an
  <interface> definition, not for those in <network> or <portgroup>.

* add a REQUIRE_ALL_ATTRIBUTES flag to the parser - if this flag is
  set the parser will call the new
  virNetDevVPortProfileCheckComplete() functions at the end of the
  parser to check for any missing attributes (based on type), and
  return failure if anything is missing. This used to be default
  behavior. Now it is only used for the virtualport defined inside an
  interface's <actual> element (by the time you've figured out the
  contents of <actual>, you should have all the necessary data to fill
  in the entire virtualport)

* add a REQUIRE_TYPE flag to the parser - if this flag is set, the
  parser will return an error if the virtualport has no type
  attribute. This also was previously the default behavior, but isn't
  needed in the case of the virtualport for a type='network' interface
  (i.e. the exact type isn't yet known), or the virtualport of a
  portgroup (i.e. the portgroup just has modifiers for the network's
  virtualport, which *does* require a type) - in those cases, the
  check will be done at domain startup, once the final virtualport is
  assembled (this is handled in the next patch).

The access, birth, modification and change times are added to
storage volumes and corresponding xml representations.  This
shows up in the XML in this format:

<timestamps>
  <atime>1341933637.027319099</atime>
  <mtime>1341933637.027319099</mtime>
</timestamps>
Signed-off-by: NEric Blake <eblake@redhat.com>

capability.rng: Guest features can be in any order.
nodedev.rng: Added <driver> element, <capability> phys_function and
virt_functions for PCI devices.
storagepool.rng: Owner or group ID can be -1.

schema tests: New capabilities and nodedev files; changed owner and
group to -1 in pool-dir.xml.
storage_conf: Print uid_t and gid_t as signed to storage pool XML.

Libvirt adds a USB controller to the guest even if the user does not
specify any in the XML. This is due to back-compat reasons.

To allow disabling USB for a guest this patch adds a new USB controller
type "none" that disables USB support for the guest.

This patch brings support to manage sheepdog pools and volumes to libvirt.
It uses the "collie" command-line utility that comes with sheepdog for that.

A sheepdog pool in libvirt maps to a sheepdog cluster.
It needs a host and port to connect to, which in most cases
is just going to be the default of localhost on port 7000.

A sheepdog volume in libvirt maps to a sheepdog vdi.
To create one specify the pool, a name and the capacity.
Volumes can also be resized later.

In the volume XML the vdi name has to be put into the <target><path>.
To use the volume as a disk source for virtual machines specify
the vdi name as "name" attribute of the <source>.
The host and port information from the pool are specified inside the host tag.

  <disk type='network'>
    ...
    <source protocol="sheepdog" name="vdi_name">
      <host name="localhost" port="7000"/>
    </source>
  </disk>

To work right this patch parses the output of collie,
so it relies on the raw output option. There recently was a bug which caused
size information to be reported wrong. This is fixed upstream already and
will be in the next release.
Signed-off-by: NSebastian Wiedenroth <wiedi@frubar.net>

Added s390-virtio machine type to the XML schema for domains in order
to not fail the domain schema tests.
Signed-off-by: NViktor Mihajlovski <mihajlov@linux.vnet.ibm.com>

Introducing the attribute vendor_id to force the CPUID instruction
in a kvm guest to return the specified vendor.

Currently you can configure LXC to bind a host directory to
a guest directory, but not to bind a guest directory to a
guest directory. While the guest container init could do
this itself, allowing it in the libvirt XML means a stricter
SELinux policy can be written

Introduce a new syntax for filesystems to allow use of a RAM
filesystem

   <filesystem type='ram'>
      <source usage='10' units='MiB'/>
      <target dir='/mnt'/>
   </filesystem>

The usage units default to KiB to limit consumption of host memory.

* docs/formatdomain.html.in: Document new syntax
* docs/schemas/domaincommon.rng: Add new attributes
* src/conf/domain_conf.c: Parsing/formatting of RAM filesystems
* src/lxc/lxc_container.c: Mounting of RAM filesystems
Signed-off-by: NDaniel P. Berrange <berrange@redhat.com>

qemu 1.1 features a xhci controller,
this patch adds support for it.
Signed-off-by: NGerd Hoffmann <kraxel@redhat.com>

as we support qed format as well.

'boot' tag shouldn't be exclusive with 'kernel', 'initrd', and 'cmdline',
though the boot sequence doesn't make sense when the guest boots from
kernel directly. But it's useful if booting from kernel is to install
a newguest, even if it's not to install a guest, there is no hurt. And
on the other hand, we allow 'boot' and the kernel tags when parsing.

This patch adds support for a new storage backend with RBD support.

RBD is the RADOS Block Device and is part of the Ceph distributed storage
system.

It comes in two flavours: Qemu-RBD and Kernel RBD, this storage backend only
supports Qemu-RBD, thus limiting the use of this storage driver to Qemu only.

To function this backend relies on librbd and librados being present on the
local system.

The backend also supports Cephx authentication for safe authentication with
the Ceph cluster.

For storing credentials it uses the built-in secret mechanism of libvirt.
Signed-off-by: NWido den Hollander <wido@widodh.nl>

This patch adds support for the recent ipset iptables extension
to libvirt's nwfilter subsystem. Ipset allows to maintain 'sets'
of IP addresses, ports and other packet parameters and allows for
faster lookup (in the order of O(1) vs. O(n)) and rule evaluation
to achieve higher throughput than what can be achieved with
individual iptables rules.

On the command line iptables supports ipset using

iptables ... -m set --match-set <ipset name> <flags> -j ...

where 'ipset name' is the name of a previously created ipset and
flags is a comma-separated list of up to 6 flags. Flags use 'src' and 'dst'
for selecting IP addresses, ports etc. from the source or
destination part of a packet. So a concrete example may look like this:

iptables -A INPUT -m set --match-set test src,src -j ACCEPT

Since ipset management is quite complex, the idea was to leave ipset 
management outside of libvirt but still allow users to reference an ipset.
The user would have to make sure the ipset is available once the VM is
started so that the iptables rule(s) referencing the ipset can be created.

Using XML to describe an ipset in an nwfilter rule would then look as
follows:

  <rule action='accept' direction='in'>
    <all ipset='test' ipsetflags='src,src'/>
  </rule>

The two parameters on the command line are also the two distinct XML attributes
'ipset' and 'ipsetflags'.

FYI: Here is the man page for ipset:

https://ipset.netfilter.org/ipset.man.html

Regards,
    Stefan

Allow specifying sound device codecs. See formatdomain.html for
more details.

Though numad will manage the memory allocation of task dynamically,
it wants management application (libvirt) to pre-set the memory
policy according to the advisory nodeset returned from querying numad,
(just like pre-bind CPU nodeset for domain process), and thus the
performance could benefit much more from it.

This patch introduces new XML tag 'placement', value 'auto' indicates
whether to set the memory policy with the advisory nodeset from numad,
and its value defaults to the value of <vcpu> placement, or 'static'
if 'nodeset' is specified. Example of the new XML tag's usage:

  <numatune>
    <memory placement='auto' mode='interleave'/>
  </numatune>

Just like what current "numatune" does, the 'auto' numa memory policy
setting uses libnuma's API too.

If <vcpu> "placement" is "auto", and <numatune> is not specified
explicitly, a default <numatume> will be added with "placement"
set as "auto", and "mode" set as "strict".

The following XML can now fully drive numad:

1) <vcpu> placement is 'auto', no <numatune> is specified.

   <vcpu placement='auto'>10</vcpu>

2) <vcpu> placement is 'auto', no 'placement' is specified for
   <numatune>.

   <vcpu placement='auto'>10</vcpu>
   <numatune>
     <memory mode='interleave'/>
   </numatune>

And it's also able to control the CPU placement and memory policy
independently. e.g.

1) <vcpu> placement is 'auto', and <numatune> placement is 'static'

   <vcpu placement='auto'>10</vcpu>
   <numatune>
     <memory mode='strict' nodeset='0-10,^7'/>
   </numatune>

2) <vcpu> placement is 'static', and <numatune> placement is 'auto'

   <vcpu placement='static' cpuset='0-24,^12'>10</vcpu>
   <numatune>
     <memory mode='interleave' placement='auto'/>
   </numatume>

A follow up patch will change the XML formatting codes to always output
'placement' for <vcpu>, even it's 'static'.

qemu's behavior in this case is to change the spice server behavior to
require secure connection to any channel not otherwise specified as
being in plaintext mode. libvirt doesn't currently allow requesting this
(via plaintext-channel=<channel name>).

RHBZ: 819499
Signed-off-by: NAlon Levy <alevy@redhat.com>

Add "usbredir" channel to list of recognized spice channels.

RHBZ: 819498
Signed-off-by: NAlon Levy <alevy@redhat.com>

In order to track a block copy job across libvirtd restarts, we
need to save internal XML that tracks the name of the file
holding the mirror.  Displaying this name in dumpxml might also
be useful to the user, even if we don't yet have a way to (re-)
start a domain with mirroring enabled up front.  This is done
with a new <mirror> sub-element to <disk>, as in:

    <disk type='file' device='disk'>
      <driver name='qemu' type='raw'/>
      <source file='/var/lib/libvirt/images/original.img'/>
      <mirror file='/var/lib/libvirt/images/copy.img' format='qcow2' ready='yes'/>
      ...
    </disk>

For now, the element is output-only, in live domains; it is ignored
when defining a domain or hot-plugging a disk (since those contexts
use VIR_DOMAIN_XML_INACTIVE in parsing).  The 'ready' attribute appears
when libvirt knows that the job has changed from the initial pulling
phase over to the mirroring phase, although absence of the attribute
is not a sure indicator of the current phase.  If we come up with a way
to make qemu start with mirroring enabled, we can relax the xml
restriction, and allow <mirror> (but not attribute 'ready') on input.
Testing active-only XML meant tweaking the testsuite slightly, but it
was worth it.

* docs/schemas/domaincommon.rng (diskspec): Add diskMirror.
* docs/formatdomain.html.in (elementsDisks): Document it.
* src/conf/domain_conf.h (_virDomainDiskDef): New members.
* src/conf/domain_conf.c (virDomainDiskDefFree): Clean them.
(virDomainDiskDefParseXML): Parse them, but only internally.
(virDomainDiskDefFormat): Output them.
* tests/qemuxml2argvdata/qemuxml2argv-disk-mirror.xml: New test file.
* tests/qemuxml2xmloutdata/qemuxml2xmlout-disk-mirror.xml: Likewise.
* tests/qemuxml2xmltest.c (testInfo): Alter members.
(testCompareXMLToXMLHelper): Allow more test control.
(mymain): Run new test.

<filesystemtgt> is redundant, as every group uses it; <address>
shouldn't be in <filesystemtgt> in case of the meaning could be
"filesystemtarget"; The elements <address>, <alias>, <target>,
... should be interleaved.

Since Xen 3.1 the clock=variable semantic is supported. In addition to
qemu/kvm Xen also knows about a variant where the offset is relative to
'localtime' instead of 'utc'.

Extends the libvirt structure with a flag 'basis' to specify, if the
offset is relative to 'localtime' or 'utc'.

Extends the libvirt structure with a flag 'reset' to force the reset
behaviour of 'localtime' and 'utc'; this is needed for backward
compatibility with previous versions of libvirt, since they report
incorrect XML.

Adapt the only user 'qemu' to the new name.
Extend the RelaxNG schema accordingly.
Document the new 'basis' attribute in the HTML documentation.
Adapt test for the new attribute.
Signed-off-by: NPhilipp Hahn <hahn@univention.de>

* docs/schemas/domaincommon.rng: Add missing lxc & openvz
  console target types. Allow arch on LXC <os> type element
* tests/domainschematest: Include tests/lxcxml2xmldata/

Pass argv to the init binary of LXC, using a new <initarg> element.

* docs/formatdomain.html.in: Document <os> usage for containers
* docs/schemas/domaincommon.rng: Add <initarg> element
* src/conf/domain_conf.c, src/conf/domain_conf.h: parsing and
  formatting of <initarg>
* src/lxc/lxc_container.c: Setup LXC argv
* tests/Makefile.am, tests/lxcxml2xmldata/lxc-systemd.xml,
  tests/lxcxml2xmltest.c, tests/testutilslxc.c,
  tests/testutilslxc.h: Test parsing/formatting of LXC related
  XML parts

Example XML:

    <disk type='file' device='cdrom'>
      <driver name='qemu' type='raw'/>
      <source file='/tmp/cdrom.img'/>
      <target dev='hdd' bus='ide' tray='open'/>
    </disk>

A few times libvirt users manually setting mac addresses have
complained of a networking failure that ends up being due to a multicast
mac address being used for a guest interface. This patch prevents that
by logging an error and failing if a multicast mac address is
encountered in each of the three following cases:

1) domain xml <interface> mac address.
2) network xml bridge mac address.
3) network xml dhcp/host mac address.

There are several other places where a mac address can be input that
aren't controlled in this manner because failure to do so has no
consequences (e.g., if the address will be used to search through
existing interfaces for a match).

The RNG has been updated to add multiMacAddr and uniMacAddr along with
the existing macAddr, and macAddr was switched to uniMacAddr where
appropriate.

If no <interface> elements are included in an LXC guest XML
description, then the LXC guest will just see the host's
network interfaces. It is desirable to be able to hide the
host interfaces, without having to define any guest interfaces.

This patch introduces a new feature flag <privnet/> to allow
forcing of a private network namespace for LXC. In the future
I also anticipate that we will add <privuser/> to force a
private user ID namespace.

* src/conf/domain_conf.c, src/conf/domain_conf.h: Add support
  for <privnet/> feature. Auto-set <privnet> if any <interface>
  devices are defined
* src/lxc/lxc_container.c: Honour request for private network
  namespace

numad is an user-level daemon that monitors NUMA topology and
processes resource consumption to facilitate good NUMA resource
alignment of applications/virtual machines to improve performance
and minimize cost of remote memory latencies. It provides a
pre-placement advisory interface, so significant processes can
be pre-bound to nodes with sufficient available resources.

More details: http://fedoraproject.org/wiki/Features/numad

"numad -w ncpus:memory_amount" is the advisory interface numad
provides currently.

This patch add the support by introducing a new XML attribute
for <vcpu>. e.g.

  <vcpu placement="auto">4</vcpu>
  <vcpu placement="static" cpuset="1-10^6">4</vcpu>

The returned advisory nodeset from numad will be printed
in domain's dumped XML. e.g.
  <vcpu placement="auto" cpuset="1-10^6">4</vcpu>

If placement is "auto", the number of vcpus and the current
memory amount specified in domain XML will be used for numad
command line (numad uses MB for memory amount):
  numad -w $num_of_vcpus:$current_memory_amount / 1024

The advisory nodeset returned from numad will be used to set
domain process CPU affinity then. (e.g. qemuProcessInitCpuAffinity).

If the user specifies both CPU affinity policy (e.g.
(<vcpu cpuset="1-10,^7,^8">4</vcpu>) and placement == "auto"
the specified CPU affinity will be overridden.

Only QEMU/KVM drivers support it now.

See docs update in patch for more details.

If there is a disk file with a comma in the name, QEmu expects a double
comma instead of a single one (e.g., the file "virtual,disk.img" needs
to be specified as "virtual,,disk.img" in QEmu's command line). This
patch fixes libvirt to work with that feature. Fix RHBZ #801036.

Based on an initial patch by Crístian Viana.

* src/util/buf.h (virBufferEscape): Alter signature.
* src/util/buf.c (virBufferEscape): Add parameter.
(virBufferEscapeSexpr): Fix caller.
* src/qemu/qemu_command.c (qemuBuildRBDString): Likewise.  Also
escape commas in file names.
(qemuBuildDriveStr): Escape commas in file names.
* docs/schemas/basictypes.rng (absFilePath): Relax RNG to allow
commas in input file names.
* tests/qemuxml2argvdata/*-disk-drive-network-sheepdog.*: Update
test.
Signed-off-by: NEric Blake <eblake@redhat.com>

spice agent-mouse support

Usage:
  <graphics type='spice'>
    <mouse mode='client'|'server'/>
  <graphics/>
Signed-off-by: NOsier Yang <jyang@redhat.com>

The schema files contained duplicate data types that can be shared from
the basictypes.rng file.

Output is still in kibibytes, but input can now be in different
scales for ease of typing.

* src/conf/domain_conf.c (virDomainParseMemory): New helper.
(virDomainDefParseXML): Use it when parsing.
* docs/schemas/domaincommon.rng: Expand XML; rename memoryKBElement
to memoryElement and update callers.
* docs/formatdomain.html.in (elementsMemoryAllocation): Document
scaling.
* tests/qemuxml2argvdata/qemuxml2argv-memtune.xml: Adjust test.
* tests/qemuxml2xmltest.c: Likewise.
* tests/qemuxml2xmloutdata/qemuxml2xmlout-memtune.xml: New file.

The test domain allows <memory>0</memory>, but the RNG was stating
that memory had to be at least 4096000 bytes.  Hypervisors should
enforce their own limits, rather than complicating the RNG.

Meanwhile, some copy and paste had introduced some fishy constructs
in various unit tests.

* docs/schemas/domaincommon.rng (memoryKB, memoryKBElement): Drop
limit that isn't enforced in code.
* src/conf/domain_conf.c (virDomainDefParseXML): Require current
<= maximum.
* tests/qemuxml2argvdata/*.xml: Fix offenders.

Disk manufacturers are fond of quoting sizes in powers of 10,
rather than powers of 2 (after all, 2.1 GB sounds larger than
2.0 GiB, even though the exact opposite is true).  So, we might
as well follow coreutils' lead in supporting three types of
suffix: single letter ${u} (which we already had) and ${u}iB
for the power of 2, and ${u}B for power of 10.

Additionally, it is impossible to create a file with more than
2**63 bytes, since off_t is signed (if you have enough storage
to even create one 8EiB file, I'm jealous).  This now reports
failure up front rather than down the road when the kernel
finally refuses an impossible size.

* docs/schemas/basictypes.rng (unit): Add suffixes.
* src/conf/storage_conf.c (virStorageSize): Use new function.
* docs/formatstorage.html.in: Document it.
* tests/storagevolxml2xmlin/vol-file-backing.xml: Test it.
* tests/storagevolxml2xmlin/vol-file.xml: Likewise.

Make it obvious to 'dumpxml' readers what unit we are using,
since our default of KiB for memory (1024) differs from qemu's
default of MiB; and differs from our use of bytes for storage.

Tests were updated via:

$ find tests/*data tests/*out -name '*.xml' | \
  xargs sed -i 's/<\(memory\|currentMemory\|hard_limit\|soft_limit\|min_guarantee\|swap_hard_limit\)>/<\1 unit='"'KiB'>/"
$ find tests/*data tests/*out -name '*.xml' | \
  xargs sed -i 's/<\(capacity\|allocation\|available\)>/<\1 unit='"'bytes'>/"

followed by a few fixes for the stragglers.

Note that with this patch, the RNG for <memory> still forbids
validation of anything except unit='KiB', since the code silently
ignores the attribute; a later patch will expand <memory> to allow
scaled input in the code and update the RNG to match.

* docs/schemas/basictypes.rng (unit): Add 'bytes'.
(scaledInteger): New define.
* docs/schemas/storagevol.rng (sizing): Use it.
* docs/schemas/storagepool.rng (sizing): Likewise.
* docs/schemas/domaincommon.rng (memoryKBElement): New define; use
for memory elements.
* src/conf/storage_conf.c (virStoragePoolDefFormat)
(virStorageVolDefFormat): Likewise.
* src/conf/domain_conf.h (_virDomainDef): Document unit used
internally.
* src/conf/storage_conf.h (_virStoragePoolDef, _virStorageVolDef):
Likewise.
* tests/*data/*.xml: Update all tests.
* tests/*out/*.xml: Likewise.
* tests/define-dev-segfault: Likewise.
* tests/openvzutilstest.c (testReadNetworkConf): Likewise.
* tests/qemuargv2xmltest.c (blankProblemElements): Likewise.