Recently CPU hardware vendors have started to support a new structure
inside the CPU package topology known as a "die". Thus the hierarchy
is now:

  sockets > dies > cores > threads

This adds support for "dies" in the XML parser, with the value
defaulting to 1 if not specified for backwards compatibility.

For example a system with 64 logical CPUs might report

   <topology sockets="4" dies="2" cores="4" threads="2"/>
Reviewed-by: NJiri Denemark <jdenemar@redhat.com>
Reviewed-by: NDaniel Henrique Barboza <danielhb413@gmail.com>
Signed-off-by: NDaniel P. Berrangé <berrange@redhat.com>

The 32-bit x86 binary is called qemu-system-i386, not
qemu-system-i686. This mistake across many test XML files was
not noticed because the mistake was also made in testutilsqemu.c
when mocking the capabilities.
Reviewed-by: NErik Skultety <eskultet@redhat.com>
Signed-off-by: NDaniel P. Berrangé <berrange@redhat.com>

Nobody should use format detection due to security implications. The
result of the change is that 'raw' format will be printed unless
specified explicitly.
Signed-off-by: NPeter Krempa <pkrempa@redhat.com>

These XMLs live in a separate directory, there's no need for them
to have a special prefix in addition. It also doesn't play nicely
with ':e' completion in Vim, finding proper file based on
qemuxml2argvtest.c is also needlessly complicated.

The files were renamed using the following commands. From
qemuxml2argvdata:

  for i in qemuxml2argv-*.xml; do mv $i ${i#qemuxml2argv-}; done

and then (to fix broken symlinks) from qemuxml2argvdata and
qemuxml2xmloutdata:

  for i in $(find . -xtype l); do \
      ln -sf $(readlink $i | sed 's/qemuxml2argv-//') $i;
  done
Signed-off-by: NMichal Privoznik <mprivozn@redhat.com>

Our test data used a lot of different qemu binary paths and some
of them were based on downstream systems.

Note that there is one file where I had to add "accel=kvm" because
the qemuargv2xml code parses "/usr/bin/kvm" as virt type="kvm".
Signed-off-by: NPavel Hrdina <phrdina@redhat.com>

Now that NVDIMM has found its way into libvirt, users might want
to fine tune some settings for each module separately. One such
setting is 'share=on|off' for the memory-backend-file object.
This setting - just like its name suggest already - enables
sharing the nvdimm module with other applications. Under the hood
it controls whether qemu mmaps() the file as MAP_PRIVATE or
MAP_SHARED.

Yet again, we have such config knob in domain XML, but it's just
an attribute to numa <cell/>. This does not give fine enough
tuning on per-memdevice basis so we need to have the attribute
for each device too.
Signed-off-by: NMichal Privoznik <mprivozn@redhat.com>

NVDIMM is new type of memory introduced into QEMU 2.6. The idea
is that we have a Non-Volatile memory module that keeps the data
persistent across domain reboots.

At the domain XML level, we already have some representation of
'dimm' modules. Long story short, NVDIMM will utilize the
existing <memory/> element that lives under <devices/> by adding
a new attribute 'nvdimm' to the existing @model and introduce a
new <path/> element for <source/> while reusing other fields. The
resulting XML would appear as:

    <memory model='nvdimm'>
      <source>
        <path>/tmp/nvdimm</path>
      </source>
      <target>
        <size unit='KiB'>523264</size>
        <node>0</node>
      </target>
      <address type='dimm' slot='0'/>
    </memory>

So far, this is just a XML parser/formatter extension. QEMU
driver implementation is in the next commit.

For more info on NVDIMM visit the following web page:

    http://pmem.io/Signed-off-by: NMichal Privoznik <mprivozn@redhat.com>

As with other devices assign the slot number right away when adding the
device. This will make the slot numbers static as we do with other
addressing elements and it will ultimately simplify allocation of the
alias in a static way which does not break with qemu.

We use the PreFormat callback for this. Many test cases need to be extended
to pass in proper qemuCaps flags so AssignAddresses doesn't throw errors.

One test case (pcie-root-port-too-many) is dropped, since it was meant
only for checking an error condition in qemuxml2argv, and one we add in
AssignAddresses it errors here too.

Long term I think AssignAddresses should be handled in qemu's PostParse
callback, but that's not entirely straightforward. Handling it here
means we can get the test suite churn over with.

Most qemuxml2xml tests expect that the input XML is unchanged after
parsing. This is unlike 99% of new qemu configs in the wild, which after
initial parsing end up with stable PCI device addresses. The xml2xml bit
doesn't currently hit that code path though, so most XML testing indeed
does not change.

Future patches will add that PCI address bits, which means most test cases
will have different output. So let's do away with the hardcoded same vs
different test split, and always track a separate output file. Tests can
still have same input and output, it just necessitates 2 separate XML files.

The current code was a little bit odd.  At first we've removed all
possible implicit input devices from domain definition to add them later
back if there was any graphics device defined while parsing XML
description.  That's not all, while formating domain definition to XML
description we at first ignore any input devices with bus different to
USB and VIRTIO and few lines later we add implicit input devices to XML.

This seems to me as a lot of code for nothing.  This patch may look
to be more complicated than original approach, but this is a preferred
way to modify/add driver specific stuff only in those drivers and not
deal with them in common parsing/formating functions.

The update is to add those implicit input devices into config XML to
follow the real HW configuration visible by guest OS.

There was also inconsistence between our behavior and QEMU's in the way,
that in QEMU there is no way how to disable those implicit input devices
for x86 architecture and they are available always, even without graphics
device.  This applies also to XEN hypervisor.  VZ driver already does its
part by putting correct implicit devices into live XML.
Signed-off-by: NPavel Hrdina <phrdina@redhat.com>

My original implementation was based on a qemu version that still did
not have all the checks in place. Using sizes that would align to odd
megabyte increments will produce the following error:

qemu-kvm: -device pc-dimm,node=0,memdev=memdimm0,id=dimm0: backend memory size must be multiple of 0x200000
qemu-kvm: -device pc-dimm,node=0,memdev=memdimm0,id=dimm0: Device 'pc-dimm' could not be initialized

Introduce an alignment retrieval function for memory devices and use it
to align the devices separately and modify a test case to verify it.

So far the argument has not much meaning and was practically ignored.
This is not good since when doing memory hotplug, the size of desired
hugepage backing is passed in that argument. Taking closer look at the
tests I'm fixing reveals the bug. For instance, while the following is
in the test:

    <memory model='dimm'>
      <source>
        <nodemask>1-3</nodemask>
        <pagesize unit='KiB'>4096</pagesize>
      </source>
      <target>
        <size unit='KiB'>524287</size>
        <node>0</node>
      </target>
      <address type='dimm' slot='0' base='0x100000000'/>
    </memory>

the generated commandline corresponding to this XML was:

    -object memory-backend-ram,id=memdimm0,size=536870912,\
    host-nodes=1-3,policy=bind

Have you noticed? Yes, memory-backend-ram! Nothing can be further away
from the right answer. The hugepage backing is requested in the XML
and we happily ignore it. This is just not right. It's
memory-backend-file which should have been used:

    -object memory-backend-file,id=memdimm0,prealloc=yes,\
    mem-path=/dev/hugepages4M/libvirt/qemu,size=536870912,\
    host-nodes=1-3,policy=bind

The problem is, that @pagesize passed to qemuBuildMemoryBackendStr
(where this part of commandline is built) was ignored. The hugepage to
back memory was searched only and only by NUMA nodes pinning. This
works only for regular guest NUMA nodes.

Then, I'm changing the hugepages size in the test XMLs too. This is
simply because in the test suite we create dummy mount points just for
2M and 1G hugepages. And in the test 4M was requested. I'm sticking to
2M, but 1G should just work too.
Signed-off-by: NMichal Privoznik <mprivozn@redhat.com>

After parsing the memory device XML the function would not restore the
XML parser context causing invalid XPath starting point for the rest of
the elements. This is a regression since 3e4230d2.

The test case addition uses the <idmap> element that is currently unused
by qemu, but parsed after the memory device definition and formatted
always.

Resolves: https://bugzilla.redhat.com/show_bug.cgi?id=1223631Signed-off-by: NLuyao Huang <lhuang@redhat.com>
Signed-off-by: NPeter Krempa <pkrempa@redhat.com>

This patch adds code that parses and formats configuration for memory
devices.

A simple configuration would be:
<memory model='dimm'>
  <target>
    <size unit='KiB'>524287</size>
    <node>0</node>
  </target>
</memory>

A complete configuration of a memory device:
<memory model='dimm'>
  <source>
    <pagesize unit='KiB'>4096</pagesize>
    <nodemask>1-3</nodemask>
  </source>
  <target>
    <size unit='KiB'>524287</size>
    <node>1</node>
  </target>
</memory>

This patch preemptively forbids use of the <memory> device in individual
drivers so the users are warned right away that the device is not
supported.

To enable memory hotplug the maximum memory size and slot count need to
be specified. As qemu supports now other units than mebibytes when
specifying memory, use the new interface in this case.

<controller type='pci' index='0' model='pci-root'/>
is auto-added to pc* machine types.
Without this controller PCI bus 0 is not available and
no PCI addresses are assigned by default.

Since older libvirt supported PCI bus 0 even without
this controller, it is removed from the XML when migrating.

With this patch, one can also fully drive numad by:

  <vcpu>2</vcpu>
  <numatune>
    <memory placement='auto'/>
  </numatune>

New tests are added.

Though numad will manage the memory allocation of task dynamically,
it wants management application (libvirt) to pre-set the memory
policy according to the advisory nodeset returned from querying numad,
(just like pre-bind CPU nodeset for domain process), and thus the
performance could benefit much more from it.

This patch introduces new XML tag 'placement', value 'auto' indicates
whether to set the memory policy with the advisory nodeset from numad,
and its value defaults to the value of <vcpu> placement, or 'static'
if 'nodeset' is specified. Example of the new XML tag's usage:

  <numatune>
    <memory placement='auto' mode='interleave'/>
  </numatune>

Just like what current "numatune" does, the 'auto' numa memory policy
setting uses libnuma's API too.

If <vcpu> "placement" is "auto", and <numatune> is not specified
explicitly, a default <numatume> will be added with "placement"
set as "auto", and "mode" set as "strict".

The following XML can now fully drive numad:

1) <vcpu> placement is 'auto', no <numatune> is specified.

   <vcpu placement='auto'>10</vcpu>

2) <vcpu> placement is 'auto', no 'placement' is specified for
   <numatune>.

   <vcpu placement='auto'>10</vcpu>
   <numatune>
     <memory mode='interleave'/>
   </numatune>

And it's also able to control the CPU placement and memory policy
independently. e.g.

1) <vcpu> placement is 'auto', and <numatune> placement is 'static'

   <vcpu placement='auto'>10</vcpu>
   <numatune>
     <memory mode='strict' nodeset='0-10,^7'/>
   </numatune>

2) <vcpu> placement is 'static', and <numatune> placement is 'auto'

   <vcpu placement='static' cpuset='0-24,^12'>10</vcpu>
   <numatune>
     <memory mode='interleave' placement='auto'/>
   </numatume>

A follow up patch will change the XML formatting codes to always output
'placement' for <vcpu>, even it's 'static'.

Make it obvious to 'dumpxml' readers what unit we are using,
since our default of KiB for memory (1024) differs from qemu's
default of MiB; and differs from our use of bytes for storage.

Tests were updated via:

$ find tests/*data tests/*out -name '*.xml' | \
  xargs sed -i 's/<\(memory\|currentMemory\|hard_limit\|soft_limit\|min_guarantee\|swap_hard_limit\)>/<\1 unit='"'KiB'>/"
$ find tests/*data tests/*out -name '*.xml' | \
  xargs sed -i 's/<\(capacity\|allocation\|available\)>/<\1 unit='"'bytes'>/"

followed by a few fixes for the stragglers.

Note that with this patch, the RNG for <memory> still forbids
validation of anything except unit='KiB', since the code silently
ignores the attribute; a later patch will expand <memory> to allow
scaled input in the code and update the RNG to match.

* docs/schemas/basictypes.rng (unit): Add 'bytes'.
(scaledInteger): New define.
* docs/schemas/storagevol.rng (sizing): Use it.
* docs/schemas/storagepool.rng (sizing): Likewise.
* docs/schemas/domaincommon.rng (memoryKBElement): New define; use
for memory elements.
* src/conf/storage_conf.c (virStoragePoolDefFormat)
(virStorageVolDefFormat): Likewise.
* src/conf/domain_conf.h (_virDomainDef): Document unit used
internally.
* src/conf/storage_conf.h (_virStoragePoolDef, _virStorageVolDef):
Likewise.
* tests/*data/*.xml: Update all tests.
* tests/*out/*.xml: Likewise.
* tests/define-dev-segfault: Likewise.
* tests/openvzutilstest.c (testReadNetworkConf): Likewise.
* tests/qemuargv2xmltest.c (blankProblemElements): Likewise.

* src/conf/domain_conf.h: Add new member "target" to struct
  _virDomainDeviceDriveAddress.

* src/conf/domain_conf.c: Parse and format "target"

* Lots of tests (.xml) in tests/domainsnapshotxml2xmlout,
  tests/qemuxml2argvdata, tests/qemuxml2xmloutdata, and
  tests/vmx2xmldata/ are modified for newly introduced
  attribute "target" for address of "drive" type.

Some tools, such as virt-manager, prefers having the default USB
controller explicit in the XML document. This patch makes sure there
is one. With this patch, it is now possible to switch from USB1 to
USB2 from the release 0.9.1 of virt-manager.

Fix tests to pass with this change.

Use it in all places where a memory or storage request size is converted
to a larger granularity. This avoids requesting too small memory or storage
sizes that could result from the truncation done by a simple division.

This extends the round up fix in 6002e040
to the whole codebase.

Instead of reporting errors for odd values in the VMX code round them up.

Update the QEMU Argv tests accordingly as the original memory size 219200
isn't a even multiple of 1024 and is rounded up to 215 megabyte now. Change
it to 219100 and 219136. Use two different values intentionally to make
sure that rounding up works.

Update virsh.pod accordingly, as rounding down and rejecting are replaced
by rounding up.

Syntax agreed on in
https://www.redhat.com/archives/libvir-list/2010-September/msg00476.html

<domain ...>
  <vcpu current='x'>y</vcpu>
...

can now be used to specify 1 <= x <= y current vcpus, in relation
to the boot-time max of y vcpus.  If current is omitted, then
current and max are assumed to be the same value.

* docs/schemas/domain.rng: Add new attribute.
* docs/formatdomain.html.in: Document it.
* tests/qemuxml2argvdata/qemuxml2argv-smp.xml: Add to
domainschematest.
* tests/xml2sexprdata/xml2sexpr-pv-vcpus.xml: Likewise.

To allow compatibility with older QEMU PCI device slot assignment
it is necessary to explicitly track the balloon device in the
XML. This introduces a new device

   <memballoon model='virtio|xen'/>

It can also have a PCI address, auto-assigned if necessary.

The memballoon will be automatically added to all Xen and QEMU
guests by default.

* docs/schemas/domain.rng: Add <memballoon> element
* src/conf/domain_conf.c, src/conf/domain_conf.h: parsing
  and formatting for memballoon device. Always add a memory
  balloon device to Xen/QEMU if none exists in XML
* src/libvirt_private.syms: Export memballoon model APIs
* src/qemu/qemu_conf.c, src/qemu/qemu_conf.h: Honour the
  PCI device address in memory balloon device
* tests/*: Update to test new functionality

Existing applications using libvirt are not aware of the disk
controller concept. Thus, after parsing the <disk> definitions
in the XML, it is neccessary to create <controller> elements
to satisfy all requested disks, as per their defined drive
addresses

* src/conf/domain_conf.c, src/conf/domain_conf.h,
  src/libvirt_private.syms: Add virDomainDefAddDiskControllers()
  method for populating disk controllers, and call it after
  parsing disk definitions.
* src/qemu/qemu_conf.c: Call virDomainDefAddDiskControllers()
  when doing ARGV -> XML conversion
* tests/qemuxml2argvdata/qemuxml2argv*.xml: Add disk controller
  data to all data files which don't have it already

When parsing the <disk> element specification, if no <address>
is provided for the disk, then automatically assign one based on
the <target dev='sdXX'/> device name. This provides for backwards
compatability with existing applications using libvirt, while also
allowing new apps to have complete fine grained control.

* src/conf/domain_conf.h, src/conf/domain_conf.c,
  src/libvirt_private.syms: Add virDomainDiskDefAssignAddress()
  for assigning a controller/bus/unit address based on disk target
* src/qemu/qemu_conf.c: Call virDomainDiskDefAssignAddress() after
  generating XML from ARGV
* tests/qemuxml2argvdata/*.xml: Add in drive address information
  to all XML files