https://bugzilla.redhat.com/show_bug.cgi?id=1572491

Commit id '78661cb1' added a physical output, but failed to update
the schema resulting in a failure from virt-xml-validate.

While at it - update the storagevolschemadata for the output.
Signed-off-by: NJohn Ferlan <jferlan@redhat.com>
ACKed-by Michal Privoznik <mprivozn@redhat.com>

Volume and pool formatting functions took different approaches to
unspecified uids/gids.  When unknown, it is always parsed as -1, but one
of the functions formatted it as unsigned int (wrong) and one as
int (better).  Due to that, our two of our XML files from tests cannot
be parsed on 32-bit machines.

RNG schema needs to be modified as well, but because both
storagepool.rng and storagevol.rng need same schema for permission
element, save some space by moving it to storagecommon.rng.
Signed-off-by: NMartin Kletzander <mkletzan@redhat.com>

Add 'nocow' to storage volume xml so that user can have an option
to set NOCOW flag to the newly created volume. It's useful on btrfs
file system to enhance performance.

Btrfs has low performance when hosting VM images, even more when the guest
in those VM are also using btrfs as file system. One way to mitigate this
bad performance is to turn off COW attributes on VM files. Generally, there
are two ways to turn off COW on btrfs: a) by mounting fs with nodatacow,
then all newly created files will be NOCOW. b) per file. Add the NOCOW file
attribute. It could only be done to empty or new files.

This patch tries the second way, according to 'nocow' option, it could set
NOCOW flag per file:
for raw file images, handle 'nocow' in libvirt code; for non-raw file images,
pass 'nocow=on' option to qemu-img, and let qemu-img to handle that (requires
qemu-img version >= 2.1).
Signed-off-by: NChunyan Liu <cyliu@suse.com>

We had incomplete RelaxNG support for storage formats listed
in virstoragefile.h: commit 027bf2ea added 'vdi' but forgot
to update the <volume> and <domain> xml lists; the <volume>
list was also missing 'fat' and 'vhd'.  Maintaining two lists
is a recipe for them getting out of sync, so make the list
common so that both contexts benefit the next time we add a
format in a single location.

* docs/schemas/domaincommon.rng (storageFormat): Move...
* docs/schemas/storagecommon.rng: ...here, and add vdi.
* docs/schemas/storagevol.rng (formatfile): Use common list.
Signed-off-by: NEric Blake <eblake@redhat.com>

Having two tiny files with a couple definitions didn't make
as much sense as one common file, especially since I plan to
add more definitions and use it in more places.

* docs/schemas/storageencryption.rng: Merge this...
* docs/schemas/storagefilefeatures.rng: ...and this, into...
* docs/schemas/storagecommon.rng: ...this new file.
* docs/schemas/Makefile.am (schema_DATA): Reflect renames.
* docs/schemas/storagevol.rng: Likewise.
* docs/schemas/domaincommon.rng: Likewise.
* libvirt.spec.in: Likewise.
* mingw-libvirt.spec.in: Likewise.
Signed-off-by: NEric Blake <eblake@redhat.com>

In the 'directory' and 'netfs' storage pools, a user can see
both 'file' and 'dir' storage volume types, to know when they
can descend into a subdirectory.  But in a network-based storage
pool, such as the upcoming 'gluster' pool, we use 'network'
instead of 'file', and did not have any counterpart for a
directory until this patch.  Adding a new volume type
'network-dir' is better than reusing 'dir', because it makes
it clear that the only way to access 'network' volumes within
that container is through the network mounting (leaving 'dir'
for something accessible in the local file system).

* include/libvirt/libvirt.h.in (virStorageVolType): Expand enum.
* docs/formatstorage.html.in: Document it.
* docs/schemasa/storagevol.rng (vol): Allow new value.
* src/conf/storage_conf.c (virStorageVol): Use new value.
* src/qemu/qemu_command.c (qemuBuildVolumeString): Fix client.
* src/qemu/qemu_conf.c (qemuTranslateDiskSourcePool): Likewise.
* tools/virsh-volume.c (vshVolumeTypeToString): Likewise.
* src/storage/storage_backend_fs.c
(virStorageBackendFileSystemVolDelete): Likewise.
Signed-off-by: NEric Blake <eblake@redhat.com>

I got annoyed at having to use both 'virsh vol-list $pool --details'
AND 'virsh vol-dumpxml $vol $pool' to learn if I had populated
the volume correctly.  Since two-thirds of the data present in
virStorageVolGetInfo() already appears in virStorageVolGetXMLDesc(),
this just adds the remaining piece of information, as:

<volume type='...'>
  ...
</volume>

* docs/formatstorage.html.in: Document new <volume type=...>.
* docs/schemas/storagevol.rng (vol): Add it to RelaxNG.
* src/conf/storage_conf.h (virStorageVolTypeToString): Declare.
* src/conf/storage_conf.c (virStorageVolTargetDefFormat): Output
the metatype.
(virStorageVolDefParseXML): Parse it, for unit tests.
* tests/storagevolxml2xmlout/vol-*.xml: Update tests to match.
Signed-off-by: NEric Blake <eblake@redhat.com>

The RNG grammar did not allow arbitrary interleaving, which makes
it harder than necessary to create a new volume from handwritten XML.
(Compare also to commit caf516db for pools).

* docs/schemas/storagevol.rng: Support interleaving.
* tests/storagevolxml2xmlin/vol-file-backing.xml: Test it.
Signed-off-by: NEric Blake <eblake@redhat.com>

Older xmllint version don't allow such characters in datatype anyURI.
In order not to change too much, I'm suggesting making a choice of
anyURI or 'absPathName' which should be fine (checked with upstream
and that old xmllint, both work fine).
Signed-off-by: NMartin Kletzander <mkletzan@redhat.com>

Add <features> and <compat> elements to volume target XML.

<compat> is a string which for qcow2 represents the QEMU version
it should be compatible with. Valid values are 0.10 and 1.1.
1.1 is implicit if the <features> element is present, otherwise
qemu-img default is used. 0.10 can be specified to explicitly
create older images after the qemu-img default changes.

<features> contains optional features, so far
<lazy_refcounts/> is available, which enables caching of reference
counters, improving performance for snapshots.

We don't parse it anyway.

https://bugzilla.redhat.com/show_bug.cgi?id=893273

The storage volume formats supported by the disk storage pool are
missing from the allowed values.

Add partition types.
Signed-off-by: NPhilipp Hahn <hahn@univention.de>

Add ocfs2 for disk vol.
Signed-off-by: NPhilipp Hahn <hahn@univention.de>

virStrToLong(..., 8, ...) already requires the mode to be octal.
Change the relax-ng schema to check for octal as well.
Signed-off-by: NPhilipp Hahn <hahn@univention.de>

The "volName" will be used by later patch.

The access, birth, modification and change times are added to
storage volumes and corresponding xml representations.  This
shows up in the XML in this format:

<timestamps>
  <atime>1341933637.027319099</atime>
  <mtime>1341933637.027319099</mtime>
</timestamps>
Signed-off-by: NEric Blake <eblake@redhat.com>

This patch brings support to manage sheepdog pools and volumes to libvirt.
It uses the "collie" command-line utility that comes with sheepdog for that.

A sheepdog pool in libvirt maps to a sheepdog cluster.
It needs a host and port to connect to, which in most cases
is just going to be the default of localhost on port 7000.

A sheepdog volume in libvirt maps to a sheepdog vdi.
To create one specify the pool, a name and the capacity.
Volumes can also be resized later.

In the volume XML the vdi name has to be put into the <target><path>.
To use the volume as a disk source for virtual machines specify
the vdi name as "name" attribute of the <source>.
The host and port information from the pool are specified inside the host tag.

  <disk type='network'>
    ...
    <source protocol="sheepdog" name="vdi_name">
      <host name="localhost" port="7000"/>
    </source>
  </disk>

To work right this patch parses the output of collie,
so it relies on the raw output option. There recently was a bug which caused
size information to be reported wrong. This is fixed upstream already and
will be in the next release.
Signed-off-by: NSebastian Wiedenroth <wiedi@frubar.net>

as we support qed format as well.

Make it obvious to 'dumpxml' readers what unit we are using,
since our default of KiB for memory (1024) differs from qemu's
default of MiB; and differs from our use of bytes for storage.

Tests were updated via:

$ find tests/*data tests/*out -name '*.xml' | \
  xargs sed -i 's/<\(memory\|currentMemory\|hard_limit\|soft_limit\|min_guarantee\|swap_hard_limit\)>/<\1 unit='"'KiB'>/"
$ find tests/*data tests/*out -name '*.xml' | \
  xargs sed -i 's/<\(capacity\|allocation\|available\)>/<\1 unit='"'bytes'>/"

followed by a few fixes for the stragglers.

Note that with this patch, the RNG for <memory> still forbids
validation of anything except unit='KiB', since the code silently
ignores the attribute; a later patch will expand <memory> to allow
scaled input in the code and update the RNG to match.

* docs/schemas/basictypes.rng (unit): Add 'bytes'.
(scaledInteger): New define.
* docs/schemas/storagevol.rng (sizing): Use it.
* docs/schemas/storagepool.rng (sizing): Likewise.
* docs/schemas/domaincommon.rng (memoryKBElement): New define; use
for memory elements.
* src/conf/storage_conf.c (virStoragePoolDefFormat)
(virStorageVolDefFormat): Likewise.
* src/conf/domain_conf.h (_virDomainDef): Document unit used
internally.
* src/conf/storage_conf.h (_virStoragePoolDef, _virStorageVolDef):
Likewise.
* tests/*data/*.xml: Update all tests.
* tests/*out/*.xml: Likewise.
* tests/define-dev-segfault: Likewise.
* tests/openvzutilstest.c (testReadNetworkConf): Likewise.
* tests/qemuargv2xmltest.c (blankProblemElements): Likewise.

The code supported unit='E' for "exabyte", but the RNG did not;
conversely, the RNG supported "z" and "y" but the code did not
(I'm jealous if you have that much storage, particularly since
it won't fit in 64-bit off_t).  Also, the code supported
<allocation unit='...'>, but not the RNG.

In an effort to make 'unit' more worthwhile in future patches,
it's easier to share it between files.

In making this factorization, note that absFilePath is more
permissive than 'path', so storage pools and storage volumes
will now validate with a wider set of file names than before.
I don't think this should be a problem in practice.

* docs/schemas/storagepool.rng: Include basic types, rather than
repeating things here.
* docs/schemas/storagevol.rng: Likewise.
* docs/schemas/basictypes.rng: Add 'unsignedLong', 'unit', and fix
to match storage code.

* docs/schemas/capability.rng: Likewise.
* docs/schemas/network.rng: Likewise.
* docs/schemas/nodedev.rng: Likewise.
* docs/schemas/storagepool.rng: Likewise.
* docs/schemas/storagevol.rng: Likewise.
Use these commands:
t=$'\t'
git ls-files | grep '\.rng$' | xargs grep -lE "^ *$t" \
  | xargs perl -MText::Tabs -ni -le \
    '$m=/^( *\t[ \t]*)(.*)/; print $m ? expand($1) . $2 : $_'

* docs/schemas/storagevol.rng: storage encryption rules require
  that UUID type be defined

The XML allows <encryption format='unencrypted'/>, this implementation
canonicalizes the internal representation so that "vol->encryption" is
non-NULL iff the volume is encrypted.

Note that partial encryption information (e.g. specifying an encryption
format, but not the key/passphrase) is valid, libvirt will automatically
choose value for the missing information during volume creation.  The
user can read the volume XML, and use the unmodified <encryption> tag in
future operations (without having to be able to understand) its contents.

* docs/formatstorage.html, docs/formatstorage.html.in: Document
  storage volume encryption options
* src/storage_conf.c, src/storage_conf.h: Hook up storage
  encryption XML handling
* tests/storagevolschemadata/vol-qcow2.xml: Test case for encryption
  schema changes

* docs/schemas/*.rng: the comments were wrong
* src/qemu_conf.c: typo in an error message

* docs/schemas/storagevol.rng docs/formatstorage.html[.in]
  tests/storagevolschemadata/vol-qcow2.xml: fix storage volume
  inconsistencies in schema and document, patch by Ryota Ozaki
daniel