- docs/formatstorage.html.in: document 'zfs' pool type, add it
   to a list of pool types that could use source physical devices
 - docs/storage.html.in: update a ZFS pool example XML with
   source physical devices, mention that starting from 1.2.9 a
   pool could be created from this devices by libvirt and in earlier
   versions user still has to create a pool manually
 - docs/drvbhyve.html.in: add an example with ZFS pools

https://bugzilla.redhat.com/show_bug.cgi?id=1092886

Rather than point off to some nefarious "pool-specific docs" page when
describing the "format" field for the target pool provide a link to the
storage driver page which describes the various valid formats for each
pool type.  Also make it a bit more clear that if a valid format isn't
specified, then the type field is ignored.

Introduce a new function to parse the provided scsi_host parent address
and unique_id value in order to find the /sys/class/scsi_host directory
which will allow a stable SCSI host address

Add a test to scsihosttest to lookup the host# name by using the PCI address
and unique_id value

Between reboots and kernel reloads, the SCSI host number used for SCSI
storage pools may change requiring modification to the storage pool XML
in order to use a specific SCSI host adapter.

This patch introduces the "parentaddr" element and "unique_id" attribute
for the SCSI host adapter in order to uniquely identify the adapter
between reboots and kernel reloads. For now the goal is to only parse
and format the XML. Both will be required to be provided in order to
uniquely identify the desired SCSI host.

The new XML is expected to be as follows:

  <adapter type='scsi_host'>
    <parentaddr unique_id='3'>
      <address domain='0x0000' bus='0x00' slot='0x1f' func='0x2'/>
    </parentaddr>
  </adapter>

where "parentaddr" is the parent device of the SCSI host using the PCI
address on which the device resides and the value from the unique_id file
for the device. Both the PCI address and unique_id values will be used
to traverse the /sys/class/scsi_host/ directories looking at each link
to match the PCI address reformatted to the directory link format where
"domain:bus:slot:function" is found.  Then for each matching directory
the unique_id file for the scsi_host will be used to match the unique_id
value in the xml.

For a PCI address listed above, this will be formatted to "0000:00:1f.2"
and the links in /sys/class/scsi_host will be used to find the host#
to be used for the 'scsi_host' device. Each entry is a link to the
/sys/bus/pci/devices directories, e.g.:

%  ls -al /sys/class/scsi_host/host2
lrwxrwxrwx. 1 root root 0 Jun  1 00:22 /sys/class/scsi_host/host2 -> ../../devices/pci0000:00/0000:00:1f.2/ata3/host2/scsi_host/host2

% cat /sys/class/scsi_host/host2/unique_id
3

The "parentaddr" and "name" attributes are mutually exclusive to identify
the SCSI host number. Use of the "parentaddr" element will be the preferred
mechanism.

This patch only supports to parse and format the XMLs. Later patches will
add code to find out the scsi host number.

Add 'nocow' to storage volume xml so that user can have an option
to set NOCOW flag to the newly created volume. It's useful on btrfs
file system to enhance performance.

Btrfs has low performance when hosting VM images, even more when the guest
in those VM are also using btrfs as file system. One way to mitigate this
bad performance is to turn off COW attributes on VM files. Generally, there
are two ways to turn off COW on btrfs: a) by mounting fs with nodatacow,
then all newly created files will be NOCOW. b) per file. Add the NOCOW file
attribute. It could only be done to empty or new files.

This patch tries the second way, according to 'nocow' option, it could set
NOCOW flag per file:
for raw file images, handle 'nocow' in libvirt code; for non-raw file images,
pass 'nocow=on' option to qemu-img, and let qemu-img to handle that (requires
qemu-img version >= 2.1).
Signed-off-by: NChunyan Liu <cyliu@suse.com>

With most of our storage backends it's possible to have two separate
volume keys to point to a single volume. (By creating sym/hard-links to
local files or by mounting remote filesystems to two different locations
and creating pools on top of them) Document this possibility.

In the 'directory' and 'netfs' storage pools, a user can see
both 'file' and 'dir' storage volume types, to know when they
can descend into a subdirectory.  But in a network-based storage
pool, such as the upcoming 'gluster' pool, we use 'network'
instead of 'file', and did not have any counterpart for a
directory until this patch.  Adding a new volume type
'network-dir' is better than reusing 'dir', because it makes
it clear that the only way to access 'network' volumes within
that container is through the network mounting (leaving 'dir'
for something accessible in the local file system).

* include/libvirt/libvirt.h.in (virStorageVolType): Expand enum.
* docs/formatstorage.html.in: Document it.
* docs/schemasa/storagevol.rng (vol): Allow new value.
* src/conf/storage_conf.c (virStorageVol): Use new value.
* src/qemu/qemu_command.c (qemuBuildVolumeString): Fix client.
* src/qemu/qemu_conf.c (qemuTranslateDiskSourcePool): Likewise.
* tools/virsh-volume.c (vshVolumeTypeToString): Likewise.
* src/storage/storage_backend_fs.c
(virStorageBackendFileSystemVolDelete): Likewise.
Signed-off-by: NEric Blake <eblake@redhat.com>

Add support for a new <pool type='gluster'>, similar to
RBD and Sheepdog.  Terminology wise, a gluster volume
forms a libvirt storage pool, within the gluster volume,
individual files are treated as libvirt storage volumes.

* docs/schemas/storagepool.rng (poolgluster): New pool type.
* docs/formatstorage.html.in: Document gluster.
* docs/storage.html.in: Likewise, and contrast it with netfs.
* tests/storagepoolxml2xmlin/pool-gluster.xml: New test.
* tests/storagepoolxml2xmlout/pool-gluster.xml: Likewise.
* tests/storagepoolxml2xmltest.c (mymain): Likewise.
Signed-off-by: NEric Blake <eblake@redhat.com>

I got annoyed at having to use both 'virsh vol-list $pool --details'
AND 'virsh vol-dumpxml $vol $pool' to learn if I had populated
the volume correctly.  Since two-thirds of the data present in
virStorageVolGetInfo() already appears in virStorageVolGetXMLDesc(),
this just adds the remaining piece of information, as:

<volume type='...'>
  ...
</volume>

* docs/formatstorage.html.in: Document new <volume type=...>.
* docs/schemas/storagevol.rng (vol): Add it to RelaxNG.
* src/conf/storage_conf.h (virStorageVolTypeToString): Declare.
* src/conf/storage_conf.c (virStorageVolTargetDefFormat): Output
the metatype.
(virStorageVolDefParseXML): Parse it, for unit tests.
* tests/storagevolxml2xmlout/vol-*.xml: Update tests to match.
Signed-off-by: NEric Blake <eblake@redhat.com>

We forgot to document several pool types.

* docs/formatstorage.html.in: Add docs for scsi, mpath, rbd, and
sheepdog.
Signed-off-by: NEric Blake <eblake@redhat.com>

Update the iSCSI storage pool example to include the secret

The description for <permissions> was copied from the storage volume
section to the storage pool section, but the semantics are different:
1. Currently only the "dir", "fs" and "netfs" storage pools use it.
2. They use it only to build the final directory.
3. A default for the storage volumes can't be set.
Signed-off-by: NPhilipp Hahn <hahn@univention.de>

The existing 'chap' XML logic was never used - just defined.  Rather than
try to insert a square peg into a round hole, blow it up and rewrite the
logic to follow the 'ceph' format.

Remove the former "chap.login" and "chap.passwd" fields and replace
with "chap.username" and "chap.secret" in _virStoragePoolAuthChap.
Adjust the virStoragePoolDefParseAuthChap() to process.

Change the rng file to describe the new layout

Update the formatstorage.html to describe the usage of the secret element
to mention that the secret type "iscsi" and "ceph" can be used
to storage pool too.

Update the formatsecret.html to include a reference to the storage pool

Update tests to handle the changes from 'login' and 'passwd' to 'username'
and '<secret>' format

Since we already have the v1.1.0-rc1 tag in git.

Add <features> and <compat> elements to volume target XML.

<compat> is a string which for qcow2 represents the QEMU version
it should be compatible with. Valid values are 0.10 and 1.1.
1.1 is implicit if the <features> element is present, otherwise
qemu-img default is used. 0.10 can be specified to explicitly
create older images after the qemu-img default changes.

<features> contains optional features, so far
<lazy_refcounts/> is available, which enables caching of reference
counters, improving performance for snapshots.

Let the pool types breathe.

The rule generating the HTML docs passing the --html flag
to xsltproc. This makes it use the legacy HTML parser, which
either ignores or tries to fix all sorts of broken XML tags.
There's no reason why we should be writing broken XML in
the first place, so removing --html and adding the XHTML
doctype to all files forces us to create good XML.

This adds the XHTML doc type and fixes many, many XML tag
problems it exposes.
Signed-off-by: NDaniel P. Berrange <berrange@redhat.com>

node device driver names the HBA like "scsi_host5", but storage
driver uses "host5", which could make the user confused. This
changes them to be consistent. However, for back-compat reason,
adapter name like "host5" is still supported.

This introduces 4 new attributes for storage pool source adapter.
E.g.

<adapter type='fc_host' parent='scsi_host5' wwnn='20000000c9831b4b' wwpn='10000000c9831b4b'/>

Attribute 'type' can be either 'scsi_host' or 'fc_host', and defaults
to 'scsi_host' if attribute 'name' is specified. I.e. It's optional
for 'scsi_host' adapter, for back-compat reason. However, mandatory
for 'fc_host' adapter and any new future adapter types. Attribute
'parent' is to specify the parent for the fc_host adapter.

* docs/formatstorage.html.in:
  - Add documents for the 4 new attrs
* docs/schemas/storagepool.rng:
  - Add RNG schema
* src/conf/storage_conf.c:
  - Parse and format the new XMLs
* src/conf/storage_conf.h:
  - New struct virStoragePoolSourceAdapter, replace "char *adapter" with it;
  - New enum virStoragePoolSourceAdapterType
* src/libvirt_private.syms:
  - Export TypeToString and TypeFromString
* src/phyp/phyp_driver.c:
  - Replace "adapter" with "adapter.data.name", which is member of the union
    of the new struct virStoragePoolSourceAdapter now. Later patch will
    add the checking, as "adapter.data.name" is only valid for "scsi_host"
    adapter.
* src/storage/storage_backend_scsi.c:
  - Like above
* tests/storagepoolxml2xmlin/pool-scsi-type-scsi-host.xml:
* tests/storagepoolxml2xmlin/pool-scsi-type-fc-host.xml:
  - New test for 'fc_host' and "scsi_host" adapter
* tests/storagepoolxml2xmlout/pool-scsi.xml:
  - Change the expected output, as the 'type' defaults to 'scsi_host' if 'name"
    specified now
* tests/storagepoolxml2xmlout/pool-scsi-type-scsi-host.xml:
* tests/storagepoolxml2xmlout/pool-scsi-type-fc-host.xml:
  - New test
* tests/storagepoolxml2xmltest.c:
  - Include the test

There was a 2 word sentence 'remote server' which is a left-over
from copy and paste.

Because of a wrong copy and paste, the documentation was saying that
'path' is the path to a block device node while it's a path to a
directory.

Sparse LVM volumes do not behave in the way one would naively expect.
The allocation does not automatically increase (which is different from
how sparse files work).

The access, birth, modification and change times are added to
storage volumes and corresponding xml representations.  This
shows up in the XML in this format:

<timestamps>
  <atime>1341933637.027319099</atime>
  <mtime>1341933637.027319099</mtime>
</timestamps>
Signed-off-by: NEric Blake <eblake@redhat.com>

Disk manufacturers are fond of quoting sizes in powers of 10,
rather than powers of 2 (after all, 2.1 GB sounds larger than
2.0 GiB, even though the exact opposite is true).  So, we might
as well follow coreutils' lead in supporting three types of
suffix: single letter ${u} (which we already had) and ${u}iB
for the power of 2, and ${u}B for power of 10.

Additionally, it is impossible to create a file with more than
2**63 bytes, since off_t is signed (if you have enough storage
to even create one 8EiB file, I'm jealous).  This now reports
failure up front rather than down the road when the kernel
finally refuses an impossible size.

* docs/schemas/basictypes.rng (unit): Add suffixes.
* src/conf/storage_conf.c (virStorageSize): Use new function.
* docs/formatstorage.html.in: Document it.
* tests/storagevolxml2xmlin/vol-file-backing.xml: Test it.
* tests/storagevolxml2xmlin/vol-file.xml: Likewise.

Yes, I like kilobytes better than kibibytes (when I say kilobytes,
I generally mean 1024).  But since the term is ambiguous, it can't
hurt to say what we mean, by using both the correct name and
calling out the numeric equivalent.

* src/libvirt.c (virDomainGetMaxMemory, virDomainSetMaxMemory)
(virDomainSetMemory, virDomainSetMemoryFlags)
(virNodeGetFreeMemory): Tweak wording.
* docs/formatdomain.html.in: Likewise.
* docs/formatstorage.html.in: Likewise.

Remove 2 words that shouldn't be here.

Add missing 'g' to '&gt;' of '<product />' source-element.
Signed-off-by: NPhilipp Hahn <hahn@univention.de>

I wrote a patch to add support for listing the Vendor and Model of a
storage pool in the storage pool XML.  This would allow vendor
extensions of specific devices.  The patch includes a test for the new
attributes as well.

Patrick Dignan

The owner and group in the documentation examples were confusingly given as
'0744'. They should be numeric uid and gid. Changed the examples to use the
default uid and gid assigned to qemu in F12.

* docs/formatstorage.html.in: Change example owner and group in volume XML

This patch is the result of running the following command in the docs
directory: sed -i 's/\t/        /g; s/\s*$//' *.html.in

* docs/*.html.in:convert tabs into 8 spaces and remove trailing whitespace

The XML allows <encryption format='unencrypted'/>, this implementation
canonicalizes the internal representation so that "vol->encryption" is
non-NULL iff the volume is encrypted.

Note that partial encryption information (e.g. specifying an encryption
format, but not the key/passphrase) is valid, libvirt will automatically
choose value for the missing information during volume creation.  The
user can read the volume XML, and use the unmodified <encryption> tag in
future operations (without having to be able to understand) its contents.

* docs/formatstorage.html, docs/formatstorage.html.in: Document
  storage volume encryption options
* src/storage_conf.c, src/storage_conf.h: Hook up storage
  encryption XML handling
* tests/storagevolschemadata/vol-qcow2.xml: Test case for encryption
  schema changes

* docs/schemas/storagevol.rng docs/formatstorage.html[.in]
  tests/storagevolschemadata/vol-qcow2.xml: fix storage volume
  inconsistencies in schema and document, patch by Ryota Ozaki
daniel

  pointed out by Ryota Ozaki
Daniel

* src/storage_backend.h src/storage_backend_logical.c
src/storage_conf.c src/storage_conf.h src/virsh.c:
Applied patches from David Lively to add storage source
elements needed for storage pool
* docs/formatstorage.html docs/formatstorage.html.in: associated
documentation
Daniel