Introduce virStoragePoolSaveState to properly format the state XML in
the same manner as virStoragePoolDefFormat, except for adding a
<poolstate> ... </poolstate> around the definition. This is similar to
virNetworkObjFormat used to save the live/active network information.

When modifying config/status XML, it might be handy to include some
additional XML elements (e.g. <poolstate>). In order to do so,
introduce new formatting function virStoragePoolDefFormatBuf and make
virStoragePoolDefFormat call it.

Make XML definition saving more generic by moving the common code into
virStoragePoolSaveXML and leave case specific code to
PoolSave{Status,Config,...} functions.

A helper that never returns an error and treats bits out of bitmap range
as false.

Use it everywhere we use ignore_value on virBitmapGetBit, or loop over
the bitmap size.

Add VIR_VOL_XML_PARSE_OPT_CAPACITY flag to virStorageVolDefParseXML.
With this flag, no error is reported when the capacity is missing
if there is a backing store.

Add VIR_VOL_XML_PARSE_NO_CAPACITY flag to the volume XML
parser. When set, it allows the capacity element to be omitted.

So we can allow omitting the capacity element if backing store is
present.

Allow the callers to pass down libvirt-internal flags.

In commit d2632d60 we agreed taht we want the parsed uid to properly
overflow but only to -1, however the value was read into long and then
wrapped into uid_t.  That meaned it failed on 32-bit systems.
Signed-off-by: NMartin Kletzander <mkletzan@redhat.com>

Volume and pool formatting functions took different approaches to
unspecified uids/gids.  When unknown, it is always parsed as -1, but one
of the functions formatted it as unsigned int (wrong) and one as
int (better).  Due to that, our two of our XML files from tests cannot
be parsed on 32-bit machines.

RNG schema needs to be modified as well, but because both
storagepool.rng and storagevol.rng need same schema for permission
element, save some space by moving it to storagecommon.rng.
Signed-off-by: NMartin Kletzander <mkletzan@redhat.com>

Since virStoragePoolFree will call virObjectUnref anyway, let's just use that
directly so as to avoid the possibility that we inadvertently clear out
a pending error message when using the public API.

https://bugzilla.redhat.com/show_bug.cgi?id=1159180

The virStoragePoolSourceFindDuplicate only checks the incoming definition
against the same type of pool as the def; however, for "scsi_host" and
"fc_host" adapter pools, it's possible that either some pool "scsi_host"
adapter definition is already using the scsi_hostN that the "fc_host"
adapter definition wants to use or some "fc_host" pool adapter definition
is using a vHBA scsi_hostN or parent scsi_hostN that an incoming "scsi_host"
definition is trying to use.

This patch adds the mismatched type checks and adds extraneous comments
to describe what each check is determining.

This patch also modifies the documentation to be describe what scsi_hostN
devices a "scsi_host" source adapter should use and which to avoid. It also
updates the parent definition to specifically call out that for mixed
environments it's better to define which parent to use so that the duplicate
pool checks can be done properly.

https://bugzilla.redhat.com/show_bug.cgi?id=1159180

Move the API from the backend to storage_conf and rename it to
virStoragePoolGetVhbaSCSIHostParent.  A future patch will need to
use this functionality from storage_conf

Signed-off-by: NMartin Kletzander <mkletzan@redhat.com>

https://bugzilla.redhat.com/show_bug.cgi?id=1160926

Introduce a 'managed' attribute to allow libvirt to decide whether to
delete a vHBA vport created via external means such as nodedev-create.
The code currently decides whether to delete the vHBA based solely on
whether the parent was provided at creation time. However, that may not
be the desired action, so rather than delete and force someone to create
another vHBA via an additional nodedev-create allow the configuration of
the storage pool to decide the desired action.

During createVport when libvirt does the VPORT_CREATE, set the managed
value to YES if not already set to indicate to the deleteVport code that
it should delete the vHBA when the pool is destroyed.

If libvirtd is restarted all the memory only state was lost, so for a
persistent storage pool, use the virStoragePoolSaveConfig in order to
write out the managed value.

Because we're now saving the current configuration, we need to be sure
to not save the parent in the output XML if it was undefined at start.
Saving the name would cause future starts to always use the same parent
which is not the expected result when not providing a parent. By not
providing a parent, libvirt is expected to find the best available
vHBA port for each subsequent (re)start.

At deleteVport, use the new managed value to decide whether to execute
the VPORT_DELETE.  Since we no longer save the parent in memory or in
XML when provided, if it was not provided, then we have to look it up.

https://bugzilla.redhat.com/show_bug.cgi?id=1160926

Introduce the ability to save a configuration of a persistent configuration
that may be changed by storage pool backend activity, such as start or stop

https://bugzilla.redhat.com/show_bug.cgi?id=1160926

Passing a copy of the storage pool adapter to a function just changes the
copy of the fields in the particular function and then when returning to
the caller those changes are discarded.  While not yet biting us in the
storage clean-up case, it did cause an issue for the fchost storage pool
startup case, createVport.  The issue was at startup, if no parent is found
in the XML, the code will search for the 'best available' parent and then
store that in the in memory copy of the adapter.  Of course, in this case
it was a copy, so when returning to the virStorageBackendSCSIStartPool that
change was discarded (or lost) from the pool->def->source.adapter which
meant at shutdown (deleteVport), the code assumed no adapter was passed
and skipped the deletion, leaving the vHBA created by libvirt still defined
requiring an additional stop of a nodedev-destroy to remove.

Adjusted the createVport to take virStoragePoolDefPtr instead of the
adapter copy. Then use the virStoragePoolSourceAdapterPtr when processing.
A future patch will need the 'def' anyway, so this just sets up for that.

Since commit 3f99d64d no new scsi_host pools can be defined
if one of the already defined scsi_host pools does not refer
to an accessible scsi_host adapter.

Relax the check by skipping over these inaccessible pools
when checking for duplicates.

If both source adapters are specified by a parent address,
just comparing the address is faster and catches even addresses
that do not refer to valid adapters.

https://bugzilla.redhat.com/show_bug.cgi?id=1146837

Resolve a crash in libvirtd resulting from commit id 'a4bd62ad' (1.0.6)
which added parentaddr and unique_id to allow unique identification of
a scsi_host, but assumed that all the pool entries and the incoming
definition would be similarly defined. If the existing pool uses the
'name' attribute and an incoming pool is using the parentaddr/unique_id,
then the code will attempt to compare the existing name string against
the incoming name string which doesn't exist (is NULL) and results in
a core (STREQ).

Conversely, if the existing pool used the parentaddr/unique_id and the
to be defined pool used the name, then the comparison would be against
the parentaddr, but since the incoming pool doesn't have one - that would
leave the comparison against a parentaddr of all 0's and a unique_id of 0,
which will always comparison to fail. This means someone could define the
same source adapter for two pools

In order to resolve this, adjust the code to get the 'host#' to be used
by the storage scsi backend in order to check/start the pool and make sure
the incoming definition doesn't match any of the existing pool defs.

 - Provide an implementation for buildPool and deletePool operations
   for the ZFS storage backend.
 - Add VIR_STORAGE_POOL_SOURCE_DEVICE flag to ZFS pool poolOptions
   as now we can specify devices to build pool from
 - storagepool.rng: add an optional 'sourceinfodev' to 'sourcezfs' and
   add an optional 'target' to 'poolzfs' entity
 - Add a couple of tests to storagepoolxml2xmltest

Since 9f781da6

Resolve a libvirtd crash in virStoragePoolSourceFindDuplicate()
when there is an existing SCSI pool defined with adapter type as
'scsi_host' and defining a new SCSI pool with adapter type as
'fc_host' and parent attribute missing or vice versa.

For example, if there is an existing SCSI pool with adapter type
as 'scsi_host' defined using the following XML

<pool type='scsi'>
  <name>TEST_SCSI_POOL</name>
    <source>
       <adapter type='scsi_host' name='scsi_host1'/>
    </source>
    <target>
        <path>/dev/disk/by-path</path>
    </target>
</pool>

When defining another SCSI pool with adapter type as 'fc_host' using the
following XML will crash libvirtd

<pool type='scsi'>
  <name>TEST_SCSI_FC_POOL</name>
  <source>
     <adapter type='fc_host' wwnn='1234567890abcdef' wwpn='abcdef1234567890'/>
  </source>
  <target>
     <path>/dev/disk/by-path</path>
  </target>
</pool>

Same is true for the reverse case as well where there exists a SCSI pool
with adapter type as 'fc_host' and another SCSI pool is defined with
adapter type as 'scsi_host'.

This happens because for fc_host 'name' is optional attribute whereas for
scsi_host its mandatory. However the check in libvirt for finding duplicate
storage pools didn't take that into account while comparing
Signed-off-by: NPradipta Kr. Banerjee <bpradip@in.ibm.com>

If there was a failure processing 'authdef' and the code went to cleanup
before the setting to source->auth, then it'd be leaked.

And add a syntax-check for '){$'.  It's not perfect, but better than
nothing.
Signed-off-by: NMartin Kletzander <mkletzan@redhat.com>

Implement ZFS storage backend driver. Currently supported
only on FreeBSD because of ZFS limitations on Linux.

Features supported:

 - pool-start, pool-stop
 - pool-info
 - vol-list
 - vol-create / vol-delete

Pool definition looks like that:

 <pool type='zfs'>
  <name>myzfspool</name>
  <source>
    <name>actualpoolname</name>
  </source>
 </pool>

The 'actualpoolname' value is a name of the pool on the system,
such as shown by 'zpool list' command. Target makes no sense
here because volumes path is always /dev/zvol/$poolname/$volname.

User has to create a pool on his own, this driver doesn't
support pool creation currently.

A volume could be used with Qemu by adding an entry like this:

    <disk type='volume' device='disk'>
      <driver name='qemu' type='raw'/>
      <source pool='myzfspool' volume='vol5'/>
      <target dev='hdc' bus='ide'/>
    </disk>

Between reboots and kernel reloads, the SCSI host number used for SCSI
storage pools may change requiring modification to the storage pool XML
in order to use a specific SCSI host adapter.

This patch introduces the "parentaddr" element and "unique_id" attribute
for the SCSI host adapter in order to uniquely identify the adapter
between reboots and kernel reloads. For now the goal is to only parse
and format the XML. Both will be required to be provided in order to
uniquely identify the desired SCSI host.

The new XML is expected to be as follows:

  <adapter type='scsi_host'>
    <parentaddr unique_id='3'>
      <address domain='0x0000' bus='0x00' slot='0x1f' func='0x2'/>
    </parentaddr>
  </adapter>

where "parentaddr" is the parent device of the SCSI host using the PCI
address on which the device resides and the value from the unique_id file
for the device. Both the PCI address and unique_id values will be used
to traverse the /sys/class/scsi_host/ directories looking at each link
to match the PCI address reformatted to the directory link format where
"domain:bus:slot:function" is found.  Then for each matching directory
the unique_id file for the scsi_host will be used to match the unique_id
value in the xml.

For a PCI address listed above, this will be formatted to "0000:00:1f.2"
and the links in /sys/class/scsi_host will be used to find the host#
to be used for the 'scsi_host' device. Each entry is a link to the
/sys/bus/pci/devices directories, e.g.:

%  ls -al /sys/class/scsi_host/host2
lrwxrwxrwx. 1 root root 0 Jun  1 00:22 /sys/class/scsi_host/host2 -> ../../devices/pci0000:00/0000:00:1f.2/ata3/host2/scsi_host/host2

% cat /sys/class/scsi_host/host2/unique_id
3

The "parentaddr" and "name" attributes are mutually exclusive to identify
the SCSI host number. Use of the "parentaddr" element will be the preferred
mechanism.

This patch only supports to parse and format the XMLs. Later patches will
add code to find out the scsi host number.

Preparation for future patches by creating a scsi_host union. For now,
just the 'name' will be present.

Add 'nocow' to storage volume xml so that user can have an option
to set NOCOW flag to the newly created volume. It's useful on btrfs
file system to enhance performance.

Btrfs has low performance when hosting VM images, even more when the guest
in those VM are also using btrfs as file system. One way to mitigate this
bad performance is to turn off COW attributes on VM files. Generally, there
are two ways to turn off COW on btrfs: a) by mounting fs with nodatacow,
then all newly created files will be NOCOW. b) per file. Add the NOCOW file
attribute. It could only be done to empty or new files.

This patch tries the second way, according to 'nocow' option, it could set
NOCOW flag per file:
for raw file images, handle 'nocow' in libvirt code; for non-raw file images,
pass 'nocow=on' option to qemu-img, and let qemu-img to handle that (requires
qemu-img version >= 2.1).
Signed-off-by: NChunyan Liu <cyliu@suse.com>

As we have a nested pointer for storing the backing store of a volume
there's no need to store it in a separate struct.

Commit dae1568c converted the perms
member of the virStorageVolTarget struct into a pointer to make it
optional. But virStorageVolTargetDefFormat did not check perms for
NULL before dereferencing it.

Replace the authType, chap, and cephx unions in virStoragePoolSource
with a single pointer to a virStorageAuthDefPtr.  Adjust all users of
the previous chap/cephx and secret unions with the source->auth data.

Replace:
if (virBufferError(&buf)) {
    virBufferFreeAndReset(&buf);
    virReportOOMError();
    ...
}

with:
if (virBufferCheckError(&buf) < 0)
    ...

This should not be a functional change (unless some callers
misused the virBuffer APIs - a different error would be reported
then)

Libvirt didn't output feature flags for images stored on native gluster.
Fix this trivially by adding a feature formatter callback.

Resolves: https://bugzilla.redhat.com/show_bug.cgi?id=1095035

The VIR_ENUM_DECL/VIR_ENUM_IMPL helper macros already append 'Type'
to the enum name being converted; it looks silly to have functions
with 'TypeType' in their name.  Even though some of our enums have
to have a 'Type' suffix, the corresponding string conversion
functions do not.

* src/conf/secret_conf.h (VIR_ENUM_DECL): Rename virSecretUsageType.
* src/conf/storage_conf.h (VIR_ENUM_DECL): Rename
virStoragePoolAuthType, virStoragePoolSourceAdapterType,
virStoragePartedFsType.
* src/conf/domain_conf.c (virDomainDiskDefParseXML)
(virDomainFSDefParseXML, virDomainFSDefFormat): Update callers.
* src/conf/secret_conf.c (virSecretDefParseUsage)
(virSecretDefFormatUsage): Likewise.
* src/conf/storage_conf.c (virStoragePoolDefParseAuth)
(virStoragePoolDefParseSource, virStoragePoolSourceFormat):
Likewise.
* src/lxc/lxc_controller.c (virLXCControllerSetupLoopDevices):
Likewise.
* src/storage/storage_backend_disk.c
(virStorageBackendDiskPartFormat): Likewise.
* src/util/virstorageencryption.c (virStorageEncryptionSecretParse)
(virStorageEncryptionSecretFormat): Likewise.
* tools/virsh-secret.c (cmdSecretList): Likewise.
* src/libvirt_private.syms (secret_conf.h, storage_conf.h): Export
corrected names.
Signed-off-by: NEric Blake <eblake@redhat.com>

When reading configuration files, we were silently ignoring
directory read failures.  While unlikely, we might as well
report them.

* src/conf/domain_conf.c (virDomainObjListLoadAllConfigs): Report
readdir errors.
* src/conf/network_conf.c (virNetworkLoadAllState)
(virNetworkLoadAllConfigs): Likewise.
* src/conf/nwfilter_conf.c (virNWFilterLoadAllConfigs): Likewise.
* src/conf/storage_conf.c (virStoragePoolLoadAllConfigs):
Likewise.
Signed-off-by: NEric Blake <eblake@redhat.com>

One of the features of qcow2 is that a wrapper file can have
more capacity than its backing file from the guest's perspective;
what's more, sparse files make tracking allocation of both
the active and backing file worthwhile.  As such, it makes
more sense to show allocation numbers for each file in a chain,
and not just the top-level file.  This sets up the fields for
the tracking, although it does not modify XML to display any
new information.

* src/util/virstoragefile.h (_virStorageSource): Add fields.
* src/conf/storage_conf.h (_virStorageVolDef): Drop redundant
fields.
* src/storage/storage_backend.c (virStorageBackendCreateBlockFrom)
(createRawFile, virStorageBackendCreateQemuImgCmd)
(virStorageBackendCreateQcowCreate): Update clients.
* src/storage/storage_driver.c (storageVolDelete)
(storageVolCreateXML, storageVolCreateXMLFrom, storageVolResize)
(storageVolWipeInternal, storageVolGetInfo): Likewise.
* src/storage/storage_backend_fs.c (virStorageBackendProbeTarget)
(virStorageBackendFileSystemRefresh)
(virStorageBackendFileSystemVolResize)
(virStorageBackendFileSystemVolRefresh): Likewise.
* src/storage/storage_backend_logical.c
(virStorageBackendLogicalMakeVol)
(virStorageBackendLogicalCreateVol): Likewise.
* src/storage/storage_backend_scsi.c
(virStorageBackendSCSINewLun): Likewise.
* src/storage/storage_backend_mpath.c
(virStorageBackendMpathNewVol): Likewise.
* src/storage/storage_backend_rbd.c
(volStorageBackendRBDRefreshVolInfo)
(virStorageBackendRBDCreateImage): Likewise.
* src/storage/storage_backend_disk.c
(virStorageBackendDiskMakeDataVol)
(virStorageBackendDiskCreateVol): Likewise.
* src/storage/storage_backend_sheepdog.c
(virStorageBackendSheepdogBuildVol)
(virStorageBackendSheepdogParseVdiList): Likewise.
* src/storage/storage_backend_gluster.c
(virStorageBackendGlusterRefreshVol): Likewise.
* src/conf/storage_conf.c (virStorageVolDefFormat)
(virStorageVolDefParseXML): Likewise.
* src/test/test_driver.c (testOpenVolumesForPool)
(testStorageVolCreateXML, testStorageVolCreateXMLFrom)
(testStorageVolDelete, testStorageVolGetInfo): Likewise.
* src/esx/esx_storage_backend_iscsi.c (esxStorageVolGetXMLDesc):
Likewise.
* src/esx/esx_storage_backend_vmfs.c (esxStorageVolGetXMLDesc)
(esxStorageVolCreateXML): Likewise.
* src/parallels/parallels_driver.c (parallelsAddHddByVolume):
Likewise.
* src/parallels/parallels_storage.c (parallelsDiskDescParseNode)
(parallelsStorageVolDefineXML, parallelsStorageVolCreateXMLFrom)
(parallelsStorageVolDefRemove, parallelsStorageVolGetInfo):
Likewise.
* src/vbox/vbox_tmpl.c (vboxStorageVolCreateXML)
(vboxStorageVolGetXMLDesc): Likewise.
* tests/storagebackendsheepdogtest.c (test_vdi_list_parser):
Likewise.
* src/phyp/phyp_driver.c (phypStorageVolCreateXML): Likewise.

A fairly smooth transition.  And now that domain disks and
storage volumes share a common struct, it opens the doors for
a future patch to expose more details in the XML for both
objects.

* src/conf/storage_conf.h (_virStorageVolTarget): Delete.
(_virStorageVolDef): Use common type.
* src/conf/storage_conf.c (virStorageVolDefFree)
(virStorageVolTargetDefFormat): Update clients.
* src/storage/storage_backend.h: Likewise.
* src/storage/storage_backend.c
(virStorageBackendDetectBlockVolFormatFD)
(virStorageBackendUpdateVolTargetInfo)
(virStorageBackendUpdateVolTargetInfoFD): Likewise.
* src/storage/storage_backend_fs.c (virStorageBackendProbeTarget):
Likewise.
Signed-off-by: NEric Blake <eblake@redhat.com>

Some preparatory work before consolidating storage volume
structs with the rest of virstoragefile.  Making these
changes allows a volume target to be much closer to (a
subset of) the virStorageSource struct.

Making perms be a pointer allows it to be optional if we
have a storage pool that doesn't expose permissions in a
way we can access.  It also allows future patches to
optionally expose permissions details learned about a disk
image via domain <disk> listings, rather than just
limiting it to storage volume listings.

Disk partition types was only used by internal code to
control what type of partition to create when carving up
an MS-DOS partition table storage pool (and is not used
for GPT partition tables or other storage pools).  It was
not exposed in volume XML, and as it is more closely
related to extent information of the overall block device
than it is to the <target> information describing the host
file.  Besides, if we ever decide to expose it in XML down
the road, we can move it back as needed.

* src/conf/storage_conf.h (_virStorageVolTarget): Change perms to
pointer, enhance comments.  Move partition type...
(_virStorageVolSource): ...here.
* src/conf/storage_conf.c (virStorageVolDefFree)
(virStorageVolDefParseXML, virStorageVolTargetDefFormat): Update
clients.
* src/storage/storage_backend_fs.c (createFileDir): Likewise.
* src/storage/storage_backend.c (virStorageBackendCreateBlockFrom)
(virStorageBackendCreateRaw, virStorageBackendCreateExecCommand)
(virStorageBackendUpdateVolTargetInfoFD): Likewise.
* src/storage/storage_backend_logical.c
(virStorageBackendLogicalCreateVol): Likewise.
* src/storage/storage_backend_disk.c
(virStorageBackendDiskMakeDataVol)
(virStorageBackendDiskPartTypeToCreate): Likewise.
Signed-off-by: NEric Blake <eblake@redhat.com>

The <source> elements need to be indented from <sources> elements.