Commit id '18642d10' refactored the call to virCommandRunRegex()
inside a new function virStorageBackendFileSystemNetFindNFSPoolSources(),
but the cut-n-paste didn't remove the "&state".  Now that state is passed
by reference, it results in a libvirtd core with a messages entry:

"...internal error: unknown storage pool type Unknow"

Without this, building an XFS pool on a formatted partition
fails with --overwrite.

https://bugzilla.redhat.com/show_bug.cgi?id=927172

Currently VolOpen notifies the user of a potentially non-fatal failure by
returning -2 and logging a VIR_WARN or VIR_INFO. Unfortunately most
callers treat -2 as fatal but don't actually report any message with
the error APIs.

Rename the VOL_OPEN_ERROR flag to VOL_OPEN_NOERROR. If NOERROR is specified,
we preserve the current behavior of returning -2 (there's only one caller
that wants this).

However in the default case, only return -1, and actually use the error
APIs. Fix up a couple callers as a result.

A future patch will merge virStorageFileMetadata and virStorageSource,
but I found it easier to do if both structs use the same information
for tracking whether a source file needs encryption keys.

* src/util/virstoragefile.h (_virStorageFileMetadata): Prepare
full encryption struct instead of just a bool.
* src/storage/storage_backend_fs.c (virStorageBackendProbeTarget):
Use transfer semantics.
* src/storage/storage_backend_gluster.c
(virStorageBackendGlusterRefreshVol): Likewise.
* src/util/virstoragefile.c (virStorageFileGetMetadataInternal):
Populate struct.
(virStorageFileFreeMetadata): Adjust clients.
* tests/virstoragetest.c (testStorageChain): Likewise.
Signed-off-by: NEric Blake <eblake@redhat.com>

Now that each virStorageSource can track allocation information,
and given that we already have the information without extra
syscalls, it's easier to just always populate the information
directly into the struct than it is to sometimes pass the address
of the struct members down the call chain.

* src/storage/storage_backend.h (virStorageBackendUpdateVolInfo)
(virStorageBackendUpdateVolTargetInfo)
(virStorageBackendUpdateVolTargetInfoFD): Update signature.
* src/storage/storage_backend.c (virStorageBackendUpdateVolInfo)
(virStorageBackendUpdateVolTargetInfo)
(virStorageBackendUpdateVolTargetInfoFD): Always populate struct
members instead.
* src/storage/storage_backend_disk.c
(virStorageBackendDiskMakeDataVol): Update client.
* src/storage/storage_backend_fs.c (virStorageBackendProbeTarget)
(virStorageBackendFileSystemRefresh)
(virStorageBackendFileSystemVolRefresh): Likewise.
* src/storage/storage_backend_gluster.c
(virStorageBackendGlusterRefreshVol): Likewise.
* src/storage/storage_backend_logical.c
(virStorageBackendLogicalMakeVol): Likewise.
* src/storage/storage_backend_mpath.c
(virStorageBackendMpathNewVol): Likewise.
* src/storage/storage_backend_scsi.c
(virStorageBackendSCSINewLun): Likewise.
Signed-off-by: NEric Blake <eblake@redhat.com>

One of the features of qcow2 is that a wrapper file can have
more capacity than its backing file from the guest's perspective;
what's more, sparse files make tracking allocation of both
the active and backing file worthwhile.  As such, it makes
more sense to show allocation numbers for each file in a chain,
and not just the top-level file.  This sets up the fields for
the tracking, although it does not modify XML to display any
new information.

* src/util/virstoragefile.h (_virStorageSource): Add fields.
* src/conf/storage_conf.h (_virStorageVolDef): Drop redundant
fields.
* src/storage/storage_backend.c (virStorageBackendCreateBlockFrom)
(createRawFile, virStorageBackendCreateQemuImgCmd)
(virStorageBackendCreateQcowCreate): Update clients.
* src/storage/storage_driver.c (storageVolDelete)
(storageVolCreateXML, storageVolCreateXMLFrom, storageVolResize)
(storageVolWipeInternal, storageVolGetInfo): Likewise.
* src/storage/storage_backend_fs.c (virStorageBackendProbeTarget)
(virStorageBackendFileSystemRefresh)
(virStorageBackendFileSystemVolResize)
(virStorageBackendFileSystemVolRefresh): Likewise.
* src/storage/storage_backend_logical.c
(virStorageBackendLogicalMakeVol)
(virStorageBackendLogicalCreateVol): Likewise.
* src/storage/storage_backend_scsi.c
(virStorageBackendSCSINewLun): Likewise.
* src/storage/storage_backend_mpath.c
(virStorageBackendMpathNewVol): Likewise.
* src/storage/storage_backend_rbd.c
(volStorageBackendRBDRefreshVolInfo)
(virStorageBackendRBDCreateImage): Likewise.
* src/storage/storage_backend_disk.c
(virStorageBackendDiskMakeDataVol)
(virStorageBackendDiskCreateVol): Likewise.
* src/storage/storage_backend_sheepdog.c
(virStorageBackendSheepdogBuildVol)
(virStorageBackendSheepdogParseVdiList): Likewise.
* src/storage/storage_backend_gluster.c
(virStorageBackendGlusterRefreshVol): Likewise.
* src/conf/storage_conf.c (virStorageVolDefFormat)
(virStorageVolDefParseXML): Likewise.
* src/test/test_driver.c (testOpenVolumesForPool)
(testStorageVolCreateXML, testStorageVolCreateXMLFrom)
(testStorageVolDelete, testStorageVolGetInfo): Likewise.
* src/esx/esx_storage_backend_iscsi.c (esxStorageVolGetXMLDesc):
Likewise.
* src/esx/esx_storage_backend_vmfs.c (esxStorageVolGetXMLDesc)
(esxStorageVolCreateXML): Likewise.
* src/parallels/parallels_driver.c (parallelsAddHddByVolume):
Likewise.
* src/parallels/parallels_storage.c (parallelsDiskDescParseNode)
(parallelsStorageVolDefineXML, parallelsStorageVolCreateXMLFrom)
(parallelsStorageVolDefRemove, parallelsStorageVolGetInfo):
Likewise.
* src/vbox/vbox_tmpl.c (vboxStorageVolCreateXML)
(vboxStorageVolGetXMLDesc): Likewise.
* tests/storagebackendsheepdogtest.c (test_vdi_list_parser):
Likewise.
* src/phyp/phyp_driver.c (phypStorageVolCreateXML): Likewise.

A fairly smooth transition.  And now that domain disks and
storage volumes share a common struct, it opens the doors for
a future patch to expose more details in the XML for both
objects.

* src/conf/storage_conf.h (_virStorageVolTarget): Delete.
(_virStorageVolDef): Use common type.
* src/conf/storage_conf.c (virStorageVolDefFree)
(virStorageVolTargetDefFormat): Update clients.
* src/storage/storage_backend.h: Likewise.
* src/storage/storage_backend.c
(virStorageBackendDetectBlockVolFormatFD)
(virStorageBackendUpdateVolTargetInfo)
(virStorageBackendUpdateVolTargetInfoFD): Likewise.
* src/storage/storage_backend_fs.c (virStorageBackendProbeTarget):
Likewise.
Signed-off-by: NEric Blake <eblake@redhat.com>

Some preparatory work before consolidating storage volume
structs with the rest of virstoragefile.  Making these
changes allows a volume target to be much closer to (a
subset of) the virStorageSource struct.

Making perms be a pointer allows it to be optional if we
have a storage pool that doesn't expose permissions in a
way we can access.  It also allows future patches to
optionally expose permissions details learned about a disk
image via domain <disk> listings, rather than just
limiting it to storage volume listings.

Disk partition types was only used by internal code to
control what type of partition to create when carving up
an MS-DOS partition table storage pool (and is not used
for GPT partition tables or other storage pools).  It was
not exposed in volume XML, and as it is more closely
related to extent information of the overall block device
than it is to the <target> information describing the host
file.  Besides, if we ever decide to expose it in XML down
the road, we can move it back as needed.

* src/conf/storage_conf.h (_virStorageVolTarget): Change perms to
pointer, enhance comments.  Move partition type...
(_virStorageVolSource): ...here.
* src/conf/storage_conf.c (virStorageVolDefFree)
(virStorageVolDefParseXML, virStorageVolTargetDefFormat): Update
clients.
* src/storage/storage_backend_fs.c (createFileDir): Likewise.
* src/storage/storage_backend.c (virStorageBackendCreateBlockFrom)
(virStorageBackendCreateRaw, virStorageBackendCreateExecCommand)
(virStorageBackendUpdateVolTargetInfoFD): Likewise.
* src/storage/storage_backend_logical.c
(virStorageBackendLogicalCreateVol): Likewise.
* src/storage/storage_backend_disk.c
(virStorageBackendDiskMakeDataVol)
(virStorageBackendDiskPartTypeToCreate): Likewise.
Signed-off-by: NEric Blake <eblake@redhat.com>

A continuation of the migration of disk details to virstoragefile.
This patch moves a single enum, but converting the name has quite
a bit of fallout.

* src/conf/domain_conf.h (virDomainDiskType): Move...
* src/util/virstoragefile.h (virStorageType): ...and rename.
* src/bhyve/bhyve_command.c (bhyveBuildDiskArgStr)
(virBhyveProcessBuildLoadCmd): Update clients.
* src/conf/domain_conf.c (virDomainDiskSourceDefParse)
(virDomainDiskDefParseXML, virDomainDiskSourceDefFormatInternal)
(virDomainDiskDefFormat, virDomainDiskGetActualType)
(virDomainDiskDefForeachPath, virDomainDiskSourceIsBlockType):
Likewise.
* src/conf/snapshot_conf.h (_virDomainSnapshotDiskDef): Likewise.
* src/conf/snapshot_conf.c (virDomainSnapshotDiskDefParseXML)
(virDomainSnapshotAlignDisks, virDomainSnapshotDiskDefFormat):
Likewise.
* src/esx/esx_driver.c (esxAutodetectSCSIControllerModel)
(esxDomainDefineXML): Likewise.
* src/locking/domain_lock.c (virDomainLockManagerAddDisk):
Likewise.
* src/lxc/lxc_controller.c
(virLXCControllerSetupLoopDeviceDisk)
(virLXCControllerSetupNBDDeviceDisk)
(virLXCControllerSetupLoopDevices, virLXCControllerSetupDisk):
Likewise.
* src/parallels/parallels_driver.c (parallelsGetHddInfo):
Likewise.
* src/phyp/phyp_driver.c (phypDiskType): Likewise.
* src/qemu/qemu_command.c (qemuGetDriveSourceString)
(qemuDomainDiskGetSourceString, qemuBuildDriveStr)
(qemuBuildCommandLine, qemuParseCommandLineDisk)
(qemuParseCommandLine): Likewise.
* src/qemu/qemu_conf.c (qemuCheckSharedDevice)
(qemuTranslateDiskSourcePool)
(qemuTranslateSnapshotDiskSourcePool): Likewise.
* src/qemu/qemu_domain.c (qemuDomainDeviceDefPostParse)
(qemuDomainDetermineDiskChain): Likewise.
* src/qemu/qemu_driver.c (qemuDomainGetBlockInfo)
(qemuDomainSnapshotPrepareDiskExternalBackingInactive)
(qemuDomainSnapshotPrepareDiskExternalBackingActive)
(qemuDomainSnapshotPrepareDiskExternalOverlayActive)
(qemuDomainSnapshotPrepareDiskExternalOverlayInactive)
(qemuDomainSnapshotPrepareDiskInternal)
(qemuDomainSnapshotPrepare)
(qemuDomainSnapshotCreateSingleDiskActive): Likewise.
* src/qemu/qemu_hotplug.c (qemuDomainChangeEjectableMedia):
Likewise.
* src/qemu/qemu_migration.c (qemuMigrationIsSafe): Likewise.
* src/security/security_apparmor.c
(AppArmorRestoreSecurityImageLabel)
(AppArmorSetSecurityImageLabel): Likewise.
* src/security/security_dac.c (virSecurityDACSetSecurityImageLabel)
(virSecurityDACRestoreSecurityImageLabelInt)
(virSecurityDACSetSecurityAllLabel): Likewise.
* src/security/security_selinux.c
(virSecuritySELinuxRestoreSecurityImageLabelInt)
(virSecuritySELinuxSetSecurityImageLabel)
(virSecuritySELinuxSetSecurityAllLabel): Likewise.
* src/storage/storage_backend.c (virStorageFileBackendForType):
Likewise.
* src/storage/storage_backend_fs.c (virStorageFileBackendFile)
(virStorageFileBackendBlock): Likewise.
* src/storage/storage_backend_gluster.c
(virStorageFileBackendGluster): Likewise.
* src/vbox/vbox_tmpl.c (vboxDomainGetXMLDesc, vboxAttachDrives)
(vboxDomainAttachDeviceImpl, vboxDomainDetachDevice): Likewise.
* src/vmware/vmware_conf.c (vmwareVmxPath): Likewise.
* src/vmx/vmx.c (virVMXParseDisk, virVMXFormatDisk)
(virVMXFormatFloppy): Likewise.
* src/xenxs/xen_sxpr.c (xenParseSxprDisks, xenParseSxpr)
(xenFormatSxprDisk): Likewise.
* src/xenxs/xen_xm.c (xenParseXM, xenFormatXMDisk): Likewise.
* tests/securityselinuxlabeltest.c (testSELinuxLoadDef):
Likewise.
* src/libvirt_private.syms (domain_conf.h): Move symbols...
(virstoragefile.h): ...as appropriate.
Signed-off-by: NEric Blake <eblake@redhat.com>

https://bugzilla.redhat.com/show_bug.cgi?id=1072714

Use the "gluster" command line tool to retrieve information about remote
volumes on a gluster server to allow storage pool source lookup.

Unfortunately gluster doesn't provide a management library so that we
could use that directly, instead the RPC calls are hardcoded in the
command line tool.

Extract the NFS related stuff into a separate function and tidy up the
rest of the code so we can reuse it to add gluster backend detection.

Additionally avoid reporting of errors from "showmount" and return an
empty source list instead. This will help when adding other detection
backends.

Remove the original VolOpen implementation, which is now only used in
one spot.

And drop the original UpdateVolInfo. Makes it a bit easier to follow
the function usage.

And change the int parameter to an explicit bool.

The only storage-specific parameter is the pool object, which
is only used for passing to the callback function.

Signed-off-by: NMartin Kletzander <mkletzan@redhat.com>

Any source file which calls the logging APIs now needs
to have a VIR_LOG_INIT("source.name") declaration at
the start of the file. This provides a static variable
of the virLogSource type.
Signed-off-by: NDaniel P. Berrange <berrange@redhat.com>

Signed-off-by: NMichal Privoznik <mprivozn@redhat.com>

Implement the "stat" and "unlink" function for "file" volumes and "stat"
for "block" volumes using the regular system calls.

The comment was talking about creating the pool while the function is
deleting it. Fix the mismatch.

Currently, 'vol-resize --allocate' allocates new space at the
vol->capacity offset. But the vol->capacity is not necessarily the same
as vol->allocation. For instance:.

	[root@localhost ~]# virsh vol-list --pool tmp-pool --details
	 Name      Path                   Type  Capacity  Allocation
	-------------------------------------------------------------
	 tmp-vol  /root/tmp-pool/tmp-vol  file  1.00 GiB  1.00 GiB

	[root@localhost ~]# virsh vol-resize tmp-vol --pool tmp-pool 2G

	[root@localhost ~]# virsh vol-list --pool tmp-pool --details
	 Name      Path                   Type  Capacity  Allocation
	-------------------------------------------------------------
	 tmp-vol  /root/tmp-pool/tmp-vol  file  2.00 GiB  1.00 GiB

So, if we want to allocate more bytes, so the file is say 3G big, the
real allocated size is 2G actually:

	[root@localhost ~]# virsh vol-resize tmp-vol --pool tmp-pool 3G --allocate

	[root@localhost ~]# virsh vol-list --pool tmp-pool --details
	 Name      Path                   Type  Capacity  Allocation
	-------------------------------------------------------------
	 tmp-vol  /root/tmp-pool/tmp-vol  file  3.00 GiB  2.00 GiB

This commit uses the correct vol->allocation instead of incorrect
vol->capacity, so the output of the commands above looks like this:

	[root@localhost ~]# virsh vol-resize tmp-vol --pool tmp-pool 3G --allocate

	[root@localhost ~]# virsh vol-list --pool tmp-pool --details
	 Name      Path                   Type  Capacity  Allocation
	-------------------------------------------------------------
	 tmp-vol  /root/tmp-pool/tmp-vol  file  3.00 GiB  3.00 GiB

Moreover, if the '--alocate' flag was used, we must update the
vol->allocation member in storageVolResize API too, not just
vol->capacity.
Reported-by: NWang Sen <wangsen@linux.vnet.ibm.com>
Signed-off-by: NMichal Privoznik <mprivozn@redhat.com>

In the 'directory' and 'netfs' storage pools, a user can see
both 'file' and 'dir' storage volume types, to know when they
can descend into a subdirectory.  But in a network-based storage
pool, such as the upcoming 'gluster' pool, we use 'network'
instead of 'file', and did not have any counterpart for a
directory until this patch.  Adding a new volume type
'network-dir' is better than reusing 'dir', because it makes
it clear that the only way to access 'network' volumes within
that container is through the network mounting (leaving 'dir'
for something accessible in the local file system).

* include/libvirt/libvirt.h.in (virStorageVolType): Expand enum.
* docs/formatstorage.html.in: Document it.
* docs/schemasa/storagevol.rng (vol): Allow new value.
* src/conf/storage_conf.c (virStorageVol): Use new value.
* src/qemu/qemu_command.c (qemuBuildVolumeString): Fix client.
* src/qemu/qemu_conf.c (qemuTranslateDiskSourcePool): Likewise.
* tools/virsh-volume.c (vshVolumeTypeToString): Likewise.
* src/storage/storage_backend_fs.c
(virStorageBackendFileSystemVolDelete): Likewise.
Signed-off-by: NEric Blake <eblake@redhat.com>

This gets rid of another stat() per volume, as well as cutting
bytes read in half, when populating the volumes of a directory
pool during a pool refresh.  Not to mention that it provides an
interface that can let gluster pools also probe file types.

* src/util/virstoragefile.h (virStorageFileProbeFormatFromFD):
Delete.
(virStorageFileProbeFormatFromBuf): New prototype.
(VIR_STORAGE_MAX_HEADER): New constant, based on...
* src/util/virstoragefile.c (STORAGE_MAX_HEAD): ...old name.
(vmdk4GetBackingStore, virStorageFileGetMetadataInternal)
(virStorageFileProbeFormat): Adjust clients.
(virStorageFileProbeFormatFromFD): Delete.
(virStorageFileProbeFormatFromBuf): Export.
* src/storage/storage_backend_fs.c (virStorageBackendProbeTarget):
Adjust client.
* src/libvirt_private.syms (virstoragefile.h): Adjust exports.
Signed-off-by: NEric Blake <eblake@redhat.com>

We are calling fstat() at least twice per storage volume in
a directory storage pool; this is rather wasteful.  Refactoring
this is also a step towards making code reusable for gluster,
where gluster can provide struct stat but cannot use fstat().

* src/storage/storage_backend.h
(virStorageBackendVolOpenCheckMode)
(virStorageBackendUpdateVolTargetInfoFD): Update signature.
* src/storage/storage_backend.c
(virStorageBackendVolOpenCheckMode): Pass stat results back.
(virStorageBackendUpdateVolTargetInfoFD): Use existing stats.
(virStorageBackendVolOpen, virStorageBackendUpdateVolTargetInfo):
Update callers.
* src/storage/storage_backend_fs.c (virStorageBackendProbeTarget):
Likewise.
* src/storage/storage_backend_scsi.c
(virStorageBackendSCSIUpdateVolTargetInfo): Likewise.
* src/storage/storage_backend_mpath.c
(virStorageBackendMpathUpdateVolTargetInfo): Likewise.
Signed-off-by: NEric Blake <eblake@redhat.com>

Most of the usage of getuid()/getgid() is in cases where we are
considering what privileges we have. As such the code should be
using the effective IDs, not real IDs.
Signed-off-by: NDaniel P. Berrange <berrange@redhat.com>

Semantics of the libvirt helper are more clear. This change also allows
to clean up some pieces of code.

Don't reuse the return value of virStorageBackendFileSystemIsMounted.
If it's 0, we'd return it even if the mount command failed.

Also, don't report another error if it's -1, since one has already
been reported.

Introduced by 258e06c8.

https://bugzilla.redhat.com/show_bug.cgi?id=981251

Convert the type of loop iterators named 'i', 'j', k',
'ii', 'jj', 'kk', to be 'size_t' instead of 'int' or
'unsigned int', also santizing 'ii', 'jj', 'kk' to use
the normal 'i', 'j', 'k' naming
Signed-off-by: NDaniel P. Berrange <berrange@redhat.com>

Add <features> and <compat> elements to volume target XML.

<compat> is a string which for qcow2 represents the QEMU version
it should be compatible with. Valid values are 0.10 and 1.1.
1.1 is implicit if the <features> element is present, otherwise
qemu-img default is used. 0.10 can be specified to explicitly
create older images after the qemu-img default changes.

<features> contains optional features, so far
<lazy_refcounts/> is available, which enables caching of reference
counters, improving performance for snapshots.

The document for "vol-resize" says the new capacity will be sparse
unless "--allocate" is specified, however, the "--allocate" flag
is never implemented. This implements the "--allocate" flag for
fs backend's raw type volume, based on posix_fallocate and the
syscall SYS_fallocate.

qemu-img resize will fail with "The new size must be a multiple of 512"
if libvirt doesn't round it first.
This fixes rhbz#951495
Signed-off-by: NChristophe Fergeau <cfergeau@redhat.com>

The source code base needs to be adapted as well. Some files
include virutil.h just for the string related functions (here,
the include is substituted to match the new file), some include
virutil.h without any need (here, the include is removed), and
some require both.

The bfree and blocks fields are supposed to be in units of frsize.  We were
calculating capacity correctly using those units, but the available
calculation was using bsize instead.  Most file systems report these as the
same value specifically because many programs are buggy, but that is no
reason to rely on that behavior, or to behave inconsistently.

This bug has been present since e266ded2 (2008) and aa296e6c, when the code
was originally introduced (the latter via cut and paste).
Signed-off-by: NSage Weil <sage@newdream.net>

Signed-off-by: NDaniel P. Berrange <berrange@redhat.com>