Commit fff74b27 moved the probing into virStorageFileGetMetadataFromBuf
but didn't update the format in volume definition.

https://bugzilla.redhat.com/show_bug.cgi?id=1104908

Jump out early if no metadata was detected (for directories).
Join the error and cleanup labels.

To match the convention:
ret - current function's return value
rc - other function's return values

Add a storage driver API equivalent of the access() function.
Implementations for the filesystem and gluster backends are provided.

Different protocols have different means to uniquely identify a storage
file. This patch implements a storage driver API to retrieve a unique
string describing a volume. The current implementation works for local
storage only and returns the canonical path of the volume.

To add caching support the local filesystem driver now has a private
structure holding the cached string, which is created only when it's
initially accessed.

This patch provides the implementation for local files only for start.

Use virStorageFileGetMetadataFromFD instead in
virStorageBackendProbeTarget as it now returns all required data and the
storage file is already open in a filedescriptor.

Also fix improper error code being returned when virFileReadHeaderFD
would fail as virStorageBackendUpdateVolTargetInfoFD would set the
return code to 0.

Add storage driver based functions to access headers of storage files
for metadata extraction. Along with this patch a local filesystem and
gluster via libgfapi implementation is provided. The gluster
implementation is based on code of the saferead_lim function.

Print the debug statements of individual file access functions from the
main API functions instead of the individual backend functions.

Also enhance initialization debug messages on a per-backend basis.

All callers of virStorageFileGetMetadataFromBuf were first calling
virStorageFileProbeFormatFromBuf, to learn what format to pass in.
But this function is already wired to do the exact same probe if
the incoming format is VIR_STORAGE_FILE_AUTO, so it's simpler to
just refactor the probing into the central function.

* src/util/virstoragefile.h (virStorageFileGetMetadataFromBuf):
Drop parameter.
(virStorageFileProbeFormatFromBuf): Drop declaration.
* src/util/virstoragefile.c (virStorageFileGetMetadataFromBuf):
Do probe here instead of in callers.
(virStorageFileProbeFormatFromBuf): Make static.
* src/libvirt_private.syms (virstoragefile.h): Drop function.
* src/storage/storage_backend_fs.c (virStorageBackendProbeTarget):
Update caller.
* src/storage/storage_backend_gluster.c
(virStorageBackendGlusterRefreshVol): Likewise.
Signed-off-by: NEric Blake <eblake@redhat.com>

Commit id 'ac9a0963' refactored out the 'withCapacity' for the
virStorageBackendUpdateVolInfo() API.  See:

http://www.redhat.com/archives/libvir-list/2014-April/msg00043.html

This resulted in a difference in how 'virsh vol-info --pool <poolName>
<volume>' or 'virsh vol-list vol-list --pool <poolName> --details' outputs
the capacity information for a directory pool with a qcow2 sparse file.

For example, using the following XML

mkdir /home/TestPool
cat testpool.xml
<pool type='dir'>
  <name>TestPool</name>
  <uuid>6bf80895-10b6-75a6-6059-89fdea2aefb7</uuid>
  <source>
  </source>
  <target>
    <path>/home/TestPool</path>
    <permissions>
      <mode>0755</mode>
      <owner>0</owner>
      <group>0</group>
    </permissions>
  </target>
</pool>

virsh pool-create testpool.xml
virsh vol-create-as --pool TestPool temp_vol_1 \
      --capacity 1048576 --allocation 1048576 --format qcow2
virsh vol-info --pool TestPool temp_vol_1

Results in listing a Capacity value.  Prior to the commit, the value would
be '1.0 MiB' (1048576 bytes). However, after the commit the output would be
(for example) '192.50 KiB', which for my system was the size of the volume
in my file system (eg 'ls -l TestPool/temp_vol_1' results in '197120' bytes
or 192.50 KiB). While perhaps technically correct, it's not necessarily
what the user expected (certainly virt-test didn't expect it).

This patch restores the code to not update the target capacity for this path

More instances of failure to report (unlikely) readdir errors.
In one case, I chose to ignore them, given that a readdir error
would be no different than timing out on the loop, where the
fallback path behaves correctly either way.

* src/storage/storage_backend.c (virStorageBackendStablePath):
Ignore readdir errors.
* src/storage/storage_backend_fs.c
(virStorageBackendFileSystemRefresh): Report readdir errors.
* src/storage/storage_backend_iscsi.c
(virStorageBackendISCSIGetHostNumber): Likewise.
* src/storage/storage_backend_scsi.c (getNewStyleBlockDevice)
(getBlockDevice, virStorageBackendSCSIFindLUs): Likewise.
Signed-off-by: NEric Blake <eblake@redhat.com>

Replace the old structure with the new one. This change is a trivial
name change operation (along with change of the freeing function).

Fix minor typos in source comments
Signed-off-by: NEric Blake <eblake@redhat.com>

Commit id '18642d10' caused a virt-test regression for NFS backend
storage error path checks when running the command:

    'virsh find-storage-pool-sources-as netfs Unknown  '

when the host did not have Gluster installed. Prior to the commit,
the test would fail with the error:

    error: internal error: Child process (/usr/sbin/showmount --no-headers
    --exports Unknown) unexpected exit status 1: clnt_create: RPC: Unknown host

After the commit, the error would be ignored, the call would succeed,
and an empty list of pool sources returned. This was tucked into the
commit message as an expected outcome.

When the target host does not have a GLUSTER_CLI this is a regression
over the previous release. Furthermore, even if Gluster CLI was present,
but had a failure to get devices, the API would return a failure even if
the NFS backend had found devices.

Modify the logic to return failure when the NFS backend check fails and
there's no GLUSTER_CLI or when both backend checks fail.

If either returns success and GLUSTER_CLI is defined, then fetch and return
a list of source devices even if it's empty

A couple pieces of virStorageFileMetadata are used only while
collecting information about the chain, and don't need to
live permanently in the struct.  This patch refactors external
callers to collect the information separately, so that the
next patch can remove the fields.

* src/util/virstoragefile.h (virStorageFileGetMetadataFromBuf):
Alter signature.
* src/util/virstoragefile.c (virStorageFileGetMetadataInternal):
Likewise.
(virStorageFileGetMetadataFromFDInternal): Adjust callers.
* src/storage/storage_backend_fs.c (virStorageBackendProbeTarget):
Likewise.
* src/storage/storage_backend_gluster.c
(virStorageBackendGlusterRefreshVol): Likewise.
Signed-off-by: NEric Blake <eblake@redhat.com>

Deciding if a user string represents a local file instead of a
network path is an operation worth exposing directly, particularly
since the next patch will be removing a redundant variable that
was caching the information.

* src/util/virstoragefile.h (virStorageIsFile): New declaration.
* src/util/virstoragefile.c (virBackingStoreIsFile): Rename...
(virStorageIsFile): ...export, and allow NULL input.
(virStorageFileGetMetadataInternal)
(virStorageFileGetMetadataRecurse, virStorageFileGetMetadata):
Update callers.
* src/conf/domain_conf.c (virDomainDiskDefForeachPath): Use it.
* src/storage/storage_backend_fs.c (virStorageBackendProbeTarget):
Likewise.
* src/libvirt_private.syms (virstoragefile.h): Export function.
Signed-off-by: NEric Blake <eblake@redhat.com>

Now that we store all metadata about a storage image in a
virStorageSource struct let's use it also to store information needed by
the storage driver to access and do operations on the files.

Commit id '18642d10' refactored the call to virCommandRunRegex()
inside a new function virStorageBackendFileSystemNetFindNFSPoolSources(),
but the cut-n-paste didn't remove the "&state".  Now that state is passed
by reference, it results in a libvirtd core with a messages entry:

"...internal error: unknown storage pool type Unknow"

Without this, building an XFS pool on a formatted partition
fails with --overwrite.

https://bugzilla.redhat.com/show_bug.cgi?id=927172

Currently VolOpen notifies the user of a potentially non-fatal failure by
returning -2 and logging a VIR_WARN or VIR_INFO. Unfortunately most
callers treat -2 as fatal but don't actually report any message with
the error APIs.

Rename the VOL_OPEN_ERROR flag to VOL_OPEN_NOERROR. If NOERROR is specified,
we preserve the current behavior of returning -2 (there's only one caller
that wants this).

However in the default case, only return -1, and actually use the error
APIs. Fix up a couple callers as a result.

A future patch will merge virStorageFileMetadata and virStorageSource,
but I found it easier to do if both structs use the same information
for tracking whether a source file needs encryption keys.

* src/util/virstoragefile.h (_virStorageFileMetadata): Prepare
full encryption struct instead of just a bool.
* src/storage/storage_backend_fs.c (virStorageBackendProbeTarget):
Use transfer semantics.
* src/storage/storage_backend_gluster.c
(virStorageBackendGlusterRefreshVol): Likewise.
* src/util/virstoragefile.c (virStorageFileGetMetadataInternal):
Populate struct.
(virStorageFileFreeMetadata): Adjust clients.
* tests/virstoragetest.c (testStorageChain): Likewise.
Signed-off-by: NEric Blake <eblake@redhat.com>

Now that each virStorageSource can track allocation information,
and given that we already have the information without extra
syscalls, it's easier to just always populate the information
directly into the struct than it is to sometimes pass the address
of the struct members down the call chain.

* src/storage/storage_backend.h (virStorageBackendUpdateVolInfo)
(virStorageBackendUpdateVolTargetInfo)
(virStorageBackendUpdateVolTargetInfoFD): Update signature.
* src/storage/storage_backend.c (virStorageBackendUpdateVolInfo)
(virStorageBackendUpdateVolTargetInfo)
(virStorageBackendUpdateVolTargetInfoFD): Always populate struct
members instead.
* src/storage/storage_backend_disk.c
(virStorageBackendDiskMakeDataVol): Update client.
* src/storage/storage_backend_fs.c (virStorageBackendProbeTarget)
(virStorageBackendFileSystemRefresh)
(virStorageBackendFileSystemVolRefresh): Likewise.
* src/storage/storage_backend_gluster.c
(virStorageBackendGlusterRefreshVol): Likewise.
* src/storage/storage_backend_logical.c
(virStorageBackendLogicalMakeVol): Likewise.
* src/storage/storage_backend_mpath.c
(virStorageBackendMpathNewVol): Likewise.
* src/storage/storage_backend_scsi.c
(virStorageBackendSCSINewLun): Likewise.
Signed-off-by: NEric Blake <eblake@redhat.com>

One of the features of qcow2 is that a wrapper file can have
more capacity than its backing file from the guest's perspective;
what's more, sparse files make tracking allocation of both
the active and backing file worthwhile.  As such, it makes
more sense to show allocation numbers for each file in a chain,
and not just the top-level file.  This sets up the fields for
the tracking, although it does not modify XML to display any
new information.

* src/util/virstoragefile.h (_virStorageSource): Add fields.
* src/conf/storage_conf.h (_virStorageVolDef): Drop redundant
fields.
* src/storage/storage_backend.c (virStorageBackendCreateBlockFrom)
(createRawFile, virStorageBackendCreateQemuImgCmd)
(virStorageBackendCreateQcowCreate): Update clients.
* src/storage/storage_driver.c (storageVolDelete)
(storageVolCreateXML, storageVolCreateXMLFrom, storageVolResize)
(storageVolWipeInternal, storageVolGetInfo): Likewise.
* src/storage/storage_backend_fs.c (virStorageBackendProbeTarget)
(virStorageBackendFileSystemRefresh)
(virStorageBackendFileSystemVolResize)
(virStorageBackendFileSystemVolRefresh): Likewise.
* src/storage/storage_backend_logical.c
(virStorageBackendLogicalMakeVol)
(virStorageBackendLogicalCreateVol): Likewise.
* src/storage/storage_backend_scsi.c
(virStorageBackendSCSINewLun): Likewise.
* src/storage/storage_backend_mpath.c
(virStorageBackendMpathNewVol): Likewise.
* src/storage/storage_backend_rbd.c
(volStorageBackendRBDRefreshVolInfo)
(virStorageBackendRBDCreateImage): Likewise.
* src/storage/storage_backend_disk.c
(virStorageBackendDiskMakeDataVol)
(virStorageBackendDiskCreateVol): Likewise.
* src/storage/storage_backend_sheepdog.c
(virStorageBackendSheepdogBuildVol)
(virStorageBackendSheepdogParseVdiList): Likewise.
* src/storage/storage_backend_gluster.c
(virStorageBackendGlusterRefreshVol): Likewise.
* src/conf/storage_conf.c (virStorageVolDefFormat)
(virStorageVolDefParseXML): Likewise.
* src/test/test_driver.c (testOpenVolumesForPool)
(testStorageVolCreateXML, testStorageVolCreateXMLFrom)
(testStorageVolDelete, testStorageVolGetInfo): Likewise.
* src/esx/esx_storage_backend_iscsi.c (esxStorageVolGetXMLDesc):
Likewise.
* src/esx/esx_storage_backend_vmfs.c (esxStorageVolGetXMLDesc)
(esxStorageVolCreateXML): Likewise.
* src/parallels/parallels_driver.c (parallelsAddHddByVolume):
Likewise.
* src/parallels/parallels_storage.c (parallelsDiskDescParseNode)
(parallelsStorageVolDefineXML, parallelsStorageVolCreateXMLFrom)
(parallelsStorageVolDefRemove, parallelsStorageVolGetInfo):
Likewise.
* src/vbox/vbox_tmpl.c (vboxStorageVolCreateXML)
(vboxStorageVolGetXMLDesc): Likewise.
* tests/storagebackendsheepdogtest.c (test_vdi_list_parser):
Likewise.
* src/phyp/phyp_driver.c (phypStorageVolCreateXML): Likewise.

A fairly smooth transition.  And now that domain disks and
storage volumes share a common struct, it opens the doors for
a future patch to expose more details in the XML for both
objects.

* src/conf/storage_conf.h (_virStorageVolTarget): Delete.
(_virStorageVolDef): Use common type.
* src/conf/storage_conf.c (virStorageVolDefFree)
(virStorageVolTargetDefFormat): Update clients.
* src/storage/storage_backend.h: Likewise.
* src/storage/storage_backend.c
(virStorageBackendDetectBlockVolFormatFD)
(virStorageBackendUpdateVolTargetInfo)
(virStorageBackendUpdateVolTargetInfoFD): Likewise.
* src/storage/storage_backend_fs.c (virStorageBackendProbeTarget):
Likewise.
Signed-off-by: NEric Blake <eblake@redhat.com>

Some preparatory work before consolidating storage volume
structs with the rest of virstoragefile.  Making these
changes allows a volume target to be much closer to (a
subset of) the virStorageSource struct.

Making perms be a pointer allows it to be optional if we
have a storage pool that doesn't expose permissions in a
way we can access.  It also allows future patches to
optionally expose permissions details learned about a disk
image via domain <disk> listings, rather than just
limiting it to storage volume listings.

Disk partition types was only used by internal code to
control what type of partition to create when carving up
an MS-DOS partition table storage pool (and is not used
for GPT partition tables or other storage pools).  It was
not exposed in volume XML, and as it is more closely
related to extent information of the overall block device
than it is to the <target> information describing the host
file.  Besides, if we ever decide to expose it in XML down
the road, we can move it back as needed.

* src/conf/storage_conf.h (_virStorageVolTarget): Change perms to
pointer, enhance comments.  Move partition type...
(_virStorageVolSource): ...here.
* src/conf/storage_conf.c (virStorageVolDefFree)
(virStorageVolDefParseXML, virStorageVolTargetDefFormat): Update
clients.
* src/storage/storage_backend_fs.c (createFileDir): Likewise.
* src/storage/storage_backend.c (virStorageBackendCreateBlockFrom)
(virStorageBackendCreateRaw, virStorageBackendCreateExecCommand)
(virStorageBackendUpdateVolTargetInfoFD): Likewise.
* src/storage/storage_backend_logical.c
(virStorageBackendLogicalCreateVol): Likewise.
* src/storage/storage_backend_disk.c
(virStorageBackendDiskMakeDataVol)
(virStorageBackendDiskPartTypeToCreate): Likewise.
Signed-off-by: NEric Blake <eblake@redhat.com>

A continuation of the migration of disk details to virstoragefile.
This patch moves a single enum, but converting the name has quite
a bit of fallout.

* src/conf/domain_conf.h (virDomainDiskType): Move...
* src/util/virstoragefile.h (virStorageType): ...and rename.
* src/bhyve/bhyve_command.c (bhyveBuildDiskArgStr)
(virBhyveProcessBuildLoadCmd): Update clients.
* src/conf/domain_conf.c (virDomainDiskSourceDefParse)
(virDomainDiskDefParseXML, virDomainDiskSourceDefFormatInternal)
(virDomainDiskDefFormat, virDomainDiskGetActualType)
(virDomainDiskDefForeachPath, virDomainDiskSourceIsBlockType):
Likewise.
* src/conf/snapshot_conf.h (_virDomainSnapshotDiskDef): Likewise.
* src/conf/snapshot_conf.c (virDomainSnapshotDiskDefParseXML)
(virDomainSnapshotAlignDisks, virDomainSnapshotDiskDefFormat):
Likewise.
* src/esx/esx_driver.c (esxAutodetectSCSIControllerModel)
(esxDomainDefineXML): Likewise.
* src/locking/domain_lock.c (virDomainLockManagerAddDisk):
Likewise.
* src/lxc/lxc_controller.c
(virLXCControllerSetupLoopDeviceDisk)
(virLXCControllerSetupNBDDeviceDisk)
(virLXCControllerSetupLoopDevices, virLXCControllerSetupDisk):
Likewise.
* src/parallels/parallels_driver.c (parallelsGetHddInfo):
Likewise.
* src/phyp/phyp_driver.c (phypDiskType): Likewise.
* src/qemu/qemu_command.c (qemuGetDriveSourceString)
(qemuDomainDiskGetSourceString, qemuBuildDriveStr)
(qemuBuildCommandLine, qemuParseCommandLineDisk)
(qemuParseCommandLine): Likewise.
* src/qemu/qemu_conf.c (qemuCheckSharedDevice)
(qemuTranslateDiskSourcePool)
(qemuTranslateSnapshotDiskSourcePool): Likewise.
* src/qemu/qemu_domain.c (qemuDomainDeviceDefPostParse)
(qemuDomainDetermineDiskChain): Likewise.
* src/qemu/qemu_driver.c (qemuDomainGetBlockInfo)
(qemuDomainSnapshotPrepareDiskExternalBackingInactive)
(qemuDomainSnapshotPrepareDiskExternalBackingActive)
(qemuDomainSnapshotPrepareDiskExternalOverlayActive)
(qemuDomainSnapshotPrepareDiskExternalOverlayInactive)
(qemuDomainSnapshotPrepareDiskInternal)
(qemuDomainSnapshotPrepare)
(qemuDomainSnapshotCreateSingleDiskActive): Likewise.
* src/qemu/qemu_hotplug.c (qemuDomainChangeEjectableMedia):
Likewise.
* src/qemu/qemu_migration.c (qemuMigrationIsSafe): Likewise.
* src/security/security_apparmor.c
(AppArmorRestoreSecurityImageLabel)
(AppArmorSetSecurityImageLabel): Likewise.
* src/security/security_dac.c (virSecurityDACSetSecurityImageLabel)
(virSecurityDACRestoreSecurityImageLabelInt)
(virSecurityDACSetSecurityAllLabel): Likewise.
* src/security/security_selinux.c
(virSecuritySELinuxRestoreSecurityImageLabelInt)
(virSecuritySELinuxSetSecurityImageLabel)
(virSecuritySELinuxSetSecurityAllLabel): Likewise.
* src/storage/storage_backend.c (virStorageFileBackendForType):
Likewise.
* src/storage/storage_backend_fs.c (virStorageFileBackendFile)
(virStorageFileBackendBlock): Likewise.
* src/storage/storage_backend_gluster.c
(virStorageFileBackendGluster): Likewise.
* src/vbox/vbox_tmpl.c (vboxDomainGetXMLDesc, vboxAttachDrives)
(vboxDomainAttachDeviceImpl, vboxDomainDetachDevice): Likewise.
* src/vmware/vmware_conf.c (vmwareVmxPath): Likewise.
* src/vmx/vmx.c (virVMXParseDisk, virVMXFormatDisk)
(virVMXFormatFloppy): Likewise.
* src/xenxs/xen_sxpr.c (xenParseSxprDisks, xenParseSxpr)
(xenFormatSxprDisk): Likewise.
* src/xenxs/xen_xm.c (xenParseXM, xenFormatXMDisk): Likewise.
* tests/securityselinuxlabeltest.c (testSELinuxLoadDef):
Likewise.
* src/libvirt_private.syms (domain_conf.h): Move symbols...
(virstoragefile.h): ...as appropriate.
Signed-off-by: NEric Blake <eblake@redhat.com>

https://bugzilla.redhat.com/show_bug.cgi?id=1072714

Use the "gluster" command line tool to retrieve information about remote
volumes on a gluster server to allow storage pool source lookup.

Unfortunately gluster doesn't provide a management library so that we
could use that directly, instead the RPC calls are hardcoded in the
command line tool.

Extract the NFS related stuff into a separate function and tidy up the
rest of the code so we can reuse it to add gluster backend detection.

Additionally avoid reporting of errors from "showmount" and return an
empty source list instead. This will help when adding other detection
backends.

Remove the original VolOpen implementation, which is now only used in
one spot.

And drop the original UpdateVolInfo. Makes it a bit easier to follow
the function usage.

And change the int parameter to an explicit bool.

The only storage-specific parameter is the pool object, which
is only used for passing to the callback function.

Signed-off-by: NMartin Kletzander <mkletzan@redhat.com>

Any source file which calls the logging APIs now needs
to have a VIR_LOG_INIT("source.name") declaration at
the start of the file. This provides a static variable
of the virLogSource type.
Signed-off-by: NDaniel P. Berrange <berrange@redhat.com>

Signed-off-by: NMichal Privoznik <mprivozn@redhat.com>

Implement the "stat" and "unlink" function for "file" volumes and "stat"
for "block" volumes using the regular system calls.

The comment was talking about creating the pool while the function is
deleting it. Fix the mismatch.

Currently, 'vol-resize --allocate' allocates new space at the
vol->capacity offset. But the vol->capacity is not necessarily the same
as vol->allocation. For instance:.

	[root@localhost ~]# virsh vol-list --pool tmp-pool --details
	 Name      Path                   Type  Capacity  Allocation
	-------------------------------------------------------------
	 tmp-vol  /root/tmp-pool/tmp-vol  file  1.00 GiB  1.00 GiB

	[root@localhost ~]# virsh vol-resize tmp-vol --pool tmp-pool 2G

	[root@localhost ~]# virsh vol-list --pool tmp-pool --details
	 Name      Path                   Type  Capacity  Allocation
	-------------------------------------------------------------
	 tmp-vol  /root/tmp-pool/tmp-vol  file  2.00 GiB  1.00 GiB

So, if we want to allocate more bytes, so the file is say 3G big, the
real allocated size is 2G actually:

	[root@localhost ~]# virsh vol-resize tmp-vol --pool tmp-pool 3G --allocate

	[root@localhost ~]# virsh vol-list --pool tmp-pool --details
	 Name      Path                   Type  Capacity  Allocation
	-------------------------------------------------------------
	 tmp-vol  /root/tmp-pool/tmp-vol  file  3.00 GiB  2.00 GiB

This commit uses the correct vol->allocation instead of incorrect
vol->capacity, so the output of the commands above looks like this:

	[root@localhost ~]# virsh vol-resize tmp-vol --pool tmp-pool 3G --allocate

	[root@localhost ~]# virsh vol-list --pool tmp-pool --details
	 Name      Path                   Type  Capacity  Allocation
	-------------------------------------------------------------
	 tmp-vol  /root/tmp-pool/tmp-vol  file  3.00 GiB  3.00 GiB

Moreover, if the '--alocate' flag was used, we must update the
vol->allocation member in storageVolResize API too, not just
vol->capacity.
Reported-by: NWang Sen <wangsen@linux.vnet.ibm.com>
Signed-off-by: NMichal Privoznik <mprivozn@redhat.com>