https://bugzilla.redhat.com/show_bug.cgi?id=1073305

When creating a volume in a pool, the creation allows the 'capacity'
value to be larger than the available space in the pool. As long as
the 'allocation' value will fit in the space, the volume will be created.

However, resizing the volume checks were made with the new absolute
capacity value against existing capacity + the available space without
regard for whether the new absolute capacity was actually allocating
space or not.  For example, a pool with 75G of available space creates
a volume of 10G using a capacity of 100G and allocation of 10G will succeed;
however, if the allocation used a capacity of 10G instead and then tried
to resize the allocation to 100G the code would fail to allow the backend
to try the resize.

Furthermore, when updating the pool "available" and "allocation" values,
the resize code would just "blindly" adjust them regardless of whether
space was "allocated" or just "capacity" was being adjusted.  This left
a scenario whereby a resize to 100G would fail; however, a resize to 50G
followed by one to 100G would both succeed.  Again, neither was adjusting
the allocation value, just the "capacity" value.

This patch adds more logic to the resize code to understand whether the
new capacity value is actually "allocating" space as well and whether it
shrinking or expanding. Since unsigned arithmatic is involved, the possibility
that we adjust the pool size values incorrectly is probable.

This patch also ensures that updates to the pool values only occur if we
actually performed the allocation.

NB: The storageVolDelete, storageVolCreateXML, and storageVolCreateXMLFrom
each only updates the pool allocation/availability values by the target
volume allocation value.

The 'checkPool' callback was originally part of the storageDriverAutostart function,
but the pools need to be checked earlier during initialization phase,
otherwise we can't start a domain which mounts a volume after the
libvirtd daemon restarted. This is because qemuProcessReconnect is called
earlier than storageDriverAutostart. Therefore the 'checkPool' logic has been
moved to storagePoolUpdateAllState which is called inside storageDriverInitialize.

We also need a valid 'conn' reference to be able to execute 'refreshPool'
during initialization phase. Though it isn't available until storageDriverAutostart
all of our storage backends do ignore 'conn' pointer, except for RBD,
but RBD doesn't support 'checkPool' callback, so it's safe to pass
conn = NULL in this case.

Resolves: https://bugzilla.redhat.com/show_bug.cgi?id=1177733

These functions operate exactly the same as their network equivalents
virNetworkLoadAllState, virNetworkLoadState.

This patch introduces new virStorageDriverState element stateDir.
Also adds necessary changes to storageStateInitialize, so that
directories initialization becomes more generic.

In order to be able to use 'checkPool' inside functions which do not
have any connection reference, 'conn' attribute needs to be discarded
from the checkPool's signature, since it's not used by any storage backend
anyway.

The tool creating the image can get the capacity from the backing
storage. Just refresh the volume afterwards.

https://bugzilla.redhat.com/show_bug.cgi?id=958510

In virStorageVolCreateXML, add VIR_VOL_XML_PARSE_NO_CAPACITY
to the call parsing the XML of the new volume to make the capacity
optional.

If the capacity is omitted, use the capacity of the old volume.
We already do that for values that are less than the original
volume capacity.

Allow the callers to pass down libvirt-internal flags.

https://bugzilla.redhat.com/show_bug.cgi?id=1176510

When storageDriverAutostart is called path virStateReload via a 'service
libvirtd reload', then because the volume list in the pool wasn't cleared
prior to the call, each volume would be listed multiple times (as many
times as we reload). I believe the issue would be introduced by commit
id '9e093f0b' at least for the libvirtd reload path, although I suppose
the introduction of virStateReload (commit id '70da0494') could be a
different cause.

Thus like other places prior to calling refreshPool, we need to call
virStoragePoolObjClearVols

When creating a RAW file, we don't take advantage
of clone of btrfs.

Add a VIR_STORAGE_VOL_CREATE_REFLINK flag to request
a reflink copy.
Signed-off-by: NChen Hanxiao <chenhanxiao@cn.fujitsu.com>
Signed-off-by: NJán Tomko <jtomko@redhat.com>

For stateless, client side drivers, it is never correct to
probe for secondary drivers. It is only ever appropriate to
use the secondary driver that is associated with the
hypervisor in question. As a result the ESX & HyperV drivers
have both been forced to do hacks where they register no-op
drivers for the ones they don't implement.

For stateful, server side drivers, we always just want to
use the same built-in shared driver. The exception is
virtualbox which is really a stateless driver and so wants
to use its own server side secondary drivers. To deal with
this virtualbox has to be built as 3 separate loadable
modules to allow registration to work in the right order.

This can all be simplified by introducing a new struct
recording the precise set of secondary drivers each
hypervisor driver wants

struct _virConnectDriver {
    virHypervisorDriverPtr hypervisorDriver;
    virInterfaceDriverPtr interfaceDriver;
    virNetworkDriverPtr networkDriver;
    virNodeDeviceDriverPtr nodeDeviceDriver;
    virNWFilterDriverPtr nwfilterDriver;
    virSecretDriverPtr secretDriver;
    virStorageDriverPtr storageDriver;
};

Instead of registering the hypervisor driver, we now
just register a virConnectDriver instead. This allows
us to remove all probing of secondary drivers. Once we
have chosen the primary driver, we immediately know the
correct secondary drivers to use.
Signed-off-by: NDaniel P. Berrange <berrange@redhat.com>

Other parts of libvirt use "%u" for formatting uid/gid and typecast to
unsigned int. Storage driver used the signed variant.

https://bugzilla.redhat.com/show_bug.cgi?id=1087104#c5

When trying to use an invalid offset to virStorageVolUpload(), libvirt
fails in virFDStreamOpenFileInternal(), although it seems libvirt does
not check the return in storageVolUpload(), and calls
virFDStreamSetInternalCloseCb() right after.  But stream doesn't have a
privateData (is NULL) yet, and the daemon crashes then.

0  0x00007f09429a9c10 in pthread_mutex_lock () from /lib64/libpthread.so.0
1  0x00007f094514dbf5 in virMutexLock (m=<optimized out>) at util/virthread.c:88
2  0x00007f09451cb211 in virFDStreamSetInternalCloseCb at fdstream.c:795
3  0x00007f092ff2c9eb in storageVolUpload at storage/storage_driver.c:2098
4  0x00007f09451f46e0 in virStorageVolUpload at libvirt.c:14000
5  0x00007f0945c78fa1 in remoteDispatchStorageVolUpload at remote_dispatch.h:14339
6  remoteDispatchStorageVolUploadHelper at remote_dispatch.h:14309
7  0x00007f094524a192 in virNetServerProgramDispatchCall at rpc/virnetserverprogram.c:437
Signed-off-by: NLuyao Huang <lhuang@redhat.com>

While this could be exposed as a public API, it's not done yet as
there's no demand for that yet. Anyway, this is just preparing
the environment for easier volume creation on the destination.
Signed-off-by: NMichal Privoznik <mprivozn@redhat.com>

Since virStoragePoolFree will call virObjectUnref anyway, let's just use that
directly so as to avoid the possibility that we inadvertently clear out
a pending error message when using the public API.

Since virStorageVolFree will call virObjectUnref anyway, let's just use that
directly so as to avoid the possibility that we inadvertently clear out
a pending error message when using the public API.

https://bugzilla.redhat.com/show_bug.cgi?id=1159180

The virStoragePoolSourceFindDuplicate only checks the incoming definition
against the same type of pool as the def; however, for "scsi_host" and
"fc_host" adapter pools, it's possible that either some pool "scsi_host"
adapter definition is already using the scsi_hostN that the "fc_host"
adapter definition wants to use or some "fc_host" pool adapter definition
is using a vHBA scsi_hostN or parent scsi_hostN that an incoming "scsi_host"
definition is trying to use.

This patch adds the mismatched type checks and adds extraneous comments
to describe what each check is determining.

This patch also modifies the documentation to be describe what scsi_hostN
devices a "scsi_host" source adapter should use and which to avoid. It also
updates the parent definition to specifically call out that for mixed
environments it's better to define which parent to use so that the duplicate
pool checks can be done properly.

s/rereshed/refreshed
Signed-off-by: NChen Hanxiao <chenhanxiao@cn.fujitsu.com>

Signed-off-by: NChen Hanxiao <chenhanxiao@cn.fujitsu.com>

Signed-off-by: NMartin Kletzander <mkletzan@redhat.com>

The shared storage driver is stateful and inside the daemon so
there is no need to use the storagePrivateData field to get the
driver handle. Just access the global driver handle directly.

Report also the name of the parent file and uid/gid used to access it to
help debugging broken storage configurations.

Add a new parameter to virStorageFileGetMetadata that will break the
backing chain detection process and report useful error message rather
than having to use virStorageFileChainGetBroken.

This patch just introduces the option, usage will be provided
separately.

There were two occurrances of attempting to initialize actualType by
calling virStorageSourceGetActualType(src) prior to a check if (!src)
resulting in Coverity complaining about the possible NULL dereference
in virStorageSourceGetActualType() of src.

Resolve by moving the actualType setting until after checking !src

Introduced by commit 395171f8.
Signed-off-by: NChen Fan <chen.fan.fnst@cn.fujitsu.com>
Signed-off-by: NJán Tomko <jtomko@redhat.com>

Currently, qemu driver uses qemuTranslateDiskSourcePool()
to translate disk volume information. This function is
general enough and could be used for other drivers as well,
so move it to conf/domain_conf.c along with its helpers.

 - qemuTranslateDiskSourcePool: move to storage/storage_driver.c
   and rename to virStorageTranslateDiskSourcePool,
 - qemuAddISCSIPoolSourceHost: move to storage/storage_driver.c
   and rename to virStorageAddISCSIPoolSourceHost,
 - qemuTranslateDiskSourcePoolAuth: move to storage/storage_driver.c
   and rename to virStorageTranslateDiskSourcePoolAuth,
 - Update users of qemuTranslateDiskSourcePool to use a
   new name.

Implement ZFS storage backend driver. Currently supported
only on FreeBSD because of ZFS limitations on Linux.

Features supported:

 - pool-start, pool-stop
 - pool-info
 - vol-list
 - vol-create / vol-delete

Pool definition looks like that:

 <pool type='zfs'>
  <name>myzfspool</name>
  <source>
    <name>actualpoolname</name>
  </source>
 </pool>

The 'actualpoolname' value is a name of the pool on the system,
such as shown by 'zpool list' command. Target makes no sense
here because volumes path is always /dev/zvol/$poolname/$volname.

User has to create a pool on his own, this driver doesn't
support pool creation currently.

A volume could be used with Qemu by adding an entry like this:

    <disk type='volume' device='disk'>
      <driver name='qemu' type='raw'/>
      <source pool='myzfspool' volume='vol5'/>
      <target dev='hdc' bus='ide'/>
    </disk>

https://bugzilla.redhat.com/show_bug.cgi?id=1072653

Upon successful upload of a volume, the target volume and storage pool
were not updated to reflect any changes as a result of the upload. Make
use of the existing stream close callback mechanism to force a backend
pool refresh to occur in a separate thread once the stream closes. The
separate thread should avoid potential deadlocks if the refresh needed
to wait on some event from the event loop which is used to perform
the stream callback.

With my intended use of storage driver assist to chown files on remote
storage we will need a witness that will tell us whether the given
storage volume supports operations needed by the storage driver.

Gluster storage works on a similar principle to NFS where it takes the
uid and gid of the actual process and uses it to access the storage
volume on the remote server. This introduces a need to chown storage
files on gluster via native API.

Rather than a unsigned int, use a 'bool' since that's how it was used.

For non-local storage drivers we can't expect to use the "scrub" tool to
wipe the volume. Split the code into a separate backend function so that
we can add protocol specific code later.

Resolves: https://bugzilla.redhat.com/show_bug.cgi?id=1118710

The next patch will move the storage volume wiping code into the
individual backends. This patch splits out the common code to wipe a
local volume into a separate backend helper so that the next patch is
simpler.

For non-local storage drivers we can't expect to use the FDStream
backend for up/downloading volumes. Split the code into a separate
backend function so that we can add protocol specific code later.

Allow de-init of null storage sources.

The parent directory doesn't necessarily need to be stored after we
don't mangle the path stored in the image. Remove it and tweak the code
to avoid using it.

Due to various refactors and compatibility with the virstoragetest the
relPath field of the virStorageSource structure was always filled either
with the relative name or the full path in case of absolutely backed
storage. Return its original purpose to store only the relative name of
the disk if it is backed relatively and tweak the tests.

Introduced by commit 2bdb8b96

Add a helper to do all the lookup steps and remove a ton of duplicated
code.

Most of the APIs now don't reach the cleanup section when the pool
object wasn't found and thus don't need to check before unlocking it.