Add a new storage driver registration function that will force the
backend code to fail if any of the storage backend modules can't be
loaded. This will make sure that they work and are present.

Add APIs that allow to dynamically register driver backends so that the
list of available drivers does not need to be known during compile time.

This will allow us to modularize the storage driver on runtime.

Added general definitions for vstorage pool backend including
the build options to add --with-storage-vstorage checking.
In order to use vstorage as a backend for a storage pool
vstorage tools (vstorage and vstorage-mount) need to be installed.
Signed-off-by: NOlga Krishtal <okrishtal@virtuozzo.com>

The file became a garbage dump for all kinds of utility functions over
time. Move them to a separate file so that the files can become a clean
interface for the storage backends.

The code at the very bottom of the DAC secdriver that calls
chown() should be fine with read-only data. If something needs to
be prepared it should have been done beforehand.
Signed-off-by: NMichal Privoznik <mprivozn@redhat.com>

https://bugzilla.redhat.com/show_bug.cgi?id=1332019

This function will essentially be a wrapper to virStorageVolInfo in order
to provide a mechanism to have the "physical" size of the volume returned
instead of the "allocation" size. This will provide similar capabilities to
the virDomainBlockInfo which can return both allocation and physical of a
domain storage volume.

NB: Since we're reusing the _virStorageVolInfo and not creating a new
_virStorageVolInfoFlags structure, we'll need to generate the rpc APIs
remoteStorageVolGetInfoFlags and remoteDispatchStorageVolGetInfoFlags
(although both were originally created from gendispatch.pl and then
just copied into daemon/remote.c and src/remote/remote_driver.c).

The new API will allow the usage of a VIR_STORAGE_VOL_GET_PHYSICAL flag
and will make the decision to return the physical or allocation value
into the allocation field.

In order to get that physical value, virStorageBackendUpdateVolTargetInfoFD
adds logic to fill in physical value matching logic in qemuStorageLimitsRefresh
used by virDomainBlockInfo when the domain is inactive.
Signed-off-by: NJohn Ferlan <jferlan@redhat.com>

Commit id '03e750f3' added support for checking the PLOOP type; however,
it used 'target.type' which no storage code ever fills in, so it will
never be set.  Change to just vol->type (could use vol->target.format
as well).

We have couple of functions that operate over NULL terminated
lits of strings. However, our naming sucks:

virStringJoin
virStringFreeList
virStringFreeListCount
virStringArrayHasString
virStringGetFirstWithPrefix

We can do better:

virStringListJoin
virStringListFree
virStringListFreeCount
virStringListHasString
virStringListGetFirstWithPrefix
Signed-off-by: NMichal Privoznik <mprivozn@redhat.com>

New line character in name of storagepool is now forbidden because it
mess virsh output and can be confusing for users.
Validation of name is done in driver, after parsing XML to avoid
problems with dissappeared pools which was already created with
new-line char in name.
Signed-off-by: NMichal Privoznik <mprivozn@redhat.com>

This way we get reference counting and we can get rid of locking
function.
Signed-off-by: NMartin Kletzander <mkletzan@redhat.com>

https://bugzilla.redhat.com/show_bug.cgi?id=1362349

When adding the ability to build the pool during the start pool processing
using the similar flags as buildPool processing would use, the code was
essentially cut-n-pasted from storagePoolCreateXML.  However, that included
a call to virStoragePoolObjRemove which shouldn't happen within the
storagePoolCreate path since that'll remove the pool from the list of
pools only to be rediscovered if libvirtd restarts.

So on failure, just fail and return as we should expect

Vz containers are able to use ploop volumes from storage pools
to work upon.

To use filesystem type volume, pool name and volume name should be
specifaed in <source> :
   <filesystem type='volume' accessmode='passthrough'>
      <driver type='ploop' format='ploop'/>
      <source pool='guest_images' volume='TEST_POOL_CT'/>
      <target dir='/'/>
   </filesystem>

The information about pool and volume is stored in ct dom configuration:
<StorageURL>libvirt://localhost/pool_name/vol_name</StorageURL>
and can be easily obtained via PrlVmDevHd_GetStorageURL sdk call.

The only shorcoming: if storage pool is moved somewhere the ct
should be redefined in order to refresh the information aboot path
to root.hdd
Signed-off-by: NOlga Krishtal <okrishtal@virtuozzo.com>

https://bugzilla.redhat.com/show_bug.cgi?id=1316370

Consider the following disk for a domain:

    <disk type='volume' device='cdrom'>
      <driver name='qemu' type='raw'/>
      <auth username='libvirt'>
        <secret type='iscsi' usage='libvirtiscsi'/>
      </auth>
      <source pool='iscsi-secret-pool' volume='unit:0:0:0' mode='direct' startupPolicy='optional'/>
      <target dev='sda' bus='scsi'/>
      <readonly/>
      <address type='drive' controller='0' bus='0' target='0' unit='0'/>
    </disk>

Now, startupPolicy is currently not allowed for iscsi disks, so
one would expect an error message to be thrown. But what a
surprise is waiting for users if they try to start up such
domain:

==15724== Invalid free() / delete / delete[] / realloc()
==15724==    at 0x4C2B1F0: free (vg_replace_malloc.c:473)
==15724==    by 0x54B7A69: virFree (viralloc.c:582)
==15724==    by 0x552DC90: virStorageAuthDefFree (virstoragefile.c:1549)
==15724==    by 0x552F023: virStorageSourceClear (virstoragefile.c:2055)
==15724==    by 0x552F054: virStorageSourceFree (virstoragefile.c:2067)
==15724==    by 0x55556AA: virDomainDiskDefFree (domain_conf.c:1562)
==15724==    by 0x5557ABE: virDomainDefFree (domain_conf.c:2547)
==15724==    by 0x1B43CC42: qemuProcessStop (qemu_process.c:5918)
==15724==    by 0x1B43BA2E: qemuProcessStart (qemu_process.c:5511)
==15724==    by 0x1B48993E: qemuDomainObjStart (qemu_driver.c:7050)
==15724==    by 0x1B489B9A: qemuDomainCreateWithFlags (qemu_driver.c:7104)
==15724==    by 0x1B489C01: qemuDomainCreate (qemu_driver.c:7122)
==15724==  Address 0x21cfbb90 is 0 bytes inside a block of size 48 free'd
==15724==    at 0x4C2B1F0: free (vg_replace_malloc.c:473)
==15724==    by 0x54B7A69: virFree (viralloc.c:582)
==15724==    by 0x552DC90: virStorageAuthDefFree (virstoragefile.c:1549)
==15724==    by 0x12D1C8D4: virStorageTranslateDiskSourcePool (storage_driver.c:3475)
==15724==    by 0x1B4396E4: qemuProcessPrepareDomain (qemu_process.c:4896)
==15724==    by 0x1B43B880: qemuProcessStart (qemu_process.c:5466)
==15724==    by 0x1B48993E: qemuDomainObjStart (qemu_driver.c:7050)
==15724==    by 0x1B489B9A: qemuDomainCreateWithFlags (qemu_driver.c:7104)
==15724==    by 0x1B489C01: qemuDomainCreate (qemu_driver.c:7122)
==15724==    by 0x561CA97: virDomainCreate (libvirt-domain.c:6787)
==15724==    by 0x12B6FD: remoteDispatchDomainCreate (remote_dispatch.h:4116)
==15724==    by 0x12B61A: remoteDispatchDomainCreateHelper (remote_dispatch.h:4092)

The problem is, in virStorageTranslateDiskSourcePool disk
def->src->auth is freed, but the pointer is not set to NULL. So
later, when qemuProcessStop starts to free the domain definition,
virStorageAuthDefFree() tries to free the memory again, instead
of jumping out immediately.
Signed-off-by: NMichal Privoznik <mprivozn@redhat.com>

Create a function to return a temporary file path to be used in a mkostemp
type call using the path to the stateDir + pool->def->name + vol->name
Signed-off-by: NJohn Ferlan <jferlan@redhat.com>

The VIR_STORAGE_POOL_EVENT_REFRESHED constant does not
reflect any change in the lifecycle of the storage pool.

It should thus not be part of the storage pool lifecycle
event set, but rather be a top level event in its own
right. Thus we introduce VIR_STORAGE_POOL_EVENT_ID_REFRESH
to replace it.
Signed-off-by: NDaniel P. Berrange <berrange@redhat.com>

After commit e808d3f2 cbdata is always available here, so the
check is pointless

Every driver provides a refreshPool impl, and many other critical
places in the code unconditionally call it without checking if
it exists, so this check is pointless

Implement storage pool event callbacks for START, STOP, DEFINE, UNDEFINED
and REFRESHED in functions when a storage pool is created/started/stopped
etc. accordingly

Replace VIR_ERROR with virReportError and virReportSystemError

Commit 5e54361c added virStoragePoolObjClearVols before refreshPool
to prevent duplicate volume entries.

However it is not needed here because we're not refreshing the pool yet,
just checking for the existence of the refresh callback.

The actual refresh is done via virStorageVolFDStreamCloseCb
in virStorageVolPoolRefreshThread, which already calls
virStoragePoolObjClearVols.

Convert to virGetLastErrorMessage() in the rest of the code

Prior to calling the 'refreshPool' during CreatePool or UploadPool
operations, we need to clear the pool; otherwise, the pool will
have duplicated entries.

https://bugzilla.redhat.com/show_bug.cgi?id=1318993

Commit id 'dd519a29' caused a regression cloning a volume into a
logical pool by removing just the 'allocation' adjustment during
storageVolCreateXMLFrom. Combined with the change to not require the
new volume input XML to have a capacity listed (commit id 'e3f1d2a8')
left the possibility that a zero allocation value (e.g., not provided)
would create a thin/sparse logical volume. When a thin lv becomes fully
populated, then LVM sets the partition 'inactive' and the subsequent
fdatasync() fails.

Add a new 'has_allocation' flag to be set at XML parse time to indicate
that allocation was provided. This is done so that if it's not provided
the create-from code uses the capacity value since we document that if
omitted, the volume will be fully allocated at time of creation.

For a logical backend, that creation time is 'createVol', while for a
file backend, creation doesn't set the size, but the 'createRaw' called
during buildVolFrom will decide whether the file is sparse or not based
on the provided capacity and allocation value.

For volume clones that provide different allocation and capacity values
to allow for sparse files, there is no change.

Signed-off-by: NYuri Chornoivan <yurchor@ukr.net>

In case of ploop volume, target path of the volume is the path to the
directory that contains image file named root.hds and DiskDescriptor.xml.
While using uploadVol and downloadVol callbacks we need to open root.hds
itself.
Upload or download operations with ploop volume are only allowed when
images do not have snapshots. Otherwise operation fails.
Signed-off-by: NOlga Krishtal <okrishtal@virtuozzo.com>
Signed-off-by: NJán Tomko <jtomko@redhat.com>

Found by inspection - after calling virStoragePoolObjAssignDef the
pool is part of the driver->pools.objs list and the failure path
for the virStoragePoolObjSaveDef will use virStoragePoolObjRemove
to remove the pool from the objs list which will unlock and free
the pool pointer (as pools->objs[i] during the loop). Since the call
doesn't clear the pool address from the callee, we need to set it
to NULL; otherwise, the virStoragePoolObjUnlock in the cleanup: code
will fail miserably.

This reverts commit 611a278f.

According to the original commit message, this is dead code:

  It is highly unlikely that a backend will know how to create a
  volume from a different volume (buildVolFrom) and not know how to
  create an empty volume (createVol).

It is highly unlikely that a backend will know how to create a
volume from a different volume (buildVolFrom) and not know how to
create an empty volume (createVol). But:
1) we call the function without any prior check so if that's the
case we would SIGSEGV immediatelly
2) it's better to be safe than sorry.
Signed-off-by: NMichal Privoznik <mprivozn@redhat.com>

Firstly, we realloc internal list to hold new item (=volume that
will be potentially created) and then we check whether we
actually know how to create it. If we don't we consume more
memory than we really need for no good reason.
Signed-off-by: NMichal Privoznik <mprivozn@redhat.com>

The virStringListLength function does not ever modify the passed
string list. It merely counts the items in it. Make sure that we
reflect this bit in the function header.
Signed-off-by: NMichal Privoznik <mprivozn@redhat.com>

(crobinso: fix up spacing and squash in sheepdog bit suggested
 by Andrea)

Commit id 'aeb1078a' added a buildPool option and failure path which
calls virStoragePoolObjRemove, which unlocks the pool, clears the 'pool'
variable, and goto cleanup.  However, at cleanup virStoragePoolObjUnlock
is called without check if pool is non NULL.

Valgrind complained:

==28277== 38 bytes in 1 blocks are definitely lost in loss record 298 of 957
==28277==    at 0x4A06A2E: malloc (vg_replace_malloc.c:270)
==28277==    by 0x82D7F57: __vasprintf_chk (in /lib64/libc-2.12.so)
==28277==    by 0x52EF16A: virVasprintfInternal (stdio2.h:199)
==28277==    by 0x52EF25C: virAsprintfInternal (virstring.c:514)
==28277==    by 0x52B1FA9: virFileBuildPath (virfile.c:2831)
==28277==    by 0x19B1947C: storageDriverAutostart (storage_driver.c:191)
==28277==    by 0x19B196A7: storageStateAutoStart (storage_driver.c:307)
==28277==    by 0x538527E: virStateInitialize (libvirt.c:793)
==28277==    by 0x11D7CF: daemonRunStateInit (libvirtd.c:947)
==28277==    by 0x52F4694: virThreadHelper (virthread.c:206)
==28277==    by 0x6E08A50: start_thread (in /lib64/libpthread-2.12.so)
==28277==    by 0x82BE93C: clone (in /lib64/libc-2.12.so)
Signed-off-by: NMichael Chapman <mike@very.puzzling.org>

https://bugzilla.redhat.com/show_bug.cgi?id=830056

Add flags handling to the virStoragePoolCreate and virStoragePoolCreateXML
API's which will allow the caller to provide the capability for the storage
pool create API's to also perform a pool build during creation rather than
requiring the additional buildPool step. This will allow transient pools
to be defined, built, and started.

The new flags are:

    * VIR_STORAGE_POOL_CREATE_WITH_BUILD
      Perform buildPool without any flags passed.

    * VIR_STORAGE_POOL_CREATE_WITH_BUILD_OVERWRITE
      Perform buildPool using VIR_STORAGE_POOL_BUILD_OVERWRITE flag.

    * VIR_STORAGE_POOL_CREATE_WITH_BUILD_NO_OVERWRITE
      Perform buildPool using VIR_STORAGE_POOL_BUILD_NO_OVERWRITE flag.

It is up to the backend to handle the processing of build flags. The
overwrite and no-overwrite flags are mutually exclusive.

NB:
This patch is loosely based upon code originally authored by Osier
Yang that were not reviewed and pushed, see:

https://www.redhat.com/archives/libvir-list/2012-July/msg01328.html

https://bugzilla.redhat.com/show_bug.cgi?id=1270709

When a volume wipe is successful, perform a volume refresh afterwards to
update any volume data that may be used in future volume commands, such as
volume resize.  For a raw file volume, a wipe could truncate the file and
a followup volume resize the capacity may fail because the volume target
allocation isn't updated to reflect the wipe activity.

Add a new API to search the currently defined pool list for a pool with
a matching UUID and return the locked pool object pointer.

Since we treat it like a boolean, let's store it that way. At least one
path had already treated as true/false anyway.

https://bugzilla.redhat.com/show_bug.cgi?id=1233003

Commit id 'fdda3760' only managed a symptom where it was possible to
create a file in a pool without libvirt's knowledge, so it was reverted.

The real fix is to have all the createVol API's which actually create
a volume (disk, logical, zfs) and the buildVol API's which handle the
real creation of some volume file (fs, rbd, sheepdog) manage deleting
any volume which they create when there is some sort of error in
processing the volume.

This way the onus isn't left up to the storage_driver to determine whether
the buildVol failure was due to some failure as a result of adjustments
made to the volume after creation such as getting sizes, changing ownership,
changing volume protections, etc. or simple a failure in creation.

Without needing to consider that the volume has to be removed, the
buildVol failure path only needs to remove the volume from the pool.
This way if a creation failed due to duplicate name, libvirt wouldn't
remove a volume that it didn't create in the pool target.

This reverts commit fdda3760.

This commit only manages a symptom of finding a buildRet failure
where a volume was not listed in the pool, but someone created the
volume outside of libvirt in the pool being managed by libvirt.

Create a helper function to remove volume from the pool.

Commit id '1b5685da' refactored the code to move buildvoldef inside
the buildVol conditional; however, the VIR_FREE of the memory was
left only when 'buildret' failed, thus we're leaking memory.
Signed-off-by: NJohn Ferlan <jferlan@redhat.com>