The helper methods for actually accessing the storage objects don't
really belong to the main storage driver implementation file. Split them
out.

If a remote call fails during event registration (more than likely from
a network failure or remote libvirtd restart timed just right), then when
calling the virObjectEventStateDeregisterID we don't want to call the
registered @freecb function because that breaks our contract that we
would only call it after succesfully returning.  If the @freecb routine
were called, it could result in a double free from properly coded
applications that free their opaque data on failure to register, as seen
in the following details:

    Program terminated with signal 6, Aborted.
    #0  0x00007fc45cba15d7 in raise
    #1  0x00007fc45cba2cc8 in abort
    #2  0x00007fc45cbe12f7 in __libc_message
    #3  0x00007fc45cbe86d3 in _int_free
    #4  0x00007fc45d8d292c in PyDict_Fini
    #5  0x00007fc45d94f46a in Py_Finalize
    #6  0x00007fc45d960735 in Py_Main
    #7  0x00007fc45cb8daf5 in __libc_start_main
    #8  0x0000000000400721 in _start

The double dereference of 'pyobj_cbData' is triggered in the following way:

    (1) libvirt_virConnectDomainEventRegisterAny is invoked.
    (2) the event is successfully added to the event callback list
        (virDomainEventStateRegisterClient in
        remoteConnectDomainEventRegisterAny returns 1 which means ok).
    (3) when function remoteConnectDomainEventRegisterAny is hit,
        network connection disconnected coincidently (or libvirtd is
        restarted) in the context of function 'call' then the connection
        is lost and the function 'call' failed, the branch
        virObjectEventStateDeregisterID is therefore taken.
    (4) 'pyobj_conn' is dereferenced the 1st time in
        libvirt_virConnectDomainEventFreeFunc.
    (5) 'pyobj_cbData' (refered to pyobj_conn) is dereferenced the
         2nd time in libvirt_virConnectDomainEventRegisterAny.
    (6) the double free error is triggered.

Resolve this by adding a @doFreeCb boolean in order to avoid calling the
freeCb in virObjectEventStateDeregisterID for any remote call failure in
a remoteConnect*EventRegister* API. For remoteConnect*EventDeregister* calls,
the passed value would be true indicating they should run the freecb if it
exists; whereas, it's false for the remote call failure path.

Patch based on the investigation and initial patch posted by
fangying <fangying1@huawei.com>.

It is necessary for some parts of the code to refresh just data
based on the based on the backing store string. Add a convenience
function that will retrieve this data.

Use ATTRIBUTE_FALLTHROUGH, introduced by commit
5d84f596, instead of comments to
indicate that the fall through is an intentional behavior.
Signed-off-by: NMarc Hartmayer <mhartmay@linux.vnet.ibm.com>
Reviewed-by: NBoris Fiuczynski <fiuczy@linux.vnet.ibm.com>
Reviewed-by: NBjoern Walk <bwalk@linux.vnet.ibm.com>

These flags to APIs will tell if caller wants to use sparse
stream for storage transfer. At the same time, it's safe to
enable them in storage driver frontend and rely on our backends
checking the flags. This way we can enable specific flags only on
some specific backends, e.g. enable
VIR_STORAGE_VOL_DOWNLOAD_SPARSE_STREAM for filesystem backend but
not iSCSI backend.
Signed-off-by: NMichal Privoznik <mprivozn@redhat.com>

Basically, what is needed here is to introduce new message type
for the messages passed between the event loop callbacks and the
worker thread that does all the I/O. The idea is that instead of
a queue of read buffers we will have a queue where "hole of size
X" messages appear. That way the event loop callbacks can just
check the head of the queue and see if the worker thread is in
data or a hole section and how long the section is.
Signed-off-by: NMichal Privoznik <mprivozn@redhat.com>

There should be no need to make dir based pools world/group readable.
So use 0711, not 0755, as the default perms for storage dirs.

Updates in v2:
 - adapt commit wording to mention dropping group readable as well
Signed-off-by: NChristian Ehrhardt <christian.ehrhardt@canonical.com>

https://bugzilla.redhat.com/show_bug.cgi?id=1439132

During 'matrix' testing of all possible combinations I found that if
device is formated with "gpt" first, then an attempt is made to format
using "mac", a startup will fail.

Deeper analysis by Peter Krempa indicates that the "mac" table fits
into the first block on the disk. Since the GPT disklabel is stored
at LBA address 1 it is not overwritten at all. Thus it's apparent that
the (blkid) detection tool then prefers GPT over a older disklabel.

The GPT disklabel has also a secondary copy at the last LBA of the disk.

So, follow the same logic as the logical pool in clearing a 1MB swath
at the beginning and end of the device to avoid potential issues with
larger sector sizes for the device.

Also fixed a minor formatting nit in virStorageBackendDeviceIsEmpty call.

Since a sector size may be larger than 512 bytes, let's just increase
the size to wipe to 1MB rather than 2KB

Rather than open code it, use the new function which uses the wipe algorithm
in order to zero the front and tail of the partition.

Create a wrapper/helper that can be used to call the storage backend
wipe helper - storageBackendVolWipeLocalFile for future use by logical
and disk backends to clear out the partition table rather than having
each open code the same algorithm.

Add bool 'zero_end' and logic that would allow a caller to wipe specific
portions of a target device either from the beginning (the default) or
from the end when zero_end is true.

This will allow for this code to wipe out partition table information
from a device.

Create a couple of helpers that will perform the same call sequence.
Signed-off-by: NJohn Ferlan <jferlan@redhat.com>

Alter virStoragePoolObjListExport in order to pass the drivers->pools
by reference
Signed-off-by: NJohn Ferlan <jferlan@redhat.com>

Mostly code motion to move storageConnectList[Defined]StoragePools
and similar test driver code into virstorageobj.c and rename to
virStoragePoolObjGetNames.

Also includes a couple of variable name adjustments to keep code consistent
with other drivers.
Signed-off-by: NJohn Ferlan <jferlan@redhat.com>

Unify the NumOf[Defined]StoragePools API into virstorageobj.c from
storage_driver and test_driver.  The only real difference between the
two is the test driver doesn't call using the aclfilter API.
Signed-off-by: NJohn Ferlan <jferlan@redhat.com>

Essentially code motion to move the storage/test driver ListAllVolumes
logic into virstorageobj.c
Signed-off-by: NJohn Ferlan <jferlan@redhat.com>

Mostly code motion to move storagePoolListVolumes code into virstorageobj.c
and rename to virStoragePoolObjVolumeGetNames.

Also includes a couple of variable name adjustments to keep code consistent
with other drivers.
Signed-off-by: NJohn Ferlan <jferlan@redhat.com>

Unify the NumOfVolumes API into virstorageobj.c from storage_driver and
test_driver.  The only real difference between the two is the test driver
doesn't call using the aclfilter API.
Signed-off-by: NJohn Ferlan <jferlan@redhat.com>

https://bugzilla.redhat.com/show_bug.cgi?id=1439132

Add "bsd" to the list of format types to not checked during blkid
processing even though it supposedly knows the format - for some
(now unknown) reason it's returning partition table not found. So
let's just let PARTED handle "bsd" too.
Signed-off-by: NJohn Ferlan <jferlan@redhat.com>

https://bugzilla.redhat.com/show_bug.cgi?id=1439132

Commit id 'a48c674f' added a check for format types "dvh" and "pc98"
to use the parted print processing instead of using blkid processing
in order to validate the label on the disk was what is expected for
disk pool startup. However, commit id 'a4cb4a74' really messed things
up by missing an else condition causing PARTEDFindLabel to always
return DIFFERENT.
Signed-off-by: NJohn Ferlan <jferlan@redhat.com>

The contents of volname would be leaked if the function were
to be passed an invalid pooltype by the caller.

Make sure the memory is released instead.

After restart of libvirtd the 'checkPool' method is supposed to validate
that the pool is online. Since libvirt then refreshes the pool contents
anyways just return whether the pool was supposed to be online so that
the code can be reached. This is necessary since if a pool does not
implement the method it's automatically considered as inactive.

Resolves: https://bugzilla.redhat.com/show_bug.cgi?id=1436065

For native gluster pools the <dir> field denotes a directory inside the
pool. For the actual pool name the <name> field has to be used.

Use the relative lookup specifier rather than the global one. Otherwise
only the first name would be looked up. Add a test case to cover the
scenario.

Resolves: https://bugzilla.redhat.com/show_bug.cgi?id=1436574

To allow testing of the algorithm, split out the extractor into a
separate helper.

The native gluster pool source list data differs from the data used for
attaching gluster volumes as netfs pools. Currently the only difference
was the format. Since native pools don't use it and later there will be
more differences add a more deterministic way to switch between the
types instead.

https://bugzilla.redhat.com/show_bug.cgi?id=1371892

The 'capacity' value (e.g. guest logical size) for a LUKS volume is
smaller than the 'physical' value of the file in the file system, so
we need to account for that.

When peeking at the encryption information about the volume add a fetch
of the payload_offset which is described as the offset to the start of
the volume data (in 512 byte sectors) in QEMU's QCryptoBlockLUKSHeader.

Then adjust the ->capacity appropriately when we determine that the
volume target encryption has a payload_offset value.

If a transient storage pool is deemed inactive after libvirtd restart it
would not be deleted from the list. Reuse virStoragePoolUpdateInactive
along with a refactor necessary to properly update the state.

Resolves: https://bugzilla.redhat.com/show_bug.cgi?id=1242801

After a pool is made inactive the definition objects need to be updated
(if a new definition is prepared) and transient pools need to be
completely removed. Split out the code doing these steps into a separate
function for later reuse.

When registering a storage poll backend, the code would use
virStorageTypeToString instead of virStoragePoolTypeToString. The
following message would be logged:

virDriverLoadModuleFunc:71 : Lookup function 'virStorageBackendSCSIRegister'
virStorageBackendRegister:174 : Registering storage backend '(null)'

off_t is signed and it's size is the same as long only on 64b archs.
Thus it cannot be formatted as %lu.
Signed-off-by: NJiri Denemark <jdenemar@redhat.com>

https://bugzilla.redhat.com/show_bug.cgi?id=1430679

As it turns out some file headers (e.g. ext4) may be larger/longer than
the 512 bytes of zeros being written prior to a pvcreate, so let's write
out 2048 bytes similar to how the pvcreate sources would peek at the first
4 sectors of the device.

Make sure there is at enough bytes on the device to clear before doing
doing the clear - just to be sure.

There is no reason for it not to be in the utils, all global symbols
under that file already have prefix vir* and there is no reason for it
to be part of DRIVER_SOURCES because that is just a leftover from
older days (pre-driver modules era, I believe).
Signed-off-by: NMartin Kletzander <mkletzan@redhat.com>

Use "virStoragePoolObj" as a prefix for any external API in virstorageobj.

Also a couple of functions were local to virstorageobj.c, so remove their
external defs iin virstorageobj.h.

NB: The virStorageVolDef* API's won't change.
Signed-off-by: NJohn Ferlan <jferlan@redhat.com>

Move all the StoragePoolObj related API's into their own module
virstorageobj from the storage_conf

Purely code motion at this point, plus adjustments to cleanly build
Signed-off-by: NJohn Ferlan <jferlan@redhat.com>

Rather than returning true/false and having the caller check if the
vHBA was actually created, let's do that check within the CreateVport
function. That way the caller can faithfully assume success based
on a name start the thread looking for the LUNs. Prior to this change
it's possible that the vHBA wasn't really created (e.g if the call to
virVHBAGetHostByWWN returned NULL), we'd claim success, but in reality
there'd be no vHBA for the pool. This also fixes a second yet seen
issue that if the nodedev was present, but the parent by name wasn't
provided (perhaps parent by wwnn/wwpn or by fabric_name), then a failure
would be returned. For this path it shouldn't be an error - we should
just be happy that something else is managing the device and we don't
have to create/delete it.

The end result is that the createVport code can now just start the
refresh thread once it gets a non NULL name back.
Signed-off-by: NJohn Ferlan <jferlan@redhat.com>

Move the bulk of createVport and rename to virNodeDeviceCreateVport.

Remove the deleteVport entirely and replace with virNodeDeviceDeleteVport
Signed-off-by: NJohn Ferlan <jferlan@redhat.com>

The function is actually in virutil.c, but prototyped in virfile.h.
This patch fixes that by renaming the function to virWaitForDevices,
adding the prototype in virutil.h and libvirt_private.syms, and then
changing the callers to use the new name.
Signed-off-by: NJohn Ferlan <jferlan@redhat.com>

Move the virStoragePoolSourceAdapter from storage_conf.h and rename
to virStorageAdapter.

Continue with code realignment for brevity and flow.
Signed-off-by: NJohn Ferlan <jferlan@redhat.com>