So far, my work has been merely preserving the status quo of
backing file analysis.  But this patch starts to tread in the
territory of making the backing chain code more powerful - we
will eventually support network storage containing non-raw
formats.  Here, we expose metadata information about a network
backing store, even if that information is still hardcoded to
a raw format for now.

* src/util/virstoragefile.c (virStorageFileGetMetadataRecurse):
Also populate struct for non-file backing.
(virStorageFileGetMetadata, virStorageFileGetMetadatainternal):
Recognize non-file top image.
(virFindBackingFile): Add comment.
(virStorageFileChainGetBroken): Adjust comment, ensure output
is set.
* tests/virstoragetest.c (mymain): Update test to reflect it.
Signed-off-by: NEric Blake <eblake@redhat.com>

Most of the APIs in CPU driver do not expect to get NULL for input
parameters. Let's mark them with ATTRIBUTE_NONNULL and also check for
some members of virCPUDef when the APIs expect them have some specific
values.
Signed-off-by: NJiri Denemark <jdenemar@redhat.com>

Signed-off-by: NJiri Denemark <jdenemar@redhat.com>

My commit 897808e7 added a parameter to virCgroupGetPercpuStats,
but didn't change the stub for systems where cgroups are not supported.

The iterator is checked for being less than or equal to need_cpus.
The 'n' variable is incremented need_cpus + 1 times.

Simplify the computation of need_cpus and make its value one larger,
to let it be used instead of 'n' and compared without the equal sign
in loop conditions.

Just index the sum_cpu_time array instead of using a helper variable.

Start the loop at start_cpu instead of continuing for all lower values.

The cpus are indexed from 0, so a startcpu value equal
to the number of CPUs is invalid.

https://bugzilla.redhat.com/show_bug.cgi?id=1070680

total_cpus is the total number of CPUs on the host
need_cpus is the number of CPUs we need to look at

(need_cpus can be larger than ncpus, because we need to look
 at CPUs before the startcpu too, even if we aren't reporting
 their stats)

Currently, virCgroupGetPercpuStats is only used by the LXC driver,
filling out the CPUTIME stats. qemuDomainGetPercpuStats does this
and also filles out VCPUTIME stats.

Extend virCgroupGetPercpuStats to also report VCPUTIME stats if
nvcpupids is non-zero. In the LXC driver, we don't have cpupids.
In the QEMU driver, there is at least one cpupid for a running domain,
so the behavior shouldn't change for QEMU either.

Also rename getSumVcpuPercpuStats to virCgroupGetPercpuVcpuSum.

We need to return the number of successfully populated stats,
not the nparams supplied by the user.

All we need is the virCgroupPtr and number of vcpupids.
This will allow the function to be moved to util/vircgroup.c.

The current use of virStorageFileMetadata is awkward; to learn
some of the information about a child node, you have to read
fields in the parent node.  This does not lend itself well to
modifying backing chains (whether inserting a new node in the
chain, or consolidating existing nodes); better would be to
learn about a child node directly in that node.  This patch
sets up some new fields which contain redundant information,
although not necessarily in the final desired state for the
new fields (see the next patch for actual tests of what is there
now).  Then later patches will do any refactoring necessary to
get the fields to their desired states, and update clients to
get the information from the new fields, so we can finally
delete the fields that are tracking information about the wrong
node.

More concretely, compare these three example backing chains:

good <- one
missing <- two
gluster://server/vol/img <- three

Pre-patch, querying the chains gives:
{ .backingStore = "/path/to/good",
  .backingStoreRaw = "good",
  .backingStoreIsFile = true,
  .backingStoreFormat = VIR_STORAGE_FILE_RAW,
  .backingMeta = {
    .backingStore = NULL,
    .backingStoreRaw = NULL,
    .backingStoreIsFile = false,
    .backingMeta = NULL,
  }
}
{ .backingStore = NULL,
  .backingStoreRaw = "missing",
  .backingStoreIsFile = false,
  .backingStoreFormat = VIR_STORAGE_FILE_NONE,
  .backingMeta = NULL,
}
{ .backingStore = "gluster://server/vol/img",
  .backingStoreRaw = NULL,
  .backingStoreIsFile = false,
  .backingStoreFormat = VIR_STORAGE_FILE_RAW,
  .backingMeta = NULL,
}

Deciding whether to ignore a missing backing file (as in virsh
vol-dumpxml) or report an error (as in security manager sVirt
labeling) requires reading multiple fields.  Plus, the format
is hard-coded to treat all network protocols as end-of-the-chain,
as if they were raw.  By the end of this patch series, the goal
is to instead represent these three situations as:

{ .path = "one",
  .canonPath = "/path/to/one",
  .type = VIR_STORAGE_TYPE_FILE,
  .format = VIR_STORAGE_FILE_QCOW2,
  .backingStoreRaw = "good",
  .backingMeta = {
    .path = "good",
    .canonPath = "/path/to/good",
    .type = VIR_STORAGE_TYPE_FILE,
    .format = VIR_STORAGE_FILE_RAW,
    .backingStoreRaw = NULL,
    .backingMeta = NULL,
  }
}
{ .path = "two",
  .canonPath = "/path/to/two",
  .type = VIR_STORAGE_TYPE_FILE,
  .format = VIR_STORAGE_FILE_QCOW2,
  .backingStoreRaw = "missing",
  .backingMeta = NULL,
}
{ .path = "three",
  .canonPath = "/path/to/three",
  .type = VIR_STORAGE_TYPE_FILE,
  .format = VIR_STORAGE_FILE_QCOW2,
  .backingStoreRaw = "gluster://server/vol/img",
  .backingMeta = {
    .path = "gluster://server/vol/img",
    .canonPath = "gluster://server/vol/img",
    .type = VIR_STORAGE_TYPE_NETWORK,
    .format = VIR_STORAGE_FILE_RAW,
    .backingStoreRaw = NULL,
    .backingMeta = NULL,
  }
}

or, for the second file, maybe also allowing:
{ .path = "two",
  .canonPath = "/path/to/two",
  .type = VIR_STORAGE_TYPE_FILE,
  .format = VIR_STORAGE_FILE_QCOW2,
  .backingStoreRaw = "missing",
  .backingMeta = {
    .path = "missing",
    .canonPath = NULL,
    .type = VIR_STORAGE_TYPE_NONE,
    .format = VIR_STORAGE_FILE_NONE,
    .backingStoreRaw = NULL,
    .backingMeta = NULL,
  }
}

* src/util/virstoragefile.h (_virStorageFileMetadata): Add
path, canonPath, relDir, type, and format fields.  Reorder
existing fields, and add lots of comments.
* src/util/virstoragefile.c (virStorageFileFreeMetadata): Clean
new fields.
(virStorageFileGetMetadataInternal)
(virStorageFileGetMetadataFromFDInternal): Start populating new
fields.
Signed-off-by: NEric Blake <eblake@redhat.com>

Right now, we are allocating virStorageFileMetadata near the bottom
of the callchain, only after we have identified that we are visiting
a file (and not a network resource).  I'm hoping to eventually
support parsing the backing chain from XML, where the backing chain
crawl then validates what was parsed rather than allocating a fresh
structure.  Likewise, I'm working towards a setup where we have a
backing element even for networks.  Both of these use cases are
easier to code if the allocation is hoisted earlier.

* src/util/virstoragefile.c (virStorageFileGetMetadataInternal)
(virStorageFileGetMetadataFromFDInternal): Change signature.
(virStorageFileGetMetadataFromBuf)
(virStorageFileGetMetadataRecurse, virStorageFileGetMetadata):
Update callers.
Signed-off-by: NEric Blake <eblake@redhat.com>

The previous patch started a separation of error messages
reported against the user-specified name, vs. tracking the
canonical path that was actually opened.  This patch extends
that notion, by hoisting directory detection up front, passing
the canonical path through the entire call chain, and
simplifying lower-level functions that can now assume that
a canonical path and directory have been supplied.

* src/util/virstoragefile.c
(virStorageFileGetMetadataFromFDInternal)
(virStorageFileGetMetadataInternal): Add parameter, require
directory.
(virFindBackingFile): Require directory.
(virStorageFileGetMetadataFromFD): Pass canonical path.
(virStorageFileGetMetadataFromBuf): Likewise.
(virStorageFileGetMetadata): Determine initial directory.
Signed-off-by: NEric Blake <eblake@redhat.com>

The function isn't used in any other file. Convert it to static.

Refactor the function to avoid multiple wrappers splitting identical
fields from the now common metadata struct.

The refactor is done by folding in the wrapper used for disk sources
which allows us to lookup secrets via the secret driver. This may allow
using stored secrets for snapshot disk images too in the future.

Now that we store all metadata about a storage image in a
virStorageSource struct let's use it also to store information needed by
the storage driver to access and do operations on the files.

Now that storage source metadata is stored in a single struct we don't
need two initialization functions for different structs.

Now that the storage source definition is uniform convert the helpers to
retrieve the actual storage type to a single one.

https://bugzilla.redhat.com/show_bug.cgi?id=1024159

If adding a volume to a storage pool fails during the CreateXML or
CreateXMLFrom API's, we don't want to adjust the available and
allocation values for the storage pool during storageVolDelete
since we haven't adjusted the values for the create.

Refactor storageVolDelete() a bit to create a storageVolDeleteInternal()
which will handle the primary deletion activities.  Add a parameter
updateMeta which will signify whether to update the values or not.

Adjust the calls from CreateXML and CreateXMLFrom to directly call the
DeleteInternal with the pool lock held.  This does bypass the call
to virStorageVolDeleteEnsureACL().

While trying to refactor the backing file chain, I noticed that
if you have a self-referential qcow2 file via a relative name:

qemu-img create -f qcow2 loop 10M
qemu-img rebase -u -f qcow2 -F qcow2 -b loop loop

then libvirt was creating a chain 2 deep before realizing it
had hit a loop; furthermore, virStorageFileChainCheckBroken
was not identifying the chain as broken.  With this patch,
the loop is detected when the chain is only 1 deep; still
enough for storage volume XML to display the file, but now
with a proper error report about where the loop was found.

This patch adds a parameter to virStorageFileGetMetadataRecurse,
so that errors at the top of the chain remain unchanged; messages
issued for backing files now use the name provided by the user
instead of the canonical name (for VDSM, which uses relative
symlinks to device mapper block devices, this is actually more
useful).

* src/util/virstoragefile.c (virStorageFileGetMetadataRecurse):
Add parameter, require canonical path up front.  Mark chain
broken on OOM or loop detection.
(virStorageFileGetMetadata): Pass in canonical name.
Signed-off-by: NEric Blake <eblake@redhat.com>

Since commit 7457cbe8 the vmware driver version isn't set anymore.
Signed-off-by: NEric Blake <eblake@redhat.com>

Implement domainSetMetadata and domainGetMetadata driver calls.

Fix incorrect ATTRIBUTE_NONNULL usage introduced in 17b17565
which caused build failure:

bhyve/bhyve_driver.c:127:48: error: expected ')'
bhyveDriverGetCapabilities(bhyveConnPtr driver ATTRIBUTE_NONNULL)
                                               ^
bhyve/bhyve_driver.c:127:27: note: to match this '('
bhyveDriverGetCapabilities(bhyveConnPtr driver ATTRIBUTE_NONNULL)

Pushed under the build breaker rule.

Commit b9dd878f (util: make it easier to grab only regular command exit)
changed the call semantics of virCommandRun() and therefore of virRun()
too. But lxcCheckNetNsSupport() was not updated.
As consequence of this lxcCheckNetNsSupport always failed and broke LXC.
Signed-off-by: NRichard Weinberger <richard@nod.at>

My commit c9123fbe introduced this copy-and-paste error.

Now that we ditched our custom pthread impl for Win32, we can
use PTHREAD_MUTEX_INITIALIZER for static mutexes. This avoids
the need to use a virOnce one-time global initializer in a
number of places.
Signed-off-by: NDaniel P. Berrange <berrange@redhat.com>

Since it is an abbreviation, PCI should always be fully
capitalized or full lower case, never Pci.
Signed-off-by: NDaniel P. Berrange <berrange@redhat.com>

Since it is an abbreviation, USB should always be fully
capitalized or full lower case, never Usb.
Signed-off-by: NDaniel P. Berrange <berrange@redhat.com>

Since it is an abbreviation, SCSI should always be fully
capitalized or full lower case, never Scsi.
Signed-off-by: NDaniel P. Berrange <berrange@redhat.com>

Functions virNetDevRestoreMacAddress() and virNetDevRestoreMacAddress()
allocate memory for variable @path using virAsprintf(), but they
haven't freed that memory before returning out.
Signed-off-by: NZhang bo <oscar.zhangbo@huawei.com>
Signed-off-by: NMichal Privoznik <mprivozn@redhat.com>

At the beginning of the function we gain a reference to the driver
capabilities. Then, we call format function (*) which if failed, unref
over caps is called. Then, at the end another unref occurs.

* - Moreover, the format was not called over gained caps, but over
privconn->caps directly which is not allowed anymore.
Signed-off-by: NMichal Privoznik <mprivozn@redhat.com>

The driver passed as the only argument to the function should never be
NULL so there's no need to check it. After removing it, the whole
function collapses to a single line doing ref over driver
capabilities.
Signed-off-by: NMichal Privoznik <mprivozn@redhat.com>

Since b15a2bbd we have the new bhyve_capabilities.[ch] files.
However, the copyright is held by both Roman and Semihalf.
Signed-off-by: NMichal Privoznik <mprivozn@redhat.com>

I noticed that the apparmor code could request metadata even
for a cdrom with no media, which would cause a memory leak of
the hash table used to look for loops in the backing chain.
But even before that, we blindly dereferenced the path for
printing a debug statement, so it is just better to enforce
that this is only used on non-NULL names.

* src/util/virstoragefile.c (virStorageFileGetMetadata): Assume
non-NULL path.
* src/util/virstoragefile.h: Annotate this.
* src/security/virt-aa-helper.c (get_files): Fix caller.
Signed-off-by: NEric Blake <eblake@redhat.com>

Implement support for connectCompareCPU.

- Move all capabilities functions to separate file
- Add initCPU

Implement bhyveConnectBaselineCPU to support OpenStack/Nova

Other drivers in libvirt (e.g. network, qemu) will automatically
return the "inactive" (persistent configuration) XML of an object when
that object is inactive. The netcf backend of the interface driver
would always try to return the live status XML of the interface, even
when it was down. Although netcf does return valid XML in that case,
for bond interfaces it is missing almost all of its content, including
the <bond> subelement itself, leading to this error message from
"virsh iface-dumpxml" of a bond interface that is inactive:

  error: XML error: bond interface misses the bond element

(this is because libvirt's validation of the XML returned by netcf
always requires a <bond> element be present).

This patch modifies the interface driver netcf backend to check if the
interface is inactive, and in that case always return the inactive XML
(which will always have a <bond> element, thus eliminating the error
message, as well as making operation more in line with other drivers.

This fixes the following bug:

  https://bugzilla.redhat.com/show_bug.cgi?id=878394

I almost wrote a hash value free function that just called
VIR_FREE, then realized I couldn't be the first person to
do that.  Sure enough, it was worth factoring into a common
helper routine.

* src/util/virhash.h (virHashValueFree): New function.
* src/util/virhash.c (virHashValueFree): Implement it.
* src/util/virobject.h (virObjectFreeHashData): New function.
* src/libvirt_private.syms (virhash.h, virobject.h): Export them.
* src/nwfilter/nwfilter_learnipaddr.c (virNWFilterLearnInit): Use
common function.
* src/qemu/qemu_capabilities.c (virQEMUCapsCacheNew): Likewise.
* src/qemu/qemu_command.c (qemuDomainCCWAddressSetCreate):
Likewise.
* src/qemu/qemu_monitor.c (qemuMonitorGetBlockInfo): Likewise.
* src/qemu/qemu_process.c (qemuProcessWaitForMonitor): Likewise.
* src/util/virclosecallbacks.c (virCloseCallbacksNew): Likewise.
* src/util/virkeyfile.c (virKeyFileParseGroup): Likewise.
* tests/qemumonitorjsontest.c
(testQemuMonitorJSONqemuMonitorJSONGetBlockInfo): Likewise.
Signed-off-by: NEric Blake <eblake@redhat.com>