This patch implements the VIR_DOMAIN_STATS_INTERFACE group of
statistics.
Signed-off-by: NFrancesco Romani <fromani@redhat.com>
Signed-off-by: NPeter Krempa <pkrempa@redhat.com>

This patch implements the VIR_DOMAIN_STATS_VCPU group of statistics. To
do so, this patch also extracts a helper to gather the vCPU information.
Signed-off-by: NFrancesco Romani <fromani@redhat.com>
Signed-off-by: NPeter Krempa <pkrempa@redhat.com>

This patch implements the VIR_DOMAIN_STATS_BALLOON group of statistics.
Signed-off-by: NFrancesco Romani <fromani@redhat.com>

This patch implements the VIR_DOMAIN_STATS_CPU_TOTAL group of
statistics.
Signed-off-by: NFrancesco Romani <fromani@redhat.com>
Signed-off-by: NPeter Krempa <pkrempa@redhat.com>

We stupidly modeled block job bandwidth after migration
bandwidth, which in turn was an 'unsigned long' and therefore
subject to 32-bit vs. 64-bit interpretations.  To work around
the fact that 10-gigabit interfaces are possible but don't fit
within 32 bits, the original interface took the number scaled
as MiB/sec.  But this scaling is rather coarse, and it might
be nice to tune bandwidth finer than in megabyte chunks.

Several of the block job calls that can set speed are fed
through a common interface, so it was easier to adjust them all
at once.  Note that there is intentionally no flag for the new
virDomainBlockCopy; there, since the API already uses a 64-bit
type always, instead of a possible 32-bit type, and is brand
new, it was easier to just avoid scaling issues.  As with the
previous patch that adjusted the query side (commit db33cc24),
omitting the new flag preserves old behavior, and the
documentation now mentions limits of what happens when a 32-bit
machine is on either client or server side.

* include/libvirt/libvirt.h.in (virDomainBlockJobSetSpeedFlags)
(virDomainBlockPullFlags)
(VIR_DOMAIN_BLOCK_REBASE_BANDWIDTH_BYTES)
(VIR_DOMAIN_BLOCK_COMMIT_BANDWIDTH_BYTES): New enums.
* src/libvirt.c (virDomainBlockJobSetSpeed, virDomainBlockPull)
(virDomainBlockRebase, virDomainBlockCommit): Document them.
* src/qemu/qemu_driver.c (qemuDomainBlockJobSetSpeed)
(qemuDomainBlockPull, qemuDomainBlockRebase)
(qemuDomainBlockCommit, qemuDomainBlockJobImpl): Support new flag.
Signed-off-by: NEric Blake <eblake@redhat.com>

Total time of a migration and total downtime transfered from a source to
a destination host do not count with the transfer time to the
destination host and with the time elapsed before guest CPUs are
resumed. Thus, source libvirtd remembers when migration started and when
guest CPUs were paused. Both timestamps are transferred to destination
libvirtd which uses them to compute total migration time and total
downtime. Obviously, this requires the time to be synchronized between
the two hosts. The reported times are useless otherwise but they would
be equally useless if we didn't do this recomputation so don't lose
anything by doing it.
Signed-off-by: NJiri Denemark <jdenemar@redhat.com>

virDomainGetJobStats gains new VIR_DOMAIN_JOB_STATS_COMPLETED flag that
can be used to fetch statistics of a completed job rather than a
currently running job.
Signed-off-by: NJiri Denemark <jdenemar@redhat.com>

To date, anyone performing a block copy and pivot ends up with
the destination being treated as <disk type='file'>.  While this
works for data access for a block device, it has at least one
noticeable shortcoming: virDomainGetBlockInfo() reports allocation
differently for block devices visited as files (the size of the
device) than for block devices visited as <disk type='block'>
(the maximum sector used, as reported by qemu); and this difference
is significant when trying to manage qcow2 format on block devices
that can be grown as needed.

Of course, the more powerful virDomainBlockCopy() API can already
express the ability to set the <disk> type.  But a new API can't
be backported, while a new flag to an existing API can; and it is
also rather inconvenient to have to resort to the full power of
generating XML when just adding a flag to the older call will do
the trick.  So this patch enhances blockcopy to let the user flag
when the resulting XML after the copy must list the device as
type='block'.

* include/libvirt/libvirt.h.in (VIR_DOMAIN_BLOCK_REBASE_COPY_DEV):
New flag.
* src/libvirt.c (virDomainBlockRebase): Document it.
* tools/virsh-domain.c (opts_block_copy, blockJobImpl): Add
--blockdev option.
* tools/virsh.pod (blockcopy): Document it.
* src/qemu/qemu_driver.c (qemuDomainBlockRebase): Allow new flag.
(qemuDomainBlockCopy): Remember the flag, and make sure it is only
used on actual block devices.
* tests/qemuxml2argvdata/qemuxml2argv-disk-mirror.xml: Test it.
Signed-off-by: NEric Blake <eblake@redhat.com>

While reviewing the new virDomainBlockCopy API, Peter Krempa
pointed out that our existing design of using MiB/s for block
job bandwidth is rather coarse, especially since qemu tracks
it in bytes/s; so virDomainBlockCopy only accepts bytes/s.
But once the new API is implemented for qemu, we will be in
the situation where it is possible to set a value that cannot
be accurately reflected back to the user, because the existing
virDomainGetBlockJobInfo defaults to the coarser units.

Fortunately, we have an escape hatch; and one that has already
served us well in the past: we can use the flags argument to
specify which scale to use (see virDomainBlockResize for prior
art).  This patch fixes the query side of the API; made easier
by previous patches that split the query side out from the
modification code.  Later patches will address the virsh
interface, as well retrofitting all other blockjob APIs to
also accept a flag for toggling bandwidth units.

* include/libvirt/libvirt.h.in (_virDomainBlockJobInfo)
(VIR_DOMAIN_BLOCK_COPY_BANDWIDTH): Document sizing issues.
(virDomainBlockJobInfoFlags): New enum.
* src/libvirt.c (virDomainGetBlockJobInfo): Document new flag.
* src/qemu/qemu_monitor.h (qemuMonitorBlockJobInfo): Add parameter.
* src/qemu/qemu_monitor.c (qemuMonitorBlockJobInfo): Likewise.
* src/qemu/qemu_monitor_json.h (qemuMonitorJSONBlockJobInfo):
Likewise.
* src/qemu/qemu_monitor_json.c (qemuMonitorJSONBlockJobInfo)
(qemuMonitorJSONGetBlockJobInfoOne): Likewise. Don't scale here.
* src/qemu/qemu_migration.c (qemuMigrationDriveMirror): Update
callers.
* src/qemu/qemu_driver.c (qemuDomainBlockPivot)
(qemuDomainBlockJobImpl): Likewise.
(qemuDomainGetBlockJobInfo): Likewise, and support new flag.
Signed-off-by: NEric Blake <eblake@redhat.com>

Add domain list filtering functions and a flag to enforce checking
whether the remote daemon supports the requested stats groups.

Let's fix this before we bake in a painful API.  Since we know
that we have exactly one non-negative fd on success, we might
as well return the fd directly instead of forcing the user to
pass in a pointer.  Furthermore, I found some memory and fd
leaks while reviewing the code - the idea is that on success,
libvirtd will have handed two fds in two different directions:
one to qemu, and one to the RPC client.

* include/libvirt/libvirt.h.in (virDomainOpenGraphicsFD): Drop
unneeded parameter.
* src/driver.h (virDrvDomainOpenGraphicsFD): Likewise.
* src/libvirt.c (virDomainOpenGraphicsFD): Adjust interface to
return fd directly.
* daemon/remote.c (remoteDispatchDomainOpenGraphicsFd): Adjust
semantics.
* src/qemu/qemu_driver.c (qemuDomainOpenGraphicsFD): Likewise,
and plug fd leak.
* src/remote/remote_driver.c (remoteDomainOpenGraphicsFD):
Likewise, and plug memory and fd leak.
Signed-off-by: NEric Blake <eblake@redhat.com>

This commit (finally) adds the virDomainBlockCopy API, with the
intent that it will provide more power to the existing 'virsh
blockcopy' command.

'virsh blockcopy' was first added in Apr 2012 (v0.9.12), which
corresponds to the upstream qemu 1.2 timeframe.  It was done as
a hack on top of the existing virDomainBlockRebase() API call,
for two reasons: 1) it was targetting a feature that landed first
in downstream RHEL qemu, but had not stabilized in upstream qemu
at the time (and indeed, 'drive-mirror' only landed upstream in
qemu 1.3 with slight differences to the first RHEL attempt,
and later gained further parameters like granularity and buf-size
that are also worth exposing), and 2) extending an existing API
allowed it to be backported without worrying about bumping .so
versions.  A virDomainBlockCopy() API was proposed at that time
[1], but we decided not to accept it into libvirt until after
upstream qemu stabilized, and it ended up getting scrapped.
Whether or not RHEL should have attempted adding a new feature
without getting it upstream first is a debate that can be held
another day; but enough time has now elapsed that we are ready to
do the interface cleanly.

[1] https://www.redhat.com/archives/libvir-list/2012-April/msg00768.html

Delaying the creation of a clean API until now has also had a
benefit: we've only recently learned of a few shortcomings in the
original design: 1) it is unable to target a network destination
(such as a gluster volume) because it hard-coded the assumption
that the destination is a local file name.  Because of all the
refactoring we've done to add virStorageSourcePtr, we are in a
better position to declare an API that parses XML describing a
host storage source as the copy destination, which was not
possible had we implemented virDomainBlockCopy as it had been
originally envisioned (although a network target will have to wait
until a later libvirt release compared to the API addition to
actually be implemented).  2) the design of using MiB/sec as the
bandwidth throttle is rather coarse; qemu is actually tuned to
bytes/second, and libvirt is preventing access to that level of
detail.  A later patch will add flags to existing block job API
that can request bytes/second instead of back-compat MiB/s, but as
this is a new API, we can get it right to begin with.

At least I had the foresight to create 'virsh blockcopy' as a
separate command at the UI level (commit 1f06c007) rather than
leaking the underlying API overload of virDomainBlockRebase onto
shell users.

A further note on the bandwidth option: virTypedParameters
intentionally lacks unsigned long (since variable-width
interaction between mixed 32- vs. 64-bit client/server setups is
nasty), but we have to deal with the fact that we are interacting
with existing older code that mistakenly chose unsigned long
bandwidth at a point before we decided to prohibit it in all new
API.  The typed parameter is therefore unsigned long long, but
the implementation (in a later patch) will have to do overflow
detection on 32-bit platforms, as well as capping the value to
match the LLONG_MAX>>20 cap of the existing MiB/s interfaces.

* include/libvirt/libvirt.h.in (virDomainBlockCopy): New API.
(virDomainBlockJobType, virConnectDomainEventBlockJobStatus):
Update related documentation.
* src/libvirt.c (virDomainBlockCopy): Implement it.
* src/libvirt_public.syms (LIBVIRT_1.2.8): Export it.
* src/driver.h (_virDriver): New driver callback.
Signed-off-by: NEric Blake <eblake@redhat.com>

The motivation for this API is that management layers that use libvirt
usually poll for statistics using various split up APIs we currently
provide. To get all the necessary stuff, the app needs to issue a lot of
calls and aggregate the results.

The APIs I'm introducing here:
1) Returns data in a format that we can expand in the future and is
(pseudo) hierarchical. The data is returned as typed parameters where
the fields are constructed as dot-separated strings containing names and
other stuff in a list of typed params.

2) Stats for multiple (all) domains can be queried at once and are
returned in one call. This will decrease the overhead necessary to issue
multiple calls per domain multiplied by the count of domains.

3) Selectable (bit mask) fields in the returned format. This will allow
to retrieve only specific stats according to the app's need.

The stats groups will be enabled using a bit field @stats passed as the
function argument. A few sample stats groups that this API will support:

VIR_DOMAIN_STATS_STATE
VIR_DOMAIN_STATS_CPU
VIR_DOMAIN_STATS_BLOCK
VIR_DOMAIN_STATS_INTERFACE

(Note that this is only an example, the initial implementation supports
 only VIR_DOMAIN_STATS_STATE while others will be added later.)

the returned typed params will use the following scheme

state.state = VIR_DOMAIN_RUNNING
state.reason = VIR_DOMAIN_RUNNING_BOOTED (the actual values according to
                                          the enum)
cpu.count = 8
cpu.0.state = running
cpu.0.time = 1234

Define the public API implementation and declare internal
driver prototype.

And add a syntax-check for '){$'.  It's not perfect, but better than
nothing.
Signed-off-by: NMartin Kletzander <mkletzan@redhat.com>

https://bugzilla.redhat.com/show_bug.cgi?id=1072653

Upon successful upload of a volume, the target volume and storage pool
were not updated to reflect any changes as a result of the upload. Make
use of the existing stream close callback mechanism to force a backend
pool refresh to occur in a separate thread once the stream closes. The
separate thread should avoid potential deadlocks if the refresh needed
to wait on some event from the event loop which is used to perform
the stream callback.

Signed-off-by: NLi Yang <liyang.fnst@cn.fujitsu.com>

Snapshots and block-copy have a flag that forces qemu to re-use existing
file. Our docs weren't exactly clear on what the existing file should
contain for this to actually work.

Re-word the docs a bit to state that the file needs to be pre-created in
the desired format and the backing chain metadata needs to be set prior
to handing it over to qemu.

Resolves: https://bugzilla.redhat.com/show_bug.cgi?id=1084360

Another patch for
https://bugzilla.redhat.com/show_bug.cgi?id=994731

Introduce flag for the block rebase API to allow the rebase operation to
leave the chain relatively addressed. Also adds a virsh switch to enable
this behavior.

Introduce flag for the block commit API to allow the commit operation to
leave the chain relatively addressed. Also adds a virsh switch to enable
this behavior.

The API should expose the information contained in virDomainCapsPtr.
Signed-off-by: NMichal Privoznik <mprivozn@redhat.com>

Instead of maintaining two very similar APIs, add the "@mac" parameter
to virNetworkGetDHCPLeases and kill virNetworkGetDHCPLeasesForMAC. Both
of those functions would return data the same way, so making @mac an
optional filter simplifies a lot of stuff.

The new VIR_CONNECT_COMPARE_CPU_FAIL_INCOMPATIBLE flag for
virConnectCompareCPU can be used to get an error
(VIR_ERR_CPU_INCOMPATIBLE) describing the incompatibility instead of the
usual VIR_CPU_COMPARE_INCOMPATIBLE return code.
Signed-off-by: NJiri Denemark <jdenemar@redhat.com>

Variables/fields named 'interface' clash with system
header symbols on some platforms.
Signed-off-by: NDaniel P. Berrange <berrange@redhat.com>

Introduce 3 new APIs, virNetworkGetDHCPLeases, virNetworkGetDHCPLeasesForMAC
and virNetworkDHCPLeaseFree.

* virNetworkGetDHCPLeases: returns the dhcp leases information for a given
     virtual network.

  For DHCPv4, the information returned:
  - Network Interface Name
  - Expiry Time
  - MAC address
  - IAID (NULL)
  - IPv4 address (with type and prefix)
  - Hostname (can be NULL)
  - Client ID (can be NULL)

  For DHCPv6, the information returned:
  - Network Interface Name
  - Expiry Time
  - MAC address
  - IAID (can be NULL, only in rare cases)
  - IPv6 address (with type and prefix)
  - Hostname (can be NULL)
  - Client DUID

  Note: @mac, @iaid, @ipaddr, @clientid are in ASCII form, not raw bytes.
  Note: @expirytime can 0, in case the lease is for infinite time.

* virNetworkGetDHCPLeasesForMAC: returns the dhcp leases information for a
     given virtual network and specified MAC Address.

* virNetworkDHCPLeaseFree: allows the upper layer application to free the
     network interface object conveniently.

There is no support for flags, so user is expected to pass 0 for
both the APIs.

include/libvirt/libvirt.h.in:
  * Define virNetworkGetDHCPLeases
  * Define virNetworkGetDHCPLeasesForMAC
  * Define virNetworkDHCPLeaseFree

src/driver.h:
  * Define networkGetDHCPLeases
  * Define networkGetDHCPLeasesForMAC

src/libvirt.c:
  * Implement virNetworkGetDHCPLeases
  * Implement virNetworkGetDHCPLeasesForMAC
  * Implement virNetworkDHCPLeaseFree

src/libvirt_public.syms:
  * Export the new symbols

The aim of the API is to get information on number of free pages
on the system. The API behaves similar to the
virNodeGetCellsFreeMemory(). User passes starting NUMA cell, the
count of nodes that he's interested in, pages sizes (yes,
multiple sizes can be queried at once) and the counts are
returned in an array.
Signed-off-by: NMichal Privoznik <mprivozn@redhat.com>

Now that qemu 2.0 allows commit of the active layer, people are
attempting to use virsh blockcommit and getting into a stuck
state, because libvirt is unprepared to handle the two-phase
commit required by qemu.

Stepping back a bit, there are two valid semantics for a
commit operation:

1. Maintain a 'golden' base, and a transient overlay. Make
changes in the overlay, and if everything appears to work,
commit those changes into the base, but still keep the overlay
for the next round of changes; repeat the cycle as desired.

2. Create an external snapshot, then back up the stable state
in the backing file. Once the backup is complete, commit the
overlay back into the base, and delete the temporary snapshot.

Since qemu doesn't know up front which of the two styles is
preferred, a block commit of the active layer merely gets
the job into a synchronized state, and sends an event; then
the user must either cancel (case 1) or complete (case 2),
where qemu then sends a second event that actually ends the
job.  However, until commit e6bcbcd3, libvirt was blindly
assuming the semantics that apply to a commit of an
intermediate image, where there is only one sane conclusion
(the job automatically ends with fewer elements in the chain);
and getting stuck because it wasn't prepared for qemu to enter
a second phase of the job.

This patch adds a flag to the libvirt API that a user MUST
supply in order to acknowledge that they will be using two-phase
semantics.  It might be possible to have a mode where if the
flag is omitted, we automatically do the case 2 semantics on
the user's behalf; but before that happens, I must do additional
patches to track the fact that we are doing an active commit
in the domain XML.  Later patches will add support of the flag,
and once 2-phase semantics are working, we can then decide
whether to relax things to allow an omitted flag to cause an
automatic pivot.

* include/libvirt/libvirt.h.in (VIR_DOMAIN_BLOCK_COMMIT_ACTIVE)
(VIR_DOMAIN_BLOCK_JOB_TYPE_ACTIVE_COMMIT): New enums.
* src/libvirt.c (virDomainBlockCommit): Document two-phase job
when committing active layer, through new flag.
(virDomainBlockJobAbort): Document that pivot also occurs after
active commit.
* tools/virsh-domain.c (vshDomainBlockJob): Cover new job.
* src/qemu/qemu_driver.c (qemuDomainBlockCommit): Explicitly
reject active copy; later patches will add it in.
Signed-off-by: NEric Blake <eblake@redhat.com>

The documentation of the functions should match the argument name in the actual
function signature.
Signed-off-by: NDan Kenigsberg <danken@redhat.com>

The current error message is

    error: use virDomainMigrateToURI3 for peer-to-peer migration

which is correct but a bit misleading because the client did not specify
VIR_MIGRATE_PEER2PEER flag. This patch changes the error message to

    error: cannot perform tunnelled migration without using peer2peer
    flag

which is consistent with the error reported by older migration APIs.

Reported by Rich Jones in
https://bugzilla.redhat.com/show_bug.cgi?id=1095924

In a number of APIs, the text implied that a user might have
<target dev='xvda'/> - but common convention is to use "vda",
not "xvda".  For example, virDomainGetDiskErrors was correct,
while virDomainBlockStats was confusing.

* src/libvirt.c: Make examples consistent.
Signed-off-by: NEric Blake <eblake@redhat.com>

These APIs allow users to get or set time in a domain, which may come
handy if the domain has been resumed just recently and NTP is not
configured or hasn't kicked in yet and the guest is running
something time critical. In addition, NTP may refuse to re-set the clock
if the skew is too big.

In addition, new ACL attribute is introduced 'set_time'.
Signed-off-by: NMichal Privoznik <mprivozn@redhat.com>

These will freeze and thaw filesystems within guest specified by
@mountpoints parameters. The parameters can be NULL and 0, then all
mounted filesystems are frozen or thawed. @flags parameter, which are
currently not used, is for future extensions.
Signed-off-by: NTomoki Sekiyama <tomoki.sekiyama@hds.com>
Signed-off-by: NEric Blake <eblake@redhat.com>

https://bugzilla.redhat.com/show_bug.cgi?id=808463

Well, libvirt doesn't distinguish between domain poweroff and
hibernation (S4). It's hard to differentiate these two on a real
machine anyway. As a result, any device that is hot(un-)plugged is
lost (appears again) when domain is started again as from our POV
it is a fresh cold boot. Instead of doing anything wise here, we
should just document this as known limitation.
Signed-off-by: NMichal Privoznik <mprivozn@redhat.com>

Each backing store of a given disk is associated with a unique index
(which is also formatted in domain XML) for easier addressing of any
particular backing store. With this patch, any backing store can be
addressed by its disk target and the index. For example, "vdc[4]"
addresses the backing store with index equal to 4 of the disk identified
by "vdc" target. Such shorthand can be used in any API in place for a
backing file path:

    virsh blockcommit domain vda --base vda[3] --top vda[2]
Signed-off-by: NJiri Denemark <jdenemar@redhat.com>

Fixes https://bugzilla.redhat.com/show_bug.cgi?id=1088748

The hotplugged vcpus could fail to show up in the guest if
they were added during boot, see:
https://bugzilla.redhat.com/show_bug.cgi?id=807508

Also change arbitrary to arbitrarily.

--memory-only option is introduced without compression supported. Now qemu
has support for dumping domain's memory in kdump-compressed format. This
patch adds a new virDomainCoreDumpWithFormat API, so that the format in
which qemu dumps domain's memory can be specified.
Signed-off-by: NQiao Nuohan <qiaonuohan@cn.fujitsu.com>
Signed-off-by: NEric Blake <eblake@redhat.com>

Busy enterprise workloads hosted on large sized VM's tend to dirty
memory faster than the transfer rate achieved via live guest migration.
Despite some good recent improvements (& using dedicated 10Gig NICs
between hosts) the live migration may NOT converge.

Recently support was added in qemu (version 1.6) to allow a user to
choose if they wish to force convergence of their migration via a
new migration capability : "auto-converge". This feature allows for qemu
to auto-detect lack of convergence and trigger a throttle-down of the
VCPUs.

This patch includes the libvirt support needed to trigger this
feature. (Testing is in progress)
Signed-off-by: NChegu Vinod <chegu_vinod@hp.com>
Signed-off-by: NJiri Denemark <jdenemar@redhat.com>

Any source file which calls the logging APIs now needs
to have a VIR_LOG_INIT("source.name") declaration at
the start of the file. This provides a static variable
of the virLogSource type.
Signed-off-by: NDaniel P. Berrange <berrange@redhat.com>