For now, disk migration via block copy job is not implemented in
libvirt.  But when we do implement it, we have to deal with the
fact that qemu does not yet provide an easy way to re-start a qemu
process with mirroring still intact.  Paolo has proposed an idea
for a persistent dirty bitmap that might make this possible, but
until that design is complete, it's hard to say what changes
libvirt would need.  Even something like 'virDomainSave' becomes
hairy, if you realize the implications that 'virDomainRestore'
would be stuck with recreating the same mirror layout.

But if we step back and look at the bigger picture, we realize that
the initial client of live storage migration via disk mirroring is
oVirt, which always uses transient domains, and that if a transient
domain is destroyed while a mirror exists, oVirt can easily restart
the storage migration by creating a new domain that visits just the
source storage, with no loss in data.

We can make life a lot easier by being cowards for now, forbidding
certain operations on a domain.  This patch guarantees that we
never get in a state where we would have to restart a domain with
a mirroring block copy, by preventing saves, snapshots, migration,
hot unplug of a disk in use, and conversion to a persistent domain
(thankfully, it is still relatively easy to 'virsh undefine' a
running domain to temporarily make it transient, run tests on
'virsh blockcopy', then 'virsh define' to restore the persistence).
Later, if the qemu design is enhanced, we can relax our code.

The change to qemudDomainDefine looks a bit odd for undoing an
assignment, rather than probing up front to avoid the assignment,
but this is because of how virDomainAssignDef combines both a
lookup and assignment into a single function call.

* src/conf/domain_conf.h (virDomainHasDiskMirror): New prototype.
* src/conf/domain_conf.c (virDomainHasDiskMirror): New function.
* src/libvirt_private.syms (domain_conf.h): Export it.
* src/qemu/qemu_driver.c (qemuDomainSaveInternal)
(qemuDomainSnapshotCreateXML, qemuDomainRevertToSnapshot)
(qemuDomainBlockJobImpl, qemudDomainDefine): Prevent dangerous
actions while block copy is already in action.
* src/qemu/qemu_hotplug.c (qemuDomainDetachDiskDevice): Likewise.
* src/qemu/qemu_migration.c (qemuMigrationIsAllowed): Likewise.

This resolves:

   https://bugzilla.redhat.com/show_bug.cgi?id=862515

which describes inconsistencies in dealing with duplicate mac
addresses on network devices in a domain.

(at any rate, it resolves *almost* everything, and prints out an
informative error message for the one problem that isn't solved, but
has a workaround.)

A synopsis of the problems:

1) you can't do a persistent attach-interface of a device with a mac
address that matches an existing device.

2) you *can* do a live attach-interface of such a device.

3) you *can* directly edit a domain and put in two devices with
matching mac addresses.

4) When running virsh detach-device (live or config), only MAC address
is checked when matching the device to remove, so the first device
with the desired mac address will be removed. This isn't always the
one that's wanted.

5) when running virsh detach-interface (live or config), the only two
items that can be specified to match against are mac address and model
type (virtio, etc) - if multiple netdevs match both of those
attributes, it again just finds the first one added and assumes that
is the only match.

Since it is completely valid to have multiple network devices with the
same MAC address (although it can cause problems in many cases, there
*are* valid use cases), what is needed is:

1) remove the restriction that prohibits doing a persistent add of a
netdev with a duplicate mac address.

2) enhance the backend of virDomainDetachDeviceFlags to check for
something that *is* guaranteed unique (but still work with just mac
address, as long as it yields only a single results.

This patch does three things:

1) removes the check for duplicate mac address during a persistent
netdev attach.

2) unifies the searching for both live and config detach of netdevices
in the subordinate functions of qemuDomainModifyDeviceFlags() to use the
new function virDomainNetFindIdx (which matches mac address and PCI
address if available, checking for duplicates if only mac address was
specified). This function returns -2 if multiple matches are found,
allowing the callers to print out an appropriate message.

Steps 1 & 2 are enough to fully fix the problem when using virsh
attach-device and detach-device (which require an XML description of
the device rather than a bunch of commandline args)

3) modifies the virsh detach-interface command to check for multiple
matches of mac address and show an error message suggesting use of the
detach-device command in cases where there are multiple matching mac
addresses.

Later we should decide how we want to input a PCI address on the virsh
commandline, and enhance detach-interface to take a --address option,
eliminating the need to use detach-device

* src/conf/domain_conf.c
* src/conf/domain_conf.h
* src/libvirt_private.syms
  * added new virDomainNetFindIdx function
  * removed now unused virDomainNetIndexByMac and
    virDomainNetRemoveByMac

* src/qemu/qemu_driver.c
  * remove check for duplicate max from qemuDomainAttachDeviceConfig
  * use virDomainNetFindIdx/virDomainNetRemove instead
    of virDomainNetRemoveByMac in qemuDomainDetachDeviceConfig
  * use virDomainNetFindIdx instead of virDomainIndexByMac
    in qemuDomainUpdateDeviceConfig

* src/qemu/qemu_hotplug.c
  * use virDomainNetFindIdx instead of a homespun loop in
    qemuDomainDetachNetDevice.

* tools/virsh-domain.c: modified detach-interface command as described
    above

It turns out that the cpuacct results properly account for offline
cpus, and always returns results for every possible cpu, not just
the online ones.  So there is no need to check the map of online
cpus in the first place, merely only a need to know the maximum
possible cpu.  Meanwhile, virNodeGetCPUBitmap had a subtle change
from returning the maximum id to instead returning the width of
the bitmap (one larger than the maximum id) in commit 2f4c5338,
which made this code encounter some off-by-one logic leading to
bad error messages when a cpu was offline:

$ virsh cpu-stats dom
error: Failed to virDomainGetCPUStats()

error: An error occurred, but the cause is unknown

Cleaning this up unraveled a chain of other unused variables.

* src/qemu/qemu_driver.c (qemuDomainGetPercpuStats): Drop
pointless check for cpumap changes, and use correct number of
cpus.  Simplify signature.
(qemuDomainGetCPUStats): Adjust caller.
* src/nodeinfo.h (nodeGetCPUCount): New prototype.
(nodeGetCPUBitmap): Drop unused parameter.
* src/nodeinfo.c (nodeGetCPUBitmap): Likewise.
(nodeGetCPUMap): Adjust caller.
(nodeGetCPUCount): New function.
* src/libvirt_private.syms (nodeinfo.h): Export it.

Added an implemention of virNodeGetCPUMap to nodeinfo.c,
(nodeGetCPUMap) which can be used by all drivers for a Linux
hypervisor host.
Signed-off-by: NViktor Mihajlovski <mihajlov@linux.vnet.ibm.com>

Callers should not need to know what the name of the file to
be read in the Linux-specific version of nodeGetCPUmap;
furthermore, qemu cares about online cpus, not present cpus,
when determining which cpus to skip.

While at it, I fixed the fact that we were computing the maximum
online cpu id by doing a slow iteration, when what we really want
to know is the max available cpu.

* src/nodeinfo.h (nodeGetCPUmap): Rename...
(nodeGetCPUBitmap): ...and simplify signature.
* src/nodeinfo.c (linuxParseCPUmax): New function.
(linuxParseCPUmap): Simplify and alter signature.
(nodeGetCPUBitmap): Change implementation.
* src/libvirt_private.syms (nodeinfo.h): Reflect rename.
* src/qemu/qemu_driver.c (qemuDomainGetPercpuStats): Update
caller.

Sometimes it's handy to know how many bits are set.

* src/util/bitmap.h (virBitmapCountBits): New prototype.
(virBitmapNextSetBit): Use correct type.
* src/util/bitmap.c (virBitmapNextSetBit): Likewise.
(virBitmapSetAll): Maintain invariant of clear tail bits.
(virBitmapCountBits): New function.
* src/libvirt_private.syms (bitmap.h): Export it.
* tests/virbitmaptest.c (test2): Test it.

Add utility functions for Open vSwitch to both save
per-port data before a live migration, and restore the
per-port data after a live migration.
Signed-off-by: NKyle Mestery <kmestery@cisco.com>

https://bugzilla.redhat.com/show_bug.cgi?id=866364

pointed out a crash due to virNetworkObjAssignDef free'ing
network->newDef without NULLing it afterward. A fix for this is in
upstream commit b7e92024. While the
NULLing of newDef was a legitimate fix, newDef should have already
been empty (NULL) anyway (as indicated in the comment that was deleted
by that commit).

The reason that newDef had a non-NULL value (i.e. the root cause) was
that networkStartNetwork() had failed after populating
network->newDef, but then neglected to free/NULL newDef in the
cleanup.

(A bit of background here: network->newDef should contain the
persistent config of a network when a network is active (and of course
only when it is persisten), and NULL at all other times. There is also
a network->def which should contain the persistent definition of the
network when it is inactive, and the current live state at all other
times. The idea is that you can make changes to network->newDef which
will take effect the next time the network is restarted, but won't
mess with the current state of the network (virDomainObj has a similar
pair of virDomainDefs that behave in the same fashion). Personally I
think there should be a network->live and network->config, and the
location of the persistent config should *always* be in
network->config, but that's for a later cleanup).

Since I love things to be symmetric, I created a new function called
virNetworkObjUnsetDefTransient(), which reverses the effects of
virNetworkObjSetDefTransient(). I don't really like the name of the
new function, but then I also didn't really like the name of the old
one either (it's just named that way to match a similar function in
the domain conf code).

In order to temporarily label files read/write during a commit
operation, we need to crawl the backing chain and find the absolute
file name that needs labeling in the first place, as well as the
name of the file that owns the backing file.

* src/util/storage_file.c (virStorageFileChainLookup): New
function.
* src/util/storage_file.h: Declare it.
* src/libvirt_private.syms (storage_file.h): Export it.

Hypervisors are starting to support HyperV Enlightenment features that
improve behavior of guests running Microsoft Windows operating systems.

This patch adds support for the "relaxed" feature that improves timer
behavior and also establishes a framework to add these features in
future.

The apic-eoi feature enum and implementation can be made more universal
to allow re-use of the enum for other features.

Add two new APIs virNetServerNewPostExecRestart and
virNetServerPreExecRestart which allow a virNetServerPtr
object to be created from a JSON object and saved to a
JSON object, for the purpose of re-exec'ing a process.

This includes serialization of all registered services
and clients
Signed-off-by: NDaniel P. Berrange <berrange@redhat.com>

Add two new APIs virNetServerClientNewPostExecRestart and
virNetServerClientPreExecRestart which allow a virNetServerClientPtr
object to be created from a JSON object and saved to a
JSON object, for the purpose of re-exec'ing a process.

This includes serialization of the connected socket associated
with the client
Signed-off-by: NDaniel P. Berrange <berrange@redhat.com>

Add two new APIs virNetServerServiceNewPostExecRestart and
virNetServerServicePreExecRestart which allow a virNetServerServicePtr
object to be created from a JSON object and saved to a
JSON object, for the purpose of re-exec'ing a process.

This includes serialization of the listening sockets associated
with the service
Signed-off-by: NDaniel P. Berrange <berrange@redhat.com>

Add two new APIs virNetSocketNewPostExecRestart and
virNetSocketPreExecRestart which allow a virNetSocketPtr
object to be created from a JSON object and saved to a
JSON object, for the purpose of re-exec'ing a process.

As well as saving the state in JSON format, the second
method will disable the O_CLOEXEC flag so that the open
file descriptors are preserved across the process re-exec()

Since it is not possible to serialize SASL or TLS encryption
state, an error will be raised if attempting to perform
serialization on non-raw sockets
Signed-off-by: NDaniel P. Berrange <berrange@redhat.com>

Add two new APIs virLockSpaceNewPostExecRestart and
virLockSpacePreExecRestart which allow a virLockSpacePtr
object to be created from a JSON object and saved to a
JSON object, for the purposes of re-exec'ing a process.

As well as saving the state in JSON format, the second
method will disable the O_CLOEXEC flag so that the open
file descriptors are preserved across the process re-exec()
Signed-off-by: NDaniel P. Berrange <berrange@redhat.com>

The previously introduced virFile{Lock,Unlock} APIs provide a
way to acquire/release fcntl() locks on individual files. For
unknown reason though, the POSIX spec says that fcntl() locks
are released when *any* file handle referring to the same path
is closed. In the following sequence

  threadA: fd1 = open("foo")
  threadB: fd2 = open("foo")
  threadA: virFileLock(fd1)
  threadB: virFileLock(fd2)
  threadB: close(fd2)

you'd expect threadA to come out holding a lock on 'foo', and
indeed it does hold a lock for a very short time. Unfortunately
when threadB does close(fd2) this releases the lock associated
with fd1. For the current libvirt use case for virFileLock -
pidfiles - this doesn't matter since the lock is acquired
at startup while single threaded an never released until
exit.

To provide a more generally useful API though, it is necessary
to introduce a slightly higher level abstraction, which is to
be referred to as a "lockspace".  This is to be provided by
a virLockSpacePtr object in src/util/virlockspace.{c,h}. The
core idea is that the lockspace keeps track of what files are
already open+locked. This means that when a 2nd thread comes
along and tries to acquire a lock, it doesn't end up opening
and closing a new FD. The lockspace just checks the current
list of held locks and immediately returns VIR_ERR_RESOURCE_BUSY.

NB, the API as it stands is designed on the basis that the
files being locked are not being otherwise opened and used
by the application code. One approach to using this API is to
acquire locks based on a hash of the filepath.

eg to lock /var/lib/libvirt/images/foo.img the application
might do

   virLockSpacePtr lockspace = virLockSpaceNew("/var/lib/libvirt/imagelocks");
   lockname = md5sum("/var/lib/libvirt/images/foo.img");
   virLockSpaceAcquireLock(lockspace, lockname);

NB, in this example, the caller should ensure that the path
is canonicalized before calculating the checksum.

It is also possible to do locks directly on resources by
using a NULL lockspace directory and then using the file
path as the lock name eg

   virLockSpacePtr lockspace = virLockSpaceNew(NULL);
   virLockSpaceAcquireLock(lockspace, "/var/lib/libvirt/images/foo.img");

This is only safe to do though if no other part of the process
will be opening the files. This will be the case when this
code is used inside the soon-to-be-reposted virlockd daemon
Signed-off-by: NDaniel P. Berrange <berrange@redhat.com>

BZ:https://bugzilla.redhat.com/show_bug.cgi?id=851981
When using macvtap, a character device gets first created by
kernel with name /dev/tapN, its selinux context is:
system_u:object_r:device_t:s0

Shortly, when udev gets notification when new file is created
in /dev, it will then jump in and relabel this file back to the
expected default context:
system_u:object_r:tun_tap_device_t:s0

There is a time gap happened.
Sometimes, it will have migration failed, AVC error message:
type=AVC msg=audit(1349858424.233:42507): avc:  denied  { read write } for
pid=19926 comm="qemu-kvm" path="/dev/tap33" dev=devtmpfs ino=131524
scontext=unconfined_u:system_r:svirt_t:s0:c598,c908
tcontext=system_u:object_r:device_t:s0 tclass=chr_file

This patch will label the tapfd device before qemu process starts:
system_u:object_r:tun_tap_device_t:MCS(MCS from seclabel->label)

This patch adds support for SUSPEND_DISK event; both lifecycle and
separated.  The support is added for QEMU, machines are changed to
PMSUSPENDED, but as QEMU sends SHUTDOWN afterwards, the state changes
to shut-off.  This and much more needs to be done in order for libvirt
to work with transient devices, wake-ups etc.  This patch is not
aiming for that functionality.

This does a shallow copy of all the bits, then strdups the two items
that are actually allocated separately.

The onlined vcpu pinning policy should inherit def->cpuset if
it's not specified explicitly, and the affinity should be set
in this case. Oppositely, the offlined vcpu pinning policy should
be free()'ed.

Using this new element, one can configure an action that should be
performed when resource locks are lost.

After it got renamed to virProcessKill (cf470068)
it is no longer needed in the private syms file.

Add support for logging to the systemd journal, using its
simple client library. The benefit over syslog is that it
accepts structured log data, so the journald can store
individual items like code file/line/func separately from
the string message. Tools which require structured log
data can then query the journal to extract exactly what
they desire without resorting to string parsing

While systemd provides a simple client library for logging,
it is more convenient for libvirt to directly write its
own client code. This lets us build up the iovec's on
the stack, avoiding the need to alloc memory when writing
log messages.
Signed-off-by: NDaniel P. Berrange <berrange@redhat.com>

In the cgroups APIs we have a virCgroupKillPainfully function
which does the loop sending SIGTERM, then SIGKILL and waiting
for the process to exit. There is similar functionality for
simple processes in qemuProcessKill, but it is tangled with
the QEMU code. Untangle it to provide a virProcessKillPainfuly
function
Signed-off-by: NDaniel P. Berrange <berrange@redhat.com>

Continue consolidation of process functions by moving some
helpers out of command.{c,h} into virprocess.{c,h}
Signed-off-by: NDaniel P. Berrange <berrange@redhat.com>

There are a number of process related functions spread
across multiple files. Start to consolidate them by
creating a virprocess.{c,h} file
Signed-off-by: NDaniel P. Berrange <berrange@redhat.com>

The virCommand prefix was inappropriate because the API
does not use any virCommandPtr object instance. This
API closely related to waitpid/exit, so use virProcess
as the prefix
Signed-off-by: NDaniel P. Berrange <berrange@redhat.com>

Change "Pid" to "Process" to align with the virProcessKill
API naming prefix
Signed-off-by: NDaniel P. Berrange <berrange@redhat.com>

Sometimes when guest machine crashes, coredump can get huge due to the
guest memory. This can be limited using madvise(2) system call and is
being used in QEMU hypervisor. This patch adds an option for configuring
that in the domain XML and related documentation.

This patch cleans up building the "-boot" parameter and while on that
fixes one inconsistency by modifying these things:

 - I completed the unfinished virDomainBootMenu enum by specifying
   LAST, declaring it and also declaring the TypeFromString and
   TypeToString parameters.
 - Previously mentioned TypeFromString and TypeToString are used when
   parsing the XML.
 - Last, but not least, visible change is that the "-boot" parameter
   is built and parsed properly:
    - The "order=" prefix is used only when additional parameters are
      used (menu, etc.).
    - It's rewritten in a way that other parameters can be added
      easily in the future (used in following patch).
    - The "order=" parameter is properly parsed regardless to where it
      is placed in the string (e.g. "menu=on,order=nc").
    - The "menu=" parameter (and others in the future) are created
      when they should be (i.e. even when bootindex is supported and
      used, but not when bootloader is selected).

Commit f36309d6 added an export with no matching implementation;
probably a misspelling of an earlier version of the final addition
of virNetworkObjSetDefTransient.

* src/libvirt_private.syms (network_conf.h): Drop bogus
virNetworkSetDefTransient.

virNetworkObjUpdate takes care of all virNetworkUpdate-related changes
to the data stored in the in-memory virNetworkObj list. It should be
called by network drivers that use this in-memory list.

virNetworkObjUpdate *does not* take care of updating any disk-based
copies of the config, nor does it perform any other operations
necessary to have the new config data take effect (e.g. it won't
re-write dnsmasq host files, nor will it send a SIGHUP to dnsmasq) -
those things should all be taken care of in the network driver
function that calls virNetworkObjUpdate (assuming that it returns
success).

These new functions are highly inspired by those in domain_conf.c (but
not identical), and are intended to make it simpler to update the
various combinations of live/persistent network configs.

The network driver wasn't previously as careful about the separation
between the live "status" in network->def and the persistent "config"
in network->newDef (or sometimes in network->def). This series
attempts to remedy some of that, but probably doesn't go all the way
(enough to get these functions working and enable continued work on
virNetworkUpdate though).

bridge_driver.c and test_driver.c were updated in a few places to take
advantage of the new functions and/or account for changes in argument
lists.

Validates the wwn while parsing, error out if it's malformed.

* src/util/util.h: Declare virValidateWWN
* src/util/util.c: Implement virValidateWWN
* src/libvirt_private.syms: Export virValidateWWN.
* src/conf/domain_conf.h: New member 'wwn' for disk def.
* src/conf/domain_conf.c: Parse and format disk <wwn>

virBitmap is recommanded to store cpuset info, and
virBitmapFormat/virBitmapParse can do the format/parse
jobs.

In many places we store bitmap info in a chunk of data
(pointed to by a char *), and have redundant codes to
set/unset bits. This patch extends virBitmap, and convert
those codes to use virBitmap in subsequent patches.

Add a new member variable map_len to store map len of bitmap.
and rename size to max_bit accordingly.

rename virBitmapAlloc to virBitmapNew.

Only implemented for linux platform.

* src/nodeinfo.h: (Declare node{Get,Set}MemoryParameters)
* src/nodeinfo.c: (Implement node{Get,Set}MemoryParameters)
* src/libvirt_private.syms: (Export those two new internal APIs to
  private symbols)