Introduce a local object virIdentity for managing security
attributes used to form a client application's identity.
Instances of this object are intended to be used as if they
were immutable, once created & populated with attributes
Signed-off-by: NDaniel P. Berrange <berrange@redhat.com>

A socket object has various pieces of security data associated
with it, such as the SELinux context, the SASL username and
the x509 distinguished name. Add new APIs to virNetServerClient
and related modules to access this data.
Signed-off-by: NDaniel P. Berrange <berrange@redhat.com>

Add necessary handling code for the new s390 CCW address type to
virDomainDeviceInfo. Further, introduce  memory management, XML
parsing, output formatting and range validation for the new
virDomainDeviceCCWAddress type.
Signed-off-by: NViktor Mihajlovski <mihajlov@linux.vnet.ibm.com>

A number of symbols are only present when GNUTLS is enabled.
Thus we must use a separate libvirt_gnutls.syms file for them
instead of libvirt_private.syms
Signed-off-by: NDaniel P. Berrange <berrange@redhat.com>

The virCaps structure gathered a ton of irrelevant data over time that.
The original reason is that it was propagated to the XML parser
functions.

This patch aims to create a new data structure virDomainXMLConf that
will contain immutable data that are used by the XML parser. This will
allow two things we need:

1) Get rid of the stuff from virCaps

2) Allow us to add callbacks to check and add driver specific stuff
after domain XML is parsed.

This first attempt removes pointers to private data allocation functions
to this new structure and update all callers and function that require
them.

Currently the server determines whether authentication of clients
is complete, by checking whether an identity is set. This patch
removes that lame hack and replaces it with an explicit method
for changing the client auth code

* daemon/remote.c: Update for new APis
* src/libvirt_private.syms, src/rpc/virnetserverclient.c,
  src/rpc/virnetserverclient.h: Remove virNetServerClientGetIdentity
  and virNetServerClientSetIdentity, adding a new method
  virNetServerClientSetAuth.
Signed-off-by: NDaniel P. Berrange <berrange@redhat.com>

Add a virThreadCancel function. This functional is inherently
dangerous and not something we want to use in general, but
integration with SELinux requires that we provide this stub.
We leave out any Win32 impl to discourage further use and
because obviously SELinux isn't enabled on Win32
Signed-off-by: NDaniel P. Berrange <berrange@redhat.com>

When reading log output from QEMU/LXC we need to skip over any
libvirt log messages. Currently the QEMU driver checks for a
fixed string, but this is better done with a regex. Add a method
virLogProbablyLogMessage to do a regex check
Signed-off-by: NDaniel P. Berrange <berrange@redhat.com>

This reverts commit 0bbbd42c.

The design for this feature is not complete, and may change the
name of the 'schid' attribute.  Revert requested by Viktor Mihajlovski.

This patch adds basic configuration support for the RNG device
supporting the virtio model with the "random" and "egd" backend types as
described in the schema in the previous patch.

For both AttachDevice and UpdateDevice APIs, if the disk device
is 'cdrom' or 'floppy', the operations could be ejecting, updating,
and inserting. For either ejecting or updating, the shared disk
entry of the original disk src has to be removed, because it's
not useful anymore.

And since the original disk def will be changed, new disk def passed
as argument will be free'ed in qemuDomainChangeEjectableMedia, so
we need to copy the orignal disk def before
qemuDomainChangeEjectableMedia, to use it for qemuRemoveSharedDisk.

Automating a sorting check is the only way to ensure we don't
regress.  Suggested by Dan Berrange.

* src/check-symsorting.pl (check_sorting): Add a parameter,
validate that groups are in order, and that files exist.
* src/Makefile.am (check-symsorting): Adjust caller.
* src/libvirt_private.syms: Fix typo.
* src/libvirt_linux.syms: Fix file name.
* src/libvirt_vmx.syms: Likewise.
* src/libvirt_xenxs.syms: Likewise.
* src/libvirt_sasl.syms: Likewise.
* src/libvirt_libssh2.syms: Likewise.
* src/libvirt_esx.syms: Mention file name.
* src/libvirt_openvz.syms: Likewise.

Purely mechanical (roughly, s/\n/~/; s/~~/\n/; sort by line;
s/~/\n/)

* src/libvirt_private.syms: Sort sections by header file name.

Recent renames were not reflected into the comments of
libvirt_private.syms; furthermore, since we mix private headers from
several directories into this file, knowing where the file lives
can be helpful.

* src/libvirt_private.sym: Reflect recent names.

Normally when a process' uid is changed to non-0, all the capabilities
bits are cleared, even those explicitly set with calls to
capng_update()/capng_apply() made immediately before setuid. And
*after* the process' uid has been changed, it no longer has the
necessary privileges to add capabilities back to the process.

In order to set a non-0 uid while still maintaining any capabilities
bits, it is necessary to either call capng_change_id() (which
unfortunately doesn't currently call initgroups to setup auxiliary
group membership), or to perform the small amount of calisthenics
contained in the new utility function virSetUIDGIDWithCaps().

Another very important difference between the capabilities
setting/clearing in virSetUIDGIDWithCaps() and virCommand's
virSetCapabilities() (which it will replace in the next patch) is that
the new function properly clears the capabilities bounding set, so it
will not be possible for a child process to set any new
capabilities.

A short description of what is done by virSetUIDGIDWithCaps():

1) clear all capabilities then set all those desired by the caller (in
capBits) plus CAP_SETGID, CAP_SETUID, and CAP_SETPCAP (which is needed
to change the capabilities bounding set).

2) call prctl(), telling it that we want to maintain current
capabilities across an upcoming setuid().

3) switch to the new uid/gid

4) again call prctl(), telling it we will no longer want capabilities
maintained if this process does another setuid().

5) clear the capabilities that we added to allow us to
setuid/setgid/change the bounding set (unless they were also requested
by the caller via the virCommand API).

Because the modification/maintaining of capabilities is intermingled
with setting the uid, this is necessarily done in a single function,
rather than having two independent functions.

Note that, due to the way that effective capabilities are computed (at
time of execve) for a process that has uid != 0, the *file*
capabilities of the binary being executed must also have the desired
capabilities bit(s) set (see "man 7 capabilities"). This can be done
with the "filecap" command. (e.g. "filecap /usr/bin/qemu-kvm sys_rawio").

The existing virSecurityManagerSetProcessLabel() API is designed so
that it must be called after forking the child process, but before
exec'ing the child. Due to the way the virCommand API works, that
means it needs to be put in a "hook" function that virCommand is told
to call out to at that time.

Setting the child process label is a basic enough need when executing
any process that virCommand should have a method of doing that. But
virCommand must be told what label to set, and only the security
driver knows the answer to that question.

The new virSecurityManagerSet*Child*ProcessLabel() API is the way to
transfer the knowledge about what label to set from the security
driver to the virCommand object. It is given a virCommandPtr, and each
security driver calls the appropriate virCommand* API to tell
virCommand what to do between fork and exec.

1) in the case of the DAC security driver, it calls
virCommandSetUID/GID() to set a uid and gid that must be set for the
child process.

2) for the SELinux security driver, it calls
virCommandSetSELinuxLabel() to save a copy of the char* that will be
sent to setexeccon_raw() *after forking the child process*.

3) for the AppArmor security drivers, it calls
virCommandSetAppArmorProfile() to save a copy of the char* that will
be sent to aa_change_profile() *after forking the child process*.

With this new API in place, we will be able to remove
virSecurityManagerSetProcessLabel() from any virCommand pre-exec
hooks.

(Unfortunately, the LXC driver uses clone() rather than virCommand, so
it can't take advantage of this new security driver API, meaning that
we need to keep around the older virSecurityManagerSetProcessLabel(),
at least for now.)

virCommand gets two new APIs: virCommandSetSELinuxLabel() and
virCommandSetAppArmorProfile(), which both save a copy of a
null-terminated string in the virCommand. During virCommandRun, if the
string is non-NULL and we've been compiled with AppArmor and/or
SELinux security driver support, the appropriate security library
function is called for the child process, using the string that was
previously set. In the case of SELinux, setexeccon_raw() is called,
and for AppArmor, aa_change_profile() is called.

This functionality has been added so that users of virCommand can use
the upcoming virSecurityManagerSetChildProcessLabel() prior to running
a child process, rather than needing to setup a hook function to be
called (and in turn call virSecurityManagerSetProcessLabel()) *during*
the setup of the child process.

If a uid and/or gid is specified for a command, it will be set just
after the user-supplied post-fork "hook" function is called.

The intent is that this can replace user hook functions that set
uid/gid. This moves the setting of uid/gid and dropping of
capabilities closer to each other, which is important since the two
should really be done at the same time (libcapng provides a single
function that does both, which we will be unable to use, but want to
mimic as closely as possible).

The hook scripts used by virCommand must be careful wrt
accessing any mutexes that may have been held by other
threads in the parent process. With the recent refactoring
there are 2 potential flaws lurking, which will become real
deadlock bugs once the global QEMU driver lock is removed.

Remove use of the QEMU driver lock from the hook function
by passing in the 'virQEMUDriverConfigPtr' instance directly.

Add functions to the virSecurityManager to be invoked before
and after fork, to ensure the mutex is held by the current
thread. This allows it to be safely used in the hook script
in the child process.
Signed-off-by: NDaniel P. Berrange <berrange@redhat.com>

Add necessary handling code for the new s390 CCW address type to
virDomainDeviceInfo. Further, introduce  memory management, XML
parsing, output formatting and range validation for the new
virDomainDeviceCCWAddress type.
Signed-off-by: NViktor Mihajlovski <mihajlov@linux.vnet.ibm.com>

To enable locking to be introduced to the security manager
objects later, turn virSecurityManager into a virObjectLockable
class
Signed-off-by: NDaniel P. Berrange <berrange@redhat.com>

To enable virCapabilities instances to be reference counted,
turn it into a virObject. All cases of virCapabilitiesFree
turn into virObjectUnref
Signed-off-by: NDaniel P. Berrange <berrange@redhat.com>

We are requesting for stderr catching for all cases in
virFileWrapperFdNew(). There is no need to have a separate
function just to report an error, esp. when we can do it in
virFileWrapperFdClose().

We had an easy way to iterate set bits, but not for iterating
cleared bits.

* src/util/virbitmap.h (virBitmapNextClearBit): New prototype.
* src/util/virbitmap.c (virBitmapNextClearBit): Implement it.
* src/libvirt_private.syms (bitmap.h): Export it.
* tests/virbitmaptest.c (test4): Test it.

To allow modifications to the lists to be synchronized, convert
virPCIDeviceList and virUSBDeviceList into virObjectLockable
classes. The locking, however, will not be self-contained. The
users of these classes will have to call virObjectLock/Unlock
in the critical regions.
Signed-off-by: NDaniel P. Berrange <berrange@redhat.com>

Rename all the usbDeviceXXX and usbXXXDevice APIs to have a
fixed virUSBDevice name prefix

Rename all the pciDeviceXXX and pciXXXDevice APIs to have a
fixed virPCIDevice name prefix

The duplicate VM checking should be done atomically with
virDomainObjListAdd, so shoud not be a separate function.
Instead just use flags to indicate what kind of checks are
required.

This pair, used in virDomainCreateXML:

   if (virDomainObjListIsDuplicate(privconn->domains, def, 1) < 0)
     goto cleanup;
   if (!(dom = virDomainObjListAdd(privconn->domains,
                                   privconn->caps,
                                   def, false)))
     goto cleanup;

Changes to

   if (!(dom = virDomainObjListAdd(privconn->domains,
                                   privconn->caps,
                                   def,
                                   VIR_DOMAIN_OBJ_LIST_ADD_CHECK_LIVE,
                                   NULL)))
     goto cleanup;

This pair, used in virDomainRestoreFlags:

   if (virDomainObjListIsDuplicate(privconn->domains, def, 1) < 0)
     goto cleanup;
   if (!(dom = virDomainObjListAdd(privconn->domains,
                                   privconn->caps,
                                   def, true)))
     goto cleanup;

Changes to

   if (!(dom = virDomainObjListAdd(privconn->domains,
                                   privconn->caps,
                                   def,
                                   VIR_DOMAIN_OBJ_LIST_ADD_LIVE |
                                   VIR_DOMAIN_OBJ_LIST_ADD_CHECK_LIVE,
                                   NULL)))
     goto cleanup;

This pair, used in virDomainDefineXML:

   if (virDomainObjListIsDuplicate(privconn->domains, def, 0) < 0)
     goto cleanup;
   if (!(dom = virDomainObjListAdd(privconn->domains,
                                   privconn->caps,
                                   def, false)))
     goto cleanup;

Changes to

   if (!(dom = virDomainObjListAdd(privconn->domains,
                                   privconn->caps,
                                   def,
                                   0, NULL)))
     goto cleanup;

As a step towards making virDomainObjList thread-safe turn it
into an opaque virObject, preventing any direct access to its
internals.

As part of this a new method virDomainObjListForEach is
introduced to replace all existing usage of virHashForEach

The APIs names for accessing the domain list object are
very inconsistent. Rename them all to have a standard
virDomainObjList prefix.

Currently, if we want to feed stdin, or catch stdout or stderr of a
virCommand we have to use virCommandRun(). When using virCommandRunAsync()
we have to register FD handles by hand. This may lead to code duplication.
Hence, introduce an internal API, which does this automatically within
virCommandRunAsync(). The intended usage looks like this:

    virCommandPtr cmd = virCommandNew*(...);
    char *buf = NULL;

    ...

    virCommandSetOutputBuffer(cmd, &buf);
    virCommandDoAsyncIO(cmd);

    if (virCommandRunAsync(cmd, NULL) < 0)
        goto cleanup;

    ...

    if (virCommandWait(cmd, NULL) < 0)
        goto cleanup;

    /* @buf now contains @cmd's stdout */
    VIR_DEBUG("STDOUT: %s", NULLSTR(buf));

    ...

cleanup:
    VIR_FREE(buf);
    virCommandFree(cmd);

Note, that both stdout and stderr buffers may change until virCommandWait()
returns.

While working with a pmsuspend vs. snapshot issue, I noticed that
the state file in /var/run/libvirt/qemu/dom.xml contained a rather
suspicious "(null)" string, which does not round-trip well through
a libvirtd restart.  Had I been on a platform other than glibc
where printf("%s",NULL) crashes instead of printing (null), we might
have noticed the problem much sooner.

And in fixing that problem, I also noticed that we had several
missing states, because we were #defining several *_LAST names
to a value _different_ than what they were already given as enums
in libvirt.h.  Yuck.  I got rid of default: labels in the case
statements, because they get in the way of gcc's -Wswitch helping
us ensure we cover all enum values.

* src/conf/domain_conf.c (virDomainStateReasonToString)
(virDomainStateReasonFromString): Fill in missing domain states;
rewrite case statement to let compiler enforce checking.
(VIR_DOMAIN_NOSTATE_LAST, VIR_DOMAIN_RUNNING_LAST)
(VIR_DOMAIN_BLOCKED_LAST, VIR_DOMAIN_PAUSED_LAST)
(VIR_DOMAIN_SHUTDOWN_LAST, VIR_DOMAIN_SHUTOFF_LAST)
(VIR_DOMAIN_CRASHED_LAST): Drop dead defines.
(VIR_DOMAIN_PMSUSPENDED_LAST): Drop dead define.
(virDomainPMSuspendedReason): Add missing enum function.
(virDomainRunningReason, virDomainPausedReason): Add missing enum
value.
* src/conf/domain_conf.h (virDomainPMSuspendedReason): Declare
missing functions.
* src/libvirt_private.syms (domain_conf.h): Export them.

I got bit by 'make check' complaining that the sort order I got
by emacs' sort-lines function differed from expectations.

* src/libvirt_private.syms: Add emacs trailer.
* src/libvirt_atomic.syms: Likewise.
* src/libvirt_daemon.syms: Likewise.
* src/libvirt_esx.syms: Likewise.
* src/libvirt_libssh2.syms: Likewise.
* src/libvirt_linux.syms: Likewise.
* src/libvirt_openvz.syms: Likewise.
* src/libvirt_sasl.syms: Likewise.
* src/libvirt_vmx.syms: Likewise.
* src/libvirt_xenxs.syms: Likewise.

This will allow storing additional topology data in the NUMA topology
definition.

This patch changes the storage type and fixes fallout of the change
across the drivers using it.

This patch also changes semantics of adding new NUMA cell information.
Until now the data were re-allocated and copied to the topology
definition. This patch changes the addition function to steal the
pointer to a pre-allocated structure to simplify the code.

The function is just a renamed public version of former
virTypedParameterArrayClear.

Introduce a virPortAllocator for managing TCP port allocations.
Signed-off-by: NDaniel P. Berrange <berrange@redhat.com>

The  virDomainObj, qemuAgent, qemuMonitor, lxcMonitor classes
all require a mutex, so can be switched to use virObjectLockable
Signed-off-by: NDaniel P. Berrange <berrange@redhat.com>

A great many virObject instances require a mutex, so introduce
a convenient class for this which provides a mutex. This avoids
repeating the tedious init/destroy code
Signed-off-by: NDaniel P. Berrange <berrange@redhat.com>

Currently all classes must directly inherit from virObject.
This allows for arbitrarily deep hierarchy. There's not much
to this aside from chaining up the 'dispose' handlers from
each class & providing APIs to check types.
Signed-off-by: NDaniel P. Berrange <berrange@redhat.com>

Add an optional 'type' attribute to <target> element of serial port
device. There are two choices for its value, 'isa-serial' and
'usb-serial'. For backward compatibility, when attribute 'type' is
missing the 'isa-serial' will be chosen as before.

Libvirt XML sample

    <serial type='pty'>
      <target type='usb-serial' port='0'/>
      <address type='usb' bus='0' port='1'/>
    </serial>

qemu commandline:

qemu ${other_vm_args}              \
    -chardev pty,id=charserial0    \
    -device usb-serial,chardev=charserial0,id=serial0,bus=usb.0,port=1