Add new API in order to set the balloon memory driver statistics collection
period in order to allow dynamic period adjustment for the virsh dommemstats to
display balloon stats data

This patch will add the qemuMonitorJSONGetMemoryStats() to execute a
"guest-stats" on the balloonpath using "get-qom" replacing the former
mechanism which looked through the "query-ballon" returned data for
the fields.  The "query-balloon" code only returns 'actual' memory.
Rather than duplicating the existing code, have the JSON API use the
GetBalloonInfo API.

A check in the qemuMonitorGetMemoryStats() will be made to ensure the
balloon driver path has been set.  Since the underlying JSON code can
return data not associated with the balloon driver, we don't fail on
a failure to get the balloonpath.  Of course since we've made the check,
we can then set the ballooninit flag.  Getting the path here is primarily
due to the process reconnect path which doesn't attempt to set the
collection period.

At vm startup and attach attempt to set the balloon driver statistics
collection period based on the value found in the domain xml file. This
is not done at reconnect since it's possible that a collection period
was set on the live guest and making the set period call would reset to
whatever value is stored in the config file.

Setting the stats collection period has a side effect of searching through
the qom-list output for the virtio balloon driver and making sure that it
has the right properties in order to allow setting of a collection period
and eventually fetching of statistics.

The walk through the qom-list is expensive and thus the balloonpath will
be saved in the monitor private structure as well as a flag indicating
that the initialization has already been attempted (in the event that a
path is not found, no sense to keep checking).

This processing model conforms to the qom object model model which
requires setting object properties after device startup. That is, it's
not possible to pass the period along via the startup code as it won't
be recognized.

Add a period in seconds to allow/enable statistics gathering from the
Balloon driver for 'virsh dommemstat <domain>'.

If users haven't configured guest agent then qemuAgentCommand() will
dereference a NULL 'mon' pointer, which causes crash of libvirtd when
using agent based cpu (un)plug.

With the patch, when the qemu-ga service isn't running in the guest,
a expected error "error: Guest agent is not responding: Guest agent
not available for now" will be raised, and the error "error: argument
unsupported: QEMU guest agent is not configured" is raised when the
guest hasn't configured guest agent.

GDB backtrace:

 (gdb) bt
 #0  virNetServerFatalSignal (sig=11, siginfo=<value optimized out>, context=<value optimized out>) at rpc/virnetserver.c:326
 #1  <signal handler called>
 #2  qemuAgentCommand (mon=0x0, cmd=0x7f39300017b0, reply=0x7f394b090910, seconds=-2) at qemu/qemu_agent.c:975
 #3  0x00007f39429507f6 in qemuAgentGetVCPUs (mon=0x0, info=0x7f394b0909b8) at qemu/qemu_agent.c:1475
 #4  0x00007f39429d9857 in qemuDomainGetVcpusFlags (dom=<value optimized out>, flags=9) at qemu/qemu_driver.c:4849
 #5  0x00007f3957dffd8d in virDomainGetVcpusFlags (domain=0x7f39300009c0, flags=8) at libvirt.c:9843

How to reproduce?

 # To start a guest without guest agent configuration
 # then run the following cmdline

 # virsh vcpucount foobar --guest
 error: End of file while reading data: Input/output error
 error: One or more references were leaked after disconnect from the hypervisor
 error: Failed to reconnect to the hypervisor

RHBZ: https://bugzilla.redhat.com/show_bug.cgi?id=984821Signed-off-by: NAlex Jia <ajia@redhat.com>
Signed-off-by: NPeter Krempa <pkrempa@redhat.com>

When using logical pools, we had to trust the target->path provided.
This parameter, however, can be completely ommited and we can use
'/dev/<source.name>' safely and populate it to target.path.

Resolves: https://bugzilla.redhat.com/show_bug.cgi?id=952973

Since previous patches has prepared everything for us, we may now
implement live hotplug of a character device.

There are two levels on which a device may be hotplugged: config
and live. The config level requires just an insert or remove from
internal domain definition structure, which is exactly what this
patch does. There is currently no implementation for a chardev
update action, as there's not much to be updated. But more
importantly, the only thing that can be updated is path or socket
address by which chardevs are distinguished. So the update action
is currently not supported.

Now that we have callbacks, we should auto fill in omitted pieces of
information. It's important for chardev hotplug to fill in the correct
/{serial,parallel,console,channel}/target/@port if no value has been
provided by user.

https://bugzilla.redhat.com/show_bug.cgi?id=799354

Until now, the "host-model" cpu mode couldn't be influenced. This patch
allows to use the <feature> elements to either enable or disable
specific CPU flags. This can be used to force flags that can be emulated
even if the host CPU doesn't support them.

This new function updates or adds a feature to a existing cpu model
definition. This function will be helpful to allow tuning of
"host-model" features in later patches.

Use VIR_ERR_XML_ERROR instead of VIR_ERR_INTERNAL_ERROR in XML parsing
code and move "%s" formating strings right after the error code.

Merge virStoragePoolDefParseAuthChap and virStoragePoolDefParseAuthCephx
into a common virStoragePoolDefParseAuthSecret.  Change the output to be
common for both by putting 'type' first followed by 'username'.

Move the auth->username processing into virStoragePoolDefParseAuth
save the resulting username into chap/cephx specific data

The existing 'chap' XML logic was never used - just defined.  Rather than
try to insert a square peg into a round hole, blow it up and rewrite the
logic to follow the 'ceph' format.

Remove the former "chap.login" and "chap.passwd" fields and replace
with "chap.username" and "chap.secret" in _virStoragePoolAuthChap.
Adjust the virStoragePoolDefParseAuthChap() to process.

Change the rng file to describe the new layout

Update the formatstorage.html to describe the usage of the secret element
to mention that the secret type "iscsi" and "ceph" can be used
to storage pool too.

Update the formatsecret.html to include a reference to the storage pool

Update tests to handle the changes from 'login' and 'passwd' to 'username'
and '<secret>' format

Split processing of "<auth" into its own function

Split out the _virStoragePoolAuthSecret data from _virStoragePoolAuthCephx
into its own structure

Generate and use the virStoragePoolAuthTypeType{To|From}String helpers

Add a new qemuMonitorJSONSetObjectProperty() method to support invocation
of the 'qom-set' JSON monitor command with a provided path, property, and
expected data type to set.

NOTE: The set API was added only for the purpose of the qemumonitorjsontest

The test code uses the same "/machine/i440fx" property as the get test and
attempts to set the "realized" property to "true" (which it should be set
at anyway).

Add a new qemuMonitorJSONGetObjectProperty() method to support invocation
of the 'qom-get' JSON monitor command with a provided path, property, and
expected data type return. The qemuMonitorJSONObjectProperty is similar to
virTypedParameter; however, a future patch will extend it a bit to include
a void pointer to balloon driver statistic data.

NOTE: The ObjectProperty structures and API are added only for the
      purpose of the qemumonitorjsontest

The provided test will execute a qom-get on "/machine/i440fx" which will
return a property "realized".

Add a new qemuMonitorJSONGetObjectListPaths() method to support invocation
of the 'qom-list' JSON monitor command with a provided path.

NOTE: The ListPath structures and API's are added only for the
      purpose of the qemumonitorjsontest

The returned list of paired data fields of "name" and "type" that can
be used to peruse QOM configuration data and eventually utilize for the
balloon statistics.

The test does a "{"execute":"qom-list", "arguments": { "path": "/"}}" which
returns "{"return": [{"name": "machine", "type": "child<container>"},
{"name": "type", "type": "string"}]}" resulting in a return of an array
of 2 elements with [0].name="machine", [0].type="child<container>".  The [1]
entry appears to be a header that could be used some day via a command such
as "virsh qemuobject --list" to format output.

If an error occurs during qemuDomainAttachNetDevice after the macvtap
was created in qemuPhysIfaceConnect, the macvtap device gets left behind.
This patch adds code to the cleanup routine to delete the macvtap.
Signed-off-by: NMatthew Rosato <mjrosato@linux.vnet.ibm.com>
Reviewed-by: NViktor Mihajlovski <mihajlov@linux.vnet.ibm.com>

I recently patches the callers to virPCIDeviceReset() to not call it
if the current driver for a device was vfio-pci (since that driver
will always reset the device itself when appropriate. At the time, Dan
Berrange suggested that I could instead modify virPCIDeviceReset
to check the currently bound driver for the device, and decide
for itself whether or not to go ahead with the reset.

This patch removes the previously added checks, and replaces them with
a check down in virPCIDeviceReset(), as suggested.

The functional difference here is that previously we were deciding
based on either the hostdev configuration or the value of
stubDriverName in the virPCIDevice object, but now we are actually
comparing to the "driver" link in the device's sysfs entry
directly. In practice, both should be the same.

virPCIDeviceGetDriverPathAndName is a static function that will need
to be called by another function that occurs above it in the
file. This patch reorders the static functions so that a forward
declaration isn't needed.

It's caller's responsibility to free return value of
qemuBuildChrDeviceStr().

https://bugzilla.redhat.com/show_bug.cgi?id=981261

Error out on negative index values.

https://bugzilla.redhat.com/show_bug.cgi?id=981261

When failing to start a container due to inaccessible root
filesystem path, we did not log any meaningful error. Add a
few debug statements to assist diagnosis
Signed-off-by: NDaniel P. Berrange <berrange@redhat.com>

The function being introduced is responsible for creating command
line argument for '-device' for given character device. Based on
the chardev type, it calls appropriate qemuBuild.*ChrDeviceStr(),
e.g.  qemuBuildSerialChrDeviceStr() for serial chardev and so on.

The chardev alias assignment is going to be needed in a separate
places, so it should be moved into a separate function rather
than copying code randomly around.

This function wraps 'chardev-remove' qemu monitor command around.
It takes chardev alias as its single argument besides qemu monitor
pointer.

The function being introduced is responsible for preparing and
executing 'chardev-add' qemu monitor command. Moreover, in case
of PTY chardev, the corresponding pty path is updated.

Currently, we are building InetSocketAddress qemu json type
within the qemuMonitorJSONNBDServerStart function. However, other
future functions may profit from the code as well. So it should
be moved into a static function.

For now, only these three helpers are needed:
virDomainChrFind - to find a duplicate chardev within VM def
virDomainChrInsert - wrapper for inserting a new chardev into VM def
virDomainChrRemove - wrapper for removing chardev from VM def

There is, however, one internal helper as well:
virDomainChrGetDomainPtrs which sets given pointers to one of
vmdef->{parallels,serials,consoles,channels} based on passed
chardev type.

This patch enables the password authentication in the libssh2 connection
driver. There are a few benefits to this step:

1) Hosts with challenge response authentication will now be supported
with the libssh2 connection driver.

2) Credential for hosts can now be stored in the authentication
credential config file

The password authentication method wasn't used as there wasn't a
pleasant way to pass the password. This patch adds the option to use
virAuth util functions to request the password either from a config file
or uses the conf callback to request it from the user.

Previously a connection object was required to retrieve the auth
credentials. This patch adds the option to call the retrieval functions
only using the connection URI or path to the configuration file. This
will allow to use this toolkit to request passwords for ssh
authentication in the libssh2 connection driver.

Changes:
*virAuthGetConfigFilePathURI(): use URI to retrieve the config file path
*virAuthGetCredential(): Remove the need to propagate conn object

virAuthGetPasswordPath():
*virAuthGetUsernamePath(): New functions, that use config file path
                           instead of conn object

nodeGetFreeMemory and nodeGetCellsFreeMemory assumed that the NUMA nodes
are contiguous and starting from 0. Unfortunately there are machines
that don't match this assumption:

available: 1 nodes (1)
node 1 cpus: 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15
node 1 size: 16340 MB
node 1 free: 11065 MB

Before this patch:
error: internal error Failed to query NUMA free memory
error: internal error Failed to query NUMA free memory for node: 0

After this patch:
Total: 15772580 KiB
0: 0 KiB
Signed-off-by: NPeter Krempa <pkrempa@redhat.com>

https://bugzilla.redhat.com/show_bug.cgi?id=964358

POSIX states that multi-threaded apps should not use functions
that are not async-signal-safe between fork and exec, yet we
were using getpwuid_r and initgroups.  Although rare, it is
possible to hit deadlock in the child, when it tries to grab
a mutex that was already held by another thread in the parent.
I actually hit this deadlock when testing multiple domains
being started in parallel with a command hook, with the following
backtrace in the child:

 Thread 1 (Thread 0x7fd56bbf2700 (LWP 3212)):
 #0  __lll_lock_wait ()
     at ../nptl/sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:136
 #1  0x00007fd5761e7388 in _L_lock_854 () from /lib64/libpthread.so.0
 #2  0x00007fd5761e7257 in __pthread_mutex_lock (mutex=0x7fd56be00360)
     at pthread_mutex_lock.c:61
 #3  0x00007fd56bbf9fc5 in _nss_files_getpwuid_r (uid=0, result=0x7fd56bbf0c70,
     buffer=0x7fd55c2a65f0 "", buflen=1024, errnop=0x7fd56bbf25b8)
     at nss_files/files-pwd.c:40
 #4  0x00007fd575aeff1d in __getpwuid_r (uid=0, resbuf=0x7fd56bbf0c70,
     buffer=0x7fd55c2a65f0 "", buflen=1024, result=0x7fd56bbf0cb0)
     at ../nss/getXXbyYY_r.c:253
 #5  0x00007fd578aebafc in virSetUIDGID (uid=0, gid=0) at util/virutil.c:1031
 #6  0x00007fd578aebf43 in virSetUIDGIDWithCaps (uid=0, gid=0, capBits=0,
     clearExistingCaps=true) at util/virutil.c:1388
 #7  0x00007fd578a9a20b in virExec (cmd=0x7fd55c231f10) at util/vircommand.c:654
 #8  0x00007fd578a9dfa2 in virCommandRunAsync (cmd=0x7fd55c231f10, pid=0x0)
     at util/vircommand.c:2247
 #9  0x00007fd578a9d74e in virCommandRun (cmd=0x7fd55c231f10, exitstatus=0x0)
     at util/vircommand.c:2100
 #10 0x00007fd56326fde5 in qemuProcessStart (conn=0x7fd53c000df0,
     driver=0x7fd55c0dc4f0, vm=0x7fd54800b100, migrateFrom=0x0, stdin_fd=-1,
     stdin_path=0x0, snapshot=0x0, vmop=VIR_NETDEV_VPORT_PROFILE_OP_CREATE,
     flags=1) at qemu/qemu_process.c:3694
 ...

The solution is to split the work of getpwuid_r/initgroups into the
unsafe portions (getgrouplist, called pre-fork) and safe portions
(setgroups, called post-fork).

* src/util/virutil.h (virSetUIDGID, virSetUIDGIDWithCaps): Adjust
signature.
* src/util/virutil.c (virSetUIDGID): Add parameters.
(virSetUIDGIDWithCaps): Adjust clients.
* src/util/vircommand.c (virExec): Likewise.
* src/util/virfile.c (virFileAccessibleAs, virFileOpenForked)
(virDirCreate): Likewise.
* src/security/security_dac.c (virSecurityDACSetProcessLabel):
Likewise.
* src/lxc/lxc_container.c (lxcContainerSetID): Likewise.
* configure.ac (AC_CHECK_FUNCS_ONCE): Check for setgroups, not
initgroups.
Signed-off-by: NEric Blake <eblake@redhat.com>