When dumping a domain, it's reasonable to save dump-file in raw format
if dump format is misconfigured or the corresponding compress program
is not available rather then fail dumping.

This patch introduces the usage of the pre-associate state of the IEEE 802.1Qbg standard on incoming VM migration on the target host. It is in response to bugzilla entry 632750.

https://bugzilla.redhat.com/show_bug.cgi?id=632750

For being able to differentiate the exact reason as to why a macvtap device is being created, either due to a VM creation or an incoming VM migration, I needed to pass that reason as a parameter from wherever qemudStartVMDaemon is being called in order to determine whether to send an ASSOCIATE (VM creation) or a PRE-ASSOCIATE (incoming VM migration) towards lldpad.

I am also fixing a problem with the virsh domainxml-to-native call on the way.

Gerhard successfully tested the patch with a recent blade network 802.1Qbg-compliant switch.

The patch should not have any side-effects on the 802.1Qbh support in libvirt, but Roopa (cc'ed) may want to verify this.

Don't overwrite errors during domain creation/definition to ease
tracking down problems.

We currently use the next free veid although there's one given in the
domain xml. This currently breaks defining new domains since vmdef->name
and veid don't match leading to the following error later on:

    error: Failed to define domain from 110.xml
    error: internal error Could not set UUID

Since silently ignoring vmdef->name is not nice respect it instead. We
avoid veid collisions in the upper levels already.

This reverts commit

 Log all errors at level INFO to stop polluting syslog
 04bd0360.

and makes virRaiseErrorFull() log errors at debug priority
when called from inside libvirtd. This stops libvirtd from
polluting it's own log with client errors at error priority
that'll be reported and logged on the client side anyway.

When we set migrate_speed by json, we receive the following
error message:
libvirtError: internal error unable to execute QEMU command
'migrate_set_speed': Invalid parameter type, expected: number

The reason is that: the arguments of migrate_set_speed
by json is json number, not json string.
Signed-off-by: NWen Congyang <wency@cn.fujitsu.com>

* src/util/virtaudit.c: Free audit string
* src/vbox/vbox_XPCOMCGlue.c: Free library name

The nodeinfo structure includes

    nodes   : the number of NUMA cell, 1 for uniform mem access
    sockets : number of CPU socket per node
    cores   : number of core per socket
    threads : number of threads per core

which does not work well for NUMA topologies where each node does not
consist of integral number of CPU sockets.

We also have VIR_NODEINFO_MAXCPUS macro in public libvirt.h which
computes maximum number of CPUs as (nodes * sockets * cores * threads).

As a result, we can't just change sockets to report total number of
sockets instead of sockets per node. This would probably be the easiest
since I doubt anyone is using the field directly. But because of the
macro, some apps might be using sockets indirectly.

This patch leaves sockets to be the number of CPU sockets per node (and
fixes qemu driver to comply with this) on machines where sockets can be
divided by nodes. If we can't divide sockets by nodes, we behave as if
there was just one NUMA node containing all sockets. Apps interested in
NUMA should consult capabilities XML, which is what they probably do
anyway.

This way, the only case in which apps that care about NUMA may break is
on machines with funky NUMA topology. And there is a chance libvirt
wasn't able to start any guests on those machines anyway (although it
depends on the topology, total number of CPUs and kernel version).
Nothing changes at all for apps that don't care about NUMA.

security_context_t happens to be a typedef for char*, and happens to
begin with a string usable as a raw context string.  But in reality,
it is an opaque type that may or may not have additional information
after the first NUL byte, where that additional information can
include pointers that can only be freed via freecon().

Proof is from this valgrind run of daemon/libvirtd:

==6028== 839,169 (40 direct, 839,129 indirect) bytes in 1 blocks are definitely lost in loss record 274 of 274
==6028==    at 0x4A0515D: malloc (vg_replace_malloc.c:195)
==6028==    by 0x3022E0D48C: selabel_open (label.c:165)
==6028==    by 0x3022E11646: matchpathcon_init_prefix (matchpathcon.c:296)
==6028==    by 0x3022E1190D: matchpathcon (matchpathcon.c:317)
==6028==    by 0x4F9D842: SELinuxRestoreSecurityFileLabel (security_selinux.c:382)

800k is a lot of memory to be leaking.

* src/storage/storage_backend.c
(virStorageBackendUpdateVolTargetInfoFD): Avoid leak on error.
* src/security/security_selinux.c
(SELinuxReserveSecurityLabel, SELinuxGetSecurityProcessLabel)
(SELinuxRestoreSecurityFileLabel): Use correct function to free
security_context_t.

* src/conf/network_conf.c (virNetworkDHCPRangeDefParseXML): Free
xml strings when no longer referenced.

Making this change makes it easier to spot the memory leaks
that will be fixed in the next patch.

* cfg.mk (sc_prohibit_xmlGetProp): New rule.
* .x-sc_prohibit_xmlGetProp: New exception.
* Makefile.am (EXTRA_DIST): Ship exception file.
* tools/virsh.c (cmdDetachInterface, cmdDetachDisk): Adjust
offenders.
* src/conf/storage_conf.c (virStoragePoolDefParseSource):
Likewise.
* src/conf/network_conf.c (virNetworkDHCPRangeDefParseXML)
(virNetworkIPParseXML): Likewise.

https://bugzilla.redhat.com/show_bug.cgi?id=656795

* src/qemu/qemu_monitor.c (qemuMonitorFree): Also free the buffer.

* cfg.mk (sc_prohibit_close): New syntax-check rule.
* src/util/pci.c (pciWaitForDeviceCleanup): Fix violation.
* .x-sc_prohibit_close: New exceptions.
* Makefile.am (EXTRA_DIST): Distribute new file.

virConnectClose calls virUnrefConnect which in turn closes
all open drivers when the refcount of that connection dropped
to zero. This works fine when you free all other objects that
hold a ref to the connection before you close it, because in
this case virUnrefConnect is the one that removes the last
ref to the connection.

But it doesn't work when you close the connection first before
freeing the other objects. This is because the other virUnref*
functions call virReleaseConnect when they detect that the
connection's refcount dropped to zero. In this case another
virUnref* function (different from virUnrefConnect) removes the
last ref to the connection. This results in not closing the
open drivers and leaking things that should have been cleaned
up in the driver close functions.

To fix this move the driver close calls to virReleaseConnect.

Except LXC and UML driver, implementations of all other drivers
simply return 0, because these drivers doesn't have config both
in memory and on disk, no need to track if the domain of these
drivers updated or not.

Rename "xenUnifiedDomainisPersistent" to "xenUnifiedDomainIsPersistent"

* esx/esx_driver.c
* lxc/lxc_driver.c
* opennebula/one_driver.c
* openvz/openvz_driver.c
* phyp/phyp_driver.c
* test/test_driver.c
* uml/uml_driver.c
* vbox/vbox_tmpl.c
* xen/xen_driver.c
* xenapi/xenapi_driver.c

* src/qemu/qemu_driver.c (add function qemuDomainIsUpdated)

* daemon/remote.c
* daemon/remote_dispatch_args.h
* daemon/remote_dispatch_prototypes.h
* daemon/remote_dispatch_ret.h
* daemon/remote_dispatch_table.h
* src/remote/remote_driver.c
* src/remote/remote_protocol.c
* src/remote/remote_protocol.h
* src/remote/remote_protocol.x
* src/remote_protocol-structs

* src/libvirt.c

* src/driver.h (new typedef, new callback member for "_virDriver")
* src/esx/esx_driver.c
* src/lxc/lxc_driver.c
* src/opennebula/one_driver.c
* src/openvz/openvz_driver.c
* src/phyp/phyp_driver.c
* src/qemu/qemu_driver.c
* src/remote/remote_driver.c
* src/test/test_driver.c
* src/uml/uml_driver.c
* src/vbox/vbox_tmpl.c
* src/xen/xen_driver.c
* src/xenapi/xenapi_driver.c

introduce new public API "virDomainIsUpdated"

* src/conf/domain_conf.h (new member "updated" for "virDomainObj")
* src/libvirt_public.syms
* include/libvirt/libvirt.h.in

* src/libvirt_private.syms (buf.h): Sort.

gnulib wraps Windows' SOCKET handle based send() and recv() functions
into file descriptor based ones that are used in libvirt.

Even though GnuTLS is using gnulib too, it explicitly doesn't use
gnulib's replacement functions on Windows. By default GnuTLS uses the
SOCKET handle based send() and recv(). This makes gnutls_handshake()
fail internally with a WSAENOTSOCK error because libvirt passes a
file descriptor; GnuTLS needs the SOCKET handle.

To avoid this mismatch make sure that GnuTLS uses gnulib's replacment
functions, by setting custom pull() and push() functions for GnuTLS.

The stdio.h header has a function called 'remove' declared. This
clashes with the 'remove' parameter in virShrinkN

* src/util/memory.c: Rename 'remove' to 'toremove'

* src/util/memory.c: Avoid 32-bit truncation extracting a 64bit int

The SCSI volumes currently get a name like '17:0:0:1' based
on $host:$bus:$target:$lun. The names are intended to be unique
per pool and stable across pool restarts. The inclusion of the
$host component breaks this, because the $host number for iSCSI
pools is dynamically allocated by the kernel at time of login.
This changes the name to be 'unit:0:0:1', ie removes the leading
host component. The 'unit:' prefix is just to ensure the volume
name doesn't start with a number and make it clearer when seen
out of context.

* src/storage/storage_backend_scsi.c: Improve volume name
  field value stability and uniqueness

Many operations are not valid on inactive storage pools. The
storage driver is currently returning VIR_ERR_INTERNAL_ERROR
in these cases, rather than the more suitable error code
VIR_ERR_OPERATION_INVALID

* src/storage/storage_driver.c: Fix error code when pool
  is not active

When libvirt starts up all storage pools default to the inactive
state, even if the underlying storage is already active on the
host. This introduces a new API into the internal storage backend
drivers that checks whether a storage pool is already active. If
the pool is active at libvirtd startup, the volume list will be
immediately populated.

* src/storage/storage_backend.h: New internal API for checking
  storage pool state
* src/storage/storage_driver.c: Check whether a pool is active
  upon driver startup
* src/storage/storage_backend_fs.c, src/storage/storage_backend_iscsi.c,
  src/storage/storage_backend_logical.c, src/storage/storage_backend_mpath.c,
  src/storage/storage_backend_scsi.c: Add checks for pool state

Since the previous patch added support for parsing the output of
the 'sendtargets' command, it is now trivial to support the
storage pool discovery API.

Given a hostname and optional portnumber and initiator IQN,
the code can return a full list of storage pool source docs,
each one representing a iSCSI target.

* src/storage/storage_backend_iscsi.c: Wire up target
  auto-discovery

The Linux iSCSI initiator toolchain has the dubious feature that
if you ever run the 'sendtargets' command to merely query what
targets are available from a server, the results will be recorded
in /var/lib/iscsi. Any time the '/etc/init.d/iscsi' script runs
in the future, it will then automatically login to all those
targets. /etc/init.d/iscsi is automatically run whenever a NIC
comes online.

So from the moment you ask a server what targets are available,
your client will forever more automatically try to login to all
targets without ever asking if you actually want it todo this.

To stop this stupid behaviour, we need to run

  iscsiadm --portal $PORTAL --target $TARGET
   --op update --name node.startup --value manual

For every target on the server.

* src/storage/storage_backend_iscsi.c: Disable automatic login
  for targets found as a result of a 'sendtargets' command

The following series of patches are adding significant
extra functionality to the iSCSI driver. THe current
internal helper methods are not sufficiently flexible
to cope with these changes. This patch refactors the
code to avoid needing to have a virStoragePoolObjPtr
instance as a parameter, instead passing individual
target, portal and initiatoriqn parameters.

It also removes hardcoding of port 3260 in the portal
address, instead using the XML value if any.

* src/storage/storage_backend_iscsi.c: Refactor internal
  helper methods

The XML docs describe a 'port' attribute for the
storage source <host> element, but the parser never
handled it.

* docs/schemas/storagepool.rng: Define port attribute
* src/conf/storage_conf.c: Add missing parsing/formatting
  of host port number
* src/conf/storage_conf.h: Remove bogus/unused 'protocol' field

When running non-root, the QEMU log file is usually opened with
truncation, since there is no logrotate for non-root usage.
This means that when libvirt logs the shutdown timestamp, the
log is accidentally truncated

* src/qemu/qemu_driver.c: Never truncate log file with shutdown
  message

The QEMU logger appends a ':' to the timestamp when it deems
it neccessary, so the virTimestamp API should not duplicate
this

* src/util/util.c: Remove trailing ':' from timestamp

Everytime a public API returns an error, libvirtd pollutes
syslog with that error message. Reduce the error logging
level to INFO so these don't appear by default.

* src/util/virterror.c: Log all errors at INFO

The virFork call resets all logging handlers that may have been
set. Re-enable them after fork in virExec, so that env variables
fir LIBVIRT_LOG_OUTPUTS and LIBVIRT_LOG_FILTERS take effect
until the execve()

* src/util/util.c: Preserve logging in child in virExec

To allow messages from different threads to be untangled,
include an integer thread identifier in log messages.

* src/util/logging.c: Include thread ID
* src/util/threads.h, src/util/threads.h, src/util/threads-pthread.c:
  Add new virThreadSelfID() function
* configure.ac: Check for sys/syscall.h

Currently changes to the persistent config aren't flushed to disk, meaning
they are lost if the domain is redefined or libvirtd is restarted.

Do this by adding a helper function to get the persistent domain config. This
should be useful for other functions that may eventually want to alter
the persistent domain config (attach/detach device). Also make similar changes
to the test drivers setvcpus command.

A caveat is that the function will return the running config for a transient
domain, rather than error. This simplifies callers, as long as they use
other methods to ensure the guest is persistent.