In a first step I am converting the netlink message construction in
macvtap code to use libnl. It's pretty much a 1:1 conversion except that
now the message needs to be allocated and deallocated.

The util/threads.c/h code already has APIs for mutexes,
condition variables and thread locals. This commit adds
in code for actually creating threads.

* src/libvirt_private.syms: Export new symbols
* src/util/threads.h: Define APIs virThreadCreate, virThreadSelf,
  virThreadIsSelf and virThreadJoin
* src/util/threads-win32.c, src/util/threads-win32.h: Win32
  impl of threads
* src/util/threads-pthread.c, src/util/threads-pthread.h: POSIX
  impl of threads

To avoid the need for duplicating implementations of virStream
drivers, provide a generic implementation that can handle any
FD based stream. This code is copied from the existing impl
in the QEMU driver, with the locking moved into the stream
impl, and addition of a read callback

The FD stream code will refuse to operate on regular files or
block devices, since those can't report EAGAIN properly when
they would block on I/O

* include/libvirt/virterror.h, include/libvirt/virterror.h: Add
  VIR_FROM_STREAM error domain
* src/qemu/qemu_driver.c: Remove code obsoleted by the new
  generic streams driver.
* src/fdstream.h, src/fdstream.c, src/fdstream.c,
  src/libvirt_private.syms: Generic reusable FD based streams

I am trying to use a qcow image with libvirt where the backing 'file' is a
qemu-nbd server.  Unfortunately virDomainDiskDefForeachPath() assumes that
backingStore is always a real file so something like 'nbd:0:3333' is rejected
because a file with that name cannot be accessed.  Note that I am not worried
about directly using nbd images.  That would require a new disk type with XML
markup, etc.  I only want it to be permitted as a backingStore

The following patch implements danpb's suggestion:
> I think I'm inclined to push the logic for skipping NBD one stage higher.
> I'd rather expect virStorageFileGetMetadata() to return all backing
> stores, even if not files. The virDomainDiskDefForeachPath() method
> should definitely ignore non-file backing stores though.
>
> So what I'm thinking is to extend the virStorageFileMetadata struct and
> just add a 'bool isFile' field to it. Default this field to true, unless
> you see the prefix of nbd: in which case set it to false. The
> virDomainDiskDefForeachPath() method can then skip over any backing
> store with isFile == false
Signed-off-by: NAdam Litke <agl@us.ibm.com>
Cc: Daniel P. Berrange <berrange@redhat.com>

Using automated replacement with sed and editing I have now replaced all
occurrences of close() with VIR_(FORCE_)CLOSE() except for one, of
course. Some replacements were straight forward, others I needed to pay
attention. I hope I payed attention in all the right places... Please
have a look. This should have at least solved one more double-close
error.

Suggested by danpb, as it's not up-to-date anymore and
lacks many functions that were added to libvirtd.

Move existing routines about virSysinfoDef to an util module,
add a new entry point virSysinfoRead() to read the host values
with dmidecode

* src/conf/domain_conf.c src/conf/domain_conf.h src/util/sysinfo.c
  src/util/sysinfo.h: move to a new module, add virSysinfoRead()
* src/Makefile.am: handle the new module build
* src/libvirt_private.syms: new internal symbols
* include/libvirt/virterror.h src/util/virterror.c: defined a new
  error code for that module
* po/POTFILES.in: add new file for translations

virPipeReadUntilEOF is used to read the stdout of exec'ed
and this could fail to capture the full output and read only
1024 bytes.
  The problem is that this is based on a poll loop, and in the
loop we read at most 1024 bytes per file descriptor, but we also
note in the loop if poll indicates that the process won't output
more than that on that fd by setting finished[i] = 1.
  The simplest way is that if we read a full buffer make sure
finished[i] is still 0 because we will need another pass in the
loop.

* src/util/virterror.c (virErrorDomainName): Always supply
trailing space if category was present.

NFS does not support file labelling, so ignore this error
for stdin_path when on NFS.

* src/security/security_selinux.c: Ignore failures on labelling
  stdin_path on NFS
* src/util/storage_file.c, src/util/storage_file.h: Refine
  virStorageFileIsSharedFS() to allow it to check for a
  specific FS type.

They only popped up during --disable-nls build. Without this configure
option, gcc wasn't able to detect them.

* src/util/util.h
* src/util/util.c
* src/libvirt_private.syms

When we mount any cgroup without "-o devices", we will fail to start vms:

error: Failed to start domain vm1
error: Unable to deny all devices for vm1: No such file or directory

When we mount any cgroup without "-o cpu", we will fail to get schedinfo:
Scheduler      : posix
error: unable to get cpu shares tunable: No such file or directory

We should only use the cgroup controllers which are mounted on host.
So I add virCgroupMounted() for qemuCgroupControllerActive()
Signed-off-by: NLai Jiangshan <laijs@cn.fujitsu.com>

fix warning
  CC     libvirt_util_la-virtaudit.lo
cc1: warnings being treated as errors
util/virtaudit.c: In function 'virAuditEncode':
util/virtaudit.c:146: error: implicit declaration of function 'virAsprintf' [-Wimplicit-function-declaration]
util/virtaudit.c:146: error: nested extern declaration of 'virAsprintf' [-Wnested-externs]

Commit 9bd3cce0 added virFork and
virDriverLoadModule to libvirt_private.syms, but virFork didn't have
a body on Win32 and virDriverLoadModule was already correctly
exported conditional via libvirt_driver_modules.syms.

* src/util/virtaudit.h: Add printf format attribute annotation

Add a helper API for ecscaping the value in audit log
messages

* src/util/virtaudit.h, src/util/virtaudit.c,
  src/libvirt_private.syms: Add virAuditEncode

The network address was being set to 192.168.122.0 instead
of 192.168.122.0/24. Fix this by removing the unneccessary
'network' field from virNetworkDef and just pass the
network address and netmask into the iptables APIs directly.

* src/conf/network_conf.h, src/conf/network_conf.c: Remove
  the 'network' field from virNEtworkDef.
* src/network/bridge_driver.c: Update for iptables API changes
* src/util/iptables.c, src/util/iptables.h: Require the
  network address + netmask pair to be passed in

* src/util/event.h: Fix copy&paste error from virEventAddHandle()

* src/util/dnsmasq.c (hostsfileAdd): Don't free uninitialized
memory on allocation failure.

Instead of storing the IP address string in virNetwork related
structs, store the parsed virSocketAddr. This will make it
easier to add IPv6 support in the future, by letting driver
code directly check what address family is present

* src/conf/network_conf.c, src/conf/network_conf.h,
  src/network/bridge_driver.c: Convert to use virSocketAddr
  in virNetwork, instead of char *.
* src/util/bridge.c, src/util/bridge.h,
  src/util/dnsmasq.c, src/util/dnsmasq.h,
  src/util/iptables.c, src/util/iptables.h: Convert to
  take a virSocketAddr instead of char * for any IP
  address parameters
* src/util/network.h: Add macros to determine if an address
  is set, and what address family is set.

It is useful to know where the client is connecting from,
so include the socket address in probe data.

* daemon/libvirtd.h: Use virSocketAddr for storing client
  address and keep printable address handy for logging
* daemon/libvirtd.c: Include socket address in client
  connect/disconnect probes
* daemon/probes.d: Add socket address to probes
* examples/systemtap/client.stp: Print socket address
* src/util/network.h: Add sockaddr_un to virSocketAddr union

The  inet_pton and inet_ntop functions are obsolete, replaced
by getaddrinfo+getnameinfo with the AI_NUMERICHOST flag set.
These can be accessed via the virSocket APIs.

The bridge.c code had methods for fetching the IP address of
a bridge which used inet_ntop. Aside from the use of inet_ntop
these methods are broken, because a NIC can have multiple
addresses and this only returns one address. Since the methods
are never used, just remove them.

* src/conf/network_conf.c, src/nwfilter/nwfilter_learnipaddr.c:
  Replace inet_pton and inet_ntop with virSocket APIs
* src/util/bridge.c, src/util/bridge.h: Remove unused methods
  which called inet_ntop.

The virSocketParse method was not doing any error reporting
which meant the true cause of the problem was lost. Remove
all error reporting from callers, and push it into virSocketParse

* src/util/network.c: Add error reporting to virSocketParse
* src/conf/domain_conf.c, src/conf/network_conf.c,
  src/network/bridge_driver.c: Remove error reporting in
  callers of virSocketParse

The getnameinfo() function is more flexible than inet_ntop()
avoiding the need to if/else the code based on socket family.
Also make it support UNIX socket addrs and allow inclusion
of a port (service) address. Finally do proper error reporting
via normal APIs.

* src/conf/domain_conf.c, src/nwfilter/nwfilter_ebiptables_driver.c,
  src/qemu/qemu_conf.c: Fix error handling with virSocketFormat
* src/util/network.c: Rewrite virSocketFormat to use getnameinfo
  and cope with UNIX socket addrs.

There was a typo in the IPv6 path of virSocketCheckNetmask which
caused it to never execute.

* src/util/network.c: s/AF_INET/AF_INET6/ in virSocketCheckNetmask

The virSocketParseAddr function was accepting any AF_* constant
and using that to set the ai_flags field in struct addrinfo.
This is invalid, since address families must go in the ai_family
field of the struct.

* src/util/network.c: Fix handling of address family
* src/conf/network_conf.c, src/network/bridge_driver.c: Pass
  AF_UNSPEC instead of relying on it being 0.

Some operations on socket addresses need to know the length of
the sockaddr struct for the particular address family. This
info was being discarded when passing around virSocketAddr
instances. Turn it from a union into a struct containing
union+socklen_t fields, so length is always kept around.

* src/util/network.h: Add socklen_t field to virSocketAddr
* src/util/network.c, src/network/bridge_driver.c,
  src/conf/domain_conf.c: Update to take account of new
  struct definition.

* src/util/virtaudit.c (virAuditSend): one less ifdef, since gcc
does not care if an ATTRIBUTE_UNUSED var gets used in some paths.

Integrate with libaudit.so for auditing of important operations.
libvirtd gains a couple of config entries for auditing. By
default it will enable auditing, if its enabled on the host.
It can be configured to force exit if auditing is disabled
on the host. It will can also send audit messages via libvirt
internal logging API

Places requiring audit reporting can use the VIR_AUDIT
macro to report data. This is a no-op unless auditing is
enabled

* autobuild.sh, mingw32-libvirt.spec.in: Disable audit
  on mingw
* configure.ac: Add check for libaudit
* daemon/libvirtd.aug, daemon/libvirtd.conf,
  daemon/test_libvirtd.aug, daemon/libvirtd.c: Add config
  options to enable auditing
* include/libvirt/virterror.h, src/util/virterror.c: Add
  VIR_FROM_AUDIT source
* libvirt.spec.in: Enable audit
* src/util/virtaudit.h, src/util/virtaudit.c: Simple internal
  API for auditing messages

Since bugs due to double-closed file descriptors are difficult to track down in a multi-threaded system, I am introducing the VIR_CLOSE(fd) macro to help avoid mistakes here.

There are lots of places where close() is being used. In this patch I am only cleaning up usage of close() in src/conf where the problems were.

I also dare to declare close() as being deprecated in libvirt code base (HACKING).

* include/libvirt/libvirt.h.in: some of the function type description
  were broken so they could not be automatically documented
* src/util/event.c docs/apibuild.py: event.c exports one public API
  so it needs to be scanned too, avoid a few warnings

Provides interfaces for setting/getting memory tunables like hard_limit,
soft_limit and swap_hard_limit

* .gnulib: Update to latest.
* bootstrap.conf (gnulib_modules): Add new termios module.
(gnulib_tool_option_extras): Make libtool usage explicit.
* src/util/util.c (includes): Gnulib now guarantees termios.h.
* bootstrap: Resync from gnulib.

Add a test for this.

Reported by Frank Dirks.

pciFindStubDriver currently returns 0 in one of the error cases.
While it's correct...NULL is more readable.
Signed-off-by: NChris Wright <chrisw@redhat.com>

* src/util/network.c (getIPv6Addr): Manually join s6_addr bytes,
instead of assuming s6_addr16 shorts.
Reported by Justin Clifton; solution suggested by Bruno Haible.

The current code will go into an infinite loop if the printf generated
string is >= 1000, AND exactly 1 character smaller than the amount of free
space in the buffer. When this happens, we are dropped into the loop body,
but nothing will actually change, because count == (buf->size - buf->use - 1),
and virBufferGrow returns unchanged if count < (buf->size - buf->use)

Fix this by removing the '- 1' bit from 'size'. The *nprintf functions handle
the NULL byte for us anyways, so we shouldn't need to manually accommodate
for it.

Here's a bug where we are actually hitting this issue:
https://bugzilla.redhat.com/show_bug.cgi?id=602772

v2: Eric's improvements: while -> if (), remove extra va_list variable,
    make sure we report buffer error if snprintf fails

v3: Add tests/virbuftest which reproduces the infinite loop before this
    patch, works correctly after

virHexToBin will be used in the .vmx handling code.