Based on a report by Chandrashekar Shastri, at
https://bugzilla.redhat.com/show_bug.cgi?id=979360

On systems where git cannot access the outside world, a developer
can instead arrange to get a copy of gnulib at the right commit
via side channels (such as NFS share drives), set GNULIB_SRCDIR,
then use ./autogen.sh --no-git.  In this setup, we will now
avoid direct use of git.  Of course, this means no automatic
gnulib updates when libvirt.git updates its submodule, but it
is expected that any developer in such a situation is already
prepared to deal with the fallout.

* .gnulib: Update to latest, for bootstrap.
* bootstrap: Synchronize from gnulib.
* autogen.sh (no_git): Avoid git when requested.
* cfg.mk (_update_required): Skip automatic rerun of bootstrap if
we can't use git.
* docs/compiling.html.in: Document this setup.
* docs/hacking.html.in: Mention this.
* HACKING: Regenerate.
Signed-off-by: NEric Blake <eblake@redhat.com>
(cherry picked from commit 1e503ee5)

The latest mingw headers on Fedora 19 fail to build with gnulib
without an update.

Meanwhile, now that upstream gnulib has better handling of -W
probing for clang, we can drop some of our own solutions in
favor of upstream; thus this reverts commit c1634100, "Correctly
detect warning flags with clang".

* .gnulib: Update to latest, for mingw and clang.
Signed-off-by: NEric Blake <eblake@redhat.com>
(cherry picked from commit cdd703f4)

FreeBSD ships an old gcc 4.2.1 which generates
bogus code, e.g. getsockopt() call returns
struct xucred with bogus values, which doesn't even
allow to connect to libvirtd:

error: Failed to find group record for gid '1284660778': No error: 0

So roll back to just -fstack-protector on FreeBSD.

(cherry picked from commit cc7cd623)

This picks up a fix for a syntax-check weakness mentioned here:
https://www.redhat.com/archives/libvir-list/2013-May/msg00811.html

* .gnulib: Update to latest, for maint.mk improvement.
Signed-off-by: NEric Blake <eblake@redhat.com>
(cherry picked from commit 12bd22c7)

Among others, this fixes a cosmetic bug where bootstrap stated:

./bootstrap: Bootstrapping from checked-out http://libvirt.org sources...

instead of the intended:

./bootstrap: Bootstrapping from checked-out libvirt sources...

* .gnulib: Update to latest, for bootstrap improvement.
* bootstrap: Resync from gnulib.

(cherry picked from commit 3dfc2b71)

https://bugzilla.redhat.com/show_bug.cgi?id=982317

maint-only patch; reported by Geert Jansen

Commit 17cdc298 tried to backport upstream 90a0c6d, but in
resolving conflicts, failed to account that upstream commit
e1d32bb9 refactored code to leave off a leading /dev.

* src/lxc/lxc_container.c (lxcContainerPopulateDevices): Use
correct device name.
Signed-off-by: NEric Blake <eblake@redhat.com>

https://bugzilla.redhat.com/show_bug.cgi?id=964358

Attempts to start a domain with both SELinux and DAC security
modules loaded will deadlock; latent problem introduced in commit
fdb3bde3 and exposed in commit 29fe5d74.  Basically, when recursing
into the security manager for other driver's prefork, we have to
undo the asymmetric lock taken at the manager level.

Reported by Jiri Denemark, with diagnosis help from Dan Berrange.

* src/security/security_stack.c (virSecurityStackPreFork): Undo
extra lock grabbed during recursion.
Signed-off-by: NEric Blake <eblake@redhat.com>
(cherry picked from commit bfc183c1)

https://bugzilla.redhat.com/show_bug.cgi?id=964358

Commit 75c12564 states that virGetGroupList must not be called
between fork and exec, then commit ee777e99 promptly violated
that for lxc's use of virSecurityManagerSetProcessLabel.  Hoist
the supplemental group detection to the time that the security
manager needs to fork.  Qemu is safe, as it uses
virSecurityManagerSetChildProcessLabel which in turn uses
virCommand to determine supplemental groups.

This does not fix the fact that virSecurityManagerSetProcessLabel
calls virSecurityDACParseIds calls parseIds which eventually
calls getpwnam_r, which also violates fork/exec async-signal-safe
safety rules, but so far no one has complained of hitting
deadlock in that case.

* src/security/security_dac.c (_virSecurityDACData): Track groups
in private data.
(virSecurityDACPreFork): New function, to set them.
(virSecurityDACClose): Clean up new fields.
(virSecurityDACGetIds): Alter signature.
(virSecurityDACSetSecurityHostdevLabelHelper)
(virSecurityDACSetChardevLabel, virSecurityDACSetProcessLabel)
(virSecurityDACSetChildProcessLabel): Update callers.
Signed-off-by: NEric Blake <eblake@redhat.com>
(cherry picked from commit 29fe5d74)

Conflicts:
	src/security/security_dac.c - virSecurityDACSetSecurityUSBLabel needed similar treatment

https://bugzilla.redhat.com/show_bug.cgi?id=964358

A future patch wants the DAC security manager to be able to safely
get the supplemental group list for a given uid, but at the time
of a fork rather than during initialization so as to pick up on
live changes to the system's group database.  This patch adds the
framework, including the possibility of a pre-fork callback
failing.

For now, any driver that implements a prefork callback must be
robust against the possibility of being part of a security stack
where a later element in the chain fails prefork.  This means
that drivers cannot do any action that requires a call to postfork
for proper cleanup (no grabbing a mutex, for example).  If this
is too prohibitive in the future, we would have to switch to a
transactioning sequence, where each driver has (up to) 3 callbacks:
PreForkPrepare, PreForkCommit, and PreForkAbort, to either clean
up or commit changes made during prepare.

* src/security/security_driver.h (virSecurityDriverPreFork): New
callback.
* src/security/security_manager.h (virSecurityManagerPreFork):
Change signature.
* src/security/security_manager.c (virSecurityManagerPreFork):
Optionally call into driver, and allow returning failure.
* src/security/security_stack.c (virSecurityDriverStack):
Wrap the handler for the stack driver.
* src/qemu/qemu_process.c (qemuProcessStart): Adjust caller.
Signed-off-by: NEric Blake <eblake@redhat.com>
(cherry picked from commit fdb3bde3)

https://bugzilla.redhat.com/show_bug.cgi?id=964358

POSIX states that multi-threaded apps should not use functions
that are not async-signal-safe between fork and exec, yet we
were using getpwuid_r and initgroups.  Although rare, it is
possible to hit deadlock in the child, when it tries to grab
a mutex that was already held by another thread in the parent.
I actually hit this deadlock when testing multiple domains
being started in parallel with a command hook, with the following
backtrace in the child:

 Thread 1 (Thread 0x7fd56bbf2700 (LWP 3212)):
 #0  __lll_lock_wait ()
     at ../nptl/sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:136
 #1  0x00007fd5761e7388 in _L_lock_854 () from /lib64/libpthread.so.0
 #2  0x00007fd5761e7257 in __pthread_mutex_lock (mutex=0x7fd56be00360)
     at pthread_mutex_lock.c:61
 #3  0x00007fd56bbf9fc5 in _nss_files_getpwuid_r (uid=0, result=0x7fd56bbf0c70,
     buffer=0x7fd55c2a65f0 "", buflen=1024, errnop=0x7fd56bbf25b8)
     at nss_files/files-pwd.c:40
 #4  0x00007fd575aeff1d in __getpwuid_r (uid=0, resbuf=0x7fd56bbf0c70,
     buffer=0x7fd55c2a65f0 "", buflen=1024, result=0x7fd56bbf0cb0)
     at ../nss/getXXbyYY_r.c:253
 #5  0x00007fd578aebafc in virSetUIDGID (uid=0, gid=0) at util/virutil.c:1031
 #6  0x00007fd578aebf43 in virSetUIDGIDWithCaps (uid=0, gid=0, capBits=0,
     clearExistingCaps=true) at util/virutil.c:1388
 #7  0x00007fd578a9a20b in virExec (cmd=0x7fd55c231f10) at util/vircommand.c:654
 #8  0x00007fd578a9dfa2 in virCommandRunAsync (cmd=0x7fd55c231f10, pid=0x0)
     at util/vircommand.c:2247
 #9  0x00007fd578a9d74e in virCommandRun (cmd=0x7fd55c231f10, exitstatus=0x0)
     at util/vircommand.c:2100
 #10 0x00007fd56326fde5 in qemuProcessStart (conn=0x7fd53c000df0,
     driver=0x7fd55c0dc4f0, vm=0x7fd54800b100, migrateFrom=0x0, stdin_fd=-1,
     stdin_path=0x0, snapshot=0x0, vmop=VIR_NETDEV_VPORT_PROFILE_OP_CREATE,
     flags=1) at qemu/qemu_process.c:3694
 ...

The solution is to split the work of getpwuid_r/initgroups into the
unsafe portions (getgrouplist, called pre-fork) and safe portions
(setgroups, called post-fork).

* src/util/virutil.h (virSetUIDGID, virSetUIDGIDWithCaps): Adjust
signature.
* src/util/virutil.c (virSetUIDGID): Add parameters.
(virSetUIDGIDWithCaps): Adjust clients.
* src/util/vircommand.c (virExec): Likewise.
* src/util/virfile.c (virFileAccessibleAs, virFileOpenForked)
(virDirCreate): Likewise.
* src/security/security_dac.c (virSecurityDACSetProcessLabel):
Likewise.
* src/lxc/lxc_container.c (lxcContainerSetID): Likewise.
* configure.ac (AC_CHECK_FUNCS_ONCE): Check for setgroups, not
initgroups.
Signed-off-by: NEric Blake <eblake@redhat.com>
(cherry picked from commit ee777e99)

Conflicts:
	src/lxc/lxc_container.c - did not use setUIDGID before 1.1.0
	src/util/virutil.c - oom handling changes not backported
	src/util/virfile.c - functions still lived in virutil.c this far back
	configure.ac - context with previous commit

https://bugzilla.redhat.com/show_bug.cgi?id=964358

Since neither getpwuid_r() nor initgroups() are safe to call in
between fork and exec (they obtain a mutex, but if some other
thread in the parent also held the mutex at the time of the fork,
the child will deadlock), we have to split out the functionality
that is unsafe.  At least glibc's initgroups() uses getgrouplist
under the hood, so the ideal split is to expose getgrouplist for
use before a fork.  Gnulib already gives us a nice wrapper via
mgetgroups; we wrap it once more to look up by uid instead of name.

* bootstrap.conf (gnulib_modules): Add mgetgroups.
* src/util/virutil.h (virGetGroupList): New declaration.
* src/util/virutil.c (virGetGroupList): New function.
* src/libvirt_private.syms (virutil.h): Export it.
Signed-off-by: NEric Blake <eblake@redhat.com>
(cherry picked from commit 75c12564)

Conflicts:
	bootstrap.conf - not updating gnulib submodule...
	configure.ac - ...so checking for getgrouplist by hand...
	src/util/virutil.c - ...and copying only the getgrouplist implementation rather than calling the gnulib function

https://bugzilla.redhat.com/show_bug.cgi?id=964358

A future patch needs to look up pw_gid; but it is wasteful
to crawl through getpwuid_r twice for two separate pieces
of information, and annoying to copy that much boilerplate
code for doing the crawl.  The current internal-only
virGetUserEnt is also a rather awkward interface; it's easier
to just design it to let callers request multiple pieces of
data as needed from one traversal.

And while at it, I noticed that virGetXDGDirectory could deref
NULL if the getpwuid_r lookup fails.

* src/util/virutil.c (virGetUserEnt): Alter signature.
(virGetUserDirectory, virGetXDGDirectory, virGetUserName): Adjust
callers.
Signed-off-by: NEric Blake <eblake@redhat.com>
(cherry picked from commit c1983ba4)

Conflicts:
	src/util/virutil.c - oom reporting/strdup changes not backported

Remove assignment of the string freed by virURIFree
to hostname, since it's not used anywhere.

Double free introduced by ddf8ad82, useless code
introduced by f03dcc5d.

https://bugzilla.redhat.com/show_bug.cgi?id=977961
(cherry picked from commit 5744d96f)

This fixes https://bugzilla.redhat.com/show_bug.cgi?id=971325

The problem was that if virPCIGetVirtualFunctions was given the name
of a non-existent interface, it would return to its caller without
initializing the pointer to the array of virtual functions to NULL,
and the caller (virNetDevGetVirtualFunctions) would try to VIR_FREE()
the invalid pointer.

The final error message before the crash would be:

 virPCIGetVirtualFunctions:2088 :
  Failed to open dir '/sys/class/net/eth2/device':
  No such file or directory

In this patch I move the initialization in virPCIGetVirtualFunctions()
to the begining of the function, and also do an explicit
initialization in virNetDevGetVirtualFunctions, just in case someone
in the future adds code into that function prior to the call to
virPCIGetVirtualFunctions.

(cherry picked from commit 2c2525ab)

We can only pass values up to LLONG_MAX through JSON
and QEMU checks if the int64_t number is not negative
at startup since 1.5.0.

https://bugzilla.redhat.com/show_bug.cgi?id=974010
(cherry picked from commit d3c87884)

This fixes the problem reported in:

   https://bugzilla.redhat.com/show_bug.cgi?id=972690

When checking for a collision of a new libvirt network's subnet with
any existing routes, we read all of /proc/net/route into memory, then
parse all the entries. The function that we use to read this file
requires a "maximum length" parameter, which had previously been set
to 64*1024. As each line in /proc/net/route is 128 bytes, this would
allow for a maximum of 512 entries in the routing table.

This patch increases that number to 128 * 100000, which allows for
100,000 routing table entries. This means that it's possible that 12MB
would be allocated, but that would only happen if there really were
100,000 route table entries on the system, it's only held for a very
short time.

Since there is no method of specifying and unlimited max (and that
would create a potential denial of service anyway) hopefully this
limit is large enough to accomodate everyone.

(cherry picked from commit 2bdf548f)

Convert input XML to migratable before using it in
qemuDomainSaveImageOpen.

XML in the save image is migratable, i.e. doesn't contain implicit
controllers. If these controllers were in a non-default order in the
input XML, the ABI check would fail. Removing and re-adding these
controllers fixes it.

https://bugzilla.redhat.com/show_bug.cgi?id=834196
(cherry picked from commit 07966f6a)

https://bugzilla.redhat.com/show_bug.cgi?id=971485

As of d7f9d827 we copy the listen
address from the qemu.conf config file in case none has been provided
via XML. But later, when migrating, we should not include such listen
address in the migratable XML as it is something autogenerated, not
requested by user. Moreover, the binding to the listen address will
likely fail, unless the address is '0.0.0.0' or its IPv6 equivalent.
This patch introduces a new boolean attribute to virDomainGraphicsListenDef
to distinguish autofilled listen addresses. However, we must keep the
attribute over libvirtd restarts, so it must be kept within status XML.

(cherry picked from commit 6546017c)

iscsiadm now supports specifying hostnames in the portal argument [1]

Instead of resolving the hostname to a single IPv4 address, pass the
hostname to isciadm, allowing IPv6 targets to work.

[1] https://bugzilla.redhat.com/show_bug.cgi?id=624437

(cherry picked from commit cbdb3c73)

https://bugzilla.redhat.com/show_bug.cgi?id=964177

Though both libvirt and QEMU's document say RTC_CHANGE returns
the offset from the host UTC, qemu actually returns the offset
from the specified date instead when specific date is provided
(-rtc base=$date).

It's not safe for qemu to fix it in code, it worked like that
for 3 years, changing it now may break other QEMU use cases.
What qemu tries to do is to fix the document:

http://lists.gnu.org/archive/html/qemu-devel/2013-05/msg04782.html

And in libvirt side, instead of replying on the value from qemu,
this converts the offset returned from qemu to the offset from
host UTC, by:

  /*
   * a: the offset from qemu RTC_CHANGE event
   * b: The specified date (-rtc base=$date)
   * c: the host date when libvirt gets the RTC_CHANGE event
   * offset: What libvirt will report
   */

  offset = a + (b - c);

The specified date (-rtc base=$date) is recorded in clock's def as
an internal only member (may be useful to exposed outside?).

Internal only XML tag "basetime" is introduced to not lose the
guest's basetime after libvirt restarting/reloading:

<clock offset='variable' adjustment='304' basis='utc' basetime='1370423588'/>

(cherry picked from commit e31b5cf3)

Instead of a unknown flag error report that metadata pre-allocation is
not supported with the requested volume creation method.

Resolves: https://bugzilla.redhat.com/show_bug.cgi?id=969982
(cherry picked from commit 77e7f69c)

For USB devices, dev->name gets formated as %.3o:%.3o even though the
numbers are decimal.

Resolves: https://bugzilla.redhat.com/show_bug.cgi?id=967712
(cherry picked from commit dc0c7915)

https://bugzilla.redhat.com/show_bug.cgi?id=903480

During domain destruction it's possible that the learnIPAddressThread has
already removed the interface prior to the teardown filter path being run.
The teardown code would only be telling the thread to terminate.

(cherry picked from commit 64919d97)

If snapshot creation failed for example due to invalid use of the
"REUSE_EXTERNAL" flag, libvirt killed access to the original image file
instead of the new image file. On machines with selinux this kills the
whole VM as the selinux context is enforced immediately.

* qemu_driver.c:qemuDomainSnapshotUndoSingleDiskActive():
    - Kill access to the new image file instead of the old one.

Partially resolves: https://bugzilla.redhat.com/show_bug.cgi?id=906639

(cherry picked from commit 17704675)

Function qemuDomainSetBlockIoTune() was checking QEMU capabilities
even when !(flags & VIR_DOMAIN_AFFECT_LIVE) and the domain was
shutoff, resulting in the following problem:

 virsh # domstate asdf; blkdeviotune asdf vda --write-bytes-sec 100
 shut off

 error: Unable to change block I/O throttle
 error: unsupported configuration: block I/O throttling not supported with this QEMU binary

Resolves: https://bugzilla.redhat.com/show_bug.cgi?id=965016
(cherry picked from commit 5af3ce82)

By default files in a FUSE mount can only be accessed by the
user which created them, even if the file permissions would
otherwise allow it. To allow other users to access the FUSE
mount the 'allow_other' mount option must be used. This bug
prevented non-root users in an LXC container from reading
the /proc/meminfo file.

https://bugzilla.redhat.com/show_bug.cgi?id=967977Signed-off-by: NDaniel P. Berrange <berrange@redhat.com>
(cherry picked from commit 922ebe4e)

Many applications use /dev/tty to read from stdin.
e.g. zypper on openSUSE.

Let's create this device node to unbreak those applications.
As /dev/tty is a synonym for the current controlling terminal
it cannot harm the host or any other containers.
Signed-off-by: NRichard Weinberger <richard@nod.at>
(cherry picked from commit 9a0ac6d9)

Conflicts:
	src/lxc/lxc_controller.c

This patch implements support for the "cpu-add" QMP command that plugs
CPUs into a live guest. The "cpu-add" command was introduced in QEMU
1.5. For the hotplug to work machine type "pc-i440fx-1.5" is required.

(cherry picked from commit c12b2be5)

Call virLogVMessage instead of virLogMessage, since libudev
called us with a va_list object, not a list of arguments.

Honor message priority and strip the trailing newline.

https://bugzilla.redhat.com/show_bug.cgi?id=969152
(cherry picked from commit f753dd62)

Before trying to mount the selinux filesystem in a container
use is_selinux_enabled() to check if the machine actually
has selinux support (eg not booted with selinux=0)
Signed-off-by: NDaniel P. Berrange <berrange@redhat.com>
(cherry picked from commit 95c6cc34)

Re-add the selinux header to lxc_container.c since other
functions now use it, beyond the patch that was just
reverted.
Signed-off-by: NDaniel P. Berrange <berrange@redhat.com>
(cherry picked from commit 63ea1e54)

Change bbe97ae9 caused the
QEMU driver to ignore ENOENT errors from cgroups, in order
to cope with missing /proc/cgroups. This is not good though
because many other things can cause ENOENT and should not
be ignored. The callers expect to see ENXIO when cgroups
are not present, so adjust the code to report that errno
when /proc/cgroups is missing
Signed-off-by: NDaniel P. Berrange <berrange@redhat.com>
(cherry picked from commit c2cf5f1c)

Found that I was unable to start existing domains after updating
to a kernel with no cgroups support

  # zgrep CGROUP /proc/config.gz
  # CONFIG_CGROUPS is not set
  # virsh start test
  error: Failed to start domain test
  error: Unable to initialize /machine cgroup: Cannot allocate memory

virCgroupPartitionNeedsEscaping() correctly returns errno (ENOENT) when
attempting to open /proc/cgroups on such a system, but it was being
dropped in virCgroupSetPartitionSuffix().

Change virCgroupSetPartitionSuffix() to propagate errors returned by
its callees.  Also check for ENOENT in qemuInitCgroup() when determining
if cgroups support is available.
(cherry picked from commit bbe97ae9)

Escaping a leading '.' with '_' in the cgroup names
Signed-off-by: NDaniel P. Berrange <berrange@redhat.com>
(cherry picked from commit 0ced83dc)

Describe the new cgroups layout, how to customize placement
of guests and what virsh commands are used to access the
parameters.
Signed-off-by: NDaniel P. Berrange <berrange@redhat.com>
(cherry picked from commit 486a86eb)

It is possible to build a kernel without swap cgroup controls
present. This causes a fatal error when querying memory
parameters. Treat missing swap controls as meaning "unlimited".
The fatal error remains if the user tries to actually change
the limit.
Signed-off-by: NDaniel P. Berrange <berrange@redhat.com>
(cherry picked from commit f493d83f)

Xen 4.3 fixes a mistake in the libxl event handler signature where the
event owned by the application was defined as const.  Detect this and
define the libvirt libxl event handler signature appropriately.
(cherry picked from commit 43b0ff5b)

If qemuMonitorBlockJob returned 0, qemuDomainBlockPivot
might return 0 even if an error occured.

https://bugzilla.redhat.com/show_bug.cgi?id=977678
(cherry picked from commit c34107df)