Given an LXC guest with a root filesystem path of

  /export/lxc/roots/helloworld/root

During startup, we will pivot the root filesystem to end up
at

  /.oldroot/export/lxc/roots/helloworld/root

We then try to open

  /.oldroot/export/lxc/roots/helloworld/root/dev/pts

Now consider if '/export/lxc' is an absolute symlink pointing
to '/media/lxc'. The kernel will try to open

  /media/lxc/roots/helloworld/root/dev/pts

whereas it should be trying to open

  /.oldroot//media/lxc/roots/helloworld/root/dev/pts

To deal with the fact that the root filesystem can be moved,
we need to resolve symlinks in *any* part of the filesystem
source path.

* src/libvirt_private.syms, src/util/util.c,
  src/util/util.h: Add virFileResolveAllLinks to resolve
  all symlinks in a path
* src/lxc/lxc_container.c: Resolve all symlinks in filesystem
  paths during startup

Move the virNetDevSetName and virNetDevSetNamespace APIs out
of LXC's veth.c and into virnetdev.c.

Move the remaining content of the file to src/util/virnetdevveth.c

* src/lxc/veth.c: Rename to src/util/virnetdevveth.c
* src/lxc/veth.h: Rename to src/util/virnetdevveth.h
* src/util/virnetdev.c, src/util/virnetdev.h: Add
  virNetDevSetName and virNetDevSetNamespace
* src/lxc/lxc_container.c, src/lxc/lxc_controller.c,
  src/lxc/lxc_driver.c: Update include paths

The src/lxc/veth.c file contains APIs for managing veth devices,
but some of the APIs duplicate stuff from src/util/virnetdev.h.
Delete thed duplicate APIs and rename the remaining ones to
follow virNetDevVethXXXX

* src/lxc/veth.c, src/lxc/veth.h: Rename APIs & delete duplicates
* src/lxc/lxc_container.c, src/lxc/lxc_controller.c,
  src/lxc/lxc_driver.c: Update for API renaming

* .gnulib: Update to latest, for improved syntax-check.
* src/lxc/lxc_container.c (includes): Drop unused include.
* src/network/bridge_driver.c: Likewise.
* src/node_device/node_device_linux_sysfs.c: Likewise.
* src/openvz/openvz_driver.c: Likewise.
* src/qemu/qemu_conf.c: Likewise.
* src/storage/storage_backend_iscsi.c: Likewise.
* src/storage/storage_backend_mpath.c: Likewise.
* src/uml/uml_conf.c: Likewise.
* src/uml/uml_driver.c: Likewise.

Currently the LXC controller only supports setup of a single
text console. This is wired up to the container init's stdio,
as well as /dev/console and /dev/tty1. Extending support for
multiple consoles, means wiring up additional PTYs to /dev/tty2,
/dev/tty3, etc, etc. The LXC controller is passed multiple open
file handles, one for each console requested.

* src/lxc/lxc_container.c, src/lxc/lxc_container.h: Wire up
  all the /dev/ttyN links required to symlink to /dev/pts/NN
* src/lxc/lxc_container.h: Open more container side /dev/pts/NN
  devices, and adapt event loop to handle I/O from all consoles
* src/lxc/lxc_driver.c: Setup multiple host side PTYs

The LXC code for mounting container filesystems from block devices
tries all filesystems in /etc/filesystems and possibly those in
/proc/filesystems. The regular mount binary, however, first tries
using libblkid to detect the format. Add support for doing the same
in libvirt, since Fedora's /etc/filesystems is missing many formats,
most notably ext4 which is the default filesystem Fedora uses!

* src/Makefile.am: Link libvirt_lxc to libblkid
* src/lxc/lxc_container.c: Probe filesystem format with libblkid

If we looped through /etc/filesystems trying to mount with each
type and failed all options, we forget to actually raise an
error message.

* src/lxc/lxc_container.c: Raise error if unable to detect
  the filesystems. Also fix existing error message

The kernel automounter is mostly broken wrt to containers. Most
notably if you start a new filesystem namespace and then attempt
to unmount any autofs filesystem, it will typically fail with a
weird error message like

  Failed to unmount '/.oldroot/sys/kernel/security':Too many levels of symbolic links

Attempting to detach the autofs mount using umount2(MNT_DETACH)
will also fail with the same error. Therefore if we get any error on
unmount()ing a filesystem from the old root FS when starting a
container, we must immediately break out and detach the entire
old root filesystem (ignoring any mounts below it).

This has the effect of making the old root filesystem inaccessible
to anything inside the container, but at the cost that the mounts
live on in the kernel until the container exits. Given that SystemD
uses autofs by default, we need LXC to be robust this scenario and
thus this tradeoff is worthwhile.

* src/lxc/lxc_container.c: Detach root filesystem if any umount
  operation fails.

The /etc/filesystems file can contain a '*' on the last line to
indicate that /proc/filessystems should be tried next. We have
a check that this '*' only occurs on the last line. Unfortunately
when we then start reading /proc/filesystems, we mistakenly think
we've seen '*' in /proc/filesystems and fail

* src/lxc/lxc_container.c: Skip '*' validation when we're reading
  /proc/filesystems

Only some of the return paths of lxcContainerWaitForContinue will
have set errno. In other paths we need to set it manually to avoid
the caller getting a random stale errno value

* src/lxc/lxc_container.c: Set errno in lxcContainerWaitForContinue

Based on a report by Coverity.  waitpid() can leak resources if it
fails with EINTR, so it should never be used without checking return
status.  But we already have a helper function that does that, so
use it in more places.

* src/lxc/lxc_container.c (lxcContainerAvailable): Use safer
virWaitPid.
* daemon/libvirtd.c (daemonForkIntoBackground): Likewise.
* tests/testutils.c (virtTestCaptureProgramOutput, virtTestMain):
Likewise.
* src/libvirt.c (virConnectAuthGainPolkit): Simplify with virCommand.

I noticed a couple typos in recent commits, and fixed the remaining
instances of them.

* docs/internals/command.html.in: Fix spelling errors.
* include/libvirt/libvirt.h.in (virConnectDomainEventCallback):
Likewise.
* python/libvirt-override.py (virEventAddHandle): Likewise.
* src/lxc/lxc_container.c (lxcContainerChild): Likewise.
* src/util/hash.c (virHashCreateFull): Likewise.
* src/storage/storage_backend_logical.c
(virStorageBackendLogicalMakeVol): Likewise.
* src/esx/esx_driver.c (esxFormatVMXFileName): Likewise.
* src/vbox/vbox_tmpl.c (vboxIIDIsEqual_v3_x): Likewise.

When booting a virtual machine with a kernel/initrd it is possible
to pass command line arguments using the <cmdline>...args...</cmdline>
element in the guest XML. These appear to the kernel / init process
in /proc/cmdline.

When booting a container we do not have a custom /proc/cmdline,
but we can easily set an environment variable for it. Ideally
we could pass individual arguments to the init process as a
regular set of 'char *argv[]' parameters, but that would involve
libvirt parsing the <cmdline> XML text. This can easily be added
later, even if we add the env variable now

* docs/drvlxc.html.in: Document env variables passed to LXC
* src/conf/domain_conf.c: Add <cmdline> to be parsed for
  guests of type='exe'
* src/lxc/lxc_container.c: Set LIBVIRT_LXC_CMDLINE env var

as they are not used with debugging turned off.

Hi,

I'm seeing an issue with udev and libvirt-lxc.  Libvirt-lxc creates
/dev/ptmx as a symlink to /dev/pts/ptmx.  When udev starts up, it
checks the device type, sees ptmx is 'not right', and replaces it
with a 'proper' ptmx.

In lxc, /dev/ptmx is bind-mounted from /dev/pts/ptmx instead of being
symlinked, so udev sees the right device type and leaves it alone.

A patch like the following seems to work for me.  Would there be
any objections to this?

>From 4c5035de52de7e06a0de9c5d0bab8c87a806cba7 Mon Sep 17 00:00:00 2001
From: Ubuntu <ubuntu@domU-12-31-39-14-F0-B3.compute-1.internal>
Date: Wed, 31 Aug 2011 18:15:54 +0000
Subject: [PATCH 1/1] make ptmx a bind mount rather than symlink

udev on some systems checks the device type of /dev/ptmx, and replaces it if
not as expected.  The symlink created by libvirt-lxc therefore gets replaced.
By creating it as a bind mount, the device type is correct and udev leaves it
alone.
Signed-off-by: NSerge Hallyn <serge.hallyn@canonical.com>

* src/qemu/qemu_driver.c (qemuDomainSaveInternal): Fix typo.
* src/conf/domain_event.c (virDomainEventDispatchMatchCallback):
Likewise.
* daemon/libvirtd.c (daemonRunStateInit): Likewise.
* src/lxc/lxc_container.c (lxcContainerChildMountSort): Likewise.
* src/util/virterror.c (virCopyError, virRaiseErrorFull): Likewise.
* src/xenxs/xen_sxpr.c (xenParseSxprSound): Likewise.

A previous commit gave the LXC driver the ability to mount
block devices for the container filesystem. Through use of
the loopback device functionality, we can build on this to
support use of plain file images for LXC filesytems.

By setting the LO_FLAGS_AUTOCLEAR flag we can ensure that
the loop device automatically disappears when the container
dies / shuts down

* src/lxc/lxc_container.c: Raise error if we see a file
  based filesystem, since it should have been turned into
  a loopback device already
* src/lxc/lxc_controller.c: Rewrite any filesystems of
  type=file, into type=block, by binding the file image
  to a free loop device

Currently the LXC driver can only populate filesystems from
host filesystems, using bind mounts. This patch allows host
block devices to be mounted. It autodetects the filesystem
format at mount time, and adds the block device to the cgroups
ACL. Example usage is

    <filesystem type='block' accessmode='passthrough'>
      <source dev='/dev/sda1'/>
      <target dir='/home'/>
    </filesystem>

* src/lxc/lxc_container.c: Mount block device filesystems
* src/lxc/lxc_controller.c: Add block device filesystems
  to cgroups ACL

An application container shouldn't get a private /dev. Fix
the regression from 6d37888e

* src/lxc/lxc_container.c: Don't mount /dev for app containers

A container should not be allowed to modify stuff in /sys
or /proc/sys so make them readonly. Make /selinux readonly
so that containers think that selinux is disabled.

Honour the readonly flag when mounting container filesystems
from the guest XML config

* src/lxc/lxc_container.c: Support readonly mounts

Even in non-virtual root filesystem mode we should be mounting
more than just a new /proc. Refactor lxcContainerMountBasicFS
so that it does everything except for /dev and /dev/pts moving
that into lxcContainerMountDevFS. Pass in a source prefix
to lxcContainerMountBasicFS() so it can be used in both shared
root and private root modes.

* src/lxc/lxc_container.c: Unify mounting code for special
  filesystems

The bind mount setup is about to get more complicated.
To avoid having to deal with several copies, pull it
out into a separate lxcContainerMountFSBind method.

Also pull out the iteration over container filesystems,
so that it will be easier to drop in support for non-bind
mount filesystems

* src/lxc/lxc_container.c: Pull bind mount code out into
  lxcContainerMountFSBind

In preparation for a future patch adding new virFile APIs.

* src/util/files.h, src/util/files.c: Move...
* src/util/virfile.h, src/util/virfile.c: ...here, and rename
functions to virFile prefix.  Macro names are intentionally
left alone.
* *.c: All '#include "files.h"' uses changed.
* src/Makefile.am (UTIL_SOURCES): Reflect rename.
* cfg.mk (exclude_file_name_regexp--sc_prohibit_close): Likewise.
* src/libvirt_private.syms: Likewise.
* docs/hacking.html.in: Likewise.
* HACKING: Regenerate.

* src/lxc/lxc_driver.c (lxcOpen, lxcDomainSetMemoryParameters)
(lxcDomainGetMemoryParameters): Reject unknown flags.
* src/lxc/lxc_container.c (lxcContainerStart): Rename flags to
cflags to reflect that it is not tied to libvirt.

Some callers expected virFileMakePath to set errno, some expected
it to return an errno value. Unify this to return 0 on success and
-1 on error. Set errno to report detailed error information.

Also optimize virFileMakePath if stat fails with an errno different
from ENOENT.

Since we can't really get useful error reporting from virCommandExec since
it needs to be the last thing we do.

Add a handshake with the cloned container process to try and detect
if it fails to start.

Add a simple handshake with the lxc_controller process so we can detect
process startup failures. We do this by adding a new --handshake cli arg
to lxc_controller for passing a file descriptor. If the process fails to
launch, we scrape all output from the logfile and report it to the user.

We will reuse these shortly, and each use should have a different error
message.

Makes it more likely we get useful error output in the logs

These VIR_XXXX0 APIs make us confused, use the non-0-suffix APIs instead.

How do these coversions works? The magic is using the gcc extension of ##.
When __VA_ARGS__ is empty, "##" will swallow the "," in "fmt," to
avoid compile error.

example: origin				after CPP
	high_level_api("%d", a_int)	low_level_api("%d", a_int)
	high_level_api("a  string")	low_level_api("a  string")

About 400 conversions.

8 special conversions:
VIR_XXXX0("") -> VIR_XXXX("msg") (avoid empty format) 2 conversions
VIR_XXXX0(string_literal_with_%) -> VIR_XXXX(%->%%) 0 conversions
VIR_XXXX0(non_string_literal) -> VIR_XXXX("%s", non_string_literal)
  (for security) 6 conversions
Signed-off-by: NLai Jiangshan <laijs@cn.fujitsu.com>

That way we can log the 'init' argv for debugging.

* src/internal.h: Define a ATTRIBUTE_NO_RETURN annotation
* src/lxc/lxc_container.c: Annotate lxcContainerDummyChild
  with ATTRIBUTE_NO_RETURN
* tests/eventtest.c: Mark async thread as ATTRIBUTE_NO_RETURN
* m4/virt-compile-warnings.m4: Enable -Wmissing-noreturn

Using the 'personality(2)' system call, we can make a container
on an x86_64 host appear to be i686. Likewise for most other
Linux 64bit arches.

* src/lxc/lxc_conf.c: Fill in 32bit capabilities for x86_64 hosts
* src/lxc/lxc_container.h, src/lxc/lxc_container.c: Add API to
  check if an arch has a 32bit alternative
* src/lxc/lxc_controller.c: Set the process personality when
  starting guest

When spawning 'init' in the container, set

  LIBVIRT_LXC_UUID=XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX
  LIBVIRT_LXC_NAME=YYYYYYYYYYYY

to allow guest software to detect & identify that they
are in a container

* src/lxc/lxc_container.c: Set LIBVIRT_LXC_UUID and
  LIBVIRT_LXC_NAME env vars

Done mechanically with:
$ git grep -l '\bDEBUG0\? *(' | xargs -L1 sed -i 's/\bDEBUG0\? *(/VIR_&/'

followed by manual deletion of qemudDebug in daemon/libvirtd.c, along
with a single 'make syntax-check' fallout in the same file, and the
actual deletion in src/util/logging.h.

* src/util/logging.h (DEBUG, DEBUG0): Delete.
* daemon/libvirtd.h (qemudDebug): Likewise.
* global: Change remaining clients over to VIR_DEBUG counterpart.

Until now, user namespaces have not done much, but (for that
reason) have been innocuous to glob in with other CLONE_
flags.  Upcoming userns development, however, will make tasks
cloned with CLONE_NEWUSER far more restricted.  In particular,
for some time they will be unable to access files with anything
other than the world access perms.

This patch assumes that noone really needs the user namespaces
to be enabled.  If that is wrong, then we can try a more
baroque patch where we create a file owned by a test userid with
700 perms and, if we can't access it after setuid'ing to that
userid, then return 0.  Otherwise, assume we are using an
older, 'harmless' user namespace implementation.

Comments appreciated.  Is it ok to do this?
Signed-off-by: NSerge Hallyn <serge.hallyn@canonical.com>

Pure cosmetic change.

The /dev/console device inside the container must NOT map
to the real /dev/console device node, since this allows the
container control over the current host console. A fun side
effect of this is that starting a container containing a
real Fedora OS will kill off your X server.

Remove the /dev/console node, and replace it with a symlink
to the primary console TTY

* src/lxc/lxc_container.c: Replace /dev/console with a
  symlink to /dev/pty/0
* src/lxc/lxc_controller.c: Remove /dev/console from cgroups
  ACL