Commit 1c275e9a accidentally dropped the storage driver from
libvirtd, because it depended on a C preprocessor macro that
was not defined.  Furthermore, if you do './configure
--without-storage-dir --with-storage-disk' or any other combination
where you explicitly build a subset of storage backends excluding
the dir backend, then the build is broken.

Based on analysis by Osier Yang.

* configure.ac (WITH_STORAGE): Define top-level conditional.
* src/Makefile.am (mod_LTLIBRARIES): Build driver even when
storage_dir is disabled.
* daemon/libvirtd.c: Pick up storage driver for any backend, not
just dir.
* daemon/Makefile.am (libvirtd_LDADD): Likewise.

We had a distributed file (remote_protocol.h, which in turn was
a prereq to remote_driver.c) depending on a generated file
(libvirt_probes.h), which is a no-no for a VPATH build from a
read-only source tree (no wonder 'make distcheck' tests precisely
that situation):

     File `libvirt_driver_remote.la' does not exist.
       File `libvirt_driver_remote_la-remote_driver.lo' does not exist.
             Prerequisite `libvirt_probes.h' is newer than target `../../src/remote/remote_protocol.h'.
            Must remake target `../../src/remote/remote_protocol.h'.
Invoking recipe from Makefile:7464 to update target `../../src/remote/remote_protocol.h'.
make[3]: Entering directory `/home/remote/eblake/libvirt-tmp2/build/libvirt-0.9.12/_build/src'
  GEN    ../../src/remote/remote_protocol.h
cannot create ../../src/remote/remote_protocol.h: Permission denied at ../../src/rpc/genprotocol.pl line 31.
make[3]: *** [../../src/remote/remote_protocol.h] Error 13

Rather than making distributed .c files depend on generated files, we
really want to ensure that compilation into .lo files is not attempted
until the generated files are present, done by this patch.  Since there
were two different sets of conditionally generated files that both
feed the .lo file, I had to introduce a new variable REMOTE_DRIVER_PREREQS
to keep automake happy.

After that fix, the next issue was that make treats './foo' and 'foo'
differently in determining whether an implicit %foo rule is applicable,
with the result that locking/qemu-sanlock.conf wasn't properly being
built at the right times.  Also, the output for using the .aug test
files was a bit verbose.

After fixing the src directory, the next error is related to the docs
directory, where the tarball is missing a stamp file and thus tries to
regenerate files that are already present:

  GEN    ../../docs/apibuild.py.stamp
Traceback (most recent call last):
  File "../../docs/apibuild.py", line 2511, in <module>
    rebuild("libvirt")
  File "../../docs/apibuild.py", line 2495, in rebuild
    builder.serialize()
  File "../../docs/apibuild.py", line 2424, in serialize
    output = open(filename, "w")
IOError: [Errno 13] Permission denied: '../../docs/libvirt-api.xml'
make[5]: *** [../../docs/apibuild.py.stamp] Error 1

and fixing that exposed another case of a distributed file (generated
html) depending on a built file (libvirt.h), but only when doing an
in-tree build, because of a file glob.

* src/Makefile.am ($(srcdir)/remote/remote_driver.c): Change...
(libvirt_driver_remote_la-remote_driver.lo): ...to the real
dependency.
($(builddir)/locking/%-sanlock.conf): Drop $(builddir), so that
rule gets run in time for test_libvirt_sanlock.aug.
(test_libvir*.aug): Cater to silent build.
(conf_DATA): Don't ship qemu-sanlock.conf in the tarball, since it
is trivial to regenerate.
* docs/Makefile.am (EXTRA_DIST): Ship our stamp file.
($(APIBUILD_STAMP)): Don't depend on generated file.

Although src/util/viratomic.h has been added to the repo, up until now
it hasn't been used. Stefan Berger is using it in his proposed dhcp
snooping patches, and an rpm build with those patches failed due to
viratomic.h not being packed up with the rest of the sources.

The goal of this patch is to prepare for support for multiple IP
addresses per interface in the DHCP snooping code.

Move the code for the IP address map that maps interface names to
IP addresses into their own file. Rename the functions on the way
but otherwise leave the code as-is. Initialize this new layer
separately before dependent layers (iplearning, dhcpsnooping)
and shut it down after them.

This patch adds DHCP snooping support to libvirt. The learning method for
IP addresses is specified by setting the "CTRL_IP_LEARNING" variable to one of
"any" [default] (existing IP learning code), "none" (static only addresses)
or "dhcp" (DHCP snooping).

Active leases are saved in a lease file and reloaded on restart or HUP.

The following interface XML activates and uses the DHCP snooping:

    <interface type='bridge'>
      <source bridge='virbr0'/>
      <filterref filter='clean-traffic'>
        <parameter name='CTRL_IP_LEARNING' value='dhcp'/>
      </filterref>
    </interface>

All filters containing the variable 'IP' are automatically adjusted when
the VM receives an IP address via DHCP. However, multiple IP addresses per
interface are silently ignored in this patch, thus only supporting one IP
address per interface. Multiple IP address support is added in a later
patch in this series.
Signed-off-by: NDavid L Stevens <dlstevens@us.ibm.com>
Signed-off-by: NStefan Berger <stefanb@linux.vnet.ibm.com>

Apart from the non-sanlock check build, there is also a little fix for
qemu (EXTRA_DIST had qemu.conf and others inside even if the build was
supposed to be without qemu).

We generate *.aug from *.aug.in by augeas-gentest.pl, so this script
should be included in dist file.

Some of our rules used $(PERL), while others used 'perl'.  Always
using the variable allows a developer to point to a different (often
better) perl than the default one found on $PATH.

* daemon/Makefile.am ($(srcdir)/remote_dispatch.h): s/perl/$(PERL).
* src/Makefile.am ($(srcdir)/remote/remote_client_bodies.h)
(PDWTAGS, %protocol.c, %_probes.stp): Likewise.

Without this fix, a VPATH build (such as used by ./autobuild.sh)
fails with messages like:

make[3]: Entering directory `/home/remote/eblake/libvirt-tmp2/build/daemon'
../../build-aux/augeas-gentest.pl libvirtd.conf ../../daemon/test_libvirtd.aug.in test_libvirtd.aug
cannot read libvirtd.conf: No such file or directory at ../../build-aux/augeas-gentest.pl line 38.

Since the test files are not part of the tarball, we can generate
them into the build dir, but rather than create a subdirectory
just for the test file, it is easier to test them directly in
libvirt.git/src.

* daemon/Makefile.am (AUG_GENTEST): Factor out definition.
(test_libvirtd.aug): Look for correct file.
* src/Makefile.am (AUG_GENTEST): Use $(PERL).
(qemu/test_libvirtd_qemu.aug, lxc/test_libvirtd_lxc.aug)
(locking/test_libvirt_sanlock.aug): Rename to avoid subdirectories.
(check-augeas-qemu, check-augeas-lxc, check-augeas-sanlock): Reflect
location of built tests.
* configure.ac (PERL): Substitute perl.

I added libvirt_qemu_probes.h into BUILT_SOURCES. That makes it
generated, but most probably it is not the clearest way how to do
that, but it fixes the build.

The previous patch fixed an incremental build, but missed that on
a fresh checkout, we now have nothing left that stops make from
nuking libvirt_qemu_probes.o.

* src/Makefile.am ($(libvirt_driver_qemu_la_SOURCES)): Delete,
since this variable is empty.
(.PRECIOUS): Add %_probes.o, so they don't get nuked as an
intermediate by-product after creating %_probes.lo.

The moment you specify a _DEPENDENCIES, older automake (stupidly)
assumes that you will specify _all_ dependencies for that target.
This stupidity has been fixed in automake 1.12, but we cannot rely on
newer automake everywhere.  For libvirt_la_DEPENDENCIES, we took
care of providing the full list, but for libvirt_qemu_la_DEPENDENCIES,
we were missing the dependency on libvirt_qemu_impl.la, which resulted
in a failed build:

make[3]: Entering directory `/home/ajia/Workspace/libvirt/src'
   CCLD   libvirt_driver_qemu.la
libtool: link: `libvirt_qemu_probes.lo' is not a valid libtool object

* src/Makefile.am (libvirt_driver_qemu_la_DEPENDENCIES): Delete;
automake does a better job if it does the entire job.

Otherwise, 'make rpm' fails with:

make[2]: *** No rule to make target `locking/test_libvirt_sanlock.aug.in', needed by `distdir'.  Stop.
make[2]: Leaving directory `/home/dipankar/src/powerpc/libvirt-ppc/src'
make[1]: *** [distdir] Error 1

Reported by Dipankar Sarma.

* src/Makefile.am (CLEANFILES): Clean only the generated file.

Libtool is picky about linking against a module library (aka a .so);
giving lots of warnings like this in the tests directory:

  CCLD   networkxml2argvtest

*** Warning: Linking the executable networkxml2argvtest against the loadable module
*** libvirt_driver_network.so is not portable!

Fix that by splitting things into a convenience library which can
be used directly by the tests, and making the real .so just wrap
the convenience library.

Based on a suggestion by Daniel P. Berrange.

* configure.ac (--with-driver-modules): Fix help test.
* src/Makefile.am (libvirt_driver_xen.la, libvirt_driver_libxl.la)
(libvirt_driver_qemu.la, libvirt_driver_lxc.la)
(libvirt_driver_uml.la): Factor into new convenience libraries.
* tests/Makefile.am (xen_LDADDS, qemu_LDADDS, lxc_LDADDS)
(networkxml2argvtest_LDADD): Link to convenience libraries, not
shared libraries.

There was no rule forcing libvirt_qemu_probes.o to be built
before libvirt_qemu_probes.lo was used. Also libvirtd was
still referencing the .o file, rather than the .lo file.

Both the .lo and .o file must be listed as DEPENDENCIES,
otherwise libtool will unhelpfully delete the .o file
once the .lo file is created.
Signed-off-by: NDaniel P. Berrange <berrange@redhat.com>

The CoTaskMemFree function requires the ole32 DLL to be
linked against. Currently this is only done for the
VirtualBox driver. Also add it to libvirt_util.la

* configure.ac: Unconditionally add ole32 DLL to Win32
* src/Makefile.am: Link old32 to libvirt_util.la

When adding new config file parameters, the corresponding
additions to the augeas lens' are constantly forgotten.
Also there are augeas test cases, these don't catch the
error, since they too are never updated.

To address this, the augeas test cases need to be auto-generated
from the example config files.

* build-aux/augeas-gentest.pl: Helper to generate an
  augeas test file, substituting in elements from the
  example config files
* src/Makefile.am, daemon/Makefile.am: Switch to
  auto-generated augeas test cases
* daemon/test_libvirtd.aug, daemon/test_libvirtd.aug.in,
  src/locking/test_libvirt_sanlock.aug,
  src/locking/test_libvirt_sanlock.aug.in,
  src/lxc/test_libvirtd_lxc.aug,
  src/lxc/test_libvirtd_lxc.aug.in,
  src/qemu/test_libvirtd_qemu.aug,
  src/qemu/test_libvirtd_qemu.aug.in: Remove example
  config file data, replacing with a ::CONFIG:: placeholder
Signed-off-by: NDaniel P. Berrange <berrange@redhat.com>

Libtool supports linking directly against .o files on some platforms
(such as Linux), which happens to be the only place where we are
actually doing that (for the dtrace-generated probes.o files).  However,
it raises a big stink about the non-portability, even though we don't
attempt it on platforms where it would actually fail:

  CCLD   libvirt_driver_qemu.la

*** Warning: Linking the shared library libvirt_driver_qemu.la against
the non-libtool
*** objects  libvirt_qemu_probes.o is not portable!

This shuts libtool up by creating a proper .lo file that matches
what libtool normally expects.

* src/Makefile.am (%_probes.lo): New rule.
(libvirt_probes.stp, libvirt_qemu_probes.stp): Simplify into...
(%_probes.stp): ...shorter rule.
(CLEANFILES): Clean new .lo files.
(libvirt_la_BUILT_LIBADD, libvirt_driver_qemu_la_LIBADD)
(libvirt_lxc_LDADD, virt_aa_helper_LDADD): Link against .lo file.
* tests/Makefile.am (PROBES_O, qemu_LDADDS): Likewise.

When building as driver modules, it is not possible for the QEMU
driver module to reference the DTrace/SystemTAP probes linked into
the main libvirt.so. Thus we need to move the QEMU probes into a
separate file 'libvirt_qemu_probes.d'. Also rename the existing
file from 'probes.d' to 'libvirt_probes.d' while we're at it

* daemon/Makefile.am, src/internal.h: Include libvirt_probes.h
  instead of probes.h
* src/Makefile.am: Add rules for libvirt_qemu_probes.d
* src/qemu/qemu_monitor.c, src/qemu/qemu_monitor_json.c,
  src/qemu/qemu_monitor_text.c: Include libvirt_qemu_probes.h
* src/libvirt_probes.d: Rename from probes.d
* src/libvirt_qemu_probes.d: QEMU specific probes formerly
  in probes.d
Signed-off-by: NDaniel P. Berrange <berrange@redhat.com>

Only libvirt_driver_storage.la links to libblkid currently. If
we are running in a scenario with driver modules, LXC must
directly link to it, since it can't assume the storage driver
is present
Signed-off-by: NDaniel P. Berrange <berrange@redhat.com>

The libvirt_test.la library was introduced to allow test suites
to reference internal-only symbols. These days, nearly every
symbol we care about is in src/libvirt_private.syms, so there
is no need for libvirt_test.la to continue to exist

* src/Makefile.am: Delete libvirt_test.la & add new .syms files
* src/libvirt_private.syms: Export symbols needed by test suite
* tests/Makefile.am: Link to libvirt_test.la. Ensure LXC tests link
  to network_driver.la
* src/libvirt_esx.syms, src/libvirt_openvz.syms: Add exports needed
  by test suite

libvirt_driver_nodedev.la should not link against either
libvirt_util.la or gnulib.la, since libvirt.so brings
in those deps.

* src/Makefile.am: Fix broken linkage of libvirt_driver_nodedev.la
Signed-off-by: NDaniel P. Berrange <berrange@redhat.com>

The driver modules all use symbols which are defined in libvirt.so.
Thus for loading of modules to work, the binary that libvirt.so
is linked to must export its symbols back to modules. If the
libvirt.so itself is dlopen()d then the RTLD_GLOBAL flag must
be set. Unfortunately few, if any, programming languages use
the RTLD_GLOBAL flag when loading modules :-( This means is it
not practical to use driver modules for any libvirt client side
drivers (OpenVZ, VMWare, Hyper-V, Remote client, test).

This patch changes the build process so only server side drivers
are built as modules (Xen, QEMU, LXC, UML)

* daemon/libvirtd.c: Add missing load of 'interface' driver
* src/Makefile.am: Only build server side drivers as modules
* src/libvirt.c: Don't load any driver modules
Signed-off-by: NDaniel P. Berrange <berrange@redhat.com>

This patch adds support for a new storage backend with RBD support.

RBD is the RADOS Block Device and is part of the Ceph distributed storage
system.

It comes in two flavours: Qemu-RBD and Kernel RBD, this storage backend only
supports Qemu-RBD, thus limiting the use of this storage driver to Qemu only.

To function this backend relies on librbd and librados being present on the
local system.

The backend also supports Cephx authentication for safe authentication with
the Ceph cluster.

For storing credentials it uses the built-in secret mechanism of libvirt.
Signed-off-by: NWido den Hollander <wido@widodh.nl>

to save some syscalls (as suggested by Eric Blake)

configure.ac: check for libnl-3 in addition to libnl-1

src/Makefile.am: link against libnl when needed

src/util/virnetlink.c:
support libnl3 api.  To minimize impact on code flow, wrap the
differences under the virNetlink* namespace.

Unfortunately libnl3 moves netlink/msg.h to
/usr/include/libnl3/netlink/msg.h, so the LIBNL_CFLAGS need to be added
to a bunch of places where they weren't needed with libnl1.
Signed-off-by: NSerge Hallyn <serge.hallyn@canonical.com>
Signed-off-by: NEric Blake <eblake@redhat.com>

commit 2223ea98 removes src/libvirt_dbus.syms, but it forgets
to remove it from EXTRA_DIST. It will cause 'make dist' failed.

DBus connection. The HAL device code further requires that
the DBus connection is integrated with the event loop and
provides such glue logic itself.

The forthcoming FirewallD integration also requires a
dbus connection with event loop integration. Thus we need
to pull the current event loop glue out of the HAL driver.

Thus we create src/util/virdbus.{c,h} files. This contains
just one method virDBusGetSystemBus() which obtains a handle
to the single shared system bus instance, with event glue
automagically setup.

* src/util/virauth.c, src/util/virauth.h: Add virAuthGetConfigFilePath
* include/libvirt/virterror.h, src/util/virterror.c: Add
  VIR_FROM_AUTH error domain
Signed-off-by: NDaniel P. Berrange <berrange@redhat.com>

To follow latest naming conventions, rename src/util/authhelper.[ch]
to src/util/virauth.[ch].

* src/util/authhelper.[ch]: Rename to src/util/virauth.[ch]
* src/esx/esx_driver.c, src/hyperv/hyperv_driver.c,
  src/phyp/phyp_driver.c, src/xenapi/xenapi_driver.c: Update
  for renamed include files
Signed-off-by: NDaniel P. Berrange <berrange@redhat.com>

The '.ini' file format is a useful alternative to the existing
config file style, when you need to have config files which
are hashes of hashes. The 'virKeyFilePtr' object provides a
way to parse these file types.

* src/Makefile.am, src/util/virkeyfile.c,
  src/util/virkeyfile.h: Add .ini file parser
* tests/Makefile.am, tests/virkeyfiletest.c: Test
  basic parsing capabilities
Signed-off-by: NDaniel P. Berrange <berrange@redhat.com>

Convert drivers currently using the qparams APIs, to instead
use the virURIPtr query parameters directly.

* src/esx/esx_util.c, src/hyperv/hyperv_util.c,
  src/remote/remote_driver.c, src/xenapi/xenapi_utils.c: Remove
  use of qparams
* src/util/qparams.h, src/util/qparams.c: Delete
* src/Makefile.am, src/libvirt_private.syms: Remove qparams
Signed-off-by: NDaniel P. Berrange <berrange@redhat.com>

Building virt-aa-helper with dtrace probes enabled, ldd complained about
undefined references:

./.libs/libvirt_util.a(libvirt_util_la-event_poll.o):(.note.stapsdt+0x24):
undefined reference to `libvirt_event_poll_purge_timeout_semaphore'
...

This patch adds a set of functions used in creating console streams for
domains using PTYs and ensures mutually exclusive access to the PTYs.

If mutually exclusive access is not used, two clients may open the same
console, which results in corruption on both clients as both of them
race to read data from the PTY.

Two approaches are used to ensure this:
1) Internal data structure holding open PTYs.
        This is used internally and enables the user to forcibly
        terminate another console connection eg. when somebody leaves
        the console open on another host.

2) UUCP style lock files:
        This uses UUCP lock files according to the  FHS
        ( http://www.pathname.com/fhs/pub/fhs-2.3.html#VARLOCKLOCKFILES )
        to check if other programs (like minicom) are not using the pty
        device of the console.

        This feature is disabled by default and may be enabled using
        configure parameter
        --with-console-lock-files=/path/to/lock/file/directory
        or --with-console-lock-files=auto (which tries to infer the
        location from OS used (currently only linux).

        On usual linux systems, normal users may not write to the
        /var/lock directory containing the locks. This poses problems
        while in session mode. If the current user has no access to the
        lockfile directory, check for presence of the file is still
        done, but no lock file is created. This does NOT result in an
        error.

Function xmlParseURI does not remove square brackets around IPv6
address when parsing. One of the solutions is making wrappers around
functions working with xmlURI*. This assures that uri->server will be
always properly assigned and it doesn't have to be changed when used
on some new place in the code.
For this purpose, functions virParseURI and virSaveURI were
added. These function are wrappers around xmlParseURI and xmlSaveUri
respectively.
Also there is one new syntax check function to prohibit these functions
anywhere else.

File changes:
 - src/util/viruri.h        -- declaration
 - src/util/viruri.c        -- definition
 - src/libvirt_private.syms -- symbol export
 - src/Makefile.am          -- added source and header files
 - cfg.mk                   -- added sc_prohibit_xmlURI
 - all others               -- ID name and include fixes

The /usr/include/python/pyconfig.h file pollutes the global
namespace with a huge number of HAVE_XXX and WITH_XXX
defines. These change what we detected in our own config.h
In particular if you try to build without DTrace, python's
headers turn it back on with predictable fail.

THe hack to workaround this is to rename WITH_DTRACE to
WITH_DTRACE_PROBES to avoid the namespace clash

This patch allows libvirt to add interfaces to already
existing Open vSwitch bridges. The following syntax in
domain XML file can be used:

    <interface type='bridge'>
      <mac address='52:54:00:d0:3f:f2'/>
      <source bridge='ovsbr'/>
      <virtualport type='openvswitch'>
        <parameters interfaceid='921a80cd-e6de-5a2e-db9c-ab27f15a6e1d'/>
      </virtualport>
      <address type='pci' domain='0x0000' bus='0x00'
                          slot='0x03' function='0x0'/>
    </interface>

or if libvirt should auto-generate the interfaceid use
following syntax:

    <interface type='bridge'>
      <mac address='52:54:00:d0:3f:f2'/>
      <source bridge='ovsbr'/>
      <virtualport type='openvswitch'>
      </virtualport>
      <address type='pci' domain='0x0000' bus='0x00'
                          slot='0x03' function='0x0'/>
    </interface>

It is also possible to pass an optional profileid. To do that
use following syntax:

   <interface type='bridge'>
     <source bridge='ovsbr'/>
     <mac address='00:55:1a:65:a2:8d'/>
     <virtualport type='openvswitch'>
       <parameters interfaceid='921a80cd-e6de-5a2e-db9c-ab27f15a6e1d'
                   profileid='test-profile'/>
     </virtualport>
   </interface>

To create Open vSwitch bridge install Open vSwitch and
run the following command:

    ovs-vsctl add-br ovsbr

Rename the src/util/netlink files to src/util/virnetlink to
better fit the naming scheme. Also rename nlComm to virNetlinkCommand.
Signed-off-by: ND. Herrendoerfer <d.herrendoerfer@herrendoerfer.name>

Our syntax checker missed all-lower-case variables (this will
be fixed by the next .gnulib update).  Additionally, anywhere
that we mix in-tree files with generated files, automake recommends
listing builddir prior to srcdir for VPATH builds.

* src/Makefile.am (*_la_CFLAGS): Favor $(top_srcdir).
(INCLUDES): Likewise, and follow automake recommendations on
builddir before srcdir.
* python/Makefile.am (INCLUDES): Swap directory order.
* tests/Makefile.am (INCLUDES): Likewise.
* tools/Makefile.am (INCLUDES): Likewise.
* daemon/Makefile.am (INCLUDES): Likewise.
(libvirtd.init, libvirtd.service): Favor $().
* examples/hellolibvirt/Makefile.am (hellolibvirt_LDADD):
Likewise.
* examples/openauth/Makefile.am (openauth_LDADD): Likewise.
* examples/dominfo/Makefile.am (INCLUDES): Drop dead include.
* examples/domsuspend/Makefile.am (INCLUDES): Likewise.