With this patch, include public headers in "" form is only allowed
for "internal.h". And only the external tools (examples|tools|python
|include/libvirt) can include the public headers in <> form.

Directories python/tools/examples should include them in <> form,
though this patch allows "" form in these directories by excluding
them, a later patch will do the cleanup.

gnulib is excluded.

Some aspects of the cgroups setup / detection code are quite subtle
and easy to break. It would greatly benefit from unit testing, but
this is difficult because the test suite won't have privileges to
play around with cgroups. The solution is to use monkey patching
via LD_PRELOAD to override the fopen, open, mkdir, access functions
to redirect access of cgroups files to some magic stubs in the
test suite.

Using this we provide custom content for the /proc/cgroup and
/proc/self/mounts files which report a fixed cgroup setup. We
then override open/mkdir/access so that access to the cgroups
filesystem gets redirected into files in a temporary directory
tree in the test suite build dir.
Signed-off-by: NDaniel P. Berrange <berrange@redhat.com>

We've already scrubbed for comparisons of 'uid_t == -1' (which fail
on platforms where uid_t is a u16), but another one snuck in.

* src/util/virutil.c (virSetUIDGIDWithCaps): Correct uid comparison.
* cfg.mk (sc_prohibit_risky_id_promotion): New rule.

testutils.c likes to print summaries after a test completes,
including if it failed.  But if the test outright exit()s,
this summary is skipped.  Enforce that we return instead of exit.

* cfg.mk (sc_prohibit_exit_in_tests): New syntax check.
* tests/commandhelper.c (main): Fix offenders.
* tests/qemumonitorjsontest.c (mymain): Likewise.
* tests/seclabeltest.c (main): Likewise.
* tests/securityselinuxlabeltest.c (mymain): Likewise.
* tests/securityselinuxtest.c (mymain): Likewise.
* tests/testutils.h (VIRT_TEST_MAIN_PRELOAD): Likewise.
* tests/testutils.c (virtTestMain): Likewise.
(virtTestCaptureProgramOutput): Use symbolic name.

hacking: Add some text around the running of Valgrind along with example
output for "real" vs. "false positives".

cfg.mk: Add hacking.in.html to sc_prohibit_raw_allocation

We allow for a trailing semicolon in full line comments since
docs/index.py has some SQL statements in it.

Otherwise constructions like "random (used by default)" end up breaking
syntax check by apparently using non-reentrant functions.

Adjust the macros to free memory allocated during various calls to
perform the check if parameter is NULL prior to really freeing and to
set the pointer to NULL after done freeing.

Nested conditionals are hard to read if they are not indented.
We can't add arbitrary whitespace to everything in spec files,
but we CAN add spaces before %if and %define.  Use this trick,
plus a fancy sed script that rewrites a spec file into a C
file, so we can use cppi to keep our spec file nice.

For reference, the sed script converts code like:

|# RHEL-5 builds are client-only for s390, ppc
|%if 0%{?rhel} == 5
|    %ifnarch %{ix86} x86_64 ia64
|        %define client_only        1
|    %endif
|%endif

into the following for cppi:

|// # RHEL-5 builds are client-only for s390, ppc
|#if a // 0%{?rhel} == 5
|# if a // %{ix86} x86_64 ia64
|#  define client_only        1
|# endif
|#endif

and errors from 'make syntax-check' look like:
spec_indentation
cppi: mingw-libvirt.spec.in: line 130: not properly indented
maint.mk: incorrect preprocessor indentation

* libvirt.spec.in: Add some indentation to make it easier to follow
various conditionals.
* mingw-libvirt-spec.in: Likewise.
* cfg.mk (sc_spec_indentation): New syntax check to enforce it.

This patch introduces support for LXC specific public APIs. In
common with what was done for QEMU, this creates a libvirt_lxc.so
library and libvirt/libvirt-lxc.h header file.

The actual APIs are

  int virDomainLxcOpenNamespace(virDomainPtr domain,
                                int **fdlist,
                                unsigned int flags);

  int virDomainLxcEnterNamespace(virDomainPtr domain,
                                 unsigned int nfdlist,
                                 int *fdlist,
                                 unsigned int *noldfdlist,
                                 int **oldfdlist,
                                 unsigned int flags);

which provide a way to use the setns() system call to move the
calling process into the container's namespace. It is not
practical to write in a generically applicable manner. The
nearest that we could get to such an API would be an API which
allows to pass a command + argv to be executed inside a
container. Even if we had such a generic API, this LXC specific
API is still useful, because it allows the caller to maintain
the current process context, in particular any I/O streams they
have open.

NB the virDomainLxcEnterNamespace() API is special in that it
runs client side, so does not involve the internal driver API.
Signed-off-by: NDaniel P. Berrange <berrange@redhat.com>

There are many aspects of the guest XML which result in the
SELinux driver applying file labelling. With the increasing
configuration options it is desirable to test this behaviour.
It is not possible to assume that the test suite has the
ability to set SELinux labels. Most filesystems though will
support extended attributes. Thus for the purpose of testing,
it is possible to extend the existing LD_PRELOAD hack to
override setfilecon() and getfilecon() to simply use the
'user.libvirt.selinux' attribute for the sake of testing.
Signed-off-by: NDaniel P. Berrange <berrange@redhat.com>

Most checks for libraries take the same format

  * --with-libFOO=yes|no|check|/some/path  argument
  * check for a function NNN in libFOO.so
  * check for a header file DDD/HHH.h
  * Define a WITH_FOO config.h symbol
  * Define a WITH_FOO make conditional
  * Substitute FOO_CFLAGS and FOO_LIBS make variables
  * Print CFLAGS & LIBS summary at the end

Doing all this correctly is rather difficult, typically
done by copy+paste of a previous usage. Further small
improvements people make are not applied to all previous
usages.

Improve this by creating some helper macros to apply
good practice. First, to perform the actual checks:

  LIBVIRT_CHECK_LIB([SELINUX], [selinux],
     [getfilecon], [selinux/selinux.h])

This checks for 'getfilecon' in -lselinux, and the
existence of 'selinux/selinux.h' header file. If successful
it sets SELINUX_CFLAGS and SELINUX_LIBS. The WITH_SELINUX
config.h macro and WITH_SELINUX make conditional are also
defined.

In some cases we need to check two variants of the same
library

  LIBVIRT_CHECK_LIB_ALT([SASL], [sasl2],
                        [sasl_client_init], [sasl/sasl.h],
                        [SASL1], [sasl],
                        [sasl_client_init], [sasl/sasl.h])

This checks for sasl_client_init in libsasl2, and if that
is not found, checks sasl_client_init in libsasl. If the
first check succeeds WITH_SASL is set, while if the second
check succeeds *both* WITH_SASL and WITH_SASL1 are set.

If the library supports pkg-config, then another variant
is available

  LIBVIRT_CHECK_PKG([AVAHI], [avahi-client], [0.6.0])

This checks for avahi-client >= 0.6.0 via pkg-config
and sets WITH_AVAHI if found.

Finally to print a summary of CFLAGS & LIBs found (if any):

  LIBVIRT_RESULT_LIB([SELINUX])
Signed-off-by: NDaniel P. Berrange <berrange@redhat.com>

Added 'usbFreeDevice' to the useless_free_options list in cfg.mk

Test cases for virSysinfoRead. Initially, there are tests for
x86 (DMI based) and s390 (/proc/... based).
In lack of PPC data, I have stubbed out the test for it, but it
can be added with a minimal effort.
Signed-off-by: NViktor Mihajlovski <mihajlov@linux.vnet.ibm.com>

The virtlockd daemon will be responsible for managing locks
on virtual machines. Communication will be via the standard
RPC infrastructure. This provides the XDR protocol definition

* src/locking/lock_protocol.x: Wire protocol for virtlockd
* src/Makefile.am: Include lock_protocol.[ch] in virtlockd
Signed-off-by: NDaniel P. Berrange <berrange@redhat.com>

The virtlockd daemon will maintain locks on behalf of libvirtd.
There are two reasons for it to be separate

 - Avoid risk of other libvirtd threads accidentally
   releasing fcntl() locks by opening + closing a file
   that is locked
 - Ensure locks can be preserved across libvirtd restarts.
   virtlockd will need to be able to re-exec itself while
   maintaining locks. This is simpler to achieve if its
   sole job is maintaining locks
Signed-off-by: NDaniel P. Berrange <berrange@redhat.com>

I noticed when writing the backend functions for virNetworkUpdate that
I was repeating the same sequence of memmove, VIR_REALLOC, nXXX-- (and
messed up the args to memmove at least once), and had seen the same
sequence in a lot of other places, so I decided to write a few
utility functions/macros - see the .h file for full documentation.

The intent is to reduce the number of lines of code, but more
importantly to eliminate the need to check the element size and
element count arithmetic every time we need to do this (I *always*
make at least one mistake.)

VIR_INSERT_ELEMENT: insert one element at an arbitrary index within an
  array of objects. The size of each object is determined
  automatically by the macro using sizeof(*array). The new element's
  contents are copied into the inserted space, then the original copy
  of contents are 0'ed out (if everything else was
  successful). Compile-time assignment and size compatibility between
  the array and the new element is guaranteed (see explanation below
  [*])

VIR_INSERT_ELEMENT_COPY: identical to VIR_INSERT_ELEMENT, except that
  the original contents of newelem are not cleared to 0 (i.e. a copy
  is made).

VIR_APPEND_ELEMENT: This is just a special case of VIR_INSERT_ELEMENT
  that "inserts" one past the current last element.

VIR_APPEND_ELEMENT_COPY: identical to VIR_APPEND_ELEMENT, except that
  the original contents of newelem are not cleared to 0 (i.e. a copy
  is made).

VIR_DELETE_ELEMENT: delete one element at an arbitrary index within an
  array of objects. It's assumed that the element being deleted is
  already saved elsewhere (or cleared, if that's what is appropriate).

All five of these macros have an _INPLACE variant, which skips the
memory re-allocation of the array, assuming that the caller has
already done it (when inserting) or will do it later (when deleting).

Note that VIR_DELETE_ELEMENT* can return a failure, but only if an
invalid index is given (index + amount to delete is > current array
size), so in most cases you can safely ignore the return (that's why
the helper function virDeleteElementsN isn't declared with
ATTRIBUTE_RETURN_CHECK). A warning is logged if this ever happens,
since it is surely a coding error.

[*] One initial problem with the INSERT and APPEND macros was that,
due to both the array pointer and newelem pointer being cast to void*
when passing to virInsertElementsN(), any chance of type-checking was
lost. If we were going to move in newelem with a memmove anyway, we
would be no worse off for this. However, most current open-coded
insert/append operations use direct struct assignment to move the new
element into place (or just populate the new element directly) - thus
use of the new macros would open a possibility for new usage errors
that didn't exist before (e.g. accidentally sending &newelemptr rather
than newelemptr - I actually did this quite a lot in my test
conversions of existing code).

But thanks to Eric Blake's clever thinking, I was able to modify the
INSERT and APPEND macros so that they *do* check for both assignment
and size compatibility of *ptr (an element in the array) and newelem
(the element being copied into the new position of the array). This is
done via clever use of the C89-guaranteed fact that the sizeof()
operator must have *no* side effects (so an assignment inside sizeof()
is checked for validity, but not actually evaluated), and the fact
that virInsertElementsN has a "# of new elements" argument that we
want to always be 1.

Commit 71d12562 tried to fix a problem where rebasing an old
branch on top of newer libvirt.git resulted in automake failing
because of a missing AUTHORS file.  However, while the fix
worked for an incremental 'make', it did not work for someone
that directly reran './autogen.sh'.  Reported by Laine Stump.

* autogen.sh (autoreconf): Check for same conditions as cfg.mk.
* cfg.mk (_update_required): Add comments.

Ever since commit 7b21981c started generating AUTHORS, we now have
the situation that if you flip between two branches in the same
git repository that cross that commit boundary, then 'make' will
fail due to automake complaining about AUTHORS not existing.  The
simplest solution is to realize that if AUTHORS does not exist,
then we flipped branches so we will need to rerun bootstrap
anyways; and rerunning bootstrap ensures AUTHORS will exist in time.

* cfg.mk (_update_required): Also depend on AUTHORS.

This documents the following whitespace rules

      if(foo)   // Bad
      if (foo)  // Good

      int foo (int wizz)  // Bad
      int foo(int wizz)   // Good

      bar = foo (wizz);  // Bad
      bar = foo(wizz);   // Good

      typedef int (*foo) (int wizz);  // Bad
      typedef int (*foo)(int wizz);   // Good

      int foo( int wizz );  // Bad
      int foo(int wizz);    // Good

There is a syntax-check rule extension to validate all these rules.
Checking for 'function (...args...)' is quite difficult since it
needs to ignore valid usage with keywords like 'if (...test...)'
and while/for/switch. It must also ignore source comments and
quoted strings.

It is not possible todo this with a simple regex in the normal
syntax-check style. So a short Perl script is created instead
to analyse the source. In practice this works well enough. The
only thing it can't cope with is multi-line quoted strings of
the form

 "start of string\
more lines\
more line\
the end"

but this can and should be written as

 "start of string"
 "more lines"
 "more line"
 "the end"

with this simple change, the bracket checking script does not
have any false positives across libvirt source, provided it
is only run against .c files. It is not practical to run it
against .h files, since those use whitespace extensively to
get alignment (though this is somewhat inconsistent and could
arguably be fixed).

The only limitation is that it cannot detect a violation where
the first arg starts with a '*', eg

   foo(*wizz);

since this generates too many false positives on function
typedefs which can't be supressed efficiently.
Signed-off-by: NDaniel P. Berrange <berrange@redhat.com>

Now that the offending code was removed, we may remove this as well.

With our fix of mkostemp (pushed as 2b435c15) we define a macro
to compile with uclibc. However, this definition is conditional
and thus needs to be properly indented. Moreover, with this definition
sc_prohibit_mkstemp syntax-check rule keeps yelling:

  src/util/logging.c:63:# define mkostemp(x,y) mkstemp(x)
  maint.mk: use mkostemp with O_CLOEXEC instead of mkstemp

Therefore we should ignore this file for this rule.

https://bugzilla.redhat.com/show_bug.cgi?id=871756

Commit cd1e8d1c assumed that systems new enough to have journald
also have mkostemp; but this is not true for uclibc.

For that matter, use of mkstemp[s] is unsafe in a multi-threaded
program.  We should prefer mkostemp[s] in the first place.

* bootstrap.conf (gnulib_modules): Add mkostemp, mkostemps; drop
mkstemp and mkstemps.
* cfg.mk (sc_prohibit_mkstemp): New syntax check.
* tools/virsh.c (vshEditWriteToTempFile): Adjust caller.
* src/qemu/qemu_driver.c (qemuDomainScreenshot)
(qemudDomainMemoryPeek): Likewise.
* src/secret/secret_driver.c (replaceFile): Likewise.
* src/vbox/vbox_tmpl.c (vboxDomainScreenshot): Likewise.

This patch resolves: https://bugzilla.redhat.com/show_bug.cgi?id=871201

If libvirt is restarted after updating the dnsmasq or radvd packages,
a subsequent "virsh net-destroy" will fail to kill the dnsmasq/radvd
process.

The problem is that when libvirtd restarts, it re-reads the dnsmasq
and radvd pidfiles, then does a sanity check on each pid it finds,
including checking that the symbolic link in /proc/$pid/exe actually
points to the same file as the path used by libvirt to execute the
binary in the first place. If this fails, libvirt assumes that the
process is no longer alive.

But if the original binary has been replaced, the link in /proc is set
to "$binarypath (deleted)" (it literally has the string " (deleted)"
appended to the link text stored in the filesystem), so even if a new
binary exists in the same location, attempts to resolve the link will
fail.

In the end, not only is the old dnsmasq/radvd not terminated when the
network is stopped, but a new dnsmasq can't be started when the
network is later restarted (because the original process is still
listening on the ports that the new process wants).

The solution is, when the initial "use stat to check for identical
inodes" check for identity between /proc/$pid/exe and $binpath fails,
to check /proc/$pid/exe for a link ending with " (deleted)" and if so,
truncate that part of the link and compare what's left with the
original binarypath.

A twist to this problem is that on systems with "merged" /sbin and
/usr/sbin (i.e. /sbin is really just a symlink to /usr/sbin; Fedora
17+ is an example of this), libvirt may have started the process using
one path, but /proc/$pid/exe lists a different path (indeed, on F17
this is the case - libvirtd uses /sbin/dnsmasq, but /proc/$pid/exe
shows "/usr/sbin/dnsmasq"). The further bit of code to resolve this is
to call virFileResolveAllLinks() on both the original binarypath and
on the truncated link we read from /proc/$pid/exe, and compare the
results.

The resulting code still succeeds in all the same cases it did before,
but also succeeds if the binary was deleted or replaced after it was
started.

AUTHORS.in tracks the maintainers, as well as some folks who were
previously in AUTHORS but don't have a git commit with proper
attribution.

Generated output is sorted alphabetically and lacks pretty spacing, so
tweak AUTHORS.in to follow the same format.

Additionally, drop the syntax-check rule that previously validated
AUTHORS against git log.

Several people have reported that if the .gnulib submodule is dirty,
then 'make' will go into an infinite loop attempting to rerun bootstrap,
because that never cleans up the dirty submodule.  By default, we
should halt and make the user investigate, but if the user doesn't
know why or care that the submodule is dirty, I also added the ability
to 'make CLEAN_SUBMODULE=1' to get things going again.

Also, while testing this, I noticed that when a submodule update was
needed, 'make' would first run autoreconf, then bootstrap (which
reruns autoreconf); adding a strategic dependency allows for less work.

* .gnulib: Update to latest, for maint.mk improvements.
* cfg.mk (_autogen): Also hook maint.mk, to run before autoreconf.
* autogen.sh (bootstrap): Refuse to run if gnulib is dirty, unless
user requests discarding gnulib changes.

Two more problems in "run.in" made the syntax-check fail.

I got an off-list report about a bad diagnostic:
Target network card mac 52:54:00:49:07:ccdoes not match source 52:54:00:49:07:b8

True to form, I've added a syntax check rule to prevent it
from recurring, and found several other offenders.

* cfg.mk (sc_require_whitespace_in_translation): New rule.
* src/conf/domain_conf.c (virDomainNetDefCheckABIStability): Add
space.
* src/esx/esx_util.c (esxUtil_ParseUri): Likewise.
* src/qemu/qemu_command.c (qemuCollectPCIAddress): Likewise.
* src/qemu/qemu_driver.c (qemuDomainSetMetadata)
(qemuDomainGetMetadata): Likewise.
* src/qemu/qemu_hotplug.c (qemuDomainChangeNetBridge): Likewise.
* src/rpc/virnettlscontext.c
(virNetTLSContextCheckCertDNWhitelist): Likewise.
* src/vmware/vmware_driver.c (vmwareDomainResume): Likewise.
* src/vbox/vbox_tmpl.c (vboxDomainGetXMLDesc, vboxAttachDrives):
Avoid false negatives.
* tools/virsh-domain.c (info_save_image_dumpxml): Reword.
Based on a report by Luwen Su.

To be able to test the QEMU monitor code, we need to have a fake
QEMU monitor server. This introduces a simple (dumb) framework
that can do this. The test case registers a series of items to
be sent back as replies to commands that will be executed. A
thread runs the event loop looking for incoming replies and
sending back this pre-registered data. This allows testing all
QEMU monitor code that deals with parsing responses and errors
from QEMU, without needing QEMU around
Signed-off-by: NDaniel P. Berrange <berrange@redhat.com>

Based on the similar gnulib commit 96ad9077.  The use of
$(_sc_search_regexp) already injects $(ME) into any output
messages, so a failure of these rules would look like this,
pre-patch:

maint.mk: maint.mk: use virStrToLong_*, not strtol variants

* cfg.mk (sc_prohibit_strncmp, sc_prohibit_strtol)
(sc_libvirt_unmarked_diagnostics): Drop redundant $(ME).

Yesterday's commit 15d2c9fa pointed out that virsh was still using
localtime(), which is not thread-safe, even though virsh is
definitely multi-threaded.  Even if we only ever triggered it from
one thread, it's better safe than sorry for maintenance purposes.

* cfg.mk (exclude_file_name_regexp--sc_prohibit_nonreentrant):
Tighten the rule.
* tools/virsh.c (vshOutputLogFile): Avoid localtime.
(vshEditWriteToTempFile, vshEditReadBackFile, cmdCd, cmdPwd)
(vshCloseLogFile): Avoid strerror.
* tools/console.c (vshMakeStdinRaw): Likewise.
* tools/virsh-domain.c (vshGenFileName): Fix spacing in previous
patch.

Nothing in the testsuite or examples directory should be translated,
as it is not part of the normally installed binary.  We already
meet this rule, but enforcing it will make it easier to remember.

Suggested by Daniel P. Berrange.

* cfg.mk (sc_prohibit_useless_translation): Enhance rule.