Test filling of nodeinfo structure if /sys/devices/system/node does not
exist. (Based on dump from a real machine)

This patch adds test data that describe a machine that has two physical
processors that don't share same core id's on their cores. On this data
the "virsh nodeinfo" reported that the machine had 10 cores per socket
while the processor had only 8. (Before fixing nodeinfo gathering code).

Make it obvious why we need Osier's patch in commit 10d9038b
to fix NUMA parsing of an AMD machine with two cores sharing
a socket id.

* tests/nodeinfotest.c (linuxTestCompareFiles): Enhance the test.
* tests/nodeinfodata/linux-nodeinfo-sysfs-test-*-output.txt: Update.

Test 2 data grabbed from a 2-core 1-node laptop.
Test 3 data grabbed from a 48-cpu AMD Magny Cours box.

* tests/nodeinfodata/linux-nodeinfo-sysfs-test-2*: New test data.
* tests/nodeinfodata/linux-nodeinfo-sysfs-test-3*: Likewise.
* tests/nodeinfotest.c (mymain): Run them.
* cfg.mk
(exclude_file_name_regexp--sc_prohibit_empty_lines_at_EOF): Exempt
new test files.

We had previously weakened our nodeinfotest in order to ignore parsed
node values, because the parse function was mistakenly relying on
host files.  A better fix is to avoid using the numactl library, but
to instead parse the same files that numactl would read, all while
allowing the files to be relative to our choice of directory.

* src/nodeinfo.c (CPU_SYS_PATH, NODE_SYS_PATH): Replace with...
(SYSFS_SYSTEM_PATH): ...parent directory.
(linuxNodeInfoCPUPopulate): Check NUMA nodes from requested
directory (by inlining numactl code).
(nodeGetCPUmap, nodeGetMemoryStats): Adjust macro use.
* tests/nodeinfotest.c (linuxTestCompareFiles, linuxTestNodeInfo):
Update test to match.

Return statements with parameter enclosed in parentheses were modified
and parentheses were removed. The whole change was scripted, here is how:

List of files was obtained using this command:
git grep -l -e '\<return\s*([^()]*\(([^()]*)[^()]*\)*)\s*;' |             \
grep -e '\.[ch]$' -e '\.py$'

Found files were modified with this command:
sed -i -e                                                                 \
's_^\(.*\<return\)\s*(\(\([^()]*([^()]*)[^()]*\)*\))\s*\(;.*$\)_\1 \2\4_' \
-e 's_^\(.*\<return\)\s*(\([^()]*\))\s*\(;.*$\)_\1 \2\3_'

Then checked for nonsense.

The whole command looks like this:
git grep -l -e '\<return\s*([^()]*\(([^()]*)[^()]*\)*)\s*;' |             \
grep -e '\.[ch]$' -e '\.py$' | xargs sed -i -e                            \
's_^\(.*\<return\)\s*(\(\([^()]*([^()]*)[^()]*\)*\))\s*\(;.*$\)_\1 \2\4_' \
-e 's_^\(.*\<return\)\s*(\([^()]*\))\s*\(;.*$\)_\1 \2\3_'

The bulk of this patch was done with:

sed -i 's/\(\bfree *(/VIR_FREE(/g' tests/*.c

followed by fixing the few compile errors that resulted.

* cfg.mk (exclude_file_name_regexp--sc_prohibit_raw_allocation):
Remove tests from exemption.
* tests/testutils.h: Add common header.
* tests/commandhelper.c: Fix offenders.
* tests/cputest.c: Likewise.
* tests/domainsnapshotxml2xmltest.c: Likewise.
* tests/interfacexml2xmltest.c: Likewise.
* tests/networkxml2argvtest.c: Likewise.
* tests/networkxml2xmltest.c: Likewise.
* tests/nodedevxml2xmltest.c: Likewise.
* tests/nodeinfotest.c: Likewise.
* tests/nwfilterxml2xmltest.c: Likewise.
* tests/qemuargv2xmltest.c: Likewise.
* tests/qemuxml2argvtest.c: Likewise.
* tests/qemuxml2xmltest.c: Likewise.
* tests/qemuxmlnstest.c: Likewise.
* tests/qparamtest.c: Likewise.
* tests/sexpr2xmltest.c: Likewise.
* tests/storagepoolxml2xmltest.c: Likewise.
* tests/storagevolxml2xmltest.c: Likewise.
* tests/testutils.c: Likewise.
* tests/virshtest.c: Likewise.
* tests/xencapstest.c: Likewise.
* tests/xmconfigtest.c: Likewise.
* tests/xml2sexprtest.c: Likewise.

Detected by valgrind. Leak introduced in commit 82ff25e1.

* tests/nodeinfotest.c: avoid memory leak on nodeinfo test case.

* how to reproduce?
  % cd tests && valgrind -v --leak-check=full ./nodeinfotest

* actual valgrind result:

==22147== 65 bytes in 1 blocks are definitely lost in loss record 14 of 29
==22147==    at 0x4A0610F: realloc (vg_replace_malloc.c:525)
==22147==    by 0x330D6FED94: __vasprintf_chk (in /lib64/libc-2.12.so)
==22147==    by 0x426697: virVasprintf (stdio2.h:199)
==22147==    by 0x426757: virAsprintf (util.c:1695)
==22147==    by 0x41585F: linuxTestNodeInfo (nodeinfotest.c:108)
==22147==    by 0x416B21: virtTestRun (testutils.c:141)
==22147==    by 0x4157EA: mymain (nodeinfotest.c:140)
==22147==    by 0x416217: virtTestMain (testutils.c:696)
==22147==    by 0x330D61ECDC: (below main) (in /lib64/libc-2.12.so)
==22147==
==22147== LEAK SUMMARY:
==22147==    definitely lost: 65 bytes in 1 blocks
==22147==    indirectly lost: 0 bytes in 0 blocks
==22147==      possibly lost: 0 bytes in 0 blocks
==22147==    still reachable: 126,126 bytes in 1,341 blocks
Signed-off-by: NAlex Jia <ajia@redhat.com>

 to proc/cpuinfo

This patch creates a new sysfs hierarchy under
tests/nodeinfodata/linux-nodeinfo-sysfs-test-1.
Output files and /proc/cpuinfo files are also respectively added for
both x86 and ppc64.
Signed-off-by: NPrerna Saxena <prerna@linux.vnet.ibm.com>

Prefer 'return EXIT_AM_SKIP' over 'exit(EXIT_AM_SKIP)'.

Prefer 'int main(void)' over 'int main(int argc, char **argv)'.

Fix mymain signature in commandtest and nodeinfotest.

In preparation for a future patch adding new virFile APIs.

* src/util/files.h, src/util/files.c: Move...
* src/util/virfile.h, src/util/virfile.c: ...here, and rename
functions to virFile prefix.  Macro names are intentionally
left alone.
* *.c: All '#include "files.h"' uses changed.
* src/Makefile.am (UTIL_SOURCES): Reflect rename.
* cfg.mk (exclude_file_name_regexp--sc_prohibit_close): Likewise.
* src/libvirt_private.syms: Likewise.
* docs/hacking.html.in: Likewise.
* HACKING: Regenerate.

since the testfiles assume a /proc/cpuinfo specific to this
architecture. We e.g. can't parse the number of cores on other
architectures.

Make virtTestLoadFile allocate the buffer to read the file into.

Fix logic error in virtTestLoadFile, stop reading on the first empty line.

Use virFileReadLimFD in virtTestCaptureProgramOutput to avoid manual
buffer handling.

A few of the tests were missing basic sanity checks, while most
of them were doing copy-and-paste initialization (in fact, some
of them pasted the argc > 1 check more than once!).  It's much
nicer to do things in one common place, and minimizes the size of
the next patch that fixes getcwd usage.

* tests/testutils.h (EXIT_AM_HARDFAIL): New define.
(progname, abs_srcdir): Define for all tests.
(VIRT_TEST_MAIN): Change callback signature.
* tests/testutils.c (virtTestMain): Do more common init.
* tests/commandtest.c (mymain): Simplify.
* tests/cputest.c (mymain): Likewise.
* tests/esxutilstest.c (mymain): Likewise.
* tests/eventtest.c (mymain): Likewise.
* tests/hashtest.c (mymain): Likewise.
* tests/networkxml2xmltest.c (mymain): Likewise.
* tests/nodedevxml2xmltest.c (myname): Likewise.
* tests/nodeinfotest.c (mymain): Likewise.
* tests/nwfilterxml2xmltest.c (mymain): Likewise.
* tests/qemuargv2xmltest.c (mymain): Likewise.
* tests/qemuhelptest.c (mymain): Likewise.
* tests/qemuxml2argvtest.c (mymain): Likewise.
* tests/qemuxml2xmltest.c (mymain): Likewise.
* tests/qparamtest.c (mymain): Likewise.
* tests/sexpr2xmltest.c (mymain): Likewise.
* tests/sockettest.c (mymain): Likewise.
* tests/statstest.c (mymain): Likewise.
* tests/storagepoolxml2xmltest.c (mymain): Likewise.
* tests/storagevolxml2xmltest.c (mymain): Likewise.
* tests/virbuftest.c (mymain): Likewise.
* tests/virshtest.c (mymain): Likewise.
* tests/vmx2xmltest.c (mymain): Likewise.
* tests/xencapstest.c (mymain): Likewise.
* tests/xmconfigtest.c (mymain): Likewise.
* tests/xml2sexprtest.c (mymain): Likewise.
* tests/xml2vmxtest.c (mymain): Likewise.

When running 'make check' under a multi-cpu Dom0 xen machine,
nodeinfotest had a spurious failure it was reading from
/sys/devices/system/cpu, but xen has no notion of topology.  The test
was intended to be isolated from reading any real system files; the
regression was introduced in Mar 2010 with commit aa2f6f96.

Fix things by allowing an early exit for the testsuite.

* src/nodeinfo.c (linuxNodeInfoCPUPopulate): Add parameter.
(nodeGetInfo): Adjust caller.
* tests/nodeinfotest.c (linuxTestCompareFiles): Likewise.

Similarly to deprecating close(), I am now deprecating fclose() and
introduce VIR_FORCE_FCLOSE() and VIR_FCLOSE(). Also, fdopen() is replaced with
VIR_FDOPEN().

Most of the files are opened in read-only mode, so usage of
VIR_FORCE_CLOSE() seemed appropriate. Others that are opened in write
mode already had the fclose()<  0 check and I converted those to
VIR_FCLOSE()<  0.

I did not find occurrences of possible double-closed files on the way.

If linuxNodeInfoCPUPopulate() fails, the test would just print "FAILED"
which is not very informative. It's better to print the real error.

* tests/nodeinfotest.c (mymain): Do not ignore virInitialize failure.
Most other callers of virInitialize test for failure.

* tests/nodeinfotest.c (linuxTestCompareFiles): Don't use
nodeinfo->member uninitialized.  linuxNodeInfoCPUPopulate requires
that some of its nodeinfo members (including threads) be initialized
upon input.  The nodeinfotest.c program lacked the initialization,
while the only other use (nodeGetInfo) did perform it.
It's not trivial to move the initialization into the function,
since nodeGetInfo sets at least one member after clearing the
buffer but before calling linuxNodeInfoCPUPopulate.

The nodeinfotest was reliant on the host NUMA topology, but all
the test data files assumed 1 single NUMA node. This test thus
failed on any NUMA machine with > 1 node

* tests/nodeinfotest.c: Hardcode 1 single numa node

* global: patch created by running:
for f in $(git ls-files '*.[ch]') ; do
    cppi $f > $f.t && mv $f.t $f
done

The current code for "nodeinfo" is pretty naive
about socket and thread information.  To determine the
sockets, it just takes the number of cpus and divides
by the number of cores.  For the thread count, it always
sets it to 1.  With more recent Intel machines, however,
hyperthreading is again an option, meaning that these
heuristics no longer work and give bogus numbers.  This
patch goes through /sys to get the additional
information so we properly report it.

Note that I had to edit the tests not to report on
socket and thread counts, since these are determined
dynamically now.

v2: As pointed out by Eric Blake, gnulib provides
    count-one-bits (which is LGPLv2+).  Use it instead
    of a hand-coded popcnt.
Signed-off-by: NChris Lalancette <clalance@redhat.com>

This appeases a new gnulib-provided "syntax-check".
* daemon/libvirtd.c (main): Use EXIT_FAILURE, not 1.
* proxy/libvirt_proxy.c (main): Likewise, and EXIT_SUCCESS, not 0.
* tests/conftest.c (main): Likewise.
* tests/reconnect.c (main): Likewise.
* tests/testutils.h (EXIT_AM_SKIP): Define.
* tests/nodeinfotest.c (mymain): Use EXIT_AM_SKIP, not 77.
* tests/qemuargv2xmltest.c: Likewise.
* tests/qemuxml2xmltest.c: Likewise.
* tests/virshtest.c (mymain): Likewise.

by running this command:
git ls-files -z | xargs -0 perl -pi -0777 -e 's/\n\n+$/\n/'
This is in preparation for a more strict make syntax-check
rule that will detect trailing blank lines.

* tests/virshtest.c (mymain): Use only one definition of this function.
[WIN32]: Call exit(77) to skip the test.
* tests/nodeinfotest.c: Likewise.

* tests/nodeinfotest.c: Include "util.h".
* src/xen_internal.c: Likewise.
* src/qemu_conf.c (qemudCapsInit): Use ARRAY_CARDINALITY.
* src/storage_backend.c (virStorageBackendForType): Likewise.
* src/storage_backend_fs.c (virStorageBackendProbeFile): Likewise.
* src/xen_internal.c (xenHypervisorMakeCapabilitiesInternal): Likewise.
* src/xend_internal.c (xenDaemonParseSxpr): Likewise.
* tests/nodeinfotest.c (mymain): Likewise.
* tests/qparamtest.c (DO_TEST): Likewise.

Done using this command (also includes .c.in and .h.in files):
for i in $(g ls-files|grep -E '\.[ch](\.in)?$'|grep -v gnulib); do
  expand -i $i > j && mv j $i;done

Done with these commands:
git grep -l Local.variab|xargs \
  perl -0x3b -pi -e 's,\n+/\*\n \* vim:(.|\n)*,\n,'

git grep -l Local.variab|xargs \
  perl -0x3b -pi -e 's,\n+/\*\n \* Local variables:\n(.|\n)*,\n,'

Use <config.h>, not "config.h", per autoconf documentation.
* Makefile.cfg (local-checks-to-skip) [sc_require_config_h]: Enable.
* .x-sc_require_config_h: New file, to list exempted files.
* Makefile.am (EXTRA_DIST): Add .x-sc_require_config_h.

New files go into these directories:
  gnulib/lib
  gnulib/m4
  gnulib/tests

* bootstrap: A wrapper around gnulib-tool.
* configure.in: Invoke gl_EARLY and gl_INIT, being careful to put gl_EARLY
before any macro that uses AC_COMPILE_IFELSE.
(AC_OUTPUT): Add lib/Makefile and gl-tests/Makefile.  Remove m4/Makefile.
* Makefile.am (SUBDIRS): Add gnulib/lib and remove m4.  Add gnulib/tests
early enough that those tests run before any libvirt unit tests.
* m4/Makefile.am: Remove file.  Not needed.
* src/Makefile.am (INCLUDES): Add -I$(top_srcdir)/gnulib/lib -I../gnulib/lib.
(LDADDS, libvirt_la_LIBADD): Add ../gnulib/lib/libgnu.la.
* src/nodeinfo.c: Include "physmem.h".
* qemud/qemud.c, src/remote_internal.c: Include "getaddrinfo.h".
(MEMINFO_PATH, linuxNodeInfoMemPopulate): Remove definitions.
(virNodeInfoPopulate): Use physmem_total, not linuxNodeInfoMemPopulate.
* tests/Makefile.am (INCLUDES): Add -I$(top_srcdir)/gnulib/lib -I../gnulib/lib.
(LDADDS): Add ../gnulib/lib/libgnu.la.
* qemud/Makefile.am (libvirtd_LDADD): Add ../gnulib/lib/libgnu.la.
* tests/nodeinfotest.c (linuxTestCompareFiles): No longer read total
memory from a file.
Update expected output not to include "Memory: NNNN"
* tests/nodeinfodata/linux-nodeinfo-1.txt:
* tests/nodeinfodata/linux-nodeinfo-2.txt:
* tests/nodeinfodata/linux-nodeinfo-3.txt:
* tests/nodeinfodata/linux-nodeinfo-4.txt:
* tests/nodeinfodata/linux-nodeinfo-5.txt:
* tests/nodeinfodata/linux-nodeinfo-6.txt:
* src/test.c [WITH_TEST]: Remove definition of _GNU_SOURCE that
would conflict with the one now in "config.h".
* autogen.sh: Add -I gnulib/m4.
* src/conf.c, src/sexpr.c: Don't define _GNU_SOURCE.
Instead, include "config.h".
* qemud/qemud.c: Remove definition of _GNU_SOURCE.
* src/openvz_driver.c: Likewise.
* src/qemu_driver.c: Likewise.
* src/remote_internal.c: Likewise.

* configure.in: Use AC_CONFIG_AUX_DIR(build-aux), so that a bunch
of gettextize-generated files go into build-aux/, rather than in
the top-level directory.
* .cvsignore: Adjust.
* build-aux/.cvsignore: New file.


Author: Jim Meyering <meyering@redhat.com>

        * python/libvir.c, python/libvirt_wrap.h, qemud/qemud.c,
        qemud/remote.c, src/internal.h, src/openvz_conf.c,
        src/openvz_driver.c, src/proxy_internal.h, src/qemu_conf.c,
        src/qemu_driver.c, src/remote_internal.h, src/test.h, src/util.c,
        src/xen_unified.c, src/xen_unified.h, tests/nodeinfotest.c,
        tests/qemuxml2argvtest.c, tests/qemuxml2xmltest.c, tests/reconnect.c,
        tests/sexpr2xmltest.c, tests/virshtest.c, tests/xencapstest.c,
        tests/xmconfigtest.c, tests/xml2sexprtest.c:
        Change #include <> to #include "" for local includes.
        Removed many includes from src/internal.h and put them in
        the C files which actually use them.
        Removed <ansidecl.h> - unused.
        Added a comment around __func__.
        Removed a clashing redefinition of VERSION symbol.
        All limits (PATH_MAX etc) now done in src/internal.h, so we
        don't need to include those headers in other files.

	* tests/Makefile.am, tests/nodeinfotest.c, tests/qemuxml2argvtest.c,
	  tests/qemuxml2xmltest.c, tests/sexpr2xmltest.c, tests/virshtest.c,
	  tests/xencapstest.c, tests/xmconfigtest.c, tests/xml2sexprtest.c:
	  Miscellaneous fixes to the tests to compile under Cygwin.

* tests/Makefile.am: Include the contents of the *data directories
in the make-dist-built tarball by adding each of that *data
directories to EXTRA_DIST.
Also add int-overflow (via $(test_scripts)) to EXTRA_DIST.
* tests/nodeinfotest.c: Prepend "$abs_top_srcdir/tests" to
each input file name.
* tests/qemuxml2argvtest.c: Likewise.
* tests/qemuxml2xmltest.c: Likewise.
* tests/sexpr2xmltest.c: Likewise.
* tests/test_conf.sh: Likewise.
* tests/virshtest.c: Likewise.
* tests/xencapstest.c: Likewise.
* tests/xmconfigtest.c: Likewise.
* tests/xml2sexprtest.c: Likewise.


Author: Jim Meyering <meyering@redhat.com>