The AMD Bulldozer architecture uses so called "Clustered integer core
modules" that count both as threads and cores. This patch expects the
cpu to be detected using the new fallback condition otherwise twice the
number of processors would be detected.

This test data was gathered on an AMD MagnyCours machine that reports it
has only one NUMA node although the hardware is consisting of 4. As
duplicate core id's are ignored the reported topology was bogous. This
should be fixed by the previous patch.

Reported and data provided by George-Cristian Bîrzan.

Commit ddd6bef4 switched to the ustar format to fix an issue where
'make dist' fails to create a tarball because we have files with
relative names longer than 100 bytes by the time you include a
'libvirt-0.9.13' prefix.  Unfortunately, even with ustar format,
the use of 'tar -ch' tries to convert symlinks to hard links,
also with a name too long (omitting the -h works, but automake
automatically passes -h); such symlinks were added in commit
6dcf98c8, which resulted in 'make dist' breaking again.  The
solution is to rename the offending symlinks to something shorter,
by shortening the entire nodeinfodata naming scheme.

* tests/nodeinfotest.c (mymain): Shorten test names.
(linuxTestNodeInfo): Accommodate new names.
* tests/nodeinfodata/*: Rename files accordingly.

Commit 80533ca2 forgot to think about offline cpus.  When a node
cpu is offline, then its topology/ subdirectory is not present,
leading to spurious error messages leaked to the user such as:

libvir:  error : cannot open /home/dummy/libvirt/tests/nodeinfodata/linux-nodeinfo-sysfs-test-6/node/node0/cpu7/topology/physical_package_id: No such file or directory

Fix that, as well as test it; the test data is gathered from a
machine with one NUMA node, hyperthreading, and with 2 of the
8 cpus offline.

* src/nodeinfo.c (virNodeParseNode): Don't parse topology of
offline cpus.
* tests/nodeinfotest.c (mymain): Run new test.
* tests/nodeinfodata/linux-nodeinfo-sysfs-test-6*: New data.

Test filling of nodeinfo structure if /sys/devices/system/node does not
exist. (Based on dump from a real machine)

This patch adds test data that describe a machine that has two physical
processors that don't share same core id's on their cores. On this data
the "virsh nodeinfo" reported that the machine had 10 cores per socket
while the processor had only 8. (Before fixing nodeinfo gathering code).

Make it obvious why we need Osier's patch in commit 10d9038b
to fix NUMA parsing of an AMD machine with two cores sharing
a socket id.

* tests/nodeinfotest.c (linuxTestCompareFiles): Enhance the test.
* tests/nodeinfodata/linux-nodeinfo-sysfs-test-*-output.txt: Update.

Test 2 data grabbed from a 2-core 1-node laptop.
Test 3 data grabbed from a 48-cpu AMD Magny Cours box.

* tests/nodeinfodata/linux-nodeinfo-sysfs-test-2*: New test data.
* tests/nodeinfodata/linux-nodeinfo-sysfs-test-3*: Likewise.
* tests/nodeinfotest.c (mymain): Run them.
* cfg.mk
(exclude_file_name_regexp--sc_prohibit_empty_lines_at_EOF): Exempt
new test files.

We had previously weakened our nodeinfotest in order to ignore parsed
node values, because the parse function was mistakenly relying on
host files.  A better fix is to avoid using the numactl library, but
to instead parse the same files that numactl would read, all while
allowing the files to be relative to our choice of directory.

* src/nodeinfo.c (CPU_SYS_PATH, NODE_SYS_PATH): Replace with...
(SYSFS_SYSTEM_PATH): ...parent directory.
(linuxNodeInfoCPUPopulate): Check NUMA nodes from requested
directory (by inlining numactl code).
(nodeGetCPUmap, nodeGetMemoryStats): Adjust macro use.
* tests/nodeinfotest.c (linuxTestCompareFiles, linuxTestNodeInfo):
Update test to match.

Return statements with parameter enclosed in parentheses were modified
and parentheses were removed. The whole change was scripted, here is how:

List of files was obtained using this command:
git grep -l -e '\<return\s*([^()]*\(([^()]*)[^()]*\)*)\s*;' |             \
grep -e '\.[ch]$' -e '\.py$'

Found files were modified with this command:
sed -i -e                                                                 \
's_^\(.*\<return\)\s*(\(\([^()]*([^()]*)[^()]*\)*\))\s*\(;.*$\)_\1 \2\4_' \
-e 's_^\(.*\<return\)\s*(\([^()]*\))\s*\(;.*$\)_\1 \2\3_'

Then checked for nonsense.

The whole command looks like this:
git grep -l -e '\<return\s*([^()]*\(([^()]*)[^()]*\)*)\s*;' |             \
grep -e '\.[ch]$' -e '\.py$' | xargs sed -i -e                            \
's_^\(.*\<return\)\s*(\(\([^()]*([^()]*)[^()]*\)*\))\s*\(;.*$\)_\1 \2\4_' \
-e 's_^\(.*\<return\)\s*(\([^()]*\))\s*\(;.*$\)_\1 \2\3_'

The bulk of this patch was done with:

sed -i 's/\(\bfree *(/VIR_FREE(/g' tests/*.c

followed by fixing the few compile errors that resulted.

* cfg.mk (exclude_file_name_regexp--sc_prohibit_raw_allocation):
Remove tests from exemption.
* tests/testutils.h: Add common header.
* tests/commandhelper.c: Fix offenders.
* tests/cputest.c: Likewise.
* tests/domainsnapshotxml2xmltest.c: Likewise.
* tests/interfacexml2xmltest.c: Likewise.
* tests/networkxml2argvtest.c: Likewise.
* tests/networkxml2xmltest.c: Likewise.
* tests/nodedevxml2xmltest.c: Likewise.
* tests/nodeinfotest.c: Likewise.
* tests/nwfilterxml2xmltest.c: Likewise.
* tests/qemuargv2xmltest.c: Likewise.
* tests/qemuxml2argvtest.c: Likewise.
* tests/qemuxml2xmltest.c: Likewise.
* tests/qemuxmlnstest.c: Likewise.
* tests/qparamtest.c: Likewise.
* tests/sexpr2xmltest.c: Likewise.
* tests/storagepoolxml2xmltest.c: Likewise.
* tests/storagevolxml2xmltest.c: Likewise.
* tests/testutils.c: Likewise.
* tests/virshtest.c: Likewise.
* tests/xencapstest.c: Likewise.
* tests/xmconfigtest.c: Likewise.
* tests/xml2sexprtest.c: Likewise.

Detected by valgrind. Leak introduced in commit 82ff25e1.

* tests/nodeinfotest.c: avoid memory leak on nodeinfo test case.

* how to reproduce?
  % cd tests && valgrind -v --leak-check=full ./nodeinfotest

* actual valgrind result:

==22147== 65 bytes in 1 blocks are definitely lost in loss record 14 of 29
==22147==    at 0x4A0610F: realloc (vg_replace_malloc.c:525)
==22147==    by 0x330D6FED94: __vasprintf_chk (in /lib64/libc-2.12.so)
==22147==    by 0x426697: virVasprintf (stdio2.h:199)
==22147==    by 0x426757: virAsprintf (util.c:1695)
==22147==    by 0x41585F: linuxTestNodeInfo (nodeinfotest.c:108)
==22147==    by 0x416B21: virtTestRun (testutils.c:141)
==22147==    by 0x4157EA: mymain (nodeinfotest.c:140)
==22147==    by 0x416217: virtTestMain (testutils.c:696)
==22147==    by 0x330D61ECDC: (below main) (in /lib64/libc-2.12.so)
==22147==
==22147== LEAK SUMMARY:
==22147==    definitely lost: 65 bytes in 1 blocks
==22147==    indirectly lost: 0 bytes in 0 blocks
==22147==      possibly lost: 0 bytes in 0 blocks
==22147==    still reachable: 126,126 bytes in 1,341 blocks
Signed-off-by: NAlex Jia <ajia@redhat.com>

 to proc/cpuinfo

This patch creates a new sysfs hierarchy under
tests/nodeinfodata/linux-nodeinfo-sysfs-test-1.
Output files and /proc/cpuinfo files are also respectively added for
both x86 and ppc64.
Signed-off-by: NPrerna Saxena <prerna@linux.vnet.ibm.com>

Prefer 'return EXIT_AM_SKIP' over 'exit(EXIT_AM_SKIP)'.

Prefer 'int main(void)' over 'int main(int argc, char **argv)'.

Fix mymain signature in commandtest and nodeinfotest.

In preparation for a future patch adding new virFile APIs.

* src/util/files.h, src/util/files.c: Move...
* src/util/virfile.h, src/util/virfile.c: ...here, and rename
functions to virFile prefix.  Macro names are intentionally
left alone.
* *.c: All '#include "files.h"' uses changed.
* src/Makefile.am (UTIL_SOURCES): Reflect rename.
* cfg.mk (exclude_file_name_regexp--sc_prohibit_close): Likewise.
* src/libvirt_private.syms: Likewise.
* docs/hacking.html.in: Likewise.
* HACKING: Regenerate.

since the testfiles assume a /proc/cpuinfo specific to this
architecture. We e.g. can't parse the number of cores on other
architectures.

Make virtTestLoadFile allocate the buffer to read the file into.

Fix logic error in virtTestLoadFile, stop reading on the first empty line.

Use virFileReadLimFD in virtTestCaptureProgramOutput to avoid manual
buffer handling.

A few of the tests were missing basic sanity checks, while most
of them were doing copy-and-paste initialization (in fact, some
of them pasted the argc > 1 check more than once!).  It's much
nicer to do things in one common place, and minimizes the size of
the next patch that fixes getcwd usage.

* tests/testutils.h (EXIT_AM_HARDFAIL): New define.
(progname, abs_srcdir): Define for all tests.
(VIRT_TEST_MAIN): Change callback signature.
* tests/testutils.c (virtTestMain): Do more common init.
* tests/commandtest.c (mymain): Simplify.
* tests/cputest.c (mymain): Likewise.
* tests/esxutilstest.c (mymain): Likewise.
* tests/eventtest.c (mymain): Likewise.
* tests/hashtest.c (mymain): Likewise.
* tests/networkxml2xmltest.c (mymain): Likewise.
* tests/nodedevxml2xmltest.c (myname): Likewise.
* tests/nodeinfotest.c (mymain): Likewise.
* tests/nwfilterxml2xmltest.c (mymain): Likewise.
* tests/qemuargv2xmltest.c (mymain): Likewise.
* tests/qemuhelptest.c (mymain): Likewise.
* tests/qemuxml2argvtest.c (mymain): Likewise.
* tests/qemuxml2xmltest.c (mymain): Likewise.
* tests/qparamtest.c (mymain): Likewise.
* tests/sexpr2xmltest.c (mymain): Likewise.
* tests/sockettest.c (mymain): Likewise.
* tests/statstest.c (mymain): Likewise.
* tests/storagepoolxml2xmltest.c (mymain): Likewise.
* tests/storagevolxml2xmltest.c (mymain): Likewise.
* tests/virbuftest.c (mymain): Likewise.
* tests/virshtest.c (mymain): Likewise.
* tests/vmx2xmltest.c (mymain): Likewise.
* tests/xencapstest.c (mymain): Likewise.
* tests/xmconfigtest.c (mymain): Likewise.
* tests/xml2sexprtest.c (mymain): Likewise.
* tests/xml2vmxtest.c (mymain): Likewise.

When running 'make check' under a multi-cpu Dom0 xen machine,
nodeinfotest had a spurious failure it was reading from
/sys/devices/system/cpu, but xen has no notion of topology.  The test
was intended to be isolated from reading any real system files; the
regression was introduced in Mar 2010 with commit aa2f6f96.

Fix things by allowing an early exit for the testsuite.

* src/nodeinfo.c (linuxNodeInfoCPUPopulate): Add parameter.
(nodeGetInfo): Adjust caller.
* tests/nodeinfotest.c (linuxTestCompareFiles): Likewise.

Similarly to deprecating close(), I am now deprecating fclose() and
introduce VIR_FORCE_FCLOSE() and VIR_FCLOSE(). Also, fdopen() is replaced with
VIR_FDOPEN().

Most of the files are opened in read-only mode, so usage of
VIR_FORCE_CLOSE() seemed appropriate. Others that are opened in write
mode already had the fclose()<  0 check and I converted those to
VIR_FCLOSE()<  0.

I did not find occurrences of possible double-closed files on the way.

If linuxNodeInfoCPUPopulate() fails, the test would just print "FAILED"
which is not very informative. It's better to print the real error.

* tests/nodeinfotest.c (mymain): Do not ignore virInitialize failure.
Most other callers of virInitialize test for failure.

* tests/nodeinfotest.c (linuxTestCompareFiles): Don't use
nodeinfo->member uninitialized.  linuxNodeInfoCPUPopulate requires
that some of its nodeinfo members (including threads) be initialized
upon input.  The nodeinfotest.c program lacked the initialization,
while the only other use (nodeGetInfo) did perform it.
It's not trivial to move the initialization into the function,
since nodeGetInfo sets at least one member after clearing the
buffer but before calling linuxNodeInfoCPUPopulate.

The nodeinfotest was reliant on the host NUMA topology, but all
the test data files assumed 1 single NUMA node. This test thus
failed on any NUMA machine with > 1 node

* tests/nodeinfotest.c: Hardcode 1 single numa node

* global: patch created by running:
for f in $(git ls-files '*.[ch]') ; do
    cppi $f > $f.t && mv $f.t $f
done

The current code for "nodeinfo" is pretty naive
about socket and thread information.  To determine the
sockets, it just takes the number of cpus and divides
by the number of cores.  For the thread count, it always
sets it to 1.  With more recent Intel machines, however,
hyperthreading is again an option, meaning that these
heuristics no longer work and give bogus numbers.  This
patch goes through /sys to get the additional
information so we properly report it.

Note that I had to edit the tests not to report on
socket and thread counts, since these are determined
dynamically now.

v2: As pointed out by Eric Blake, gnulib provides
    count-one-bits (which is LGPLv2+).  Use it instead
    of a hand-coded popcnt.
Signed-off-by: NChris Lalancette <clalance@redhat.com>

This appeases a new gnulib-provided "syntax-check".
* daemon/libvirtd.c (main): Use EXIT_FAILURE, not 1.
* proxy/libvirt_proxy.c (main): Likewise, and EXIT_SUCCESS, not 0.
* tests/conftest.c (main): Likewise.
* tests/reconnect.c (main): Likewise.
* tests/testutils.h (EXIT_AM_SKIP): Define.
* tests/nodeinfotest.c (mymain): Use EXIT_AM_SKIP, not 77.
* tests/qemuargv2xmltest.c: Likewise.
* tests/qemuxml2xmltest.c: Likewise.
* tests/virshtest.c (mymain): Likewise.

by running this command:
git ls-files -z | xargs -0 perl -pi -0777 -e 's/\n\n+$/\n/'
This is in preparation for a more strict make syntax-check
rule that will detect trailing blank lines.

* tests/virshtest.c (mymain): Use only one definition of this function.
[WIN32]: Call exit(77) to skip the test.
* tests/nodeinfotest.c: Likewise.

* tests/nodeinfotest.c: Include "util.h".
* src/xen_internal.c: Likewise.
* src/qemu_conf.c (qemudCapsInit): Use ARRAY_CARDINALITY.
* src/storage_backend.c (virStorageBackendForType): Likewise.
* src/storage_backend_fs.c (virStorageBackendProbeFile): Likewise.
* src/xen_internal.c (xenHypervisorMakeCapabilitiesInternal): Likewise.
* src/xend_internal.c (xenDaemonParseSxpr): Likewise.
* tests/nodeinfotest.c (mymain): Likewise.
* tests/qparamtest.c (DO_TEST): Likewise.

Done using this command (also includes .c.in and .h.in files):
for i in $(g ls-files|grep -E '\.[ch](\.in)?$'|grep -v gnulib); do
  expand -i $i > j && mv j $i;done

Done with these commands:
git grep -l Local.variab|xargs \
  perl -0x3b -pi -e 's,\n+/\*\n \* vim:(.|\n)*,\n,'

git grep -l Local.variab|xargs \
  perl -0x3b -pi -e 's,\n+/\*\n \* Local variables:\n(.|\n)*,\n,'

Use <config.h>, not "config.h", per autoconf documentation.
* Makefile.cfg (local-checks-to-skip) [sc_require_config_h]: Enable.
* .x-sc_require_config_h: New file, to list exempted files.
* Makefile.am (EXTRA_DIST): Add .x-sc_require_config_h.

New files go into these directories:
  gnulib/lib
  gnulib/m4
  gnulib/tests

* bootstrap: A wrapper around gnulib-tool.
* configure.in: Invoke gl_EARLY and gl_INIT, being careful to put gl_EARLY
before any macro that uses AC_COMPILE_IFELSE.
(AC_OUTPUT): Add lib/Makefile and gl-tests/Makefile.  Remove m4/Makefile.
* Makefile.am (SUBDIRS): Add gnulib/lib and remove m4.  Add gnulib/tests
early enough that those tests run before any libvirt unit tests.
* m4/Makefile.am: Remove file.  Not needed.
* src/Makefile.am (INCLUDES): Add -I$(top_srcdir)/gnulib/lib -I../gnulib/lib.
(LDADDS, libvirt_la_LIBADD): Add ../gnulib/lib/libgnu.la.
* src/nodeinfo.c: Include "physmem.h".
* qemud/qemud.c, src/remote_internal.c: Include "getaddrinfo.h".
(MEMINFO_PATH, linuxNodeInfoMemPopulate): Remove definitions.
(virNodeInfoPopulate): Use physmem_total, not linuxNodeInfoMemPopulate.
* tests/Makefile.am (INCLUDES): Add -I$(top_srcdir)/gnulib/lib -I../gnulib/lib.
(LDADDS): Add ../gnulib/lib/libgnu.la.
* qemud/Makefile.am (libvirtd_LDADD): Add ../gnulib/lib/libgnu.la.
* tests/nodeinfotest.c (linuxTestCompareFiles): No longer read total
memory from a file.
Update expected output not to include "Memory: NNNN"
* tests/nodeinfodata/linux-nodeinfo-1.txt:
* tests/nodeinfodata/linux-nodeinfo-2.txt:
* tests/nodeinfodata/linux-nodeinfo-3.txt:
* tests/nodeinfodata/linux-nodeinfo-4.txt:
* tests/nodeinfodata/linux-nodeinfo-5.txt:
* tests/nodeinfodata/linux-nodeinfo-6.txt:
* src/test.c [WITH_TEST]: Remove definition of _GNU_SOURCE that
would conflict with the one now in "config.h".
* autogen.sh: Add -I gnulib/m4.
* src/conf.c, src/sexpr.c: Don't define _GNU_SOURCE.
Instead, include "config.h".
* qemud/qemud.c: Remove definition of _GNU_SOURCE.
* src/openvz_driver.c: Likewise.
* src/qemu_driver.c: Likewise.
* src/remote_internal.c: Likewise.

* configure.in: Use AC_CONFIG_AUX_DIR(build-aux), so that a bunch
of gettextize-generated files go into build-aux/, rather than in
the top-level directory.
* .cvsignore: Adjust.
* build-aux/.cvsignore: New file.


Author: Jim Meyering <meyering@redhat.com>

        * python/libvir.c, python/libvirt_wrap.h, qemud/qemud.c,
        qemud/remote.c, src/internal.h, src/openvz_conf.c,
        src/openvz_driver.c, src/proxy_internal.h, src/qemu_conf.c,
        src/qemu_driver.c, src/remote_internal.h, src/test.h, src/util.c,
        src/xen_unified.c, src/xen_unified.h, tests/nodeinfotest.c,
        tests/qemuxml2argvtest.c, tests/qemuxml2xmltest.c, tests/reconnect.c,
        tests/sexpr2xmltest.c, tests/virshtest.c, tests/xencapstest.c,
        tests/xmconfigtest.c, tests/xml2sexprtest.c:
        Change #include <> to #include "" for local includes.
        Removed many includes from src/internal.h and put them in
        the C files which actually use them.
        Removed <ansidecl.h> - unused.
        Added a comment around __func__.
        Removed a clashing redefinition of VERSION symbol.
        All limits (PATH_MAX etc) now done in src/internal.h, so we
        don't need to include those headers in other files.

	* tests/Makefile.am, tests/nodeinfotest.c, tests/qemuxml2argvtest.c,
	  tests/qemuxml2xmltest.c, tests/sexpr2xmltest.c, tests/virshtest.c,
	  tests/xencapstest.c, tests/xmconfigtest.c, tests/xml2sexprtest.c:
	  Miscellaneous fixes to the tests to compile under Cygwin.