Update the nodeinfo helper code to use virReportError instead
of the nodeReportError custom macro
Signed-off-by: NDaniel P. Berrange <berrange@redhat.com>

Commit 80533ca2 forgot to think about offline cpus.  When a node
cpu is offline, then its topology/ subdirectory is not present,
leading to spurious error messages leaked to the user such as:

libvir:  error : cannot open /home/dummy/libvirt/tests/nodeinfodata/linux-nodeinfo-sysfs-test-6/node/node0/cpu7/topology/physical_package_id: No such file or directory

Fix that, as well as test it; the test data is gathered from a
machine with one NUMA node, hyperthreading, and with 2 of the
8 cpus offline.

* src/nodeinfo.c (virNodeParseNode): Don't parse topology of
offline cpus.
* tests/nodeinfotest.c (mymain): Run new test.
* tests/nodeinfodata/linux-nodeinfo-sysfs-test-6*: New data.

This patch changes the way data to fill the nodeinfo structure are
gathered. We've gathere the test data by iterating processors an sockets
separately from nodes. The reported data was based solely on information
about core id. Problems arise when eg cores in mulit-processor machines
don't have same id's on both processors or maybe one physical processor
contains more NUMA nodes.

This patch changes the approach how we detect processors and nodes. Now
we start at enumerating nodes and for each node processors, sockets and
threads are enumerated separately. This approach provides acurate data
that comply to docs about the nodeinfo structure. This also enables to
get rid of hacks: see commits 10d9038b,
ac9dd4a6. (Those changes in nodeinfo.c
are efectively reverted by this patch).

This patch also changes output of one of the tests, as the processor
topology is now acquired more precisely.

For the s390x architecture the sysfs core_id alone is not unique. As a
result it can happen that libvirt thinks there are less host CPUs available
than really present.
Currently, a logical CPU is equivalent to a core for s390x. We therefore
produce a fake core id from the CPU number.
Signed-off-by: NViktor Mihajlovski <mihajlov@linux.vnet.ibm.com>

Minimal CPU "parser" for s390 to avoid compile time warning.
Signed-off-by: NThang Pham <thang.pham@us.ibm.com>
Signed-off-by: NViktor Mihajlovski <mihajlov@linux.vnet.ibm.com>

To ensure consistent error reporting of invalid arguments,
provide a number of predefined helper methods & macros.

 - An arg which must not be NULL:

   virCheckNonNullArgReturn(argname, retvalue)
   virCheckNonNullArgGoto(argname, label)

 - An arg which must be NULL

   virCheckNullArgGoto(argname, label)

 - An arg which must be positive (ie 1 or greater)

   virCheckPositiveArgGoto(argname, label)

 - An arg which must not be 0

   virCheckNonZeroArgGoto(argname, label)

 - An arg which must be zero

   virCheckZeroArgGoto(argname, label)

 - An arg which must not be negative (ie 0 or greater)

   virCheckNonNegativeArgGoto(argname, label)

* src/libvirt.c, src/libvirt-qemu.c,
  src/nodeinfo.c, src/datatypes.c: Update to use
  virCheckXXXX macros
* po/POTFILES.in: Add libvirt-qemu.c and virterror_internal.h
* src/internal.h: Define macros for checking invalid args
* src/util/virterror_internal.h: Define macros for reporting
  invalid args
Signed-off-by: NDaniel P. Berrange <berrange@redhat.com>

"Instead of developing one CPU with 12 cores, the Magny Cours is
actually two 6 core “Bulldozer” CPUs combined in to one package"

I.e, each package has two NUMA nodes, and the two numa nodes share
the same core ID set (0-6), which means parsing the cores number
from sysfs doesn't work in this case.

And the wrong CPU number could cause three problems for libvirt:

1) performance lost

  A domain without "cpuset" or "placement='auto'" (to drive numad)
specified will be only pinned to part of the CPUs.

2) domain can be started

  If a domain uses numad, and the advisory nodeset returned from
numad contains node which exceeds the range of wrong total CPU
number. The domain will fail to start, as the bitmask passed to
sched_setaffinity could be fully filled with zero.

3) wrong CPU number affects lots of stuffs.

  E.g. for command "virsh vcpuinfo", "virsh vcpupin", it will always
output with the truncated CPU list.

For more details:

https://www.redhat.com/archives/libvir-list/2012-May/msg00607.html

This patch is to fix the problem by parsing /proc/cpuinfo to get
the value of field "cpu cores", and use it as nodeinfo->cores if
it's greater than the cores number from sysfs.

We had previously weakened our nodeinfotest in order to ignore parsed
node values, because the parse function was mistakenly relying on
host files.  A better fix is to avoid using the numactl library, but
to instead parse the same files that numactl would read, all while
allowing the files to be relative to our choice of directory.

* src/nodeinfo.c (CPU_SYS_PATH, NODE_SYS_PATH): Replace with...
(SYSFS_SYSTEM_PATH): ...parent directory.
(linuxNodeInfoCPUPopulate): Check NUMA nodes from requested
directory (by inlining numactl code).
(nodeGetCPUmap, nodeGetMemoryStats): Adjust macro use.
* tests/nodeinfotest.c (linuxTestCompareFiles, linuxTestNodeInfo):
Update test to match.

We were wasting time to malloc a copy of a constant string, then
copy it into static storage, for every call to nodeGetInfo.  At
least we were lucky that it was a constant source, and thus not
subject to even worse issues with one thread clobbering the static
storage while another was using it.  This gets rid of the waste,
by passing the string through the stack instead, as well as renaming
internal functions to better match our conventions.

* src/nodeinfo.c (sysfs_path): Delete.
(get_cpu_value, count_thread_siblings, parse_socket): Add
parameter, and rename...
(virNodeGetCpuValue, virNodeCountThreadSiblings)
(virNodeParseSocket): ... into a common namespace.
(cpu_online, parse_core): Inline into callers.
(linuxNodeInfoCPUPopulate): Update caller.
(nodeGetInfo): Drop a useless malloc.

Error: RESOURCE_LEAK:
/builddir/build/BUILD/libvirt-0.9.10/src/nodeinfo.c:629: alloc_fn: Calling allocation function "fopen".
/builddir/build/BUILD/libvirt-0.9.10/src/nodeinfo.c:629: var_assign: Assigning: "cpuinfo" =  storage returned from "fopen("/proc/cpuinfo", "r")".
/builddir/build/BUILD/libvirt-0.9.10/src/nodeinfo.c:638: leaked_storage: Variable "cpuinfo" going out of scope leaks the storage it points to.

I noticed these compiler warnings when building for the s390 architecture.

* src/node_device/node_device_udev.c (udevDeviceMonitorStartup):
Mark unused variable.
* src/nodeinfo.c (linuxNodeInfoCPUPopulate): Avoid unused variable.

The code is splattered with a mix of

  sizeof foo
  sizeof (foo)
  sizeof(foo)

Standardize on sizeof(foo) and add a syntax check rule to
enforce it
Signed-off-by: NDaniel P. Berrange <berrange@redhat.com>

Signed-off-by: NKAMEZAWA Hiroyuki <kamezawa.hiroyu@jp.fujitsu.com>
Signed-off-by: NLai Jiangshan <laijs@cn.fujitsu.com>

If "conf/domain_conf.h" included in file nodeinfo.c, gcc complains about
a variable shadowing. fix it for potential further including of the file.

ref: https://www.redhat.com/archives/libvir-list/2012-February/msg00955.html

The blocks to extract node information on a per-arch
basis wasn't well balanced leading to a compilation
failure if not on one of the handled arches (PCs and PPCs)

Fix memory leak:

==27534== 24 bytes in 1 blocks are definitely lost in loss record 207 of 530
==27534==    at 0x4A05E46: malloc (vg_replace_malloc.c:195)
==27534==    by 0x38EC26EC37: vasprintf (in /lib64/libc-2.13.so)
==27534==    by 0x4E998E6: virVasprintf (util.c:1677)
==27534==    by 0x4E999F1: virAsprintf (util.c:1695)
==27534==    by 0x4F1EAAC: nodeGetInfo (nodeinfo.c:593)
==27534==    by 0x47948F: qemuCapsInitCPU (qemu_capabilities.c:855)
==27534==    by 0x4796B1: qemuCapsInit (qemu_capabilities.c:915)
==27534==    by 0x456550: qemuCreateCapabilities (qemu_driver.c:245)
==27534==    by 0x4578C4: qemudStartup (qemu_driver.c:580)
==27534==    by 0x4F20886: virStateInitialize (libvirt.c:852)
==27534==    by 0x420E55: daemonRunStateInit (libvirtd.c:1156)
==27534==    by 0x4E94C56: virThreadHelper (threads-pthread.c:157)

Mark this leaked variable as const char * when it is passed into another
function. 

 /proc/cpuinfo

Libvirt at present depends on /proc/cpuinfo to gather host
details such as CPUs, cores, threads, etc. This is an architecture-
dependent approach. An alternative is to use 'Sysfs', which provides
a platform-agnostic interface to parse host CPU topology.
Signed-off-by: NPrerna Saxena <prerna@linux.vnet.ibm.com>

Introduced by 5e495c8b, except the ones for checking if numa
is supported by host, all the NO_SUPPORT are changed back. For
the ones about numa checking, change them into INTERNAL_ERROR.

In preparation for a future patch adding new virFile APIs.

* src/util/files.h, src/util/files.c: Move...
* src/util/virfile.h, src/util/virfile.c: ...here, and rename
functions to virFile prefix.  Macro names are intentionally
left alone.
* *.c: All '#include "files.h"' uses changed.
* src/Makefile.am (UTIL_SOURCES): Reflect rename.
* cfg.mk (exclude_file_name_regexp--sc_prohibit_close): Likewise.
* src/libvirt_private.syms: Likewise.
* docs/hacking.html.in: Likewise.
* HACKING: Regenerate.

When building libvirt without libvirtd, we will receive the following error
message:

make[3]: Entering directory `/home/wency/rpmbuild/BUILD/libvirt-0.9.2/tools'
  CC     virsh-virsh.o
  CC     virsh-console.o
  GEN    virt-xml-validate
  GEN    virt-pki-validate
  CCLD   virsh
./src/.libs/libvirt.so: undefined reference to `numa_available'
./src/.libs/libvirt.so: undefined reference to `numa_max_node'
collect2: ld returned 1 exit status

The reason is that: we check numactl only when building qemu driver, and qemu
driver will not be built when bulding without libvirtd. So with_numactl's
value is check and we will not link libnuma.so.

In the other function, we call numa_available() and numa_max_node() only
when HAVE_NUMACTL is 1. We should do the same check in the function nodeGetMemoryStats().

Detected by autobuild.sh when cross-building for mingw.
Introduced in commits ce76e853 and af35cece.

* src/nodeinfo.c (nodeGetCPUStats, nodeGetMemoryStats): Mark
parameters as potentially unused.

Signed-off-by: NMinoru Usui <usui@mxm.nes.nec.co.jp>

Signed-off-by: NMinoru Usui <usui@mxm.nes.nec.co.jp>

that break compilation on non intel architectures:

mips:
https://buildd.debian.org/status/fetch.php?pkg=libvirt&arch=mips&ver=0.9.2-1&stamp=1307570195
powerpc:
https://buildd.debian.org/status/fetch.php?pkg=libvirt&arch=powerpc&ver=0.9.2-1&stamp=1307550913
s390:
https://buildd.debian.org/status/fetch.php?pkg=libvirt&arch=s390&ver=0.9.2-1&stamp=1307641748
sparc:
https://buildd.debian.org/status/fetch.php?pkg=libvirt&arch=sparc&ver=0.9.2-1&stamp=1307552926

Up to now we missed parser for cpuinfo on x390(x) machines. Those machines
have only 1 thread, core, socket. What is missing is information about
CPU frequency.

Inspired by Eric Blake

And from all related macros and functions.

* src/nodeinfo.c (linuxNodeInfoCPUPopulate): Fix indentation of
last patch.

Due to differences in /proc/cpuinfo the parsing of the cpu data is
different between architectures. On PPC /proc/cpuinfo looks like this:

[original formatting with tabs]

processor    : 0
cpu          : PPC970MP, altivec supported
clock        : 2297.700000MHz
revision     : 1.1 (pvr 0044 0101)

processor    : 1
cpu          : PPC970MP, altivec supported
clock        : 2297.700000MHz
revision     : 1.1 (pvr 0044 0101)

[..]

timebase     : 14318000
platform     : pSeries
model        : IBM,8844-AC1
machine      : CHRP IBM,8844-AC1

The patch adapts the parsing of the data found in /proc/cpuinfo.

/sys/devices/system/cpu/cpuX/topology/physical_package_id also
always returns -1. Check for it on ppc and make it '0' if found negative.

Remove the <stdbool.h> header from all source files / headers
and just put it into internal.h

* src/internal.h: Add <stdbool.h>

Virsh freecell --all was not only getting wrong NUMA nodes count, but
even the NUMA nodes IDs. They doesn't have to be continuous, as I've
found out during testing this. Therefore a modification of
nodeGetCellsFreeMemory() error message.

When running 'make check' under a multi-cpu Dom0 xen machine,
nodeinfotest had a spurious failure it was reading from
/sys/devices/system/cpu, but xen has no notion of topology.  The test
was intended to be isolated from reading any real system files; the
regression was introduced in Mar 2010 with commit aa2f6f96.

Fix things by allowing an early exit for the testsuite.

* src/nodeinfo.c (linuxNodeInfoCPUPopulate): Add parameter.
(nodeGetInfo): Adjust caller.
* tests/nodeinfotest.c (linuxTestCompareFiles): Likewise.

The nodeinfo structure includes

    nodes   : the number of NUMA cell, 1 for uniform mem access
    sockets : number of CPU socket per node
    cores   : number of core per socket
    threads : number of threads per core

which does not work well for NUMA topologies where each node does not
consist of integral number of CPU sockets.

We also have VIR_NODEINFO_MAXCPUS macro in public libvirt.h which
computes maximum number of CPUs as (nodes * sockets * cores * threads).

As a result, we can't just change sockets to report total number of
sockets instead of sockets per node. This would probably be the easiest
since I doubt anyone is using the field directly. But because of the
macro, some apps might be using sockets indirectly.

This patch leaves sockets to be the number of CPU sockets per node (and
fixes qemu driver to comply with this) on machines where sockets can be
divided by nodes. If we can't divide sockets by nodes, we behave as if
there was just one NUMA node containing all sockets. Apps interested in
NUMA should consult capabilities XML, which is what they probably do
anyway.

This way, the only case in which apps that care about NUMA may break is
on machines with funky NUMA topology. And there is a chance libvirt
wasn't able to start any guests on those machines anyway (although it
depends on the topology, total number of CPUs and kernel version).
Nothing changes at all for apps that don't care about NUMA.

Similarly to deprecating close(), I am now deprecating fclose() and
introduce VIR_FORCE_FCLOSE() and VIR_FCLOSE(). Also, fdopen() is replaced with
VIR_FDOPEN().

Most of the files are opened in read-only mode, so usage of
VIR_FORCE_CLOSE() seemed appropriate. Others that are opened in write
mode already had the fclose()<  0 check and I converted those to
VIR_FCLOSE()<  0.

I did not find occurrences of possible double-closed files on the way.

Fixes https://bugzilla.redhat.com/show_bug.cgi?id=635857.

* src/nodeinfo.c (cpu_online): Allow missing directory for all
CPUs, not just cpu0.
Signed-off-by: NEric Blake <eblake@redhat.com>