The cpu polarization member is the only per cpu state that is not part
of the pcpu structure. So add it there and have everything in one place.
Signed-off-by: NHeiko Carstens <heiko.carstens@de.ibm.com>
Signed-off-by: NMartin Schwidefsky <schwidefsky@de.ibm.com>

The maximum nesting of the cpu topology is evaluated when /proc/sysinfo
is the first time read. This happens without a lock and a concurrent
reader on a different cpu can see and use an invalid intermediate value.
Besides the fact that this race is quite unlikely the worst thing that
could happen is that /proc/sysinfo would contain bogus information about
the machine's cpu topology.
Nevertheless this should be fixed. So move the detection code to the
early machine detection code and since now the value is early available
use it in the topology code as well.
Signed-off-by: NHeiko Carstens <heiko.carstens@de.ibm.com>
Signed-off-by: NMartin Schwidefsky <schwidefsky@de.ibm.com>

Any change to sysinfo.h causes a whole kernel recompile since sysinfo.h is
included by topology.h, which again is used nearly everywhere.
So remove that include and add a forward declaration instead.
Signed-off-by: NHeiko Carstens <heiko.carstens@de.ibm.com>
Signed-off-by: NMartin Schwidefsky <schwidefsky@de.ibm.com>

Remove the file name from the comment at top of many files. In most
cases the file name was wrong anyway, so it's rather pointless.

Also unify the IBM copyright statement. We did have a lot of sightly
different statements and wanted to change them one after another
whenever a file gets touched. However that never happened. Instead
people start to take the old/"wrong" statements to use as a template
for new files.
So unify all of them in one go.
Signed-off-by: NHeiko Carstens <heiko.carstens@de.ibm.com>

Define struct pcpu and merge some of the NR_CPUS arrays into it, including
__cpu_logical_map, current_set and smp_cpu_state. Split smp related
functions to those operating on physical cpus and the functions operating
on a logical cpu number. Make the functions for physical cpus use a
pointer to a struct pcpu. This hides the knowledge about cpu addresses in
smp.c, entry[64].S and swsusp_asm64.S, thus remove the sigp.h header.

The PSW restart mechanism is used to start secondary cpus, calling a
function on an online cpu, calling a function on the ipl cpu, and for
the nmi signal. Replace the different assembler functions with a
single function restart_int_handler. The new entry point calls a function
whose pointer is stored in the lowcore of the target cpu and it can wait
for the source cpu to stop. This covers all existing use cases.

Overall the code is now simpler and there are ~380 lines less code.
Reviewed-by: NHeiko Carstens <heiko.carstens@de.ibm.com>
Signed-off-by: NMartin Schwidefsky <schwidefsky@de.ibm.com>

Increase cpu topology change poll frequency if a change is anticipated.
Otherwise a user might be a bit confused to have to wait up to a minute
in order to see a change this should be visible immediatly.
However there is no guarantee that the change will happen during the
time frame the poll frequency is increased.
Signed-off-by: NHeiko Carstens <heiko.carstens@de.ibm.com>
Signed-off-by: NMartin Schwidefsky <schwidefsky@de.ibm.com>

Cleanup z10 topology handling. This adds some more code but hopefully
the result is more readable and easier to maintain.
Signed-off-by: NHeiko Carstens <heiko.carstens@de.ibm.com>
Signed-off-by: NMartin Schwidefsky <schwidefsky@de.ibm.com>

Remove all ifdefs from topology code and also only compile it for the
CONFIG_SCHED_BOOK case. The new code selects SCHED_MC if SCHED_BOOK is
selected. SCHED_MC without SCHED_BOOK is not possible anymore.
Furthermore various sysfs attributes are not available anymore for the
!SCHED_BOOK case. In particular all attributes that correspond to
CPU polarization.
But since all real world kernels have SCHED_BOOK selected anyway this
doesn't matter too much.
Signed-off-by: NHeiko Carstens <heiko.carstens@de.ibm.com>
Signed-off-by: NMartin Schwidefsky <schwidefsky@de.ibm.com>

This moves the 'cpu sysdev_class' over to a regular 'cpu' subsystem
and converts the devices to regular devices. The sysdev drivers are
implemented as subsystem interfaces now.

After all sysdev classes are ported to regular driver core entities, the
sysdev implementation will be entirely removed from the kernel.

Userspace relies on events and generic sysfs subsystem infrastructure
from sysdev devices, which are made available with this conversion.

Cc: Haavard Skinnemoen <hskinnemoen@gmail.com>
Cc: Hans-Christian Egtvedt <egtvedt@samfundet.no>
Cc: Tony Luck <tony.luck@intel.com>
Cc: Fenghua Yu <fenghua.yu@intel.com>
Cc: Arnd Bergmann <arnd@arndb.de>
Cc: Benjamin Herrenschmidt <benh@kernel.crashing.org>
Cc: Paul Mackerras <paulus@samba.org>
Cc: Martin Schwidefsky <schwidefsky@de.ibm.com>
Cc: Heiko Carstens <heiko.carstens@de.ibm.com>
Cc: Paul Mundt <lethal@linux-sh.org>
Cc: "David S. Miller" <davem@davemloft.net>
Cc: Chris Metcalf <cmetcalf@tilera.com>
Cc: Thomas Gleixner <tglx@linutronix.de>
Cc: Ingo Molnar <mingo@redhat.com>
Cc: "H. Peter Anvin" <hpa@zytor.com>
Cc: Borislav Petkov <bp@amd64.org>
Cc: Tigran Aivazian <tigran@aivazian.fsnet.co.uk>
Cc: Len Brown <lenb@kernel.org>
Cc: Zhang Rui <rui.zhang@intel.com>
Cc: Dave Jones <davej@redhat.com>
Cc: Peter Zijlstra <peterz@infradead.org>
Cc: Russell King <rmk+kernel@arm.linux.org.uk>
Cc: Andrew Morton <akpm@linux-foundation.org>
Cc: Arjan van de Ven <arjan@linux.intel.com>
Cc: "Rafael J. Wysocki" <rjw@sisk.pl>
Cc: "Srivatsa S. Bhat" <srivatsa.bhat@linux.vnet.ibm.com>
Signed-off-by: NKay Sievers <kay.sievers@vrfy.org>
Signed-off-by: NGreg Kroah-Hartman <gregkh@suse.de>

Make sure that all cpus in a book on a z10 appear as book siblings
and not as core siblings. This fixes some performance regressions that
appeared after the book scheduling domain got introduced.
Signed-off-by: NHeiko Carstens <heiko.carstens@de.ibm.com>
Signed-off-by: NMartin Schwidefsky <schwidefsky@de.ibm.com>

Fix this warning:
WARNING: vmlinux.o(.text+0x199b6): Section mismatch in reference from
the function alloc_masks() to the function .init.text:__alloc_bootmem()
Signed-off-by: NSebastian Ott <sebott@linux.vnet.ibm.com>
Signed-off-by: NMartin Schwidefsky <schwidefsky@de.ibm.com>

Merge irq.c and s390_ext.c into irq.c. That way all external interrupt
related functions are together.
Signed-off-by: NHeiko Carstens <heiko.carstens@de.ibm.com>
Signed-off-by: NMartin Schwidefsky <schwidefsky@de.ibm.com>

Adapt new API.
Signed-off-by: NKOSAKI Motohiro <kosaki.motohiro@jp.fujitsu.com>
Signed-off-by: NMartin Schwidefsky <schwidefsky@de.ibm.com>

Fix cpu masks for 'topology=off' case. Folding of the scheduling domains
happen in such a way that everything belongs to the MC domain instead
of the CPU doimain.
This should fix a performance regression introduced with
eafd2b6d "[S390] topology: use default MC domain initializer" and also
makes sure we have the same behavious as if CONFIG_SCHED_MC was not
selected at all.
Signed-off-by: NHeiko Carstens <heiko.carstens@de.ibm.com>
Signed-off-by: NMartin Schwidefsky <schwidefsky@de.ibm.com>

Export the cpu configuration topology via sysinfo. Two new lines are
introduced:

CPU Topology HW:      0 0 0 4 6 4
CPU Topology SW:      0 0 0 0 4 24

The HW line describes the cpu topology nesting levels when the maximum
nesting level is used to get the corresponding SYSIB.
The SW line describes what Linux is actually using. In this case it
supports only two levels (CONFIG_SCHED_BOOK off) and therefore the
hardware folded the two lower levels in the SYSIB response block.
Signed-off-by: NHeiko Carstens <heiko.carstens@de.ibm.com>
Signed-off-by: NMartin Schwidefsky <schwidefsky@de.ibm.com>

Move the topology sysinfo SYSIB definitions to the proper place in
asm/sysinfo.h where they should be.
Signed-off-by: NHeiko Carstens <heiko.carstens@de.ibm.com>
Signed-off-by: NMartin Schwidefsky <schwidefsky@de.ibm.com>

Move cpu topology facility detection to early setup code where it
should be.
Signed-off-by: NHeiko Carstens <heiko.carstens@de.ibm.com>
Signed-off-by: NMartin Schwidefsky <schwidefsky@de.ibm.com>

Store the facility list once at system startup with stfl/stfle and
reuse the result for all facility tests.
Signed-off-by: NMartin Schwidefsky <schwidefsky@de.ibm.com>

Switch default value of the kernel parameter 'topology' from off to on.
Various performance measurements have finally shown that there are no
(known) regressions anywhere.
Signed-off-by: NHeiko Carstens <heiko.carstens@de.ibm.com>
Signed-off-by: NMartin Schwidefsky <schwidefsky@de.ibm.com>

Use the extended cpu topology information that z11 machines provide
to improve the scheduler's decision making.
Signed-off-by: NHeiko Carstens <heiko.carstens@de.ibm.com>
Signed-off-by: NPeter Zijlstra <a.p.zijlstra@chello.nl>
LKML-Reference: <20100831082844.604956770@de.ibm.com>
Signed-off-by: NIngo Molnar <mingo@elte.hu>

Provide a topology_core_id define which makes sure that the contents of
/sys/devices/system/cpu/cpuX/topology/core_id
indeed do contain the core id and not always 0.
Signed-off-by: NHeiko Carstens <heiko.carstens@de.ibm.com>
Signed-off-by: NMartin Schwidefsky <schwidefsky@de.ibm.com>

In the default case the lock is not unlocked.  The return is
converted to a goto, to share the unlock at the end of the function.

A simplified version of the semantic patch that finds this problem is as
follows: (http://coccinelle.lip6.fr/)

// <smpl>
@r exists@
expression E1;
identifier f;
@@

f (...) { <+...
* spin_lock_irq (E1,...);
... when != E1
* return ...;
...+> }
// </smpl>
Signed-off-by: NJulia Lawall <julia@diku.dk>
Signed-off-by: NHeiko Carstens <heiko.carstens@de.ibm.com>
Signed-off-by: NMartin Schwidefsky <schwidefsky@de.ibm.com>

Finally move it to the place where it belongs to and make get rid of
it for !CONFIG_SMP.
Signed-off-by: NHeiko Carstens <heiko.carstens@de.ibm.com>
Signed-off-by: NMartin Schwidefsky <schwidefsky@de.ibm.com>

Impact: cleanup

cpu_coregroup_mask is the New Hotness.

As S/390 uses theirs internally, so we just make it static.
Signed-off-by: NRusty Russell <rusty@rustcorp.com.au>
Signed-off-by: NMartin Schwidefsky <schwidefsky@de.ibm.com>

Like cpu_coregroup_map, but returns a (const) pointer.

Compile-tested on s390 (defconfig).
Signed-off-by: NRusty Russell <rusty@rustcorp.com.au>
Signed-off-by: NMike Travis <travis@sgi.com>

Signed-off-by: NMartin Schwidefsky <schwidefsky@de.ibm.com>

Interrupts haven't been implemented. So remove the dead code.
Signed-off-by: NHeiko Carstens <heiko.carstens@de.ibm.com>
Signed-off-by: NMartin Schwidefsky <schwidefsky@de.ibm.com>

Introduce a topology=[on|off] kernel parameter which allows to switch
cpu topology on/off. Default will be off, since it looks like that for
some workloards this doesn't behave very well (on s390).
Signed-off-by: NHeiko Carstens <heiko.carstens@de.ibm.com>
Signed-off-by: NMartin Schwidefsky <schwidefsky@de.ibm.com>

Call rebuild_sched_domains instead of arch_reinit_sched_domains if
cpu topology changes. This leaves cpu sets alone which otherwise would
be destroyed.
If and how it makes sense to define cpu sets on a virtualized
architecture is another question.
Signed-off-by: NHeiko Carstens <heiko.carstens@de.ibm.com>
Signed-off-by: NMartin Schwidefsky <schwidefsky@de.ibm.com>

Common code doesn't call arch_update_cpu_topology() anymore on
cpu hotplug. But our architecture backend relied on that in order to
update the cpu_core_map. For machines without cpu topology support
this leads uninitialized cpu_core_maps for later on added cpus.

To solve this just initialize the maps with cpu_possible_map, since
that will be always valid for machines without topology support.
Signed-off-by: NHeiko Carstens <heiko.carstens@de.ibm.com>
Signed-off-by: NMartin Schwidefsky <schwidefsky@de.ibm.com>

Change arch_update_cpu_topology so it returns 1 if the cpu topology changed
and 0 if it didn't change. This will be useful for the next patch which adds
a call to this function in partition_sched_domains.
Signed-off-by: NHeiko Carstens <heiko.carstens@de.ibm.com>
Signed-off-by: NIngo Molnar <mingo@elte.hu>

cpu_coregroup_map used to grab a mutex on s390 since it was only
called from process context.
Since c7c22e4d "block: add support
for IO CPU affinity" this is not true anymore.
It now also gets called from softirq context.

To prevent possible deadlocks change this in architecture code and
use a spinlock instead of a mutex.

Cc: stable@kernel.org
Cc: Jens Axboe <jens.axboe@oracle.com>
Signed-off-by: NHeiko Carstens <heiko.carstens@de.ibm.com>
Signed-off-by: NMartin Schwidefsky <schwidefsky@de.ibm.com>

Now that it is safe to use get_online_cpus() we can revert

	[S390] cpu topology: Fix possible deadlock.
	commit: fd781fa2

and call arch_reinit_sched_domains() directly from topology_work_fn().
Signed-off-by: NOleg Nesterov <oleg@tv-sign.ru>
Cc: Gautham R Shenoy <ego@in.ibm.com>
Tested-by: NHeiko Carstens <heiko.carstens@de.ibm.com>
Cc: Max Krasnyansky <maxk@qualcomm.com>
Cc: Paul Jackson <pj@sgi.com>
Cc: Paul Menage <menage@google.com>
Cc: Peter Zijlstra <peterz@infradead.org>
Cc: Vegard Nossum <vegard.nossum@gmail.com>
Cc: Martin Schwidefsky <schwidefsky@de.ibm.com>
Cc: Ingo Molnar <mingo@elte.hu>
Signed-off-by: NAndrew Morton <akpm@linux-foundation.org>
Signed-off-by: NLinus Torvalds <torvalds@linux-foundation.org>

As noted by Akinobu Mita in patch b1fceac2,
alloc_bootmem and related functions never return NULL and always return a
zeroed region of memory.  Thus a NULL test or memset after calls to these
functions is unnecessary.

 arch/s390/kernel/topology.c |    2 --
 1 file changed, 2 deletions(-)

This was fixed using the following semantic patch.
(http://www.emn.fr/x-info/coccinelle/)

// <smpl>
@@
expression E;
statement S;
@@

E = \(alloc_bootmem\|alloc_bootmem_low\|alloc_bootmem_pages\|alloc_bootmem_low_pages\)(...)
... when != E
(
- BUG_ON (E == NULL);
|
- if (E == NULL) S
)

@@
expression E,E1;
@@

E = \(alloc_bootmem\|alloc_bootmem_low\|alloc_bootmem_pages\|alloc_bootmem_low_pages\)(...)
... when != E
- memset(E,0,E1);
// </smpl>
Signed-off-by: NJulia Lawall <julia@diku.dk>
Signed-off-by: NHeiko Carstens <heiko.carstens@de.ibm.com>
Signed-off-by: NMartin Schwidefsky <schwidefsky@de.ibm.com>

When we get a notification that cpu topology changed, we schedule a
work struct which just calls arch_reinit_sched_domains. This function
in turn calls get_online_cpus() which results int the lockdep warning
below.

After all it turnded out that it's not legal to call get_online_cpus()
from the context of a multi-threaded work queue.
It could deadlock this way:

process 0 (events/cpu-x):
-> run_workqueue
-> removes my work_struct from the work queue
-> calls work_struct->fn
-> get_online_cpus()
-> locks on cpu_hotplug.lock since process 1 below is doing cpu hotplug

process 1:
-> cpu_down (for cpu-x)
-> cpu_hotplug_begin (holds cpu_hotplug.lock now)
-> cpu-x dead
-> notifier_call_chain with CPU_DEAD
-> cleanup_workqueue_thread
-> flush_cpu_workqueue (succeeds)
-> kthread_stop for events/cpu-x
  -> now kthread_stop waits for my work_struct to complete from within
     process 0. -> dead.

A single threaded workqueue wouldn't have such problems, however there is
no such common queue available and it's not worth to create one for the
very rare calls to arch_reinit_sched_domains.

So we just create a kernel thread from our work struct which calls
arch_reinit_sched_domains and are done with it.

Thanks to Oleg Nesterov and Peter Zijlstra for helping me figuring out
that this isn't a false positive lockdep warning:

=======================================================
[ INFO: possible circular locking dependency detected ]
2.6.25-03562-g3dc50637-dirty #12
-------------------------------------------------------
events/3/14 is trying to acquire lock:
 (&cpu_hotplug.lock){--..}, at: [<0000000000076094>] get_online_cpus+0x50/0x78

but task is already holding lock:
 (topology_work){--..}, at: [<0000000000059cde>] run_workqueue+0x106/0x278

which lock already depends on the new lock.

the existing dependency chain (in reverse order) is:

-> #2 (topology_work){--..}:
       [<000000000006fc74>] __lock_acquire+0x1010/0x111c
       [<000000000006fe40>] lock_acquire+0xc0/0xf8
       [<0000000000059d48>] run_workqueue+0x170/0x278
       [<0000000000059edc>] worker_thread+0x8c/0xf0
       [<000000000005f5bc>] kthread+0x68/0xa0
       [<000000000001a33e>] kernel_thread_starter+0x6/0xc
       [<000000000001a338>] kernel_thread_starter+0x0/0xc

-> #1 (events){--..}:
       [<000000000006fc74>] __lock_acquire+0x1010/0x111c
       [<000000000006fe40>] lock_acquire+0xc0/0xf8
       [<000000000005a23c>] cleanup_workqueue_thread+0x60/0xa8
       [<00000000003b2ab8>] workqueue_cpu_callback+0xbc/0x170
       [<00000000003bba80>] notifier_call_chain+0x5c/0xa4
       [<00000000000655a2>] __raw_notifier_call_chain+0x26/0x38
       [<00000000000655e2>] raw_notifier_call_chain+0x2e/0x40
       [<0000000000075e00>] cpu_down+0x228/0x31c
       [<00000000003b1dd8>] store_online+0x64/0xb8
       [<00000000001e7128>] sysdev_store+0x48/0x58
       [<0000000000121cd2>] sysfs_write_file+0x126/0x1c0
       [<00000000000c1944>] vfs_write+0xb0/0x15c
       [<00000000000c20e6>] sys_write+0x56/0x88
       [<0000000000027a68>] sys32_write+0x34/0x4c
       [<0000000000023f70>] sysc_noemu+0x10/0x16
       [<0000000077f3f186>] 0x77f3f186

-> #0 (&cpu_hotplug.lock){--..}:
       [<000000000006fa84>] __lock_acquire+0xe20/0x111c
       [<000000000006fe40>] lock_acquire+0xc0/0xf8
       [<00000000003b701c>] mutex_lock_nested+0xd0/0x364
       [<0000000000076094>] get_online_cpus+0x50/0x78
       [<000000000003a03e>] arch_reinit_sched_domains+0x26/0x58
       [<000000000002700e>] topology_work_fn+0x26/0x34
       [<0000000000059d4e>] run_workqueue+0x176/0x278
       [<0000000000059edc>] worker_thread+0x8c/0xf0
       [<000000000005f5bc>] kthread+0x68/0xa0
       [<000000000001a33e>] kernel_thread_starter+0x6/0xc
       [<000000000001a338>] kernel_thread_starter+0x0/0xc

other info that might help us debug this:

2 locks held by events/3/14:
 #0:  (events){--..}, at: [<0000000000059cde>] run_workqueue+0x106/0x278
 #1:  (topology_work){--..}, at: [<0000000000059cde>] run_workqueue+0x106/0x278

stack backtrace:
CPU: 3 Not tainted 2.6.25-03562-g3dc50637-dirty #12
Process events/3 (pid: 14, task: 000000002fb04038, ksp: 000000002fb0bd70)
0400000000000000 000000002fb0ba40 0000000000000002 0000000000000000
       000000002fb0bae0 000000002fb0ba58 000000002fb0ba58 0000000000016488
       0000000000000000 000000002fb0bd70 0000000000000000 0000000000000000
       000000002fb0ba40 000000000000000c 000000002fb0ba40 000000002fb0bab0
       00000000003c99e0 0000000000016488 000000002fb0ba40 000000002fb0ba90
Call Trace:
([<00000000000163fc>] show_trace+0x138/0x158)
 [<00000000000164e2>] show_stack+0xc6/0xf8
 [<0000000000016624>] dump_stack+0xb0/0xc0
 [<000000000006cd36>] print_circular_bug_tail+0xa2/0xb4
 [<000000000006fa84>] __lock_acquire+0xe20/0x111c
 [<000000000006fe40>] lock_acquire+0xc0/0xf8
 [<00000000003b701c>] mutex_lock_nested+0xd0/0x364
 [<0000000000076094>] get_online_cpus+0x50/0x78
 [<000000000003a03e>] arch_reinit_sched_domains+0x26/0x58
 [<000000000002700e>] topology_work_fn+0x26/0x34
 [<0000000000059d4e>] run_workqueue+0x176/0x278
 [<0000000000059edc>] worker_thread+0x8c/0xf0
 [<000000000005f5bc>] kthread+0x68/0xa0
 [<000000000001a33e>] kernel_thread_starter+0x6/0xc
 [<000000000001a338>] kernel_thread_starter+0x0/0xc
INFO: lockdep is turned off.

Cc: Oleg Nesterov <oleg@tv-sign.ru>
Cc: Peter Zijlstra <peterz@infradead.org>
Signed-off-by: NHeiko Carstens <heiko.carstens@de.ibm.com>
Signed-off-by: NMartin Schwidefsky <schwidefsky@de.ibm.com>

This exposes the core siblings to user space via sysfs.
Signed-off-by: NHeiko Carstens <heiko.carstens@de.ibm.com>

If vertical cpu polarization is active then the hypervisor will
dispatch certain cpus for a longer time than other cpus for maximum
performance. For example if a guest would have three virtual cpus,
each of them with a share of 33 percent, then in case of vertical
cpu polarization all of the processing time would be combined to a
single cpu which would run all the time, while the other two cpus
would get nearly no cpu time.

There are three different types of vertical cpus: high, medium and
low. Low cpus hardly get any real cpu time, while high cpus get a
full real cpu. Medium cpus get something in between.

In order to switch between the two possible modes (default is
horizontal) a 0 for horizontal polarization or a 1 for vertical
polarization must be written to the dispatching sysfs attribute:

/sys/devices/system/cpu/dispatching

The polarization of each single cpu can be figured out by the
polarization sysfs attribute of each cpu:

/sys/devices/system/cpu/cpuX/polarization

horizontal, vertical:high, vertical:medium, vertical:low or unknown.

When switching polarization the polarization attribute may contain
the value unknown until the configuration change is done and the
kernel has figured out the new polarization of each cpu.

Note that running a system with different types of vertical cpus may
result in significant performance regressions. If possible only one
type of vertical cpus should be used. All other cpus should be
offlined.
Signed-off-by: NMartin Schwidefsky <schwidefsky@de.ibm.com>
Signed-off-by: NHeiko Carstens <heiko.carstens@de.ibm.com>

Add s390 backend so we can give the scheduler some hints about the
cpu topology.
Signed-off-by: NMartin Schwidefsky <schwidefsky@de.ibm.com>
Signed-off-by: NHeiko Carstens <heiko.carstens@de.ibm.com>