When using systemtap it was observed that our udelay implementation is
rather suboptimal if being called from a kprobe handler installed by
systemtap.

The problem observed when a kprobe was installed on lock_acquired().
When the probe was hit the kprobe handler did call udelay, which set
up an (internal) timer and reenabled interrupts (only the clock comparator
interrupt) and waited for the interrupt.
This is an optimization to avoid that the cpu is busy looping while waiting
that enough time passes. The problem is that the interrupt handler still
does call irq_enter()/irq_exit() which then again can lead to a deadlock,
since some accounting functions may take locks as well.

If one of these locks is the same, which caused lock_acquired() to be
called, we have a nice deadlock.

This patch reworks the udelay code for the interrupts disabled case to
immediately leave the low level interrupt handler when the clock
comparator interrupt happens. That way no C code is being called and the
deadlock cannot happen anymore.
Signed-off-by: NHeiko Carstens <heiko.carstens@de.ibm.com>
Reviewed-by: NMartin Schwidefsky <schwidefsky@de.ibm.com>
Signed-off-by: NMartin Schwidefsky <schwidefsky@de.ibm.com>

The program parameter can be used to mark hardware samples with
some token.  Previously, it was used to mark guest samples only.

Improve the program parameter doubleword by combining two parts,
the leftmost LPP part and the rightmost PID part.  Set the PID
part for processes by using the task PID.
To distinguish host and guest samples for the kernel (PID part
is zero), the guest must always set the program paramater to a
non-zero value.  Use the leftmost bit in the LPP part of the
program parameter to be able to detect guest kernel samples.

[brueckner@linux.vnet.ibm.com]: Split __LC_CURRENT and introduced
__LC_LPP. Corrected __LC_CURRENT users and adjusted assembler parts.
And updated the commit message accordingly.
Signed-off-by: NChristian Borntraeger <borntraeger@de.ibm.com>
Signed-off-by: NHendrik Brueckner <brueckner@linux.vnet.ibm.com>
Reviewed-by: NHeiko Carstens <heiko.carstens@de.ibm.com>
Signed-off-by: NMartin Schwidefsky <schwidefsky@de.ibm.com>

The use of OFFSET instead of DEFINE makes the definitions in asm-offsets.c
more readable. While we are at it sort the defines for struct _lowcore
according to the field order and remove some unneeded defines.
Signed-off-by: NMartin Schwidefsky <schwidefsky@de.ibm.com>

Various functions in entry.S perform test-under-mask instructions
to test for particular bits in memory.  Because test-under-mask uses
a mask value of one byte, the mask value and the offset into the
memory must be calculated manually.  This easily introduces errors
and is hard to review and read.

Introduce the TSTMSK assembler macro to specify a mask constant and
let the macro calculate the offset and the byte mask to generate a
test-under-mask instruction.  The benefit is that existing symbolic
constants can now be used for tests.  Also the macro checks for
zero mask values and mask values that consist of multiple bytes.
Signed-off-by: NHendrik Brueckner <brueckner@linux.vnet.ibm.com>
Reviewed-by: NHeiko Carstens <heiko.carstens@de.ibm.com>
Signed-off-by: NMartin Schwidefsky <schwidefsky@de.ibm.com>

Previously, the init task did not have an allocated FPU save area and
saving an FPU state was not possible.  Now if the vector extension is
always enabled, provide a static FPU save area to save FPU states of
vector instructions that can be executed quite early.
Signed-off-by: NHendrik Brueckner <brueckner@linux.vnet.ibm.com>
Reviewed-by: NHeiko Carstens <heiko.carstens@de.ibm.com>
Signed-off-by: NMartin Schwidefsky <schwidefsky@de.ibm.com>

If the kernel detects that the s390 hardware supports the vector
facility, it is enabled by default at an early stage.  To force
it off, use the novx kernel parameter.  Note that there is a small
time window, where the vector facility is enabled before it is
forced to be off.

With enabling the vector facility by default, the FPU save and
restore functions can be improved.  They do not longer require
to manage expensive control register updates to enable or disable
the vector enablement control for particular processes.
Signed-off-by: NHendrik Brueckner <brueckner@linux.vnet.ibm.com>
Reviewed-by: NHeiko Carstens <heiko.carstens@de.ibm.com>
Signed-off-by: NMartin Schwidefsky <schwidefsky@de.ibm.com>

To be able to analyse problems in regard to hypervisor overhead
add a tracepoing for diagnose calls. It reports the number of
the diagnose issued, e.g.

            sshd-1385  [002] ....    42.701431: diagnose: nr=0x9c
          <idle>-0     [001] ..s.    43.587528: diagnose: nr=0x9c
Signed-off-by: NMartin Schwidefsky <schwidefsky@de.ibm.com>

Introduce /sys/debug/kernel/diag_stat with a statistic how many diagnose
calls have been done by each CPU in the system.
Signed-off-by: NMartin Schwidefsky <schwidefsky@de.ibm.com>

The <linux/memblock.h> already provides for_each_mem_range() macro that
iterates through memblock areas from type_a and not included in type_b.
We can remove custom for_each_dump_mem_range() macro and use the
for_each_mem_range() instead.
Signed-off-by: NAlexander Kuleshov <kuleshovmail@gmail.com>
Signed-off-by: NHeiko Carstens <heiko.carstens@de.ibm.com>

By definition smp_wmb only orders writes against writes. (Finish all
previous writes, and do not start any future write). To protect the
vdso init code against early reads on other CPUs, let's use a full
smp_mb at the end of vdso init. As right now smp_wmb is implemented
as full serialization, this needs no stable backport, but this change
will be necessary if we reimplement smp_wmb.
Signed-off-by: NChristian Borntraeger <borntraeger@de.ibm.com>
Signed-off-by: NMartin Schwidefsky <schwidefsky@de.ibm.com>

The calculation for the SMT scaling factor for a hardware thread
which has been partially idle needs to disregard the cycles spent
by the other threads of the core while the thread is idle.
Signed-off-by: NMartin Schwidefsky <schwidefsky@de.ibm.com>

As discussed on linux-arch all architectures should wire up the separate
system calls that are hidden behind the socketcall multiplexer system call.

It's just a couple more system calls and gives us a very small performance
improvement.
Signed-off-by: NHeiko Carstens <heiko.carstens@de.ibm.com>
Signed-off-by: NMartin Schwidefsky <schwidefsky@de.ibm.com>

A couple of compat wrapper functions are simply trampolines to the real
system call. This happened because the compat wrapper defines will only
sign and zero extend system call parameters which are of different size
on s390/s390x (longs and pointers).
All other parameters will be correctly sign and zero extended by the
normal system call wrappers.
Signed-off-by: NHeiko Carstens <heiko.carstens@de.ibm.com>
Signed-off-by: NMartin Schwidefsky <schwidefsky@de.ibm.com>

Add notrace to the compat wrapper define to disable tracing of compat
wrapper functions. These are supposed to be very small and more or less
just a trampoline to the real system call.

Also fix indentation.
Signed-off-by: NHeiko Carstens <heiko.carstens@de.ibm.com>
Signed-off-by: NMartin Schwidefsky <schwidefsky@de.ibm.com>

Signed-off-by: NMathieu Desnoyers <mathieu.desnoyers@efficios.com>
CC: Andrew Morton <akpm@linux-foundation.org>
CC: linux-api@vger.kernel.org
CC: Heiko Carstens <heiko.carstens@de.ibm.com>
CC: linux-s390@vger.kernel.org
Signed-off-by: NMartin Schwidefsky <schwidefsky@de.ibm.com>

Signed-off-by: NHeiko Carstens <heiko.carstens@de.ibm.com>
Signed-off-by: NMartin Schwidefsky <schwidefsky@de.ibm.com>

The scaled cputime is supposed to be derived from the normal per-thread
cputime by dividing it with the average thread density in the last interval.

The calculation of the scaling values for the average thread density is
incorrect. The current, incorrect calculation:

    Ci = cycle count with i active threads
    T = unscaled cputime, sT = scaled cputime
    sT = T * (C1 + C2 + ... + Cn) / (1*C1 + 2*C2 + ... + n*Cn)

The calculation happens to yield the correct numbers for the simple cases
with only one Ci value not zero. But for cases with multiple Ci values not
zero it fails. E.g. on a SMT-2 system with one thread active half the time
and two threads active for the other half of the time it fails, the scaling
factor should be 3/4 but the formula gives 2/3.

The correct formula is

    sT = T * (C1/1 + C2/2 + ... + Cn/n) / (C1 + C2 + ... + Cn)
Signed-off-by: NMartin Schwidefsky <schwidefsky@de.ibm.com>

Previously, the cpum_cf PMU returned -EPERM if a counter is requested and
the counter set to which the counter belongs is not authorized.  According
to the perf_event_open() system call manual, an error code of EPERM indicates
an unsupported exclude setting or CAP_SYS_ADMIN is missing.

Use ENOENT to indicate that particular counters are not available when the
counter set which contains the counter is not authorized.  For generic events,
this might trigger a fall back, for example, to a software event.
Signed-off-by: NHendrik Brueckner <brueckner@linux.vnet.ibm.com>
Signed-off-by: NMartin Schwidefsky <schwidefsky@de.ibm.com>

The uc_sigmask in the ucontext structure is an array of words to keep
the 64 signal bits (or 1024 if you ask glibc but the kernel sigset_t
only has 64 bits).

For 64 bit the sigset_t contains a single 8 byte word, but for 31 bit
there are two 4 byte words. The compat signal handler code uses a
simple copy of the 64 bit sigset_t to the 31 bit compat_sigset_t.
As s390 is a big-endian architecture this is incorrect, the two words
in the 31 bit sigset_t array need to be swapped.

Cc: <stable@vger.kernel.org>
Reported-by: NStefan Liebler <stli@linux.vnet.ibm.com>
Signed-off-by: NMartin Schwidefsky <schwidefsky@de.ibm.com>

The critical section cleanup code misses to add the offset of the
thread_struct to the task address.
Therefore, if the critical section code gets executed, it may corrupt
the task struct or restore the contents of the floating point registers
from the wrong memory location.
Fixes d0164ee2 "s390/kernel: remove save_fpu_regs() parameter and use
__LC_CURRENT instead".
Signed-off-by: NHeiko Carstens <heiko.carstens@de.ibm.com>
Reviewed-by: NHendrik Brueckner <brueckner@linux.vnet.ibm.com>
Signed-off-by: NMartin Schwidefsky <schwidefsky@de.ibm.com>

The swsusp_arch_suspend()/swsusp_arch_resume() functions currently only
save and restore the floating point registers. If the task that started
the hibernation process is using vector registers they can get lost.
To fix this just call save_fpu_regs in swsusp_arch_suspend(), the restore
will happen automatically on return to user space.
Reported-by: NHeiko Carstens <heiko.carstens@de.ibm.com>
Signed-off-by: NMartin Schwidefsky <schwidefsky@de.ibm.com>

printk() supports %*ph format specifier for printing a small buffers,
let's use it intead of %02x %02x...
Signed-off-by: NAlexander Kuleshov <kuleshovmail@gmail.com>
Signed-off-by: NMartin Schwidefsky <schwidefsky@de.ibm.com>

Change machine_check_init() to an early_initcall(). This makes sure it will
be called before all other cpus are online and therfore saves us a lot of
pointless smp_call_function() calls.
The control register settings will be forwarded to the other cpus when they
will be brought online.
Signed-off-by: NHeiko Carstens <heiko.carstens@de.ibm.com>
Reviewed-by: NHendrik Brueckner <brueckner@linux.vnet.ibm.com>
Signed-off-by: NMartin Schwidefsky <schwidefsky@de.ibm.com>

The novx parameter disables the vector facility but the HWCAP_S390_VXRS
bit in the ELf hardware capabilies is always set if the machine has
the vector facility. If the user space program uses the "vx" string
in the features field of /proc/cpuinfo to utilize vector instruction
it will crash if the novx kernel paramter is set.

Convert setup_hwcaps to an arch_initcall and use MACHINE_HAS_VX to
decide if the HWCAPS_S390_VXRS bit needs to be set.

Cc: stable@vger.kernel.org # 3.18+
Reported-by: NUlrich Weigand <uweigand@de.ibm.com>
Reviewed-by: NHeiko Carstens <heiko.carstens@de.ibm.com>
Signed-off-by: NMartin Schwidefsky <schwidefsky@de.ibm.com>

Remove the two facility bits

50 - constrained transactional-execution facility
74 - transactional-execution facility

from the required facilities if the kernel is built with -march=zEC12.

E.g. z/VM 6.3 doesn't virtualize the TX facility yet. Therefore a kernel
built with -march=zEC12 and ipl'ed on a zEC12 machine as a z/VM 6.3 guest
will emit a message about the missing facilities and stop working.

The kernel however doesn't make use of the TX facility, therefore remove
the two TX related facility bits and fix this unpleasant behavior.
Signed-off-by: NHeiko Carstens <heiko.carstens@de.ibm.com>
Reviewed-by: NDavid Hildenbrand <dahi@linux.vnet.ibm.com>
Signed-off-by: NMartin Schwidefsky <schwidefsky@de.ibm.com>

By accident this level has been removed by the NUMA infrastructure patch.
For non-NUMA systems with CPUs that span more than one book, this makes
the scheduler only use one of the books and the other books remain idle.

Fix this and re-add the missing level.

For NUMA and non-NUMA we have the following scheduling domains and groups:

 - SMT  (Groups: CPU threads)
 - MC   (Groups: Cores)
 - BOOK (Groups: Books)

For the non-NUMA case we have one last level scheduling domain:

 - DIE  (Groups: Whole system, has all CPUs -> cpu_cpu_mask)

For the NUMA case we have the following two last level scheduling domains:

 - DIE  (Groups: NUMA nodes -> cpu_cpu_mask -> returns node siblings)
 - NUMA (Groups: Whole system, has all CPUs -> created in sched_init_numa())

Fixes: e8054b65 ("s390/numa: add topology tree infrastructure")
Reported-and-tested-by: NEvgeny Cherkashin <Eugene.Crosser@ru.ibm.com>
Signed-off-by: NMichael Holzheu <holzheu@linux.vnet.ibm.com>
Acked-by: NHeiko Carstens <heiko.carstens@de.ibm.com>
Signed-off-by: NMartin Schwidefsky <schwidefsky@de.ibm.com>

Migrate s390 driver to the new 'set-state' interface provided by
clockevents core, the earlier 'set-mode' interface is marked obsolete
now.

This also enables us to implement callbacks for new states of clockevent
devices, for example: ONESHOT_STOPPED.

We weren't doing anything in the ->set_mode() callback. So, this patch
doesn't provide any set-state callbacks.

Cc: Martin Schwidefsky <schwidefsky@de.ibm.com>
Cc: Heiko Carstens <heiko.carstens@de.ibm.com>
Cc: linux390@de.ibm.com
Cc: linux-s390@vger.kernel.org
Signed-off-by: NViresh Kumar <viresh.kumar@linaro.org>
Signed-off-by: NDaniel Lezcano <daniel.lezcano@linaro.org>

As proposed by Andy Lutomirski create the SysV and the GNU hash
for the vdso objects. This may make some dynamic loaders a bit
faster.
Signed-off-by: NMartin Schwidefsky <schwidefsky@de.ibm.com>

Since we are already protected by the "sched_domains_mutex" lock, we can
safely remove the topology lock.
Reviewed-by: NHeiko Carstens <heiko.carstens@de.ibm.com>
Signed-off-by: NMichael Holzheu <holzheu@linux.vnet.ibm.com>
Signed-off-by: NMartin Schwidefsky <schwidefsky@de.ibm.com>

If the host has STP enabled, the TOD of the host will be changed during
synchronization phases. These are performed during a stop_machine() call.

As the guest TOD is based on the host TOD, we have to make sure that:
- no VCPU is in the SIE (implicitly guaranteed via stop_machine())
- manual guest TOD calculations are not affected

"Epoch" is the guest TOD clock delta to the host TOD clock. We have to
adjust that value during the STP synchronization and make sure that code
that accesses the epoch won't get interrupted in between (via disabling
preemption).
Signed-off-by: NFan Zhang <zhangfan@linux.vnet.ibm.com>
Reviewed-by: NDavid Hildenbrand <dahi@linux.vnet.ibm.com>
Reviewed-by: NChristian Borntraeger <borntraeger@de.ibm.com>
Acked-by: NMartin Schwidefsky <schwidefsky@de.ibm.com>
Signed-off-by: NChristian Borntraeger <borntraeger@de.ibm.com>

The MT scaling values are updated on each calll to do_account_vtime.
This function is called for each HZ interrupt and for each context
switch. Context switch can happen often, the STCCTM instruction
on this path is noticeable. Limit the updates to once per jiffy.
Signed-off-by: NMartin Schwidefsky <schwidefsky@de.ibm.com>

Enable core NUMA support for s390 and add one simple default mode "plain"
that creates one single NUMA node.

This patch contains several changes from Michael Holzheu.
Signed-off-by: NPhilipp Hachtmann <phacht@linux.vnet.ibm.com>
Signed-off-by: NMichael Holzheu <holzheu@linux.vnet.ibm.com>
Signed-off-by: NMartin Schwidefsky <schwidefsky@de.ibm.com>

Since we've already stepped away from ENABLE is a JMP and DISABLE is a
NOP with the branch_default bits, and are going to make it even worse,
rename it to make it all clearer.

This way we don't mix multiple levels of logic attributes, but have a
plain 'physical' name for what the current instruction patching status
of a jump label is.

This is a first step in removing the naming confusion that has led to
a stream of avoidable bugs such as:

  a833581e ("x86, perf: Fix static_key bug in load_mm_cr4()")
Signed-off-by: NPeter Zijlstra (Intel) <peterz@infradead.org>
Cc: Andrew Morton <akpm@linux-foundation.org>
Cc: Linus Torvalds <torvalds@linux-foundation.org>
Cc: Paul E. McKenney <paulmck@linux.vnet.ibm.com>
Cc: Peter Zijlstra <peterz@infradead.org>
Cc: Thomas Gleixner <tglx@linutronix.de>
Cc: linux-kernel@vger.kernel.org
[ Beefed up the changelog. ]
Signed-off-by: NIngo Molnar <mingo@kernel.org>

Right now we use the address of the sie control block as tag for
the sampling data. This is hard to get for users. Let's just use
the PID of the cpu thread to mark the hardware samples.
Reviewed-by: NDavid Hildenbrand <dahi@linux.vnet.ibm.com>
Signed-off-by: NChristian Borntraeger <borntraeger@de.ibm.com>
Signed-off-by: NMartin Schwidefsky <schwidefsky@de.ibm.com>

All calls to save_fpu_regs() specify the fpu structure of the current task
pointer as parameter.  The task pointer of the current task can also be
retrieved from the CPU lowcore directly.  Remove the parameter definition,
load the __LC_CURRENT task pointer from the CPU lowcore, and rebase the FPU
structure onto the task structure.  Apply the same approach for the
load_fpu_regs() function.
Reviewed-by: NHeiko Carstens <heiko.carstens@de.ibm.com>
Signed-off-by: NHendrik Brueckner <brueckner@linux.vnet.ibm.com>
Signed-off-by: NMartin Schwidefsky <schwidefsky@de.ibm.com>

 Section mismatch in reference from the function __smp_store_cpu_state()
  to the function .init.text:memblock_alloc()
The function __smp_store_cpu_state() references
the function __init memblock_alloc().
Reviewed-by: NMichael Holzheu <holzheu@linux.vnet.ibm.com>
Signed-off-by: NHeiko Carstens <heiko.carstens@de.ibm.com>
Signed-off-by: NMartin Schwidefsky <schwidefsky@de.ibm.com>

The 31-bit assembler code for the early sclp console is error
prone as git commit fde24b54d976cc123506695c17db01438a11b673
"s390/sclp: clear upper register halves in _sclp_print_early"
has shown.

Convert the assembler code to C.
Signed-off-by: NMartin Schwidefsky <schwidefsky@de.ibm.com>

Stephen Powell reported the following crash on a z890 machine:

Kernel BUG at 00000000001219d0 [verbose debug info unavailable]
illegal operation: 0001 ilc:3 [#1] SMP
Krnl PSW : 0704e00180000000 00000000001219d0 (init_cache_level+0x38/0xe0)
	   R:0 T:1 IO:1 EX:1 Key:0 M:1 W:0 P:0 AS:3 CC:2 PM:0 EA:3
Krnl Code: 00000000001219c2: a7840056		brc	8,121a6e
	   00000000001219c6: a7190000		lghi	%r1,0
	  #00000000001219ca: eb101000004c	ecag	%r1,%r0,0(%r1)
	  >00000000001219d0: a7390000		lghi	%r3,0
	   00000000001219d4: e310f0a00024	stg	%r1,160(%r15)
	   00000000001219da: a7080000		lhi	%r0,0
	   00000000001219de: a7b9f000		lghi	%r11,-4096
	   00000000001219e2: c0a0002899d9	larl	%r10,634d94
Call Trace:
 [<0000000000478ee2>] detect_cache_attributes+0x2a/0x2b8
 [<000000000097c9b0>] cacheinfo_sysfs_init+0x60/0xc8
 [<00000000001001c0>] do_one_initcall+0x98/0x1c8
 [<000000000094fdc2>] kernel_init_freeable+0x212/0x2d8
 [<000000000062352e>] kernel_init+0x26/0x118
 [<000000000062fd2e>] kernel_thread_starter+0x6/0xc

The illegal operation was executed because of a missing facility check,
which should have made sure that the ECAG execution would only be executed
on machines which have the general-instructions-extension facility
installed.
Reported-and-tested-by: NStephen Powell <zlinuxman@wowway.com>
Cc: stable@vger.kernel.org # v4.0+
Signed-off-by: NHeiko Carstens <heiko.carstens@de.ibm.com>
Signed-off-by: NMartin Schwidefsky <schwidefsky@de.ibm.com>

Replace the deprecated atomic_{set,clear}_mask() usage with the now
ubiquous atomic_{or,andnot}() functions.
Signed-off-by: NPeter Zijlstra (Intel) <peterz@infradead.org>
Signed-off-by: NThomas Gleixner <tglx@linutronix.de>

If a machine checks is received while the CPU is in the kernel, only
the s390_do_machine_check function will be called. The call to
s390_handle_mcck is postponed until the CPU returns to user space.
Because of this it is safe to use the asynchronous stack for machine
checks even if the CPU is already handling an interrupt.
Signed-off-by: NMartin Schwidefsky <schwidefsky@de.ibm.com>