For non-QEBSM devices, get_buf_states() merges PENDING and EMPTY buffers
into a single group of finished buffers. To allow the upper-layer driver
to differentiate between the two states, qdio_check_pending() looks at
each buffer's state again and sets the sbal_state flag to
QDIO_OUTBUF_STATE_FLAG_PENDING accordingly.

So effectively we're spending overhead on _every_ Output Queue
inspection, just to avoid some additional TX completion calls in case
a group of buffers has completed with mixed EMPTY / PENDING state.
Given that PENDING buffers should rarely occur, this is a bad trade-off.
In particular so as the additional checks in get_buf_states() affect
_all_ device types (even those that don't use the PENDING state).

Rip it all out, and just report the PENDING completions separately as
we already do for QEBSM devices.
Signed-off-by: NJulian Wiedmann <jwi@linux.ibm.com>
Reviewed-by: NBenjamin Block <bblock@linux.ibm.com>
Signed-off-by: NVasily Gorbik <gor@linux.ibm.com>

For QEBSM devices the 'merge_pending' mechanism in get_buf_states()
doesn't apply, and we can actually get SLSB_P_OUTPUT_PENDING returned.

So for this case propagating the PENDING state to the driver via the
queue's sbal_state doesn't make sense and creates unnecessary overhead.
Instead introduce a new QDIO_ERROR_* flag that gets passed to the
driver, and triggers the same processing as if the buffers were flagged
as QDIO_OUTBUF_STATE_FLAG_PENDING.
Signed-off-by: NJulian Wiedmann <jwi@linux.ibm.com>
Reviewed-by: NBenjamin Block <bblock@linux.ibm.com>
Signed-off-by: NVasily Gorbik <gor@linux.ibm.com>

When inspecting a queue, any error is currently returned back through
the queue's qdio_error field. Turn this into a proper variable that gets
passed through the call chain, so that the lifetime is clear and the
error state can be accessed along the way.
Signed-off-by: NJulian Wiedmann <jwi@linux.ibm.com>
Reviewed-by: NBenjamin Block <bblock@linux.ibm.com>
Signed-off-by: NVasily Gorbik <gor@linux.ibm.com>

We don't kick the handler for Input Queues anymore. Move the remaining
code into its only caller.
Signed-off-by: NJulian Wiedmann <jwi@linux.ibm.com>
Reviewed-by: NBenjamin Block <bblock@linux.ibm.com>
Signed-off-by: NVasily Gorbik <gor@linux.ibm.com>

Remove get_tod_clock_ext() and the STORE_CLOCK_EXT_SIZE define. This
enforces all users of the existing low level functions to use a union
tod_clock.

This way there is now a compile time check for the correct time and
therefore also if the size of the argument matches what will be
written to by the STORE CLOCK EXTENDED instruction.
Signed-off-by: NHeiko Carstens <hca@linux.ibm.com>
Signed-off-by: NVasily Gorbik <gor@linux.ibm.com>

Use store_tod_clock_ext() in order to be able to get rid
get_tod_clock_ext().
Signed-off-by: NHeiko Carstens <hca@linux.ibm.com>
Signed-off-by: NVasily Gorbik <gor@linux.ibm.com>

Use store_tod_clock_ext() instead of get_tod_clock_ext().
Unfortunately one usage has to be converted to a cast, since
otherwise a uapi header file would have to be changed.
Signed-off-by: NHeiko Carstens <hca@linux.ibm.com>
Signed-off-by: NVasily Gorbik <gor@linux.ibm.com>

Signed-off-by: NHeiko Carstens <hca@linux.ibm.com>
Signed-off-by: NVasily Gorbik <gor@linux.ibm.com>

Use union tod_clock and get rid of the kvm specific struct
kvm_s390_tod_clock_ext which apparently was introduced for the same
purpose.
Signed-off-by: NHeiko Carstens <hca@linux.ibm.com>
Signed-off-by: NVasily Gorbik <gor@linux.ibm.com>

Signed-off-by: NHeiko Carstens <hca@linux.ibm.com>
Signed-off-by: NVasily Gorbik <gor@linux.ibm.com>

Convert tod_clock_base to union tod_clock. This simplifies quite a bit
of code and also fixes a bug in read_persistent_clock64();

void read_persistent_clock64(struct timespec64 *ts)
{
        __u64 delta;

        delta = initial_leap_seconds + TOD_UNIX_EPOCH;
        get_tod_clock_ext(clk);
        *(__u64 *) &clk[1] -= delta;
        if (*(__u64 *) &clk[1] > delta)
                clk[0]--;
        ext_to_timespec64(clk, ts);
}

Assume &clk[1] == 3 and delta == 2; then after the substraction the if
condition becomes true and the epoch part of the clock is decremented
by one because of an assumed overflow, even though there is none.

Fix this by using 128 bit arithmetics and let the compiler do the
right thing:

void read_persistent_clock64(struct timespec64 *ts)
{
        union tod_clock clk;
        u64 delta;

        delta = initial_leap_seconds + TOD_UNIX_EPOCH;
        store_tod_clock_ext(&clk);
        clk.eitod -= delta;
        ext_to_timespec64(&clk, ts);
}
Signed-off-by: NHeiko Carstens <hca@linux.ibm.com>
Signed-off-by: NVasily Gorbik <gor@linux.ibm.com>

Introduce new store_tod_clock_ext() function, which is the same like
store_tod_clock_ext_cc() except that it doesn't return a condition
code.
Signed-off-by: NHeiko Carstens <hca@linux.ibm.com>
Signed-off-by: NVasily Gorbik <gor@linux.ibm.com>

Rename store_tod_clock_ext() to store_tod_clock_ext_cc() to reflect
that it returns a condition code and also use union tod_clock as
parameter.
Signed-off-by: NHeiko Carstens <hca@linux.ibm.com>
Signed-off-by: NVasily Gorbik <gor@linux.ibm.com>

Introduce union tod_clock which is supposed to be used to decode and
access various fields of the result of STORE CLOCK EXTENDED.
Signed-off-by: NHeiko Carstens <hca@linux.ibm.com>
Signed-off-by: NVasily Gorbik <gor@linux.ibm.com>

s390 and alpha are the only 64 bit architectures with a 32-bit ino_t.
Since this is quite unusual this causes bugs from time to time.

See e.g. commit ebce3eb2 ("ceph: fix inode number handling on
arches with 32-bit ino_t") for an example.

This (obviously) also prevents s390 and alpha to use 64-bit ino_t for
tmpfs. See commit b85a7a8b ("tmpfs: disallow CONFIG_TMPFS_INODE64
on s390").

Therefore switch both s390 and alpha to 64-bit ino_t. This should only
have an effect on the ustat system call. To prevent ABI breakage
define struct ustat compatible to the old layout and change
sys_ustat() accordingly.
Acked-by: NLinus Torvalds <torvalds@linux-foundation.org>
Signed-off-by: NHeiko Carstens <hca@linux.ibm.com>
Signed-off-by: NVasily Gorbik <gor@linux.ibm.com>

The current code uses the address in %r11 to figure out whether
it was called from the machine check handler or from a normal
interrupt handler. Instead of doing this implicit logic (which
is mostly a leftover from the old critical cleanup approach)
just add a second label and use that.
Signed-off-by: NSven Schnelle <svens@linux.ibm.com>
Reviewed-by: NHeiko Carstens <hca@linux.ibm.com>
Signed-off-by: NVasily Gorbik <gor@linux.ibm.com>

Instead of thrashing r11 which is normally our pointer to struct
pt_regs on the stack, use r13 as temporary register in the BR_EX
macro. r13 is already used in cleanup_sie, so no need to thrash
another register.
Signed-off-by: NSven Schnelle <svens@linux.ibm.com>
Reviewed-by: NHeiko Carstens <hca@linux.ibm.com>
Signed-off-by: NVasily Gorbik <gor@linux.ibm.com>

If a machine check is coming in during sie, the PU saves the
control registers to the machine check save area. Afterwards
mcck_int_handler is called, which loads __LC_KERNEL_ASCE into
%cr1. Later the code restores %cr1 from the machine check area,
but that is wrong when SIE was interrupted because the machine
check area still contains the gmap asce. Instead it should return
with either __KERNEL_ASCE in %cr1 when interrupted in SIE or
the previous %cr1 content saved in the machine check save area.

Fixes: 87d59863 ("s390/mm: remove set_fs / rework address space handling")
Signed-off-by: NSven Schnelle <svens@linux.ibm.com>
Cc: <stable@kernel.org> # v5.8+
Reviewed-by: NHeiko Carstens <hca@linux.ibm.com>
Signed-off-by: NVasily Gorbik <gor@linux.ibm.com>

The previous code used the normal kernel stack for machine checks.
This is problematic when a machine check interrupts a system call
or interrupt handler right at the beginning where registers are set up.

Assume system_call is interrupted at the first instruction and a machine
check is triggered. The machine check handler is called, checks the PSW
to see whether it is coming from user space, notices that it is already
in kernel mode but %r15 still contains the user space stack. This would
lead to a kernel crash.

There are basically two ways of fixing that: Either using the 'critical
cleanup' approach which compares the address in the PSW to see whether
it is already at a point where the stack has been set up, or use an extra
stack for the machine check handler.

For simplicity, we will go with the second approach and allocate an extra
stack. This adds some memory overhead for large systems, but usually large
system have plenty of memory so this isn't really a concern. But it keeps
the mchk stack setup simple and less error prone.

Fixes: 0b0ed657 ("s390: remove critical section cleanup from entry.S")
Signed-off-by: NSven Schnelle <svens@linux.ibm.com>
Cc: <stable@kernel.org> # v5.8+
Reviewed-by: NHeiko Carstens <hca@linux.ibm.com>
Signed-off-by: NVasily Gorbik <gor@linux.ibm.com>

The code does:

S390_lowcore.async_stack = new + STACK_INIT_OFFSET;

But the compiler is free to first assign one value and
add the other value later. If a IRQ would be coming in
between these two operations, it would run with an invalid
stack. Prevent this by using WRITE_ONCE.
Signed-off-by: NSven Schnelle <svens@linux.ibm.com>
Signed-off-by: NVasily Gorbik <gor@linux.ibm.com>

This is a preparation patch for two later bugfixes. In the past both
int_handler and machine check handler used SWITCH_KERNEL to switch to
the kernel stack. However, SWITCH_KERNEL doesn't work properly in machine
check context. So instead of adding more complexity to this macro, just
remove it.
Signed-off-by: NSven Schnelle <svens@linux.ibm.com>
Cc: <stable@kernel.org> # v5.8+
Reviewed-by: NHeiko Carstens <hca@linux.ibm.com>
Signed-off-by: NVasily Gorbik <gor@linux.ibm.com>

Use a cpu alternative to switch between stck and stckf instead of
making it compile time dependent. This will also make kernels compiled
for old machines, but running on newer machines, use stckf.
Signed-off-by: NHeiko Carstens <hca@linux.ibm.com>
Signed-off-by: NVasily Gorbik <gor@linux.ibm.com>

Add support for alternative inline assemblies with input and output
arguments. This is consistent to x86.
Signed-off-by: NHeiko Carstens <hca@linux.ibm.com>
Signed-off-by: NVasily Gorbik <gor@linux.ibm.com>

Use a cpu alternative to switch between stck and stckf instead of
making it compile time dependent. This will also make kernels compiled
for old machines, but running on newer machines, use stckf.
Reviewed-by: NChristian Borntraeger <borntraeger@de.ibm.com>
Signed-off-by: NHeiko Carstens <hca@linux.ibm.com>
Signed-off-by: NVasily Gorbik <gor@linux.ibm.com>

Use STORE CLOCK EXTENDED instead of STORE CLOCK in early tod clock
setup. This is just to remove another usage of stck, trying to remove
all usages of STORE CLOCK.  This doesn't fix anything.
Reviewed-by: NChristian Borntraeger <borntraeger@de.ibm.com>
Signed-off-by: NHeiko Carstens <hca@linux.ibm.com>
Signed-off-by: NVasily Gorbik <gor@linux.ibm.com>

Use get_tod_clock_fast() instead of store_tod_clock(), since
store_tod_clock() can be very slow.
Reviewed-by: NChristian Borntraeger <borntraeger@de.ibm.com>
Signed-off-by: NHeiko Carstens <hca@linux.ibm.com>
Signed-off-by: NVasily Gorbik <gor@linux.ibm.com>

The stck/stckf instruction used within the inline assembly within
do_account_vtime() changes the condition code. This is not reflected
with the clobber list, and therefore might result in incorrect code
generation.

It seems unlikely that the compiler could generate incorrect code
considering the surrounding C code, but it must still be fixed.

Cc: <stable@vger.kernel.org>
Reviewed-by: NChristian Borntraeger <borntraeger@de.ibm.com>
Signed-off-by: NHeiko Carstens <hca@linux.ibm.com>
Signed-off-by: NVasily Gorbik <gor@linux.ibm.com>

This is the s390 variant of commit e6b28ec6 ("x86/vdso: On timens
page fault prefault also VVAR page").
Reviewed-by: NAlexander Gordeev <agordeev@linux.ibm.com>
Signed-off-by: NHeiko Carstens <hca@linux.ibm.com>
Signed-off-by: NVasily Gorbik <gor@linux.ibm.com>

Implement generic vdso time namespace support which also enables time
namespaces for s390. This is quite similar to what arm64 has.
Reviewed-by: NAlexander Gordeev <agordeev@linux.ibm.com>
Signed-off-by: NHeiko Carstens <hca@linux.ibm.com>
Signed-off-by: NVasily Gorbik <gor@linux.ibm.com>

Use the passed in vdso_data pointer instead of calculating it again.
This is also required as a prerequisite for vdso time namespaces: if a
process is part of a time namespace __arch_get_vdso_data() will return
a pointer to the time namespace data page instead of the vdso data
page, which is not what __arch_get_hw_counter() expects.
Reviewed-by: NAlexander Gordeev <agordeev@linux.ibm.com>
Signed-off-by: NHeiko Carstens <hca@linux.ibm.com>
Signed-off-by: NVasily Gorbik <gor@linux.ibm.com>

For consistency with x86 and arm64 move the data page before code
pages. Similar to commit 601255ae ("arm64: vdso: move data page
before code pages").
Reviewed-by: NAlexander Gordeev <agordeev@linux.ibm.com>
Signed-off-by: NHeiko Carstens <hca@linux.ibm.com>
Signed-off-by: NVasily Gorbik <gor@linux.ibm.com>

Add a separate "[vvar]" mapping for the vdso datapage, since it
doesn't need to be executable or COW-able.

This is actually the s390 implementation of commit 87154938
("arm64: vdso: put vdso datapage in a separate vma")
Reviewed-by: NAlexander Gordeev <agordeev@linux.ibm.com>
Signed-off-by: NHeiko Carstens <hca@linux.ibm.com>
Signed-off-by: NVasily Gorbik <gor@linux.ibm.com>

Implement vdso mapping similar to arm64 and powerpc.
Reviewed-by: NAlexander Gordeev <agordeev@linux.ibm.com>
Signed-off-by: NHeiko Carstens <hca@linux.ibm.com>
Signed-off-by: NVasily Gorbik <gor@linux.ibm.com>

- remove unneeded includes
- move functions around
- remove obvious and/or incorrect comments
- shorten some if conditions

No functional change.
Signed-off-by: NHeiko Carstens <hca@linux.ibm.com>
Signed-off-by: NVasily Gorbik <gor@linux.ibm.com>

A few local variables exist only so the contents of a global variable
can be copied to them, and use that value only for reading.
Just remove them and rename some global variables. Also change
vdso64_[start|end] to be character arrays to be consistent with other
architectures, and get rid of the global variable vdso64_kbase.
Reviewed-by: NAlexander Gordeev <agordeev@linux.ibm.com>
Signed-off-by: NHeiko Carstens <hca@linux.ibm.com>
Signed-off-by: NVasily Gorbik <gor@linux.ibm.com>

vdso_pages (aka vdso64_pages) is never 0, therefore remove the check.
Reviewed-by: NAlexander Gordeev <agordeev@linux.ibm.com>
Signed-off-by: NHeiko Carstens <hca@linux.ibm.com>
Signed-off-by: NVasily Gorbik <gor@linux.ibm.com>

Handle allocation error gracefully and simply disable vdso instead of
leaving the system in an undefined state.
Reviewed-by: NAlexander Gordeev <agordeev@linux.ibm.com>
Signed-off-by: NHeiko Carstens <hca@linux.ibm.com>
Signed-off-by: NVasily Gorbik <gor@linux.ibm.com>

The vdso is (and must) be page aligned and its size must also be
a multiple of PAGE_SIZE. Therefore no need to round upwards.
Reviewed-by: NAlexander Gordeev <agordeev@linux.ibm.com>
Signed-off-by: NHeiko Carstens <hca@linux.ibm.com>
Signed-off-by: NVasily Gorbik <gor@linux.ibm.com>

Convert vdso_init() to arch_initcall like it is on all other architectures.
This requires to remove the vdso_getcpu_init() call from vdso_init()
since it must be called before smp is enabled.
vdso_getcpu_init() is now an early_initcall like on powerpc.
Reviewed-by: NAlexander Gordeev <agordeev@linux.ibm.com>
Signed-off-by: NHeiko Carstens <hca@linux.ibm.com>
Signed-off-by: NVasily Gorbik <gor@linux.ibm.com>

The vdso data page actually contains an array. Fix that.
This doesn't fix a real bug, just reflects reality.
Reviewed-by: NAlexander Gordeev <agordeev@linux.ibm.com>
Signed-off-by: NHeiko Carstens <hca@linux.ibm.com>
Signed-off-by: NVasily Gorbik <gor@linux.ibm.com>