Allow the way we access members of the processor vtable to be changed
at compile time.  We will need to move to per-CPU vtables to fix the
Spectre variant 2 issues on big.Little systems.

However, we have a couple of calls that do not need the vtable
treatment, and indeed cause a kernel warning due to the (later) use
of smp_processor_id(), so also introduce the PROC_TABLE macro for
these which always use CPU 0's function pointers.
Reviewed-by: NJulien Thierry <julien.thierry@arm.com>
Signed-off-by: NRussell King <rmk+kernel@armlinux.org.uk>

Call the per-processor type check_bugs() method in the same way as we
do other per-processor functions - move the "processor." detail into
proc-fns.h.
Reviewed-by: NJulien Thierry <julien.thierry@arm.com>
Signed-off-by: NRussell King <rmk+kernel@armlinux.org.uk>

Split out the lookup of the processor type and associated error handling
from the rest of setup_processor() - we will need to use this in the
secondary CPU bringup path for big.Little Spectre variant 2 mitigation.
Reviewed-by: NJulien Thierry <julien.thierry@arm.com>
Signed-off-by: NRussell King <rmk+kernel@armlinux.org.uk>

Move lookup_processor_type() out of the __init section so it is callable
from (eg) the secondary startup code during hotplug.
Reviewed-by: NJulien Thierry <julien.thierry@arm.com>
Signed-off-by: NRussell King <rmk+kernel@armlinux.org.uk>

In vfp_preserve_user_clear_hwstate, ufp_exc->fpinst2 gets assigned to
itself. It should actually be hwstate->fpinst2 that gets assigned to the
ufp_exc field.

Fixes commit 3aa2df6e ("ARM: 8791/1:
vfp: use __copy_to_user() when saving VFP state").
Reported-by: NDavid Binderman <dcb314@hotmail.com>
Signed-off-by: NJulien Thierry <julien.thierry@arm.com>
Signed-off-by: NRussell King <rmk+kernel@armlinux.org.uk>

Due to what appears to be a copy/paste error, the opening ENTRY()
of cpu_v7_hvc_switch_mm() lacks a matching ENDPROC(), and instead,
the one for cpu_v7_smc_switch_mm() is duplicated.

Given that it is ENDPROC() that emits the Thumb annotation, the
cpu_v7_hvc_switch_mm() routine will be called in ARM mode on a
Thumb2 kernel, resulting in the following splat:

  Internal error: Oops - undefined instruction: 0 [#1] SMP THUMB2
  Modules linked in:
  CPU: 0 PID: 1 Comm: swapper/0 Not tainted 4.18.0-rc1-00030-g4d28ad89189d-dirty #488
  Hardware name: QEMU KVM Virtual Machine, BIOS 0.0.0 02/06/2015
  PC is at cpu_v7_hvc_switch_mm+0x12/0x18
  LR is at flush_old_exec+0x31b/0x570
  pc : [<c0316efe>]    lr : [<c04117c7>]    psr: 00000013
  sp : ee899e50  ip : 00000000  fp : 00000001
  r10: eda28f34  r9 : eda31800  r8 : c12470e0
  r7 : eda1fc00  r6 : eda53000  r5 : 00000000  r4 : ee88c000
  r3 : c0316eec  r2 : 00000001  r1 : eda53000  r0 : 6da6c000
  Flags: nzcv  IRQs on  FIQs on  Mode SVC_32  ISA ARM  Segment none

Note the 'ISA ARM' in the last line.

Fix this by using the correct name in ENDPROC().

Cc: <stable@vger.kernel.org>
Fixes: 10115105 ("ARM: spectre-v2: add firmware based hardening")
Reviewed-by: NDave Martin <Dave.Martin@arm.com>
Acked-by: NMarc Zyngier <marc.zyngier@arm.com>
Signed-off-by: NArd Biesheuvel <ard.biesheuvel@linaro.org>
Signed-off-by: NRussell King <rmk+kernel@armlinux.org.uk>

Sanitize user pointer given to __copy_to_user, both for standard version
and memcopy version of the user accessor.
Signed-off-by: NJulien Thierry <julien.thierry@arm.com>
Signed-off-by: NRussell King <rmk+kernel@armlinux.org.uk>

Introduce C and asm helpers to sanitize user address, taking the
address range they target into account.

Use asm helper for existing sanitization in __copy_from_user().
Signed-off-by: NJulien Thierry <julien.thierry@arm.com>
Signed-off-by: NRussell King <rmk+kernel@armlinux.org.uk>

When Spectre mitigation is required, __put_user() needs to include
check_uaccess. This is already the case for put_user(), so just make
__put_user() an alias of put_user().
Signed-off-by: NJulien Thierry <julien.thierry@arm.com>
Signed-off-by: NRussell King <rmk+kernel@armlinux.org.uk>

A mispredicted conditional call to set_fs could result in the wrong
addr_limit being forwarded under speculation to a subsequent access_ok
check, potentially forming part of a spectre-v1 attack using uaccess
routines.

This patch prevents this forwarding from taking place, but putting heavy
barriers in set_fs after writing the addr_limit.

Porting commit c2f0ad4f ("arm64: uaccess: Prevent speculative use
of the current addr_limit").
Signed-off-by: NJulien Thierry <julien.thierry@arm.com>
Signed-off-by: NRussell King <rmk+kernel@armlinux.org.uk>

With Spectre-v1.1 mitigations, __put_user_error is pointless. In an attempt
to remove it, replace its references in frame setups with __put_user.
Signed-off-by: NJulien Thierry <julien.thierry@arm.com>
Signed-off-by: NRussell King <rmk+kernel@armlinux.org.uk>

Copy events to user using __copy_to_user() rather than copy members of
individually with __put_user_error().
This has the benefit of disabling/enabling PAN once per event intead of
once per event member.
Signed-off-by: NJulien Thierry <julien.thierry@arm.com>
Signed-off-by: NRussell King <rmk+kernel@armlinux.org.uk>

Use __copy_to_user() rather than __put_user_error() for individual
members when saving VFP state.
This has the benefit of disabling/enabling PAN once per copied struct
intead of once per write.
Signed-off-by: NJulien Thierry <julien.thierry@arm.com>
Signed-off-by: NRussell King <rmk+kernel@armlinux.org.uk>

When setting a dummy iwmmxt context, create a local instance and
use __copy_to_user both cases whether iwmmxt is being used or not.
This has the benefit of disabling/enabling PAN once for the whole copy
intead of once per write.
Signed-off-by: NJulien Thierry <julien.thierry@arm.com>
Signed-off-by: NRussell King <rmk+kernel@armlinux.org.uk>

When saving the ARM integer registers, use __copy_to_user() to
copy them into user signal frame, rather than __put_user_error().
This has the benefit of disabling/enabling PAN once for the whole copy
intead of once per write.
Signed-off-by: NJulien Thierry <julien.thierry@arm.com>
Signed-off-by: NRussell King <rmk+kernel@armlinux.org.uk>

Spectre variant 1 attacks are about this sequence of pseudo-code:

	index = load(user-manipulated pointer);
	access(base + index * stride);

In order for the cache side-channel to work, the access() must me made
to memory which userspace can detect whether cache lines have been
loaded.  On 32-bit ARM, this must be either user accessible memory, or
a kernel mapping of that same user accessible memory.

The problem occurs when the load() speculatively loads privileged data,
and the subsequent access() is made to user accessible memory.

Any load() which makes use of a user-maniplated pointer is a potential
problem if the data it has loaded is used in a subsequent access.  This
also applies for the access() if the data loaded by that access is used
by a subsequent access.

Harden the get_user() accessors against Spectre attacks by forcing out
of bounds addresses to a NULL pointer.  This prevents get_user() being
used as the load() step above.  As a side effect, put_user() will also
be affected even though it isn't implicated.

Also harden copy_from_user() by redoing the bounds check within the
arm_copy_from_user() code, and NULLing the pointer if out of bounds.
Acked-by: NMark Rutland <mark.rutland@arm.com>
Signed-off-by: NRussell King <rmk+kernel@armlinux.org.uk>

Fixing __get_user() for spectre variant 1 is not sane: we would have to
add address space bounds checking in order to validate that the location
should be accessed, and then zero the address if found to be invalid.

Since __get_user() is supposed to avoid the bounds check, and this is
exactly what get_user() does, there's no point having two different
implementations that are doing the same thing.  So, when the Spectre
workarounds are required, make __get_user() an alias of get_user().
Acked-by: NMark Rutland <mark.rutland@arm.com>
Signed-off-by: NRussell King <rmk+kernel@armlinux.org.uk>

Borrow the x86 implementation of __inttype() to use in get_user() to
select an integer type suitable to temporarily hold the result value.
This is necessary to avoid propagating the volatile nature of the
result argument, which can cause the following warning:

lib/iov_iter.c:413:5: warning: optimization may eliminate reads and/or writes to register variables [-Wvolatile-register-var]
Acked-by: NMark Rutland <mark.rutland@arm.com>
Signed-off-by: NRussell King <rmk+kernel@armlinux.org.uk>

__get_user_error() is used as a fast accessor to make copying structure
members as efficient as possible.  However, with software PAN and the
recent Spectre variant 1, the efficiency is reduced as these are no
longer fast accessors.

In the case of software PAN, it has to switch the domain register around
each access, and with Spectre variant 1, it would have to repeat the
access_ok() check for each access.

Rather than using __get_user_error() to copy each semops element member,
copy each semops element in full using __copy_from_user().
Acked-by: NMark Rutland <mark.rutland@arm.com>
Signed-off-by: NRussell King <rmk+kernel@armlinux.org.uk>

__get_user_error() is used as a fast accessor to make copying structure
members in the signal handling path as efficient as possible.  However,
with software PAN and the recent Spectre variant 1, the efficiency is
reduced as these are no longer fast accessors.

In the case of software PAN, it has to switch the domain register around
each access, and with Spectre variant 1, it would have to repeat the
access_ok() check for each access.

Use __copy_from_user() rather than __get_user_err() for individual
members when restoring VFP state.
Acked-by: NMark Rutland <mark.rutland@arm.com>
Signed-off-by: NRussell King <rmk+kernel@armlinux.org.uk>

__get_user_error() is used as a fast accessor to make copying structure
members in the signal handling path as efficient as possible.  However,
with software PAN and the recent Spectre variant 1, the efficiency is
reduced as these are no longer fast accessors.

In the case of software PAN, it has to switch the domain register around
each access, and with Spectre variant 1, it would have to repeat the
access_ok() check for each access.

It becomes much more efficient to use __copy_from_user() instead, so
let's use this for the ARM integer registers.
Acked-by: NMark Rutland <mark.rutland@arm.com>
Signed-off-by: NRussell King <rmk+kernel@armlinux.org.uk>

Prevent speculation at the syscall table decoding by clamping the index
used to zero on invalid system call numbers, and using the csdb
speculative barrier.
Signed-off-by: NRussell King <rmk+kernel@armlinux.org.uk>
Acked-by: NMark Rutland <mark.rutland@arm.com>
Boot-tested-by: NTony Lindgren <tony@atomide.com>
Reviewed-by: NTony Lindgren <tony@atomide.com>

Add an implementation of the array_index_mask_nospec() function for
mitigating Spectre variant 1 throughout the kernel.
Signed-off-by: NRussell King <rmk+kernel@armlinux.org.uk>
Acked-by: NMark Rutland <mark.rutland@arm.com>
Boot-tested-by: NTony Lindgren <tony@atomide.com>
Reviewed-by: NTony Lindgren <tony@atomide.com>

Add assembly and C macros for the new CSDB instruction.
Signed-off-by: NRussell King <rmk+kernel@armlinux.org.uk>
Acked-by: NMark Rutland <mark.rutland@arm.com>
Boot-tested-by: NTony Lindgren <tony@atomide.com>
Reviewed-by: NTony Lindgren <tony@atomide.com>

Report support for SMCCC_ARCH_WORKAROUND_1 to KVM guests for affected
CPUs.
Signed-off-by: NRussell King <rmk+kernel@armlinux.org.uk>
Boot-tested-by: NTony Lindgren <tony@atomide.com>
Reviewed-by: NTony Lindgren <tony@atomide.com>
Reviewed-by: NMarc Zyngier <marc.zyngier@arm.com>

We want SMCCC_ARCH_WORKAROUND_1 to be fast. As fast as possible.
So let's intercept it as early as we can by testing for the
function call number as soon as we've identified a HVC call
coming from the guest.
Signed-off-by: NRussell King <rmk+kernel@armlinux.org.uk>
Boot-tested-by: NTony Lindgren <tony@atomide.com>
Reviewed-by: NTony Lindgren <tony@atomide.com>
Reviewed-by: NMarc Zyngier <marc.zyngier@arm.com>

Include Brahma B15 in the Spectre v2 KVM workarounds.
Signed-off-by: NRussell King <rmk+kernel@armlinux.org.uk>
Acked-by: NFlorian Fainelli <f.fainelli@gmail.com>
Boot-tested-by: NTony Lindgren <tony@atomide.com>
Reviewed-by: NTony Lindgren <tony@atomide.com>
Acked-by: NMarc Zyngier <marc.zyngier@arm.com>

In order to avoid aliasing attacks against the branch predictor
on Cortex-A15, let's invalidate the BTB on guest exit, which can
only be done by invalidating the icache (with ACTLR[0] being set).

We use the same hack as for A12/A17 to perform the vector decoding.
Signed-off-by: NMarc Zyngier <marc.zyngier@arm.com>
Signed-off-by: NRussell King <rmk+kernel@armlinux.org.uk>
Boot-tested-by: NTony Lindgren <tony@atomide.com>
Reviewed-by: NTony Lindgren <tony@atomide.com>

In order to avoid aliasing attacks against the branch predictor,
let's invalidate the BTB on guest exit. This is made complicated
by the fact that we cannot take a branch before invalidating the
BTB.

We only apply this to A12 and A17, which are the only two ARM
cores on which this useful.
Signed-off-by: NMarc Zyngier <marc.zyngier@arm.com>
Signed-off-by: NRussell King <rmk+kernel@armlinux.org.uk>
Boot-tested-by: NTony Lindgren <tony@atomide.com>
Reviewed-by: NTony Lindgren <tony@atomide.com>

Warn at error level if the context switching function is not what we
are expecting.  This can happen with big.Little systems, which we
currently do not support.
Signed-off-by: NRussell King <rmk+kernel@armlinux.org.uk>
Boot-tested-by: NTony Lindgren <tony@atomide.com>
Reviewed-by: NTony Lindgren <tony@atomide.com>
Acked-by: NMarc Zyngier <marc.zyngier@arm.com>

Add firmware based hardening for cores that require more complex
handling in firmware.
Signed-off-by: NRussell King <rmk+kernel@armlinux.org.uk>
Boot-tested-by: NTony Lindgren <tony@atomide.com>
Reviewed-by: NTony Lindgren <tony@atomide.com>
Reviewed-by: NMarc Zyngier <marc.zyngier@arm.com>

In order to prevent aliasing attacks on the branch predictor,
invalidate the BTB or instruction cache on CPUs that are known to be
affected when taking an abort on a address that is outside of a user
task limit:

Cortex A8, A9, A12, A17, A73, A75: flush BTB.
Cortex A15, Brahma B15: invalidate icache.

If the IBE bit is not set, then there is little point to enabling the
workaround.
Signed-off-by: NRussell King <rmk+kernel@armlinux.org.uk>
Boot-tested-by: NTony Lindgren <tony@atomide.com>
Reviewed-by: NTony Lindgren <tony@atomide.com>

When the branch predictor hardening is enabled, firmware must have set
the IBE bit in the auxiliary control register.  If this bit has not
been set, the Spectre workarounds will not be functional.

Add validation that this bit is set, and print a warning at alert level
if this is not the case.
Signed-off-by: NRussell King <rmk+kernel@armlinux.org.uk>
Reviewed-by: NFlorian Fainelli <f.fainelli@gmail.com>
Boot-tested-by: NTony Lindgren <tony@atomide.com>
Reviewed-by: NTony Lindgren <tony@atomide.com>

Harden the branch predictor against Spectre v2 attacks on context
switches for ARMv7 and later CPUs.  We do this by:

Cortex A9, A12, A17, A73, A75: invalidating the BTB.
Cortex A15, Brahma B15: invalidating the instruction cache.

Cortex A57 and Cortex A72 are not addressed in this patch.

Cortex R7 and Cortex R8 are also not addressed as we do not enforce
memory protection on these cores.
Signed-off-by: NRussell King <rmk+kernel@armlinux.org.uk>
Boot-tested-by: NTony Lindgren <tony@atomide.com>
Reviewed-by: NTony Lindgren <tony@atomide.com>
Acked-by: NMarc Zyngier <marc.zyngier@arm.com>

Add a Kconfig symbol for CPUs which are vulnerable to the Spectre
attacks.
Signed-off-by: NRussell King <rmk+kernel@armlinux.org.uk>
Reviewed-by: NFlorian Fainelli <f.fainelli@gmail.com>
Boot-tested-by: NTony Lindgren <tony@atomide.com>
Reviewed-by: NTony Lindgren <tony@atomide.com>
Acked-by: NMarc Zyngier <marc.zyngier@arm.com>

Add support for per-processor bug checking - each processor function
descriptor gains a function pointer for this check, which must not be
an __init function.  If non-NULL, this will be called whenever a CPU
enters the kernel via which ever path (boot CPU, secondary CPU startup,
CPU resuming, etc.)

This allows processor specific bug checks to validate that workaround
bits are properly enabled by firmware via all entry paths to the kernel.
Signed-off-by: NRussell King <rmk+kernel@armlinux.org.uk>
Reviewed-by: NFlorian Fainelli <f.fainelli@gmail.com>
Boot-tested-by: NTony Lindgren <tony@atomide.com>
Reviewed-by: NTony Lindgren <tony@atomide.com>
Acked-by: NMarc Zyngier <marc.zyngier@arm.com>

Check for CPU bugs when secondary processors are being brought online,
and also when CPUs are resuming from a low power mode.  This gives an
opportunity to check that processor specific bug workarounds are
correctly enabled for all paths that a CPU re-enters the kernel.
Signed-off-by: NRussell King <rmk+kernel@armlinux.org.uk>
Reviewed-by: NFlorian Fainelli <f.fainelli@gmail.com>
Boot-tested-by: NTony Lindgren <tony@atomide.com>
Reviewed-by: NTony Lindgren <tony@atomide.com>
Acked-by: NMarc Zyngier <marc.zyngier@arm.com>

Prepare the processor bug infrastructure so that it can be expanded to
check for per-processor bugs.
Signed-off-by: NRussell King <rmk+kernel@armlinux.org.uk>
Reviewed-by: NFlorian Fainelli <f.fainelli@gmail.com>
Boot-tested-by: NTony Lindgren <tony@atomide.com>
Reviewed-by: NTony Lindgren <tony@atomide.com>
Acked-by: NMarc Zyngier <marc.zyngier@arm.com>

Add CPU part numbers for Cortex A53, A57, A72, A73, A75 and the
Broadcom Brahma B15 CPU.
Signed-off-by: NRussell King <rmk+kernel@armlinux.org.uk>
Acked-by: NFlorian Fainelli <f.fainelli@gmail.com>
Boot-tested-by: NTony Lindgren <tony@atomide.com>
Reviewed-by: NTony Lindgren <tony@atomide.com>
Acked-by: NMarc Zyngier <marc.zyngier@arm.com>