Commit 0f54b14e ("arm64: cpufeature: Change read_cpuid() to use
sysreg's mrs_s macro") changed read_cpuid to require a SYS_ prefix on
register names, to allow manual assembly of registers unknown by the
toolchain, using tables in sysreg.h.

This interacts poorly with commit 42b55734 ("efi/arm64: Check
for h/w support before booting a >4 KB granular kernel"), which is
curretly queued via the tip tree, and uses read_cpuid without a SYS_
prefix. Due to this, a build of next-20160304 fails if EFI and 64K pages
are selected.

To avoid this issue when trees are merged, move the required SYS_
prefixing into read_cpuid, and revert all of the updated callsites to
pass plain register names. This effectively reverts the bulk of commit
0f54b14e.
Signed-off-by: NMark Rutland <mark.rutland@arm.com>
Cc: James Morse <james.morse@arm.com>
Signed-off-by: NCatalin Marinas <catalin.marinas@arm.com>

On ThunderX T88 pass 1.x through 2.1 parts, broadcast TLBI
instructions may cause the icache to become corrupted if it contains
data for a non-current ASID.

This patch implements the workaround (which invalidates the local
icache when switching the mm) by using code patching.
Signed-off-by: NAndrew Pinski <apinski@cavium.com>
Signed-off-by: NDavid Daney <david.daney@cavium.com>
Reviewed-by: NWill Deacon <will.deacon@arm.com>
Signed-off-by: NCatalin Marinas <catalin.marinas@arm.com>

Now that we have a clear understanding of the sign of a feature,
rename the routines to reflect the sign, so that it is not misused.
The cpuid_feature_extract_field() now accepts a 'sign' parameter.
Signed-off-by: NSuzuki K. Poulose <suzuki.poulose@arm.com>
Acked-by: NWill Deacon <will.deacon@arm.com>
Acked-by: NMarc Zyngier <marc.zyngier@arm.com>
Signed-off-by: NCatalin Marinas <catalin.marinas@arm.com>

Use the appropriate accessor for the feature bit by keeping
track of the sign of the feature
Signed-off-by: NSuzuki K. Poulose <suzuki.poulose@arm.com>
Acked-by: NWill Deacon <will.deacon@arm.com>
Signed-off-by: NCatalin Marinas <catalin.marinas@arm.com>

We verify the capabilities of the secondary CPUs only when
hotplug is enabled. The boot time activated CPUs do not
go through the verification by checking whether the system
wide capabilities were initialised or not.

This patch removes the capability check dependency on CONFIG_HOTPLUG_CPU,
to make sure that all the secondary CPUs go through the check.
The boot time activated CPUs will still skip the system wide
capability check. The plan is to hook in a check for CPU features
used by the kernel at early boot up, based on the Boot CPU values.

Cc: Mark Rutland <mark.rutland@arm.com>
Acked-by: NWill Deacon <will.deacon@arm.com>
Signed-off-by: NSuzuki K Poulose <suzuki.poulose@arm.com>
Signed-off-by: NCatalin Marinas <catalin.marinas@arm.com>

If a CPU supports both Privileged Access Never (PAN) and User Access
Override (UAO), we don't need to disable/re-enable PAN round all
copy_to_user() like calls.

UAO alternatives cause these calls to use the 'unprivileged' load/store
instructions, which are overridden to be the privileged kind when
fs==KERNEL_DS.

This patch changes the copy_to_user() calls to have their PAN toggling
depend on a new composite 'feature' ARM64_ALT_PAN_NOT_UAO.

If both features are detected, PAN will be enabled, but the copy_to_user()
alternatives will not be applied. This means PAN will be enabled all the
time for these functions. If only PAN is detected, the toggling will be
enabled as normal.

This will save the time taken to disable/re-enable PAN, and allow us to
catch copy_to_user() accesses that occur with fs==KERNEL_DS.

Futex and swp-emulation code continue to hang their PAN toggling code on
ARM64_HAS_PAN.
Signed-off-by: NJames Morse <james.morse@arm.com>
Signed-off-by: NCatalin Marinas <catalin.marinas@arm.com>

'User Access Override' is a new ARMv8.2 feature which allows the
unprivileged load and store instructions to be overridden to behave in
the normal way.

This patch converts {get,put}_user() and friends to use ldtr*/sttr*
instructions - so that they can only access EL0 memory, then enables
UAO when fs==KERNEL_DS so that these functions can access kernel memory.

This allows user space's read/write permissions to be checked against the
page tables, instead of testing addr<USER_DS, then using the kernel's
read/write permissions.
Signed-off-by: NJames Morse <james.morse@arm.com>
[catalin.marinas@arm.com: move uao_thread_switch() above dsb()]
Signed-off-by: NCatalin Marinas <catalin.marinas@arm.com>

Older assemblers may not have support for newer feature registers. To get
round this, sysreg.h provides a 'mrs_s' macro that takes a register
encoding and generates the raw instruction.

Change read_cpuid() to use mrs_s in all cases so that new registers
don't have to be a special case. Including sysreg.h means we need to move
the include and definition of read_cpuid() after the #ifndef __ASSEMBLY__
to avoid syntax errors in vmlinux.lds.
Signed-off-by: NJames Morse <james.morse@arm.com>
Acked-by: NMark Rutland <mark.rutland@arm.com>
Signed-off-by: NCatalin Marinas <catalin.marinas@arm.com>

Most CPUs have a hardware prefetcher which generally performs better
without explicit prefetch instructions issued by software, however
some CPUs (e.g. Cavium ThunderX) rely solely on explicit prefetch
instructions.

This patch adds an alternative pattern (ARM64_HAS_NO_HW_PREFETCH) to
allow our library code to make use of explicit prefetch instructions
during things like copy routines only when the CPU does not have the
capability to perform the prefetching itself.
Signed-off-by: NWill Deacon <will.deacon@arm.com>
Tested-by: NAndrew Pinski <apinski@cavium.com>
Signed-off-by: NCatalin Marinas <catalin.marinas@arm.com>

Some of the feature bits have unsigned values and need
to be treated accordingly to avoid errors. Adds the property
to the feature bits and use the appropriate field extract helpers.
Reported-by: NAKASHI Takahiro <takahiro.akashi@linaro.org>
Cc: Ard Biesheuvel <ard.biesheuvel@linaro.org>
Signed-off-by: NSuzuki K. Poulose <suzuki.poulose@arm.com>
Signed-off-by: NCatalin Marinas <catalin.marinas@arm.com>

The cpuid_feature_extract_field() extracts the feature value
as a signed integer. This could be problematic for features
whose values are unsigned. e.g, ID_AA64DFR0_EL1:BRPs. Add
an unsigned variant for the unsigned fields.

Cc: Ard Biesheuvel <ard.biesheuvel@linaro.org>
Reported-by: NAKASHI Takahiro <takahiro.akashi@linaro.org>
Signed-off-by: NSuzuki K. Poulose <suzuki.poulose@arm.com>
Signed-off-by: NCatalin Marinas <catalin.marinas@arm.com>

Cortex-A57 parts up to r1p2 can misreport Stage 2 translation faults
when a Stage 1 permission fault or device alignment fault should
have been reported.

This patch implements the workaround (which is to validate that the
Stage-1 translation actually succeeds) by using code patching.

Cc: stable@vger.kernel.org
Reviewed-by: NWill Deacon <will.deacon@arm.com>
Signed-off-by: NMarc Zyngier <marc.zyngier@arm.com>
Signed-off-by: NChristoffer Dall <christoffer.dall@linaro.org>

Extend struct arm64_cpu_capabilities to handle the HWCAP detection
and make use of the system wide value of the feature registers for
a reliable set of HWCAPs.
Signed-off-by: NSuzuki K. Poulose <suzuki.poulose@arm.com>
Tested-by: NDave Martin <Dave.Martin@arm.com>
Signed-off-by: NCatalin Marinas <catalin.marinas@arm.com>

Now that we can reliably read the system wide safe value for a
feature register, use that to compute the system capability.
This patch also replaces the 'feature-register-specific'
methods with a generic routine to check the capability.
Signed-off-by: NSuzuki K. Poulose <suzuki.poulose@arm.com>
Tested-by: NDave Martin <Dave.Martin@arm.com>
Signed-off-by: NCatalin Marinas <catalin.marinas@arm.com>

At the moment we run through the arm64_features capability list for
each CPU and set the capability if one of the CPU supports it. This
could be problematic in a heterogeneous system with differing capabilities.
Delay the CPU feature checks until all the enabled CPUs are up(i.e,
smp_cpus_done(), so that we can make better decisions based on the
overall system capability. Once we decide and advertise the capabilities
the alternatives can be applied. From this state, we cannot roll back
a feature to disabled based on the values from a new hotplugged CPU,
due to the runtime patching and other reasons. So, for all new CPUs,
we need to make sure that they have the established system capabilities.
Failing which, we bring the CPU down, preventing it from turning online.
Once the capabilities are decided, any new CPU booting up goes through
verification to ensure that it has all the enabled capabilities and also
invokes the respective enable() method on the CPU.

The CPU errata checks are not delayed and is still executed per-CPU
to detect the respective capabilities. If we ever come across a non-errata
capability that needs to be checked on each-CPU, we could introduce them via
a new capability table(or introduce a flag), which can be processed per CPU.

The next patch will make the feature checks use the system wide
safe value of a feature register.

NOTE: The enable() methods associated with the capability is scheduled
on all the CPUs (which is the only use case at the moment). If we need
a different type of 'enable()' which only needs to be run once on any CPU,
we should be able to handle that when needed.
Signed-off-by: NSuzuki K. Poulose <suzuki.poulose@arm.com>
Tested-by: NDave Martin <Dave.Martin@arm.com>
[catalin.marinas@arm.com: static variable and coding style fixes]
Signed-off-by: NCatalin Marinas <catalin.marinas@arm.com>

check_cpu_capabilities runs through a given list of caps and
checks if the system has the cap, updates the system capability
bitmap and also runs any enable() methods associated with them.
All of this is not quite obvious from the name 'check'. This
patch splits the check_cpu_capabilities into two parts :

1) update_cpu_capabilities
 => Runs through the given list and updates the system
    wide capability map.
2) enable_cpu_capabilities
 => Runs through the given list and invokes enable() (if any)
    for the caps enabled on the system.

Cc: Andre Przywara <andre.przywara@arm.com>
Cc: Will Deacon <will.deacon@arm.com>
Suggested-by: NCatalin Marinas <catalin.marinsa@arm.com>
Signed-off-by: NSuzuki K. Poulose <suzuki.poulose@arm.com>
Tested-by: NDave Martin <Dave.Martin@arm.com>
Signed-off-by: NCatalin Marinas <catalin.marinas@arm.com>

Make use of the system wide safe register to decide the support
for mixed endian.
Signed-off-by: NSuzuki K. Poulose <suzuki.poulose@arm.com>
Tested-by: NDave Martin <Dave.Martin@arm.com>
Signed-off-by: NCatalin Marinas <catalin.marinas@arm.com>

Add an API for reading the safe CPUID value across the
system from the new infrastructure.
Signed-off-by: NSuzuki K. Poulose <suzuki.poulose@arm.com>
Tested-by: NDave Martin <Dave.Martin@arm.com>
Signed-off-by: NCatalin Marinas <catalin.marinas@arm.com>

This patch adds an infrastructure to keep track of the CPU feature
registers on the system. For each register, the infrastructure keeps
track of the system wide safe value of the feature bits. Also, tracks
the which fields of a register should be matched strictly across all
the CPUs on the system for the SANITY check infrastructure.

The feature bits are classified into following 3 types depending on
the implication of the possible values. This information is used to
decide the safe value for a feature.

LOWER_SAFE  - The smaller value is safer
HIGHER_SAFE - The bigger value is safer
EXACT       - We can't decide between the two, so a predefined safe_value is used.

This infrastructure will be later used to make better decisions for:

 - Kernel features (e.g, KVM, Debug)
 - SANITY Check
 - CPU capability
 - ELF HWCAP
 - Exposing CPU Feature register to userspace.
Signed-off-by: NSuzuki K. Poulose <suzuki.poulose@arm.com>
Tested-by: NDave Martin <Dave.Martin@arm.com>
[catalin.marinas@arm.com: whitespace fix]
Signed-off-by: NCatalin Marinas <catalin.marinas@arm.com>

Introduce a helper to extract cpuid feature for any given
width.
Signed-off-by: NSuzuki K. Poulose <suzuki.poulose@arm.com>
Tested-by: NDave Martin <Dave.Martin@arm.com>
Signed-off-by: NCatalin Marinas <catalin.marinas@arm.com>

Move the mixed endian support detection code to cpufeature.c
from cpuinfo.c. This also moves the update_cpu_features()
used by mixed endian detection code, which will get more
functionality.

Also moves the ID register field shifts to asm/sysreg.h,
where all the useful definitions will end up in later patches.
Signed-off-by: NSuzuki K. Poulose <suzuki.poulose@arm.com>
Tested-by: NDave Martin <Dave.Martin@arm.com>
Signed-off-by: NCatalin Marinas <catalin.marinas@arm.com>

Delay the ELF HWCAP initialisation until all the (enabled) CPUs are
up, i.e, smp_cpus_done(). This is in preparation for detecting the
common features across the CPUS and creating a consistent ELF HWCAP
for the system.
Signed-off-by: NSuzuki K. Poulose <suzuki.poulose@arm.com>
Tested-by: NDave Martin <Dave.Martin@arm.com>
Signed-off-by: NCatalin Marinas <catalin.marinas@arm.com>

This patch implements Cavium ThunderX erratum 23154.

The gicv3 of ThunderX requires a modified version for reading the IAR
status to ensure data synchronization. Since this is in the fast-path
and called with each interrupt, runtime patching is used using jump
label patching for smallest overhead (no-op). This is the same
technique as used for tracepoints.
Signed-off-by: NRobert Richter <rrichter@cavium.com>
Reviewed-by: NMarc Zygnier <marc.zyngier@arm.com>
Acked-by: NCatalin Marinas <catalin.marinas@arm.com>
Cc: Tirumalesh Chalamarla <tchalamarla@cavium.com>
Cc: linux-arm-kernel@lists.infradead.org
Cc: Jason Cooper <jason@lakedaemon.net>
Cc: Will Deacon <will.deacon@arm.com>
Link: http://lkml.kernel.org/r/1442869119-1814-3-git-send-email-rric@kernel.orgSigned-off-by: NThomas Gleixner <tglx@linutronix.de>

Other CPU features follow an 'ARM64_HAS_*' naming scheme, so do the same
for the LSE atomics.
Signed-off-by: NWill Deacon <will.deacon@arm.com>

Add a CPU feature for the LSE atomic instructions, so that they can be
patched in at runtime when we detect that they are supported.
Reviewed-by: NSteve Capper <steve.capper@arm.com>
Reviewed-by: NCatalin Marinas <catalin.marinas@arm.com>
Signed-off-by: NWill Deacon <will.deacon@arm.com>

cpufeature.h makes use of DECLARE_BITMAP, which in turn relies on the
BITS_TO_LONGS and DIV_ROUND_UP macros.

This patch includes kernel.h in cpufeature.h to prevent all users having
to do the same thing.
Acked-by: NCatalin Marinas <catalin.marinas@arm.com>
Signed-off-by: NWill Deacon <will.deacon@arm.com>

'Privileged Access Never' is a new arm8.1 feature which prevents
privileged code from accessing any virtual address where read or write
access is also permitted at EL0.

This patch enables the PAN feature on all CPUs, and modifies {get,put}_user
helpers temporarily to permit access.

This will catch kernel bugs where user memory is accessed directly.
'Unprivileged loads and stores' using ldtrb et al are unaffected by PAN.
Reviewed-by: NCatalin Marinas <catalin.marinas@arm.com>
Signed-off-by: NJames Morse <james.morse@arm.com>
[will: use ALTERNATIVE in asm and tidy up pan_enable check]
Signed-off-by: NWill Deacon <will.deacon@arm.com>

When a new cpu feature is available, the cpu feature bits will have some
initial value, which is incremented when the feature is updated.
This patch changes 'register_value' to be 'min_field_value', and checks
the feature bits value (interpreted as a signed int) is greater than this
minimum.
Acked-by: NCatalin Marinas <catalin.marinas@arm.com>
Signed-off-by: NJames Morse <james.morse@arm.com>
Signed-off-by: NWill Deacon <will.deacon@arm.com>

This patch adds an 'enable()' callback to cpu capability/feature
detection, allowing features that require some setup or configuration
to get this opportunity once the feature has been detected.
Acked-by: NCatalin Marinas <catalin.marinas@arm.com>
Signed-off-by: NJames Morse <james.morse@arm.com>
Signed-off-by: NWill Deacon <will.deacon@arm.com>

Based on arch/arm/include/asm/cputype.h, this function does the
shifting and sign extension necessary when accessing cpu feature fields.
Signed-off-by: NJames Morse <james.morse@arm.com>
Suggested-by: NRussell King <linux@arm.linux.org.uk>
Acked-by: NCatalin Marinas <catalin.marinas@arm.com>
Signed-off-by: NWill Deacon <will.deacon@arm.com>

Add a new item to the feature set (ARM64_HAS_SYSREG_GIC_CPUIF)
to indicate that we have a system register GIC CPU interface

This will help KVM switching to alternative instruction patching.
Reviewed-by: NAndre Przywara <andre.przywara@arm.com>
Reviewed-by: NChristoffer Dall <christoffer.dall@linaro.org>
Acked-by: NWill Deacon <will.deacon@arm.com>
Signed-off-by: NMarc Zyngier <marc.zyngier@arm.com>
Signed-off-by: NCatalin Marinas <catalin.marinas@arm.com>

When running a compat (AArch32) userspace on Cortex-A53, a load at EL0
from a virtual address that matches the bottom 32 bits of the virtual
address used by a recent load at (AArch64) EL1 might return incorrect
data.

This patch works around the issue by writing to the contextidr_el1
register on the exception return path when returning to a 32-bit task.
This workaround is patched in at runtime based on the MIDR value of the
processor.
Reviewed-by: NMarc Zyngier <marc.zyngier@arm.com>
Tested-by: NMark Rutland <mark.rutland@arm.com>
Signed-off-by: NWill Deacon <will.deacon@arm.com>

As we detect more architectural features at runtime, it makes
sense to reuse the existing framework whilst avoiding to call
a feature an erratum...

This patch extract the core capability parsing, moves it into
a new file (cpufeature.c), and let the CPU errata detection code
use it.
Reviewed-by: NAndre Przywara <andre.przywara@arm.com>
Acked-by: NWill Deacon <will.deacon@arm.com>
Signed-off-by: NMarc Zyngier <marc.zyngier@arm.com>
Signed-off-by: NWill Deacon <will.deacon@arm.com>

This patch keeps track of the mixed endian EL0 support across
the system and provides helper functions to export it. The status
is a boolean indicating whether all the CPUs on the system supports
mixed endian at EL0.
Signed-off-by: NSuzuki K. Poulose <suzuki.poulose@arm.com>
Cc: Will Deacon <will.deacon@arm.com>
Signed-off-by: NCatalin Marinas <catalin.marinas@arm.com>

Building arm64.allmodconfig leads to the following warning:

usb/gadget/function/f_ncm.c:203:0: warning: "NCAPS" redefined
 #define NCAPS (USB_CDC_NCM_NCAP_ETH_FILTER | USB_CDC_NCM_NCAP_CRC_MODE)
 ^
In file included from /home/build/work/batch/arch/arm64/include/asm/io.h:32:0,
                 from /home/build/work/batch/include/linux/clocksource.h:19,
                 from /home/build/work/batch/include/clocksource/arm_arch_timer.h:19,
                 from /home/build/work/batch/arch/arm64/include/asm/arch_timer.h:27,
                 from /home/build/work/batch/arch/arm64/include/asm/timex.h:19,
                 from /home/build/work/batch/include/linux/timex.h:65,
                 from /home/build/work/batch/include/linux/sched.h:19,
                 from /home/build/work/batch/arch/arm64/include/asm/compat.h:25,
                 from /home/build/work/batch/arch/arm64/include/asm/stat.h:23,
                 from /home/build/work/batch/include/linux/stat.h:5,
                 from /home/build/work/batch/include/linux/module.h:10,
                 from /home/build/work/batch/drivers/usb/gadget/function/f_ncm.c:19:
arch/arm64/include/asm/cpufeature.h:27:0: note: this is the location of the previous definition
 #define NCAPS     2

So add a ARM64 prefix to avoid such problem.
Reported-by: NOlof's autobuilder <build@lixom.net>
Signed-off-by: NFabio Estevam <fabio.estevam@freescale.com>
Signed-off-by: NWill Deacon <will.deacon@arm.com>

The ARM erratum 832075 applies to certain revisions of Cortex-A57,
one of the workarounds is to change device loads into using
load-aquire semantics.
This is achieved using the alternatives framework.
Signed-off-by: NAndre Przywara <andre.przywara@arm.com>
Signed-off-by: NWill Deacon <will.deacon@arm.com>

The ARM errata 819472, 826319, 827319 and 824069 define the same
workaround for these hardware issues in certain Cortex-A53 parts.
Use the new alternatives framework and the CPU MIDR detection to
patch "cache clean" into "cache clean and invalidate" instructions if
an affected CPU is detected at runtime.
Signed-off-by: NAndre Przywara <andre.przywara@arm.com>
[will: add __maybe_unused to squash gcc warning]
Signed-off-by: NWill Deacon <will.deacon@arm.com>

After each CPU has been started, we iterate through a list of
CPU features or bugs to detect CPUs which need (or could benefit
from) kernel code patches.
For each feature/bug there is a function which checks if that
particular CPU is affected. We will later provide some more generic
functions for common things like testing for certain MIDR ranges.
We do this for every CPU to cover big.LITTLE systems properly as
well.
If a certain feature/bug has been detected, the capability bit will
be set, so that later the call to apply_alternatives() will trigger
the actual code patching.
Signed-off-by: NAndre Przywara <andre.przywara@arm.com>
Signed-off-by: NWill Deacon <will.deacon@arm.com>

For taking note if at least one CPU in the system needs a bug
workaround or would benefit from a code optimization, we create a new
bitmap to hold (artificial) feature bits.
Since elf_hwcap is part of the userland ABI, we keep it alone and
introduce a new data structure for that (along with some accessors).
Signed-off-by: NAndre Przywara <andre.przywara@arm.com>
Signed-off-by: NWill Deacon <will.deacon@arm.com>

This enables support for the generic CPU feature modalias implementation that
wires up optional CPU features to udev based module autoprobing.

A file <asm/cpufeature.h> is provided that maps CPU feature numbers to
elf_hwcap bits, which is the standard way on arm64 to advertise optional CPU
features both internally and to user space.
Signed-off-by: NArd Biesheuvel <ard.biesheuvel@linaro.org>
[catalin.marinas@arm.com: removed unnecessary "!!"]
Signed-off-by: NCatalin Marinas <catalin.marinas@arm.com>