The ARMv7 processor setup function __v7_setup() cleans and invalidates the
CPU cache before enabling MMU to start the CPU with a clean CPU local cache.

But on ARMv7 architectures like Cortex-[A15/A8], this code will end
up flushing the L2 caches(up to level of Coherency) which is undesirable
and expensive. The setup functions are used in the CPU hotplug scenario too
and hence flushing all cache levels should be avoided.

This patch replaces the cache flushing call with the newly introduced
v7 dcache LoUIS API where only cache levels up to LoUIS are cleaned and
invalidated when a processors executes __v7_setup which is the expected
behavior.

For processors like A9 and A5 where the L2 cache is an outer one the
behavior should be unchanged.
Reviewed-by: NNicolas Pitre <nico@linaro.org>
Signed-off-by: NSantosh Shilimkar <santosh.shilimkar@ti.com>
Signed-off-by: NLorenzo Pieralisi <lorenzo.pieralisi@arm.com>
Tested-by: NShawn Guo <shawn.guo@linaro.org>

Currently when ThumbEE is not enabled (!CONFIG_ARM_THUMBEE) the ThumbEE
register states are not saved/restored at context switch. The default state
of the ThumbEE Ctrl register (TEECR) allows userspace accesses to the
ThumbEE Base Handler register (TEEHBR). This can cause unexpected behaviour
when people use ThumbEE on !CONFIG_ARM_THUMBEE kernels, as well as allowing
covert communication - eg between userspace tasks running inside chroot
jails.

This patch sets up TEECR in order to prevent user-space access to TEEHBR
when !CONFIG_ARM_THUMBEE. In this case, tasks are sent SIGILL if they try to
access TEEHBR.

Cc: stable@vger.kernel.org
Reviewed-by: NWill Deacon <will.deacon@arm.com>
Signed-off-by: NJonathan Austin <jonathan.austin@arm.com>
Signed-off-by: NRussell King <rmk+kernel@arm.linux.org.uk>

Erratum #743622 affects all r2 variants of the Cortex-A9 processor, so
ensure that the workaround is applied regardless of the revision.

Cc: <stable@vger.kernel.org>
Reported-by: NRussell King <rmk+kernel@arm.linux.org.uk>
Signed-off-by: NWill Deacon <will.deacon@arm.com>
Signed-off-by: NRussell King <rmk+kernel@arm.linux.org.uk>

On v7, we use the same cache maintenance instructions for data lines
as for unified lines. This was not the case for v6, where HARVARD_CACHE
was defined to indicate the L1 cache topology.

This patch removes the erroneous compile-time check for HARVARD_CACHE in
proc-v7.S, ensuring that we perform I-side invalidation at boot.
Reported-and-Acked-by: NShawn Guo <shawn.guo@linaro.org>

Cc: stable <stable@vger.kernel.org>
Acked-by: NCatalin Marinas <Catalin.Marinas@arm.com>
Signed-off-by: NWill Deacon <will.deacon@arm.com>
Signed-off-by: NRussell King <rmk+kernel@arm.linux.org.uk>

The merging of commits 1b6ba46b ("ARM: LPAE: MMU setup for the 3-level
page table format") and b4244738 ("ARM: 7202/1: Add Cortex-A7 proc info")
during the merge window ended up putting the Cortex-A7 proc_info into a
code block guarded by !CONFIG_ARM_LPAE. This makes Cortex-A7 platforms
unbootable when LPAE is enabled.

This patch moves the proc_info structure for Cortex-A7 outside of the
guarded block.

Cc: Pawel Moll <pawel.moll@arm.com>
Acked-by: NCatalin Marinas <catalin.marinas@arm.com>
Signed-off-by: NWill Deacon <will.deacon@arm.com>
Signed-off-by: NRussell King <rmk+kernel@arm.linux.org.uk>

Activation conditions for a workaround should not be encoded in the
workaround's direct dependencies if this makes otherwise reasonable
configuration choices impossible.

This patches uses the SMP/UP patching facilities instead to compile
out the workaround if the configuration means that it is definitely
not needed.

This means that configs for buggy silicon can simply select
ARM_ERRATA_751472, without preventing a UP kernel from being built
or duplicatiing knowledge about when to activate the workaround.
This seems the correct way to do things, because the erratum is a
property of the silicon, irrespective of what the kernel config
happens to be.
Signed-off-by: NDave Martin <dave.martin@linaro.org>
Signed-off-by: NRussell King <rmk+kernel@arm.linux.org.uk>

This patch adds processor info for ARM Ltd. Cortex-A7.

A7 is architecturally identical to A15 so it shares the
same SMP initialization code and hwcaps.
Tested-by: NWill Deacon <will.deacon@arm.com>
Signed-off-by: NPawel Moll <pawel.moll@arm.com>
Signed-off-by: NRussell King <rmk+kernel@arm.linux.org.uk>

This patch adds the MMU initialisation for the LPAE page table format.
The swapper_pg_dir size with LPAE is 5 rather than 4 pages. A new
proc-v7-3level.S file contains the TTB initialisation, context switch
and PTE setting code with the LPAE. The TTBRx split is based on the
PAGE_OFFSET with TTBR1 used for the kernel mappings. The 36-bit mappings
(supersections) and a few other memory types in mmu.c are conditionally
compiled.
Signed-off-by: NCatalin Marinas <catalin.marinas@arm.com>

This patch modifies the proc-v7.S file so that it only contains code
shared between classic MMU and LPAE. The non-common code is factored out
into a separate file.
Signed-off-by: NCatalin Marinas <catalin.marinas@arm.com>

The CPU reset functions disable the MMU and therefore must be executed
with an identity mapping in place.

This patch places the CPU reset functions into the .idmap.text section,
causing the idmap code to include them as part of the identity mapping.
Acked-by: NDave Martin <dave.martin@linaro.org>
Signed-off-by: NWill Deacon <will.deacon@arm.com>

Support for the cpu_suspend functions is only built-in
when CONFIG_PM_SLEEP is enabled, but omap3/4, exynos4
and pxa always call cpu_suspend when CONFIG_PM is enabled.
Signed-off-by: NArnd Bergmann <arnd@arndb.de>

There is no need to save and restore the context ID register on ARMv6
and ARMv7 with a temporary page table as we write the context ID
register when we switch back to the real page tables for the thread.

Moreover, the temporary page tables do not contain any non-global
mappings, so the context ID value should not be used.  To be safe,
initialize the register to a reserved context ID value.
Tested-by: NSantosh Shilimkar <santosh.shilimkar@ti.com>
Tested-by: NShawn Guo <shawn.guo@linaro.org>
Tested-by: NLorenzo Pieralisi <lorenzo.pieralisi@arm.com>
Signed-off-by: NRussell King <rmk+kernel@arm.linux.org.uk>

Only use the preallocated page table during the resume, not while
suspending.  This avoids the overhead of having to switch unnecessarily
to the resume page table in the suspend path.
Tested-by: NSantosh Shilimkar <santosh.shilimkar@ti.com>
Tested-by: NShawn Guo <shawn.guo@linaro.org>
Tested-by: NLorenzo Pieralisi <lorenzo.pieralisi@arm.com>
Signed-off-by: NRussell King <rmk+kernel@arm.linux.org.uk>

Preallocate a page table and setup an identity mapping for the MMU
enable code.  This means we don't have to "borrow" a page table to
do this, avoiding complexities with L2 cache coherency.
Tested-by: NSantosh Shilimkar <santosh.shilimkar@ti.com>
Tested-by: NShawn Guo <shawn.guo@linaro.org>
Tested-by: NLorenzo Pieralisi <lorenzo.pieralisi@arm.com>
Signed-off-by: NRussell King <rmk+kernel@arm.linux.org.uk>

For ARMv7 kernels running in the non-secure world, writing to the
auxillary control register causes an abort, so we must avoid directly
writing the auxillary control register.  If the ACR has already been
reinitialized by SoC code, don't try to restore it.
Signed-off-by: NRussell King <rmk+kernel@arm.linux.org.uk>

Add a dsb after the isb to ensure that the previous writes to the
CP15 registers take effect before we enable the MMU.
Signed-off-by: NRussell King <rmk+kernel@arm.linux.org.uk>

cpu_v7_reset disables the MMU and then branches to the provided address.
On Thumb-2 kernels, we should take care to clear the Thumb Exception
enable bit in the System Control Register, otherwise this may wreak
havok in the code to which we are branching (for example, an ARM kernel
image via kexec).
Reviewed-by: NDave Martin <dave.martin@linaro.org>
Signed-off-by: NWill Deacon <will.deacon@arm.com>
Signed-off-by: NRussell King <rmk+kernel@arm.linux.org.uk>

Commit 66a625a8 (ARM: mm: proc-macros: Add generic proc/cache/tlb struct
definition macros) introduced build errors when PM_SLEEP is not enabled.
The per-CPU do_suspend/do_resume functions are defined via the
preprocessor to constant 0.  However, the macros which use these were
converted to assembly, resulting in undefined references to these
functions.  Fix that by moving the ! ifdef section into proc-macros.S
and deleting it from all effected proc-*.S files.
Acked-by: NDave Martin <dave.martin@linaro.org>
Signed-off-by: NRussell King <rmk+kernel@arm.linux.org.uk>

This patch adds simple definitions of cpu_reset for ARMv6 and ARMv7
cores, which disable the MMU via the SCTLR.
Signed-off-by: NWill Deacon <will.deacon@arm.com>

Multicore implementations of the Cortex-A15 require bit 6 of the
auxiliary control register to be set in order for cache and TLB
maintenance operations to be broadcast between CPUs.

This patch adds a new proc_info structure for Cortex-A15, which enables
the SMP bit during setup and includes the new HWCAP for integer
division.
Signed-off-by: NWill Deacon <will.deacon@arm.com>

This patch adds processor info for ARM Ltd. Cortex A5,
which has SCU initialisation procedure identical to A9.
Signed-off-by: NPawel Moll <pawel.moll@arm.com>
Signed-off-by: NWill Deacon <will.deacon@arm.com>

As most of the proc info content is common across all v7
processors, this patch converts existing A9 and generic v7
descriptions into a macro (allowing extra flags in future).
Signed-off-by: NPawel Moll <pawel.moll@arm.com>
Signed-off-by: NWill Deacon <will.deacon@arm.com>

Signed-off-by: NDave Martin <dave.martin@linaro.org>

Ensure that the TLS register is saved and restored over a suspend
cycle, so that userspace programs don't see a corrupted TLS value.
Tested-by: NKevin Hilman <khilman@ti.com>
Acked-by: NJean Pihet <j-pihet@ti.com>
Signed-off-by: NRussell King <rmk+kernel@arm.linux.org.uk>

Add the missing suspend/resume pointers for the suspend code.  This
is needed when building for multiple CPUs.
Tested-by: NKevin Hilman <khilman@ti.com>
Acked-by: NJean Pihet <j-pihet@ti.com>
Signed-off-by: NRussell King <rmk+kernel@arm.linux.org.uk>

This reverts commit 52af9c6c.

Will Deacon reports that:

 In 52af9c6c ("ARM: 6943/1: mm: use TTBR1 instead of reserved context ID")
 I updated the ASID rollover code to use only the kernel page tables
 whilst updating the ASID.

 Unfortunately, the code to restore the user page tables was part of a
 later patch which isn't yet in mainline, so this leaves the code
 quite broken.

We're also in the process of eliminating __ARCH_WANT_INTERRUPTS_ON_CTXSW
from ARM, so lets revert these until we can properly sort out what we're
doing with the ARM context switching.
Signed-off-by: NRussell King <rmk+kernel@arm.linux.org.uk>

On ARMv7 CPUs that cache first level page table entries (like the
Cortex-A15), using a reserved ASID while changing the TTBR or flushing
the TLB is unsafe.

This is because the CPU may cache the first level entry as the result of
a speculative memory access while the reserved ASID is assigned. After
the process owning the page tables dies, the memory will be reallocated
and may be written with junk values which can be interpreted as global,
valid PTEs by the processor. This will result in the TLB being populated
with bogus global entries.

This patch avoids the use of a reserved context ID in the v7 switch_mm
and ASID rollover code by temporarily using the swapper_pg_dir pointed
at by TTBR1, which contains only global entries that are not tagged
with ASIDs.
Acked-by: NCatalin Marinas <catalin.marinas@arm.com>
Signed-off-by: NWill Deacon <will.deacon@arm.com>
Signed-off-by: NRussell King <rmk+kernel@arm.linux.org.uk>

This patch makes TTBR1 point to swapper_pg_dir so that global, kernel
mappings can be used exclusively on v6 and v7 cores where they are
needed.
Signed-off-by: NCatalin Marinas <catalin.marinas@arm.com>
Signed-off-by: NWill Deacon <will.deacon@arm.com>
Signed-off-by: NRussell King <rmk+kernel@arm.linux.org.uk>

CONFIG_PM is now set whenever we support either runtime PM in addition
to suspend and hibernate.  This causes build errors when runtime PM is
enabled on a platform, but the CPU does not have the appropriate support
for suspend.

So, switch this code to use CONFIG_PM_SLEEP rather than CONFIG_PM to
allow runtime PM to be enabled without causing build errors.
Signed-off-by: NRussell King <rmk+kernel@arm.linux.org.uk>

Fixes generated by 'codespell' and manually reviewed.
Signed-off-by: NLucas De Marchi <lucas.demarchi@profusion.mobi>

On the r2p* and r3p* versions of the Cortex-A9, a speculative memory
access may cause a page table walk which starts prior to an ASID switch
but completes afterwards. This can populate the micro-TLB with a stale
entry which may be hit with the new ASID.

This workaround places two dsb instructions in the mm switching code so
that no page table walks can cross the ASID switch.
Acked-by: NCatalin Marinas <catalin.marinas@arm.com>
Signed-off-by: NWill Deacon <will.deacon@arm.com>
Signed-off-by: NRussell King <rmk+kernel@arm.linux.org.uk>

This adds core support for saving and restoring CPU coprocessor
registers for suspend/resume support.  This contains support for suspend
with ARM920, ARM926, SA11x0, PXA25x, PXA27x, PXA3xx, V6 and V7 CPUs.
Tested on Assabet and Tegra 2.
Tested-by: NColin Cross <ccross@android.com>
Tested-by: NKukjin Kim <kgene.kim@samsung.com>
Signed-off-by: NRussell King <rmk+kernel@arm.linux.org.uk>

On versions of the Cortex-A9 prior to r3p0, an interrupted ICIALLUIS
operation may prevent the completion of a following broadcasted
operation if the second operation is received by a CPU before the
ICIALLUIS has completed, potentially leading to corrupted entries in
the cache or TLB.

This workaround sets a bit in the diagnostic register of the Cortex-A9,
causing CP15 maintenance operations to be uninterruptible.
Acked-by: NCatalin Marinas <catalin.marinas@arm.com>
Signed-off-by: NWill Deacon <will.deacon@arm.com>
Signed-off-by: NRussell King <rmk+kernel@arm.linux.org.uk>

Commit d30e45ee (ARM: pgtable: switch order of Linux vs hardware page tables)
introduced a pre-increment addressing offset which is out of range for
Thumb-2.  Thumb-2 only permits offsets <256.  So split the intruction in
two for Thumb-2.
Signed-off-by: NDave Martin <dave.martin@linaro.org>
Signed-off-by: NRussell King <rmk+kernel@arm.linux.org.uk>

Signed-off-by: NRussell King <rmk+kernel@arm.linux.org.uk>

The hardware page tables use an XN bit 'execute never'.  Historically,
we've had a Linux 'execute allow' bit, in the positive sense.  Get rid
of this artifact as future hardware will continue to have the XN sense.
Signed-off-by: NRussell King <rmk+kernel@arm.linux.org.uk>

This switches the ordering of the Linux vs hardware page tables in
each page, thereby eliminating some of the arithmetic in the page
table walks.  As we now place the Linux page table at the beginning
of the page, we can deal with the offset in the pgt by simply masking
it away, along with the other control bits.

This also makes the arithmetic all be positive, rather than a mixture.
Signed-off-by: NRussell King <rmk+kernel@arm.linux.org.uk>

Directives such as .long and .word do not magically cause the
assembler location counter to become aligned in gas.  As a result,
using these directives in code sections can result in misaligned
data words when building a Thumb-2 kernel (CONFIG_THUMB2_KERNEL).

This is a Bad Thing, since the ABI permits the compiler to assume
that fundamental types of word size or above are word- aligned when
accessing them from C.  If the data is not really word-aligned,
this can cause impaired performance and stray alignment faults in
some circumstances.

In general, the following rules should be applied when using data
word declaration directives inside code sections:

    * .quad and .double:
         .align 3

    * .long, .word, .single, .float:
         .align (or .align 2)

    * .short:
        No explicit alignment required, since Thumb-2
        instructions are always 2 or 4 bytes in size.
        immediately after an instruction.

In this specific case, we can achieve the desired alignment by
forcing a 32-bit branch instruction using the W() macro, since the
assembler location counter is already 32-bit aligned in this case.
Reviewed-by: NWill Deacon <will.deacon@arm.com>
Signed-off-by: NDave Martin <dave.martin@linaro.org>
Acked-by: NCatalin Marinas <catalin.marinas@arm.com>
Signed-off-by: NRussell King <rmk+kernel@arm.linux.org.uk>

The SWP instruction was deprecated in the ARMv6 architecture,
superseded by the LDREX/STREX family of instructions for
load-linked/store-conditional operations. The ARMv7 multiprocessing
extensions mandate that SWP/SWPB instructions are treated as undefined
from reset, with the ability to enable them through the System Control
Register SW bit.

This patch adds the alternative solution to emulate the SWP and SWPB
instructions using LDREX/STREX sequences, and log statistics to
/proc/cpu/swp_emulation. To correctly deal with copy-on-write, it also
modifies cpu_v7_set_pte_ext to change the mappings to priviliged RO when
user RO.
Signed-off-by: NLeif Lindholm <leif.lindholm@arm.com>
Acked-by: NCatalin Marinas <catalin.marinas@arm.com>
Acked-by: NKirill A. Shutemov <kirill@shutemov.name>
Signed-off-by: NRussell King <rmk+kernel@arm.linux.org.uk>

This patch removes the domain switching functionality via the set_fs and
__switch_to functions on cores that have a TLS register.

Currently, the ioremap and vmalloc areas share the same level 1 page
tables and therefore have the same domain (DOMAIN_KERNEL). When the
kernel domain is modified from Client to Manager (via the __set_fs or in
the __switch_to function), the XN (eXecute Never) bit is overridden and
newer CPUs can speculatively prefetch the ioremap'ed memory.

Linux performs the kernel domain switching to allow user-specific
functions (copy_to/from_user, get/put_user etc.) to access kernel
memory. In order for these functions to work with the kernel domain set
to Client, the patch modifies the LDRT/STRT and related instructions to
the LDR/STR ones.

The user pages access rights are also modified for kernel read-only
access rather than read/write so that the copy-on-write mechanism still
works. CPU_USE_DOMAINS gets disabled only if the hardware has a TLS register
(CPU_32v6K is defined) since writing the TLS value to the high vectors page
isn't possible.

The user addresses passed to the kernel are checked by the access_ok()
function so that they do not point to the kernel space.
Tested-by: NAnton Vorontsov <cbouatmailru@gmail.com>
Cc: Tony Lindgren <tony@atomide.com>
Signed-off-by: NCatalin Marinas <catalin.marinas@arm.com>
Signed-off-by: NRussell King <rmk+kernel@arm.linux.org.uk>