Some platforms always enter the kernel in the ARM state even if
the kernel is compiled for THUMB2. Add a small wrapper on top of
cpu_resume() that switches into THUMB2 state.

This provides the functionality to fix a problem reported by Kevin
Hilman on next-20150601 where the ifc6410 fails to boot a THUMB2
kernel because the platform's firmware always enters the kernel in
ARM mode from deep idle states.

(rmk: tweaked to work without BSYM->badr changes.)
Reported-by: NKevin Hilman <khilman@linaro.org>
Cc: Ard Biesheuvel <ard.biesheuvel@linaro.org>
Cc: Lina Iyer <lina.iyer@linaro.org>
Signed-off-by: NStephen Boyd <sboyd@codeaurora.org>
Signed-off-by: NRussell King <rmk+kernel@arm.linux.org.uk>

Move cpu_resume() to the .text section where it belongs. Change
the adr reference to sleep_save_sp to an explicit PC relative
reference so sleep_save_sp itself can remain in .data.

This helps prevent linker failure on large kernels, as the code
in the .data section may be too far away to be in range for normal
b/bl instructions.
Reviewed-by: NNicolas Pitre <nico@linaro.org>
Tested-by: NSudeep Holla <sudeep.holla@arm.com>
Signed-off-by: NArd Biesheuvel <ard.biesheuvel@linaro.org>
Signed-off-by: NRussell King <rmk+kernel@arm.linux.org.uk>

ARMv6 and greater introduced a new instruction ("bx") which can be used
to return from function calls.  Recent CPUs perform better when the
"bx lr" instruction is used rather than the "mov pc, lr" instruction,
and this sequence is strongly recommended to be used by the ARM
architecture manual (section A.4.1.1).

We provide a new macro "ret" with all its variants for the condition
code which will resolve to the appropriate instruction.

Rather than doing this piecemeal, and miss some instances, change all
the "mov pc" instances to use the new macro, with the exception of
the "movs" instruction and the kprobes code.  This allows us to detect
the "mov pc, lr" case and fix it up - and also gives us the possibility
of deploying this for other registers depending on the CPU selection.
Reported-by: NWill Deacon <will.deacon@arm.com>
Tested-by: Stephen Warren <swarren@nvidia.com> # Tegra Jetson TK1
Tested-by: Robert Jarzmik <robert.jarzmik@free.fr> # mioa701_bootresume.S
Tested-by: Andrew Lunn <andrew@lunn.ch> # Kirkwood
Tested-by: NShawn Guo <shawn.guo@freescale.com>
Tested-by: Tony Lindgren <tony@atomide.com> # OMAPs
Tested-by: Gregory CLEMENT <gregory.clement@free-electrons.com> # Armada XP, 375, 385
Acked-by: Sekhar Nori <nsekhar@ti.com> # DaVinci
Acked-by: Christoffer Dall <christoffer.dall@linaro.org> # kvm/hyp
Acked-by: Haojian Zhuang <haojian.zhuang@gmail.com> # PXA3xx
Acked-by: Stefano Stabellini <stefano.stabellini@eu.citrix.com> # Xen
Tested-by: Uwe Kleine-König <u.kleine-koenig@pengutronix.de> # ARMv7M
Tested-by: Simon Horman <horms+renesas@verge.net.au> # Shmobile
Signed-off-by: NRussell King <rmk+kernel@arm.linux.org.uk>

On CPUs with virtualization extensions the kernel installs HYP mode
configuration on both primary and secondary cpus upon cold boot.

On platforms where CPUs are shutdown in idle paths (ie CPU core gating),
when a CPU resumes from low-power states it currently does not execute
code that reinstalls the HYP configuration, which means that the kernel
cannot run eg KVM properly on such machines.

This patch, mirroring cold-boot behaviour, executes position independent
code that reinstalls HYP configuration and drops to SVC mode safely on
warmboot, so that deep idle states can be enabled in kernel running as
hosts on platforms with power management HW.

Cc: Christoffer Dall <christoffer.dall@linaro.org>
Cc: Dave Martin <dave.martin@arm.com>
Cc: Marc Zyngier <marc.zyngier@arm.com>
Cc: Nicolas Pitre <nico@linaro.org>
Signed-off-by: NLorenzo Pieralisi <lorenzo.pieralisi@arm.com>
Acked-by: NMarc Zyngier <marc.zyngier@arm.com>
Reviewed-by: NDave Martin <Dave.Martin@arm.com>
Signed-off-by: NRussell King <rmk+kernel@arm.linux.org.uk>

If we are booting in LE and compiled for BE8, then add code to
set the state to bE8. Since the instruction stream is always LE,
we do not need to do anything special to the instruction.

Also ensure that the secondary processors are started in the same mode.

Note, we do add about 20 bytes to the kernel image, but it seems easier
to do this than adding another configuration to change.
Signed-off-by: NBen Dooks <ben.dooks@codethink.co.uk>
Reviewed-by: NDave Martin <Dave.Martin@arm.com>
Tested-by: NThomas Petazzoni <thomas.petazzoni@free-electrons.com>

Currently we hash the MPIDR of the CPU being suspended to determine which
entry in the sleep_save_sp array to use. In some situations, such as when
we want to resume on another physical CPU, the MPIDR of another CPU should
be used instead.

So let's use the value of cpu_logical_map(smp_processor_id()) in place
of the MPIDR in the suspend path.  This will result in the same index
being used as with the previous code unless the caller has modified
cpu_logical_map() beforehand with the MPIDR of the physical CPU the
suspending logical CPU will resume on.

Consequently, if doing a physical CPU migration, cpu_logical_map() must
be updated appropriately somewhere between cpu_pm_enter() and
cpu_suspend().

The register allocation in __cpu_suspend is reworked in order to better
accommodate the additional argument.
Signed-off-by: NNicolas Pitre <nico@linaro.org>
Acked-by: NLorenzo Pieralisi <lorenzo.pieralisi@arm.com>
Reviewed-by: NDave Martin <Dave.Martin@arm.com>

Current implementation of cpu_{suspend}/cpu_{resume} relies on the MPIDR
to index the array of pointers where the context is saved and restored.
The current approach works as long as the MPIDR can be considered a
linear index, so that the pointers array can simply be dereferenced by
using the MPIDR[7:0] value.
On ARM multi-cluster systems, where the MPIDR may not be a linear index,
to properly dereference the stack pointer array, a mapping function should
be applied to it so that it can be used for arrays look-ups.

This patch adds code in the cpu_{suspend}/cpu_{resume} implementation
that relies on shifting and ORing hashing method to map a MPIDR value to a
set of buckets precomputed at boot to have a collision free mapping from
MPIDR to context pointers.

The hashing algorithm must be simple, fast, and implementable with few
instructions since in the cpu_resume path the mapping is carried out with
the MMU off and the I-cache off, hence code and data are fetched from DRAM
with no-caching available. Simplicity is counterbalanced with a little
increase of memory (allocated dynamically) for stack pointers buckets, that
should be anyway fairly limited on most systems.

Memory for context pointers is allocated in a early_initcall with
size precomputed and stashed previously in kernel data structures.
Memory for context pointers is allocated through kmalloc; this
guarantees contiguous physical addresses for the allocated memory which
is fundamental to the correct functioning of the resume mechanism that
relies on the context pointer array to be a chunk of contiguous physical
memory. Virtual to physical address conversion for the context pointer
array base is carried out at boot to avoid fiddling with virt_to_phys
conversions in the cpu_resume path which is quite fragile and should be
optimized to execute as few instructions as possible.
Virtual and physical context pointer base array addresses are stashed in a
struct that is accessible from assembly using values generated through the
asm-offsets.c mechanism.

Cc: Will Deacon <will.deacon@arm.com>
Cc: Catalin Marinas <catalin.marinas@arm.com>
Cc: Russell King <linux@arm.linux.org.uk>
Cc: Colin Cross <ccross@android.com>
Cc: Santosh Shilimkar <santosh.shilimkar@ti.com>
Cc: Daniel Lezcano <daniel.lezcano@linaro.org>
Cc: Amit Kucheria <amit.kucheria@linaro.org>
Signed-off-by: NLorenzo Pieralisi <lorenzo.pieralisi@arm.com>
Reviewed-by: NDave Martin <Dave.Martin@arm.com>
Reviewed-by: NNicolas Pitre <nico@linaro.org>
Tested-by: NShawn Guo <shawn.guo@linaro.org>
Tested-by: NKevin Hilman <khilman@linaro.org>
Tested-by: NStephen Warren <swarren@wwwdotorg.org>

Disintegrate asm/system.h for ARM.
Signed-off-by: NDavid Howells <dhowells@redhat.com>
cc: Russell King <linux@arm.linux.org.uk>
cc: linux-arm-kernel@lists.infradead.org

Before we enable the MMU, we must ensure that the TTBR registers contain
sane values. After the MMU has been enabled, we jump to the *virtual*
address of the following function, so we also need to ensure that the
SCTLR write has taken effect.

This patch adds ISB instructions around the SCTLR write to ensure the
visibility of the above.
Signed-off-by: NWill Deacon <will.deacon@arm.com>
Signed-off-by: NCatalin Marinas <catalin.marinas@arm.com>

The ARM CPU suspend code requires cpu_resume_mmu to be identity mapped
in order to re-enable the MMU when coming out of suspend. Currently,
this is accomplished by maintaining a suspend_pgd with the relevant
mapping put in place at init time.

This patch replaces the use of suspend_pgd with the new idmap_pgd.
cpu_resume_mmu is placed in the .idmap.text section so that it is
included in the identity map.
Reviewed-by: NCatalin Marinas <catalin.marinas@arm.com>
Acked-by: NDave Martin <dave.martin@linaro.org>
Tested-by: NLorenzo Pieralisi <Lorenzo.Pieralisi@arm.com>
Signed-off-by: NWill Deacon <will.deacon@arm.com>

Convert some of the sleep.S guts to C code, which makes it easier to
use our macros and to add L2 cache handling.  We provide a helper
function, __cpu_suspend_save(), which deals with saving the common
state, setting up for resume, and flushing caches.

The remainder left as assembly code is the saving of the CPU general
purpose registers, and allocating space on the stack to save the CPU
specific registers and resume state.
Tested-by: NSantosh Shilimkar <santosh.shilimkar@ti.com>
Tested-by: NShawn Guo <shawn.guo@linaro.org>
Tested-by: NLorenzo Pieralisi <lorenzo.pieralisi@arm.com>
Signed-off-by: NRussell King <rmk+kernel@arm.linux.org.uk>

We don't require cpu_resume_turn_mmu_on as we can combine the ldr
instruction with the following code provided we ensure that
cpu_resume_mmu is aligned for older CPUs.  Note that we also align
to a 32-byte boundary to ensure that the code can't cross a section
boundary.
Tested-by: NSantosh Shilimkar <santosh.shilimkar@ti.com>
Tested-by: NShawn Guo <shawn.guo@linaro.org>
Tested-by: NLorenzo Pieralisi <lorenzo.pieralisi@arm.com>
Signed-off-by: NRussell King <rmk+kernel@arm.linux.org.uk>

Only use the preallocated page table during the resume, not while
suspending.  This avoids the overhead of having to switch unnecessarily
to the resume page table in the suspend path.
Tested-by: NSantosh Shilimkar <santosh.shilimkar@ti.com>
Tested-by: NShawn Guo <shawn.guo@linaro.org>
Tested-by: NLorenzo Pieralisi <lorenzo.pieralisi@arm.com>
Signed-off-by: NRussell King <rmk+kernel@arm.linux.org.uk>

Preallocate a page table and setup an identity mapping for the MMU
enable code.  This means we don't have to "borrow" a page table to
do this, avoiding complexities with L2 cache coherency.
Tested-by: NSantosh Shilimkar <santosh.shilimkar@ti.com>
Tested-by: NShawn Guo <shawn.guo@linaro.org>
Tested-by: NLorenzo Pieralisi <lorenzo.pieralisi@arm.com>
Signed-off-by: NRussell King <rmk+kernel@arm.linux.org.uk>

Ensure that the return value from __cpu_suspend is non-zero when
aborting.  Zero indicates a successful suspend occurred.
Tested-by: NSantosh Shilimkar <santosh.shilimkar@ti.com>
Tested-by: NShawn Guo <shawn.guo@linaro.org>
Tested-by: NLorenzo Pieralisi <lorenzo.pieralisi@arm.com>
Signed-off-by: NRussell King <rmk+kernel@arm.linux.org.uk>

There are SoCs where attempting to enter a low power state is ignored,
and the CPU continues executing instructions with all state preserved.
It is over-complex at that point to disable the MMU just to call the
resume path.

Instead, allow the suspend finisher to return error codes to abort
suspend in this circumstance, where the cpu_suspend internals will then
unwind the saved state on the stack.  Also omit the tlb flush as no
changes to the page tables will have happened.
Signed-off-by: NRussell King <rmk+kernel@arm.linux.org.uk>

The first and second arguments shouldn't concern platform code, so
hide them from each platforms caller.
Tested-by: NKevin Hilman <khilman@ti.com>
Acked-by: NJean Pihet <j-pihet@ti.com>
Signed-off-by: NRussell King <rmk+kernel@arm.linux.org.uk>

As we have core code dealing with CPU suspend/resume, we can
re-initialize the CPUs exception banked registers via that code rather
than having platforms deal with that level of detail.  So, move the
call to cpu_init() out of platform code into core code.
Tested-by: NKevin Hilman <khilman@ti.com>
Acked-by: NJean Pihet <j-pihet@ti.com>
Signed-off-by: NRussell King <rmk+kernel@arm.linux.org.uk>

cpu_suspend() has a weird calling method which makes it only possible to
call from assembly code: it returns with a modified stack pointer to
finish the suspend, but on resume, it 'returns' via a provided pointer.

We can make cpu_suspend() appear to be a normal function merely by
swapping the resume pointer argument and the link register.

Do so, and update all callers to take account of this more traditional
behaviour.
Acked-by: NFrank Hofmann <frank.hofmann@tomtom.com>
Tested-by: NKevin Hilman <khilman@ti.com>
Acked-by: NJean Pihet <j-pihet@ti.com>
Signed-off-by: NRussell King <rmk+kernel@arm.linux.org.uk>

Save the suspend function pointer onto the stack for use when returning.
Allocate r2 to pass an argument to the suspend function.
Acked-by: NFrank Hofmann <frank.hofmann@tomtom.com>
Tested-by: NKevin Hilman <khilman@ti.com>
Acked-by: NJean Pihet <j-pihet@ti.com>
Signed-off-by: NRussell King <rmk+kernel@arm.linux.org.uk>

Avoid using r2 and r3 in the suspend code, allowing these to be
passed further into the function as arguments.
Acked-by: NFrank Hofmann <frank.hofmann@tomtom.com>
Tested-by: NKevin Hilman <khilman@ti.com>
Acked-by: NJean Pihet <j-pihet@ti.com>
Signed-off-by: NRussell King <rmk+kernel@arm.linux.org.uk>

Make cpu_suspend()..return function preserve r4 to r11 across a suspend
cycle.  This is in preparation of relieving platform support code from
this task.
Acked-by: NFrank Hofmann <frank.hofmann@tomtom.com>
Tested-by: NKevin Hilman <khilman@ti.com>
Acked-by: NJean Pihet <j-pihet@ti.com>
Signed-off-by: NRussell King <rmk+kernel@arm.linux.org.uk>

Very little code is different between these two paths now, so extract
the common code.
Acked-by: NFrank Hofmann <frank.hofmann@tomtom.com>
Tested-by: NKevin Hilman <khilman@ti.com>
Acked-by: NJean Pihet <j-pihet@ti.com>
Signed-off-by: NRussell King <rmk+kernel@arm.linux.org.uk>

Move the return address for cpu_resume to the top of stack so that
cpu_resume looks more like a normal function.
Acked-by: NFrank Hofmann <frank.hofmann@tomtom.com>
Tested-by: NKevin Hilman <khilman@ti.com>
Acked-by: NJean Pihet <j-pihet@ti.com>
Signed-off-by: NRussell King <rmk+kernel@arm.linux.org.uk>

Eliminate the differences between MULTI_CPU and non-MULTI_CPU resume
paths, making the saved structure identical irrespective of the way
the kernel was configured.
Acked-by: NFrank Hofmann <frank.hofmann@tomtom.com>
Tested-by: NKevin Hilman <khilman@ti.com>
Acked-by: NJean Pihet <j-pihet@ti.com>
Signed-off-by: NRussell King <rmk+kernel@arm.linux.org.uk>

Signed-off-by: NNicolas Pitre <nicolas.pitre@linaro.org>
Reviewed-by: NDave Martin <dave.martin@linaro.org>
Signed-off-by: NRussell King <rmk+kernel@arm.linux.org.uk>

Allow the generic sleep code to be used with SMP CPU idle by storing
N CPU stack pointers rather than just one.  Tested on Assabet and
Tegra 2.
Tested-by: NColin Cross <ccross@android.com>
Signed-off-by: NRussell King <rmk+kernel@arm.linux.org.uk>

This adds core support for saving and restoring CPU coprocessor
registers for suspend/resume support.  This contains support for suspend
with ARM920, ARM926, SA11x0, PXA25x, PXA27x, PXA3xx, V6 and V7 CPUs.
Tested on Assabet and Tegra 2.
Tested-by: NColin Cross <ccross@android.com>
Tested-by: NKukjin Kim <kgene.kim@samsung.com>
Signed-off-by: NRussell King <rmk+kernel@arm.linux.org.uk>