With the advent of CPU_HOTPLUG, the enable-method property for CPU0 may
tells us something useful (i.e. how to hotplug it back on), so we must
read it along with all the enable-method for all the other CPUs.  Even
on UP the enable-method may tell us useful information (e.g. if a core
has some mechanism that might be usable for cpuidle), so we should
always read it.

This patch factors out the reading of the enable method, and ensures
that CPU0's enable method is read regardless of whether the kernel is
built with SMP support.
Signed-off-by: NMark Rutland <mark.rutland@arm.com>
Signed-off-by: NCatalin Marinas <catalin.marinas@arm.com>

The arm64 kernel has an internal holding pen, which is necessary for
some systems where we can't bring CPUs online individually and must hold
multiple CPUs in a safe area until the kernel is able to handle them.
The current SMP infrastructure for arm64 is closely coupled to this
holding pen, and alternative boot methods must launch CPUs into the pen,
where they sit before they are launched into the kernel proper.

With PSCI (and possibly other future boot methods), we can bring CPUs
online individually, and need not perform the secondary_holding_pen
dance. Instead, this patch factors the holding pen management code out
to the spin-table boot method code, as it is the only boot method
requiring the pen.

A new entry point for secondaries, secondary_entry is added for other
boot methods to use, which bypasses the holding pen and its associated
overhead when bringing CPUs online. The smp.pen.text section is also
removed, as the pen can live in head.text without problem.

The cpu_operations structure is extended with two new functions,
cpu_boot and cpu_postboot, for bringing a cpu into the kernel and
performing any post-boot cleanup required by a bootmethod (e.g.
resetting the secondary_holding_pen_release to INVALID_HWID).
Documentation is added for cpu_operations.
Signed-off-by: NMark Rutland <mark.rutland@arm.com>
Signed-off-by: NCatalin Marinas <catalin.marinas@arm.com>

For hotplug support, we're going to want a place to store operations
that do more than bring CPUs online, and it makes sense to group these
with our current smp_enable_ops. For cpuidle support, we'll want to
group additional functions, and we may want them even for UP kernels.

This patch renames smp_enable_ops to the more general cpu_operations,
and pulls the definitions out of smp code such that they can be used in
UP kernels. While we're at it, fix up instances of the cpu parameter to
be an unsigned int, drop the init markings and rename the *_cpu
functions to cpu_* to reduce future churn when cpu_operations is
extended.
Signed-off-by: NMark Rutland <mark.rutland@arm.com>
Signed-off-by: NCatalin Marinas <catalin.marinas@arm.com>

The functions in psci.c are only used from smp_psci.c, and smp_psci
cannot function without psci.c. Additionally psci.c is built when !SMP,
where it's expected that cpu_suspend may be useful.

This patch unifies the two files, removing pointless duplication and
paving the way for PSCI support in UP systems.
Signed-off-by: NMark Rutland <mark.rutland@arm.com>
Signed-off-by: NCatalin Marinas <catalin.marinas@arm.com>

This function may be called from loadable modules, so it needs
exporting.
Signed-off-by: NCatalin Marinas <catalin.marinas@arm.com>
Reported-by: NLoc Ho <lho@apm.com>

If context switching happens during executing fpsimd_flush_thread(),
stale value in FPSIMD registers will be saved into current thread's
fpsimd_state by fpsimd_thread_switch(). That may cause invalid
initialization state for the new process, so disable preemption
when executing fpsimd_flush_thread().
Signed-off-by: NJiang Liu <jiang.liu@huawei.com>
Cc: Jiang Liu <liuj97@gmail.com>
Signed-off-by: NCatalin Marinas <catalin.marinas@arm.com>

Under arm64 elf_hwcap is a 32 bit quantity, but it is stored in
a 64 bit auxiliary ELF field and glibc reads hwcap as 64 bit.

This patch widens elf_hwcap to be 64 bit.
Signed-off-by: NSteve Capper <steve.capper@arm.com>
Signed-off-by: NCatalin Marinas <catalin.marinas@arm.com>

When a task crashes and we print debugging information, ensure that
compat tasks show the actual AArch32 LR and SP registers rather than the
AArch64 ones.
Signed-off-by: NCatalin Marinas <catalin.marinas@arm.com>

TCR.TBI0 can be used to cause hardware address translation to ignore the
top byte of userspace virtual addresses. Whilst not especially useful in
standard C programs, this can be used by JITs to `tag' pointers with
various pieces of metadata.

This patch enables this bit for AArch64 Linux, and adds a new file to
Documentation/arm64/ which describes some potential caveats when using
tagged virtual addresses.
Signed-off-by: NWill Deacon <will.deacon@arm.com>
Signed-off-by: NCatalin Marinas <catalin.marinas@arm.com>

We always use a timer-backed delay loop for arm64, so don't bother
reporting a bogomips value which appears to confuse some people.
Signed-off-by: NWill Deacon <will.deacon@arm.com>
Signed-off-by: NCatalin Marinas <catalin.marinas@arm.com>

Most architectures use the same implementation. Collapse the common ones
into a single weak function that can be overridden.
Signed-off-by: NGrant Likely <grant.likely@linaro.org>

The current vmlinux.lds.S places the notes sections between the
end of rw data and start of bss. This means that _edata doesn't
really point to the end of data. Since notes are read-only, this
patch moves them to the read-only segment so that _edata does
point to the end of initialized rw data.
Signed-off-by: NMark Salter <msalter@redhat.com>
Signed-off-by: NCatalin Marinas <catalin.marinas@arm.com>

do_undefinstr() has to be called with interrupts disabled since it may
read the instruction from the user address space which could lead to a
data abort and subsequent might_sleep() warning in do_page_fault().
Signed-off-by: NCatalin Marinas <catalin.marinas@arm.com>

Expand the arm64 image header to allow for co-existance with
PE/COFF header required by the EFI stub.  The PE/COFF format
requires the "MZ" header to be at offset 0, and the offset
to the PE/COFF header to be at offset 0x3c.  The image
header is expanded to allow 2 instructions at the beginning
to accommodate a benign intruction at offset 0 that includes
the "MZ" header, a magic number, and the offset to the PE/COFF
header.
Signed-off-by: NRoy Franz <roy.franz@linaro.org>
Signed-off-by: NCatalin Marinas <catalin.marinas@arm.com>

Add <asm/neon.h> containing kernel_neon_begin/kernel_neon_end function
declarations and corresponding definitions in fpsimd.c

These are needed to wrap uses of NEON in kernel mode. The names are
identical to the ones used in arm/ so code using intrinsics or
vectorized by GCC can be shared between arm and arm64.
Signed-off-by: NArd Biesheuvel <ard.biesheuvel@linaro.org>
Signed-off-by: NCatalin Marinas <catalin.marinas@arm.com>

This is a port of f2fe09b0 ("ARM: 7663/1: perf: fix ARMv7 EVTYPE_MASK
to include NSH bit") to arm64, which fixes the broken evtype mask to
include the NSH bit, allowing profiling at EL2.

Cc: <stable@vger.kernel.org>
Signed-off-by: NWill Deacon <will.deacon@arm.com>
Signed-off-by: NCatalin Marinas <catalin.marinas@arm.com>

This is a port of cb2d8b34 ("ARM: 7698/1: perf: fix group validation
when using enable_on_exec") to arm64, which fixes the event validation
checking so that events in the OFF state are still considered when
enable_on_exec is true.

Cc: <stable@vger.kernel.org>
Signed-off-by: NWill Deacon <will.deacon@arm.com>
Signed-off-by: NCatalin Marinas <catalin.marinas@arm.com>

This is a port of c95eb318 ("ARM: 7809/1: perf: fix event validation
for software group leaders") to arm64, which fixes a panic in the arm64
perf backend found as a result of Vince's fuzzing tool.

Cc: <stable@vger.kernel.org>
Signed-off-by: NWill Deacon <will.deacon@arm.com>
Signed-off-by: NCatalin Marinas <catalin.marinas@arm.com>

This is a port of d9f96635 ("ARM: 7810/1: perf: Fix array out of
bounds access in armpmu_map_hw_event()") to arm64, which fixes an oops
in the arm64 perf backend found as a result of Vince's fuzzing tool.

Cc: <stable@vger.kernel.org>
Signed-off-by: NWill Deacon <will.deacon@arm.com>
Signed-off-by: NCatalin Marinas <catalin.marinas@arm.com>

Written by Catalin Marinas, tested by APM on storm platform. This is needed
because of the failures encountered when running SpecWeb benchmark test.
Signed-off-by: NFeng Kan <fkan@apm.com>
Acked-by: NKumar Sankaran <ksankaran@apm.com>
Signed-off-by: NCatalin Marinas <catalin.marinas@arm.com>

Commit ff701306 (arm64: use common reboot infrastructure) converted the
arm_pm_restart declaration to the new reboot infrastructure but missed
the actual definition.
Signed-off-by: NCatalin Marinas <catalin.marinas@arm.com>

Commit 7b6d864b (reboot: arm: change reboot_mode to use enum
reboot_mode) changed the way reboot is handled on arm, which has a
direct impact on arm64 as we share the reset driver on the VE platform.

The obvious fix is to move arm64 to use the same infrastructure.
Signed-off-by: NMarc Zyngier <marc.zyngier@arm.com>
[catalin.marinas@arm.com: removed reboot_mode = REBOOT_HARD default setting]
Signed-off-by: NCatalin Marinas <catalin.marinas@arm.com>

There is a slight chance that (timer) interrupts are triggered before a
secondary CPU has been marked online with implications on softirq thread
affinity.
Signed-off-by: NCatalin Marinas <catalin.marinas@arm.com>
Reported-by: NKirill Tkhai <tkhai@yandex.ru>

The __cpuinit type of throwaway sections might have made sense
some time ago when RAM was more constrained, but now the savings
do not offset the cost and complications.  For example, the fix in
commit 5e427ec2 ("x86: Fix bit corruption at CPU resume time")
is a good example of the nasty type of bugs that can be created
with improper use of the various __init prefixes.

After a discussion on LKML[1] it was decided that cpuinit should go
the way of devinit and be phased out.  Once all the users are gone,
we can then finally remove the macros themselves from linux/init.h.

Note that some harmless section mismatch warnings may result, since
notify_cpu_starting() and cpu_up() are arch independent (kernel/cpu.c)
are flagged as __cpuinit  -- so if we remove the __cpuinit from
arch specific callers, we will also get section mismatch warnings.
As an intermediate step, we intend to turn the linux/init.h cpuinit
content into no-ops as early as possible, since that will get rid
of these warnings.  In any case, they are temporary and harmless.

This removes all the arch/arm64 uses of the __cpuinit macros from
all C files.  Currently arm64 does not have any __CPUINIT used in
assembly files.

[1] https://lkml.org/lkml/2013/5/20/589

Cc: Catalin Marinas <catalin.marinas@arm.com>
Cc: Will Deacon <will.deacon@arm.com>
Acked-by: NCatalin Marinas <catalin.marinas@arm.com>
Signed-off-by: NPaul Gortmaker <paul.gortmaker@windriver.com>

Finally plug KVM/arm64 into the config system, making it possible
to enable KVM support on AArch64 CPUs.
Signed-off-by: NMarc Zyngier <marc.zyngier@arm.com>
Signed-off-by: NPaolo Bonzini <pbonzini@redhat.com>

With this change, we no longer lose the innermost entry in the user-mode
part of the call chain.  See also the x86 port, which includes the ip,
and the corresponding change in arch/arm.
Signed-off-by: NJed Davis <jld@mozilla.com>
Acked-by: NIngo Molnar <mingo@kernel.org>
Acked-by: NWill Deacon <will.deacon@arm.com>
Cc: stable@vger.kernel.org
Signed-off-by: NCatalin Marinas <catalin.marinas@arm.com>

The HYP mode world switch in all its glory.

Implements save/restore of host/guest registers, EL2 trapping,
IPA resolution, and additional services (tlb invalidation).
Reviewed-by: NChristopher Covington <cov@codeaurora.org>
Signed-off-by: NMarc Zyngier <marc.zyngier@arm.com>

The software breakpoint handlers are hooked in directly from ptrace,
which makes it difficult to add additional handlers for things like
kprobes and kgdb.

This patch moves the handling code into debug-monitors.c, where we can
dispatch to different debug subsystems more easily.
Signed-off-by: NWill Deacon <will.deacon@arm.com>
Signed-off-by: NCatalin Marinas <catalin.marinas@arm.com>

As is done for other architectures, sort the exception table at
build-time rather than during boot.

Since sortextable appears to be a standalone C program relying on the
host elf.h to provide EM_AARCH64, I've had to add a conditional check in
order to allow cross-compilation on machines that aren't running a
bleeding-edge libc-dev.
Signed-off-by: NWill Deacon <will.deacon@arm.com>
Signed-off-by: NCatalin Marinas <catalin.marinas@arm.com>

Under arm64, we will calibrate the delay loop statically using a known
timer frequency, so delete read_current_timer(), or it will cause
compiling issue with allmodconfig.

The related error:
  ERROR: "read_current_timer" [lib/rbtree_test.ko] undefined!
  ERROR: "read_current_timer" [lib/interval_tree_test.ko] undefined!
  ERROR: "read_current_timer" [fs/ext4/ext4.ko] undefined!
  ERROR: "read_current_timer" [crypto/tcrypt.ko] undefined!
Signed-off-by: NChen Gang <gang.chen@asianux.com>
Acked-by: NMarc Zyngier <marc.zyngier@arm.com>
Signed-off-by: NCatalin Marinas <catalin.marinas@arm.com>

Add the necessary infrastructure for identity-mapped HYP page
tables. Idmap-ed code must be in the ".hyp.idmap.text" linker
section.

The rest of the HYP ends up in ".hyp.text".
Reviewed-by: NCatalin Marinas <catalin.marinas@arm.com>
Signed-off-by: NMarc Zyngier <marc.zyngier@arm.com>

Rather than completely killing the kernel if we receive an esr value we
can't deal with in the el0 handlers, send the process a SIGILL and log
the esr value in the hope that we can debug it. If we receive a bad esr
from el1, we'll die() as before.
Signed-off-by: NMark Rutland <mark.rutland@arm.com>
Signed-off-by: NCatalin Marinas <catalin.marinas@arm.com>
Cc: stable@vger.kernel.org

Currently, if a compat process reads or writes from/to a disabled
cp15/cp14 register, the trap is not handled by the el0_sync_compat
handler, and the kernel will head to bad_mode, where it will die(), and
oops(). For 64 bit processes, disabled system register accesses are
currently treated as unhandled instructions.

This patch modifies entry.S to treat these unhandled traps as undefined
instructions, sending a SIGILL to userspace. This gives processes a
chance to handle this and stop using inaccessible registers, and
prevents further issues in the kernel as a result of the die().
Reported-by: NJohannes Jensen <Johannes.Jensen@arm.com>
Signed-off-by: NMark Rutland <mark.rutland@arm.com>
Signed-off-by: NCatalin Marinas <catalin.marinas@arm.com>

Currently user faults (page, undefined instruction) are always reported
even though the user may have a signal handler for them. This patch adds
unhandled_signal() check together with printk_ratelimit() for these
cases.
Signed-off-by: NCatalin Marinas <catalin.marinas@arm.com>

Need 'EXPORT_SYMBOL(clear_page)' if building with allmodconfig.

The related errors:
  ERROR: "clear_page" [fs/ocfs2/dlm/ocfs2_dlm.ko] undefined!
  ERROR: "clear_page" [fs/ntfs/ntfs.ko] undefined!
  ERROR: "clear_page" [fs/gfs2/gfs2.ko] undefined!
  ERROR: "clear_page" [fs/fuse/fuse.ko] undefined!
  ERROR: "clear_page" [fs/ext3/ext3.ko] undefined!
  ERROR: "clear_page" [fs/ext2/ext2.ko] undefined!
  ERROR: "clear_page" [fs/exofs/libore.ko] undefined!
  ERROR: "clear_page" [fs/exofs/exofs.ko] undefined!
  ERROR: "clear_page" [drivers/block/brd.ko] undefined!
Signed-off-by: NChen Gang <gang.chen@asianux.com>
Signed-off-by: NCatalin Marinas <catalin.marinas@arm.com>

The of_platform_populate() is currently invoked at device_initcall()
level. There are however drivers that use platform_driver_probe()
directly and they need the devices to be populated. This patch makes the
of_platform_populate() and arch_initcall().
Signed-off-by: NCatalin Marinas <catalin.marinas@arm.com>
Reported-by: NBenoit Lecardonnel <Benoit.Lecardonnel@synopsys.com>
Tested-by: NBenoit Lecardonnel <Benoit.Lecardonnel@synopsys.com>

During boot, we take the debug OS lock before interrupts are enabled.
This is required to prevent clearing of PSTATE.D on the interrupt entry
path, which could result in spurious debug exceptions before we've got
round to resetting things like the hardware breakpoints registers to a
sane state.

A problem with this approach is that taking the OS lock prevents an
external JTAG debugger from debugging the system, which is especially
irritating during boot, where JTAG debugging can be most useful.

This patch clears mdscr_el1 rather than taking the lock, clearing the
MDE and KDE bits and preventing self-hosted hardware debug exceptions
from occurring.
Tested-by: NMark Rutland <mark.rutland@arm.com>
Signed-off-by: NWill Deacon <will.deacon@arm.com>
Signed-off-by: NCatalin Marinas <catalin.marinas@arm.com>
Cc: stable@vger.kernel.org

When compiling with allmodconfig. early_console is already defined as an
extern global pointer. Need let it point to the object which we intend
to (like arm32 done).
Signed-off-by: NChen Gang <gang.chen@asianux.com>
Acked-by: NArnd Bergmann <arnd@arndb.de>
Signed-off-by: NCatalin Marinas <catalin.marinas@arm.com>

Signed-off-by: NAl Viro <viro@zeniv.linux.org.uk>

This patch adds the arm_pm_poweroff definition expected by the
vexpress-poweroff.c driver and enables the latter for arm64.
Signed-off-by: NCatalin Marinas <catalin.marinas@arm.com>
Acked-by: NPawel Moll <pawel.moll@arm.com>