We write idmap_t0sz with SCTLR_EL1.{C,M} clear, but we only have the
guarnatee that the kernel Image is clean, not invalid in the caches, and
therefore we might read a stale value once the MMU is enabled.

This patch ensures we invalidate the corresponding cacheline after the
write as we do for all other data written before we set SCTLR_EL1.{C.M},
guaranteeing that the value will be visible later. We rely on the DSBs
in __create_page_tables to complete the maintenance.
Signed-off-by: NMark Rutland <mark.rutland@arm.com>
CC: Catalin Marinas <catalin.marinas@arm.com>
Cc: Will Deacon <will.deacon@arm.com>
Signed-off-by: NWill Deacon <will.deacon@arm.com>

Historically, the PMU devicetree bindings have expected SPIs to be
listed in order of *logical* CPU number. This is problematic for
bootloaders, especially when the boot CPU (logical ID 0) isn't listed
first in the devicetree.

This patch adds a new optional property, interrupt-affinity, to the
PMU node which allows the interrupt affinity to be described using
a list of phandled to CPU nodes, with each entry in the list
corresponding to the SPI at the same index in the interrupts property.

Cc: Mark Rutland <mark.rutland@arm.com>
Signed-off-by: NWill Deacon <will.deacon@arm.com>

After writing the page tables, we use __inval_cache_range to invalidate
any stale cache entries. Strongly Ordered memory accesses are not
ordered w.r.t. cache maintenance instructions, and hence explicit memory
barriers are required to provide this ordering. However,
__inval_cache_range was written to be used on Normal Cacheable memory
once the MMU and caches are on, and does not have any barriers prior to
the DC instructions.

This patch adds a DMB between the page tables being written and the
corresponding cachelines being invalidated, ensuring that the
invalidation makes the new data visible to subsequent cacheable
accesses. A barrier is not required before the prior invalidate as we do
not access the page table memory area prior to this, and earlier
barriers in preserve_boot_args and set_cpu_boot_mode_flag ensures
ordering w.r.t. any stores performed prior to entering Linux.
Signed-off-by: NMark Rutland <mark.rutland@arm.com>
Cc: Catalin Marinas <catalin.marinas@arm.com>
Cc: Will Deacon <will.deacon@arm.com>
Fixes: c218bca7 ("arm64: Relax the kernel cache requirements for boot")
Signed-off-by: NWill Deacon <will.deacon@arm.com>

This patch modifies the HYP init code so it can deal with system
RAM residing at an offset which exceeds the reach of VA_BITS.

Like for EL1, this involves configuring an additional level of
translation for the ID map. However, in case of EL2, this implies
that all translations use the extra level, as we cannot seamlessly
switch between translation tables with different numbers of
translation levels.

So add an extra translation table at the root level. Since the
ID map and the runtime HYP map are guaranteed not to overlap, they
can share this root level, and we can essentially merge these two
tables into one.
Tested-by: NMarc Zyngier <marc.zyngier@arm.com>
Reviewed-by: NMarc Zyngier <marc.zyngier@arm.com>
Signed-off-by: NArd Biesheuvel <ard.biesheuvel@linaro.org>
Signed-off-by: NWill Deacon <will.deacon@arm.com>

The page size and the number of translation levels, and hence the supported
virtual address range, are build-time configurables on arm64 whose optimal
values are use case dependent. However, in the current implementation, if
the system's RAM is located at a very high offset, the virtual address range
needs to reflect that merely because the identity mapping, which is only used
to enable or disable the MMU, requires the extended virtual range to map the
physical memory at an equal virtual offset.

This patch relaxes that requirement, by increasing the number of translation
levels for the identity mapping only, and only when actually needed, i.e.,
when system RAM's offset is found to be out of reach at runtime.
Tested-by: NLaura Abbott <lauraa@codeaurora.org>
Reviewed-by: NCatalin Marinas <catalin.marinas@arm.com>
Tested-by: NMarc Zyngier <marc.zyngier@arm.com>
Signed-off-by: NArd Biesheuvel <ard.biesheuvel@linaro.org>
Signed-off-by: NWill Deacon <will.deacon@arm.com>

cpu_get_pgd isn't used anywhere and is Probably Not What You Want.
Remove it before anybody decides to use it.
Signed-off-by: NWill Deacon <will.deacon@arm.com>

According to the arm64 boot protocol, registers x1 to x3 should be
zero upon kernel entry, and non-zero values are reserved for future
use. This future use is going to be problematic if we never enforce
the current rules, so start enforcing them now, by emitting a warning
if non-zero values are detected.
Acked-by: NMark Rutland <mark.rutland@arm.com>
Signed-off-by: NArd Biesheuvel <ard.biesheuvel@linaro.org>
Signed-off-by: NWill Deacon <will.deacon@arm.com>

This removes the function __calc_phys_offset and all open coded
virtual to physical address translations using the offset kept
in x28.

Instead, just use absolute or PC-relative symbol references as
appropriate when referring to virtual or physical addresses,
respectively.
Tested-by: NMark Rutland <mark.rutland@arm.com>
Reviewed-by: NMark Rutland <mark.rutland@arm.com>
Signed-off-by: NArd Biesheuvel <ard.biesheuvel@linaro.org>
Signed-off-by: NWill Deacon <will.deacon@arm.com>

Enabling of the MMU is split into two functions, with an align and
a branch in the middle. On arm64, the entire kernel Image is ID mapped
so this is really not necessary, and we can just merge it into a
single function.

Also replaces an open coded adrp/add reference to __enable_mmu pair
with adr_l.
Tested-by: NMark Rutland <mark.rutland@arm.com>
Reviewed-by: NMark Rutland <mark.rutland@arm.com>
Signed-off-by: NArd Biesheuvel <ard.biesheuvel@linaro.org>
Signed-off-by: NWill Deacon <will.deacon@arm.com>

Replace the confusing virtual/physical address arithmetic with a simple
PC-relative reference.
Tested-by: NMark Rutland <mark.rutland@arm.com>
Reviewed-by: NMark Rutland <mark.rutland@arm.com>
Signed-off-by: NArd Biesheuvel <ard.biesheuvel@linaro.org>
Signed-off-by: NWill Deacon <will.deacon@arm.com>

This removes the confusing __switch_data object from head.S,
and replaces it with standard PC-relative references to the
various symbols it encapsulates.
Reviewed-by: NCatalin Marinas <catalin.marinas@arm.com>
Tested-by: NMark Rutland <mark.rutland@arm.com>
Reviewed-by: NMark Rutland <mark.rutland@arm.com>
Signed-off-by: NArd Biesheuvel <ard.biesheuvel@linaro.org>
Signed-off-by: NWill Deacon <will.deacon@arm.com>

The global processor_id is assigned the MIDR_EL1 value of the boot
CPU in the early init code, but is never referenced afterwards.

As the relevance of the MIDR_EL1 value of the boot CPU is debatable
anyway, especially under big.LITTLE, let's remove it before anyone
starts using it.
Tested-by: NMark Rutland <mark.rutland@arm.com>
Reviewed-by: NMark Rutland <mark.rutland@arm.com>
Reviewed-by: NCatalin Marinas <catalin.marinas@arm.com>
Signed-off-by: NArd Biesheuvel <ard.biesheuvel@linaro.org>
Signed-off-by: NWill Deacon <will.deacon@arm.com>

The adrp instruction is mostly used in combination with either
an add, a ldr or a str instruction with the low bits of the
referenced symbol in the 12-bit immediate of the followup
instruction.

Introduce the macros adr_l, ldr_l and str_l that encapsulate
these common patterns.
Tested-by: NMark Rutland <mark.rutland@arm.com>
Reviewed-by: NMark Rutland <mark.rutland@arm.com>
Signed-off-by: NArd Biesheuvel <ard.biesheuvel@linaro.org>
Signed-off-by: NWill Deacon <will.deacon@arm.com>

struct cpu_table is an artifact left from the (very) early days of
the arm64 port, and its only real use is to allow the most beautiful
"AArch64 Processor" string to be displayed at boot time.

Really? Yes, really.

Let's get rid of it. In order to avoid another BogoMips-gate, the
aforementioned string is preserved.
Acked-by: NMark Rutland <mark.rutland@arm.com>
Acked-by: NCatalin Marinas <catalin.marinas@arm.com>
Acked-by: NArd Biesheuvel <ard.biesheuvel@linaro.org>
Signed-off-by: NMarc Zyngier <marc.zyngier@arm.com>
Signed-off-by: NArd Biesheuvel <ard.biesheuvel@linaro.org>
Signed-off-by: NWill Deacon <will.deacon@arm.com>

Raise the maximum CPU limit to 4096 in preparation for upcoming
platforms with large core counts.
Acked-by: NArnd Bergmann <arnd@arndb.de>
Signed-off-by: NGanapatrao Kulkarni <gkulkarni@caviumnetworks.com>
Signed-off-by: NWill Deacon <will.deacon@arm.com>

The perf core implicitly rejects events spanning multiple HW PMUs, as in
these cases the event->ctx will differ. However this validation is
performed after pmu::event_init() is called in perf_init_event(), and
thus pmu::event_init() may be called with a group leader from a
different HW PMU.

The ARM64 PMU driver does not take this fact into account, and when
validating groups assumes that it can call to_arm_pmu(event->pmu) for
any HW event. When the event in question is from another HW PMU this is
wrong, and results in dereferencing garbage.

This patch updates the ARM64 PMU driver to first test for and reject
events from other PMUs, moving the to_arm_pmu and related logic after
this test. Fixes a crash triggered by perf_fuzzer on Linux-4.0-rc2, with
a CCI PMU present:

Bad mode in Synchronous Abort handler detected, code 0x86000006 -- IABT (current EL)
CPU: 0 PID: 1371 Comm: perf_fuzzer Not tainted 3.19.0+ #249
Hardware name: V2F-1XV7 Cortex-A53x2 SMM (DT)
task: ffffffc07c73a280 ti: ffffffc07b0a0000 task.ti: ffffffc07b0a0000
PC is at 0x0
LR is at validate_event+0x90/0xa8
pc : [<0000000000000000>] lr : [<ffffffc000090228>] pstate: 00000145
sp : ffffffc07b0a3ba0

[<          (null)>]           (null)
[<ffffffc0000907d8>] armpmu_event_init+0x174/0x3cc
[<ffffffc00015d870>] perf_try_init_event+0x34/0x70
[<ffffffc000164094>] perf_init_event+0xe0/0x10c
[<ffffffc000164348>] perf_event_alloc+0x288/0x358
[<ffffffc000164c5c>] SyS_perf_event_open+0x464/0x98c
Code: bad PC value

Also cleans up the code to use the arm_pmu only when we know
that we are dealing with an arm pmu event.

Cc: Will Deacon <will.deacon@arm.com>
Acked-by: NMark Rutland <mark.rutland@arm.com>
Acked-by: NPeter Ziljstra (Intel) <peterz@infradead.org>
Signed-off-by: NSuzuki K. Poulose <suzuki.poulose@arm.com>
Signed-off-by: NWill Deacon <will.deacon@arm.com>

The HYP init bounce page is a runtime construct that ensures that the
HYP init code does not cross a page boundary. However, this is something
we can do perfectly well at build time, by aligning the code appropriately.

For arm64, we just align to 4 KB, and enforce that the code size is less
than 4 KB, regardless of the chosen page size.

For ARM, the whole code is less than 256 bytes, so we tweak the linker
script to align at a power of 2 upper bound of the code size

Note that this also fixes a benign off-by-one error in the original bounce
page code, where a bounce page would be allocated unnecessarily if the code
was exactly 1 page in size.

On ARM, it also fixes an issue with very large kernels reported by Arnd
Bergmann, where stub sections with linker emitted veneers could erroneously
trigger the size/alignment ASSERT() in the linker script.
Tested-by: NMarc Zyngier <marc.zyngier@arm.com>
Reviewed-by: NMarc Zyngier <marc.zyngier@arm.com>
Signed-off-by: NArd Biesheuvel <ard.biesheuvel@linaro.org>
Signed-off-by: NWill Deacon <will.deacon@arm.com>

This changes the AES core transform implementations to issue aese/aesmc
(and aesd/aesimc) in pairs. This enables a micro-architectural optimization
in recent Cortex-A5x cores that improves performance by 50-90%.

Measured performance in cycles per byte (Cortex-A57):

                CBC enc         CBC dec         CTR
  before        3.64            1.34            1.32
  after         1.95            0.85            0.93

Note that this results in a ~5% performance decrease for older cores.
Signed-off-by: NArd Biesheuvel <ard.biesheuvel@linaro.org>
Signed-off-by: NWill Deacon <will.deacon@arm.com>

Fixmap indices are in the interval (FIX_HOLE, __end_of_fixed_addresses),
but in __set_fixmap we only check idx <= __end_of_fixed_addresses, and
therefore indices <= FIX_HOLE are erroneously accepted. If called with
such an idx, __set_fixmap may corrupt page tables outside of the fixmap
region.

This patch ensures that we validate the idx against both endpoints of
the interval.

Cc: Catalin Marinas <catalin.marinas@arm.com>
Cc: Kees Cook <keescook@chromium.org>
Acked-by: NArd Biesheuvel <ard.biesheuvel@linaro.org>
Acked-by: NLaura Abbott <lauraa@codeaurora.org>
Signed-off-by: NMark Rutland <mark.rutland@arm.com>
Signed-off-by: NWill Deacon <will.deacon@arm.com>

The FIX_TEXT_POKE0 is currently at the end of the temporary fixmap
slots, despite the fact that it can be used at any point during runtime
(e.g. for poking the text of loaded modules), and thus should be a
permanent fixmap slot (as is the case on arm and x86).

This patch moves FIX_TEXT_POKE0 into the set of permanent fixmap slots.

Cc: Catalin Marinas <catalin.marinas@arm.com>
Cc: Kees Cook <keescook@chromium.org>
Acked-by: NArd Biesheuvel <ard.biesheuvel@linaro.org>
Acked-by: NLaura Abbott <lauraa@codeaurora.org>
Signed-off-by: NMark Rutland <mark.rutland@arm.com>
Signed-off-by: NWill Deacon <will.deacon@arm.com>

This effectively unexports set_memory_ro and set_memory_rw functions from
commit 11d91a77 ("arm64: Add CONFIG_DEBUG_SET_MODULE_RONX support").

No module user of those is in mainline kernel and we explicitly do not want
modules to use these functions, as they i.e. RO-protect eBPF (interpreted and
JIT'ed) images from malicious modifications/bugs.

Outside of eBPF scope, I believe also other set_memory_* functions should
be unexported on arm64 due to non-existant mainline module user. Laura
mentioned that they have some uses for modules doing set_memory_*, but
none that are in mainline and it's unclear if they would ever get there.
Signed-off-by: NDaniel Borkmann <daniel@iogearbox.net>
Acked-by: NAlexei Starovoitov <ast@plumgrid.com>
Acked-by: NLaura Abbott <lauraa@codeaurora.org>
Signed-off-by: NWill Deacon <will.deacon@arm.com>

With binutils 2.25 the default alignment for 32bit arm sections changed to
have everything 64k aligned. Armv7 binaries built with this binutils version
run successfully on an arm64 system.

Since effectively there is now the chance to run armv7 code on arm64 even
with 64k page size, it doesn't make sense to block people from enabling
CONFIG_COMPAT on those configurations.
Signed-off-by: NAlexander Graf <agraf@suse.de>
Signed-off-by: NWill Deacon <will.deacon@arm.com>

The arm mmap2 syscall takes the offset in units of 4K, thus with 64K pages
the offset needs to be scaled to units of pages.
Signed-off-by: NAndreas Schwab <schwab@suse.de>
Signed-off-by: NAlexander Graf <agraf@suse.de>
[will: removed redundant lr parameter, localised PAGE_SHIFT #if check]
Signed-off-by: NWill Deacon <will.deacon@arm.com>

Commit f4f75ad5 ("efi: efistub: Convert into static library")
introduced a static library for EFI stub, libstub.

The EFI libstub directory is referenced by the kernel build system via
a obj subdirectory rule in:
drivers/firmware/efi/Makefile

Unfortunately, arm64 also references the EFI libstub via:
libs-$(CONFIG_EFI_STUB) += drivers/firmware/efi/libstub/

If we're unlucky, the kernel build system can enter libstub via two
simultaneous threads resulting in build failures such as:

fixdep: error opening depfile: drivers/firmware/efi/libstub/.efi-stub-helper.o.d: No such file or directory
scripts/Makefile.build:257: recipe for target 'drivers/firmware/efi/libstub/efi-stub-helper.o' failed
make[1]: *** [drivers/firmware/efi/libstub/efi-stub-helper.o] Error 2
Makefile:939: recipe for target 'drivers/firmware/efi/libstub' failed
make: *** [drivers/firmware/efi/libstub] Error 2
make: *** Waiting for unfinished jobs....

This patch adjusts the arm64 Makefile to reference the compiled library
explicitly (as is currently done in x86), rather than the directory.

Fixes: f4f75ad5 efi: efistub: Convert into static library
Signed-off-by: NSteve Capper <steve.capper@linaro.org>
Signed-off-by: NWill Deacon <will.deacon@arm.com>

We currently don't log the boot mode for arm64 as we do for arm, and
without KVM the user is provided with no indication as to which mode(s)
CPUs were booted in, which can seriously hinder debugging in some cases.

Add logging to the boot path once all CPUs are up. Where CPUs are
mismatched in violation of the boot protocol, WARN and set a taint (as
we do for CPU other CPU feature mismatches) given that the
firmware/bootloader is buggy and should be fixed.

Cc: Catalin Marinas <catalin.marinas@arm.com>
Cc: Marc Zyngier <marc.zyngier@arm.com>
Tested-by: NArd Biesheuvel <ard.biesheuvel@linaro.org>
Reviewed-by: NArd Biesheuvel <ard.biesheuvel@linaro.org>
Signed-off-by: NMark Rutland <mark.rutland@arm.com>
Signed-off-by: NWill Deacon <will.deacon@arm.com>

Commit 828e9834 ("arm64: head: create a new function for setting
the boot_cpu_mode flag") added BOOT_CPU_MODE_EL1, a nonzero value
replacing uses of zero. However it failed to update __boot_cpu_mode
appropriately.

A CPU booted at EL2 writes BOOT_CPU_MODE_EL2 to __boot_cpu_mode[0], and
a CPU booted at EL1 writes BOOT_CPU_MODE_EL1 to __boot_cpu_mode[1].
Later is_hyp_mode_mismatched() determines there to be a mismatch if
__boot_cpu_mode[0] != __boot_cpu_mode[1].

If all CPUs are booted at EL1, __boot_cpu_mode[0] will be set to
BOOT_CPU_MODE_EL1, but __boot_cpu_mode[1] will retain its initial value
of zero, and is_hyp_mode_mismatched will erroneously determine that the
boot modes are mismatched. This hasn't been a problem so far, but later
patches which will make use of is_hyp_mode_mismatched() expect it to
work correctly.

This patch initialises __boot_cpu_mode[1] to BOOT_CPU_MODE_EL1, fixing
the erroneous mismatch detection when all CPUs are booted at EL1.

Cc: Catalin Marinas <catalin.marinas@arm.com>
Cc: Marc Zyngier <marc.zyngier@arm.com>
Tested-by: NArd Biesheuvel <ard.biesheuvel@linaro.org>
Reviewed-by: NArd Biesheuvel <ard.biesheuvel@linaro.org>
Signed-off-by: NMark Rutland <mark.rutland@arm.com>
Signed-off-by: NWill Deacon <will.deacon@arm.com>

Currently we only perform alternative patching for kernels built with
CONFIG_SMP, as we call apply_alternatives_all() in smp.c, which is only
built for CONFIG_SMP. Thus !SMP kernels may not have necessary
alternatives patched in.

This patch ensures that we call apply_alternatives_all() once all CPUs
are booted, even for !SMP kernels, by having the smp_init_cpus() stub
call this for !SMP kernels via up_late_init. A new wrapper,
do_post_cpus_up_work, is added so we can hook other calls here later
(e.g. boot mode logging).

Cc: Andre Przywara <andre.przywara@arm.com>
Cc: Catalin Marinas <catalin.marinas@arm.com>
Fixes: e039ee4e ("arm64: add alternative runtime patching")
Tested-by: NArd Biesheuvel <ard.biesheuvel@linaro.org>
Reviewed-by: NArd Biesheuvel <ard.biesheuvel@linaro.org>
Signed-off-by: NMark Rutland <mark.rutland@arm.com>
Signed-off-by: NWill Deacon <will.deacon@arm.com>

ARM64 has the yield nop hint which has the intended semantics of
cpu_relax. Implement.

The immediate application is ARM CPU emulators. An emulator can take
advantage of the yield hint to de-prioritise an emulated CPU in favor
of other emulation tasks. QEMU A64 SMP emulation has yield awareness,
and sees a significant boot time performance increase with this change.
Signed-off-by: NPeter Crosthwaite <peter.crosthwaite@xilinx.com>
Acked-by: NWill Deacon <will.deacon@arm.com>
Signed-off-by: NWill Deacon <will.deacon@arm.com>

Another one for the big head.S spring cleaning: the label should
be after the .align or it may point to the padding.
Signed-off-by: NArd Biesheuvel <ard.biesheuvel@linaro.org>
Signed-off-by: NCatalin Marinas <catalin.marinas@arm.com>

If UEFI Runtime Services are available, they are preferred over direct
PSCI calls or other methods to reset the system.

For the reset case, we need to hook into machine_restart(), as the
arm_pm_restart function pointer may be overwritten by modules.
Tested-by: NMark Rutland <mark.rutland@arm.com>
Reviewed-by: NMark Rutland <mark.rutland@arm.com>
Reviewed-by: NMatt Fleming <matt.fleming@intel.com>
Signed-off-by: NArd Biesheuvel <ard.biesheuvel@linaro.org>
Signed-off-by: NCatalin Marinas <catalin.marinas@arm.com>

The ARM architecture allows the caching of intermediate page table
levels and page table freeing requires a sequence like:

	pmd_clear()
	TLB invalidation
	pte page freeing

With commit 5e5f6dc1 (arm64: mm: enable HAVE_RCU_TABLE_FREE logic),
the page table freeing batching was moved from tlb_remove_page() to
tlb_remove_table(). The former takes care of TLB invalidation as this is
also shared with pte clearing and page cache page freeing. The latter,
however, does not invalidate the TLBs for intermediate page table levels
as it probably relies on the architecture code to do it if required.
When the mm->mm_users < 2, tlb_remove_table() does not do any batching
and page table pages are freed before tlb_finish_mmu() which performs
the actual TLB invalidation.

This patch introduces __tlb_flush_pgtable() for arm64 and calls it from
the {pte,pmd,pud}_free_tlb() directly without relying on deferred page
table freeing.

Fixes: 5e5f6dc1 arm64: mm: enable HAVE_RCU_TABLE_FREE logic
Reported-by: NJon Masters <jcm@redhat.com>
Tested-by: NJon Masters <jcm@redhat.com>
Tested-by: NSteve Capper <steve.capper@linaro.org>
Signed-off-by: NCatalin Marinas <catalin.marinas@arm.com>

The set_memory_* functions currently only support module
addresses. The addresses are validated using is_module_addr.
That function is special though and relies on internal state
in the module subsystem to work properly. At the time of
module initialization and calling set_memory_*, it's too early
for is_module_addr to work properly so it always returns
false. Rather than be subject to the whims of the module state,
just bounds check against the module virtual address range.
Signed-off-by: NLaura Abbott <lauraa@codeaurora.org>
Signed-off-by: NCatalin Marinas <catalin.marinas@arm.com>

This patch fixes the backward compatibility of the older driver with the
newer firmware by making the binding unique so that the older driver won't
recognize the non-supported interfaces.

The new bindings are in sync with the newer firmware.
Signed-off-by: NIyappan Subramanian <isubramanian@apm.com>
Signed-off-by: NKeyur Chudgar <kchudgar@apm.com>
Tested-by: NMark Langsdorf <mlangsdo@redhat.com>
Signed-off-by: NDavid S. Miller <davem@davemloft.net>

ARM64 CPUidle driver requires the cpu_do_idle function so that it can
be used to enter the shallowest idle state, and it is declared in
asm/proc-fns.h.

The current ARM64 CPUidle driver does not include asm/proc-fns.h
explicitly and it has so far relied on implicit inclusion from other
header files.

Owing to some header dependencies reshuffling this currently triggers
build failures when CONFIG_ARM64_64K_PAGES=y:

drivers/cpuidle/cpuidle-arm64.c: In function "arm64_enter_idle_state"
drivers/cpuidle/cpuidle-arm64.c:42:3: error: implicit declaration of
function "cpu_do_idle" [-Werror=implicit-function-declaration]
   cpu_do_idle();
   ^

This patch adds the explicit inclusion of the asm/proc-fns.h header file
in the arm64 asm/cpuidle.h header file, so that the build breakage is fixed
and the required header inclusion is added to the appropriate arch back-end
CPUidle header, already included by the CPUidle arm64 driver, where
CPUidle arch related function declarations belong.
Reported-by: NLaura Abbott <lauraa@codeaurora.org>
Signed-off-by: NLorenzo Pieralisi <lorenzo.pieralisi@arm.com>
Acked-by: NWill Deacon <will.deacon@arm.com>
Tested-by: NMark Rutland <mark.rutland@arm.com>
Signed-off-by: NCatalin Marinas <catalin.marinas@arm.com>

The native (64-bit) sigval_t union contains sival_int (32-bit) and
sival_ptr (64-bit). When a compat application invokes a syscall that
takes a sigval_t value (as part of a larger structure, e.g.
compat_sys_mq_notify, compat_sys_timer_create), the compat_sigval_t
union is converted to the native sigval_t with sival_int overlapping
with either the least or the most significant half of sival_ptr,
depending on endianness. When the corresponding signal is delivered to a
compat application, on big endian the current (compat_uptr_t)sival_ptr
cast always returns 0 since sival_int corresponds to the top part of
sival_ptr. This patch fixes copy_siginfo_to_user32() so that sival_int
is copied to the compat_siginfo_t structure.

Cc: <stable@vger.kernel.org>
Reported-by: NBamvor Jian Zhang <bamvor.zhangjian@huawei.com>
Tested-by: NBamvor Jian Zhang <bamvor.zhangjian@huawei.com>
Signed-off-by: NCatalin Marinas <catalin.marinas@arm.com>

With commit 3690951f (arm64: Use swiotlb late initialisation), the
swiotlb buffer size is limited to MAX_ORDER_NR_PAGES. However, there are
platforms with 32-bit only devices that require bounce buffering via
swiotlb. This patch changes the swiotlb initialisation to an early 64MB
memblock allocation. In order to get the swiotlb buffer correctly
allocated (via memblock_virt_alloc_low_nopanic), this patch also defines
ARCH_LOW_ADDRESS_LIMIT to the maximum physical address capable of 32-bit
DMA.
Reported-by: NKefeng Wang <wangkefeng.wang@huawei.com>
Tested-by: NKefeng Wang <wangkefeng.wang@huawei.com>
Signed-off-by: NCatalin Marinas <catalin.marinas@arm.com>

Patch 2f896d58 ("arm64: use fixmap for text patching") changed
the way we patch the kernel text, using a fixmap when the kernel or
modules are flagged as read only.

Unfortunately, a flaw in the logic makes it fall over when patching
modules without CONFIG_DEBUG_SET_MODULE_RONX enabled:

[...]
[   32.032636] Call trace:
[   32.032716] [<fffffe00003da0dc>] __copy_to_user+0x2c/0x60
[   32.032837] [<fffffe0000099f08>] __aarch64_insn_write+0x94/0xf8
[   32.033027] [<fffffe000009a0a0>] aarch64_insn_patch_text_nosync+0x18/0x58
[   32.033200] [<fffffe000009c3ec>] ftrace_modify_code+0x58/0x84
[   32.033363] [<fffffe000009c4e4>] ftrace_make_nop+0x3c/0x58
[   32.033532] [<fffffe0000164420>] ftrace_process_locs+0x3d0/0x5c8
[   32.033709] [<fffffe00001661cc>] ftrace_module_init+0x28/0x34
[   32.033882] [<fffffe0000135148>] load_module+0xbb8/0xfc4
[   32.034044] [<fffffe0000135714>] SyS_finit_module+0x94/0xc4
[...]

This is triggered by the use of virt_to_page() on a module address,
which ends to pointing to Nowhereland if you're lucky, or corrupt
your precious data if not.

This patch fixes the logic by mimicking what is done on arm:
- If we're patching a module and CONFIG_DEBUG_SET_MODULE_RONX is set,
  use vmalloc_to_page().
- If we're patching the kernel and CONFIG_DEBUG_RODATA is set,
  use virt_to_page().
- Otherwise, use the provided address, as we can write to it directly.

Tested on 4.0-rc1 as a KVM guest.
Reported-by: NRichard W.M. Jones <rjones@redhat.com>
Reviewed-by: NKees Cook <keescook@chromium.org>
Acked-by: NMark Rutland <mark.rutland@arm.com>
Acked-by: NLaura Abbott <lauraa@codeaurora.org>
Tested-by: NRichard W.M. Jones <rjones@redhat.com>
Cc: Will Deacon <will.deacon@arm.com>
Signed-off-by: NMarc Zyngier <marc.zyngier@arm.com>
Signed-off-by: NCatalin Marinas <catalin.marinas@arm.com>

This patch increases the interleave factor for parallel AES modes
to 4x. This improves performance on Cortex-A57 by ~35%. This is
due to the 3-cycle latency of AES instructions on the A57's
relatively deep pipeline (compared to Cortex-A53 where the AES
instruction latency is only 2 cycles).

At the same time, disable inline expansion of the core AES functions,
as the performance benefit of this feature is negligible.

  Measured on AMD Seattle (using tcrypt.ko mode=500 sec=1):

  Baseline (2x interleave, inline expansion)
  ------------------------------------------
  testing speed of async cbc(aes) (cbc-aes-ce) decryption
  test 4 (128 bit key, 8192 byte blocks): 95545 operations in 1 seconds
  test 14 (256 bit key, 8192 byte blocks): 68496 operations in 1 seconds

  This patch (4x interleave, no inline expansion)
  -----------------------------------------------
  testing speed of async cbc(aes) (cbc-aes-ce) decryption
  test 4 (128 bit key, 8192 byte blocks): 124735 operations in 1 seconds
  test 14 (256 bit key, 8192 byte blocks): 92328 operations in 1 seconds
Signed-off-by: NArd Biesheuvel <ard.biesheuvel@linaro.org>
Signed-off-by: NCatalin Marinas <catalin.marinas@arm.com>

Caught during Trinity testing. The pte_modify does not allow
modification for PTE type bit. This cause the test to hang
the system. It is found that the PTE can't transit from an
inaccessible page (b00) to a valid page (b11) because the mask
does not allow it. This happens when a big block of mmaped
memory is set the PROT_NONE, then the a small piece is broken
off and set to PROT_WRITE | PROT_READ cause a huge page split.
Signed-off-by: NFeng Kan <fkan@apm.com>
Signed-off-by: NCatalin Marinas <catalin.marinas@arm.com>

The functions __cpu_flush_user_tlb_range and __cpu_flush_kern_tlb_range
were removed in commit fa48e6f7 'arm64: mm: Optimise tlb flush logic
where we have >4K granule'. Global variable cpu_tlb was never used in
arm64.

Remove them.
Signed-off-by: NYingjoe Chen <yingjoe.chen@mediatek.com>
Acked-by: NWill Deacon <will.deacon@arm.com>
Signed-off-by: NCatalin Marinas <catalin.marinas@arm.com>