Skip to content

cloud-kernel
cloud-kernel

openanolis / cloud-kernel
大约 1 年 前同步成功

通知 158
Star 36
Fork 7
cloud-kernel
cloud-kernel

体验新版 GitCode，发现更多精彩内容 >>

提交 5167d09f 编写于 作者: L Linus Torvalds
浏览文件
操作

Merge tag 'arm64-upstream' of git://git.kernel.org/pub/scm/linux/kernel/git/arm64/linux 

Pull arm64 updates from Will Deacon:
 "Once again, Catalin's off on holiday and I'm looking after the arm64
  tree.  Please can you pull the following arm64 updates for 3.17?

  Note that this branch also includes the new GICv3 driver (merged via a
  stable tag from Jason's irqchip tree), since there is a fix for older
  binutils on top.

  Changes include:
   - context tracking support (NO_HZ_FULL) which narrowly missed 3.16
   - vDSO layout rework following Andy's work on x86
   - TEXT_OFFSET fuzzing for bootloader testing
   - /proc/cpuinfo tidy-up
   - preliminary work to support 48-bit virtual addresses, but this is
     currently disabled until KVM has been ported to use it (the patches
     do, however, bring some nice clean-up)
   - boot-time CPU sanity checks (especially useful on heterogenous
     systems)
   - support for syscall auditing
   - support for CC_STACKPROTECTOR
   - defconfig updates"

* tag 'arm64-upstream' of git://git.kernel.org/pub/scm/linux/kernel/git/arm64/linux: (55 commits)
  arm64: add newline to I-cache policy string
  Revert "arm64: dmi: Add SMBIOS/DMI support"
  arm64: fpsimd: fix a typo in fpsimd_save_partial_state ENDPROC
  arm64: don't call break hooks for BRK exceptions from EL0
  arm64: defconfig: enable devtmpfs mount option
  arm64: vdso: fix build error when switching from LE to BE
  arm64: defconfig: add virtio support for running as a kvm guest
  arm64: gicv3: Allow GICv3 compilation with older binutils
  arm64: fix soft lockup due to large tlb flush range
  arm64/crypto: fix makefile rule for aes-glue-%.o
  arm64: Do not invoke audit_syscall_* functions if !CONFIG_AUDIT_SYSCALL
  arm64: Fix barriers used for page table modifications
  arm64: Add support for 48-bit VA space with 64KB page configuration
  arm64: asm/pgtable.h pmd/pud definitions clean-up
  arm64: Determine the vmalloc/vmemmap space at build time based on VA_BITS
  arm64: Clean up the initial page table creation in head.S
  arm64: Remove asm/pgtable-*level-types.h files
  arm64: Remove asm/pgtable-*level-hwdef.h files
  arm64: Convert bool ARM64_x_LEVELS to int ARM64_PGTABLE_LEVELS
  arm64: mm: Implement 4 levels of translation tables
  ...
上级 8533ce72 ea171967
展开全部 隐藏空白更改
内联 并排
Showing with 3061 addition and 798 deletion
Documentation/arm64/booting.txt
浏览文件 @ 5167d09f
......@@ -72,27 +72,54 @@ The decompressed kernel image contains a 64-byte header as follows:
u32 code0; /* Executable code */
u32 code1; /* Executable code */
u64 text_offset; /* Image load offset */
u64 res0 = 0; /* reserved */
u64 res1 = 0; /* reserved */
u64 text_offset; /* Image load offset, little endian */
u64 image_size; /* Effective Image size, little endian */
u64 flags; /* kernel flags, little endian */
u64 res2 = 0; /* reserved */
u64 res3 = 0; /* reserved */
u64 res4 = 0; /* reserved */
u32 magic = 0x644d5241; /* Magic number, little endian, "ARM\x64" */
u32 res5 = 0; /* reserved */
u32 res5; /* reserved (used for PE COFF offset) */
Header notes:
- As of v3.17, all fields are little endian unless stated otherwise.
- code0/code1 are responsible for branching to stext.
- when booting through EFI, code0/code1 are initially skipped.
res5 is an offset to the PE header and the PE header has the EFI
entry point (efi_stub_entry). When the stub has done its work, it
entry point (efi_stub_entry). When the stub has done its work, it
jumps to code0 to resume the normal boot process.
The image must be placed at the specified offset (currently 0x80000)
from the start of the system RAM and called there. The start of the
system RAM must be aligned to 2MB.
- Prior to v3.17, the endianness of text_offset was not specified. In
these cases image_size is zero and text_offset is 0x80000 in the
endianness of the kernel. Where image_size is non-zero image_size is
little-endian and must be respected. Where image_size is zero,
text_offset can be assumed to be 0x80000.
- The flags field (introduced in v3.17) is a little-endian 64-bit field
composed as follows:
Bit 0: Kernel endianness. 1 if BE, 0 if LE.
Bits 1-63: Reserved.
- When image_size is zero, a bootloader should attempt to keep as much
memory as possible free for use by the kernel immediately after the
end of the kernel image. The amount of space required will vary
depending on selected features, and is effectively unbound.
The Image must be placed text_offset bytes from a 2MB aligned base
address near the start of usable system RAM and called there. Memory
below that base address is currently unusable by Linux, and therefore it
is strongly recommended that this location is the start of system RAM.
At least image_size bytes from the start of the image must be free for
use by the kernel.
Any memory described to the kernel (even that below the 2MB aligned base
address) which is not marked as reserved from the kernel e.g. with a
memreserve region in the device tree) will be considered as available to
the kernel.
Before jumping into the kernel, the following conditions must be met:
......
Documentation/arm64/memory.txt
浏览文件 @ 5167d09f
......@@ -2,18 +2,18 @@
==============================
Author: Catalin Marinas <catalin.marinas@arm.com>
Date : 20 February 2012
This document describes the virtual memory layout used by the AArch64
Linux kernel. The architecture allows up to 4 levels of translation
tables with a 4KB page size and up to 3 levels with a 64KB page size.
AArch64 Linux uses 3 levels of translation tables with the 4KB page
configuration, allowing 39-bit (512GB) virtual addresses for both user
and kernel. With 64KB pages, only 2 levels of translation tables are
used but the memory layout is the same.
AArch64 Linux uses either 3 levels or 4 levels of translation tables
with the 4KB page configuration, allowing 39-bit (512GB) or 48-bit
(256TB) virtual addresses, respectively, for both user and kernel. With
64KB pages, only 2 levels of translation tables, allowing 42-bit (4TB)
virtual address, are used but the memory layout is the same.
User addresses have bits 63:39 set to 0 while the kernel addresses have
User addresses have bits 63:48 set to 0 while the kernel addresses have
the same bits set to 1. TTBRx selection is given by bit 63 of the
virtual address. The swapper_pg_dir contains only kernel (global)
mappings while the user pgd contains only user (non-global) mappings.
......@@ -21,58 +21,40 @@ The swapper_pgd_dir address is written to TTBR1 and never written to
TTBR0.
AArch64 Linux memory layout with 4KB pages:
AArch64 Linux memory layout with 4KB pages + 3 levels:
Start End Size Use
-----------------------------------------------------------------------
0000000000000000 0000007fffffffff 512GB user
ffffff8000000000 ffffffffffffffff 512GB kernel
ffffff8000000000 ffffffbbfffeffff ~240GB vmalloc
ffffffbbffff0000 ffffffbbffffffff 64KB [guard page]
AArch64 Linux memory layout with 4KB pages + 4 levels:
ffffffbc00000000 ffffffbdffffffff 8GB vmemmap
ffffffbe00000000 ffffffbffbbfffff ~8GB [guard, future vmmemap]
ffffffbffa000000 ffffffbffaffffff 16MB PCI I/O space
ffffffbffb000000 ffffffbffbbfffff 12MB [guard]
ffffffbffbc00000 ffffffbffbdfffff 2MB fixed mappings
ffffffbffbe00000 ffffffbffbffffff 2MB [guard]
ffffffbffc000000 ffffffbfffffffff 64MB modules
ffffffc000000000 ffffffffffffffff 256GB kernel logical memory map
Start End Size Use
-----------------------------------------------------------------------
0000000000000000 0000ffffffffffff 256TB user
ffff000000000000 ffffffffffffffff 256TB kernel
AArch64 Linux memory layout with 64KB pages:
AArch64 Linux memory layout with 64KB pages + 2 levels:
Start End Size Use
-----------------------------------------------------------------------
0000000000000000 000003ffffffffff 4TB user
fffffc0000000000 ffffffffffffffff 4TB kernel
fffffc0000000000 fffffdfbfffeffff ~2TB vmalloc
fffffdfbffff0000 fffffdfbffffffff 64KB [guard page]
AArch64 Linux memory layout with 64KB pages + 3 levels:
fffffdfc00000000 fffffdfdffffffff 8GB vmemmap
fffffdfe00000000 fffffdfffbbfffff ~8GB [guard, future vmmemap]
fffffdfffa000000 fffffdfffaffffff 16MB PCI I/O space
fffffdfffb000000 fffffdfffbbfffff 12MB [guard]
fffffdfffbc00000 fffffdfffbdfffff 2MB fixed mappings
fffffdfffbe00000 fffffdfffbffffff 2MB [guard]
Start End Size Use
-----------------------------------------------------------------------
0000000000000000 0000ffffffffffff 256TB user
ffff000000000000 ffffffffffffffff 256TB kernel
fffffdfffc000000 fffffdffffffffff 64MB modules
fffffe0000000000 ffffffffffffffff 2TB kernel logical memory map
For details of the virtual kernel memory layout please see the kernel
booting log.
Translation table lookup with 4KB pages:
......@@ -86,7 +68,7 @@ Translation table lookup with 4KB pages:
| | | | +-> [20:12] L3 index
| | | +-----------> [29:21] L2 index
| | +---------------------> [38:30] L1 index
| +-------------------------------> [47:39] L0 index (not used)
| +-------------------------------> [47:39] L0 index
+-------------------------------------------------> [63] TTBR0/1
......@@ -99,10 +81,11 @@ Translation table lookup with 64KB pages:
| | | | v
| | | | [15:0] in-page offset
| | | +----------> [28:16] L3 index
| | +--------------------------> [41:29] L2 index (only 38:29 used)
| +-------------------------------> [47:42] L1 index (not used)
| | +--------------------------> [41:29] L2 index
| +-------------------------------> [47:42] L1 index
+-------------------------------------------------> [63] TTBR0/1
When using KVM, the hypervisor maps kernel pages in EL2, at a fixed
offset from the kernel VA (top 24bits of the kernel VA set to zero):
......
arch/arm64/Kconfig
浏览文件 @ 5167d09f
config ARM64
def_bool y
select ARCH_HAS_ATOMIC64_DEC_IF_POSITIVE
select ARCH_HAS_OPP
select ARCH_HAS_TICK_BROADCAST if GENERIC_CLOCKEVENTS_BROADCAST
select ARCH_USE_CMPXCHG_LOCKREF
select ARCH_SUPPORTS_ATOMIC_RMW
......@@ -11,6 +10,8 @@ config ARM64
select ARM_AMBA
select ARM_ARCH_TIMER
select ARM_GIC
select AUDIT_ARCH_COMPAT_GENERIC
select ARM_GIC_V3
select BUILDTIME_EXTABLE_SORT
select CLONE_BACKWARDS
select COMMON_CLK
......@@ -29,10 +30,12 @@ config ARM64
select GENERIC_STRNLEN_USER
select GENERIC_TIME_VSYSCALL
select HARDIRQS_SW_RESEND
select HAVE_ARCH_AUDITSYSCALL
select HAVE_ARCH_JUMP_LABEL
select HAVE_ARCH_KGDB
select HAVE_ARCH_TRACEHOOK
select HAVE_C_RECORDMCOUNT
select HAVE_CC_STACKPROTECTOR
select HAVE_DEBUG_BUGVERBOSE
select HAVE_DEBUG_KMEMLEAK
select HAVE_DMA_API_DEBUG
......@@ -63,6 +66,7 @@ config ARM64
select RTC_LIB
select SPARSE_IRQ
select SYSCTL_EXCEPTION_TRACE
select HAVE_CONTEXT_TRACKING
help
ARM 64-bit (AArch64) Linux support.
......@@ -155,14 +159,63 @@ endmenu
menu "Kernel Features"
choice
prompt "Page size"
default ARM64_4K_PAGES
help
Page size (translation granule) configuration.
config ARM64_4K_PAGES
bool "4KB"
help
This feature enables 4KB pages support.
config ARM64_64K_PAGES
bool "Enable 64KB pages support"
bool "64KB"
help
This feature enables 64KB pages support (4KB by default)
allowing only two levels of page tables and faster TLB
look-up. AArch32 emulation is not available when this feature
is enabled.
endchoice
choice
prompt "Virtual address space size"
default ARM64_VA_BITS_39 if ARM64_4K_PAGES
default ARM64_VA_BITS_42 if ARM64_64K_PAGES
help
Allows choosing one of multiple possible virtual address
space sizes. The level of translation table is determined by
a combination of page size and virtual address space size.
config ARM64_VA_BITS_39
bool "39-bit"
depends on ARM64_4K_PAGES
config ARM64_VA_BITS_42
bool "42-bit"
depends on ARM64_64K_PAGES
config ARM64_VA_BITS_48
bool "48-bit"
depends on BROKEN
endchoice
config ARM64_VA_BITS
int
default 39 if ARM64_VA_BITS_39
default 42 if ARM64_VA_BITS_42
default 48 if ARM64_VA_BITS_48
config ARM64_PGTABLE_LEVELS
int
default 2 if ARM64_64K_PAGES && ARM64_VA_BITS_42
default 3 if ARM64_64K_PAGES && ARM64_VA_BITS_48
default 3 if ARM64_4K_PAGES && ARM64_VA_BITS_39
default 4 if ARM64_4K_PAGES && ARM64_VA_BITS_48
config CPU_BIG_ENDIAN
bool "Build big-endian kernel"
help
......
arch/arm64/Kconfig.debug
浏览文件 @ 5167d09f
......@@ -28,4 +28,19 @@ config PID_IN_CONTEXTIDR
instructions during context switch. Say Y here only if you are
planning to use hardware trace tools with this kernel.
config ARM64_RANDOMIZE_TEXT_OFFSET
bool "Randomize TEXT_OFFSET at build time"
help
Say Y here if you want the image load offset (AKA TEXT_OFFSET)
of the kernel to be randomized at build-time. When selected,
this option will cause TEXT_OFFSET to be randomized upon any
build of the kernel, and the offset will be reflected in the
text_offset field of the resulting Image. This can be used to
fuzz-test bootloaders which respect text_offset.
This option is intended for bootloader and/or kernel testing
only. Bootloaders must make no assumptions regarding the value
of TEXT_OFFSET and platforms must not require a specific
value.
endmenu
arch/arm64/Makefile
浏览文件 @ 5167d09f
......@@ -38,7 +38,11 @@ CHECKFLAGS += -D__aarch64__
head-y := arch/arm64/kernel/head.o
# The byte offset of the kernel image in RAM from the start of RAM.
ifeq ($(CONFIG_ARM64_RANDOMIZE_TEXT_OFFSET), y)
TEXT_OFFSET := $(shell awk 'BEGIN {srand(); printf "0x%04x0\n", int(65535 * rand())}')
else
TEXT_OFFSET := 0x00080000
endif
export TEXT_OFFSET GZFLAGS
......
arch/arm64/configs/defconfig
浏览文件 @ 5167d09f
......@@ -52,8 +52,11 @@ CONFIG_IP_PNP_BOOTP=y
# CONFIG_INET_LRO is not set
# CONFIG_IPV6 is not set
# CONFIG_WIRELESS is not set
CONFIG_NET_9P=y
CONFIG_NET_9P_VIRTIO=y
CONFIG_UEVENT_HELPER_PATH="/sbin/hotplug"
CONFIG_DEVTMPFS=y
CONFIG_DEVTMPFS_MOUNT=y
CONFIG_DMA_CMA=y
CONFIG_BLK_DEV_LOOP=y
CONFIG_VIRTIO_BLK=y
......@@ -65,6 +68,7 @@ CONFIG_PATA_PLATFORM=y
CONFIG_PATA_OF_PLATFORM=y
CONFIG_NETDEVICES=y
CONFIG_TUN=y
CONFIG_VIRTIO_NET=y
CONFIG_SMC91X=y
CONFIG_SMSC911X=y
# CONFIG_WLAN is not set
......@@ -76,6 +80,7 @@ CONFIG_SERIAL_8250_CONSOLE=y
CONFIG_SERIAL_AMBA_PL011=y
CONFIG_SERIAL_AMBA_PL011_CONSOLE=y
CONFIG_SERIAL_OF_PLATFORM=y
CONFIG_VIRTIO_CONSOLE=y
# CONFIG_HW_RANDOM is not set
# CONFIG_HWMON is not set
CONFIG_REGULATOR=y
......@@ -90,6 +95,7 @@ CONFIG_USB_ISP1760_HCD=y
CONFIG_USB_STORAGE=y
CONFIG_MMC=y
CONFIG_MMC_ARMMMCI=y
CONFIG_VIRTIO_BALLOON=y
CONFIG_VIRTIO_MMIO=y
# CONFIG_IOMMU_SUPPORT is not set
CONFIG_EXT2_FS=y
......@@ -107,6 +113,7 @@ CONFIG_HUGETLBFS=y
# CONFIG_MISC_FILESYSTEMS is not set
CONFIG_NFS_FS=y
CONFIG_ROOT_NFS=y
CONFIG_9P_FS=y
CONFIG_NLS_CODEPAGE_437=y
CONFIG_NLS_ISO8859_1=y
CONFIG_VIRTUALIZATION=y
......
arch/arm64/crypto/Makefile
浏览文件 @ 5167d09f
......@@ -35,4 +35,4 @@ AFLAGS_aes-neon.o := -DINTERLEAVE=4
CFLAGS_aes-glue-ce.o := -DUSE_V8_CRYPTO_EXTENSIONS
$(obj)/aes-glue-%.o: $(src)/aes-glue.c FORCE
$(call if_changed_dep,cc_o_c)
$(call if_changed_rule,cc_o_c)
arch/arm64/include/asm/cacheflush.h
浏览文件 @ 5167d09f
......@@ -138,19 +138,10 @@ static inline void __flush_icache_all(void)
#define flush_icache_page(vma,page) do { } while (0)
/*
* flush_cache_vmap() is used when creating mappings (eg, via vmap,
* vmalloc, ioremap etc) in kernel space for pages. On non-VIPT
* caches, since the direct-mappings of these pages may contain cached
* data, we need to do a full cache flush to ensure that writebacks
* don't corrupt data placed into these pages via the new mappings.
* Not required on AArch64 (PIPT or VIPT non-aliasing D-cache).
*/
static inline void flush_cache_vmap(unsigned long start, unsigned long end)
{
/*
* set_pte_at() called from vmap_pte_range() does not
* have a DSB after cleaning the cache line.
*/
dsb(ish);
}
static inline void flush_cache_vunmap(unsigned long start, unsigned long end)
......
arch/arm64/include/asm/cachetype.h
浏览文件 @ 5167d09f
......@@ -30,10 +30,14 @@
#ifndef __ASSEMBLY__
static inline u32 icache_policy(void)
{
return (read_cpuid_cachetype() >> CTR_L1IP_SHIFT) & CTR_L1IP_MASK;
}
#include <linux/bitops.h>
#define CTR_L1IP(ctr) (((ctr) >> CTR_L1IP_SHIFT) & CTR_L1IP_MASK)
#define ICACHEF_ALIASING BIT(0)
#define ICACHEF_AIVIVT BIT(1)
extern unsigned long __icache_flags;
/*
* Whilst the D-side always behaves as PIPT on AArch64, aliasing is
......@@ -41,12 +45,12 @@ static inline u32 icache_policy(void)
*/
static inline int icache_is_aliasing(void)
{
return icache_policy() != ICACHE_POLICY_PIPT;
return test_bit(ICACHEF_ALIASING, &__icache_flags);
}
static inline int icache_is_aivivt(void)
{
return icache_policy() == ICACHE_POLICY_AIVIVT;
return test_bit(ICACHEF_AIVIVT, &__icache_flags);
}
static inline u32 cache_type_cwg(void)
......
arch/arm64/include/asm/pgtable-2level-hwdef.h → arch/arm64/include/asm/cpu.h
浏览文件 @ 5167d09f
/*
* Copyright (C) 2012 ARM Ltd.
* Copyright (C) 2014 ARM Ltd.
*
* This program is free software; you can redistribute it and/or modify
* it under the terms of the GNU General Public License version 2 as
......@@ -13,31 +13,47 @@
* You should have received a copy of the GNU General Public License
* along with this program. If not, see <http://www.gnu.org/licenses/>.
*/
#ifndef __ASM_PGTABLE_2LEVEL_HWDEF_H
#define __ASM_PGTABLE_2LEVEL_HWDEF_H
#ifndef __ASM_CPU_H
#define __ASM_CPU_H
/*
* With LPAE and 64KB pages, there are 2 levels of page tables. Each level has
* 8192 entries of 8 bytes each, occupying a 64KB page. Levels 0 and 1 are not
* used. The 2nd level table (PGD for Linux) can cover a range of 4TB, each
* entry representing 512MB. The user and kernel address spaces are limited to
* 4TB in the 64KB page configuration.
*/
#define PTRS_PER_PTE 8192
#define PTRS_PER_PGD 8192
#include <linux/cpu.h>
#include <linux/init.h>
#include <linux/percpu.h>
/*
* PGDIR_SHIFT determines the size a top-level page table entry can map.
* Records attributes of an individual CPU.
*/
#define PGDIR_SHIFT 29
#define PGDIR_SIZE (_AC(1, UL) << PGDIR_SHIFT)
#define PGDIR_MASK (~(PGDIR_SIZE-1))
struct cpuinfo_arm64 {
struct cpu cpu;
u32 reg_ctr;
u32 reg_cntfrq;
u32 reg_dczid;
u32 reg_midr;
/*
* section address mask and size definitions.
*/
#define SECTION_SHIFT 29
#define SECTION_SIZE (_AC(1, UL) << SECTION_SHIFT)
#define SECTION_MASK (~(SECTION_SIZE-1))
u64 reg_id_aa64isar0;
u64 reg_id_aa64isar1;
u64 reg_id_aa64mmfr0;
u64 reg_id_aa64mmfr1;
u64 reg_id_aa64pfr0;
u64 reg_id_aa64pfr1;
u32 reg_id_isar0;
u32 reg_id_isar1;
u32 reg_id_isar2;
u32 reg_id_isar3;
u32 reg_id_isar4;
u32 reg_id_isar5;
u32 reg_id_mmfr0;
u32 reg_id_mmfr1;
u32 reg_id_mmfr2;
u32 reg_id_mmfr3;
u32 reg_id_pfr0;
u32 reg_id_pfr1;
};
DECLARE_PER_CPU(struct cpuinfo_arm64, cpu_data);
void cpuinfo_store_cpu(void);
void __init cpuinfo_store_boot_cpu(void);
#endif
#endif /* __ASM_CPU_H */
arch/arm64/include/asm/cputype.h
浏览文件 @ 5167d09f
......@@ -18,6 +18,8 @@
#define INVALID_HWID ULONG_MAX
#define MPIDR_UP_BITMASK (0x1 << 30)
#define MPIDR_MT_BITMASK (0x1 << 24)
#define MPIDR_HWID_BITMASK 0xff00ffffff
#define MPIDR_LEVEL_BITS_SHIFT 3
......@@ -36,15 +38,34 @@
__val; \
})
#define MIDR_REVISION_MASK 0xf
#define MIDR_REVISION(midr) ((midr) & MIDR_REVISION_MASK)
#define MIDR_PARTNUM_SHIFT 4
#define MIDR_PARTNUM_MASK (0xfff << MIDR_PARTNUM_SHIFT)
#define MIDR_PARTNUM(midr) \
(((midr) & MIDR_PARTNUM_MASK) >> MIDR_PARTNUM_SHIFT)
#define MIDR_ARCHITECTURE_SHIFT 16
#define MIDR_ARCHITECTURE_MASK (0xf << MIDR_ARCHITECTURE_SHIFT)
#define MIDR_ARCHITECTURE(midr) \
(((midr) & MIDR_ARCHITECTURE_MASK) >> MIDR_ARCHITECTURE_SHIFT)
#define MIDR_VARIANT_SHIFT 20
#define MIDR_VARIANT_MASK (0xf << MIDR_VARIANT_SHIFT)
#define MIDR_VARIANT(midr) \
(((midr) & MIDR_VARIANT_MASK) >> MIDR_VARIANT_SHIFT)
#define MIDR_IMPLEMENTOR_SHIFT 24
#define MIDR_IMPLEMENTOR_MASK (0xff << MIDR_IMPLEMENTOR_SHIFT)
#define MIDR_IMPLEMENTOR(midr) \
(((midr) & MIDR_IMPLEMENTOR_MASK) >> MIDR_IMPLEMENTOR_SHIFT)
#define ARM_CPU_IMP_ARM 0x41
#define ARM_CPU_IMP_APM 0x50
#define ARM_CPU_PART_AEM_V8 0xD0F0
#define ARM_CPU_PART_FOUNDATION 0xD000
#define ARM_CPU_PART_CORTEX_A53 0xD030
#define ARM_CPU_PART_CORTEX_A57 0xD070
#define ARM_CPU_PART_AEM_V8 0xD0F
#define ARM_CPU_PART_FOUNDATION 0xD00
#define ARM_CPU_PART_CORTEX_A57 0xD07
#define ARM_CPU_PART_CORTEX_A53 0xD03
#define APM_CPU_PART_POTENZA 0x0000
#define APM_CPU_PART_POTENZA 0x000
#ifndef __ASSEMBLY__
......@@ -65,12 +86,12 @@ static inline u64 __attribute_const__ read_cpuid_mpidr(void)
static inline unsigned int __attribute_const__ read_cpuid_implementor(void)
{
return (read_cpuid_id() & 0xFF000000) >> 24;
return MIDR_IMPLEMENTOR(read_cpuid_id());
}
static inline unsigned int __attribute_const__ read_cpuid_part_number(void)
{
return (read_cpuid_id() & 0xFFF0);
return MIDR_PARTNUM(read_cpuid_id());
}
static inline u32 __attribute_const__ read_cpuid_cachetype(void)
......
arch/arm64/include/asm/fpsimdmacros.h
浏览文件 @ 5167d09f
......@@ -40,6 +40,19 @@
str w\tmpnr, [\state, #16 * 2 + 4]
.endm
.macro fpsimd_restore_fpcr state, tmp
/*
* Writes to fpcr may be self-synchronising, so avoid restoring
* the register if it hasn't changed.
*/
mrs \tmp, fpcr
cmp \tmp, \state
b.eq 9999f
msr fpcr, \state
9999:
.endm
/* Clobbers \state */
.macro fpsimd_restore state, tmpnr
ldp q0, q1, [\state, #16 * 0]
ldp q2, q3, [\state, #16 * 2]
......@@ -60,7 +73,7 @@
ldr w\tmpnr, [\state, #16 * 2]
msr fpsr, x\tmpnr
ldr w\tmpnr, [\state, #16 * 2 + 4]
msr fpcr, x\tmpnr
fpsimd_restore_fpcr x\tmpnr, \state
.endm
.altmacro
......@@ -84,7 +97,7 @@
.macro fpsimd_restore_partial state, tmpnr1, tmpnr2
ldp w\tmpnr1, w\tmpnr2, [\state]
msr fpsr, x\tmpnr1
msr fpcr, x\tmpnr2
fpsimd_restore_fpcr x\tmpnr2, x\tmpnr1
adr x\tmpnr1, 0f
ldr w\tmpnr2, [\state, #8]
add \state, \state, x\tmpnr2, lsl #4
......
arch/arm64/include/asm/memory.h
浏览文件 @ 5167d09f
......@@ -41,11 +41,7 @@
* The module space lives between the addresses given by TASK_SIZE
* and PAGE_OFFSET - it must be within 128MB of the kernel text.
*/
#ifdef CONFIG_ARM64_64K_PAGES
#define VA_BITS (42)
#else
#define VA_BITS (39)
#endif
#define VA_BITS (CONFIG_ARM64_VA_BITS)
#define PAGE_OFFSET (UL(0xffffffffffffffff) << (VA_BITS - 1))
#define MODULES_END (PAGE_OFFSET)
#define MODULES_VADDR (MODULES_END - SZ_64M)
......
arch/arm64/include/asm/page.h
浏览文件 @ 5167d09f
......@@ -31,14 +31,26 @@
/* We do define AT_SYSINFO_EHDR but don't use the gate mechanism */
#define __HAVE_ARCH_GATE_AREA 1
#ifndef __ASSEMBLY__
/*
* The idmap and swapper page tables need some space reserved in the kernel
* image. Both require pgd, pud (4 levels only) and pmd tables to (section)
* map the kernel. With the 64K page configuration, swapper and idmap need to
* map to pte level. The swapper also maps the FDT (see __create_page_tables
* for more information).
*/
#ifdef CONFIG_ARM64_64K_PAGES
#include <asm/pgtable-2level-types.h>
#define SWAPPER_PGTABLE_LEVELS (CONFIG_ARM64_PGTABLE_LEVELS)
#else
#include <asm/pgtable-3level-types.h>
#define SWAPPER_PGTABLE_LEVELS (CONFIG_ARM64_PGTABLE_LEVELS - 1)
#endif
#define SWAPPER_DIR_SIZE (SWAPPER_PGTABLE_LEVELS * PAGE_SIZE)
#define IDMAP_DIR_SIZE (SWAPPER_DIR_SIZE)
#ifndef __ASSEMBLY__
#include <asm/pgtable-types.h>
extern void __cpu_clear_user_page(void *p, unsigned long user);
extern void __cpu_copy_user_page(void *to, const void *from,
unsigned long user);
......
arch/arm64/include/asm/pgalloc.h
浏览文件 @ 5167d09f
......@@ -26,7 +26,7 @@
#define check_pgt_cache() do { } while (0)
#ifndef CONFIG_ARM64_64K_PAGES
#if CONFIG_ARM64_PGTABLE_LEVELS > 2
static inline pmd_t *pmd_alloc_one(struct mm_struct *mm, unsigned long addr)
{
......@@ -44,7 +44,27 @@ static inline void pud_populate(struct mm_struct *mm, pud_t *pud, pmd_t *pmd)
set_pud(pud, __pud(__pa(pmd) | PMD_TYPE_TABLE));
}
#endif /* CONFIG_ARM64_64K_PAGES */
#endif /* CONFIG_ARM64_PGTABLE_LEVELS > 2 */
#if CONFIG_ARM64_PGTABLE_LEVELS > 3
static inline pud_t *pud_alloc_one(struct mm_struct *mm, unsigned long addr)
{
return (pud_t *)get_zeroed_page(GFP_KERNEL | __GFP_REPEAT);
}
static inline void pud_free(struct mm_struct *mm, pud_t *pud)
{
BUG_ON((unsigned long)pud & (PAGE_SIZE-1));
free_page((unsigned long)pud);
}
static inline void pgd_populate(struct mm_struct *mm, pgd_t *pgd, pud_t *pud)
{
set_pgd(pgd, __pgd(__pa(pud) | PUD_TYPE_TABLE));
}
#endif /* CONFIG_ARM64_PGTABLE_LEVELS > 3 */
extern pgd_t *pgd_alloc(struct mm_struct *mm);
extern void pgd_free(struct mm_struct *mm, pgd_t *pgd);
......
arch/arm64/include/asm/pgtable-hwdef.h
浏览文件 @ 5167d09f
......@@ -16,18 +16,50 @@
#ifndef __ASM_PGTABLE_HWDEF_H
#define __ASM_PGTABLE_HWDEF_H
#ifdef CONFIG_ARM64_64K_PAGES
#include <asm/pgtable-2level-hwdef.h>
#else
#include <asm/pgtable-3level-hwdef.h>
#define PTRS_PER_PTE (1 << (PAGE_SHIFT - 3))
/*
* PMD_SHIFT determines the size a level 2 page table entry can map.
*/
#if CONFIG_ARM64_PGTABLE_LEVELS > 2
#define PMD_SHIFT ((PAGE_SHIFT - 3) * 2 + 3)
#define PMD_SIZE (_AC(1, UL) << PMD_SHIFT)
#define PMD_MASK (~(PMD_SIZE-1))
#define PTRS_PER_PMD PTRS_PER_PTE
#endif
/*
* PUD_SHIFT determines the size a level 1 page table entry can map.
*/
#if CONFIG_ARM64_PGTABLE_LEVELS > 3
#define PUD_SHIFT ((PAGE_SHIFT - 3) * 3 + 3)
#define PUD_SIZE (_AC(1, UL) << PUD_SHIFT)
#define PUD_MASK (~(PUD_SIZE-1))
#define PTRS_PER_PUD PTRS_PER_PTE
#endif
/*
* PGDIR_SHIFT determines the size a top-level page table entry can map
* (depending on the configuration, this level can be 0, 1 or 2).
*/
#define PGDIR_SHIFT ((PAGE_SHIFT - 3) * CONFIG_ARM64_PGTABLE_LEVELS + 3)
#define PGDIR_SIZE (_AC(1, UL) << PGDIR_SHIFT)
#define PGDIR_MASK (~(PGDIR_SIZE-1))
#define PTRS_PER_PGD (1 << (VA_BITS - PGDIR_SHIFT))
/*
* Section address mask and size definitions.
*/
#define SECTION_SHIFT PMD_SHIFT
#define SECTION_SIZE (_AC(1, UL) << SECTION_SHIFT)
#define SECTION_MASK (~(SECTION_SIZE-1))
/*
* Hardware page table definitions.
*
* Level 1 descriptor (PUD).
*/
#define PUD_TYPE_TABLE (_AT(pudval_t, 3) << 0)
#define PUD_TABLE_BIT (_AT(pgdval_t, 1) << 1)
#define PUD_TYPE_MASK (_AT(pgdval_t, 3) << 0)
#define PUD_TYPE_SECT (_AT(pgdval_t, 1) << 0)
......
arch/arm64/include/asm/pgtable-3level-types.h → arch/arm64/include/asm/pgtable-types.h
浏览文件 @ 5167d09f
/*
* Copyright (C) 2012 ARM Ltd.
* Page table types definitions.
*
* This program is free software; you can redistribute it and/or modify
* Copyright (C) 2014 ARM Ltd.
* Author: Catalin Marinas <catalin.marinas@arm.com>
*
* This program is free software: you can redistribute it and/or modify
* it under the terms of the GNU General Public License version 2 as
* published by the Free Software Foundation.
*
......@@ -13,13 +16,15 @@
* You should have received a copy of the GNU General Public License
* along with this program. If not, see <http://www.gnu.org/licenses/>.
*/
#ifndef __ASM_PGTABLE_3LEVEL_TYPES_H
#define __ASM_PGTABLE_3LEVEL_TYPES_H
#ifndef __ASM_PGTABLE_TYPES_H
#define __ASM_PGTABLE_TYPES_H
#include <asm/types.h>
typedef u64 pteval_t;
typedef u64 pmdval_t;
typedef u64 pudval_t;
typedef u64 pgdval_t;
#undef STRICT_MM_TYPECHECKS
......@@ -30,39 +35,61 @@ typedef u64 pgdval_t;
* These are used to make use of C type-checking..
*/
typedef struct { pteval_t pte; } pte_t;
#define pte_val(x) ((x).pte)
#define __pte(x) ((pte_t) { (x) } )
#if CONFIG_ARM64_PGTABLE_LEVELS > 2
typedef struct { pmdval_t pmd; } pmd_t;
typedef struct { pgdval_t pgd; } pgd_t;
typedef struct { pteval_t pgprot; } pgprot_t;
#define pmd_val(x) ((x).pmd)
#define __pmd(x) ((pmd_t) { (x) } )
#endif
#define pte_val(x) ((x).pte)
#define pmd_val(x) ((x).pmd)
#define pgd_val(x) ((x).pgd)
#define pgprot_val(x) ((x).pgprot)
#if CONFIG_ARM64_PGTABLE_LEVELS > 3
typedef struct { pudval_t pud; } pud_t;
#define pud_val(x) ((x).pud)
#define __pud(x) ((pud_t) { (x) } )
#endif
#define __pte(x) ((pte_t) { (x) } )
#define __pmd(x) ((pmd_t) { (x) } )
typedef struct { pgdval_t pgd; } pgd_t;
#define pgd_val(x) ((x).pgd)
#define __pgd(x) ((pgd_t) { (x) } )
#define __pgprot(x) ((pgprot_t) { (x) } )
typedef struct { pteval_t pgprot; } pgprot_t;
#define pgprot_val(x) ((x).pgprot)
#define __pgprot(x) ((pgprot_t) { (x) } )
#else /* !STRICT_MM_TYPECHECKS */
typedef pteval_t pte_t;
typedef pmdval_t pmd_t;
typedef pgdval_t pgd_t;
typedef pteval_t pgprot_t;
#define pte_val(x) (x)
#define pmd_val(x) (x)
#define pgd_val(x) (x)
#define pgprot_val(x) (x)
#define __pte(x) (x)
#if CONFIG_ARM64_PGTABLE_LEVELS > 2
typedef pmdval_t pmd_t;
#define pmd_val(x) (x)
#define __pmd(x) (x)
#endif
#if CONFIG_ARM64_PGTABLE_LEVELS > 3
typedef pudval_t pud_t;
#define pud_val(x) (x)
#define __pud(x) (x)
#endif
typedef pgdval_t pgd_t;
#define pgd_val(x) (x)
#define __pgd(x) (x)
typedef pteval_t pgprot_t;
#define pgprot_val(x) (x)
#define __pgprot(x) (x)
#endif /* STRICT_MM_TYPECHECKS */
#endif /* STRICT_MM_TYPECHECKS */
#if CONFIG_ARM64_PGTABLE_LEVELS == 2
#include <asm-generic/pgtable-nopmd.h>
#elif CONFIG_ARM64_PGTABLE_LEVELS == 3
#include <asm-generic/pgtable-nopud.h>
#endif
#endif /* __ASM_PGTABLE_3LEVEL_TYPES_H */
#endif /* __ASM_PGTABLE_TYPES_H */
arch/arm64/include/asm/pgtable.h
浏览文件 @ 5167d09f
......@@ -33,9 +33,16 @@
/*
* VMALLOC and SPARSEMEM_VMEMMAP ranges.
*
* VMEMAP_SIZE: allows the whole VA space to be covered by a struct page array
* (rounded up to PUD_SIZE).
* VMALLOC_START: beginning of the kernel VA space
* VMALLOC_END: extends to the available space below vmmemmap, PCI I/O space,
* fixed mappings and modules
*/
#define VMEMMAP_SIZE ALIGN((1UL << (VA_BITS - PAGE_SHIFT)) * sizeof(struct page), PUD_SIZE)
#define VMALLOC_START (UL(0xffffffffffffffff) << VA_BITS)
#define VMALLOC_END (PAGE_OFFSET - UL(0x400000000) - SZ_64K)
#define VMALLOC_END (PAGE_OFFSET - PUD_SIZE - VMEMMAP_SIZE - SZ_64K)
#define vmemmap ((struct page *)(VMALLOC_END + SZ_64K))
......@@ -44,14 +51,9 @@
#ifndef __ASSEMBLY__
extern void __pte_error(const char *file, int line, unsigned long val);
extern void __pmd_error(const char *file, int line, unsigned long val);
extern void __pud_error(const char *file, int line, unsigned long val);
extern void __pgd_error(const char *file, int line, unsigned long val);
#define pte_ERROR(pte) __pte_error(__FILE__, __LINE__, pte_val(pte))
#ifndef CONFIG_ARM64_64K_PAGES
#define pmd_ERROR(pmd) __pmd_error(__FILE__, __LINE__, pmd_val(pmd))
#endif
#define pgd_ERROR(pgd) __pgd_error(__FILE__, __LINE__, pgd_val(pgd))
#ifdef CONFIG_SMP
#define PROT_DEFAULT (PTE_TYPE_PAGE | PTE_AF | PTE_SHARED)
#define PROT_SECT_DEFAULT (PMD_TYPE_SECT | PMD_SECT_AF | PMD_SECT_S)
......@@ -112,6 +114,8 @@ extern void __pgd_error(const char *file, int line, unsigned long val);
extern struct page *empty_zero_page;
#define ZERO_PAGE(vaddr) (empty_zero_page)
#define pte_ERROR(pte) __pte_error(__FILE__, __LINE__, pte_val(pte))
#define pte_pfn(pte) ((pte_val(pte) & PHYS_MASK) >> PAGE_SHIFT)
#define pfn_pte(pfn,prot) (__pte(((phys_addr_t)(pfn) << PAGE_SHIFT) | pgprot_val(prot)))
......@@ -119,6 +123,10 @@ extern struct page *empty_zero_page;
#define pte_none(pte) (!pte_val(pte))
#define pte_clear(mm,addr,ptep) set_pte(ptep, __pte(0))
#define pte_page(pte) (pfn_to_page(pte_pfn(pte)))
/* Find an entry in the third-level page table. */
#define pte_index(addr) (((addr) >> PAGE_SHIFT) & (PTRS_PER_PTE - 1))
#define pte_offset_kernel(dir,addr) (pmd_page_vaddr(*(dir)) + pte_index(addr))
#define pte_offset_map(dir,addr) pte_offset_kernel((dir), (addr))
......@@ -138,6 +146,8 @@ extern struct page *empty_zero_page;
#define pte_valid_user(pte) \
((pte_val(pte) & (PTE_VALID | PTE_USER)) == (PTE_VALID | PTE_USER))
#define pte_valid_not_user(pte) \
((pte_val(pte) & (PTE_VALID | PTE_USER)) == PTE_VALID)
static inline pte_t pte_wrprotect(pte_t pte)
{
......@@ -184,6 +194,15 @@ static inline pte_t pte_mkspecial(pte_t pte)
static inline void set_pte(pte_t *ptep, pte_t pte)
{
*ptep = pte;
/*
* Only if the new pte is valid and kernel, otherwise TLB maintenance
* or update_mmu_cache() have the necessary barriers.
*/
if (pte_valid_not_user(pte)) {
dsb(ishst);
isb();
}
}
extern void __sync_icache_dcache(pte_t pteval, unsigned long addr);
......@@ -303,6 +322,7 @@ static inline void set_pmd(pmd_t *pmdp, pmd_t pmd)
{
*pmdp = pmd;
dsb(ishst);
isb();
}
static inline void pmd_clear(pmd_t *pmdp)
......@@ -323,7 +343,9 @@ static inline pte_t *pmd_page_vaddr(pmd_t pmd)
*/
#define mk_pte(page,prot) pfn_pte(page_to_pfn(page),prot)
#ifndef CONFIG_ARM64_64K_PAGES
#if CONFIG_ARM64_PGTABLE_LEVELS > 2
#define pmd_ERROR(pmd) __pmd_error(__FILE__, __LINE__, pmd_val(pmd))
#define pud_none(pud) (!pud_val(pud))
#define pud_bad(pud) (!(pud_val(pud) & 2))
......@@ -333,6 +355,7 @@ static inline void set_pud(pud_t *pudp, pud_t pud)
{
*pudp = pud;
dsb(ishst);
isb();
}
static inline void pud_clear(pud_t *pudp)
......@@ -345,7 +368,51 @@ static inline pmd_t *pud_page_vaddr(pud_t pud)
return __va(pud_val(pud) & PHYS_MASK & (s32)PAGE_MASK);
}
#endif /* CONFIG_ARM64_64K_PAGES */
/* Find an entry in the second-level page table. */
#define pmd_index(addr) (((addr) >> PMD_SHIFT) & (PTRS_PER_PMD - 1))
static inline pmd_t *pmd_offset(pud_t *pud, unsigned long addr)
{
return (pmd_t *)pud_page_vaddr(*pud) + pmd_index(addr);
}
#endif /* CONFIG_ARM64_PGTABLE_LEVELS > 2 */
#if CONFIG_ARM64_PGTABLE_LEVELS > 3
#define pud_ERROR(pud) __pud_error(__FILE__, __LINE__, pud_val(pud))
#define pgd_none(pgd) (!pgd_val(pgd))
#define pgd_bad(pgd) (!(pgd_val(pgd) & 2))
#define pgd_present(pgd) (pgd_val(pgd))
static inline void set_pgd(pgd_t *pgdp, pgd_t pgd)
{
*pgdp = pgd;
dsb(ishst);
}
static inline void pgd_clear(pgd_t *pgdp)
{
set_pgd(pgdp, __pgd(0));
}
static inline pud_t *pgd_page_vaddr(pgd_t pgd)
{
return __va(pgd_val(pgd) & PHYS_MASK & (s32)PAGE_MASK);
}
/* Find an entry in the frst-level page table. */
#define pud_index(addr) (((addr) >> PUD_SHIFT) & (PTRS_PER_PUD - 1))
static inline pud_t *pud_offset(pgd_t *pgd, unsigned long addr)
{
return (pud_t *)pgd_page_vaddr(*pgd) + pud_index(addr);
}
#endif /* CONFIG_ARM64_PGTABLE_LEVELS > 3 */
#define pgd_ERROR(pgd) __pgd_error(__FILE__, __LINE__, pgd_val(pgd))
/* to find an entry in a page-table-directory */
#define pgd_index(addr) (((addr) >> PGDIR_SHIFT) & (PTRS_PER_PGD - 1))
......@@ -355,18 +422,6 @@ static inline pmd_t *pud_page_vaddr(pud_t pud)
/* to find an entry in a kernel page-table-directory */
#define pgd_offset_k(addr) pgd_offset(&init_mm, addr)
/* Find an entry in the second-level page table.. */
#ifndef CONFIG_ARM64_64K_PAGES
#define pmd_index(addr) (((addr) >> PMD_SHIFT) & (PTRS_PER_PMD - 1))
static inline pmd_t *pmd_offset(pud_t *pud, unsigned long addr)
{
return (pmd_t *)pud_page_vaddr(*pud) + pmd_index(addr);
}
#endif
/* Find an entry in the third-level page table.. */
#define pte_index(addr) (((addr) >> PAGE_SHIFT) & (PTRS_PER_PTE - 1))
static inline pte_t pte_modify(pte_t pte, pgprot_t newprot)
{
const pteval_t mask = PTE_USER | PTE_PXN | PTE_UXN | PTE_RDONLY |
......@@ -383,9 +438,6 @@ static inline pmd_t pmd_modify(pmd_t pmd, pgprot_t newprot)
extern pgd_t swapper_pg_dir[PTRS_PER_PGD];
extern pgd_t idmap_pg_dir[PTRS_PER_PGD];
#define SWAPPER_DIR_SIZE (3 * PAGE_SIZE)
#define IDMAP_DIR_SIZE (2 * PAGE_SIZE)
/*
* Encode and decode a swap entry:
* bits 0-1: present (must be zero)
......
arch/arm64/include/asm/processor.h
浏览文件 @ 5167d09f
......@@ -137,8 +137,8 @@ extern struct task_struct *cpu_switch_to(struct task_struct *prev,
#define task_pt_regs(p) \
((struct pt_regs *)(THREAD_START_SP + task_stack_page(p)) - 1)
#define KSTK_EIP(tsk) task_pt_regs(tsk)->pc
#define KSTK_ESP(tsk) task_pt_regs(tsk)->sp
#define KSTK_EIP(tsk) ((unsigned long)task_pt_regs(tsk)->pc)
#define KSTK_ESP(tsk) ((unsigned long)task_pt_regs(tsk)->sp)
/*
* Prefetching support
......
arch/arm64/include/asm/stackprotector.h 0 → 100644
浏览文件 @ 5167d09f
/*
* GCC stack protector support.
*
* Stack protector works by putting predefined pattern at the start of
* the stack frame and verifying that it hasn't been overwritten when
* returning from the function. The pattern is called stack canary
* and gcc expects it to be defined by a global variable called
* "__stack_chk_guard" on ARM. This unfortunately means that on SMP
* we cannot have a different canary value per task.
*/
#ifndef __ASM_STACKPROTECTOR_H
#define __ASM_STACKPROTECTOR_H
#include <linux/random.h>
#include <linux/version.h>
extern unsigned long __stack_chk_guard;
/*
* Initialize the stackprotector canary value.
*
* NOTE: this must only be called from functions that never return,
* and it must always be inlined.
*/
static __always_inline void boot_init_stack_canary(void)
{
unsigned long canary;
/* Try to get a semi random initial value. */
get_random_bytes(&canary, sizeof(canary));
canary ^= LINUX_VERSION_CODE;
current->stack_canary = canary;
__stack_chk_guard = current->stack_canary;
}
#endif /* _ASM_STACKPROTECTOR_H */
arch/arm64/include/asm/syscall.h
浏览文件 @ 5167d09f
......@@ -16,6 +16,8 @@
#ifndef __ASM_SYSCALL_H
#define __ASM_SYSCALL_H
#include <uapi/linux/audit.h>
#include <linux/compat.h>
#include <linux/err.h>
extern const void *sys_call_table[];
......@@ -105,4 +107,16 @@ static inline void syscall_set_arguments(struct task_struct *task,
memcpy(&regs->regs[i], args, n * sizeof(args[0]));
}
/*
* We don't care about endianness (__AUDIT_ARCH_LE bit) here because
* AArch64 has the same system calls both on little- and big- endian.
*/
static inline int syscall_get_arch(void)
{
if (is_compat_task())
return AUDIT_ARCH_ARM;
return AUDIT_ARCH_AARCH64;
}
#endif /* __ASM_SYSCALL_H */
arch/arm64/include/asm/pgtable-2level-types.h → arch/arm64/include/asm/sysreg.h
浏览文件 @ 5167d09f
/*
* Copyright (C) 2012 ARM Ltd.
* Macros for accessing system registers with older binutils.
*
* This program is free software; you can redistribute it and/or modify
* Copyright (C) 2014 ARM Ltd.
* Author: Catalin Marinas <catalin.marinas@arm.com>
*
* This program is free software: you can redistribute it and/or modify
* it under the terms of the GNU General Public License version 2 as
* published by the Free Software Foundation.
*
......@@ -13,50 +16,45 @@
* You should have received a copy of the GNU General Public License
* along with this program. If not, see <http://www.gnu.org/licenses/>.
*/
#ifndef __ASM_PGTABLE_2LEVEL_TYPES_H
#define __ASM_PGTABLE_2LEVEL_TYPES_H
#include <asm/types.h>
typedef u64 pteval_t;
typedef u64 pgdval_t;
typedef pgdval_t pmdval_t;
#undef STRICT_MM_TYPECHECKS
#ifdef STRICT_MM_TYPECHECKS
/*
* These are used to make use of C type-checking..
*/
typedef struct { pteval_t pte; } pte_t;
typedef struct { pgdval_t pgd; } pgd_t;
typedef struct { pteval_t pgprot; } pgprot_t;
#ifndef __ASM_SYSREG_H
#define __ASM_SYSREG_H
#define pte_val(x) ((x).pte)
#define pgd_val(x) ((x).pgd)
#define pgprot_val(x) ((x).pgprot)
#define sys_reg(op0, op1, crn, crm, op2) \
((((op0)-2)<<19)|((op1)<<16)|((crn)<<12)|((crm)<<8)|((op2)<<5))
#define __pte(x) ((pte_t) { (x) } )
#define __pgd(x) ((pgd_t) { (x) } )
#define __pgprot(x) ((pgprot_t) { (x) } )
#ifdef __ASSEMBLY__
#else /* !STRICT_MM_TYPECHECKS */
.irp num,0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23,24,25,26,27,28,29,30
.equ __reg_num_x\num, \num
.endr
.equ __reg_num_xzr, 31
typedef pteval_t pte_t;
typedef pgdval_t pgd_t;
typedef pteval_t pgprot_t;
.macro mrs_s, rt, sreg
.inst 0xd5300000|(\sreg)|(__reg_num_\rt)
.endm
#define pte_val(x) (x)
#define pgd_val(x) (x)
#define pgprot_val(x) (x)
.macro msr_s, sreg, rt
.inst 0xd5100000|(\sreg)|(__reg_num_\rt)
.endm
#define __pte(x) (x)
#define __pgd(x) (x)
#define __pgprot(x) (x)
#else
#endif /* STRICT_MM_TYPECHECKS */
asm(
" .irp num,0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23,24,25,26,27,28,29,30\n"
" .equ __reg_num_x\\num, \\num\n"
" .endr\n"
" .equ __reg_num_xzr, 31\n"
"\n"
" .macro mrs_s, rt, sreg\n"
" .inst 0xd5300000|(\\sreg)|(__reg_num_\\rt)\n"
" .endm\n"
"\n"
" .macro msr_s, sreg, rt\n"
" .inst 0xd5100000|(\\sreg)|(__reg_num_\\rt)\n"
" .endm\n"
);
#include <asm-generic/pgtable-nopmd.h>
#endif
#endif /* __ASM_PGTABLE_2LEVEL_TYPES_H */
#endif /* __ASM_SYSREG_H */
arch/arm64/include/asm/thread_info.h
浏览文件 @ 5167d09f
......@@ -103,6 +103,7 @@ static inline struct thread_info *current_thread_info(void)
#define TIF_NEED_RESCHED 1
#define TIF_NOTIFY_RESUME 2 /* callback before returning to user */
#define TIF_FOREIGN_FPSTATE 3 /* CPU's FP state is not current's */
#define TIF_NOHZ 7
#define TIF_SYSCALL_TRACE 8
#define TIF_SYSCALL_AUDIT 9
#define TIF_SYSCALL_TRACEPOINT 10
......@@ -118,6 +119,7 @@ static inline struct thread_info *current_thread_info(void)
#define _TIF_NEED_RESCHED (1 << TIF_NEED_RESCHED)
#define _TIF_NOTIFY_RESUME (1 << TIF_NOTIFY_RESUME)
#define _TIF_FOREIGN_FPSTATE (1 << TIF_FOREIGN_FPSTATE)
#define _TIF_NOHZ (1 << TIF_NOHZ)
#define _TIF_SYSCALL_TRACE (1 << TIF_SYSCALL_TRACE)
#define _TIF_SYSCALL_AUDIT (1 << TIF_SYSCALL_AUDIT)
#define _TIF_SYSCALL_TRACEPOINT (1 << TIF_SYSCALL_TRACEPOINT)
......@@ -128,7 +130,8 @@ static inline struct thread_info *current_thread_info(void)
_TIF_NOTIFY_RESUME | _TIF_FOREIGN_FPSTATE)
#define _TIF_SYSCALL_WORK (_TIF_SYSCALL_TRACE | _TIF_SYSCALL_AUDIT | \
_TIF_SYSCALL_TRACEPOINT | _TIF_SECCOMP)
_TIF_SYSCALL_TRACEPOINT | _TIF_SECCOMP | \
_TIF_NOHZ)
#endif /* __KERNEL__ */
#endif /* __ASM_THREAD_INFO_H */
arch/arm64/include/asm/tlb.h
浏览文件 @ 5167d09f
......@@ -91,7 +91,7 @@ static inline void __pte_free_tlb(struct mmu_gather *tlb, pgtable_t pte,
tlb_remove_page(tlb, pte);
}
#ifndef CONFIG_ARM64_64K_PAGES
#if CONFIG_ARM64_PGTABLE_LEVELS > 2
static inline void __pmd_free_tlb(struct mmu_gather *tlb, pmd_t *pmdp,
unsigned long addr)
{
......@@ -100,6 +100,15 @@ static inline void __pmd_free_tlb(struct mmu_gather *tlb, pmd_t *pmdp,
}
#endif
#if CONFIG_ARM64_PGTABLE_LEVELS > 3
static inline void __pud_free_tlb(struct mmu_gather *tlb, pud_t *pudp,
unsigned long addr)
{
tlb_add_flush(tlb, addr);
tlb_remove_page(tlb, virt_to_page(pudp));
}
#endif
static inline void __tlb_remove_pmd_tlb_entry(struct mmu_gather *tlb, pmd_t *pmdp,
unsigned long address)
{
......
arch/arm64/include/asm/tlbflush.h
浏览文件 @ 5167d09f
......@@ -98,8 +98,8 @@ static inline void flush_tlb_page(struct vm_area_struct *vma,
dsb(ish);
}
static inline void flush_tlb_range(struct vm_area_struct *vma,
unsigned long start, unsigned long end)
static inline void __flush_tlb_range(struct vm_area_struct *vma,
unsigned long start, unsigned long end)
{
unsigned long asid = (unsigned long)ASID(vma->vm_mm) << 48;
unsigned long addr;
......@@ -112,7 +112,7 @@ static inline void flush_tlb_range(struct vm_area_struct *vma,
dsb(ish);
}
static inline void flush_tlb_kernel_range(unsigned long start, unsigned long end)
static inline void __flush_tlb_kernel_range(unsigned long start, unsigned long end)
{
unsigned long addr;
start >>= 12;
......@@ -122,6 +122,30 @@ static inline void flush_tlb_kernel_range(unsigned long start, unsigned long end
for (addr = start; addr < end; addr += 1 << (PAGE_SHIFT - 12))
asm("tlbi vaae1is, %0" : : "r"(addr));
dsb(ish);
isb();
}
/*
* This is meant to avoid soft lock-ups on large TLB flushing ranges and not
* necessarily a performance improvement.
*/
#define MAX_TLB_RANGE (1024UL << PAGE_SHIFT)
static inline void flush_tlb_range(struct vm_area_struct *vma,
unsigned long start, unsigned long end)
{
if ((end - start) <= MAX_TLB_RANGE)
__flush_tlb_range(vma, start, end);
else
flush_tlb_mm(vma->vm_mm);
}
static inline void flush_tlb_kernel_range(unsigned long start, unsigned long end)
{
if ((end - start) <= MAX_TLB_RANGE)
__flush_tlb_kernel_range(start, end);
else
flush_tlb_all();
}
/*
......@@ -131,8 +155,8 @@ static inline void update_mmu_cache(struct vm_area_struct *vma,
unsigned long addr, pte_t *ptep)
{
/*
* set_pte() does not have a DSB, so make sure that the page table
* write is visible.
* set_pte() does not have a DSB for user mappings, so make sure that
* the page table write is visible.
*/
dsb(ishst);
}
......
arch/arm64/include/asm/unistd.h
浏览文件 @ 5167d09f
......@@ -26,7 +26,24 @@
#define __ARCH_WANT_COMPAT_SYS_SENDFILE
#define __ARCH_WANT_SYS_FORK
#define __ARCH_WANT_SYS_VFORK
/*
* Compat syscall numbers used by the AArch64 kernel.
*/
#define __NR_compat_restart_syscall 0
#define __NR_compat_sigreturn 119
#define __NR_compat_rt_sigreturn 173
/*
* The following SVCs are ARM private.
*/
#define __ARM_NR_COMPAT_BASE 0x0f0000
#define __ARM_NR_compat_cacheflush (__ARM_NR_COMPAT_BASE+2)
#define __ARM_NR_compat_set_tls (__ARM_NR_COMPAT_BASE+5)
#define __NR_compat_syscalls 383
#endif
#define __ARCH_WANT_SYS_CLONE
#include <uapi/asm/unistd.h>
......
arch/arm64/include/asm/unistd32.h
浏览文件 @ 5167d09f
此差异已折叠。
arch/arm64/kernel/Makefile
浏览文件 @ 5167d09f
......@@ -15,7 +15,8 @@ CFLAGS_REMOVE_return_address.o = -pg
arm64-obj-y := cputable.o debug-monitors.o entry.o irq.o fpsimd.o \
entry-fpsimd.o process.o ptrace.o setup.o signal.o \
sys.o stacktrace.o time.o traps.o io.o vdso.o \
hyp-stub.o psci.o cpu_ops.o insn.o return_address.o
hyp-stub.o psci.o cpu_ops.o insn.o return_address.o \
cpuinfo.o
arm64-obj-$(CONFIG_COMPAT) += sys32.o kuser32.o signal32.o \
sys_compat.o
......
arch/arm64/kernel/cpu_ops.c
浏览文件 @ 5167d09f
......@@ -30,8 +30,8 @@ const struct cpu_operations *cpu_ops[NR_CPUS];
static const struct cpu_operations *supported_cpu_ops[] __initconst = {
#ifdef CONFIG_SMP
&smp_spin_table_ops,
&cpu_psci_ops,
#endif
&cpu_psci_ops,
NULL,
};
......
arch/arm64/kernel/cpuinfo.c 0 → 100644
浏览文件 @ 5167d09f
/*
* Record and handle CPU attributes.
*
* Copyright (C) 2014 ARM Ltd.
* This program is free software; you can redistribute it and/or modify
* it under the terms of the GNU General Public License version 2 as
* published by the Free Software Foundation.
*
* This program is distributed in the hope that it will be useful,
* but WITHOUT ANY WARRANTY; without even the implied warranty of
* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
* GNU General Public License for more details.
*
* You should have received a copy of the GNU General Public License
* along with this program. If not, see <http://www.gnu.org/licenses/>.
*/
#include <asm/arch_timer.h>
#include <asm/cachetype.h>
#include <asm/cpu.h>
#include <asm/cputype.h>
#include <linux/bitops.h>
#include <linux/init.h>
#include <linux/kernel.h>
#include <linux/printk.h>
#include <linux/smp.h>
/*
* In case the boot CPU is hotpluggable, we record its initial state and
* current state separately. Certain system registers may contain different
* values depending on configuration at or after reset.
*/
DEFINE_PER_CPU(struct cpuinfo_arm64, cpu_data);
static struct cpuinfo_arm64 boot_cpu_data;
static char *icache_policy_str[] = {
[ICACHE_POLICY_RESERVED] = "RESERVED/UNKNOWN",
[ICACHE_POLICY_AIVIVT] = "AIVIVT",
[ICACHE_POLICY_VIPT] = "VIPT",
[ICACHE_POLICY_PIPT] = "PIPT",
};
unsigned long __icache_flags;
static void cpuinfo_detect_icache_policy(struct cpuinfo_arm64 *info)
{
unsigned int cpu = smp_processor_id();
u32 l1ip = CTR_L1IP(info->reg_ctr);
if (l1ip != ICACHE_POLICY_PIPT)
set_bit(ICACHEF_ALIASING, &__icache_flags);
if (l1ip == ICACHE_POLICY_AIVIVT);
set_bit(ICACHEF_AIVIVT, &__icache_flags);
pr_info("Detected %s I-cache on CPU%d\n", icache_policy_str[l1ip], cpu);
}
static int check_reg_mask(char *name, u64 mask, u64 boot, u64 cur, int cpu)
{
if ((boot & mask) == (cur & mask))
return 0;
pr_warn("SANITY CHECK: Unexpected variation in %s. Boot CPU: %#016lx, CPU%d: %#016lx\n",
name, (unsigned long)boot, cpu, (unsigned long)cur);
return 1;
}
#define CHECK_MASK(field, mask, boot, cur, cpu) \
check_reg_mask(#field, mask, (boot)->reg_ ## field, (cur)->reg_ ## field, cpu)
#define CHECK(field, boot, cur, cpu) \
CHECK_MASK(field, ~0ULL, boot, cur, cpu)
/*
* Verify that CPUs don't have unexpected differences that will cause problems.
*/
static void cpuinfo_sanity_check(struct cpuinfo_arm64 *cur)
{
unsigned int cpu = smp_processor_id();
struct cpuinfo_arm64 *boot = &boot_cpu_data;
unsigned int diff = 0;
/*
* The kernel can handle differing I-cache policies, but otherwise
* caches should look identical. Userspace JITs will make use of
* *minLine.
*/
diff |= CHECK_MASK(ctr, 0xffff3fff, boot, cur, cpu);
/*
* Userspace may perform DC ZVA instructions. Mismatched block sizes
* could result in too much or too little memory being zeroed if a
* process is preempted and migrated between CPUs.
*/
diff |= CHECK(dczid, boot, cur, cpu);
/* If different, timekeeping will be broken (especially with KVM) */
diff |= CHECK(cntfrq, boot, cur, cpu);
/*
* Even in big.LITTLE, processors should be identical instruction-set
* wise.
*/
diff |= CHECK(id_aa64isar0, boot, cur, cpu);
diff |= CHECK(id_aa64isar1, boot, cur, cpu);
/*
* Differing PARange support is fine as long as all peripherals and
* memory are mapped within the minimum PARange of all CPUs.
* Linux should not care about secure memory.
* ID_AA64MMFR1 is currently RES0.
*/
diff |= CHECK_MASK(id_aa64mmfr0, 0xffffffffffff0ff0, boot, cur, cpu);
diff |= CHECK(id_aa64mmfr1, boot, cur, cpu);
/*
* EL3 is not our concern.
* ID_AA64PFR1 is currently RES0.
*/
diff |= CHECK_MASK(id_aa64pfr0, 0xffffffffffff0fff, boot, cur, cpu);
diff |= CHECK(id_aa64pfr1, boot, cur, cpu);
/*
* If we have AArch32, we care about 32-bit features for compat. These
* registers should be RES0 otherwise.
*/
diff |= CHECK(id_isar0, boot, cur, cpu);
diff |= CHECK(id_isar1, boot, cur, cpu);
diff |= CHECK(id_isar2, boot, cur, cpu);
diff |= CHECK(id_isar3, boot, cur, cpu);
diff |= CHECK(id_isar4, boot, cur, cpu);
diff |= CHECK(id_isar5, boot, cur, cpu);
diff |= CHECK(id_mmfr0, boot, cur, cpu);
diff |= CHECK(id_mmfr1, boot, cur, cpu);
diff |= CHECK(id_mmfr2, boot, cur, cpu);
diff |= CHECK(id_mmfr3, boot, cur, cpu);
diff |= CHECK(id_pfr0, boot, cur, cpu);
diff |= CHECK(id_pfr1, boot, cur, cpu);
/*
* Mismatched CPU features are a recipe for disaster. Don't even
* pretend to support them.
*/
WARN_TAINT_ONCE(diff, TAINT_CPU_OUT_OF_SPEC,
"Unsupported CPU feature variation.");
}
static void __cpuinfo_store_cpu(struct cpuinfo_arm64 *info)
{
info->reg_cntfrq = arch_timer_get_cntfrq();
info->reg_ctr = read_cpuid_cachetype();
info->reg_dczid = read_cpuid(DCZID_EL0);
info->reg_midr = read_cpuid_id();
info->reg_id_aa64isar0 = read_cpuid(ID_AA64ISAR0_EL1);
info->reg_id_aa64isar1 = read_cpuid(ID_AA64ISAR1_EL1);
info->reg_id_aa64mmfr0 = read_cpuid(ID_AA64MMFR0_EL1);
info->reg_id_aa64mmfr1 = read_cpuid(ID_AA64MMFR1_EL1);
info->reg_id_aa64pfr0 = read_cpuid(ID_AA64PFR0_EL1);
info->reg_id_aa64pfr1 = read_cpuid(ID_AA64PFR1_EL1);
info->reg_id_isar0 = read_cpuid(ID_ISAR0_EL1);
info->reg_id_isar1 = read_cpuid(ID_ISAR1_EL1);
info->reg_id_isar2 = read_cpuid(ID_ISAR2_EL1);
info->reg_id_isar3 = read_cpuid(ID_ISAR3_EL1);
info->reg_id_isar4 = read_cpuid(ID_ISAR4_EL1);
info->reg_id_isar5 = read_cpuid(ID_ISAR5_EL1);
info->reg_id_mmfr0 = read_cpuid(ID_MMFR0_EL1);
info->reg_id_mmfr1 = read_cpuid(ID_MMFR1_EL1);
info->reg_id_mmfr2 = read_cpuid(ID_MMFR2_EL1);
info->reg_id_mmfr3 = read_cpuid(ID_MMFR3_EL1);
info->reg_id_pfr0 = read_cpuid(ID_PFR0_EL1);
info->reg_id_pfr1 = read_cpuid(ID_PFR1_EL1);
cpuinfo_detect_icache_policy(info);
}
void cpuinfo_store_cpu(void)
{
struct cpuinfo_arm64 *info = this_cpu_ptr(&cpu_data);
__cpuinfo_store_cpu(info);
cpuinfo_sanity_check(info);
}
void __init cpuinfo_store_boot_cpu(void)
{
struct cpuinfo_arm64 *info = &per_cpu(cpu_data, 0);
__cpuinfo_store_cpu(info);
boot_cpu_data = *info;
}
arch/arm64/kernel/debug-monitors.c
浏览文件 @ 5167d09f
......@@ -315,20 +315,20 @@ static int brk_handler(unsigned long addr, unsigned int esr,
{
siginfo_t info;
if (call_break_hook(regs, esr) == DBG_HOOK_HANDLED)
return 0;
if (user_mode(regs)) {
info = (siginfo_t) {
.si_signo = SIGTRAP,
.si_errno = 0,
.si_code = TRAP_BRKPT,
.si_addr = (void __user *)instruction_pointer(regs),
};
if (!user_mode(regs))
force_sig_info(SIGTRAP, &info, current);
} else if (call_break_hook(regs, esr) != DBG_HOOK_HANDLED) {
pr_warning("Unexpected kernel BRK exception at EL1\n");
return -EFAULT;
}
info = (siginfo_t) {
.si_signo = SIGTRAP,
.si_errno = 0,
.si_code = TRAP_BRKPT,
.si_addr = (void __user *)instruction_pointer(regs),
};
force_sig_info(SIGTRAP, &info, current);
return 0;
}
......
arch/arm64/kernel/entry-fpsimd.S
浏览文件 @ 5167d09f
......@@ -52,7 +52,7 @@ ENDPROC(fpsimd_load_state)
ENTRY(fpsimd_save_partial_state)
fpsimd_save_partial x0, 1, 8, 9
ret
ENDPROC(fpsimd_load_partial_state)
ENDPROC(fpsimd_save_partial_state)
/*
* Load the bottom n FP registers.
......
arch/arm64/kernel/entry.S
浏览文件 @ 5167d09f
......@@ -27,7 +27,32 @@
#include <asm/esr.h>
#include <asm/thread_info.h>
#include <asm/unistd.h>
#include <asm/unistd32.h>
/*
* Context tracking subsystem. Used to instrument transitions
* between user and kernel mode.
*/
.macro ct_user_exit, syscall = 0
#ifdef CONFIG_CONTEXT_TRACKING
bl context_tracking_user_exit
.if \syscall == 1
/*
* Save/restore needed during syscalls. Restore syscall arguments from
* the values already saved on stack during kernel_entry.
*/
ldp x0, x1, [sp]
ldp x2, x3, [sp, #S_X2]
ldp x4, x5, [sp, #S_X4]
ldp x6, x7, [sp, #S_X6]
.endif
#endif
.endm
.macro ct_user_enter
#ifdef CONFIG_CONTEXT_TRACKING
bl context_tracking_user_enter
#endif
.endm
/*
* Bad Abort numbers
......@@ -91,6 +116,7 @@
.macro kernel_exit, el, ret = 0
ldp x21, x22, [sp, #S_PC] // load ELR, SPSR
.if \el == 0
ct_user_enter
ldr x23, [sp, #S_SP] // load return stack pointer
.endif
.if \ret
......@@ -353,7 +379,6 @@ el0_sync:
lsr x24, x25, #ESR_EL1_EC_SHIFT // exception class
cmp x24, #ESR_EL1_EC_SVC64 // SVC in 64-bit state
b.eq el0_svc
adr lr, ret_to_user
cmp x24, #ESR_EL1_EC_DABT_EL0 // data abort in EL0
b.eq el0_da
cmp x24, #ESR_EL1_EC_IABT_EL0 // instruction abort in EL0
......@@ -382,7 +407,6 @@ el0_sync_compat:
lsr x24, x25, #ESR_EL1_EC_SHIFT // exception class
cmp x24, #ESR_EL1_EC_SVC32 // SVC in 32-bit state
b.eq el0_svc_compat
adr lr, ret_to_user
cmp x24, #ESR_EL1_EC_DABT_EL0 // data abort in EL0
b.eq el0_da
cmp x24, #ESR_EL1_EC_IABT_EL0 // instruction abort in EL0
......@@ -425,48 +449,59 @@ el0_da:
/*
* Data abort handling
*/
mrs x0, far_el1
bic x0, x0, #(0xff << 56)
mrs x26, far_el1
// enable interrupts before calling the main handler
enable_dbg_and_irq
ct_user_exit
bic x0, x26, #(0xff << 56)
mov x1, x25
mov x2, sp
adr lr, ret_to_user
b do_mem_abort
el0_ia:
/*
* Instruction abort handling
*/
mrs x0, far_el1
mrs x26, far_el1
// enable interrupts before calling the main handler
enable_dbg_and_irq
ct_user_exit
mov x0, x26
orr x1, x25, #1 << 24 // use reserved ISS bit for instruction aborts
mov x2, sp
adr lr, ret_to_user
b do_mem_abort
el0_fpsimd_acc:
/*
* Floating Point or Advanced SIMD access
*/
enable_dbg
ct_user_exit
mov x0, x25
mov x1, sp
adr lr, ret_to_user
b do_fpsimd_acc
el0_fpsimd_exc:
/*
* Floating Point or Advanced SIMD exception
*/
enable_dbg
ct_user_exit
mov x0, x25
mov x1, sp
adr lr, ret_to_user
b do_fpsimd_exc
el0_sp_pc:
/*
* Stack or PC alignment exception handling
*/
mrs x0, far_el1
mrs x26, far_el1
// enable interrupts before calling the main handler
enable_dbg_and_irq
mov x0, x26
mov x1, x25
mov x2, sp
adr lr, ret_to_user
b do_sp_pc_abort
el0_undef:
/*
......@@ -474,7 +509,9 @@ el0_undef:
*/
// enable interrupts before calling the main handler
enable_dbg_and_irq
ct_user_exit
mov x0, sp
adr lr, ret_to_user
b do_undefinstr
el0_dbg:
/*
......@@ -486,12 +523,15 @@ el0_dbg:
mov x2, sp
bl do_debug_exception
enable_dbg
ct_user_exit
b ret_to_user
el0_inv:
enable_dbg
ct_user_exit
mov x0, sp
mov x1, #BAD_SYNC
mrs x2, esr_el1
adr lr, ret_to_user
b bad_mode
ENDPROC(el0_sync)
......@@ -504,6 +544,7 @@ el0_irq_naked:
bl trace_hardirqs_off
#endif
ct_user_exit
irq_handler
#ifdef CONFIG_TRACE_IRQFLAGS
......@@ -608,6 +649,7 @@ el0_svc:
el0_svc_naked: // compat entry point
stp x0, scno, [sp, #S_ORIG_X0] // save the original x0 and syscall number
enable_dbg_and_irq
ct_user_exit 1
ldr x16, [tsk, #TI_FLAGS] // check for syscall hooks
tst x16, #_TIF_SYSCALL_WORK
......
arch/arm64/kernel/head.S
浏览文件 @ 5167d09f
......@@ -22,6 +22,7 @@
#include <linux/linkage.h>
#include <linux/init.h>
#include <linux/irqchip/arm-gic-v3.h>
#include <asm/assembler.h>
#include <asm/ptrace.h>
......@@ -35,37 +36,31 @@
#include <asm/page.h>
#include <asm/virt.h>
/*
* swapper_pg_dir is the virtual address of the initial page table. We place
* the page tables 3 * PAGE_SIZE below KERNEL_RAM_VADDR. The idmap_pg_dir has
* 2 pages and is placed below swapper_pg_dir.
*/
#define KERNEL_RAM_VADDR (PAGE_OFFSET + TEXT_OFFSET)
#if (KERNEL_RAM_VADDR & 0xfffff) != 0x80000
#error KERNEL_RAM_VADDR must start at 0xXXX80000
#if (TEXT_OFFSET & 0xf) != 0
#error TEXT_OFFSET must be at least 16B aligned
#elif (PAGE_OFFSET & 0xfffff) != 0
#error PAGE_OFFSET must be at least 2MB aligned
#elif TEXT_OFFSET > 0xfffff
#error TEXT_OFFSET must be less than 2MB
#endif
#define SWAPPER_DIR_SIZE (3 * PAGE_SIZE)
#define IDMAP_DIR_SIZE (2 * PAGE_SIZE)
.globl swapper_pg_dir
.equ swapper_pg_dir, KERNEL_RAM_VADDR - SWAPPER_DIR_SIZE
.globl idmap_pg_dir
.equ idmap_pg_dir, swapper_pg_dir - IDMAP_DIR_SIZE
.macro pgtbl, ttb0, ttb1, phys
add \ttb1, \phys, #TEXT_OFFSET - SWAPPER_DIR_SIZE
sub \ttb0, \ttb1, #IDMAP_DIR_SIZE
.macro pgtbl, ttb0, ttb1, virt_to_phys
ldr \ttb1, =swapper_pg_dir
ldr \ttb0, =idmap_pg_dir
add \ttb1, \ttb1, \virt_to_phys
add \ttb0, \ttb0, \virt_to_phys
.endm
#ifdef CONFIG_ARM64_64K_PAGES
#define BLOCK_SHIFT PAGE_SHIFT
#define BLOCK_SIZE PAGE_SIZE
#define TABLE_SHIFT PMD_SHIFT
#else
#define BLOCK_SHIFT SECTION_SHIFT
#define BLOCK_SIZE SECTION_SIZE
#define TABLE_SHIFT PUD_SHIFT
#endif
#define KERNEL_START KERNEL_RAM_VADDR
......@@ -120,9 +115,9 @@ efi_head:
b stext // branch to kernel start, magic
.long 0 // reserved
#endif
.quad TEXT_OFFSET // Image load offset from start of RAM
.quad 0 // reserved
.quad 0 // reserved
.quad _kernel_offset_le // Image load offset from start of RAM, little-endian
.quad _kernel_size_le // Effective size of kernel image, little-endian
.quad _kernel_flags_le // Informative flags, little-endian
.quad 0 // reserved
.quad 0 // reserved
.quad 0 // reserved
......@@ -295,6 +290,23 @@ CPU_LE( bic x0, x0, #(3 << 24) ) // Clear the EE and E0E bits for EL1
msr cnthctl_el2, x0
msr cntvoff_el2, xzr // Clear virtual offset
#ifdef CONFIG_ARM_GIC_V3
/* GICv3 system register access */
mrs x0, id_aa64pfr0_el1
ubfx x0, x0, #24, #4
cmp x0, #1
b.ne 3f
mrs_s x0, ICC_SRE_EL2
orr x0, x0, #ICC_SRE_EL2_SRE // Set ICC_SRE_EL2.SRE==1
orr x0, x0, #ICC_SRE_EL2_ENABLE // Set ICC_SRE_EL2.Enable==1
msr_s ICC_SRE_EL2, x0
isb // Make sure SRE is now set
msr_s ICH_HCR_EL2, xzr // Reset ICC_HCR_EL2 to defaults
3:
#endif
/* Populate ID registers. */
mrs x0, midr_el1
mrs x1, mpidr_el1
......@@ -413,7 +425,7 @@ ENTRY(secondary_startup)
mov x23, x0 // x23=current cpu_table
cbz x23, __error_p // invalid processor (x23=0)?
pgtbl x25, x26, x24 // x25=TTBR0, x26=TTBR1
pgtbl x25, x26, x28 // x25=TTBR0, x26=TTBR1
ldr x12, [x23, #CPU_INFO_SETUP]
add x12, x12, x28 // __virt_to_phys
blr x12 // initialise processor
......@@ -455,8 +467,13 @@ ENDPROC(__enable_mmu)
* x27 = *virtual* address to jump to upon completion
*
* other registers depend on the function called upon completion
*
* We align the entire function to the smallest power of two larger than it to
* ensure it fits within a single block map entry. Otherwise were PHYS_OFFSET
* close to the end of a 512MB or 1GB block we might require an additional
* table to map the entire function.
*/
.align 6
.align 4
__turn_mmu_on:
msr sctlr_el1, x0
isb
......@@ -479,17 +496,38 @@ ENDPROC(__calc_phys_offset)
.quad PAGE_OFFSET
/*
* Macro to populate the PGD for the corresponding block entry in the next
* level (tbl) for the given virtual address.
* Macro to create a table entry to the next page.
*
* tbl: page table address
* virt: virtual address
* shift: #imm page table shift
* ptrs: #imm pointers per table page
*
* Preserves: virt
* Corrupts: tmp1, tmp2
* Returns: tbl -> next level table page address
*/
.macro create_table_entry, tbl, virt, shift, ptrs, tmp1, tmp2
lsr \tmp1, \virt, #\shift
and \tmp1, \tmp1, #\ptrs - 1 // table index
add \tmp2, \tbl, #PAGE_SIZE
orr \tmp2, \tmp2, #PMD_TYPE_TABLE // address of next table and entry type
str \tmp2, [\tbl, \tmp1, lsl #3]
add \tbl, \tbl, #PAGE_SIZE // next level table page
.endm
/*
* Macro to populate the PGD (and possibily PUD) for the corresponding
* block entry in the next level (tbl) for the given virtual address.
*
* Preserves: pgd, tbl, virt
* Preserves: tbl, next, virt
* Corrupts: tmp1, tmp2
*/
.macro create_pgd_entry, pgd, tbl, virt, tmp1, tmp2
lsr \tmp1, \virt, #PGDIR_SHIFT
and \tmp1, \tmp1, #PTRS_PER_PGD - 1 // PGD index
orr \tmp2, \tbl, #3 // PGD entry table type
str \tmp2, [\pgd, \tmp1, lsl #3]
.macro create_pgd_entry, tbl, virt, tmp1, tmp2
create_table_entry \tbl, \virt, PGDIR_SHIFT, PTRS_PER_PGD, \tmp1, \tmp2
#if SWAPPER_PGTABLE_LEVELS == 3
create_table_entry \tbl, \virt, TABLE_SHIFT, PTRS_PER_PTE, \tmp1, \tmp2
#endif
.endm
/*
......@@ -522,7 +560,7 @@ ENDPROC(__calc_phys_offset)
* - pgd entry for fixed mappings (TTBR1)
*/
__create_page_tables:
pgtbl x25, x26, x24 // idmap_pg_dir and swapper_pg_dir addresses
pgtbl x25, x26, x28 // idmap_pg_dir and swapper_pg_dir addresses
mov x27, lr
/*
......@@ -550,10 +588,10 @@ __create_page_tables:
/*
* Create the identity mapping.
*/
add x0, x25, #PAGE_SIZE // section table address
mov x0, x25 // idmap_pg_dir
ldr x3, =KERNEL_START
add x3, x3, x28 // __pa(KERNEL_START)
create_pgd_entry x25, x0, x3, x5, x6
create_pgd_entry x0, x3, x5, x6
ldr x6, =KERNEL_END
mov x5, x3 // __pa(KERNEL_START)
add x6, x6, x28 // __pa(KERNEL_END)
......@@ -562,9 +600,9 @@ __create_page_tables:
/*
* Map the kernel image (starting with PHYS_OFFSET).
*/
add x0, x26, #PAGE_SIZE // section table address
mov x0, x26 // swapper_pg_dir
mov x5, #PAGE_OFFSET
create_pgd_entry x26, x0, x5, x3, x6
create_pgd_entry x0, x5, x3, x6
ldr x6, =KERNEL_END
mov x3, x24 // phys offset
create_block_map x0, x7, x3, x5, x6
......@@ -585,13 +623,6 @@ __create_page_tables:
sub x6, x6, #1 // inclusive range
create_block_map x0, x7, x3, x5, x6
1:
/*
* Create the pgd entry for the fixed mappings.
*/
ldr x5, =FIXADDR_TOP // Fixed mapping virtual address
add x0, x26, #2 * PAGE_SIZE // section table address
create_pgd_entry x26, x0, x5, x6, x7
/*
* Since the page tables have been populated with non-cacheable
* accesses (MMU disabled), invalidate the idmap and swapper page
......@@ -611,7 +642,7 @@ ENDPROC(__create_page_tables)
__switch_data:
.quad __mmap_switched
.quad __bss_start // x6
.quad _end // x7
.quad __bss_stop // x7
.quad processor_id // x4
.quad __fdt_pointer // x5
.quad memstart_addr // x6
......
arch/arm64/kernel/hyp-stub.S
浏览文件 @ 5167d09f
......@@ -19,6 +19,7 @@
#include <linux/init.h>
#include <linux/linkage.h>
#include <linux/irqchip/arm-gic-v3.h>
#include <asm/assembler.h>
#include <asm/ptrace.h>
......
arch/arm64/kernel/image.h 0 → 100644
浏览文件 @ 5167d09f
/*
* Linker script macros to generate Image header fields.
*
* Copyright (C) 2014 ARM Ltd.
*
* This program is free software; you can redistribute it and/or modify
* it under the terms of the GNU General Public License version 2 as
* published by the Free Software Foundation.
*
* This program is distributed in the hope that it will be useful,
* but WITHOUT ANY WARRANTY; without even the implied warranty of
* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
* GNU General Public License for more details.
*
* You should have received a copy of the GNU General Public License
* along with this program. If not, see <http://www.gnu.org/licenses/>.
*/
#ifndef __ASM_IMAGE_H
#define __ASM_IMAGE_H
#ifndef LINKER_SCRIPT
#error This file should only be included in vmlinux.lds.S
#endif
/*
* There aren't any ELF relocations we can use to endian-swap values known only
* at link time (e.g. the subtraction of two symbol addresses), so we must get
* the linker to endian-swap certain values before emitting them.
*/
#ifdef CONFIG_CPU_BIG_ENDIAN
#define DATA_LE64(data) \
((((data) & 0x00000000000000ff) << 56) | \
(((data) & 0x000000000000ff00) << 40) | \
(((data) & 0x0000000000ff0000) << 24) | \
(((data) & 0x00000000ff000000) << 8) | \
(((data) & 0x000000ff00000000) >> 8) | \
(((data) & 0x0000ff0000000000) >> 24) | \
(((data) & 0x00ff000000000000) >> 40) | \
(((data) & 0xff00000000000000) >> 56))
#else
#define DATA_LE64(data) ((data) & 0xffffffffffffffff)
#endif
#ifdef CONFIG_CPU_BIG_ENDIAN
#define __HEAD_FLAG_BE 1
#else
#define __HEAD_FLAG_BE 0
#endif
#define __HEAD_FLAGS (__HEAD_FLAG_BE << 0)
/*
* These will output as part of the Image header, which should be little-endian
* regardless of the endianness of the kernel. While constant values could be
* endian swapped in head.S, all are done here for consistency.
*/
#define HEAD_SYMBOLS \
_kernel_size_le = DATA_LE64(_end - _text); \
_kernel_offset_le = DATA_LE64(TEXT_OFFSET); \
_kernel_flags_le = DATA_LE64(__HEAD_FLAGS);
#endif /* __ASM_IMAGE_H */
arch/arm64/kernel/kuser32.S
浏览文件 @ 5167d09f
......@@ -28,7 +28,7 @@
* See Documentation/arm/kernel_user_helpers.txt for formal definitions.
*/
#include <asm/unistd32.h>
#include <asm/unistd.h>
.align 5
.globl __kuser_helper_start
......
arch/arm64/kernel/process.c
浏览文件 @ 5167d09f
......@@ -51,6 +51,12 @@
#include <asm/processor.h>
#include <asm/stacktrace.h>
#ifdef CONFIG_CC_STACKPROTECTOR
#include <linux/stackprotector.h>
unsigned long __stack_chk_guard __read_mostly;
EXPORT_SYMBOL(__stack_chk_guard);
#endif
static void setup_restart(void)
{
/*
......
arch/arm64/kernel/psci.c
浏览文件 @ 5167d09f
......@@ -235,7 +235,7 @@ static void psci_sys_poweroff(void)
* PSCI Function IDs for v0.2+ are well defined so use
* standard values.
*/
static int psci_0_2_init(struct device_node *np)
static int __init psci_0_2_init(struct device_node *np)
{
int err, ver;
......@@ -296,7 +296,7 @@ static int psci_0_2_init(struct device_node *np)
/*
* PSCI < v0.2 get PSCI Function IDs via DT.
*/
static int psci_0_1_init(struct device_node *np)
static int __init psci_0_1_init(struct device_node *np)
{
u32 id;
int err;
......@@ -434,9 +434,11 @@ static int cpu_psci_cpu_kill(unsigned int cpu)
return 0;
}
#endif
#endif
const struct cpu_operations cpu_psci_ops = {
.name = "psci",
#ifdef CONFIG_SMP
.cpu_init = cpu_psci_cpu_init,
.cpu_prepare = cpu_psci_cpu_prepare,
.cpu_boot = cpu_psci_cpu_boot,
......@@ -445,6 +447,6 @@ const struct cpu_operations cpu_psci_ops = {
.cpu_die = cpu_psci_cpu_die,
.cpu_kill = cpu_psci_cpu_kill,
#endif
#endif
};
#endif
arch/arm64/kernel/ptrace.c
浏览文件 @ 5167d09f
......@@ -19,6 +19,7 @@
* along with this program. If not, see <http://www.gnu.org/licenses/>.
*/
#include <linux/audit.h>
#include <linux/compat.h>
#include <linux/kernel.h>
#include <linux/sched.h>
......@@ -39,6 +40,7 @@
#include <asm/compat.h>
#include <asm/debug-monitors.h>
#include <asm/pgtable.h>
#include <asm/syscall.h>
#include <asm/traps.h>
#include <asm/system_misc.h>
......@@ -1113,11 +1115,20 @@ asmlinkage int syscall_trace_enter(struct pt_regs *regs)
if (test_thread_flag(TIF_SYSCALL_TRACEPOINT))
trace_sys_enter(regs, regs->syscallno);
#ifdef CONFIG_AUDITSYSCALL
audit_syscall_entry(syscall_get_arch(), regs->syscallno,
regs->orig_x0, regs->regs[1], regs->regs[2], regs->regs[3]);
#endif
return regs->syscallno;
}
asmlinkage void syscall_trace_exit(struct pt_regs *regs)
{
#ifdef CONFIG_AUDITSYSCALL
audit_syscall_exit(regs);
#endif
if (test_thread_flag(TIF_SYSCALL_TRACEPOINT))
trace_sys_exit(regs, regs_return_value(regs));
......
arch/arm64/kernel/setup.c
浏览文件 @ 5167d09f
......@@ -45,6 +45,7 @@
#include <linux/efi.h>
#include <asm/fixmap.h>
#include <asm/cpu.h>
#include <asm/cputype.h>
#include <asm/elf.h>
#include <asm/cputable.h>
......@@ -77,7 +78,6 @@ unsigned int compat_elf_hwcap2 __read_mostly;
#endif
static const char *cpu_name;
static const char *machine_name;
phys_addr_t __fdt_pointer __initdata;
/*
......@@ -219,6 +219,8 @@ static void __init setup_processor(void)
sprintf(init_utsname()->machine, ELF_PLATFORM);
elf_hwcap = 0;
cpuinfo_store_boot_cpu();
/*
* Check for sane CTR_EL0.CWG value.
*/
......@@ -307,8 +309,6 @@ static void __init setup_machine_fdt(phys_addr_t dt_phys)
while (true)
cpu_relax();
}
machine_name = of_flat_dt_get_machine_name();
}
/*
......@@ -417,14 +417,12 @@ static int __init arm64_device_init(void)
}
arch_initcall_sync(arm64_device_init);
static DEFINE_PER_CPU(struct cpu, cpu_data);
static int __init topology_init(void)
{
int i;
for_each_possible_cpu(i) {
struct cpu *cpu = &per_cpu(cpu_data, i);
struct cpu *cpu = &per_cpu(cpu_data.cpu, i);
cpu->hotpluggable = 1;
register_cpu(cpu, i);
}
......@@ -449,10 +447,21 @@ static int c_show(struct seq_file *m, void *v)
{
int i;
seq_printf(m, "Processor\t: %s rev %d (%s)\n",
cpu_name, read_cpuid_id() & 15, ELF_PLATFORM);
/*
* Dump out the common processor features in a single line. Userspace
* should read the hwcaps with getauxval(AT_HWCAP) rather than
* attempting to parse this.
*/
seq_puts(m, "features\t:");
for (i = 0; hwcap_str[i]; i++)
if (elf_hwcap & (1 << i))
seq_printf(m, " %s", hwcap_str[i]);
seq_puts(m, "\n\n");
for_each_online_cpu(i) {
struct cpuinfo_arm64 *cpuinfo = &per_cpu(cpu_data, i);
u32 midr = cpuinfo->reg_midr;
/*
* glibc reads /proc/cpuinfo to determine the number of
* online processors, looking for lines beginning with
......@@ -461,25 +470,13 @@ static int c_show(struct seq_file *m, void *v)
#ifdef CONFIG_SMP
seq_printf(m, "processor\t: %d\n", i);
#endif
seq_printf(m, "implementer\t: 0x%02x\n",
MIDR_IMPLEMENTOR(midr));
seq_printf(m, "variant\t\t: 0x%x\n", MIDR_VARIANT(midr));
seq_printf(m, "partnum\t\t: 0x%03x\n", MIDR_PARTNUM(midr));
seq_printf(m, "revision\t: 0x%x\n\n", MIDR_REVISION(midr));
}
/* dump out the processor features */
seq_puts(m, "Features\t: ");
for (i = 0; hwcap_str[i]; i++)
if (elf_hwcap & (1 << i))
seq_printf(m, "%s ", hwcap_str[i]);
seq_printf(m, "\nCPU implementer\t: 0x%02x\n", read_cpuid_id() >> 24);
seq_printf(m, "CPU architecture: AArch64\n");
seq_printf(m, "CPU variant\t: 0x%x\n", (read_cpuid_id() >> 20) & 15);
seq_printf(m, "CPU part\t: 0x%03x\n", (read_cpuid_id() >> 4) & 0xfff);
seq_printf(m, "CPU revision\t: %d\n", read_cpuid_id() & 15);
seq_puts(m, "\n");
seq_printf(m, "Hardware\t: %s\n", machine_name);
return 0;
}
......
arch/arm64/kernel/signal32.c
浏览文件 @ 5167d09f
......@@ -27,7 +27,7 @@
#include <asm/fpsimd.h>
#include <asm/signal32.h>
#include <asm/uaccess.h>
#include <asm/unistd32.h>
#include <asm/unistd.h>
struct compat_sigcontext {
/* We always set these two fields to 0 */
......
arch/arm64/kernel/smp.c
浏览文件 @ 5167d09f
......@@ -39,6 +39,7 @@
#include <asm/atomic.h>
#include <asm/cacheflush.h>
#include <asm/cpu.h>
#include <asm/cputype.h>
#include <asm/cpu_ops.h>
#include <asm/mmu_context.h>
......@@ -154,6 +155,11 @@ asmlinkage void secondary_start_kernel(void)
if (cpu_ops[cpu]->cpu_postboot)
cpu_ops[cpu]->cpu_postboot();
/*
* Log the CPU info before it is marked online and might get read.
*/
cpuinfo_store_cpu();
/*
* Enable GIC and timers.
*/
......
arch/arm64/kernel/suspend.c
浏览文件 @ 5167d09f
......@@ -119,7 +119,7 @@ int cpu_suspend(unsigned long arg)
extern struct sleep_save_sp sleep_save_sp;
extern phys_addr_t sleep_idmap_phys;
static int cpu_suspend_init(void)
static int __init cpu_suspend_init(void)
{
void *ctx_ptr;
......
arch/arm64/kernel/sys_compat.c
浏览文件 @ 5167d09f
......@@ -26,7 +26,7 @@
#include <linux/uaccess.h>
#include <asm/cacheflush.h>
#include <asm/unistd32.h>
#include <asm/unistd.h>
static inline void
do_compat_cache_op(unsigned long start, unsigned long end, int flags)
......
arch/arm64/kernel/topology.c
浏览文件 @ 5167d09f
......@@ -20,6 +20,7 @@
#include <linux/of.h>
#include <linux/sched.h>
#include <asm/cputype.h>
#include <asm/topology.h>
static int __init get_cpu_for_node(struct device_node *node)
......@@ -188,13 +189,9 @@ static int __init parse_dt_topology(void)
* Check that all cores are in the topology; the SMP code will
* only mark cores described in the DT as possible.
*/
for_each_possible_cpu(cpu) {
if (cpu_topology[cpu].cluster_id == -1) {
pr_err("CPU%d: No topology information specified\n",
cpu);
for_each_possible_cpu(cpu)
if (cpu_topology[cpu].cluster_id == -1)
ret = -EINVAL;
}
}
out_map:
of_node_put(map);
......@@ -219,14 +216,6 @@ static void update_siblings_masks(unsigned int cpuid)
struct cpu_topology *cpu_topo, *cpuid_topo = &cpu_topology[cpuid];
int cpu;
if (cpuid_topo->cluster_id == -1) {
/*
* DT does not contain topology information for this cpu.
*/
pr_debug("CPU%u: No topology information configured\n", cpuid);
return;
}
/* update core and thread sibling masks */
for_each_possible_cpu(cpu) {
cpu_topo = &cpu_topology[cpu];
......@@ -249,6 +238,36 @@ static void update_siblings_masks(unsigned int cpuid)
void store_cpu_topology(unsigned int cpuid)
{
struct cpu_topology *cpuid_topo = &cpu_topology[cpuid];
u64 mpidr;
if (cpuid_topo->cluster_id != -1)
goto topology_populated;
mpidr = read_cpuid_mpidr();
/* Uniprocessor systems can rely on default topology values */
if (mpidr & MPIDR_UP_BITMASK)
return;
/* Create cpu topology mapping based on MPIDR. */
if (mpidr & MPIDR_MT_BITMASK) {
/* Multiprocessor system : Multi-threads per core */
cpuid_topo->thread_id = MPIDR_AFFINITY_LEVEL(mpidr, 0);
cpuid_topo->core_id = MPIDR_AFFINITY_LEVEL(mpidr, 1);
cpuid_topo->cluster_id = MPIDR_AFFINITY_LEVEL(mpidr, 2);
} else {
/* Multiprocessor system : Single-thread per core */
cpuid_topo->thread_id = -1;
cpuid_topo->core_id = MPIDR_AFFINITY_LEVEL(mpidr, 0);
cpuid_topo->cluster_id = MPIDR_AFFINITY_LEVEL(mpidr, 1);
}
pr_debug("CPU%u: cluster %d core %d thread %d mpidr %#016llx\n",
cpuid, cpuid_topo->cluster_id, cpuid_topo->core_id,
cpuid_topo->thread_id, mpidr);
topology_populated:
update_siblings_masks(cpuid);
}
......
arch/arm64/kernel/traps.c
浏览文件 @ 5167d09f
......@@ -156,7 +156,7 @@ static void dump_backtrace(struct pt_regs *regs, struct task_struct *tsk)
frame.pc = thread_saved_pc(tsk);
}
printk("Call trace:\n");
pr_emerg("Call trace:\n");
while (1) {
unsigned long where = frame.pc;
int ret;
......@@ -331,17 +331,22 @@ asmlinkage void bad_mode(struct pt_regs *regs, int reason, unsigned int esr)
void __pte_error(const char *file, int line, unsigned long val)
{
printk("%s:%d: bad pte %016lx.\n", file, line, val);
pr_crit("%s:%d: bad pte %016lx.\n", file, line, val);
}
void __pmd_error(const char *file, int line, unsigned long val)
{
printk("%s:%d: bad pmd %016lx.\n", file, line, val);
pr_crit("%s:%d: bad pmd %016lx.\n", file, line, val);
}
void __pud_error(const char *file, int line, unsigned long val)
{
pr_crit("%s:%d: bad pud %016lx.\n", file, line, val);
}
void __pgd_error(const char *file, int line, unsigned long val)
{
printk("%s:%d: bad pgd %016lx.\n", file, line, val);
pr_crit("%s:%d: bad pgd %016lx.\n", file, line, val);
}
void __init trap_init(void)
......
arch/arm64/kernel/vdso.c
浏览文件 @ 5167d09f
......@@ -88,22 +88,29 @@ int aarch32_setup_vectors_page(struct linux_binprm *bprm, int uses_interp)
{
struct mm_struct *mm = current->mm;
unsigned long addr = AARCH32_VECTORS_BASE;
int ret;
static struct vm_special_mapping spec = {
.name = "[vectors]",
.pages = vectors_page,
};
void *ret;
down_write(&mm->mmap_sem);
current->mm->context.vdso = (void *)addr;
/* Map vectors page at the high address. */
ret = install_special_mapping(mm, addr, PAGE_SIZE,
VM_READ|VM_EXEC|VM_MAYREAD|VM_MAYEXEC,
vectors_page);
ret = _install_special_mapping(mm, addr, PAGE_SIZE,
VM_READ|VM_EXEC|VM_MAYREAD|VM_MAYEXEC,
&spec);
up_write(&mm->mmap_sem);
return ret;
return PTR_ERR_OR_ZERO(ret);
}
#endif /* CONFIG_COMPAT */
static struct vm_special_mapping vdso_spec[2];
static int __init vdso_init(void)
{
int i;
......@@ -114,8 +121,8 @@ static int __init vdso_init(void)
}
vdso_pages = (&vdso_end - &vdso_start) >> PAGE_SHIFT;
pr_info("vdso: %ld pages (%ld code, %ld data) at base %p\n",
vdso_pages + 1, vdso_pages, 1L, &vdso_start);
pr_info("vdso: %ld pages (%ld code @ %p, %ld data @ %p)\n",
vdso_pages + 1, vdso_pages, &vdso_start, 1L, vdso_data);
/* Allocate the vDSO pagelist, plus a page for the data. */
vdso_pagelist = kcalloc(vdso_pages + 1, sizeof(struct page *),
......@@ -123,12 +130,23 @@ static int __init vdso_init(void)
if (vdso_pagelist == NULL)
return -ENOMEM;
/* Grab the vDSO data page. */
vdso_pagelist[0] = virt_to_page(vdso_data);
/* Grab the vDSO code pages. */
for (i = 0; i < vdso_pages; i++)
vdso_pagelist[i] = virt_to_page(&vdso_start + i * PAGE_SIZE);
vdso_pagelist[i + 1] = virt_to_page(&vdso_start + i * PAGE_SIZE);
/* Grab the vDSO data page. */
vdso_pagelist[i] = virt_to_page(vdso_data);
/* Populate the special mapping structures */
vdso_spec[0] = (struct vm_special_mapping) {
.name = "[vvar]",
.pages = vdso_pagelist,
};
vdso_spec[1] = (struct vm_special_mapping) {
.name = "[vdso]",
.pages = &vdso_pagelist[1],
};
return 0;
}
......@@ -138,52 +156,42 @@ int arch_setup_additional_pages(struct linux_binprm *bprm,
int uses_interp)
{
struct mm_struct *mm = current->mm;
unsigned long vdso_base, vdso_mapping_len;
int ret;
unsigned long vdso_base, vdso_text_len, vdso_mapping_len;
void *ret;
vdso_text_len = vdso_pages << PAGE_SHIFT;
/* Be sure to map the data page */
vdso_mapping_len = (vdso_pages + 1) << PAGE_SHIFT;
vdso_mapping_len = vdso_text_len + PAGE_SIZE;
down_write(&mm->mmap_sem);
vdso_base = get_unmapped_area(NULL, 0, vdso_mapping_len, 0, 0);
if (IS_ERR_VALUE(vdso_base)) {
ret = vdso_base;
ret = ERR_PTR(vdso_base);
goto up_fail;
}
mm->context.vdso = (void *)vdso_base;
ret = install_special_mapping(mm, vdso_base, vdso_mapping_len,
VM_READ|VM_EXEC|
VM_MAYREAD|VM_MAYWRITE|VM_MAYEXEC,
vdso_pagelist);
if (ret) {
mm->context.vdso = NULL;
ret = _install_special_mapping(mm, vdso_base, PAGE_SIZE,
VM_READ|VM_MAYREAD,
&vdso_spec[0]);
if (IS_ERR(ret))
goto up_fail;
}
up_fail:
up_write(&mm->mmap_sem);
vdso_base += PAGE_SIZE;
mm->context.vdso = (void *)vdso_base;
ret = _install_special_mapping(mm, vdso_base, vdso_text_len,
VM_READ|VM_EXEC|
VM_MAYREAD|VM_MAYWRITE|VM_MAYEXEC,
&vdso_spec[1]);
if (IS_ERR(ret))
goto up_fail;
return ret;
}
const char *arch_vma_name(struct vm_area_struct *vma)
{
/*
* We can re-use the vdso pointer in mm_context_t for identifying
* the vectors page for compat applications. The vDSO will always
* sit above TASK_UNMAPPED_BASE and so we don't need to worry about
* it conflicting with the vectors base.
*/
if (vma->vm_mm && vma->vm_start == (long)vma->vm_mm->context.vdso) {
#ifdef CONFIG_COMPAT
if (vma->vm_start == AARCH32_VECTORS_BASE)
return "[vectors]";
#endif
return "[vdso]";
}
up_write(&mm->mmap_sem);
return 0;
return NULL;
up_fail:
mm->context.vdso = NULL;
up_write(&mm->mmap_sem);
return PTR_ERR(ret);
}
/*
......
arch/arm64/kernel/vdso/Makefile
浏览文件 @ 5167d09f
......@@ -43,13 +43,13 @@ $(obj)/vdso-offsets.h: $(obj)/vdso.so.dbg FORCE
$(call if_changed,vdsosym)
# Assembly rules for the .S files
$(obj-vdso): %.o: %.S
$(obj-vdso): %.o: %.S FORCE
$(call if_changed_dep,vdsoas)
# Actual build commands
quiet_cmd_vdsold = VDSOL $@
quiet_cmd_vdsold = VDSOL $@
cmd_vdsold = $(CC) $(c_flags) -Wl,-n -Wl,-T $^ -o $@
quiet_cmd_vdsoas = VDSOA $@
quiet_cmd_vdsoas = VDSOA $@
cmd_vdsoas = $(CC) $(a_flags) -c -o $@ $<
# Install commands for the unstripped file
......
arch/arm64/kernel/vdso/vdso.lds.S
浏览文件 @ 5167d09f
......@@ -28,6 +28,7 @@ OUTPUT_ARCH(aarch64)
SECTIONS
{
PROVIDE(_vdso_data = . - PAGE_SIZE);
. = VDSO_LBASE + SIZEOF_HEADERS;
.hash : { *(.hash) } :text
......@@ -57,9 +58,6 @@ SECTIONS
_end = .;
PROVIDE(end = .);
. = ALIGN(PAGE_SIZE);
PROVIDE(_vdso_data = .);
/DISCARD/ : {
*(.note.GNU-stack)
*(.data .data.* .gnu.linkonce.d.* .sdata*)
......
arch/arm64/kernel/vmlinux.lds.S
浏览文件 @ 5167d09f
......@@ -9,6 +9,8 @@
#include <asm/memory.h>
#include <asm/page.h>
#include "image.h"
#define ARM_EXIT_KEEP(x)
#define ARM_EXIT_DISCARD(x) x
......@@ -104,9 +106,18 @@ SECTIONS
_edata = .;
BSS_SECTION(0, 0, 0)
. = ALIGN(PAGE_SIZE);
idmap_pg_dir = .;
. += IDMAP_DIR_SIZE;
swapper_pg_dir = .;
. += SWAPPER_DIR_SIZE;
_end = .;
STABS_DEBUG
HEAD_SYMBOLS
}
/*
......@@ -114,3 +125,8 @@ SECTIONS
*/
ASSERT(((__hyp_idmap_text_start + PAGE_SIZE) > __hyp_idmap_text_end),
"HYP init code too big")
/*
* If padding is applied before .head.text, virt<->phys conversions will fail.
*/
ASSERT(_text == (PAGE_OFFSET + TEXT_OFFSET), "HEAD is misaligned")
arch/arm64/mm/fault.c
浏览文件 @ 5167d09f
......@@ -62,6 +62,7 @@ void show_pte(struct mm_struct *mm, unsigned long addr)
break;
pud = pud_offset(pgd, addr);
printk(", *pud=%016llx", pud_val(*pud));
if (pud_none(*pud) || pud_bad(*pud))
break;
......
arch/arm64/mm/init.c
浏览文件 @ 5167d09f
......@@ -33,6 +33,7 @@
#include <linux/dma-mapping.h>
#include <linux/dma-contiguous.h>
#include <asm/fixmap.h>
#include <asm/sections.h>
#include <asm/setup.h>
#include <asm/sizes.h>
......@@ -137,20 +138,16 @@ void __init arm64_memblock_init(void)
{
phys_addr_t dma_phys_limit = 0;
/* Register the kernel text, kernel data and initrd with memblock */
/*
* Register the kernel text, kernel data, initrd, and initial
* pagetables with memblock.
*/
memblock_reserve(__pa(_text), _end - _text);
#ifdef CONFIG_BLK_DEV_INITRD
if (initrd_start)
memblock_reserve(__virt_to_phys(initrd_start), initrd_end - initrd_start);
#endif
/*
* Reserve the page tables. These are already in use,
* and can only be in node 0.
*/
memblock_reserve(__pa(swapper_pg_dir), SWAPPER_DIR_SIZE);
memblock_reserve(__pa(idmap_pg_dir), IDMAP_DIR_SIZE);
early_init_fdt_scan_reserved_mem();
/* 4GB maximum for 32-bit only capable devices */
......@@ -269,26 +266,33 @@ void __init mem_init(void)
#define MLK(b, t) b, t, ((t) - (b)) >> 10
#define MLM(b, t) b, t, ((t) - (b)) >> 20
#define MLG(b, t) b, t, ((t) - (b)) >> 30
#define MLK_ROUNDUP(b, t) b, t, DIV_ROUND_UP(((t) - (b)), SZ_1K)
pr_notice("Virtual kernel memory layout:\n"
" vmalloc : 0x%16lx - 0x%16lx (%6ld MB)\n"
" vmalloc : 0x%16lx - 0x%16lx (%6ld GB)\n"
#ifdef CONFIG_SPARSEMEM_VMEMMAP
" vmemmap : 0x%16lx - 0x%16lx (%6ld MB)\n"
" vmemmap : 0x%16lx - 0x%16lx (%6ld GB maximum)\n"
" 0x%16lx - 0x%16lx (%6ld MB actual)\n"
#endif
" PCI I/O : 0x%16lx - 0x%16lx (%6ld MB)\n"
" fixed : 0x%16lx - 0x%16lx (%6ld KB)\n"
" modules : 0x%16lx - 0x%16lx (%6ld MB)\n"
" memory : 0x%16lx - 0x%16lx (%6ld MB)\n"
" .init : 0x%p" " - 0x%p" " (%6ld kB)\n"
" .text : 0x%p" " - 0x%p" " (%6ld kB)\n"
" .data : 0x%p" " - 0x%p" " (%6ld kB)\n",
MLM(VMALLOC_START, VMALLOC_END),
" .init : 0x%p" " - 0x%p" " (%6ld KB)\n"
" .text : 0x%p" " - 0x%p" " (%6ld KB)\n"
" .data : 0x%p" " - 0x%p" " (%6ld KB)\n",
MLG(VMALLOC_START, VMALLOC_END),
#ifdef CONFIG_SPARSEMEM_VMEMMAP
MLG((unsigned long)vmemmap,
(unsigned long)vmemmap + VMEMMAP_SIZE),
MLM((unsigned long)virt_to_page(PAGE_OFFSET),
(unsigned long)virt_to_page(high_memory)),
#endif
MLM((unsigned long)PCI_IOBASE, (unsigned long)PCI_IOBASE + SZ_16M),
MLK(FIXADDR_START, FIXADDR_TOP),
MLM(MODULES_VADDR, MODULES_END),
MLM(PAGE_OFFSET, (unsigned long)high_memory),
MLK_ROUNDUP(__init_begin, __init_end),
MLK_ROUNDUP(_text, _etext),
MLK_ROUNDUP(_sdata, _edata));
......
arch/arm64/mm/ioremap.c
浏览文件 @ 5167d09f
......@@ -103,19 +103,28 @@ void __iomem *ioremap_cache(phys_addr_t phys_addr, size_t size)
}
EXPORT_SYMBOL(ioremap_cache);
#ifndef CONFIG_ARM64_64K_PAGES
static pte_t bm_pte[PTRS_PER_PTE] __page_aligned_bss;
#if CONFIG_ARM64_PGTABLE_LEVELS > 2
static pte_t bm_pmd[PTRS_PER_PMD] __page_aligned_bss;
#endif
#if CONFIG_ARM64_PGTABLE_LEVELS > 3
static pte_t bm_pud[PTRS_PER_PUD] __page_aligned_bss;
#endif
static inline pmd_t * __init early_ioremap_pmd(unsigned long addr)
static inline pud_t * __init early_ioremap_pud(unsigned long addr)
{
pgd_t *pgd;
pud_t *pud;
pgd = pgd_offset_k(addr);
BUG_ON(pgd_none(*pgd) || pgd_bad(*pgd));
pud = pud_offset(pgd, addr);
return pud_offset(pgd, addr);
}
static inline pmd_t * __init early_ioremap_pmd(unsigned long addr)
{
pud_t *pud = early_ioremap_pud(addr);
BUG_ON(pud_none(*pud) || pud_bad(*pud));
return pmd_offset(pud, addr);
......@@ -132,13 +141,18 @@ static inline pte_t * __init early_ioremap_pte(unsigned long addr)
void __init early_ioremap_init(void)
{
pgd_t *pgd;
pud_t *pud;
pmd_t *pmd;
unsigned long addr = fix_to_virt(FIX_BTMAP_BEGIN);
pmd = early_ioremap_pmd(fix_to_virt(FIX_BTMAP_BEGIN));
#ifndef CONFIG_ARM64_64K_PAGES
/* need to populate pmd for 4k pagesize only */
pgd = pgd_offset_k(addr);
pgd_populate(&init_mm, pgd, bm_pud);
pud = pud_offset(pgd, addr);
pud_populate(&init_mm, pud, bm_pmd);
pmd = pmd_offset(pud, addr);
pmd_populate_kernel(&init_mm, pmd, bm_pte);
#endif
/*
* The boot-ioremap range spans multiple pmds, for which
* we are not prepared:
......
arch/arm64/mm/mmu.c
浏览文件 @ 5167d09f
......@@ -32,6 +32,7 @@
#include <asm/setup.h>
#include <asm/sizes.h>
#include <asm/tlb.h>
#include <asm/memblock.h>
#include <asm/mmu_context.h>
#include "mm.h"
......@@ -204,9 +205,16 @@ static void __init alloc_init_pud(pgd_t *pgd, unsigned long addr,
unsigned long end, unsigned long phys,
int map_io)
{
pud_t *pud = pud_offset(pgd, addr);
pud_t *pud;
unsigned long next;
if (pgd_none(*pgd)) {
pud = early_alloc(PTRS_PER_PUD * sizeof(pud_t));
pgd_populate(&init_mm, pgd, pud);
}
BUG_ON(pgd_bad(*pgd));
pud = pud_offset(pgd, addr);
do {
next = pud_addr_end(addr, end);
......@@ -290,10 +298,10 @@ static void __init map_mem(void)
* memory addressable from the initial direct kernel mapping.
*
* The initial direct kernel mapping, located at swapper_pg_dir,
* gives us PGDIR_SIZE memory starting from PHYS_OFFSET (which must be
* gives us PUD_SIZE memory starting from PHYS_OFFSET (which must be
* aligned to 2MB as per Documentation/arm64/booting.txt).
*/
limit = PHYS_OFFSET + PGDIR_SIZE;
limit = PHYS_OFFSET + PUD_SIZE;
memblock_set_current_limit(limit);
/* map all the memory banks */
......
drivers/irqchip/Kconfig
浏览文件 @ 5167d09f
......@@ -10,6 +10,11 @@ config ARM_GIC
config GIC_NON_BANKED
bool
config ARM_GIC_V3
bool
select IRQ_DOMAIN
select MULTI_IRQ_HANDLER
config ARM_NVIC
bool
select IRQ_DOMAIN
......
drivers/irqchip/Makefile
浏览文件 @ 5167d09f
......@@ -15,7 +15,8 @@ obj-$(CONFIG_ORION_IRQCHIP) += irq-orion.o
obj-$(CONFIG_ARCH_SUNXI) += irq-sun4i.o
obj-$(CONFIG_ARCH_SUNXI) += irq-sunxi-nmi.o
obj-$(CONFIG_ARCH_SPEAR3XX) += spear-shirq.o
obj-$(CONFIG_ARM_GIC) += irq-gic.o
obj-$(CONFIG_ARM_GIC) += irq-gic.o irq-gic-common.o
obj-$(CONFIG_ARM_GIC_V3) += irq-gic-v3.o irq-gic-common.o
obj-$(CONFIG_ARM_NVIC) += irq-nvic.o
obj-$(CONFIG_ARM_VIC) += irq-vic.o
obj-$(CONFIG_IMGPDC_IRQ) += irq-imgpdc.o
......
drivers/irqchip/irq-gic-common.c 0 → 100644
浏览文件 @ 5167d09f
/*
* Copyright (C) 2002 ARM Limited, All Rights Reserved.
*
* This program is free software; you can redistribute it and/or modify
* it under the terms of the GNU General Public License version 2 as
* published by the Free Software Foundation.
*
* This program is distributed in the hope that it will be useful,
* but WITHOUT ANY WARRANTY; without even the implied warranty of
* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
* GNU General Public License for more details.
*
* You should have received a copy of the GNU General Public License
* along with this program. If not, see <http://www.gnu.org/licenses/>.
*/
#include <linux/interrupt.h>
#include <linux/io.h>
#include <linux/irq.h>
#include <linux/irqchip/arm-gic.h>
#include "irq-gic-common.h"
void gic_configure_irq(unsigned int irq, unsigned int type,
void __iomem *base, void (*sync_access)(void))
{
u32 enablemask = 1 << (irq % 32);
u32 enableoff = (irq / 32) * 4;
u32 confmask = 0x2 << ((irq % 16) * 2);
u32 confoff = (irq / 16) * 4;
bool enabled = false;
u32 val;
/*
* Read current configuration register, and insert the config
* for "irq", depending on "type".
*/
val = readl_relaxed(base + GIC_DIST_CONFIG + confoff);
if (type == IRQ_TYPE_LEVEL_HIGH)
val &= ~confmask;
else if (type == IRQ_TYPE_EDGE_RISING)
val |= confmask;
/*
* As recommended by the spec, disable the interrupt before changing
* the configuration
*/
if (readl_relaxed(base + GIC_DIST_ENABLE_SET + enableoff) & enablemask) {
writel_relaxed(enablemask, base + GIC_DIST_ENABLE_CLEAR + enableoff);
if (sync_access)
sync_access();
enabled = true;
}
/*
* Write back the new configuration, and possibly re-enable
* the interrupt.
*/
writel_relaxed(val, base + GIC_DIST_CONFIG + confoff);
if (enabled)
writel_relaxed(enablemask, base + GIC_DIST_ENABLE_SET + enableoff);
if (sync_access)
sync_access();
}
void __init gic_dist_config(void __iomem *base, int gic_irqs,
void (*sync_access)(void))
{
unsigned int i;
/*
* Set all global interrupts to be level triggered, active low.
*/
for (i = 32; i < gic_irqs; i += 16)
writel_relaxed(0, base + GIC_DIST_CONFIG + i / 4);
/*
* Set priority on all global interrupts.
*/
for (i = 32; i < gic_irqs; i += 4)
writel_relaxed(0xa0a0a0a0, base + GIC_DIST_PRI + i);
/*
* Disable all interrupts. Leave the PPI and SGIs alone
* as they are enabled by redistributor registers.
*/
for (i = 32; i < gic_irqs; i += 32)
writel_relaxed(0xffffffff, base + GIC_DIST_ENABLE_CLEAR + i / 8);
if (sync_access)
sync_access();
}
void gic_cpu_config(void __iomem *base, void (*sync_access)(void))
{
int i;
/*
* Deal with the banked PPI and SGI interrupts - disable all
* PPI interrupts, ensure all SGI interrupts are enabled.
*/
writel_relaxed(0xffff0000, base + GIC_DIST_ENABLE_CLEAR);
writel_relaxed(0x0000ffff, base + GIC_DIST_ENABLE_SET);
/*
* Set priority on PPI and SGI interrupts
*/
for (i = 0; i < 32; i += 4)
writel_relaxed(0xa0a0a0a0, base + GIC_DIST_PRI + i * 4 / 4);
if (sync_access)
sync_access();
}
arch/arm64/include/asm/pgtable-3level-hwdef.h → drivers/irqchip/irq-gic-common.h
浏览文件 @ 5167d09f
/*
* Copyright (C) 2012 ARM Ltd.
* Copyright (C) 2002 ARM Limited, All Rights Reserved.
*
* This program is free software; you can redistribute it and/or modify
* it under the terms of the GNU General Public License version 2 as
......@@ -13,38 +13,17 @@
* You should have received a copy of the GNU General Public License
* along with this program. If not, see <http://www.gnu.org/licenses/>.
*/
#ifndef __ASM_PGTABLE_3LEVEL_HWDEF_H
#define __ASM_PGTABLE_3LEVEL_HWDEF_H
/*
* With LPAE and 4KB pages, there are 3 levels of page tables. Each level has
* 512 entries of 8 bytes each, occupying a 4K page. The first level table
* covers a range of 512GB, each entry representing 1GB. The user and kernel
* address spaces are limited to 512GB each.
*/
#define PTRS_PER_PTE 512
#define PTRS_PER_PMD 512
#define PTRS_PER_PGD 512
#ifndef _IRQ_GIC_COMMON_H
#define _IRQ_GIC_COMMON_H
/*
* PGDIR_SHIFT determines the size a top-level page table entry can map.
*/
#define PGDIR_SHIFT 30
#define PGDIR_SIZE (_AC(1, UL) << PGDIR_SHIFT)
#define PGDIR_MASK (~(PGDIR_SIZE-1))
#include <linux/of.h>
#include <linux/irqdomain.h>
/*
* PMD_SHIFT determines the size a middle-level page table entry can map.
*/
#define PMD_SHIFT 21
#define PMD_SIZE (_AC(1, UL) << PMD_SHIFT)
#define PMD_MASK (~(PMD_SIZE-1))
/*
* section address mask and size definitions.
*/
#define SECTION_SHIFT 21
#define SECTION_SIZE (_AC(1, UL) << SECTION_SHIFT)
#define SECTION_MASK (~(SECTION_SIZE-1))
void gic_configure_irq(unsigned int irq, unsigned int type,
void __iomem *base, void (*sync_access)(void));
void gic_dist_config(void __iomem *base, int gic_irqs,
void (*sync_access)(void));
void gic_cpu_config(void __iomem *base, void (*sync_access)(void));
#endif
#endif /* _IRQ_GIC_COMMON_H */
drivers/irqchip/irq-gic-v3.c 0 → 100644
浏览文件 @ 5167d09f
/*
* Copyright (C) 2013, 2014 ARM Limited, All Rights Reserved.
* Author: Marc Zyngier <marc.zyngier@arm.com>
*
* This program is free software; you can redistribute it and/or modify
* it under the terms of the GNU General Public License version 2 as
* published by the Free Software Foundation.
*
* This program is distributed in the hope that it will be useful,
* but WITHOUT ANY WARRANTY; without even the implied warranty of
* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
* GNU General Public License for more details.
*
* You should have received a copy of the GNU General Public License
* along with this program. If not, see <http://www.gnu.org/licenses/>.
*/
#include <linux/cpu.h>
#include <linux/delay.h>
#include <linux/interrupt.h>
#include <linux/of.h>
#include <linux/of_address.h>
#include <linux/of_irq.h>
#include <linux/percpu.h>
#include <linux/slab.h>
#include <linux/irqchip/arm-gic-v3.h>
#include <asm/cputype.h>
#include <asm/exception.h>
#include <asm/smp_plat.h>
#include "irq-gic-common.h"
#include "irqchip.h"
struct gic_chip_data {
void __iomem *dist_base;
void __iomem **redist_base;
void __percpu __iomem **rdist;
struct irq_domain *domain;
u64 redist_stride;
u32 redist_regions;
unsigned int irq_nr;
};
static struct gic_chip_data gic_data __read_mostly;
#define gic_data_rdist() (this_cpu_ptr(gic_data.rdist))
#define gic_data_rdist_rd_base() (*gic_data_rdist())
#define gic_data_rdist_sgi_base() (gic_data_rdist_rd_base() + SZ_64K)
/* Our default, arbitrary priority value. Linux only uses one anyway. */
#define DEFAULT_PMR_VALUE 0xf0
static inline unsigned int gic_irq(struct irq_data *d)
{
return d->hwirq;
}
static inline int gic_irq_in_rdist(struct irq_data *d)
{
return gic_irq(d) < 32;
}
static inline void __iomem *gic_dist_base(struct irq_data *d)
{
if (gic_irq_in_rdist(d)) /* SGI+PPI -> SGI_base for this CPU */
return gic_data_rdist_sgi_base();
if (d->hwirq <= 1023) /* SPI -> dist_base */
return gic_data.dist_base;
if (d->hwirq >= 8192)
BUG(); /* LPI Detected!!! */
return NULL;
}
static void gic_do_wait_for_rwp(void __iomem *base)
{
u32 count = 1000000; /* 1s! */
while (readl_relaxed(base + GICD_CTLR) & GICD_CTLR_RWP) {
count--;
if (!count) {
pr_err_ratelimited("RWP timeout, gone fishing\n");
return;
}
cpu_relax();
udelay(1);
};
}
/* Wait for completion of a distributor change */
static void gic_dist_wait_for_rwp(void)
{
gic_do_wait_for_rwp(gic_data.dist_base);
}
/* Wait for completion of a redistributor change */
static void gic_redist_wait_for_rwp(void)
{
gic_do_wait_for_rwp(gic_data_rdist_rd_base());
}
/* Low level accessors */
static u64 gic_read_iar(void)
{
u64 irqstat;
asm volatile("mrs_s %0, " __stringify(ICC_IAR1_EL1) : "=r" (irqstat));
return irqstat;
}
static void gic_write_pmr(u64 val)
{
asm volatile("msr_s " __stringify(ICC_PMR_EL1) ", %0" : : "r" (val));
}
static void gic_write_ctlr(u64 val)
{
asm volatile("msr_s " __stringify(ICC_CTLR_EL1) ", %0" : : "r" (val));
isb();
}
static void gic_write_grpen1(u64 val)
{
asm volatile("msr_s " __stringify(ICC_GRPEN1_EL1) ", %0" : : "r" (val));
isb();
}
static void gic_write_sgi1r(u64 val)
{
asm volatile("msr_s " __stringify(ICC_SGI1R_EL1) ", %0" : : "r" (val));
}
static void gic_enable_sre(void)
{
u64 val;
asm volatile("mrs_s %0, " __stringify(ICC_SRE_EL1) : "=r" (val));
val |= ICC_SRE_EL1_SRE;
asm volatile("msr_s " __stringify(ICC_SRE_EL1) ", %0" : : "r" (val));
isb();
/*
* Need to check that the SRE bit has actually been set. If
* not, it means that SRE is disabled at EL2. We're going to
* die painfully, and there is nothing we can do about it.
*
* Kindly inform the luser.
*/
asm volatile("mrs_s %0, " __stringify(ICC_SRE_EL1) : "=r" (val));
if (!(val & ICC_SRE_EL1_SRE))
pr_err("GIC: unable to set SRE (disabled at EL2), panic ahead\n");
}
static void gic_enable_redist(void)
{
void __iomem *rbase;
u32 count = 1000000; /* 1s! */
u32 val;
rbase = gic_data_rdist_rd_base();
/* Wake up this CPU redistributor */
val = readl_relaxed(rbase + GICR_WAKER);
val &= ~GICR_WAKER_ProcessorSleep;
writel_relaxed(val, rbase + GICR_WAKER);
while (readl_relaxed(rbase + GICR_WAKER) & GICR_WAKER_ChildrenAsleep) {
count--;
if (!count) {
pr_err_ratelimited("redist didn't wake up...\n");
return;
}
cpu_relax();
udelay(1);
};
}
/*
* Routines to disable, enable, EOI and route interrupts
*/
static void gic_poke_irq(struct irq_data *d, u32 offset)
{
u32 mask = 1 << (gic_irq(d) % 32);
void (*rwp_wait)(void);
void __iomem *base;
if (gic_irq_in_rdist(d)) {
base = gic_data_rdist_sgi_base();
rwp_wait = gic_redist_wait_for_rwp;
} else {
base = gic_data.dist_base;
rwp_wait = gic_dist_wait_for_rwp;
}
writel_relaxed(mask, base + offset + (gic_irq(d) / 32) * 4);
rwp_wait();
}
static int gic_peek_irq(struct irq_data *d, u32 offset)
{
u32 mask = 1 << (gic_irq(d) % 32);
void __iomem *base;
if (gic_irq_in_rdist(d))
base = gic_data_rdist_sgi_base();
else
base = gic_data.dist_base;
return !!(readl_relaxed(base + offset + (gic_irq(d) / 32) * 4) & mask);
}
static void gic_mask_irq(struct irq_data *d)
{
gic_poke_irq(d, GICD_ICENABLER);
}
static void gic_unmask_irq(struct irq_data *d)
{
gic_poke_irq(d, GICD_ISENABLER);
}
static void gic_eoi_irq(struct irq_data *d)
{
gic_write_eoir(gic_irq(d));
}
static int gic_set_type(struct irq_data *d, unsigned int type)
{
unsigned int irq = gic_irq(d);
void (*rwp_wait)(void);
void __iomem *base;
/* Interrupt configuration for SGIs can't be changed */
if (irq < 16)
return -EINVAL;
if (type != IRQ_TYPE_LEVEL_HIGH && type != IRQ_TYPE_EDGE_RISING)
return -EINVAL;
if (gic_irq_in_rdist(d)) {
base = gic_data_rdist_sgi_base();
rwp_wait = gic_redist_wait_for_rwp;
} else {
base = gic_data.dist_base;
rwp_wait = gic_dist_wait_for_rwp;
}
gic_configure_irq(irq, type, base, rwp_wait);
return 0;
}
static u64 gic_mpidr_to_affinity(u64 mpidr)
{
u64 aff;
aff = (MPIDR_AFFINITY_LEVEL(mpidr, 3) << 32 |
MPIDR_AFFINITY_LEVEL(mpidr, 2) << 16 |
MPIDR_AFFINITY_LEVEL(mpidr, 1) << 8 |
MPIDR_AFFINITY_LEVEL(mpidr, 0));
return aff;
}
static asmlinkage void __exception_irq_entry gic_handle_irq(struct pt_regs *regs)
{
u64 irqnr;
do {
irqnr = gic_read_iar();
if (likely(irqnr > 15 && irqnr < 1020)) {
u64 irq = irq_find_mapping(gic_data.domain, irqnr);
if (likely(irq)) {
handle_IRQ(irq, regs);
continue;
}
WARN_ONCE(true, "Unexpected SPI received!\n");
gic_write_eoir(irqnr);
}
if (irqnr < 16) {
gic_write_eoir(irqnr);
#ifdef CONFIG_SMP
handle_IPI(irqnr, regs);
#else
WARN_ONCE(true, "Unexpected SGI received!\n");
#endif
continue;
}
} while (irqnr != ICC_IAR1_EL1_SPURIOUS);
}
static void __init gic_dist_init(void)
{
unsigned int i;
u64 affinity;
void __iomem *base = gic_data.dist_base;
/* Disable the distributor */
writel_relaxed(0, base + GICD_CTLR);
gic_dist_wait_for_rwp();
gic_dist_config(base, gic_data.irq_nr, gic_dist_wait_for_rwp);
/* Enable distributor with ARE, Group1 */
writel_relaxed(GICD_CTLR_ARE_NS | GICD_CTLR_ENABLE_G1A | GICD_CTLR_ENABLE_G1,
base + GICD_CTLR);
/*
* Set all global interrupts to the boot CPU only. ARE must be
* enabled.
*/
affinity = gic_mpidr_to_affinity(cpu_logical_map(smp_processor_id()));
for (i = 32; i < gic_data.irq_nr; i++)
writeq_relaxed(affinity, base + GICD_IROUTER + i * 8);
}
static int gic_populate_rdist(void)
{
u64 mpidr = cpu_logical_map(smp_processor_id());
u64 typer;
u32 aff;
int i;
/*
* Convert affinity to a 32bit value that can be matched to
* GICR_TYPER bits [63:32].
*/
aff = (MPIDR_AFFINITY_LEVEL(mpidr, 3) << 24 |
MPIDR_AFFINITY_LEVEL(mpidr, 2) << 16 |
MPIDR_AFFINITY_LEVEL(mpidr, 1) << 8 |
MPIDR_AFFINITY_LEVEL(mpidr, 0));
for (i = 0; i < gic_data.redist_regions; i++) {
void __iomem *ptr = gic_data.redist_base[i];
u32 reg;
reg = readl_relaxed(ptr + GICR_PIDR2) & GIC_PIDR2_ARCH_MASK;
if (reg != GIC_PIDR2_ARCH_GICv3 &&
reg != GIC_PIDR2_ARCH_GICv4) { /* We're in trouble... */
pr_warn("No redistributor present @%p\n", ptr);
break;
}
do {
typer = readq_relaxed(ptr + GICR_TYPER);
if ((typer >> 32) == aff) {
gic_data_rdist_rd_base() = ptr;
pr_info("CPU%d: found redistributor %llx @%p\n",
smp_processor_id(),
(unsigned long long)mpidr, ptr);
return 0;
}
if (gic_data.redist_stride) {
ptr += gic_data.redist_stride;
} else {
ptr += SZ_64K * 2; /* Skip RD_base + SGI_base */
if (typer & GICR_TYPER_VLPIS)
ptr += SZ_64K * 2; /* Skip VLPI_base + reserved page */
}
} while (!(typer & GICR_TYPER_LAST));
}
/* We couldn't even deal with ourselves... */
WARN(true, "CPU%d: mpidr %llx has no re-distributor!\n",
smp_processor_id(), (unsigned long long)mpidr);
return -ENODEV;
}
static void gic_cpu_init(void)
{
void __iomem *rbase;
/* Register ourselves with the rest of the world */
if (gic_populate_rdist())
return;
gic_enable_redist();
rbase = gic_data_rdist_sgi_base();
gic_cpu_config(rbase, gic_redist_wait_for_rwp);
/* Enable system registers */
gic_enable_sre();
/* Set priority mask register */
gic_write_pmr(DEFAULT_PMR_VALUE);
/* EOI deactivates interrupt too (mode 0) */
gic_write_ctlr(ICC_CTLR_EL1_EOImode_drop_dir);
/* ... and let's hit the road... */
gic_write_grpen1(1);
}
#ifdef CONFIG_SMP
static int gic_secondary_init(struct notifier_block *nfb,
unsigned long action, void *hcpu)
{
if (action == CPU_STARTING || action == CPU_STARTING_FROZEN)
gic_cpu_init();
return NOTIFY_OK;
}
/*
* Notifier for enabling the GIC CPU interface. Set an arbitrarily high
* priority because the GIC needs to be up before the ARM generic timers.
*/
static struct notifier_block gic_cpu_notifier = {
.notifier_call = gic_secondary_init,
.priority = 100,
};
static u16 gic_compute_target_list(int *base_cpu, const struct cpumask *mask,
u64 cluster_id)
{
int cpu = *base_cpu;
u64 mpidr = cpu_logical_map(cpu);
u16 tlist = 0;
while (cpu < nr_cpu_ids) {
/*
* If we ever get a cluster of more than 16 CPUs, just
* scream and skip that CPU.
*/
if (WARN_ON((mpidr & 0xff) >= 16))
goto out;
tlist |= 1 << (mpidr & 0xf);
cpu = cpumask_next(cpu, mask);
if (cpu == nr_cpu_ids)
goto out;
mpidr = cpu_logical_map(cpu);
if (cluster_id != (mpidr & ~0xffUL)) {
cpu--;
goto out;
}
}
out:
*base_cpu = cpu;
return tlist;
}
static void gic_send_sgi(u64 cluster_id, u16 tlist, unsigned int irq)
{
u64 val;
val = (MPIDR_AFFINITY_LEVEL(cluster_id, 3) << 48 |
MPIDR_AFFINITY_LEVEL(cluster_id, 2) << 32 |
irq << 24 |
MPIDR_AFFINITY_LEVEL(cluster_id, 1) << 16 |
tlist);
pr_debug("CPU%d: ICC_SGI1R_EL1 %llx\n", smp_processor_id(), val);
gic_write_sgi1r(val);
}
static void gic_raise_softirq(const struct cpumask *mask, unsigned int irq)
{
int cpu;
if (WARN_ON(irq >= 16))
return;
/*
* Ensure that stores to Normal memory are visible to the
* other CPUs before issuing the IPI.
*/
smp_wmb();
for_each_cpu_mask(cpu, *mask) {
u64 cluster_id = cpu_logical_map(cpu) & ~0xffUL;
u16 tlist;
tlist = gic_compute_target_list(&cpu, mask, cluster_id);
gic_send_sgi(cluster_id, tlist, irq);
}
/* Force the above writes to ICC_SGI1R_EL1 to be executed */
isb();
}
static void gic_smp_init(void)
{
set_smp_cross_call(gic_raise_softirq);
register_cpu_notifier(&gic_cpu_notifier);
}
static int gic_set_affinity(struct irq_data *d, const struct cpumask *mask_val,
bool force)
{
unsigned int cpu = cpumask_any_and(mask_val, cpu_online_mask);
void __iomem *reg;
int enabled;
u64 val;
if (gic_irq_in_rdist(d))
return -EINVAL;
/* If interrupt was enabled, disable it first */
enabled = gic_peek_irq(d, GICD_ISENABLER);
if (enabled)
gic_mask_irq(d);
reg = gic_dist_base(d) + GICD_IROUTER + (gic_irq(d) * 8);
val = gic_mpidr_to_affinity(cpu_logical_map(cpu));
writeq_relaxed(val, reg);
/*
* If the interrupt was enabled, enabled it again. Otherwise,
* just wait for the distributor to have digested our changes.
*/
if (enabled)
gic_unmask_irq(d);
else
gic_dist_wait_for_rwp();
return IRQ_SET_MASK_OK;
}
#else
#define gic_set_affinity NULL
#define gic_smp_init() do { } while(0)
#endif
static struct irq_chip gic_chip = {
.name = "GICv3",
.irq_mask = gic_mask_irq,
.irq_unmask = gic_unmask_irq,
.irq_eoi = gic_eoi_irq,
.irq_set_type = gic_set_type,
.irq_set_affinity = gic_set_affinity,
};
static int gic_irq_domain_map(struct irq_domain *d, unsigned int irq,
irq_hw_number_t hw)
{
/* SGIs are private to the core kernel */
if (hw < 16)
return -EPERM;
/* PPIs */
if (hw < 32) {
irq_set_percpu_devid(irq);
irq_set_chip_and_handler(irq, &gic_chip,
handle_percpu_devid_irq);
set_irq_flags(irq, IRQF_VALID | IRQF_NOAUTOEN);
}
/* SPIs */
if (hw >= 32 && hw < gic_data.irq_nr) {
irq_set_chip_and_handler(irq, &gic_chip,
handle_fasteoi_irq);
set_irq_flags(irq, IRQF_VALID | IRQF_PROBE);
}
irq_set_chip_data(irq, d->host_data);
return 0;
}
static int gic_irq_domain_xlate(struct irq_domain *d,
struct device_node *controller,
const u32 *intspec, unsigned int intsize,
unsigned long *out_hwirq, unsigned int *out_type)
{
if (d->of_node != controller)
return -EINVAL;
if (intsize < 3)
return -EINVAL;
switch(intspec[0]) {
case 0: /* SPI */
*out_hwirq = intspec[1] + 32;
break;
case 1: /* PPI */
*out_hwirq = intspec[1] + 16;
break;
default:
return -EINVAL;
}
*out_type = intspec[2] & IRQ_TYPE_SENSE_MASK;
return 0;
}
static const struct irq_domain_ops gic_irq_domain_ops = {
.map = gic_irq_domain_map,
.xlate = gic_irq_domain_xlate,
};
static int __init gic_of_init(struct device_node *node, struct device_node *parent)
{
void __iomem *dist_base;
void __iomem **redist_base;
u64 redist_stride;
u32 redist_regions;
u32 reg;
int gic_irqs;
int err;
int i;
dist_base = of_iomap(node, 0);
if (!dist_base) {
pr_err("%s: unable to map gic dist registers\n",
node->full_name);
return -ENXIO;
}
reg = readl_relaxed(dist_base + GICD_PIDR2) & GIC_PIDR2_ARCH_MASK;
if (reg != GIC_PIDR2_ARCH_GICv3 && reg != GIC_PIDR2_ARCH_GICv4) {
pr_err("%s: no distributor detected, giving up\n",
node->full_name);
err = -ENODEV;
goto out_unmap_dist;
}
if (of_property_read_u32(node, "#redistributor-regions", &redist_regions))
redist_regions = 1;
redist_base = kzalloc(sizeof(*redist_base) * redist_regions, GFP_KERNEL);
if (!redist_base) {
err = -ENOMEM;
goto out_unmap_dist;
}
for (i = 0; i < redist_regions; i++) {
redist_base[i] = of_iomap(node, 1 + i);
if (!redist_base[i]) {
pr_err("%s: couldn't map region %d\n",
node->full_name, i);
err = -ENODEV;
goto out_unmap_rdist;
}
}
if (of_property_read_u64(node, "redistributor-stride", &redist_stride))
redist_stride = 0;
gic_data.dist_base = dist_base;
gic_data.redist_base = redist_base;
gic_data.redist_regions = redist_regions;
gic_data.redist_stride = redist_stride;
/*
* Find out how many interrupts are supported.
* The GIC only supports up to 1020 interrupt sources (SGI+PPI+SPI)
*/
gic_irqs = readl_relaxed(gic_data.dist_base + GICD_TYPER) & 0x1f;
gic_irqs = (gic_irqs + 1) * 32;
if (gic_irqs > 1020)
gic_irqs = 1020;
gic_data.irq_nr = gic_irqs;
gic_data.domain = irq_domain_add_tree(node, &gic_irq_domain_ops,
&gic_data);
gic_data.rdist = alloc_percpu(typeof(*gic_data.rdist));
if (WARN_ON(!gic_data.domain) || WARN_ON(!gic_data.rdist)) {
err = -ENOMEM;
goto out_free;
}
set_handle_irq(gic_handle_irq);
gic_smp_init();
gic_dist_init();
gic_cpu_init();
return 0;
out_free:
if (gic_data.domain)
irq_domain_remove(gic_data.domain);
free_percpu(gic_data.rdist);
out_unmap_rdist:
for (i = 0; i < redist_regions; i++)
if (redist_base[i])
iounmap(redist_base[i]);
kfree(redist_base);
out_unmap_dist:
iounmap(dist_base);
return err;
}
IRQCHIP_DECLARE(gic_v3, "arm,gic-v3", gic_of_init);
drivers/irqchip/irq-gic.c
浏览文件 @ 5167d09f
......@@ -47,6 +47,7 @@
#include <asm/exception.h>
#include <asm/smp_plat.h>
#include "irq-gic-common.h"
#include "irqchip.h"
union gic_base {
......@@ -189,12 +190,6 @@ static int gic_set_type(struct irq_data *d, unsigned int type)
{
void __iomem *base = gic_dist_base(d);
unsigned int gicirq = gic_irq(d);
u32 enablemask = 1 << (gicirq % 32);
u32 enableoff = (gicirq / 32) * 4;
u32 confmask = 0x2 << ((gicirq % 16) * 2);
u32 confoff = (gicirq / 16) * 4;
bool enabled = false;
u32 val;
/* Interrupt configuration for SGIs can't be changed */
if (gicirq < 16)
......@@ -208,25 +203,7 @@ static int gic_set_type(struct irq_data *d, unsigned int type)
if (gic_arch_extn.irq_set_type)
gic_arch_extn.irq_set_type(d, type);
val = readl_relaxed(base + GIC_DIST_CONFIG + confoff);
if (type == IRQ_TYPE_LEVEL_HIGH)
val &= ~confmask;
else if (type == IRQ_TYPE_EDGE_RISING)
val |= confmask;
/*
* As recommended by the spec, disable the interrupt before changing
* the configuration
*/
if (readl_relaxed(base + GIC_DIST_ENABLE_SET + enableoff) & enablemask) {
writel_relaxed(enablemask, base + GIC_DIST_ENABLE_CLEAR + enableoff);
enabled = true;
}
writel_relaxed(val, base + GIC_DIST_CONFIG + confoff);
if (enabled)
writel_relaxed(enablemask, base + GIC_DIST_ENABLE_SET + enableoff);
gic_configure_irq(gicirq, type, base, NULL);
raw_spin_unlock(&irq_controller_lock);
......@@ -387,12 +364,6 @@ static void __init gic_dist_init(struct gic_chip_data *gic)
writel_relaxed(0, base + GIC_DIST_CTRL);
/*
* Set all global interrupts to be level triggered, active low.
*/
for (i = 32; i < gic_irqs; i += 16)
writel_relaxed(0, base + GIC_DIST_CONFIG + i * 4 / 16);
/*
* Set all global interrupts to this CPU only.
*/
......@@ -402,18 +373,7 @@ static void __init gic_dist_init(struct gic_chip_data *gic)
for (i = 32; i < gic_irqs; i += 4)
writel_relaxed(cpumask, base + GIC_DIST_TARGET + i * 4 / 4);
/*
* Set priority on all global interrupts.
*/
for (i = 32; i < gic_irqs; i += 4)
writel_relaxed(0xa0a0a0a0, base + GIC_DIST_PRI + i * 4 / 4);
/*
* Disable all interrupts. Leave the PPI and SGIs alone
* as these enables are banked registers.
*/
for (i = 32; i < gic_irqs; i += 32)
writel_relaxed(0xffffffff, base + GIC_DIST_ENABLE_CLEAR + i * 4 / 32);
gic_dist_config(base, gic_irqs, NULL);
writel_relaxed(1, base + GIC_DIST_CTRL);
}
......@@ -440,18 +400,7 @@ static void gic_cpu_init(struct gic_chip_data *gic)
if (i != cpu)
gic_cpu_map[i] &= ~cpu_mask;
/*
* Deal with the banked PPI and SGI interrupts - disable all
* PPI interrupts, ensure all SGI interrupts are enabled.
*/
writel_relaxed(0xffff0000, dist_base + GIC_DIST_ENABLE_CLEAR);
writel_relaxed(0x0000ffff, dist_base + GIC_DIST_ENABLE_SET);
/*
* Set priority on PPI and SGI interrupts
*/
for (i = 0; i < 32; i += 4)
writel_relaxed(0xa0a0a0a0, dist_base + GIC_DIST_PRI + i * 4 / 4);
gic_cpu_config(dist_base, NULL);
writel_relaxed(0xf0, base + GIC_CPU_PRIMASK);
writel_relaxed(1, base + GIC_CPU_CTRL);
......
include/linux/irqchip/arm-gic-v3.h 0 → 100644
浏览文件 @ 5167d09f
此差异已折叠。
include/uapi/linux/audit.h
浏览文件 @ 5167d09f
......@@ -342,6 +342,7 @@ enum {
#define __AUDIT_ARCH_64BIT 0x80000000
#define __AUDIT_ARCH_LE 0x40000000
#define AUDIT_ARCH_AARCH64 (EM_AARCH64|__AUDIT_ARCH_64BIT|__AUDIT_ARCH_LE)
#define AUDIT_ARCH_ALPHA (EM_ALPHA|__AUDIT_ARCH_64BIT|__AUDIT_ARCH_LE)
#define AUDIT_ARCH_ARM (EM_ARM|__AUDIT_ARCH_LE)
#define AUDIT_ARCH_ARMEB (EM_ARM)
......
Markdown is supported
0% 或 .
You are about to add 0 people to the discussion. Proceed with caution.
先完成此消息的编辑！
想要评论请 注册 或