This is an eBPF JIT for RV32G, adapted from the JIT for RV64G and
the 32-bit ARM JIT.

There are two main changes required for this to work compared to
the RV64 JIT.

First, eBPF registers are 64-bit, while RV32G registers are 32-bit.
BPF registers either map directly to 2 RISC-V registers, or reside
in stack scratch space and are saved and restored when used.

Second, many 64-bit ALU operations do not trivially map to 32-bit
operations. Operations that move bits between high and low words,
such as ADD, LSH, MUL, and others must emulate the 64-bit behavior
in terms of 32-bit instructions.

This patch also makes related changes to bpf_jit.h, such
as adding RISC-V instructions required by the RV32 JIT.

Supported features:

The RV32 JIT supports the same features and instructions as the
RV64 JIT, with the following exceptions:

- ALU64 DIV/MOD: Requires loops to implement on 32-bit hardware.

- BPF_XADD | BPF_DW: There's no 8-byte atomic instruction in RV32.

These features are also unsupported on other BPF JITs for 32-bit
architectures.

Testing:

- lib/test_bpf.c
test_bpf: Summary: 378 PASSED, 0 FAILED, [349/366 JIT'ed]
test_bpf: test_skb_segment: Summary: 2 PASSED, 0 FAILED

The tests that are not JITed are all due to use of 64-bit div/mod
or 64-bit xadd.

- tools/testing/selftests/bpf/test_verifier.c
Summary: 1415 PASSED, 122 SKIPPED, 43 FAILED

Tested both with and without BPF JIT hardening.

This is the same set of tests that pass using the BPF interpreter
with the JIT disabled.

Verification and synthesis:

We developed the RV32 JIT using our automated verification tool,
Serval. We have used Serval in the past to verify patches to the
RV64 JIT. We also used Serval to superoptimize the resulting code
through program synthesis.

You can find the tool and a guide to the approach and results here:
https://github.com/uw-unsat/serval-bpf/tree/rv32-jit-v5Co-developed-by: NXi Wang <xi.wang@gmail.com>
Signed-off-by: NXi Wang <xi.wang@gmail.com>
Signed-off-by: NLuke Nelson <luke.r.nels@gmail.com>
Signed-off-by: NDaniel Borkmann <daniel@iogearbox.net>
Reviewed-by: NBjörn Töpel <bjorn.topel@gmail.com>
Acked-by: NBjörn Töpel <bjorn.topel@gmail.com>
Link: https://lore.kernel.org/bpf/20200305050207.4159-3-luke.r.nels@gmail.com

This patch factors out code that can be used by both the RV64 and RV32
BPF JITs to a common bpf_jit.h and bpf_jit_core.c.

Move struct definitions and macro-like functions to header. Rename
rv_sb_insn/rv_uj_insn to rv_b_insn/rv_j_insn to match the RISC-V
specification.

Move reusable functions emit_body() and bpf_int_jit_compile() to
bpf_jit_core.c with minor simplifications. Rename emit_insn() and
build_{prologue,epilogue}() to be prefixed with "bpf_jit_" as they are
no longer static.

Rename bpf_jit_comp.c to bpf_jit_comp64.c to be more explicit.
Co-developed-by: NXi Wang <xi.wang@gmail.com>
Signed-off-by: NXi Wang <xi.wang@gmail.com>
Signed-off-by: NLuke Nelson <luke.r.nels@gmail.com>
Signed-off-by: NDaniel Borkmann <daniel@iogearbox.net>
Reviewed-by: NBjörn Töpel <bjorn.topel@gmail.com>
Acked-by: NBjörn Töpel <bjorn.topel@gmail.com>
Link: https://lore.kernel.org/bpf/20200305050207.4159-2-luke.r.nels@gmail.com

Adjust the indent to match Linux coding style.
Signed-off-by: NZong Li <zong.li@sifive.com>
Signed-off-by: NPalmer Dabbelt <palmerdabbelt@google.com>

Each page table should be created by allocating a complete page size
for it. Otherwise, the content of the page table would be corrupted
somewhere through memory allocation which allocates the memory at the
middle of the page table for other use.
Signed-off-by: NZong Li <zong.li@sifive.com>
Signed-off-by: NPalmer Dabbelt <palmerdabbelt@google.com>

Tell git to not track the compiled boot/loader and boot/loader.lds
files.
Signed-off-by: NDamien Le Moal <damien.lemoal@wdc.com>
Signed-off-by: NPalmer Dabbelt <palmerdabbelt@google.com>

Historically, we have been enabling all interrupts for each
HART in trap_init(). Ideally, we should only enable M-mode
interrupts for M-mode kernel and S-mode interrupts for S-mode
kernel in trap_init().

Currently, we get suprious S-mode interrupts on Kendryte K210
board running M-mode NO-MMU kernel because we are enabling all
interrupts in trap_init(). To fix this, we only enable software
and external interrupt in trap_init(). In future, trap_init()
will only enable software interrupt and PLIC driver will enable
external interrupt using CPU notifiers.

Fixes: a4c3733d ("riscv: abstract out CSR names for supervisor vs machine mode")
Signed-off-by: NAnup Patel <anup.patel@wdc.com>
Reviewed-by: NAtish Patra <atish.patra@wdc.com>
Tested-by: Palmer Dabbelt <palmerdabbelt@google.com> [QMEU virt machine with SMP]
[Palmer: Move the Fixes up to a newer commit]
Reviewed-by: NPalmer Dabbelt <palmerdabbelt@google.com>
Signed-off-by: NPalmer Dabbelt <palmerdabbelt@google.com>

When the kernel is running in S-mode, the expectation is that the
bootloader or SBI layer will configure the PMP to allow the kernel to
access physical memory.  But, when the kernel is running in M-mode and is
started with the ELF "loader", there's probably no bootloader or SBI layer
involved to configure the PMP.  Thus, we need to configure the PMP
ourselves to enable the kernel to access all regions.
Signed-off-by: NGreentime Hu <greentime.hu@sifive.com>
Reviewed-by: NPalmer Dabbelt <palmerdabbelt@google.com>
Signed-off-by: NPalmer Dabbelt <palmerdabbelt@google.com>

dma-continuguous.h is generic for all architectures except arm32 which has
its own version.

Similar change was done for msi.h by commit a1b39bae
("asm-generic: Make msi.h a mandatory include/asm header")
Suggested-by: NChristoph Hellwig <hch@infradead.org>
Link: https://lore.kernel.org/linux-arm-kernel/20200117080446.GA8980@lst.de/T/#m92bb56b04161057635d4142e1b3b9b6b0a70122eSigned-off-by: NMichal Simek <michal.simek@xilinx.com>
Reviewed-by: NChristoph Hellwig <hch@lst.de>
Acked-by: NThomas Gleixner <tglx@linutronix.de>
Acked-by: NArnd Bergmann <arnd@arndb.de>
Acked-by: Paul Walmsley <paul.walmsley@sifive.com> # for arch/riscv

walk_page_range() is going to be allowed to walk page tables other than
those of user space.  For this it needs to know when it has reached a
'leaf' entry in the page tables.  This information is provided by the
p?d_leaf() functions/macros.

For riscv a page is a leaf page when it has a read, write or execute bit
set on it.

Link: http://lkml.kernel.org/r/20191218162402.45610-8-steven.price@arm.comSigned-off-by: NSteven Price <steven.price@arm.com>
Reviewed-by: NAlexandre Ghiti <alex@ghiti.fr>
Reviewed-by: NZong Li <zong.li@sifive.com>
Acked-by: Paul Walmsley <paul.walmsley@sifive.com>	[arch/riscv]
Cc: Albert Ou <aou@eecs.berkeley.edu>
Cc: Andy Lutomirski <luto@kernel.org>
Cc: Ard Biesheuvel <ard.biesheuvel@linaro.org>
Cc: Arnd Bergmann <arnd@arndb.de>
Cc: Benjamin Herrenschmidt <benh@kernel.crashing.org>
Cc: Borislav Petkov <bp@alien8.de>
Cc: Catalin Marinas <catalin.marinas@arm.com>
Cc: Christian Borntraeger <borntraeger@de.ibm.com>
Cc: Dave Hansen <dave.hansen@linux.intel.com>
Cc: David S. Miller <davem@davemloft.net>
Cc: Heiko Carstens <heiko.carstens@de.ibm.com>
Cc: "H. Peter Anvin" <hpa@zytor.com>
Cc: Ingo Molnar <mingo@redhat.com>
Cc: James Hogan <jhogan@kernel.org>
Cc: James Morse <james.morse@arm.com>
Cc: Jerome Glisse <jglisse@redhat.com>
Cc: "Liang, Kan" <kan.liang@linux.intel.com>
Cc: Mark Rutland <mark.rutland@arm.com>
Cc: Michael Ellerman <mpe@ellerman.id.au>
Cc: Paul Burton <paul.burton@mips.com>
Cc: Paul Mackerras <paulus@samba.org>
Cc: Peter Zijlstra <peterz@infradead.org>
Cc: Ralf Baechle <ralf@linux-mips.org>
Cc: Russell King <linux@armlinux.org.uk>
Cc: Thomas Gleixner <tglx@linutronix.de>
Cc: Vasily Gorbik <gor@linux.ibm.com>
Cc: Vineet Gupta <vgupta@synopsys.com>
Cc: Will Deacon <will@kernel.org>
Signed-off-by: NAndrew Morton <akpm@linux-foundation.org>
Signed-off-by: NLinus Torvalds <torvalds@linux-foundation.org>

Add the gpio DT node in SiFive FU540 soc-specific DT file.
Enable the gpio node in HiFive Unleashed board-specific DT file.
Signed-off-by: NYash Shah <yash.shah@sifive.com>
Signed-off-by: NPalmer Dabbelt <palmerdabbelt@google.com>

This patch implements CONFIG_DEBUG_VIRTUAL to do additional checks on
virt_to_phys and __pa_symbol calls. virt_to_phys used for linear mapping
check, and __pa_symbol used for kernel symbol check. In current RISC-V,
kernel image maps to linear mapping area. If CONFIG_DEBUG_VIRTUAL is
disable, these two functions calculate the offset on the address feded
directly without any checks.

The result of test_debug_virtual as follows:

[    0.358456] ------------[ cut here ]------------
[    0.358738] virt_to_phys used for non-linear address: (____ptrval____) (0xffffffd000000000)
[    0.359174] WARNING: CPU: 0 PID: 1 at arch/riscv/mm/physaddr.c:16 __virt_to_phys+0x3c/0x50
[    0.359409] Modules linked in:
[    0.359630] CPU: 0 PID: 1 Comm: swapper/0 Not tainted 5.5.0-rc3-00002-g5133c5c0ca13 #57
[    0.359861] epc: ffffffe000253d1a ra : ffffffe000253d1a sp : ffffffe03aa87da0
[    0.360019]  gp : ffffffe000ae03b0 tp : ffffffe03aa88000 t0 : ffffffe000af2660
[    0.360175]  t1 : 0000000000000064 t2 : 00000000000000b7 s0 : ffffffe03aa87dc0
[    0.360330]  s1 : ffffffd000000000 a0 : 000000000000004f a1 : 0000000000000000
[    0.360492]  a2 : 0000000000000000 a3 : 0000000000000000 a4 : ffffffe000a84358
[    0.360672]  a5 : 0000000000000000 a6 : 0000000000000000 a7 : 0000000000000000
[    0.360876]  s2 : ffffffe000ae0600 s3 : ffffffe00000fc7c s4 : ffffffe0000224b0
[    0.361067]  s5 : ffffffe000030890 s6 : ffffffe000022470 s7 : 0000000000000008
[    0.361267]  s8 : ffffffe0000002c4 s9 : ffffffe000ae0640 s10: ffffffe000ae0630
[    0.361453]  s11: 0000000000000000 t3 : 0000000000000000 t4 : 000000000001e6d0
[    0.361636]  t5 : ffffffe000ae0a18 t6 : ffffffe000aee54e
[    0.361806] status: 0000000000000120 badaddr: 0000000000000000 cause: 0000000000000003
[    0.362056] ---[ end trace aec0bf78d4978122 ]---
[    0.362404] PA: 0xfffffff080200000 for VA: 0xffffffd000000000
[    0.362607] PA: 0x00000000baddd2d0 for VA: 0xffffffe03abdd2d0
Signed-off-by: NZong Li <zong.li@sifive.com>
Reviewed-by: NPaul Walmsley <paul.walmsley@sifive.com>
Tested-by: NPaul Walmsley <paul.walmsley@sifive.com>
Signed-off-by: NPalmer Dabbelt <palmerdabbelt@google.com>

While rv32 technically has 34-bit physical addresses, no current platforms
use it and it's likely to shake out driver bugs.

Let's keep 64-bit phys_addr_t off on 32-bit builds until one shows up,
since other work will be needed to make such a system useful anyway.

PHYS_ADDR_T_64BIT is def_bool 64BIT, so just remove the select.
Signed-off-by: NOlof Johansson <olof@lixom.net>
Signed-off-by: NPalmer Dabbelt <palmerdabbelt@google.com>

This patch ports the feature Kernel Address SANitizer (KASAN).

Note: The start address of shadow memory is at the beginning of kernel
space, which is 2^64 - (2^39 / 2) in SV39. The size of the kernel space is
2^38 bytes so the size of shadow memory should be 2^38 / 8. Thus, the
shadow memory would not overlap with the fixmap area.

There are currently two limitations in this port,

1. RV64 only: KASAN need large address space for extra shadow memory
region.

2. KASAN can't debug the modules since the modules are allocated in VMALLOC
area. We mapped the shadow memory, which corresponding to VMALLOC area, to
the kasan_early_shadow_page because we don't have enough physical space for
all the shadow memory corresponding to VMALLOC area.
Signed-off-by: NNick Hu <nickhu@andestech.com>
Reported-by: NGreentime Hu <green.hu@gmail.com>
Signed-off-by: NPalmer Dabbelt <palmerdabbelt@google.com>

The existing __lshrti3 was really inefficient, and the other two helpers
are also needed to compile some modules.

Add the missing versions, and export all of the symbols like arm64
already does.

This code is based on the assembly generated by libgcc builds.

This fixes a build break triggered by ubsan:

riscv64-unknown-linux-gnu-ld: lib/ubsan.o: in function `.L2':
ubsan.c:(.text.unlikely+0x38): undefined reference to `__ashlti3'
riscv64-unknown-linux-gnu-ld: ubsan.c:(.text.unlikely+0x42): undefined reference to `__ashrti3'
Signed-off-by: NOlof Johansson <olof@lixom.net>
[paul.walmsley@sifive.com: use SYM_FUNC_{START,END} instead of
 ENTRY/ENDPROC; note libgcc origin]
Signed-off-by: NPaul Walmsley <paul.walmsley@sifive.com>

Temporary files used in the VDSO build process linger on even after make
mrproper: vdso-dummy.o.tmp, vdso.so.dbg.tmp.

Delete them once they're no longer needed.
Signed-off-by: NIlie Halip <ilie.halip@gmail.com>
Signed-off-by: NPaul Walmsley <paul.walmsley@sifive.com>

The code in secondary_park is currently placed in the .init section. The
kernel reclaims and clears this code when it finishes booting. That
causes the cores parked in it to go to somewhere unpredictable, so we
move this function out of init to make sure the cores stay looping there.

The instruction bgeu a0, t0, .Lsecondary_park may have "a relocation
truncated to fit" issue during linking time. It is because that sections
are too far to jump. Let's use tail to jump to the .Lsecondary_park.
Signed-off-by: NGreentime Hu <greentime.hu@sifive.com>
Reviewed-by: NAnup Patel <anup.patel@sifive.com>
Cc: Andreas Schwab <schwab@suse.de>
Cc: stable@vger.kernel.org
Fixes: 76d2a049 ("RISC-V: Init and Halt Code")
Signed-off-by: NPaul Walmsley <paul.walmsley@sifive.com>

con_init in tty/vt.c will now set conswitchp to dummy_con if it's unset.
Drop it from arch setup code.
Signed-off-by: NArvind Sankar <nivedita@alum.mit.edu>
Link: https://lore.kernel.org/r/20191218214506.49252-19-nivedita@alum.mit.eduSigned-off-by: NGreg Kroah-Hartman <gregkh@linuxfoundation.org>

CSR_MISA is defined in Privileged Architectures' spec: 3.1.1 Machine
ISA Register misa. Every bit:1 indicate a feature, so we should beqz
reset_done when there is no F/D bit in csr_misa register.
Signed-off-by: NGuo Ren <ren_guo@c-sky.com>
[paul.walmsley@sifive.com: fix typo in commit message]
Fixes: 9e806356 ("riscv: clear the instruction cache and all registers when booting")
Signed-off-by: NPaul Walmsley <paul.walmsley@sifive.com>

The commit 9209fb51 ("riscv: move sifive_l2_cache.c to drivers/soc")
moves the sifive L2 cache driver to driver/soc. It did not move the
header file along with the driver. Therefore this patch moves the header
file to driver/soc
Signed-off-by: NYash Shah <yash.shah@sifive.com>
Reviewed-by: NAnup Patel <anup@brainfault.org>
[paul.walmsley@sifive.com: updated to fix the include guard]
Fixes: 9209fb51 ("riscv: move sifive_l2_cache.c to drivers/soc")
Signed-off-by: NPaul Walmsley <paul.walmsley@sifive.com>

This is required for clone3 which passes the TLS value through a
struct rather than a register.
Signed-off-by: NAmanieu d'Antras <amanieu@gmail.com>
Cc: linux-riscv@lists.infradead.org
Cc: <stable@vger.kernel.org> # 5.3.x
Link: https://lore.kernel.org/r/20200102172413.654385-6-amanieu@gmail.comSigned-off-by: NChristian Brauner <christian.brauner@ubuntu.com>

"IRQ_TIMER", used in the arch/riscv CSR header file, is a sufficiently
generic macro name that it's used by several source files across the
Linux code base.  Some of these other files ultimately include the
arch/riscv CSR include file, causing collisions.  Fix by prefixing the
RISC-V csr.h IRQ_ macro names with an RV_ prefix.

Fixes: a4c3733d ("riscv: abstract out CSR names for supervisor vs machine mode")
Reported-by: NOlof Johansson <olof@lixom.net>
Acked-by: NOlof Johansson <olof@lixom.net>
Signed-off-by: NPaul Walmsley <paul.walmsley@sifive.com>

The condition should be logical NOT to assign the hook address to parent
address. Because the return value 0 of function_graph_enter upon
success.

Fixes: e949b6db (riscv/function_graph: Simplify with function_graph_enter())
Signed-off-by: NZong Li <zong.li@sifive.com>
Reviewed-by: NSteven Rostedt (VMware) <rostedt@goodmis.org>
Cc: stable@vger.kernel.org
Signed-off-by: NPaul Walmsley <paul.walmsley@sifive.com>

Add the L2 cache controller DT node in SiFive FU540 soc-specific DT file
Signed-off-by: NYash Shah <yash.shah@sifive.com>
Reviewed-by: NPalmer Dabbelt <palmerdabbelt@google.com>
Signed-off-by: NPaul Walmsley <paul.walmsley@sifive.com>

This patch enables GCOV code coverage measurement on RISC-V.
Lightly tested on QEMU and Hifive Unleashed board, seems to work as
expected.
Signed-off-by: NZong Li <zong.li@sifive.com>
Reviewed-by: NAnup Patel <anup@brainfault.org>
Acked-by: NJonathan Corbet <corbet@lwn.net>
Signed-off-by: NPaul Walmsley <paul.walmsley@sifive.com>

__pa_symbol is the marcro that should be used for kernel symbols. It is
also a pre-requisite for DEBUG_VIRTUAL which will do bounds checking.
Signed-off-by: NZong Li <zong.li@sifive.com>
Reviewed-by: NAnup Patel <anup@brainfault.org>
Signed-off-by: NPaul Walmsley <paul.walmsley@sifive.com>

This is needed by LKDTM (crash dump test module), it calls
flush_icache_range(), which on RISC-V turns into flush_icache_all(). On
other architectures, the actual implementation is exported, so follow
that precedence and export it here too.

Fixes build of CONFIG_LKDTM that fails with:
ERROR: "flush_icache_all" [drivers/misc/lkdtm/lkdtm.ko] undefined!
Signed-off-by: NOlof Johansson <olof@lixom.net>
Signed-off-by: NPaul Walmsley <paul.walmsley@sifive.com>

Running "stress-ng --enosys 4 -t 20 -v" showed a large number of kernel oops
with "Unable to handle kernel paging request at virtual address" message. This
happens when enosys stressor starts testing random non-valid syscalls.

I forgot to redirect any syscall below -1 to sys_ni_syscall.

With the patch kernel oops messages are gone while running stress-ng enosys
stressor.
Signed-off-by: NDavid Abdurachmanov <david.abdurachmanov@sifive.com>
Fixes: 5340627e ("riscv: add support for SECCOMP and SECCOMP_FILTER")
Signed-off-by: NPaul Walmsley <paul.walmsley@sifive.com>

When support for !MMU was added, the declaration of
__asm_copy_to_user() & __asm_copy_from_user() were #ifdefed
out hence their EXPORT_SYMBOL() give an error message like:
  .../riscv_ksyms.c:13:15: error: '__asm_copy_to_user' undeclared here
  .../riscv_ksyms.c:14:15: error: '__asm_copy_from_user' undeclared here

Since these symbols are not defined with !MMU it's wrong to export them.
Same for __clear_user() (even though this one is also declared in
include/asm-generic/uaccess.h and thus doesn't give an error message).

Fix this by doing the EXPORT_SYMBOL() directly where these symbols
are defined: inside lib/uaccess.S itself.

Fixes: 6bd33e1e ("riscv: fix compile failure with EXPORT_SYMBOL() & !MMU")
Reported-by: Nkbuild test robot <lkp@intel.com>
Cc: Christoph Hellwig <hch@lst.de>
Cc: Palmer Dabbelt <palmer@dabbelt.com>
Cc: Paul Walmsley <paul.walmsley@sifive.com>
Signed-off-by: NLuc Van Oostenryck <luc.vanoostenryck@gmail.com>
Signed-off-by: NPaul Walmsley <paul.walmsley@sifive.com>

The sifive_l2_cache.c is in no way related to RISC-V architecture
memory management.  It is a little stub driver working around the fact
that the EDAC maintainers prefer their drivers to be structured in a
certain way that doesn't fit the SiFive SOCs.

Move the file to drivers/soc and add a Kconfig option for it, as well
as the whole drivers/soc boilerplate for CONFIG_SOC_SIFIVE.

Fixes: a967a289 ("RISC-V: sifive_l2_cache: Add L2 cache controller driver for SiFive SoCs")
Signed-off-by: NChristoph Hellwig <hch@lst.de>
Reviewed-by: NBorislav Petkov <bp@suse.de>
[paul.walmsley@sifive.com: keep the MAINTAINERS change specific to the L2$ controller code]
Signed-off-by: NPaul Walmsley <paul.walmsley@sifive.com>

pfn_to_page & page_to_pfn depend on vmemmap being available before the calls
if kernel is configured with CONFIG_SPARSEMEM_VMEMMAP=y. This was caused
by NOMMU changes which moved vmemmap definition bellow functions definitions
calling pfn_to_page & page_to_pfn.

Noticed while compiled 5.5-rc2 kernel for Fedora/RISCV.

v2:
- Add a comment for vmemmap in source
Signed-off-by: NDavid Abdurachmanov <david.abdurachmanov@sifive.com>
Fixes: 6bd33e1e ("riscv: add nommu support")
Reviewed-by: NAnup Patel <anup@brainfault.org>
Signed-off-by: NPaul Walmsley <paul.walmsley@sifive.com>

This patch fixes that the sscratch register clearing in M-mode. It cleared
sscratch register in M-mode, but it should clear mscratch register. That will
cause kernel trap if the CPU core doesn't support S-mode when trying to access
sscratch.

Fixes: 9e806356 ("riscv: clear the instruction cache and all registers when booting")
Signed-off-by: NGreentime Hu <greentime.hu@sifive.com>
Reviewed-by: NAnup Patel <anup@brainfault.org>
Signed-off-by: NPaul Walmsley <paul.walmsley@sifive.com>

In Kconfig files, config options are written without the CONFIG_ prefix.

Fixes: 6bd33e1e ("riscv: add nommu support")
Signed-off-by: NAndreas Schwab <schwab@suse.de>
Reviewed-by: NAnup Patel <anup@brainfault.org>
Signed-off-by: NPaul Walmsley <paul.walmsley@sifive.com>

There are some typos in boot image header and riscv boot documentation.

Fix the typos.
Signed-off-by: NAtish Patra <atish.patra@wdc.com>
Reviewed-by: NPalmer Dabbelt <palmerdabbelt@google.com>
Link: https://lore.kernel.org/r/20191009010637.9955-1-atish.patra@wdc.comSigned-off-by: NJonathan Corbet <corbet@lwn.net>

RISC-V was missing a proper perf_arch_bpf_user_pt_regs macro for
CONFIG_PERF_EVENT builds.
Signed-off-by: NBjörn Töpel <bjorn.topel@gmail.com>
Signed-off-by: NDaniel Borkmann <daniel@iogearbox.net>
Link: https://lore.kernel.org/bpf/20191216091343.23260-10-bjorn.topel@gmail.com

Add missing uapi header the BPF_PROG_TYPE_PERF_EVENT programs by
exporting struct user_regs_struct instead of struct pt_regs which is
in-kernel only.
Signed-off-by: NBjörn Töpel <bjorn.topel@gmail.com>
Signed-off-by: NDaniel Borkmann <daniel@iogearbox.net>
Link: https://lore.kernel.org/bpf/20191216091343.23260-9-bjorn.topel@gmail.com

Instead of using emit_imm() and emit_jalr() which can expand to six
instructions, start using jal or auipc+jalr.
Signed-off-by: NBjörn Töpel <bjorn.topel@gmail.com>
Signed-off-by: NDaniel Borkmann <daniel@iogearbox.net>
Link: https://lore.kernel.org/bpf/20191216091343.23260-8-bjorn.topel@gmail.com

This commit makes sure that the JIT images is kept close to the kernel
text, so BPF calls can use relative calling with auipc/jalr or jal
instead of loading the full 64-bit address and jalr.

The BPF JIT image region is 128 MB before the kernel text.
Signed-off-by: NBjörn Töpel <bjorn.topel@gmail.com>
Signed-off-by: NDaniel Borkmann <daniel@iogearbox.net>
Link: https://lore.kernel.org/bpf/20191216091343.23260-7-bjorn.topel@gmail.com

Remove one addi, and instead use the offset part of jalr.
Signed-off-by: NBjörn Töpel <bjorn.topel@gmail.com>
Signed-off-by: NDaniel Borkmann <daniel@iogearbox.net>
Link: https://lore.kernel.org/bpf/20191216091343.23260-6-bjorn.topel@gmail.com

This commit add support for far (offset > 21b) jumps and exits.
Signed-off-by: NBjörn Töpel <bjorn.topel@gmail.com>
Signed-off-by: NDaniel Borkmann <daniel@iogearbox.net>
Reviewed-by: NLuke Nelson <lukenels@cs.washington.edu>
Link: https://lore.kernel.org/bpf/20191216091343.23260-5-bjorn.topel@gmail.com

Start use the emit_branch() function in the tail call emitter in order
to support far branching.
Signed-off-by: NBjörn Töpel <bjorn.topel@gmail.com>
Signed-off-by: NDaniel Borkmann <daniel@iogearbox.net>
Link: https://lore.kernel.org/bpf/20191216091343.23260-4-bjorn.topel@gmail.com