This patch converts bpf_prog_array from an array of prog pointers
to the array of struct bpf_prog_array_item elements.

This allows to save a cgroup storage pointer for each bpf program
efficiently attached to a cgroup.
Signed-off-by: NRoman Gushchin <guro@fb.com>
Cc: Alexei Starovoitov <ast@kernel.org>
Cc: Daniel Borkmann <daniel@iogearbox.net>
Acked-by: NMartin KaFai Lau <kafai@fb.com>
Signed-off-by: NDaniel Borkmann <daniel@iogearbox.net>

If a bpf program is using cgroup local storage, allocate
a bpf_cgroup_storage structure automatically on attaching the program
to a cgroup and save the pointer into the corresponding bpf_prog_list
entry.
Analogically, release the cgroup local storage on detaching
of the bpf program.
Signed-off-by: NRoman Gushchin <guro@fb.com>
Cc: Alexei Starovoitov <ast@kernel.org>
Cc: Daniel Borkmann <daniel@iogearbox.net>
Acked-by: NMartin KaFai Lau <kafai@fb.com>
Signed-off-by: NDaniel Borkmann <daniel@iogearbox.net>

This commit introduces the bpf_cgroup_storage_set() helper,
which will be used to pass a pointer to a cgroup storage
to the bpf helper.
Signed-off-by: NRoman Gushchin <guro@fb.com>
Cc: Alexei Starovoitov <ast@kernel.org>
Cc: Daniel Borkmann <daniel@iogearbox.net>
Acked-by: NMartin KaFai Lau <kafai@fb.com>
Signed-off-by: NDaniel Borkmann <daniel@iogearbox.net>

This commit introduces BPF_MAP_TYPE_CGROUP_STORAGE maps:
a special type of maps which are implementing the cgroup storage.

>From the userspace point of view it's almost a generic
hash map with the (cgroup inode id, attachment type) pair
used as a key.

The only difference is that some operations are restricted:
  1) a user can't create new entries,
  2) a user can't remove existing entries.

The lookup from userspace is o(log(n)).
Signed-off-by: NRoman Gushchin <guro@fb.com>
Cc: Alexei Starovoitov <ast@kernel.org>
Cc: Daniel Borkmann <daniel@iogearbox.net>
Acked-by: NMartin KaFai Lau <kafai@fb.com>
Signed-off-by: NDaniel Borkmann <daniel@iogearbox.net>

This commits extends existing bpf maps memory charging API
to support dynamic charging/uncharging.

This is required to account memory used by maps,
if all entries are created dynamically after
the map initialization.
Signed-off-by: NRoman Gushchin <guro@fb.com>
Cc: Alexei Starovoitov <ast@kernel.org>
Cc: Daniel Borkmann <daniel@iogearbox.net>
Acked-by: NMartin KaFai Lau <kafai@fb.com>
Signed-off-by: NDaniel Borkmann <daniel@iogearbox.net>

When check_alu_op() handles a BPF_MOV64 between two registers,
it calls check_reg_arg(DST_OP) on the dst register, marking it
as unbounded. If the src and dst register are the same, this
marks the src as unbounded, which can lead to unexpected errors
for further checks that rely on bounds info. For example:

	BPF_MOV64_IMM(BPF_REG_2, 0),
	BPF_MOV64_REG(BPF_REG_2, BPF_REG_2),
	BPF_ALU64_REG(BPF_ADD, BPF_REG_1, BPF_REG_2),
	BPF_MOV64_IMM(BPF_REG_0, 0),
	BPF_EXIT_INSN(),

Results in:

	"math between ctx pointer and register with unbounded
	min value is not allowed"

check_alu_op() now uses check_reg_arg(DST_OP_NO_MARK), and MOVs
that need to mark the dst register (MOVIMM, MOV32) do so.

Added a test case for MOV64 dst == src, and dst != src.
Signed-off-by: NArthur Fabre <afabre@cloudflare.com>
Acked-by: NEdward Cree <ecree@solarflare.com>
Signed-off-by: NDaniel Borkmann <daniel@iogearbox.net>

Add test to use get_socket_cookie() from BPF programs of types
BPF_PROG_TYPE_SOCK_OPS and BPF_PROG_TYPE_CGROUP_SOCK_ADDR.

The test attaches two programs to cgroup, runs TCP server and client in
the cgroup and checks that two operations are done properly on client
socket when user calls connect(2):

1. In BPF_CGROUP_INET6_CONNECT socket cookie is used as the key to write
   new value in a map for client socket.

2. In BPF_CGROUP_SOCK_OPS (BPF_SOCK_OPS_TCP_CONNECT_CB callback) the
   value written in "1." is found by socket cookie, since it's the same
   socket, and updated.

Finally the test verifies the value in the map.
Signed-off-by: NAndrey Ignatov <rdna@fb.com>
Acked-by: NYonghong Song <yhs@fb.com>
Signed-off-by: NDaniel Borkmann <daniel@iogearbox.net>

Add missing helper to bpf_helpers.h that is used in tests and samples.
Signed-off-by: NAndrey Ignatov <rdna@fb.com>
Acked-by: NYonghong Song <yhs@fb.com>
Signed-off-by: NDaniel Borkmann <daniel@iogearbox.net>

Sync bpf_get_socket_cookie() related bpf UAPI changes to tools/.
Signed-off-by: NAndrey Ignatov <rdna@fb.com>
Acked-by: NYonghong Song <yhs@fb.com>
Signed-off-by: NDaniel Borkmann <daniel@iogearbox.net>

bpf_get_socket_cookie() helper can be used to identify skb that
correspond to the same socket.

Though socket cookie can be useful in many other use-cases where socket is
available in program context. Specifically BPF_PROG_TYPE_CGROUP_SOCK_ADDR
and BPF_PROG_TYPE_SOCK_OPS programs can benefit from it so that one of
them can augment a value in a map prepared earlier by other program for
the same socket.

The patch adds support to call bpf_get_socket_cookie() from
BPF_PROG_TYPE_CGROUP_SOCK_ADDR and BPF_PROG_TYPE_SOCK_OPS.

It doesn't introduce new helpers. Instead it reuses same helper name
bpf_get_socket_cookie() but adds support to this helper to accept
`struct bpf_sock_addr` and `struct bpf_sock_ops`.

Documentation in bpf.h is changed in a way that should not break
automatic generation of markdown.
Signed-off-by: NAndrey Ignatov <rdna@fb.com>
Acked-by: NYonghong Song <yhs@fb.com>
Signed-off-by: NDaniel Borkmann <daniel@iogearbox.net>

'app' is dereferenced before used for the devlink trace point.
In case FW is buggy and sends a control message to a VF queue
we should make sure app is not NULL.
Reported-by: NDan Carpenter <dan.carpenter@oracle.com>
Signed-off-by: NJakub Kicinski <jakub.kicinski@netronome.com>
Reviewed-by: NDirk van der Merwe <dirk.vandermerwe@netronome.com>
Signed-off-by: NDaniel Borkmann <daniel@iogearbox.net>

run_lwt_bpf is called by bpf_{input/output/xmit}.
These functions are already protected by rcu_read_lock.
because lwtunnel_{input/output/xmit} holds rcu_read_lock
and then calls bpf_{input/output/xmit}.
So that rcu_read_lock in the run_lwt_bpf is unnecessary.
Signed-off-by: NTaehee Yoo <ap420073@gmail.com>
Acked-by: NYonghong Song <yhs@fb.com>
Signed-off-by: NDaniel Borkmann <daniel@iogearbox.net>

The seg6local LWT provides the End.DT6 action, which allows to
decapsulate an outer IPv6 header containing a Segment Routing Header
(SRH), full specification is available here:

https://tools.ietf.org/html/draft-filsfils-spring-srv6-network-programming-05

This patch adds this action now to the seg6local BPF
interface. Since it is not mandatory that the inner IPv6 header also
contains a SRH, seg6_bpf_srh_state has been extended with a pointer to
a possible SRH of the outermost IPv6 header. This helps assessing if the
validation must be triggered or not, and avoids some calls to
ipv6_find_hdr.

v3: s/1/true, s/0/false for boolean values
v2: - changed true/false -> 1/0
    - preempt_enable no longer called in first conditional block
Signed-off-by: NMathieu Xhonneux <m.xhonneux@gmail.com>
Signed-off-by: NDaniel Borkmann <daniel@iogearbox.net>

Commit 531b014e ("tools: bpf: make use of reallocarray") causes
a compiler error when building the perf tool in the linux-next tree.
Compile file tools/lib/bpf/libbpf.c on a FEDORA 28 installation with
gcc compiler version: gcc (GCC) 8.0.1 20180324 (Red Hat 8.0.1-0.20)
shows this error message:

  [root@p23lp27] # make V=1 EXTRA_CFLAGS="-Wp,-D_FORTIFY_SOURCE=2 -O2"
  [...]
  make -f /home6/tmricht/linux-next/tools/build/Makefile.build
  	dir=./util/scripting-engines obj=libperf
  libbpf.c: In function ‘bpf_object__elf_collect’:
  libbpf.c:811:15: error: ignoring return value of ‘strerror_r’,
  		declared with attribute warn_unused_result [-Werror=unused-result]
       strerror_r(-err, errmsg, sizeof(errmsg));
                 ^
  cc1: all warnings being treated as errors
  mv: cannot stat './.libbpf.o.tmp': No such file or directory
  /home6/tmricht/linux-next/tools/build/Makefile.build:96: recipe for target 'libbpf.o' failed

Replace all occurrences of strerror() by calls to strerror_r(). To
keep the compiler quiet also use the return value from strerror_r()
otherwise a 'variable set but not use' warning which is treated as
error terminates the compile.

Fixes: 531b014e ("tools: bpf: make use of reallocarray")
Suggested-by: NJakub Kicinski <jakub.kicinski@netronome.com>
Suggested-by: NDaniel Borkmann <daniel@iogearbox.net>
Signed-off-by: NThomas Richter <tmricht@linux.ibm.com>
Reviewed-by: NHendrik Brueckner <brueckner@linux.ibm.com>
Reviewed-by: NJakub Kicinski <jakub.kicinski@netronome.com>
Signed-off-by: NDaniel Borkmann <daniel@iogearbox.net>

Tobin C. Harding says:

====================
Recently BPF docs were converted to RST format. A couple of things were
missed:

 - Use 'index.rst' instead of 'README.rst'.  Although README.rst will
   work just fine it is more typical to keep the subdirectory indices
   in a file called 'index.rst'.

 - Integrate files Documentation/bpf/*.rst into build system using
   toctree in Documentation/bpf/index.rst

 - Include bpf/index in top level toctree so bpf is indexed in the main
   kernel docs.

 - Make anal change to heading format (inline with rest of Documentation/).
====================
Signed-off-by: NDaniel Borkmann <daniel@iogearbox.net>

The majority of files in the kernel documentation index use
capitalisation for all words, especially the shorter ones.  BPF docs
better fit in with the rest of the documentation if the heading is all
capitalised.

Capitalise document heading.
Signed-off-by: NTobin C. Harding <me@tobin.cc>
Signed-off-by: NDaniel Borkmann <daniel@iogearbox.net>

Recently bpf docs were converted to RST format.  The new files were not
added to the top level toctree.  This causes build system to emit a
warning of type

	WARNING: document isn't included in any toctree

Add bpf/index.rst to Documentation/index.rst
Signed-off-by: NTobin C. Harding <me@tobin.cc>
Signed-off-by: NDaniel Borkmann <daniel@iogearbox.net>

Recently bpf/ docs were converted to us RST format.  bp/index.rst was
created out of README but toctree was not added to include files within
Documentation/bpf/

Add toctree to Documentation/bpf/index.rst
Signed-off-by: NTobin C. Harding <me@tobin.cc>
Signed-off-by: NDaniel Borkmann <daniel@iogearbox.net>

Recently bpf/ docs were converted to use RST format.  'README.rst' was
created but in order to fit in with the Sphinx build system this file
should be named 'index.rst'.  Rename file, fixes to integrate into
Sphinx build system in following patches.

docs: Rename Documentation/bpf/README.rst to Documentation/bpf/index.rst
Signed-off-by: NTobin C. Harding <me@tobin.cc>
Signed-off-by: NDaniel Borkmann <daniel@iogearbox.net>

Jakub Kicinski says:

====================
This set converts xdpsock_user.c and xdp_fwd_user.c to use libbpf instead
of bpf_load.o.  First two patches are minor improvements to libbpf to make
the conversion (and use of libbpf in general) nicer.
====================
Acked-by: NYonghong Song <yhs@fb.com>
Signed-off-by: NDaniel Borkmann <daniel@iogearbox.net>

Convert xdpsock_user.c to use libbpf instead of bpf_load.o.
Signed-off-by: NJakub Kicinski <jakub.kicinski@netronome.com>
Reviewed-by: NQuentin Monnet <quentin.monnet@netronome.com>
Signed-off-by: NDaniel Borkmann <daniel@iogearbox.net>

Convert xdp_fwd_user.c to use libbpf instead of bpf_load.o.
Signed-off-by: NJakub Kicinski <jakub.kicinski@netronome.com>
Reviewed-by: NQuentin Monnet <quentin.monnet@netronome.com>
Signed-off-by: NDaniel Borkmann <daniel@iogearbox.net>

Allow users to find programs by section names.
Signed-off-by: NJakub Kicinski <jakub.kicinski@netronome.com>
Reviewed-by: NQuentin Monnet <quentin.monnet@netronome.com>
Signed-off-by: NDaniel Borkmann <daniel@iogearbox.net>

bpf_map__fd() handles NULL map gracefully and returns -EINVAL.
bpf_program__fd() and bpf_program__nth_fd() crash in this case.
Make the behaviour more consistent by validating prog pointer
as well.
Signed-off-by: NJakub Kicinski <jakub.kicinski@netronome.com>
Reviewed-by: NQuentin Monnet <quentin.monnet@netronome.com>
Signed-off-by: NDaniel Borkmann <daniel@iogearbox.net>

Jakub Kicinski says:

====================
This set is focused on improving the performance of perf events
reported from BPF offload.  Perf events can now be received on
packet data queues, which significantly improves the performance
(from total of 0.5 Msps to 5Msps per core).  To get to this
performance we need a fast path for control messages which will
operate on raw buffers and recycle them immediately.

Patch 5 replaces the map pointers for perf maps with map IDs.
We look the pointers up in a hashtable, anyway, to validate they
are correct, so there is no performance difference.  Map IDs
have the advantage of being easier to understand for users in
case of errors (we no longer print raw pointers to the logs).

Last patch improves info messages about map offload.
====================
Signed-off-by: NDaniel Borkmann <daniel@iogearbox.net>

FW can put constraints on map element size to maximize resource
use and efficiency.  When user attempts offload of a map which
does not fit into those constraints an informational message is
printed to kernel logs to inform user about the reason offload
failed.  Map offload does not have access to any advanced error
reporting like verifier log or extack.  There is also currently
no way for us to nicely expose the FW capabilities to user
space.  Given all those constraints we should make sure log
messages are as informative as possible.  Improve them.
Signed-off-by: NJakub Kicinski <jakub.kicinski@netronome.com>
Reviewed-by: NDirk van der Merwe <dirk.vandermerwe@netronome.com>
Signed-off-by: NDaniel Borkmann <daniel@iogearbox.net>

Record perf maps by map ID, not raw kernel pointer.  This helps
with debug messages, because printing pointers to logs is frowned
upon, and makes debug easier for the users, as map ID is something
they should be more familiar with.  Note that perf maps are offload
neutral, therefore IDs won't be orphaned.

While at it use a rate limited print helper for the error message.
Reported-by: NKees Cook <keescook@chromium.org>
Signed-off-by: NJakub Kicinski <jakub.kicinski@netronome.com>
Reviewed-by: NDirk van der Merwe <dirk.vandermerwe@netronome.com>
Reviewed-by: NQuentin Monnet <quentin.monnet@netronome.com>
Signed-off-by: NDaniel Borkmann <daniel@iogearbox.net>

Control queue is fairly low latency, and requires SKB allocations,
which means we can't even reach 0.5Msps with perf events.  Allow
perf events to be delivered to data queues.  This allows us to not
only use multiple queues, but also receive and deliver to user space
more than 5Msps per queue (Xeon E5-2630 v4 2.20GHz, no retpolines).
Signed-off-by: NJakub Kicinski <jakub.kicinski@netronome.com>
Reviewed-by: NDirk van der Merwe <dirk.vandermerwe@netronome.com>
Reviewed-by: NQuentin Monnet <quentin.monnet@netronome.com>
Signed-off-by: NDaniel Borkmann <daniel@iogearbox.net>

In preparation for SKB-less perf event handling make
nfp_bpf_event_output() take buffer address and length,
not SKB as parameters.
Signed-off-by: NJakub Kicinski <jakub.kicinski@netronome.com>
Reviewed-by: NDirk van der Merwe <dirk.vandermerwe@netronome.com>
Reviewed-by: NQuentin Monnet <quentin.monnet@netronome.com>
Signed-off-by: NDaniel Borkmann <daniel@iogearbox.net>

Port id 0xffffffff is reserved for control messages.  Allow reception
of messages with this id on data queues.  Hand off a raw buffer to
the higher layer code, without allocating SKB for max efficiency.
The RX handle can't modify or keep the buffer, after it returns
buffer is handed back over to the NIC RX free buffer list.
Signed-off-by: NJakub Kicinski <jakub.kicinski@netronome.com>
Reviewed-by: NDirk van der Merwe <dirk.vandermerwe@netronome.com>
Signed-off-by: NDaniel Borkmann <daniel@iogearbox.net>

Representor packets are received on PF queues with special metadata tag
for demux.  There is no reason to resolve the representor ID -> netdev
after the skb has been allocated.  Move the code, this will allow us to
handle special FW messages without SKB allocation overhead.
Signed-off-by: NJakub Kicinski <jakub.kicinski@netronome.com>
Reviewed-by: NDirk van der Merwe <dirk.vandermerwe@netronome.com>
Signed-off-by: NDaniel Borkmann <daniel@iogearbox.net>

To smoothly test BTF supported binary on samples/bpf,
let samples/bpf/Makefile probe llc, pahole and
llvm-objcopy for BPF support and use them
like tools/testing/selftests/bpf/Makefile
changed from the commit c0fa1b6c ("bpf: btf:
Add BTF tests").
Signed-off-by: NTaeung Song <treeze.taeung@gmail.com>
Acked-by: NMartin KaFai Lau <kafai@fb.com>
Reviewed-by: NJakub Kicinski <jakub.kicinski@netronome.com>
Signed-off-by: NDaniel Borkmann <daniel@iogearbox.net>

Define u_smp_rmb() and u_smp_wmb() to respective barrier instructions.
This ensures the processor will order accesses to queue indices against
accesses to queue ring entries.
Signed-off-by: NBrian Brooks <brian.brooks@linaro.org>
Acked-by: NBjörn Töpel <bjorn.topel@intel.com>
Signed-off-by: NDaniel Borkmann <daniel@iogearbox.net>

For untracked things of tools/bpf, add this.
Reviewed-by: NJakub Kicinski <jakub.kicinski@netronome.com>
Signed-off-by: NTaeung Song <treeze.taeung@gmail.com>
Signed-off-by: NDaniel Borkmann <daniel@iogearbox.net>

Adjust tcp_client.py and tcp_server.py to work with Python 3 by using
the print function, marking string literals as bytes, and using the
newer exception syntax. This should be functionally equivalent and
supports Python 3+.
Signed-off-by: NJeremy Cline <jcline@redhat.com>
Signed-off-by: NDaniel Borkmann <daniel@iogearbox.net>

Fix inconsistent IS_ERR and PTR_ERR in get_btf,
the proper pointer to be passed as argument is '*btf'

This issue was detected with the help of Coccinelle.

Fixes: 2d3feca8 ("bpf: btf: print map dump and lookup with btf info")
Signed-off-by: NYueHaibing <yuehaibing@huawei.com>
Acked-by: NDavid S. Miller <davem@davemloft.net>
Acked-by: NJakub Kicinski <jakub.kicinski@netronome.com>
Acked-by: NMartin KaFai Lau <kafai@fb.com>
Signed-off-by: NDaniel Borkmann <daniel@iogearbox.net>

Skbprio (SKB Priority Queue) is a queueing discipline that prioritizes packets
according to their skb->priority field. Under congestion, already-enqueued lower
priority packets will be dropped to make space available for higher priority
packets. Skbprio was conceived as a solution for denial-of-service defenses that
need to route packets with different priorities as a means to overcome DoS
attacks.

v5
*Do not reference qdisc_dev(sch)->tx_queue_len for setting limit. Instead set
default sch->limit to 64.

v4
*Drop Documentation/networking/sch_skbprio.txt doc file to move it to tc man
page for Skbprio, in iproute2.

v3
*Drop max_limit parameter in struct skbprio_sched_data and instead use
sch->limit.

*Reference qdisc_dev(sch)->tx_queue_len only once, during initialisation for
qdisc (previously being referenced every time qdisc changes).

*Move qdisc's detailed description from in-code to Documentation/networking.

*When qdisc is saturated, enqueue incoming packet first before dequeueing
lowest priority packet in queue - improves usage of call stack registers.

*Introduce and use overlimit stat to keep track of number of dropped packets.

v2
*Use skb->priority field rather than DS field. Rename queueing discipline as
SKB Priority Queue (previously Gatekeeper Priority Queue).

*Queueing discipline is made classful to expose Skbprio's internal priority
queues.
Signed-off-by: NNishanth Devarajan <ndev2021@gmail.com>
Reviewed-by: NSachin Paryani <sachin.paryani@gmail.com>
Reviewed-by: NCody Doucette <doucette@bu.edu>
Reviewed-by: NMichel Machado <michel@digirati.com.br>
Acked-by: NCong Wang <xiyou.wangcong@gmail.com>
Signed-off-by: NDavid S. Miller <davem@davemloft.net>

Certain PHY's have issues when operating in GBit slave mode and can
be forced to master mode. Examples are RTL8211C, also the Micrel PHY
driver has a DT setting to force master mode.
If two such chips are link partners the autonegotiation will fail.
Standard defines a self-clearing on read, latched-high bit to
indicate this error. Check this bit to inform the user.
Signed-off-by: NHeiner Kallweit <hkallweit1@gmail.com>
Signed-off-by: NDavid S. Miller <davem@davemloft.net>

Stephen Hemminger says:

====================
net whitespace cleanups

Ran script that I use to check for trailing whitespace and
blank lines at end of files across all files in net/ directory.
These are errors that checkpatch reports and git flags.

These are the resulting fixes broken up mostly by subsystem.
====================
Signed-off-by: NDavid S. Miller <davem@davemloft.net>

Several files have extra line at end of file.
Signed-off-by: NStephen Hemminger <stephen@networkplumber.org>
Signed-off-by: NDavid S. Miller <davem@davemloft.net>