Change the action related offload functions to take in flow_rule *
as input instead of flow_cls_offload * as input. The flow_rule
parts of flow_cls_offload is the only part that is used in any
case, and this is required for more conntrack offload patches
which will follow later.
Signed-off-by: NLouis Peens <louis.peens@corigine.com>
Signed-off-by: NYinjun Zhang <yinjun.zhang@corigine.com>
Signed-off-by: NSimon Horman <simon.horman@corigine.com>
Signed-off-by: NDavid S. Miller <davem@davemloft.net>

This is a small cleanup to pass in flow->rule to some of the compile
functions instead of extracting it every time. This is will also be
useful for conntrack patches later.
Signed-off-by: NLouis Peens <louis.peens@corigine.com>
Signed-off-by: NYinjun Zhang <yinjun.zhang@corigine.com>
Signed-off-by: NSimon Horman <simon.horman@corigine.com>
Signed-off-by: NDavid S. Miller <davem@davemloft.net>

Expose and refactor the match compilation functions so that they
can be invoked externally. Also update the functions so they can
be called multiple times with the results OR'd together. This is
applicable for the flows-merging scenario, in which there could be
overlapped and non-conflicting match fields. This will be used
in upcoming conntrack patches. This is safe to do in the in the
single call case as well since both unmasked_data and mask_data
gets initialised to 0.
Signed-off-by: NYinjun Zhang <yinjun.zhang@corigine.com>
Signed-off-by: NLouis Peens <louis.peens@corigine.com>
Signed-off-by: NSimon Horman <simon.horman@corigine.com>
Signed-off-by: NDavid S. Miller <davem@davemloft.net>

The current location of the callback delete can lead to a race
condition where deleting the callback requires a write_lock on
the nf_table, but at the same time another thread from netfilter
could have taken a read lock on the table before trying to offload.
Since the driver is taking a rtnl_lock this can lead into a deadlock
situation, where the netfilter offload will wait for the cls_flower
rtnl_lock to be released, but this cannot happen since this is
waiting for the nf_table read_lock to be released before it can
delete the callback.

Solve this by completely removing the nf_flow_table_offload_del_cb
call, as this will now be cleaned up by act_ct itself when cleaning
up the specific nf_table.

Fixes: 62268e78 ("nfp: flower-ct: add nft callback stubs")
Signed-off-by: NLouis Peens <louis.peens@corigine.com>
Signed-off-by: NYinjun Zhang <yinjun.zhang@corigine.com>
Signed-off-by: NSimon Horman <simon.horman@corigine.com>
Signed-off-by: NDavid S. Miller <davem@davemloft.net>

The nfp driver has rcu_read_lock()/rcu_read_unlock() pairs around XDP
program invocations. However, the actual lifetime of the objects referred
by the XDP program invocation is longer, all the way through to the call to
xdp_do_flush(), making the scope of the rcu_read_lock() too small.

While this is not actually an issue for the nfp driver because it doesn't
support XDP_REDIRECT (and thus doesn't call xdp_do_flush()), the
rcu_read_lock() is still unneeded. And With the addition of RCU annotations
to the XDP_REDIRECT map types that take bh execution into account, lockdep
even understands this to be safe, so there's really no reason to keep it
around.
Signed-off-by: NToke Høiland-Jørgensen <toke@redhat.com>
Signed-off-by: NDaniel Borkmann <daniel@iogearbox.net>
Reviewed-by: NSimon Horman <simon.horman@netronome.com>
Cc: oss-drivers@netronome.com
Link: https://lore.kernel.org/bpf/20210624160609.292325-15-toke@redhat.com

The nfp_fl_ct_add_flow() function can fail so we need to check for
failure.

Fixes: 95255017 ("nfp: flower-ct: add nft flows to nft list")
Signed-off-by: NDan Carpenter <dan.carpenter@oracle.com>
Reviewed-by: NLouis Peens <louis.peens@corigine.com>
Reviewed-by: NSimon Horman <simon.horman@corigine.com>
Signed-off-by: NDavid S. Miller <davem@davemloft.net>

Fill in code stub to check that the flow actions are valid for
merge. The actions of the flow X should not conflict with the
matches of flow X+1. For now this check is quite strict and
set_actions are very limited, will need to update this when
NAT support is added.
Signed-off-by: NLouis Peens <louis.peens@corigine.com>
Signed-off-by: NYinjun Zhang <yinjun.zhang@corigine.com>
Signed-off-by: NSimon Horman <simon.horman@corigine.com>
Signed-off-by: NDavid S. Miller <davem@davemloft.net>

Fill in check_meta stub to check that ct_metadata action fields in
the nft flow matches the ct_match data of the post_ct flow.
Signed-off-by: NLouis Peens <louis.peens@corigine.com>
Signed-off-by: NYinjun Zhang <yinjun.zhang@corigine.com>
Signed-off-by: NSimon Horman <simon.horman@corigine.com>
Signed-off-by: NDavid S. Miller <davem@davemloft.net>

Replace merge check stub code with the actual implementation. This
checks that the match parts of two tc flows does not conflict.
Only overlapping keys needs to be checked, and only the narrowest
masked parts needs to be checked, so each key is masked with the
AND'd result of both masks before comparing.
Signed-off-by: NLouis Peens <louis.peens@corigine.com>
Signed-off-by: NYinjun Zhang <yinjun.zhang@corigine.com>
Signed-off-by: NSimon Horman <simon.horman@corigine.com>
Signed-off-by: NDavid S. Miller <davem@davemloft.net>

Add in the code to merge the tc_merge objects with the flows
received from nft. At the moment flows are just merged blindly
as the validity check functions are stubbed out, this will
be populated in follow-up patches.
Signed-off-by: NLouis Peens <louis.peens@corigine.com>
Signed-off-by: NYinjun Zhang <yinjun.zhang@corigine.com>
Signed-off-by: NSimon Horman <simon.horman@corigine.com>
Signed-off-by: NDavid S. Miller <davem@davemloft.net>

Add table and struct to save the result of the three-way merge
between pre_ct,post_ct, and nft flows. Merging code is to be
added in follow-up patches.
Signed-off-by: NLouis Peens <louis.peens@corigine.com>
Signed-off-by: NYinjun Zhang <yinjun.zhang@corigine.com>
Signed-off-by: NSimon Horman <simon.horman@corigine.com>
Signed-off-by: NDavid S. Miller <davem@davemloft.net>

The nft flow will be destroyed after offload cb returns. This means
we need save a full copy of it since it can be referenced through
other paths other than just the offload cb, for example when a new
pre_ct or post_ct entry is added, and it needs to be merged with
an existing nft entry.
Signed-off-by: NYinjun Zhang <yinjun.zhang@corigine.com>
Signed-off-by: NLouis Peens <louis.peens@corigine.com>
Signed-off-by: NSimon Horman <simon.horman@corigine.com>
Signed-off-by: NDavid S. Miller <davem@davemloft.net>

Implement code to add and remove nft flows to the relevant list.
Registering and deregistering the callback function for the nft
table is quite complicated. The safest is to delete the callback
on the removal of the last pre_ct flow. This is because if this
is also the latest pre_ct flow in software it means that this
specific nft table will be freed, so there will not be a later
opportunity to do this. Another place where it looks possible
to delete the callback is when the last nft_flow is deleted,
but this happens under the flow_table lock, which is also taken
when deregistering the callback, leading to a deadlock situation.

This means the final solution here is to delete the callback
when removing the last pre_ct flow, and then clean up any
remaining nft_flow entries which may still be present, since
there will never be a callback now to do this, leaving them
orphaned if not cleaned up here as well.
Signed-off-by: NLouis Peens <louis.peens@corigine.com>
Signed-off-by: NYinjun Zhang <yinjun.zhang@corigine.com>
Signed-off-by: NSimon Horman <simon.horman@corigine.com>
Signed-off-by: NDavid S. Miller <davem@davemloft.net>

Add register/unregister of the nft callback. For now just add
stub code to accept the flows, but don't do anything with it.
Decided to accept the flows since netfilter will keep on trying
to offload a flow if it was rejected, which is quite noisy.
Follow-up patches will start implementing the functions to add
nft flows to the relevant tables.
Signed-off-by: NLouis Peens <louis.peens@corigine.com>
Signed-off-by: NYinjun Zhang <yinjun.zhang@corigine.com>
Signed-off-by: NSimon Horman <simon.horman@corigine.com>
Signed-off-by: NDavid S. Miller <davem@davemloft.net>

Add functions to handle delete flow callbacks for ct flows. Also
accept the flows for offloading by returning 0 instead of -EOPNOTSUPP.
Flows will still not actually be offloaded to hw, but at this point
it's difficult to not accept the flows and also exercise the cleanup
paths properly. Traffic will still be handled safely through the
fallback path.
Signed-off-by: NLouis Peens <louis.peens@corigine.com>
Signed-off-by: NYinjun Zhang <yinjun.zhang@corigine.com>
Signed-off-by: NSimon Horman <simon.horman@corigine.com>
Signed-off-by: NDavid S. Miller <davem@davemloft.net>

Using list_move() instead of list_del() + list_add() in nfp_cppcore.c.
Reported-by: NHulk Robot <hulkci@huawei.com>
Signed-off-by: NBaokun Li <libaokun1@huawei.com>
Reviewed-by: NSimon Horman <simon.horman@corigine.com>
Signed-off-by: NDavid S. Miller <davem@davemloft.net>

Add merging of pre/post_ct flow rules into the tc_merge table.
Pre_ct flows needs to be merge with post_ct flows and vice versa.

This needs to be done for all flows in the same zone table, as well
as with the wc_zone_table, which is for flows masking out ct_zone
info.

Cleanup is happening when all the tables are cleared up and prints
a warning traceback as this is not expected in the final version.
At this point we are not actually returning success for the offload,
so we do not get any delete requests for flows, so we can't delete
them that way yet. This means that cleanup happens in what would
usually be an exception path.
Signed-off-by: NLouis Peens <louis.peens@corigine.com>
Signed-off-by: NYinjun Zhang <yinjun.zhang@corigine.com>
Signed-off-by: NSimon Horman <simon.horman@corigine.com>
Signed-off-by: NDavid S. Miller <davem@davemloft.net>

Add the table required to store the merge result of pre_ct and post_ct
flows. This is just the initial setup and teardown of the table,
the implementation will be in follow-up patches.
Signed-off-by: NLouis Peens <louis.peens@corigine.com>
Signed-off-by: NYinjun Zhang <yinjun.zhang@corigine.com>
Signed-off-by: NSimon Horman <simon.horman@corigine.com>
Signed-off-by: NDavid S. Miller <davem@davemloft.net>

Add a hashtable which contains entries to map flow cookies to ct
flow entries. Currently the entries are added and not used, but
follow-up patches will use this for stats updates and flow deletes.
Signed-off-by: NLouis Peens <louis.peens@corigine.com>
Signed-off-by: NYinjun Zhang <yinjun.zhang@corigine.com>
Signed-off-by: NSimon Horman <simon.horman@corigine.com>
Signed-off-by: NDavid S. Miller <davem@davemloft.net>

This commit starts adding the structures and lists that will
be used in follow up commits to enable offloading of conntrack.
Some stub functions are also introduced as placeholders by
this commit.
Signed-off-by: NLouis Peens <louis.peens@corigine.com>
Signed-off-by: NYinjun Zhang <yinjun.zhang@corigine.com>
Signed-off-by: NSimon Horman <simon.horman@corigine.com>
Signed-off-by: NDavid S. Miller <davem@davemloft.net>

Start populating the pre/post_ct handler functions. Add a zone entry
to the zone table, based on the zone information from the flow. In
the case of a post_ct flow which has a wildcarded match on the zone
create a special entry.
Signed-off-by: NLouis Peens <louis.peens@corigine.com>
Signed-off-by: NYinjun Zhang <yinjun.zhang@corigine.com>
Signed-off-by: NSimon Horman <simon.horman@corigine.com>
Signed-off-by: NDavid S. Miller <davem@davemloft.net>

Add initial zone table to nfp_flower_priv. This table will be used
to store all the information required to offload conntrack.
Signed-off-by: NLouis Peens <louis.peens@corigine.com>
Signed-off-by: NYinjun Zhang <yinjun.zhang@corigine.com>
Signed-off-by: NSimon Horman <simon.horman@corigine.com>
Signed-off-by: NDavid S. Miller <davem@davemloft.net>

Add checks to see if a flow is a conntrack flow we can potentially
handle. Just stub out the handling the different conntrack flows.
Signed-off-by: NLouis Peens <louis.peens@corigine.com>
Signed-off-by: NYinjun Zhang <yinjun.zhang@corigine.com>
Signed-off-by: NSimon Horman <simon.horman@corigine.com>
Signed-off-by: NDavid S. Miller <davem@davemloft.net>

This is in preparation for conntrack offload support which makes
used of different chains. Add explicit checks for conntrack and
non-zero chains in the add_offload path.
Signed-off-by: NLouis Peens <louis.peens@corigine.com>
Signed-off-by: NYinjun Zhang <yinjun.zhang@corigine.com>
Signed-off-by: NSimon Horman <simon.horman@corigine.com>
Signed-off-by: NDavid S. Miller <davem@davemloft.net>

In preparation to enable -Wimplicit-fallthrough for Clang, fix a warning
by explicitly adding a break statement instead of letting the code fall
through to the next case.

Link: https://github.com/KSPP/linux/issues/115Acked-by: NSimon Horman <simon.horman@netronome.com>
Signed-off-by: NGustavo A. R. Silva <gustavoars@kernel.org>

Fixes the following W=1 kernel build warning(s):

 drivers/net/ethernet/netronome/nfp/ccm_mbox.c:52: warning: expecting prototype for struct nfp_ccm_mbox_skb_cb. Prototype was for struct nfp_ccm_mbox_cmsg_cb instead
 drivers/net/ethernet/netronome/nfp/flower/tunnel_conf.c:35: warning: expecting prototype for struct nfp_tun_pre_run_rule. Prototype was for struct nfp_tun_pre_tun_rule instead
 drivers/net/ethernet/netronome/nfp/nfpcore/nfp_nffw.c:38: warning: expecting prototype for NFFW_INFO_VERSION history(). Prototype was for NFFW_INFO_VERSION_CURRENT() instead

Cc: Simon Horman <simon.horman@netronome.com>
Cc: Jakub Kicinski <kuba@kernel.org>
Signed-off-by: NYang Shen <shenyang39@huawei.com>
Reviewed-by: NSimon Horman <simon.horman@netronome.com>
Signed-off-by: NDavid S. Miller <davem@davemloft.net>

The number of lanes of devlink port should be correctly initialized
when registering the port, so that the input check when running
"devlink port split <port> count <N>" can pass.

Fixes: a21cf0a8 ("devlink: Add a new devlink port lanes attribute and pass to netlink")
Signed-off-by: NYinjun Zhang <yinjun.zhang@corigine.com>
Signed-off-by: NLouis Peens <louis.peens@corigine.com>
Signed-off-by: NSimon Horman <simon.horman@netronome.com>
Signed-off-by: NDavid S. Miller <davem@davemloft.net>

Allow hardware offload of a policer action attached to a matchall filter
which enforces a packets-per-second rate-limit.

e.g.
tc filter add dev tap1 parent ffff: u32 match \
        u32 0 0 police pkts_rate 3000 pkts_burst 1000
Signed-off-by: NPeng Zhang <peng.zhang@corigine.com>
Signed-off-by: NBaowen Zheng <baowen.zheng@corigine.com>
Signed-off-by: NSimon Horman <simon.horman@netronome.com>
Signed-off-by: NLouis Peens <louis.peens@netronome.com>
Signed-off-by: NDavid S. Miller <davem@davemloft.net>

A merge hint message needs some time to process before the merged
flow actually reaches the firmware, during which we may get duplicate
merge hints if there're more than one packet that hit the pre-merged
flow. And processing duplicate merge hints will cost extra host_ctx's
which are a limited resource.

Avoid the duplicate merge by using hash table to store the sub_flows
to be merged.

Fixes: 8af56f40 ("nfp: flower: offload merge flows")
Signed-off-by: NYinjun Zhang <yinjun.zhang@corigine.com>
Signed-off-by: NLouis Peens <louis.peens@corigine.com>
Signed-off-by: NSimon Horman <simon.horman@netronome.com>
Signed-off-by: NDavid S. Miller <davem@davemloft.net>

In nfp_bpf_ctrl_msg_rx, if
nfp_ccm_get_type(skb) == NFP_CCM_TYPE_BPF_BPF_EVENT is true, the skb
will be freed. But the skb is still used by nfp_ccm_rx(&bpf->ccm, skb).

My patch adds a return when the skb was freed.

Fixes: bcf0cafa ("nfp: split out common control message handling code")
Signed-off-by: NLv Yunlong <lyl2019@mail.ustc.edu.cn>
Reviewed-by: NJakub Kicinski <kuba@kernel.org>
Signed-off-by: NDavid S. Miller <davem@davemloft.net>

struct sk_buff has been declared. Remove the duplicate.
Signed-off-by: NWan Jiabing <wanjiabing@vivo.com>
Reviewed-by: NSimon Horman <simon.horman@netronome.com>
Signed-off-by: NDavid S. Miller <davem@davemloft.net>

The nfp_pr_et function is nearly identical to ethtool_sprintf except for
the fact that it passes the pointer by value and as a return whereas
ethtool_sprintf passes it as a pointer.

Since they are so close just update nfp to make use of ethtool_sprintf
Reviewed-by: NSimon Horman <simon.horman@netronome.com>
Signed-off-by: NAlexander Duyck <alexanderduyck@fb.com>
Signed-off-by: NDavid S. Miller <davem@davemloft.net>

pre_tun_rule flows does not follow the usual add-flow path, instead
they are used to update the pre_tun table on the firmware. This means
that if the mask-id gets allocated here the firmware will never see the
"NFP_FL_META_FLAG_MANAGE_MASK" flag for the specific mask id, which
triggers the allocation on the firmware side. This leads to the firmware
mask being corrupted and causing all sorts of strange behaviour.

Fixes: f12725d9 ("nfp: flower: offload pre-tunnel rules")
Signed-off-by: NLouis Peens <louis.peens@corigine.com>
Signed-off-by: NSimon Horman <simon.horman@netronome.com>
Signed-off-by: NDavid S. Miller <davem@davemloft.net>

Differentiate between ipv4 and ipv6 flows when configuring the pre_tunnel
table to prevent them trampling each other in the table.

Fixes: 78346160 ("nfp: flower: update flow merge code to support IPv6 tunnels")
Signed-off-by: NLouis Peens <louis.peens@corigine.com>
Signed-off-by: NSimon Horman <simon.horman@netronome.com>
Signed-off-by: NDavid S. Miller <davem@davemloft.net>

There are some pre_tunnel flows combinations which are incorrectly being
offloaded without proper support, fix these.

- Matching on MPLS is not supported for pre_tun.
- Match on IPv4/IPv6 layer must be present.
- Destination MAC address must match pre_tun.dev MAC

Fixes: 120ffd84 ("nfp: flower: verify pre-tunnel rules")
Signed-off-by: NLouis Peens <louis.peens@corigine.com>
Signed-off-by: NSimon Horman <simon.horman@netronome.com>
Signed-off-by: NDavid S. Miller <davem@davemloft.net>

A follow-up patch will allow users to configures packet-per-second policing
in the software datapath. In preparation for this, teach all drivers that
support offload of the policer action to reject such configuration as
currently none of them support it.
Signed-off-by: NBaowen Zheng <baowen.zheng@corigine.com>
Signed-off-by: NSimon Horman <simon.horman@netronome.com>
Signed-off-by: NLouis Peens <louis.peens@netronome.com>
Signed-off-by: NDavid S. Miller <davem@davemloft.net>

A subsequent patch will add additional atomic operations. These new
operations will use the same opcode field as the existing XADD, with
the immediate discriminating different operations.

In preparation, rename the instruction mode BPF_ATOMIC and start
calling the zero immediate BPF_ADD.

This is possible (doesn't break existing valid BPF progs) because the
immediate field is currently reserved MBZ and BPF_ADD is zero.

All uses are removed from the tree but the BPF_XADD definition is
kept around to avoid breaking builds for people including kernel
headers.
Signed-off-by: NBrendan Jackman <jackmanb@google.com>
Signed-off-by: NAlexei Starovoitov <ast@kernel.org>
Acked-by: NBjörn Töpel <bjorn.topel@gmail.com>
Link: https://lore.kernel.org/bpf/20210114181751.768687-5-jackmanb@google.com

Introduce xdp_prepare_buff utility routine to initialize per-descriptor
xdp_buff fields (e.g. xdp_buff pointers). Rely on xdp_prepare_buff() in
all XDP capable drivers.
Signed-off-by: NLorenzo Bianconi <lorenzo@kernel.org>
Signed-off-by: NDaniel Borkmann <daniel@iogearbox.net>
Reviewed-by: NAlexander Duyck <alexanderduyck@fb.com>
Acked-by: NJesper Dangaard Brouer <brouer@redhat.com>
Acked-by: NJohn Fastabend <john.fastabend@gmail.com>
Acked-by: NShay Agroskin <shayagr@amazon.com>
Acked-by: NMartin Habets <habetsm.xilinx@gmail.com>
Acked-by: NCamelia Groza <camelia.groza@nxp.com>
Acked-by: NMarcin Wojtas <mw@semihalf.com>
Link: https://lore.kernel.org/bpf/45f46f12295972a97da8ca01990b3e71501e9d89.1608670965.git.lorenzo@kernel.orgSigned-off-by: NAlexei Starovoitov <ast@kernel.org>

Introduce xdp_init_buff utility routine to initialize xdp_buff fields
const over NAPI iterations (e.g. frame_sz or rxq pointer). Rely on
xdp_init_buff in all XDP capable drivers.
Signed-off-by: NLorenzo Bianconi <lorenzo@kernel.org>
Signed-off-by: NDaniel Borkmann <daniel@iogearbox.net>
Reviewed-by: NAlexander Duyck <alexanderduyck@fb.com>
Acked-by: NJesper Dangaard Brouer <brouer@redhat.com>
Acked-by: NJohn Fastabend <john.fastabend@gmail.com>
Acked-by: NShay Agroskin <shayagr@amazon.com>
Acked-by: NMartin Habets <habetsm.xilinx@gmail.com>
Acked-by: NCamelia Groza <camelia.groza@nxp.com>
Acked-by: NMarcin Wojtas <mw@semihalf.com>
Link: https://lore.kernel.org/bpf/7f8329b6da1434dc2b05a77f2e800b29628a8913.1608670965.git.lorenzo@kernel.orgSigned-off-by: NAlexei Starovoitov <ast@kernel.org>

All UDP tunnel port management is now routed via udp_tunnel_nic
infra directly. Remove the old callbacks.
Reviewed-by: NAlexander Duyck <alexanderduyck@fb.com>
Reviewed-by: NJacob Keller <jacob.e.keller@intel.com>
Signed-off-by: NJakub Kicinski <kuba@kernel.org>