Remove one function call whose result was not used anywhere.
Signed-off-by: NIoana Radulescu <ruxandra.radulescu@nxp.com>
Signed-off-by: NIoana Ciornei <ioana.ciornei@nxp.com>
Signed-off-by: NDavid S. Miller <davem@davemloft.net>

Starting with firmware version MC10.18.0, a new counter for in flight
Tx frames is offered. Use it when bringing down the interface to
determine when all pending Tx frames have been processed by hardware
instead of sleeping a fixed amount of time.
Signed-off-by: NIoana Radulescu <ruxandra.radulescu@nxp.com>
Signed-off-by: NDavid S. Miller <davem@davemloft.net>

Starting with firmware version MC10.18.0, we have support for
L2 flow control. Asymmetrical configuration (Rx or Tx only) is
supported, but not pause frame autonegotioation.

Pause frame configuration is done via ethtool. By default, we start
with flow control enabled on both Rx and Tx. Changes are propagated
to hardware through firmware commands, using two flags (PAUSE,
ASYM_PAUSE) to specify Rx and Tx pause configuration, as follows:

PAUSE | ASYM_PAUSE | Rx pause | Tx pause
----------------------------------------
  0   |     0      | disabled | disabled
  0   |     1      | disabled | enabled
  1   |     0      | enabled  | enabled
  1   |     1      | enabled  | disabled

The hardware can automatically send pause frames when the number
of buffers in the pool goes below a predefined threshold. Due to
this, flow control is incompatible with Rx frame queue taildrop
(both mechanisms target the case when processing of ingress
frames can't keep up with the Rx rate; for large frames, the number
of buffers in the pool may never get low enough to trigger pause
frames as long as taildrop is enabled). So we set pause frame
generation and Rx FQ taildrop as mutually exclusive.
Signed-off-by: NIoana Radulescu <ruxandra.radulescu@nxp.com>
Reviewed-by: NAndrew Lunn <andrew@lunn.ch>
Signed-off-by: NDavid S. Miller <davem@davemloft.net>

Whenever a link state change occurs, we get notified and save
the new link settings in the device's private data. In ethtool
get_link_ksettings, use the stored state instead of interrogating
the firmware each time.
Signed-off-by: NIoana Radulescu <ruxandra.radulescu@nxp.com>
Reviewed-by: NAndrew Lunn <andrew@lunn.ch>
Signed-off-by: NDavid S. Miller <davem@davemloft.net>

Implement mqprio qdisc support by mapping traffic classes to
different hardware enqueue priorities. The maximum number of
supported traffic classes is an attribute of each DPNI object.

The traffic classes map to hardware priorities from highest (0)
to lowest (highest prio number). The skb priority information
received from the stack is used to select the hardware Tx queue
on which to enqueue the frame.
Signed-off-by: NIoana Radulescu <ruxandra.radulescu@nxp.com>
Signed-off-by: NBogdan Purcareata <bogdan.purcareata@nxp.com>
Signed-off-by: NDavid S. Miller <davem@davemloft.net>

DPNI objects can have multiple traffic classes, as reflected by
the num_tc attribute. Until now we ignored its value and only
used traffic class 0.

This patch adds support for multiple Tx traffic classes; we have
num_queues x num_tcs hardware queues available for each interface.
Signed-off-by: NIoana Radulescu <ruxandra.radulescu@nxp.com>
Signed-off-by: NDavid S. Miller <davem@davemloft.net>

Move the code configuring xps on the netdev TX queues to a
separate function. A subsequent patch will need to call
this in another context as well.
Signed-off-by: NIoana Radulescu <ruxandra.radulescu@nxp.com>
Signed-off-by: NDavid S. Miller <davem@davemloft.net>

The driver is using netdev_alloc_frag() for allocation in the
->ndo_start_xmit() path. That one is always invoked in a BH disabled
region so we could also use napi_alloc_frag().

Use napi_alloc_frag() for skb allocation.

Cc: Ioana Radulescu <ruxandra.radulescu@nxp.com>
Acked-by: NIoana Radulescu <ruxandra.radulescu@nxp.com>
Signed-off-by: NSebastian Andrzej Siewior <bigeasy@linutronix.de>
Signed-off-by: NDavid S. Miller <davem@davemloft.net>

According to the comment, the preempt_disable() statement is required
due to synchronisation in napi_alloc_frag(). The awful truth is that
local_bh_disable() is required because otherwise the NAPI poll callback
can be invoked while the open function setup buffers. This isn't
unlikely since the dpaa2 provides multiple devices.

The usage of napi_alloc_frag() has been removed in commit

 27c87486 ("dpaa2-eth: Use a single page per Rx buffer")

which means that the comment is not accurate and the preempt_disable()
statement is not required.

Remove the outdated comment and the no longer required
preempt_disable().

Cc: Ioana Radulescu <ruxandra.radulescu@nxp.com>
Acked-by: NIoana Radulescu <ruxandra.radulescu@nxp.com>
Signed-off-by: NSebastian Andrzej Siewior <bigeasy@linutronix.de>
Signed-off-by: NDavid S. Miller <davem@davemloft.net>

Use PTR_ERR_OR_ZERO instead of PTR_ERR in cases where
zero is a valid input. Reported by smatch.
Signed-off-by: NIoana Radulescu <ruxandra.radulescu@nxp.com>
Signed-off-by: NDavid S. Miller <davem@davemloft.net>

This reverts commit f8b99585.

The reverted change instructed the QMan hardware block to fetch
RX frame annotation and beginning of frame data to cache before
the core would read them.

It turns out that in rare cases, it's possible that a QMan
stashing transaction is delayed long enough such that, by the time
it gets executed, the frame in question had already been dequeued
by the core and software processing began on it. If the core
manages to unmap the frame buffer _before_ the stashing transaction
is executed, an SMMU exception will be raised.

Unfortunately there is no easy way to work around this while keeping
the performance advantages brought by QMan stashing, so disable
it altogether.
Signed-off-by: NIoana Radulescu <ruxandra.radulescu@nxp.com>
Signed-off-by: NDavid S. Miller <davem@davemloft.net>

On platforms that lack a TCAM (like LS1088A), masking of
flow steering keys is not supported. Until now we didn't
offer flow steering capabilities at all on these platforms,
since our driver implementation configured a "comprehensive"
FS key (containing all supported header fields), with masks
used to ignore the fields not present in the rules provided
by the user.

We now allow ethtool rules that share a common key (i.e. have
the same header fields). The FS key is now kept in the driver
private data and initialized when the first rule is added to
an empty table, rather than at probe time. If a rule with a new
composition key is wanted, the user must first manually delete
all previous rules.

When building a FS table entry to pass to firmware, we still use
the old building algorithm, which assumes an all-supported-fields
key, and later collapse the fields which aren't actually needed.

Masked rules are not supported; if provided, the mask value
will be ignored. For firmware versions older than MC10.7.0
(that only offer the legacy ABIs for configuring distribution
keys) flow steering without masking support remains unavailable.
Signed-off-by: NIoana Radulescu <ruxandra.radulescu@nxp.com>
Signed-off-by: NDavid S. Miller <davem@davemloft.net>

Introduce an internal id bitfield to uniquely identify header fields
supported by the Rx distribution keys. For the hash key, add a
conversion from the RXH_* bitmask provided by ethtool to the internal
ids.
Signed-off-by: NIoana Radulescu <ruxandra.radulescu@nxp.com>
Signed-off-by: NDavid S. Miller <davem@davemloft.net>

Add two macros to simplify reading DPNI options.
Signed-off-by: NIoana Radulescu <ruxandra.radulescu@nxp.com>
Signed-off-by: NDavid S. Miller <davem@davemloft.net>

Set the Rx flow classification enable flag only if key config
operation is successful.

Fixes 3f9b5c9 ("dpaa2-eth: Configure Rx flow classification key")
Signed-off-by: NIoana Radulescu <ruxandra.radulescu@nxp.com>
Signed-off-by: NDavid S. Miller <davem@davemloft.net>

Take advantage of the software Rx batching by using
netif_receive_skb_list instead of napi_gro_receive.
Signed-off-by: NIoana Ciornei <ioana.ciornei@nxp.com>
Signed-off-by: NDavid S. Miller <davem@davemloft.net>

It might happen that Tx conf acknowledges a frame before it was
subscribed in bql, as subscribing was previously done after the enqueue
operation.

This patch moves the netdev_tx_sent_queue call before the actual frame
enqueue, so that this can never happen.

Fixes: 569dac6a ("dpaa2-eth: bql support")
Signed-off-by: NIoana Ciornei <ioana.ciornei@nxp.com>
Signed-off-by: NDavid S. Miller <davem@davemloft.net>

Make sure we don't try to enqueue XDP_REDIRECT frames to an
inexistent FQ.

While it is guaranteed not to have more than one queue per core,
having fewer queues than CPUs on an interface is a valid
configuration.

Fixes: d678be1d ("dpaa2-eth: add XDP_REDIRECT support")
Reported-by: NJesper Dangaard Brouer <brouer@redhat.com>
Signed-off-by: NIoana Radulescu <ruxandra.radulescu@nxp.com>
Signed-off-by: NDavid S. Miller <davem@davemloft.net>

Implement support for the XDP_REDIRECT action.

The redirected frame is transmitted and confirmed on the regular Tx/Tx
conf queues. Frame is marked with the "XDP" type in the software
annotation, since it requires special treatment.

We don't have good hardware support for TX batching, so the
XDP_XMIT_FLUSH flag doesn't make a difference for now; ndo_xdp_xmit
performs the actual Tx operation on the spot.
Signed-off-by: NIoana Ciornei <ioana.ciornei@nxp.com>
Signed-off-by: NIoana Radulescu <ruxandra.radulescu@nxp.com>
Signed-off-by: NDavid S. Miller <davem@davemloft.net>

We write different metadata information in the software annotation
area of Tx frames, depending on frame type. Make this more explicit
by introducing a type field and separate structures for single buffer
and scatter-gather frames.
Signed-off-by: NIoana Radulescu <ruxandra.radulescu@nxp.com>
Signed-off-by: NIoana Ciornei <ioana.ciornei@nxp.com>
Signed-off-by: NDavid S. Miller <davem@davemloft.net>

Configure the amount of 64 bytes of frame, annotation and context data
that will be cache stashed for a specific frame queue.  Since the frame
context is not used, configure that only 64 bytes of frame data and 64
bytes of annotation will be stashed.
Signed-off-by: NIoana Ciornei <ioana.ciornei@nxp.com>
Acked-by: NDavid S. Miller <davem@davemloft.net>
Signed-off-by: NLi Yang <leoyang.li@nxp.com>

Starting with MC10.14.0, dpaa2_io_service_enqueue_fq() API is
functional. Since there are a number of cases where it offers
better performance compared to the currently used enqueue
function, switch to it for firmware versions that support it.
Signed-off-by: NIoana Radulescu <ruxandra.radulescu@nxp.com>
Signed-off-by: NDavid S. Miller <davem@davemloft.net>

While in NAPI context, free skbs by calling napi_consume_skb()
instead of dev_kfree_skb(), to take advantage of the bulk freeing
mechanism.
Signed-off-by: NIoana Radulescu <ruxandra.radulescu@nxp.com>
Signed-off-by: NDavid S. Miller <davem@davemloft.net>

Instead of allocating page fragments via the network stack,
use the page allocator directly. For now, we consume one page
for each Rx buffer.

With the new memory model we are free to consider adding more
XDP support.

Performance decreases slightly in some IP forwarding cases.
No visible effect on termination traffic. The driver memory
footprint increases as a result of this change, but it is
still small enough to not really matter.

Another side effect is that now Rx buffer alignment requirements
are naturally satisfied without any additional actions needed.
Remove alignment related code, except in the buffer layout
information conveyed to MC, as hardware still needs to know the
alignment value we guarantee.
Signed-off-by: NIoana Ciornei <ioana.ciornei@nxp.com>
Signed-off-by: NIoana Radulescu <ruxandra.radulescu@nxp.com>
Signed-off-by: NDavid S. Miller <davem@davemloft.net>

Export detailed driver counters through debugfs.

Statistics already available in ethtool are presented in a
structured manner. Includes per-core, per-FQ and per-channel statistics.

Also transition from module_fsl_mc_driver to explicit module_init/exit
in order to create the debugfs directory besides registering the driver.
Signed-off-by: NIoana Radulescu <ruxandra.radulescu@nxp.com>
Signed-off-by: NIoana Ciornei <ioana.ciornei@nxp.com>
Signed-off-by: NDavid S. Miller <davem@davemloft.net>

In the current implementation, on interface down we disabled NAPI and
then manually drained any remaining ingress frames. This could lead
to a situation when, under heavy traffic, the data availability
notification for some of the channels would not get rearmed correctly.

Change the implementation such that we let all remaining ingress frames
be processed as usual and only disable NAPI once the hardware queues
are empty.

We also add a wait on the Tx side, to allow hardware time to process
all in-flight Tx frames before issueing the disable command.
Signed-off-by: NIoana Radulescu <ruxandra.radulescu@nxp.com>
Signed-off-by: NDavid S. Miller <davem@davemloft.net>

Automatically add a device link between the actual device requesting the
dpaa2_io_service_register and the underlying dpaa2_io used. This link
will ensure that when a DPIO device, which is indirectly used by other
devices, is unbound any consumer devices will be also unbound from their
drivers.

For example, any DPNI, bound to the dpaa2-eth driver, which is using
DPIO devices will be unbound before its supplier device.

Also, add a new parameter to the dpaa2_io_service_[de]register functions
to specify the requesting device (ie the consumer).
Signed-off-by: NIoana Ciornei <ioana.ciornei@nxp.com>
Reviewed-by: NHoria Geanta <horia.geanta@nxp.com>
Reviewed-by: NIoana Radulescu <ruxandra.radulescu@nxp.com>
Signed-off-by: NLi Yang <leoyang.li@nxp.com>

Add comments in the switch statement for XDP action to indicate
fallthrough is intended.
Signed-off-by: NIoana Radulescu <ruxandra.radulescu@nxp.com>
Signed-off-by: NDavid S. Miller <davem@davemloft.net>

Add counters for xdp processed frames to the channel statistics.
Signed-off-by: NIoana Radulescu <ruxandra.radulescu@nxp.com>
Reviewed-by: NDavid Ahern <dsahern@gmail.com>
Signed-off-by: NDavid S. Miller <davem@davemloft.net>

Remove unused counter. Reorder fields in channel stats structure
to match the ethtool strings order and make it easier to print them
with ethtool -S.
Signed-off-by: NIoana Radulescu <ruxandra.radulescu@nxp.com>
Signed-off-by: NDavid S. Miller <davem@davemloft.net>

Send frames back on the same port for XDP_TX action.
Since the frame buffers have been allocated by us, we can recycle
them directly into the Rx buffer pool instead of requesting a
confirmation frame upon transmission complete.
Signed-off-by: NIoana Radulescu <ruxandra.radulescu@nxp.com>
Signed-off-by: NDavid S. Miller <davem@davemloft.net>

In order to support enqueueing Rx FDs back to hardware, we need to
DMA map Rx buffers as bidirectional.
Signed-off-by: NIoana Radulescu <ruxandra.radulescu@nxp.com>
Signed-off-by: NDavid S. Miller <davem@davemloft.net>

Instead of freeing the RX buffers, release them back into the pool.
We wait for the maximum number of buffers supported by a single
release command to accumulate before issuing the command.

Also, don't unmap the Rx buffers at the beginning of the Rx routine
anymore, since that would require remapping them before release.
Instead, just do a DMA sync at first and only unmap if the frame is
meant for the stack.
Signed-off-by: NIoana Radulescu <ruxandra.radulescu@nxp.com>
Signed-off-by: NDavid S. Miller <davem@davemloft.net>

We'll use function free_bufs() on the XDP path as well, so move
it higher in order to avoid a forward declaration.
Signed-off-by: NIoana Radulescu <ruxandra.radulescu@nxp.com>
Signed-off-by: NDavid S. Miller <davem@davemloft.net>

Reserve XDP_PACKET_HEADROOM bytes in Rx buffers to allow XDP
programs to increase frame header size.
Signed-off-by: NIoana Radulescu <ruxandra.radulescu@nxp.com>
Signed-off-by: NDavid S. Miller <davem@davemloft.net>

We keep one XDP program reference per channel. The only actions
supported for now are XDP_DROP and XDP_PASS.

Until now we didn't enforce a maximum size for Rx frames based
on MTU value. Change that, since for XDP mode we must ensure no
scatter-gather frames can be received.
Signed-off-by: NIoana Radulescu <ruxandra.radulescu@nxp.com>
Acked-by: NCamelia Groza <camelia.groza@nxp.com>
Reviewed-by: NDavid Ahern <dsahern@gmail.com>
Signed-off-by: NDavid S. Miller <davem@davemloft.net>

Add support for byte queue limit.

On NAPI poll, we save the total number of Tx confirmed frames/bytes
and register them with bql at the end of the poll function.
Signed-off-by: NIoana Radulescu <ruxandra.radulescu@nxp.com>
Signed-off-by: NDavid S. Miller <davem@davemloft.net>

Change the frame consume callback signature:
* the entire FQ structure is passed to the callback instead
of just the queue index
* the NAPI structure can be easily obtained from the channel
it is associated to, so we don't need to pass it explicitly
Signed-off-by: NIoana Radulescu <ruxandra.radulescu@nxp.com>
Signed-off-by: NDavid S. Miller <davem@davemloft.net>

The DPNI object on which we build a network interface has a
certain number of {Rx, Tx, Tx confirmation} frame queues as
resources. The default hardware setup offers one queue of each
type, as well as one DPCON channel, for each core available
in the system.

There are however cases where the number of queues is greater
than the number of cores or channels. Until now, we configured
and used all the frame queues associated with a DPNI, even if it
meant assigning multiple queues of one type to the same channel.

Update the driver to only use a number of queues equal to the
number of channels, ensuring each channel will contain exactly
one Rx and one Tx confirmation queue.

>From the user viewpoint, this change is completely transparent.
Performance wise there is no impact in most scenarios. In case
the number of queues is larger than and not a multiple of the
number of channels, Rx hash distribution offers now better load
balancing between cores, which can have a positive impact on
overall system performance.
Signed-off-by: NIoana Radulescu <ruxandra.radulescu@nxp.com>
Signed-off-by: NDavid S. Miller <davem@davemloft.net>

The fsl_mc_object_allocate function can fail because not all allocatable
objects are probed by the fsl_mc_allocator at the call time. Defer the
dpaa2-eth probe when this happens.
Signed-off-by: NIoana Ciornei <ioana.ciornei@nxp.com>
Reviewed-by: NAndrew Lunn <andrew@lunn.ch>
Signed-off-by: NDavid S. Miller <davem@davemloft.net>