Adds support for XDP_TX i.e transmits packet out of
the XDP TX queue mapped to the corresponding Rx queue
on which packet is received.

Since SQ for XDP TX will be used only on a single cpu i.e
SQ description creation and freeing, using atomic free count
is not necessary and will become a bottleneck. Hence added
a separate 'xdp_free_cnt' used for SQs designated for XDP
to track descriptor free count.

Changes also include
- A new entry 'xdp_page' is added to save transmitted packet's
  page pointer for later cleanup.
- XDP Tx SQ's doorbell is ringed once per NAPI instance.
- Retrieving designated SQ for packets being sent out by stack
  via 'nicvf_xmit'.
Signed-off-by: NSunil Goutham <sgoutham@cavium.com>
Signed-off-by: NDavid S. Miller <davem@davemloft.net>

Adds support for XDP_DROP.
Also since in XDP mode there is just a single buffer per page,
made changes to recycle DMA mapping info as well along with pages.
Signed-off-by: NSunil Goutham <sgoutham@cavium.com>
Signed-off-by: NDavid S. Miller <davem@davemloft.net>

Adds basic XDP support i.e attaching a BPF program to an
interface. Also takes care of allocating separate Tx queues
for XDP path and for network stack packet transmission.

This patch doesn't support handling of any of the XDP actions,
all are treated as XDP_PASS i.e packets will be handed over to
the network stack.

Changes also involve allocating one receive buffer per page in XDP
mode and multiple in normal mode i.e when no BPF program is attached.
Signed-off-by: NSunil Goutham <sgoutham@cavium.com>
Signed-off-by: NDavid S. Miller <davem@davemloft.net>

Optimized CQE handling with below changes
- Feeing descriptors back to SQ in bulk i.e once per NAPI
  instance instead for every CQE_TX, this will reduce number
  of atomic updates to 'sq->free_cnt'.
- Checking errors in CQE_TX and CQE_RX before calling appropriate
  fn()s to update error stats i.e reduce branching.

Also removed debug messages in packet handling path which otherwise
causes issues if DEBUG is enabled.
Signed-off-by: NSunil Goutham <sgoutham@cavium.com>
Signed-off-by: NDavid S. Miller <davem@davemloft.net>

Remove deprecated pci_enable_msix API in favour of its
successor pci_alloc_irq_vectors.
Signed-off-by: NThanneeru Srinivasulu <tsrinivasulu@cavium.com>
Signed-off-by: NSunil Goutham <sgoutham@cavium.com>
Reviewed-by: NChristoph Hellwig <hch@lst.de>
Signed-off-by: NDavid S. Miller <davem@davemloft.net>

Adding support for TSO and checksum hardware offloads for ipv6.
Signed-off-by: NThanneeru Srinivasulu <tsrinivasulu@cavium.com>
Signed-off-by: NSunil Goutham <sgoutham@cavium.com>
Signed-off-by: NDavid S. Miller <davem@davemloft.net>

ACPI support has been added to ARM IOMMU driver in 4.10 kernel
and that has resulted in VNIC interfaces throwing translation
faults when kernel is booted with ACPI as driver was not using
DMA API. This patch fixes the issue by using DMA API which inturn
will create translation tables when IOMMU is enabled.

Also VNIC doesn't have a seperate receive buffer ring per receive
queue, so there is no 1:1 descriptor index matching between CQE_RX
and the index in buffer ring from where a buffer has been used for
DMA'ing. Unlike other NICs, here it's not possible to maintain dma
address to virt address mappings within the driver. This leaves us
no other choice but to use IOMMU's IOVA address conversion API to
get buffer's virtual address which can be given to network stack
for processing.
Signed-off-by: NSunil Goutham <sgoutham@cavium.com>
Signed-off-by: NDavid S. Miller <davem@davemloft.net>

napi_complete_done() allows to opt-in for gro_flush_timeout,
added back in linux-3.19, commit 3b47d303
("net: gro: add a per device gro flush timer")

This allows for more efficient GRO aggregation without
sacrifying latencies.
Signed-off-by: NEric Dumazet <edumazet@google.com>
Signed-off-by: NDavid S. Miller <davem@davemloft.net>

Fix to return a negative error code from the error handling
case instead of 0, as done elsewhere in this function.

Fixes: 712c3185 ("net: thunderx: Program LMAC credits based on MTU")
Signed-off-by: NWei Yongjun <weiyongjun1@huawei.com>
Signed-off-by: NDavid S. Miller <davem@davemloft.net>

The network device operation for reading statistics is only called
in one place, and it ignores the return value. Having a structure
return value is potentially confusing because some future driver could
incorrectly assume that the return value was used.

Fix all drivers with ndo_get_stats64 to have a void function.
Signed-off-by: NStephen Hemminger <sthemmin@microsoft.com>
Signed-off-by: NDavid S. Miller <davem@davemloft.net>

Transmit queue timeout issue is seen in two cases
- Due to a race condition btw setting stop_queue at xmit()
  and checking for stopped_queue in NAPI poll routine, at times
  transmission from a SQ comes to a halt. This is fixed
  by using barriers and also added a check for SQ free descriptors,
  incase SQ is stopped and there are only CQE_RX i.e no CQE_TX.
- Contrary to an assumption, a HW errata where HW doesn't stop transmission
  even though there are not enough CQEs available for a CQE_TX is
  not fixed in T88 pass 2.x. This results in a Qset error with
  'CQ_WR_FULL' stalling transmission. This is fixed by adjusting
  RXQ's  RED levels for CQ level such that there is always enough
  space left for CQE_TXs.
Signed-off-by: NSunil Goutham <sgoutham@cavium.com>
Signed-off-by: NDavid S. Miller <davem@davemloft.net>

Enable pause frames on both Rx and Tx side, configure pause
interval e.t.c. Also support for enable/disable pause frames
on Rx/Tx via ethtool has been added.
Signed-off-by: NSunil Goutham <sgoutham@cavium.com>
Signed-off-by: NDavid S. Miller <davem@davemloft.net>

Signed-off-by: NThanneeru Srinivasulu <tsrinivasulu@cavium.com>
Signed-off-by: NSunil Goutham <sgoutham@cavium.com>
Signed-off-by: NDavid S. Miller <davem@davemloft.net>

This patch fixes the following
1. When interface is being teardown and queues are being cleaned up,
   free pending SKBs that are in SQ which are either not transmitted
   or freed as NAPI is disabled by that time.
2. While interface initialization, delay CFG_DONE notification till
   the end to avoid corner cases where TXQs are enabled but CQ
   interrupts are not which results blocking transmission and kicking
   off watchdog.
3. Check for IFF_UP while re-enabling RBDR interrupts from tasklet.
Signed-off-by: NSunil Goutham <sgoutham@cavium.com>
Signed-off-by: NDavid S. Miller <davem@davemloft.net>

This patch fixes multiple issues
1. Convert all driver statistics to percpu counters for accuracy.
2. To avoid multiple CQEs posted by a TSO packet appended to HW,
   TSO pkt's SQE has 'post_cqe' not set but a dummy SQE is added
   for getting HW transmit completion notification. This dummy
   SQE has 'dont_send' set and HW drops the pkt pointed to in this
   thus Tx drop counter increases. This patch fixes this by subtracting
   SW tx tso counter from HW Tx drop counter for actual packet drop counter.
3. Reset all individual queue's and VNIC HW stats when interface is going down.
4. Getrid off unnecessary counters in hot path.
5. Bringout all CQE error stats i.e both Rx and Tx.
Signed-off-by: NSunil Goutham <sgoutham@cavium.com>
Signed-off-by: NDavid S. Miller <davem@davemloft.net>

Programming LMAC credits taking 9K frame size by default is incorrect
as for an interface which is one of the many on the same BGX/QLM
no of credits available will be less as Tx FIFO will be divided
across all interfaces. So let's say a BGX with 40G interface and another
BGX with multiple 10G, bandwidth of 10G interfaces will be effected when
traffic is running on both 40G and 10G interfaces simultaneously.

This patch fixes this issue by programming credits based on netdev's MTU.
Also fixed configuring MTU to HW and added CQE counter for pkts which
exceed this value.
Signed-off-by: NSunil Goutham <sgoutham@cavium.com>
Signed-off-by: NDavid S. Miller <davem@davemloft.net>

liquidio: min_mtu 68, max_mtu 16000

thunder: min_mtu 64, max_mtu 9200

CC: netdev@vger.kernel.org
CC: Sunil Goutham <sgoutham@cavium.com>
CC: Robert Richter <rric@kernel.org>
CC: Derek Chickles <derek.chickles@caviumnetworks.com>
CC: Satanand Burla <satananda.burla@caviumnetworks.com>
CC: Felix Manlunas <felix.manlunas@caviumnetworks.com>
CC: Raghu Vatsavayi <raghu.vatsavayi@caviumnetworks.com>
Signed-off-by: NJarod Wilson <jarod@redhat.com>
Signed-off-by: NDavid S. Miller <davem@davemloft.net>

This patch adds support for byte queue limits
Signed-off-by: NSunil Goutham <sgoutham@cavium.com>
Signed-off-by: NDavid S. Miller <davem@davemloft.net>

On ThunderX 88xx pass 2.x chips when TSO is offloaded to HW,
HW posts a CQE for every TSO segment transmitted. Current code
does handles this, but is prone to issues when segment sizes are
small resulting in SW processing too many CQEs and also at times
frees a SKB which is not yet transmitted.

This patch handles the errata in a different way and eliminates issues
with earlier approach, TSO packet is submitted to HW with post_cqe=0,
so that no CQE is posted upon completion of transmission of TSO packet
but a additional HDR + IMMEDIATE descriptors are added to SQ due to
which a CQE is posted and will have required info to be used while
cleanup in napi. This way only one CQE is posted for a TSO packet.
Signed-off-by: NSunil Goutham <sgoutham@cavium.com>
Signed-off-by: NDavid S. Miller <davem@davemloft.net>

Use standard API to generate a random RSS hash key
on every boot.
Signed-off-by: NSunil Goutham <sgoutham@cavium.com>
Signed-off-by: NDavid S. Miller <davem@davemloft.net>

This patch enables bulk freeing on the Tx side.
Signed-off-by: NSunil Goutham <sgoutham@cavium.com>
Signed-off-by: NDavid S. Miller <davem@davemloft.net>

Set MAC addresses only for primary VF's and don't for
secondary VFs.
Signed-off-by: NSunil Goutham <sgoutham@cavium.com>
Signed-off-by: NDavid S. Miller <davem@davemloft.net>

Adding debug messages in case of NACK for a mailbox message, also
did small cleanups.
Signed-off-by: NRadoslaw Biernacki <rad@semihalf.com>
Signed-off-by: NSunil Goutham <sgoutham@cavium.com>
Signed-off-by: NDavid S. Miller <davem@davemloft.net>

This patch changes the way VF's irqs are visible in /proc/interrupts.
Instead of VF id, logical interface's netdev name is used for IRQ
naming and also all secondary VF's interrupts in multiqset config
use primary VF's netdev name.
Signed-off-by: NSunil Goutham <sgoutham@cavium.com>
Signed-off-by: NDavid S. Miller <davem@davemloft.net>

81xx has only 4 CPUs, so it doesn't make sense to initialize
entire Qset i.e 8 queues by default. Made changes to queue
initialization to init queues equal to number of CPUs or
8 queues whichever is lesser. Also this will be applicable to
VMs with VNIC VF attached and having less VCPUs
Signed-off-by: NSunil Goutham <sgoutham@cavium.com>
Signed-off-by: NDavid S. Miller <davem@davemloft.net>

This patch adds PCI device IDs of VNIC on newer chips and also
registers VF driver with them. Device id remains same for all
versions of chips but subsystem device id changes.
Signed-off-by: NSunil Goutham <sgoutham@cavium.com>
Signed-off-by: NDavid S. Miller <davem@davemloft.net>

Replace all trans_start updates with netif_trans_update helper.
change was done via spatch:

struct net_device *d;
@@
- d->trans_start = jiffies
+ netif_trans_update(d)

Compile tested only.

Cc: user-mode-linux-devel@lists.sourceforge.net
Cc: linux-xtensa@linux-xtensa.org
Cc: linux1394-devel@lists.sourceforge.net
Cc: linux-rdma@vger.kernel.org
Cc: netdev@vger.kernel.org
Cc: MPT-FusionLinux.pdl@broadcom.com
Cc: linux-scsi@vger.kernel.org
Cc: linux-can@vger.kernel.org
Cc: linux-parisc@vger.kernel.org
Cc: linux-omap@vger.kernel.org
Cc: linux-hams@vger.kernel.org
Cc: linux-usb@vger.kernel.org
Cc: linux-wireless@vger.kernel.org
Cc: linux-s390@vger.kernel.org
Cc: devel@driverdev.osuosl.org
Cc: b.a.t.m.a.n@lists.open-mesh.org
Cc: linux-bluetooth@vger.kernel.org
Signed-off-by: NFlorian Westphal <fw@strlen.de>
Acked-by: NFelipe Balbi <felipe.balbi@linux.intel.com>
Acked-by: NMugunthan V N <mugunthanvnm@ti.com>
Acked-by: NAntonio Quartulli <a@unstable.cc>
Signed-off-by: NDavid S. Miller <davem@davemloft.net>

Counting rx packets for every CQE_RX in CQ irq handler is incorrect.
Synchronization is missing when multiple queues are receiving packets
simultaneously. Like transmit packet stats use HW stats here.

Also removed unused 'cqe_type' parameter in nicvf_rcv_pkt_handler().
Signed-off-by: NSunil Goutham <sgoutham@cavium.com>
Signed-off-by: NDavid S. Miller <davem@davemloft.net>

For secondary Qsets 'hw_tso' is not getting set as probe() returns
much earlier. Fixed it by moving silicon revision check.
Signed-off-by: NSunil Goutham <sgoutham@cavium.com>
Signed-off-by: NDavid S. Miller <davem@davemloft.net>

When a interface is assigned morethan 8 queues and the logical interface
is toggled i.e down & up, additional queues or qsets are not initialized
as secondary qset count is being set to zero while tearing down.
Signed-off-by: NSunil Goutham <sgoutham@cavium.com>
Signed-off-by: NDavid S. Miller <davem@davemloft.net>

This affinity hint can be used by user space irqbalance tool to set
preferred CPU mask for irqs registered by this VF. Irqbalance needs
to be in 'exact' mode to set irq affinity same as indicated by
affinity hint.
Signed-off-by: NSunil Goutham <sgoutham@cavium.com>
Signed-off-by: NDavid S. Miller <davem@davemloft.net>

napi_schedule is being called from hard irq context, hence
switch to napi_schedule_irqoff which avoids unneeded call
to local_irq_save and local_irq_restore.
Signed-off-by: NSunil Goutham <sgoutham@cavium.com>
Signed-off-by: NDavid S. Miller <davem@davemloft.net>

When system is low on atomic memory, too many error messages are logged.
Since this is not a total failure but a simple switch to non-atomic allocation
better to have a stat.

Also add a stat for reset, kicked due to transmit watchdog timeout.
Signed-off-by: NThanneeru Srinivasulu <tsrinivasulu@caviumnetworks.com>
Signed-off-by: NSunil Goutham <sgoutham@cavium.com>
Signed-off-by: NDavid S. Miller <davem@davemloft.net>

This adds support for offloading TCP segmentation to HW in pass-2
revision of hardware. Both driver level SW TSO for pass1.x chips
and HW TSO for pass-2 chip will co-exist. Modified SQ descriptor
structures to reflect pass-2 hw implementation.
Signed-off-by: NSunil Goutham <sgoutham@cavium.com>
Signed-off-by: NDavid S. Miller <davem@davemloft.net>

Call netif_carrier_on() only if interface's link is up. Switching this on
upon IFF_UP by default, is causing issues with ethernet channel bonding
in LACP mode. Initial NETDEV_CHANGE notification was being skipped.

Also fixed some issues with link/speed/duplex reporting via ethtool.
Signed-off-by: NSunil Goutham <sgoutham@cavium.com>
Signed-off-by: NDavid S. Miller <davem@davemloft.net>

In some cases the crash is caused by nicvf_remove() being called from
outside. For example, if we try to feed the device to vfio after the
probe has failed for some reason. So, move the check to better place.
Signed-off-by: NPavel Fedin <p.fedin@samsung.com>
Signed-off-by: NDavid S. Miller <davem@davemloft.net>

If device probe fails, driver remains bound to the PCI device. However,
driver data has been reset to NULL. This causes crash upon dereferencing
it in nicvf_remove()
Signed-off-by: NPavel Fedin <p.fedin@samsung.com>
Signed-off-by: NDavid S. Miller <davem@davemloft.net>

Signed-off-by: NSunil Goutham <sgoutham@cavium.com>
Signed-off-by: NDavid Daney <david.daney@cavium.com>
Signed-off-by: NDavid S. Miller <davem@davemloft.net>

Support for setting VF's corresponding BGX LMAC in internal
loopback mode. This mode can be used for verifying basic HW
functionality such as packet I/O, RX checksum validation,
CQ/RBDR interrupts, stats e.t.c. Useful when DUT has no external
network connectivity.

'loopback' mode can be enabled or disabled via ethtool.

Note: This feature is not supported when no of VFs enabled are
morethan no of physical interfaces i.e active BGX LMACs
Signed-off-by: NSunil Goutham <sgoutham@cavium.com>
Signed-off-by: NAleksey Makarov <aleksey.makarov@caviumnetworks.com>
Signed-off-by: NDavid S. Miller <davem@davemloft.net>

This patch adds support for handling multiple qsets assigned to a
single VF. There by increasing no of queues from earlier 8 to max
no of CPUs in the system i.e 48 queues on a single node and 96 on
dual node system. User doesn't have option to assign which Qsets/VFs
 to be merged. Upon request from VF, PF assigns next free Qsets as
secondary qsets. To maintain current behavior no of queues is kept
to 8 by default which can be increased via ethtool.

If user wants to unbind NICVF driver from a secondary Qset then it
should be done after tearing down primary VF's interface.
Signed-off-by: NSunil Goutham <sgoutham@cavium.com>
Signed-off-by: NAleksey Makarov <aleksey.makarov@caviumnetworks.com>
Signed-off-by: NRobert Richter <rrichter@cavium.com>
Signed-off-by: NDavid S. Miller <davem@davemloft.net>