Future devices will allow for more queue pairs, so allocate a netdev
that can handle them. While we're at it, get rid of the separate
MAX_TX/MAX_RX defines. Since we always get matched queue pairs, having
these makes no sense.

Change-ID: I0e3556cd9a962506e509eb7c0afa36b329e8cb51
Signed-off-by: NMitch Williams <mitch.a.williams@intel.com>
Tested-by: NAndrew Bowers <andrewx.bowers@intel.com>
Signed-off-by: NJeff Kirsher <jeffrey.t.kirsher@intel.com>

Bump.

Change-ID: I00ebbb2e5e5572f947502b8f6db4d94f666d6b14
Signed-off-by: NCatherine Sullivan <catherine.sullivan@intel.com>
Tested-by: NAndrew Bowers <andrewx.bowers@intel.com>
Signed-off-by: NJeff Kirsher <jeffrey.t.kirsher@intel.com>

Instead of awkwardly keeping a fixed array of pointers in the adapter
struct and then allocating ring structs individually, just keep a single
pointer and allocate a single blob for the arrays. This simplifies code,
shrinks the adapter structure, and future-proofs the driver by not
limiting the number of rings we can handle.

Change-ID: I31334ff911a6474954232cfe4bc98ccca3c769ff
Signed-off-by: NMitch Williams <mitch.a.williams@intel.com>
Tested-by: NAndrew Bowers <andrewx.bowers@intel.com>
Signed-off-by: NJeff Kirsher <jeffrey.t.kirsher@intel.com>

Change the queue_vector array from a statically-sized member of the
adapter structure to a dynamically-allocated and -sized array.

This reduces the size of the adapter structure, and allows us to support
any number of queue vectors in the future without changing the code.

Change-ID: I08dc622cb2f2ad01e832e51c1ad9b86524730693
Signed-off-by: NMitch Williams <mitch.a.williams@intel.com>
Tested-by: NAndrew Bowers <andrewx.bowers@intel.com>
Signed-off-by: NJeff Kirsher <jeffrey.t.kirsher@intel.com>

If, upon a midnight dreary, the PF returns ERR_PARAM when the VF is
requesting resources, that's fatal. Either the firmware or NVM is badly,
badly misconfigured, or this VF has been disabled due to a previous VF
driver sending a bunch of bogus messages.

Either way, there is no recovery from this. Don't ponder weak and weary,
just quit.

Change-ID: I09d9f16cc4ee7fec3b57646a289d33838c1c5bf5
Signed-off-by: NMitch Williams <mitch.a.williams@intel.com>
Tested-by: NAndrew Bowers <andrewx.bowers@intel.com>
Signed-off-by: NJeff Kirsher <jeffrey.t.kirsher@intel.com>

This patch adds new fields to i40e_vsi to store user configured
RSS config data and code to use it.

Change-ID: Ic5d3db8d9df52182b560248f8cdca9c5c7546879
Signed-off-by: NHelin Zhang <helin.zhang@intel.com>
Tested-by: NAndrew Bowers <andrewx.bowers@intel.com>
Signed-off-by: NJeff Kirsher <jeffrey.t.kirsher@intel.com>

There are two ways to get RSS, this patch implements two functions
with the same input parameters, and creates a more generic function
for getting RSS configuration.

Change-ID: I12d3b712c21455d47dd0a5aae58fc9b7c680db59
Signed-off-by: NHelin Zhang <helin.zhang@intel.com>
Tested-by: NAndrew Bowers <andrewx.bowers@intel.com>
Signed-off-by: NJeff Kirsher <jeffrey.t.kirsher@intel.com>

There are two ways to configure RSS, this patch adjusts those two
functions with the same input parameters, and creates a more
generic function for configuring RSS.

Change-ID: Iace73bdeba4831909979bef221011060ab327f71
Signed-off-by: NHelin Zhang <helin.zhang@intel.com>
Tested-by: NAndrew Bowers <andrewx.bowers@intel.com>
Signed-off-by: NJeff Kirsher <jeffrey.t.kirsher@intel.com>

This patch renames old VF adapter specific RSS function to clarify
its scope.

Change-ID: Ie5253083a44c677ebb7709a8a3a18402ad2dc6a6
Signed-off-by: NHelin Zhang <helin.zhang@intel.com>
Tested-by: NAndrew Bowers <andrewx.bowers@intel.com>
Signed-off-by: NJeff Kirsher <jeffrey.t.kirsher@intel.com>

Issue a prefetch for data early in the transmit path.
This should not be generally needed for Tx traffic, but
it helps immensely for pktgen workloads and should help
for forwarding workloads as well.

Change-ID: Iefee870c20599e0c4240e1d8637e4f16b625f83a
Signed-off-by: NJesse Brandeburg <jesse.brandeburg@intel.com>
Tested-by: NAndrew Bowers <andrewx.bowers@intel.com>
Signed-off-by: NJeff Kirsher <jeffrey.t.kirsher@intel.com>

This patch fixes the issue of forcing WB too often causing us to not
benefit from NAPI.

Without this patch we were forcing WB/arming interrupt too often taking
away the benefits of NAPI and causing a performance impact.

With this patch we disable force WB in the clean routine for X710
and XL710 adapters. X722 adapters do not enable interrupt to force
a WB and benefit from WB_ON_ITR and hence force WB is left enabled
for those adapters.
For XL710 and X710 adapters if we have less than 4 packets pending
a software Interrupt triggered from service task will force a WB.

This patch also changes the conditions for setting RS bit as described
in code comments. This optimizes when the HW does a tail bump amd when
it does a WB. It also optimizes when we do a wmb.

Change-ID: Id831e1ae7d3e2ec3f52cd0917b41ce1d22d75d9d
Signed-off-by: NAnjali Singhai Jain <anjali.singhai@intel.com>
Tested-by: NAndrew Bowers <andrewx.bowers@intel.com>
Signed-off-by: NJeff Kirsher <jeffrey.t.kirsher@intel.com>

When a lot (many hundreds) of MAC or VLAN filters are added at one time,
we can overflow the Admin Queue buffer size with all the requests.
Unfortunately, the driver would then calculate the message size
incorrectly, causing it to be rejected by the PF. Furthermore, there was
no mechanism to trigger another request to allow for configuring the
rest of the filters that didn't fit into the first request.

To fix this, recalculate the correct buffer size when we detect the
overflow condition instead of just assuming the max buffer size. Also,
don't clear the request bit in adapter->aq_required when we have an
overflow, so that the rest of the filters can be processed later.

Change-ID: Idd7cbbc5af31315e0dcb1b10e6a02ad9817ce65c
Signed-off-by: NMitch Williams <mitch.a.williams@intel.com>
Tested-by: NAndrew Bowers <andrewx.bowers@intel.com>
Signed-off-by: NJeff Kirsher <jeffrey.t.kirsher@intel.com>

Clean up and enhance error messages related to VF MAC/VLAN filters.
Indicate which VF is having issues, and if possible indicate the MAC
address or VLAN involved.

Also, when an error is returned from the PF driver, print useful
information about what went wrong, for the most likely cases.

Change-ID: Ib3d15eef9e3369a78fd142948671e5fa26d921b8
Signed-off-by: NMitch Williams <mitch.a.williams@intel.com>
Tested-by: NAndrew Bowers <andrewx.bowers@intel.com>
Signed-off-by: NJeff Kirsher <jeffrey.t.kirsher@intel.com>

Add a comment to the #endif to more easily match it with its #if.

Change-ID: I47eb0a60a17dc6d2f01a930e45006d2dc82e044f
Signed-off-by: NHelin Zhang <helin.zhang@intel.com>
Tested-by: NAndrew Bowers <andrewx.bowers@intel.com>
Signed-off-by: NJeff Kirsher <jeffrey.t.kirsher@intel.com>

When in NAPI with interrupts disabled, the HW needs to be forced to do a
write back on TX if the number of descriptors pending are less than a
cache line.

This stat helps keep track of how many times we get into this situation.

Change-ID: I76c1bcc7ebccd6bffcc5aa33bfe05f2fa1c9a984
Signed-off-by: NAnjali Singhai Jain <anjali.singhai@intel.com>
Tested-by: NAndrew Bowers <andrewx.bowers@intel.com>
Signed-off-by: NJeff Kirsher <jeffrey.t.kirsher@intel.com>

Code was moved into a separate function some time ago.

Change-ID: Icabbe71ce05cf5d716d3e1152cdd9cd41d11bcb5
Signed-off-by: NShannon Nelson <shannon.nelson@intel.com>
Tested-by: NAndrew Bowers <andrewx.bowers@intel.com>
Signed-off-by: NJeff Kirsher <jeffrey.t.kirsher@intel.com>

Compiler complained of an unused variable, which the driver was just
using to store the result of a rd32 which is used to clear a register
unconditionally.  Just drop the unused variable and re-use one.
Signed-off-by: NJesse Brandeburg <jesse.brandeburg@intel.com>
Tested-by: NAndrew Bowers <andrewx.bowers@intel.com>
Signed-off-by: NJeff Kirsher <jeffrey.t.kirsher@intel.com>

We would like to automatically provide busy polling support
to all NAPI drivers, without them having to implement anything.

skb_mark_napi_id() can be called from napi_gro_receive() and
napi_get_frags().

Few drivers are still calling skb_mark_napi_id() because
they use netif_receive_skb(). They should eventually call
napi_gro_receive() instead. I will leave this to drivers
maintainers.
Signed-off-by: NEric Dumazet <edumazet@google.com>
Signed-off-by: NDavid S. Miller <davem@davemloft.net>

In i40evf_msix_aq(), the first two lines of rd32() are mainly to clear
the registers. If we initialize 'val' at this point, it will be
overwritten immediately. We shall simply discard the return value here.

When we initialize 'val', we might as well include the mask in one step.
Signed-off-by: NJean Sacren <sakiwit@gmail.com>
Tested-by: NAndrew Bowers <andrewx.bowers@intel.com>
Signed-off-by: NJeff Kirsher <jeffrey.t.kirsher@intel.com>

@flush has been missing since the inception of i40evf_irq_enable(). Add
it for the kernel doc.
Signed-off-by: NJean Sacren <sakiwit@gmail.com>
Tested-by: NAndrew Bowers <andrewx.bowers@intel.com>
Acked-by: NJesse Brandeburg <jesse.brandeburg@intel.com>
Signed-off-by: NJeff Kirsher <jeffrey.t.kirsher@intel.com>

Instead of using a custom approach change the code to use %*ph format
specifier.
Signed-off-by: NAndy Shevchenko <andriy.shevchenko@linux.intel.com>
Tested-by: NAndrew Bowers <andrewx.bowers@intel.com>
Acked-by: NJesse Brandeburg <jesse.brandeburg@intel.com>
Signed-off-by: NJeff Kirsher <jeffrey.t.kirsher@intel.com>

Bump up the version...

Change-ID: Ib8d501021671ba20250115ed54330e2c182255b7
Signed-off-by: NCatherine Sullivan <catherine.sullivan@intel.com>
Tested-by: NAndrew Bowers <andrewx.bowers@intel.com>
Signed-off-by: NJeff Kirsher <jeffrey.t.kirsher@intel.com>

Not sure how this compiles at all. Use the correct struct for
manipulating the VLAN filter list. Without this, the VLAN filter
list doesn't get processed correctly, and VLAN filters will not
be re-enabled after any kind of reset.

Change-ID: Iceff2dc089f303058fb71ecb08419eed471e0e90
Signed-off-by: NMitch Williams <mitch.a.williams@intel.com>
Tested-by: NAndrew Bowers <andrewx.bowers@intel.com>
Signed-off-by: NJeff Kirsher <jeffrey.t.kirsher@intel.com>

This patch adds a workaround to drop any flow control frames from being
transmitted from any VSI. FW can still send flow control frames if flow
control is enabled.

With this patch in place a malicious VF cannot send flow control or PFC
packets out on the wire.

Change-ID: I4303b24e98b93066d2767fec24dfe78be591c277
Signed-off-by: NAnjali Singhai Jain <anjali.singhai@intel.com>
Tested-by: NAndrew Bowers <andrewx.bowers@intel.com>
Signed-off-by: NJeff Kirsher <jeffrey.t.kirsher@intel.com>

Bump.

Change-ID: Id0a7ecaa491f88ce94c9eba4901e592a56044ee0
Signed-off-by: NCatherine Sullivan <catherine.sullivan@intel.com>
Tested-by: NAndrew Bowers <andrewx.bowers@intel.com>
Signed-off-by: NJeff Kirsher <jeffrey.t.kirsher@intel.com>

A mask value of 0x1FF was accidentally replaced with a bit mask
causing flow director sideband to be broken.

Change-ID: Id3387f67dd1b567b41692b570b383c58671e1eae
Signed-off-by: NAnjali Singhai Jain <anjali.singhai@intel.com>
Tested-by: NAndrew Bowers <andrewx.bowers@intel.com>
Signed-off-by: NJeff Kirsher <jeffrey.t.kirsher@intel.com>

The adaptive ITR (interrupt throttle rate) algorithm was adjusting
the hardware's interrupt rate too frequently.  This caused a lot
of variation in the interrupt rate for fairly constant workloads.

Change the code to have a counter and adjust only once every N
number of interrupts.

Change-ID: I0460f1f86571037484eca5aca36ac4d889cb8389
Signed-off-by: NJesse Brandeburg <jesse.brandeburg@intel.com>
Tested-by: NAndrew Bowers <andrewx.bowers@intel.com>
Signed-off-by: NJeff Kirsher <jeffrey.t.kirsher@intel.com>

The dynamic algorithm, while now working, doesn't have good
performance in 40G mode.

One part of this patch addresses the high CPU utilization of some small
streaming workloads that the driver should reduce CPU in.

It also changes the minimum ITR that the dynamic algorithm
will settle on, causing our minimum latency to go from 12us
to about 14us, when using adaptive mode.

It also changes the BULK interrupt rate to allow maximum throughput
on a 40Gb connection with a single thread of transmit, clamping
interrupt rate to 8000 for TX makes single thread traffic go too
slow.

The new ULTRA bulk setting is introduced and is used
when the Rx packet rate on this queue exceeds 40000 packets per
second.  This value of 40000 was chosen because the automatic tuning
of minimum ITR=20us means that a single queue can't quite achieve
that many packets per second from a round-robin test.

Change-ID: Icce8faa128688ca5fd2c4229bdd9726877a92ea2
Signed-off-by: NJesse Brandeburg <jesse.brandeburg@intel.com>
Tested-by: NAndrew Bowers <andrewx.bowers@intel.com>
Signed-off-by: NJeff Kirsher <jeffrey.t.kirsher@intel.com>

The driver was using a value expressed in 2us increments
for the divisor to figure out our bytes/usec values.

Fix the usecs variable to contain a value in microseconds.

Change-ID: I5c20493103c295d6f201947bb908add7040b7c41
Signed-off-by: NJesse Brandeburg <jesse.brandeburg@intel.com>
Tested-by: NAndrew Bowers <andrewx.bowers@intel.com>
Signed-off-by: NJeff Kirsher <jeffrey.t.kirsher@intel.com>

This change moves a multi-line register setting into a function
which simplifies reading the flow of the enable function.

This also fixes a bug where the enable function was enabling
the interrupt twice while trying to update the two interrupt
throttle rate thresholds for Rx and Tx.

Change-ID: Ie308f9d0d48540204590cb9d7a5a7b1196f959bb
Signed-off-by: NJesse Brandeburg <jesse.brandeburg@intel.com>
Tested-by: NAndrew Bowers <andrewx.bowers@intel.com>
Signed-off-by: NJeff Kirsher <jeffrey.t.kirsher@intel.com>

When the VF driver is unable to communicate with the PF, it just gives
up and never tries again. Aside from the obvious character flaw that
this shows, it's also a lousy user experience.

When PF communications fail, wait five seconds, and try again. And
again. Don't give up, little VF driver! Your prince will come!

Change-ID: Ia1378a39879883563b8faffce819f375821f9585
Signed-off-by: NMitch Williams <mitch.a.williams@intel.com>
Tested-by: NAndrew Bowers <andrewx.bowers@intel.com>
Signed-off-by: NJeff Kirsher <jeffrey.t.kirsher@intel.com>

The i40e_intr and i40e/i40evf_msix_clean_rings functions run from hard
interrupt context or with interrupts already disabled in netpoll.

They can use napi_schedule_irqoff() instead of napi_schedule()
Signed-off-by: NAlexander Duyck <aduyck@mirantis.com>
Tested-by: NAndrew Bowers <andrewx.bowers@intel.com>
Signed-off-by: NJeff Kirsher <jeffrey.t.kirsher@intel.com>

The defines from the RSS enabling call were mistakenly
missed in the patches to the i40e which should have been
to i40evf as well.

This is a follow up to (commit ed921559886dd40528) "fix
32 bit build warnings".
Signed-off-by: NJesse Brandeburg <jesse.brandeburg@intel.com>
Tested-by: NAndrew Bowers <andrewx.bowers@intel.com>
Signed-off-by: NJeff Kirsher <jeffrey.t.kirsher@intel.com>

Bump.

Change-ID: I7ec818a507554648675b9b245ced9e6b6bd9ed4e
Signed-off-by: NCatherine Sullivan <catherine.sullivan@intel.com>
Tested-by: NAndrew Bowers <andrewx.bowers@intel.com>
Signed-off-by: NJeff Kirsher <jeffrey.t.kirsher@intel.com>

On some devices, in some systems, in some configurations, the VFs would
fail to initialize the first time you loaded the driver.

To correct this, increase the delay time for the init task slightly, and
wait longer before giving up.

If we enable VFs and load the VF driver in the same kernel as the PF
driver, we can totally overwhelm the PF driver with AQ requests because
all of the instances try to initialize at the same time.

To help alleviate this, stagger the initial scheduling of the init task
using the PCIe function as a multiplier. We mask off the function to
only three bits so no instance has to wait too long.

With these two changes, initializing 128 VFs on a single device goes
from four minutes to just a few seconds.

Change-ID: If3d8720c1c4e838ab36d8781d9ec295a62380936
Signed-off-by: NMitch Williams <mitch.a.williams@intel.com>
Tested-by: NAndrew Bowers <andrewx.bowers@intel.com>
Signed-off-by: NJeff Kirsher <jeffrey.t.kirsher@intel.com>

The vlan_features field was correctly being set to the same value as the
netdev features field. However, this was being done before the features
were actually being set up, leaving the vlan_features empty.

Also, after a reset, vlan_features will be incorrectly assigned the
previous netdev feature flags, which can contain VLAN feature bits. This
makes the VLAN code angry and will cause a stack dump.

To fix these issues, set up the netdev features first, then mask out the
VLAN feature bits when assigning vlan_features.

Change-ID: Ib0548869dc83cf6a841cb8697dd94c12359ba4d2
Signed-off-by: NMitch Williams <mitch.a.williams@intel.com>
Tested-by: NAndrew Bowers <andrewx.bowers@intel.com>
Signed-off-by: NJeff Kirsher <jeffrey.t.kirsher@intel.com>

The XL710 hardware has a different interrupt moderation design
that can support a limit of total interrupts per second per
vector, in addition to the "number of interrupts per second"
controls already established in the driver.  This combination
of hardware features allows us to set very low default latency
settings but minimize the total CPU utilization by not
making too many interrupts, should the user desire.

The current driver implementation is still enabling the dynamic
moderation in the driver, and only using the rx/tx-usecs
limit in ethtool to limit the interrupt rate per second, by default.

The new code implemented in this patch
2) adds init/use of the new "Interrupt Limit" register
3) adds ethtool knob to control/report the limits above

Usage is ethtool -C ethx rx-usecs-high <value> Where <value> is number
of microseconds to create a rate of 1/N interrupts per second,
regardless of rx-usecs or tx-usecs values. Since there is a credit based
scheme in the hardware, the rx-usecs and tx-usecs can be configured for
very low latency for short bursts, but once the credit runs out the
refill rate on the credits is limited by rx-usecs-high.

Change-ID: I3a1075d3296123b0f4f50623c779b027af5b188d
Signed-off-by: NJesse Brandeburg <jesse.brandeburg@intel.com>
Tested-by: NAndrew Bowers <andrewx.bowers@intel.com>
Signed-off-by: NJeff Kirsher <jeffrey.t.kirsher@intel.com>

Add const to functions that return strings that aren't going to be
modified. This addresses some reported compile complaints.

Change-ID: Ic56b1e814ab4d23a50480e7fdec652445f776ee8
Signed-off-by: NJingjing Wu <jingjing.wu@intel.com>
Tested-by: NAndrew Bowers <andrewx.bowers@intel.com>
Signed-off-by: NJeff Kirsher <jeffrey.t.kirsher@intel.com>

As per Eric Dumazet's previous patches:
(see commit (24d2e4a5) - tg3: use napi_complete_done())

Quoting verbatim:
Using napi_complete_done() instead of napi_complete() allows
us to use /sys/class/net/ethX/gro_flush_timeout

GRO layer can aggregate more packets if the flush is delayed a bit,
without having to set too big coalescing parameters that impact
latencies.
</end quote>

Tested
configuration: low latency via ethtool -C ethx adaptive-rx off
				rx-usecs 10 adaptive-tx off tx-usecs 15
workload: streaming rx using netperf TCP_MAERTS

igb:
MIGRATED TCP MAERTS TEST from 0.0.0.0 (0.0.0.0) port 0 AF_INET to 10.0.0.1 () port 0 AF_INET : demo
...
Interim result:  941.48 10^6bits/s over 1.000 seconds ending at 1440193171.589

Alignment      Offset         Bytes    Bytes       Recvs   Bytes    Sends
Local  Remote  Local  Remote  Xfered   Per                 Per
Recv   Send    Recv   Send             Recv (avg)          Send (avg)
    8       8      0       0 1176930056  1475.36    797726   16384.00  71905

MIGRATED TCP MAERTS TEST from 0.0.0.0 (0.0.0.0) port 0 AF_INET to 10.0.0.1 () port 0 AF_INET : demo
...
Interim result:  941.49 10^6bits/s over 0.997 seconds ending at 1440193142.763

Alignment      Offset         Bytes    Bytes       Recvs   Bytes    Sends
Local  Remote  Local  Remote  Xfered   Per                 Per
Recv   Send    Recv   Send             Recv (avg)          Send (avg)
    8       8      0       0 1175182320  50476.00     23282   16384.00  71816

i40e:
Hard to test because the traffic is incoming so fast (24Gb/s) that GRO
always receives 87kB, even at the highest interrupt rate.

Other drivers were only compile tested.
Signed-off-by: NJesse Brandeburg <jesse.brandeburg@intel.com>
Tested-by: NAndrew Bowers <andrewx.bowers@intel.com>
Signed-off-by: NJeff Kirsher <jeffrey.t.kirsher@intel.com>

Signed-off-by: NAlexander Duyck <aduyck@mirantis.com>
Tested-by: NAndrew Bowers <andrewx.bowers@intel.com>
Signed-off-by: NJeff Kirsher <jeffrey.t.kirsher@intel.com>