Fix a bug introduced in the force writeback code, where the interrupt
rate was set to 0 (maximum) by accident.

The driver must correctly set the NOITR fields to avoid ITR update
as a side effect of triggering the software interrupt.

Change-ID: I290851ae04ef3811c43aab5ee33242029f26c1a3
Signed-off-by: NJesse Brandeburg <jesse.brandeburg@intel.com>
Signed-off-by: NAnjali Singhai Jain <anjali.singhai@intel.com>
Signed-off-by: NJeff Kirsher <jeffrey.t.kirsher@intel.com>

Performance can be improved a bit by imitating ixgbe and using
prefetch to get us the next Tx descriptor.

Change-ID: Ice7ffd4cd0ce87c35295059bdb7972a7f53723aa
Signed-off-by: NJesse Brandeburg <jesse.brandeburg@intel.com>
Signed-off-by: NJeff Kirsher <jeffrey.t.kirsher@intel.com>

Use l4_tunnel type generically to keep code flow simple.

Change-ID: Ic52287e3b1ca4204e6b6e13431890c1a6ae9c422
Signed-off-by: NAnjali Singhai Jain <anjali.singhai@intel.com>
Signed-off-by: NJeff Kirsher <jeffrey.t.kirsher@intel.com>

A sparse complaint in i40e_debug_aq in a funky buffer write goes away by
straightening out the code out to something less convoluted.

Also fix some other sparse warnings while we are at it, making some
functions static and using NULL instead of 0.

Change-ID: I93907534fe1f1f675830774b3d14ecf1c6ffc9a0
Signed-off-by: NShannon Nelson <shannon.nelson@intel.com>
Signed-off-by: NJeff Kirsher <jeffrey.t.kirsher@intel.com>

We were missing a few packet types for VXLAN offload. This patch fixes
that.

Change-ID: I4b23aa0b08e40ed49d0df6c49a5ed9f2009b44ce
Signed-off-by: NAnjali Singhai Jain <anjali.singhai@intel.com>
Signed-off-by: NJeff Kirsher <jeffrey.t.kirsher@intel.com>

We were checking the outer Protocol flags and deciding the flow for
inner header. This patch fixes that.
This fixes the Tx checksum offload for TCP/IPv6 over vxlan.

Change-ID: I837aaea921d34f71b24c2bc32aaadea5001ddf78
Signed-off-by: NAnjali Singhai Jain <anjali.singhai@intel.com>
Signed-off-by: NJeff Kirsher <jeffrey.t.kirsher@intel.com>

The driver was having some issues with false Tx hang detection. This
makes the driver a little more direct with the checks for progress
forward by directly checking the head write back address and tail register
when determining progress.  This avoids Tx hangs where the software
gets behind, because we are directly checking hardware state when
determining hang state.

Change-ID: I774f0e861c9e8ab5ccb213634100fe15440ae24a
Signed-off-by: NJesse Brandeburg <jesse.brandeburg@intel.com>
Tested-by: NJim Young <james.m.young@intel.com>
Signed-off-by: NJeff Kirsher <jeffrey.t.kirsher@intel.com>

The hardware has some limitations the driver needs to adhere to,
that we found in extended testing.
  1) no more than 8 descriptors per packet on the wire
  2) no header can span more than 3 descriptors

If one of these events occurs, the hardware will generate an internal
error and freeze the Tx queue.

This patch linearizes the skb to avoid these situations.

Change-ID: I37dab7d3966e14895a9663ec4d0aaa8eb0d9e115
Signed-off-by: NAnjali Singhai Jain <anjali.singhai@intel.com>
Tested-by: NJim Young <james.m.young@intel.com>
Signed-off-by: NJeff Kirsher <jeffrey.t.kirsher@intel.com>

Split the receive hot path code into two, one for packet split and one
for single buffer. This improves receive performance since we only need
to check if the ring is in packet split mode once per NAPI poll time,
not several times per packet. The single buffer code is further improved
by the removal of a bunch of code and several variables that are not
needed. On a receive-oriented test this can improve single-threaded
throughput.

Also refactor the packet split receive path to use a fixed buffer for
headers, like ixgbe does. This vastly reduces the number of DMA mappings
and unmappings we need to do, allowing for much better performance in
the presence of an IOMMU.

Lastly, correct packet split descriptor types now that we are actually
using them.

Change-ID: I3a194a93af3d2c31e77ff17644ac7376da6f3e4b
Signed-off-by: NMitch Williams <mitch.a.williams@intel.com>
Tested-by: NJim Young <james.m.young@intel.com>
Signed-off-by: NJeff Kirsher <jeffrey.t.kirsher@intel.com>

This patch fixes indentation issue and error found in argument
reported by static analysis.  Without this patch, sparse and other
static analysis errors will be found.
Reported-by: NFengguang Wu <fengguang.wu@intel.com>
Reported-by: NJulia Lawall <julia.lawall@lip6.fr>
Signed-off-by: NCarolyn Wyborny <carolyn.wyborny@intel.com>
Signed-off-by: NJeff Kirsher <jeffrey.t.kirsher@intel.com>
Signed-off-by: NDavid S. Miller <davem@davemloft.net>

This patch adds a call to u64_stats_init to Rx setup.
This done in order to avoid lockdep errors with seqcount on newer kernels.

Change-ID: Ia8ba8f0bcbd1c0e926f97d70aeee4ce4fd055e93
Signed-off-by: NCarolyn Wyborny <carolyn.wyborny@intel.com>
Signed-off-by: NJeff Kirsher <jeffrey.t.kirsher@intel.com>

This patch forces Tx descriptor writebacks on ITR by kicking
off the SWINT interrupt when we notice that there are non-cache-aligned
Tx descriptors waiting in the ring while interrupts are disabled
under NAPI.

Change-ID: dd6d9675629bf266c7515ad7a201394618c35444
Signed-off-by: NAnjali Singhai Jain <anjali.singhai@intel.com>
Signed-off-by: NJeff Kirsher <jeffrey.t.kirsher@intel.com>

The same macros are used for rx as well. So rename it.
Signed-off-by: NJiri Pirko <jiri@resnulli.us>
Signed-off-by: NDavid S. Miller <davem@davemloft.net>

Set skb->csum_level instead of skb->encapsulation when indicating
CHECKSUM_UNNECESSARY for an encapsulated checksum.
Signed-off-by: NTom Herbert <therbert@google.com>
Signed-off-by: NDavid S. Miller <davem@davemloft.net>

We are seeing situations where the driver sees a hang with less than 4
desc pending, if the driver chooses to ignore it the queue progresses
forward and the stack never experiences a real hang.
With this patch we will log a stat when this situation happens
"tx_sluggish" will increment and we can see some more details
at a higher debug level. Other than that we will ignore this
particular case of Tx hang.

Change-ID: I7d1d1666d990e2b12f4f6bed0d17d22e1b6410d5
Signed-off-by: NAnjali Singhai Jain <anjali.singhai@intel.com>
Signed-off-by: NJeff Kirsher <jeffrey.t.kirsher@intel.com>

This device claims TSO and checksum support for vlans.  It also
allows a user to control vlan acceleration offloading.  As such,
it is possible to turn off vlan acceleration and configure a vlan
which will continue to support TSO and hw checksums.

In such situation the packet passed down the the device will contain
a vlan header and skb->protocol will be set to ETH_P_8021Q.
The device assumes that skb->protocol contains network protocol
value and uses that value to set up TSO and checksum information.
This results in corrupted frames sent on the wire.

This patch extract the protocol value correctly and corrects TSO
and checksums for non-accelerated traffic.

Fix this by using vlan_get_protocol() helper.

CC: Jeff Kirsher <jeffrey.t.kirsher@intel.com>
CC: Jesse Brandeburg <jesse.brandeburg@intel.com>
CC: Bruce Allan <bruce.w.allan@intel.com>
CC: Carolyn Wyborny <carolyn.wyborny@intel.com>
CC: Don Skidmore <donald.c.skidmore@intel.com>
CC: Greg Rose <gregory.v.rose@intel.com>
CC: Alex Duyck <alexander.h.duyck@intel.com>
CC: John Ronciak <john.ronciak@intel.com>
CC: Mitch Williams <mitch.a.williams@intel.com>
CC: Linux NICS <linux.nics@intel.com>
CC: e1000-devel@lists.sourceforge.net
Signed-off-by: NVladislav Yasevich <vyasevic@redhat.com>
Signed-off-by: NDavid S. Miller <davem@davemloft.net>

The hardware design requires that the driver avoid indicating
checksum offload success on some ipv6 frames with extension
headers.

The code needs to just check for the IPV6EXADD bit and if
it is set punt the checksum to the stack.  I don't know why
the code was checking TCP on inner protocol, as that code
doesn't make any sense to me but seems wrong, so remove it.

Change-ID: I10d3aacdbb1819fb60b4b0eb80e6cc67ef2c9599
Signed-off-by: NJesse Brandeburg <jesse.brandeburg@intel.com>
Tested-By: NJim Young <jamesx.m.young@intel.com>
Tested-by: NSibai Li <sibai.li@intel.com>
Signed-off-by: NJeff Kirsher <jeffrey.t.kirsher@intel.com>

The HW still needs to consume it and freeing it in the function
that created it would mean we will be racing with the HW. The
i40e_clean_tx_ring() routine will free up the buffer attached once
the HW has consumed it.  The clean_fdir_tx_irq function had to be fixed
to handle the freeing correctly.

Cases where we program more than one filter per flow (Ipv4), the
code had to be changed to allocate dummy buffer multiple times
since it will be freed by the clean routine.  This also fixes an issue
where the filter program routine was not checking if there were
descriptors available for programming a filter.

Change-ID: Idf72028fd873221934e319d021ef65a1e51acaf7
Signed-off-by: NAnjali Singhai Jain <anjali.singhai@intel.com>
Tested-by: NJim Young <jamesx.m.young@intel.com>
Signed-off-by: NJeff Kirsher <jeffrey.t.kirsher@intel.com>

Driver needs to initialize all members of context descriptor. Stale
data is possible otherwise.

Change-ID: Idc6b53af45583509da42d5ec0824cbaf78aee64f
Signed-off-by: NJesse Brandeburg <jesse.brandeburg@intel.com>
Signed-off-by: NJeff Kirsher <jeffrey.t.kirsher@intel.com>

The driver was not marking packets with bad checksums
correctly, especially IPv6 packets with a bad checksum.
To do this correctly we need a define that may be set by
hardware in rare cases.

Change-ID: I1a997b72b491ded27a78ac3bce1197b2d2611130
Signed-off-by: NJesse Brandeburg <jesse.brandeburg@intel.com>
Signed-off-by: NJeff Kirsher <jeffrey.t.kirsher@intel.com>

The TSO logic in the transmit path had some assumptions that
have been broken now that the kernel can send as much as 32kB
in a single skb->frag[.] entry, even on a system with 4kB pages.

This fixes the assumptions and allows the kernel to operate
as efficiently as possible with both SENDFILE and SEND.

In addition, the hardware limit of data contained in a descriptor is
changed to the next power of two below where it currently is in
order to align to a power of two value, preventing a single byte
of data in a descriptor.

Change-ID: I6af1f0b87c1458e10644dbd47541591075a52651
Signed-off-by: NJesse Brandeburg <jesse.brandeburg@intel.com>
Signed-off-by: NJeff Kirsher <jeffrey.t.kirsher@intel.com>

Adding the appropriate GNU General Public License header and
update copyright year to 2014.

Change-ID: I769dd2d37d70350afd0c8727ae2859c0fd340361
Signed-off-by: NJesse Brandeburg <jesse.brandeburg@intel.com>
Signed-off-by: NCatherine Sullivan <catherine.sullivan@intel.com>
Tested-by: NSibai Li <sibai.li@intel.com>
Signed-off-by: NJeff Kirsher <jeffrey.t.kirsher@intel.com>

Signed-off-by: NFrancois Romieu <romieu@fr.zoreil.com>
Cc: Jesse Brandeburg <jesse.brandeburg@intel.com>
Tested-by: NSibai Li <sibai.li@intel.com>
Signed-off-by: NJeff Kirsher <jeffrey.t.kirsher@intel.com>

We don't need context descriptors for every packet, only tso
or timesync.  This fixes a bug in the driver where it would
always add a context even if all the passed in values
to the context descriptor function were 0/default values.

Change-ID: I0101d2b893380707b5c2de61aab3e16d4310e9a1
Signed-off-by: NJesse Brandeburg <jesse.brandeburg@intel.com>
Signed-off-by: NCatherine Sullivan <catherine.sullivan@intel.com>
Tested-by: NKavindya Deegala <kavindya.s.deegala@intel.com>
Signed-off-by: NJeff Kirsher <jeffrey.t.kirsher@intel.com>

The hardware supports a feature to avoid updating the descriptor
ring by marking each descriptor with a DD bit, and instead
writes a memory location with an update to where the driver
should clean up to.  Enable this feature.

Change-ID: I5da4e0681f0b581a6401c950a81808792267fe57
Signed-off-by: NJesse Brandeburg <jesse.brandeburg@intel.com>
Signed-off-by: NMitch Williams <mitch.a.williams@intel.com>
Signed-off-by: NCatherine Sullivan <catherine.sullivan@intel.com>
Tested-by: NKavindya Deegala <kavindya.s.deegala@intel.com>
Signed-off-by: NJeff Kirsher <jeffrey.t.kirsher@intel.com>

We need to make sure that we stay away from the cache line
where the DD bit (done) may be getting written back for
the transmit ring since the hardware may be writing the
whole cache line for a partial update.

Change-ID: Id0b6dfc01f654def6a2a021af185803be1915d7e
Signed-off-by: NJesse Brandeburg <jesse.brandeburg@intel.com>
Signed-off-by: NCatherine Sullivan <catherine.sullivan@intel.com>
Tested-by: NKavindya Deegala <kavindya.s.deegala@intel.com>
Signed-off-by: NJeff Kirsher <jeffrey.t.kirsher@intel.com>

Original comment from Tom Herbert <therbert@google.com>

   Drivers should call skb_set_hash to set the hash and its type
   in an skbuff.

This patch builds upon Tom's original implementation and adds
the L4 type return when we know it is an L4 hash.
This requires use of the ptype decoder ring, so enable it.

Change-ID: I2f9fa86d1a6add58cff13386f7f4238b1abcc468
CC: Tom Herbert <therbert@google.com>
Signed-off-by: NJesse Brandeburg <jesse.brandeburg@intel.com>
Acked-by: NShannon Nelson <shannon.nelson@intel.com>
Acked-by: NMitch Williams <mitch.a.williams@intel.com>
Signed-off-by: NCatherine Sullivan <catherine.sullivan@intel.com>
Tested-by: NKavindya Deegala <kavindya.s.deegala@intel.com>
Signed-off-by: NJeff Kirsher <jeffrey.t.kirsher@intel.com>

This change moves one operator up to the previous line and deletes
the duplicate declaration of ETH_ALEN.

Also update copyrights.

Change-ID: I88de73093b584e0f3b29d481ccd83fc4b1a1afa5
Signed-off-by: NJesse Brandeburg <jesse.brandeburg@intel.com>
Tested-by: NKavindya Deegala <kavindya.s.deegala@intel.com>
Signed-off-by: NAaron Brown <aaron.f.brown@intel.com>
Signed-off-by: NDavid S. Miller <davem@davemloft.net>

Fix a spelling error, s/extention/extension/.
Signed-off-by: NJesse Brandeburg <jesse.brandeburg@intel.com>
Tested-by: NKavindya Deegala <kavindya.s.deegala@intel.com>
Signed-off-by: NAaron Brown <aaron.f.brown@intel.com>
Signed-off-by: NDavid S. Miller <davem@davemloft.net>

As of commit 7f12ad74 ("i40evf: transmit
and receive functionality") the s390 builds (allyesconfig) fail with:

drivers/net/ethernet/intel/i40evf/i40e_txrx.c: In function 'i40e_clean_rx_irq':
drivers/net/ethernet/intel/i40evf/i40e_txrx.c:818:3: error: implicit declaration of function 'prefetch'
make[5]: *** [drivers/net/ethernet/intel/i40evf/i40e_txrx.o] Error 1

due to an implicit assumption that the prototype from linux/prefetch.h
will be present.

Cc: Mitch Williams <mitch.a.williams@intel.com>
Cc: Greg Rose <gregory.v.rose@intel.com>
Signed-off-by: NPaul Gortmaker <paul.gortmaker@windriver.com>
Acked-by: NShannon Nelson <shannon.nelson@intel.com>
Signed-off-by: NJeff Kirsher <jeffrey.t.kirsher@intel.com>

This file contains the transmit, receive, and NAPI functionality.

Some of the functions in this module are extracted from the i40e driver
but functions that are not appropriate for virtual function devices have
been removed.
Signed-off-by: NMitch Williams <mitch.a.williams@intel.com>
Signed-off-by: NGreg Rose <gregory.v.rose@intel.com>
Tested-by: NSibai Li <sibai.li@intel.com>
Signed-off-by: NJeff Kirsher <jeffrey.t.kirsher@intel.com>

The hash is reported correctly in the rss field if and only if
the filter status is 3.  Other values of filter status mean
different things and we shouldn't depend on a bitwise result.

The issue was that
a & b --> returns true for b={1,2,3}
the fix is
a & b == b

Also refactor this function to use constant operations because we
are in fast path.

Change-Id: I4e29be87439c1cf8b60bc31bea29dff89596c013
Signed-off-by: NJesse Brandeburg <jesse.brandeburg@intel.com>
Tested-by: NKavindya Deegala <kavindya.s.deegala@intel.com>
Signed-off-by: NJeff Kirsher <jeffrey.t.kirsher@intel.com>

Fix the overactive irq issue seen in testing and allow use of
the legacy interrupt.
Signed-off-by: NShannon Nelson <shannon.nelson@intel.com>
Signed-off-by: NJesse Brandeburg <jesse.brandeburg@intel.com>
Tested-by: NKavindya Deegala <kavindya.s.deegala@intel.com>
Signed-off-by: NJeff Kirsher <jeffrey.t.kirsher@intel.com>

This function did a lot of unnecessary cpu_to_xxx(foo) and making it
worse, each of these calls caused a lot of line wrapping.

Fix look and feel via a refactor of this function.  No functional
changes.
Signed-off-by: NJesse Brandeburg <jesse.brandeburg@intel.com>
Tested-by: NKavindya Deegala <kavindya.s.deegala@intel.com>
Signed-off-by: NJeff Kirsher <jeffrey.t.kirsher@intel.com>

This is a fix for an issue reported by coverity, reported
by Hannes Frederic Sowa.
Signed-off-by: NJesse Brandeburg <jesse.brandeburg@intel.com>
CC: Hannes Frederic Sowa <hannes@stressinduktion.org>
Tested-by: NKavindya Deegala <kavindya.s.deegala@intel.com>
Signed-off-by: NJeff Kirsher <jeffrey.t.kirsher@intel.com>

Hot path doesn't need read-flush after interrupt enable, and this
flush really causes a lot of extra cpu utilization.
Signed-off-by: NJesse Brandeburg <jesse.brandeburg@intel.com>
Tested-by: NKavindya Deegala <kavindya.s.deegala@intel.com>
Signed-off-by: NJeff Kirsher <jeffrey.t.kirsher@intel.com>

This change brings support for 64 bit netstats to the driver. Previously
the stats were 64 bit but highly racy due to the fact that 64 bit
transactions are not atomic on 32 bit systems.  This change makes is so
that the 64 bit byte and packet stats are reliable on all architectures.
Signed-off-by: NAlexander Duyck <alexander.h.duyck@intel.com>
Signed-off-by: NJesse Brandeburg <jesse.brandeburg@intel.com>
Tested-by: NKavindya Deegala <kavindya.s.deegala@intel.com>
Signed-off-by: NJeff Kirsher <jeffrey.t.kirsher@intel.com>

Allocate the queue pairs individually instead of as a group.  This
allows for much easier queue management as it is possible to dynamically
resize the queues without having to free and allocate the entire block.

Ease statistic collection by treating Tx/Rx queue pairs as a single
unit.  Each pair is allocated together and starts with a Tx queue and
ends with an Rx queue.  By ordering them this way it is possible to know
the Rx offset based on a pointer to the Tx queue.
Signed-off-by: NAlexander Duyck <alexander.h.duyck@intel.com>
Signed-off-by: NJesse Brandeburg <jesse.brandeburg@intel.com>
Tested-by: NKavindya Deegala <kavindya.s.deegala@intel.com>
Signed-off-by: NJeff Kirsher <jeffrey.t.kirsher@intel.com>

This replaces the ring container array with a linked list.  The idea is
to make the logic much easier to deal with since this will allow us to
call a simple helper function from the q_vectors to go through the
entire list.
Signed-off-by: NAlexander Duyck <alexander.h.duyck@intel.com>
Signed-off-by: NJesse Brandeburg <jesse.brandeburg@intel.com>
Tested-by: NKavindya Deegala <kavindya.s.deegala@intel.com>
Signed-off-by: NJeff Kirsher <jeffrey.t.kirsher@intel.com>

This makes it so that the Tx and Rx byte and packet counts are
separated from the rest of the statistics.  This allows for better
isolation of these stats when we move them into the 64 bit statistics.

Simplify things by re-ordering how the stats display in ethtool.
Instead of displaying all of the Tx queues as a block, followed by all
the Rx queues, the new order is Tx[0], Rx[0], Tx[1], Rx[1], ..., Tx[n],
Rx[n].  This reduces the loops and cleans up the display for testing
purposes since it is very easy to verify if flow director is doing the
right thing as the Tx and Rx queue pair are shown in pairs.
Signed-off-by: NAlexander Duyck <alexander.h.duyck@intel.com>
Signed-off-by: NJesse Brandeburg <jesse.brandeburg@intel.com>
Tested-by: NKavindya Deegala <kavindya.s.deegala@intel.com>
Signed-off-by: NJeff Kirsher <jeffrey.t.kirsher@intel.com>