Try to allocate a segment of PIO buffer to each TX channel.  If
allocation fails, log an error but continue.

PIO buffers must be mapped separately from the NIC registers, with
write-combining enabled.  Where the host page size is 4K, we could
potentially map each VI's registers and PIO buffer separately.
However, this would add significant complexity, and we also need to
support architectures such as POWER which have a greater page size.
So make a single contiguous write-combining mapping after the
uncacheable mapping, aligned to the host page size, and link PIO
buffers there.  Where necessary, allocate additional VIs within
the write-combining mapping purely for access to PIO buffers.

Link all TX buffers to TX queues and the additional VIs in
efx_ef10_dimension_resources() and in efx_ef10_init_nic() after
an MC reboot.
Signed-off-by: NBen Hutchings <bhutchings@solarflare.com>

Segmentation remains in the driver, but we generate option descriptors
describing the required packet editing rather than making our own
copies.

Reduce tso_state::ipv4_id to 16 bits, so it doesn't overflow into the
TCP_FLAGS field of the option descriptor.
Signed-off-by: NBen Hutchings <bhutchings@solarflare.com>

Signed-off-by: NBen Hutchings <bhutchings@solarflare.com>

Update the dates for files that have been added to in 2012-2013.
Drop the 'Solarstorm' brand name that's still lingering here.
Signed-off-by: NBen Hutchings <bhutchings@solarflare.com>

The TX path firmware for EF10 supports 'option descriptors' to control
offloads and various other features.  Add a flag and field for these
in struct efx_tx_buffer, and don't treat them as DMA descriptors on
completion.
Signed-off-by: NBen Hutchings <bhutchings@solarflare.com>

Add a counter for TX merged completion events.

This is implemented in the common TX path, because the NIC event
handlers only know how many descriptors were completed, not how many
packets.
Signed-off-by: NBen Hutchings <bhutchings@solarflare.com>

Currently efx_stop_datapath() will try to flush our DMA queues (if DMA
is enabled), then finalise software and hardware state for each queue.
However, for EF10 we must ask the MC to finalise each queue, which
implicitly starts flushing it, and then wait for the flush events.
We therefore need to delegate more of this to the NIC type.

Combine all the hardware operations into a new NIC-type operation
efx_nic_type::fini_dmaq, and call this before tearing down the
software state and buffers for all the DMA queues.
Signed-off-by: NBen Hutchings <bhutchings@solarflare.com>

Most call sites for efx_nic_alloc_buffer() are part of the probe or
reconfiguration paths and can allocate with GFP_KERNEL.  A few others
should use GFP_NOIO (I think).  Only one is in atomic context and
must use the current GFP_ATOMIC.
Signed-off-by: NBen Hutchings <bhutchings@solarflare.com>

Add PTP IEEE-1588 support and make accesible via the PHC subsystem.

This work is based on prior code by Andrew Jackson
Signed-off-by: NStuart Hodgson <smhodgson@solarflare.com>
[bwh:
 - Add byte order conversion in efx_ptp_send_times()
 - Simplify conversion of PPS event times
 - Add the built-in vs module check to CONFIG_SFC_PTP dependencies]
Signed-off-by: NBen Hutchings <bhutchings@solarflare.com>

Signed-off-by: NBen Hutchings <bhutchings@solarflare.com>

We only use tso_state::full_packet_space to calculate the IPv4 tot_len
or IPv6 payload_len, not to set tso_state::packet_space.  Replace it
with an ip_base_len field holding the value of tot_len or payload_len
before including the TCP payload, which is much more useful when
constructing the new headers.
Signed-off-by: NBen Hutchings <bhutchings@solarflare.com>

TSO header buffers contain a control structure immediately followed by
the packet headers, and are kept on a free list when not in use.  This
complicates buffer management and tends to result in cache read misses
when we recycle such buffers (particularly if DMA-coherent memory
requires caches to be disabled).

Replace the free list with a simple mapping by descriptor index.  We
know that there is always a payload descriptor between any two
descriptors with TSO header buffers, so we can allocate only one
such buffer for each two descriptors.

While we're at it, use a standard error code for allocation failure,
not -1.
Signed-off-by: NBen Hutchings <bhutchings@solarflare.com>

We now have a definite upper bound on the number of descriptors per
skb; use that to stop the queue when the next packet might not fit.
Signed-off-by: NBen Hutchings <bhutchings@solarflare.com>

Add a flags field to struct efx_tx_buffer, replacing the
continuation and map_single booleans.

Since a single descriptor cannot be both a TSO header and the last
descriptor for an skb, unionise efx_tx_buffer::{skb,tsoh} and add
flags for validity of these fields.

Clear all flags in free buffers (whereas previously the continuation
flag would be set).
Signed-off-by: NBen Hutchings <bhutchings@solarflare.com>

Currently an skb requiring TSO may not fit within a minimum-size TX
queue.  The TX queue selected for the skb may stall and trigger the TX
watchdog repeatedly (since the problem skb will be retried after the
TX reset).  This issue is designated as CVE-2012-3412.

Set the maximum number of TSO segments for our devices to 100.  This
should make no difference to behaviour unless the actual MSS is less
than about 700.  Increase the minimum TX queue size accordingly to
allow for 2 worst-case skbs, so that there will definitely be space
to add an skb after we wake a queue.

To avoid invalidating existing configurations, change
efx_ethtool_set_ringparam() to fix up values that are too small rather
than returning -EINVAL.
Signed-off-by: NBen Hutchings <bhutchings@solarflare.com>
Signed-off-by: NDavid S. Miller <davem@davemloft.net>

There is nothing in the VLAN driver or core VLAN support that
invalidates the TCP and IP header offsets.
Signed-off-by: NBen Hutchings <bhutchings@solarflare.com>

tso_state::packet_space is always set in tso_start_packet(); the
value set in tso_start() is not used, and is also incorrect.
Signed-off-by: NBen Hutchings <bhutchings@solarflare.com>

Signed-off-by: NBen Hutchings <bhutchings@solarflare.com>

Fix some indentation and line continuations.
Signed-off-by: NBen Hutchings <bhutchings@solarflare.com>

The 'page size' for PCIe DMA, i.e. the alignment of boundaries at
which DMA must be broken, is 4KB.  Name this value as EFX_PAGE_SIZE
and use it in efx_max_tx_len().  Redefine EFX_BUF_SIZE as
EFX_PAGE_SIZE since its value is also a result of that requirement,
and use it in efx_init_special_buffer().
Signed-off-by: NBen Hutchings <bhutchings@solarflare.com>

The out-of-tree version of the sfc driver used to run a self-test on
each device before registering it.  Although this was never included
in-tree, some functions have checks for this special case which is not
really possible.
Signed-off-by: NBen Hutchings <bhutchings@solarflare.com>

The advantage of kcalloc is, that will prevent integer overflows which could
result from the multiplication of number of elements and size and it is also
a bit nicer to read.

The semantic patch that makes this change is available
in https://lkml.org/lkml/2011/11/25/107Signed-off-by: NThomas Meyer <thomas@m3y3r.de>
Signed-off-by: NDavid S. Miller <davem@davemloft.net>

As soon as skb is pushed to hardware, it can be completed and freed, so
we should not dereference skb anymore.
Signed-off-by: NEric Dumazet <eric.dumazet@gmail.com>
Signed-off-by: NDavid S. Miller <davem@davemloft.net>

Changes to sfc to use byte queue limits.
Signed-off-by: NTom Herbert <therbert@google.com>
Acked-by: NEric Dumazet <eric.dumazet@gmail.com>
Signed-off-by: NDavid S. Miller <davem@davemloft.net>

To ease skb->truesize sanitization, its better to be able to localize
all references to skb frags size.

Define accessors : skb_frag_size() to fetch frag size, and
skb_frag_size_{set|add|sub}() to manipulate it.
Signed-off-by: NEric Dumazet <eric.dumazet@gmail.com>
Signed-off-by: NDavid S. Miller <davem@davemloft.net>

When I converted some drivers from pci_map_page to skb_frag_dma_map I
neglected to convert PCI_DMA_xDEVICE into DMA_x_DEVICE and
pci_dma_mapping_error into dma_mapping_error.
Signed-off-by: NIan Campbell <ian.campbell@citrix.com>
Signed-off-by: NDavid S. Miller <davem@davemloft.net>

Signed-off-by: NIan Campbell <ian.campbell@citrix.com>
Cc: Solarflare linux maintainers <linux-net-drivers@solarflare.com>
Cc: Steve Hodgson <shodgson@solarflare.com>
Cc: Ben Hutchings <bhutchings@solarflare.com>
Cc: netdev@vger.kernel.org
Signed-off-by: NDavid S. Miller <davem@davemloft.net>

Moves the Solarflare drivers into drivers/net/ethernet/sfc/ and
make the necessary Kconfig and Makefile changes.

CC: Steve Hodgson <shodgson@solarflare.com>
CC: Ben Hutchings <bhutchings@solarflare.com>
Signed-off-by: NJeff Kirsher <jeffrey.t.kirsher@intel.com>

We need to keep the TX queues stopped throughout a reset, without
triggering the TX watchdog and regardless of the link state.  The
proper way to do this is to use netif_device_{detach,attach}() just as
we do around suspend/resume, rather than the current bodge of faking
link-down.

Since we also need to do this during an offline self-test and we
perform a reset during that, add these function calls outside of
efx_reset_down() and efx_reset_up().
Signed-off-by: NBen Hutchings <bhutchings@solarflare.com>

If the TX queues are running during loopback self tests, host
traffic gets looped back which causes the test to fail.  Avoid
restarting the TX queues after the port reset so that any packets
sent by the host get held back until after the tests have completed.

[bwh: Also wake all TX queues at the end of self-tests.]
Signed-off-by: NBen Hutchings <bhutchings@solarflare.com>

Signed-off-by: NBen Hutchings <bhutchings@solarflare.com>

Implement the ndo_setup_tc() operation with 2 traffic classes.

Current Solarstorm controllers do not implement TX queue priority, but
they do allow queues to be 'paced' with an enforced delay between
packets.  Paced and unpaced queues are scheduled in round-robin within
two separate hardware bins (paced queues with a large delay may be
placed into a third bin temporarily, but we won't use that).  If there
are queues in both bins, the TX scheduler will alternate between them.

If we make high-priority queues unpaced and best-effort queues paced,
and high-priority queues are mostly empty, a single high-priority queue
can then instantly take 50% of the packet rate regardless of how many
of the best-effort queues have descriptors outstanding.

We do not actually want an enforced delay between packets on best-
effort queues, so we set the pace value to a reserved value that
actually results in a delay of 0.
Signed-off-by: NBen Hutchings <bhutchings@solarflare.com>

efx_hard_start_xmit() needs to implement a mapping which is the
inverse of tx_queue::core_txq.  Move the initialisation of
tx_queue::core_txq next to efx_hard_start_xmit() to make the
connection more obvious.
Signed-off-by: NBen Hutchings <bhutchings@solarflare.com>

Long before this driver went into mainline, it had support for
multiple TX queues per port, with lockless TX enabled.  Since Linux
did not know anything of this, filling up any hardware TX queue would
stop the core TX queue and multiple hardware TX queues could fill up
before the scheduler reacted.  Thus it was necessary to keep a count
of how many TX queues were stopped and to wake the core TX queue only
when all had free space again.

The driver also previously (ab)used the per-hardware-queue stopped
flag as a counter to deal with various things that can inhibit TX, but
it no longer does that.

Remove the per-channel tx_stop_count, tx_stop_lock and
per-hardware-queue stopped count and just use the networking core
queue state directly.
Signed-off-by: NBen Hutchings <bhutchings@solarflare.com>

Whenever we add DMA descriptors to a TX ring and update the ring
pointer, the TX DMA engine must first read the new DMA descriptors and
then start reading packet data.  However, all released Solarflare 10G
controllers have a 'TX push' feature that allows us to reduce latency
by writing the first new DMA descriptor along with the pointer update.
This is only useful when the queue is empty.  The hardware should
ignore the pushed descriptor if the queue is not empty, but this check
is buggy, so we must do it in software.

In order to tell whether a TX queue is empty, we need to compare the
previous transmission count (write_count) and completion count
(read_count).  However, if we do that every time we update the ring
pointer then read_count may ping-pong between the caches of two CPUs
running the transmission and completion paths for the queue.
Therefore, we split the check for an empty queue between the
completion path and the transmission path:

- Add an empty_read_count field representing a point at which the
  completion path saw the TX queue as empty.
- Add an old_write_count field for use on the completion path.
- On the completion path, whenever read_count reaches or passes
  old_write_count the TX queue may be empty.  We then read
  write_count, set empty_read_count if read_count == write_count,
  and update old_write_count.
- On the transmission path, we read empty_read_count.  If it's set, we
  compare it with the value of write_count before the current set of
  descriptors was added.  If they match, the queue really is empty and
  we can use TX push.
Signed-off-by: NBen Hutchings <bhutchings@solarflare.com>

Signed-off-by: NBen Hutchings <bhutchings@solarflare.com>

Signed-off-by: NBen Hutchings <bhutchings@solarflare.com>
Signed-off-by: NDavid S. Miller <davem@davemloft.net>

- Allow the ring size to be specified in non
   power-of-two sizes (for instance to limit
   the amount of receive buffers).
 - Automatically size the event queue.
Signed-off-by: NBen Hutchings <bhutchings@solarflare.com>
Signed-off-by: NDavid S. Miller <davem@davemloft.net>

In preparation for changes to the way channels and queue structures
are allocated, revise the macros and functions used to look up and
iterator over them.

- Replace efx_for_each_tx_queue() with iteration over channels then TX
  queues
- Replace efx_for_each_rx_queue() with iteration over channels then RX
  queues (with one exception, shortly to be removed)
- Introduce efx_get_{channel,rx_queue,tx_queue}() functions to look up
  channels and queues by index
- Introduce efx_channel_get_{rx,tx}_queue() functions to look up a
  channel's queues
Signed-off-by: NBen Hutchings <bhutchings@solarflare.com>
Signed-off-by: NDavid S. Miller <davem@davemloft.net>

Replace EFX_ERR() with netif_err(), EFX_INFO() with netif_info(),
EFX_LOG() with netif_dbg() and EFX_TRACE() and EFX_REGDUMP() with
netif_vdbg().

Replace EFX_ERR_RL(), EFX_INFO_RL() and EFX_LOG_RL() using explicit
calls to net_ratelimit().

Implement the ethtool operations to get and set message level flags,
and add a 'debug' module parameter for the initial value.
Signed-off-by: NBen Hutchings <bhutchings@solarflare.com>
Signed-off-by: NDavid S. Miller <davem@davemloft.net>