Support DQA queue sharing when no free queue exists for
allocation to a STA that already exists. This means that
a single queue will serve more than a single TID (although
the RA will be the same for all TIDs served).

We try to choose the lowest AC possible, to ensure the
shared queues have the lowest possible combined AC
requirements. The queue to share is chosen only from the
same RA's DATA queues as follows (in descending priority):
 1. An AC_BE queue
 2. Same AC queue
 3. Highest AC queue that is lower than new AC
 4. Any existing AC (there always is at least 1 DATA queue)

If any aggregations existed for any of the TIDs of the
shared queue - they are stopped (the FW is notified), but
no delBA is sent.
Signed-off-by: NLiad Kaufman <liad.kaufman@intel.com>
Signed-off-by: NLuca Coelho <luciano.coelho@intel.com>

Currently code calls restock for mq devices during the init
function, unlike sq where restock is called after init.
This causes an harmless but alarming deadlock warning from
lockdep, to fix this - unify the init code.
Rename the restock functions while at it.
Signed-off-by: NSara Sharon <sara.sharon@intel.com>
Signed-off-by: NLuca Coelho <luciano.coelho@intel.com>

Add a warning in case packet didn't end up in the HW
destined queue.
Signed-off-by: NSara Sharon <sara.sharon@intel.com>
Signed-off-by: NLuca Coelho <luciano.coelho@intel.com>

We now have 9000 devices that support multiple frames in
a single RB. Enable it.
Signed-off-by: NSara Sharon <sara.sharon@intel.com>
Signed-off-by: NLuca Coelho <luciano.coelho@intel.com>

For 9000 devices we can have PCIe bus for discrete
devices and IOSF bus for integrated devices.
PCIe supports maximum transfer size of 128B while IOSF
bus supports maximum transfer size of 64B.
Configure RB size accordingly.
Signed-off-by: NSara Sharon <sara.sharon@intel.com>
Signed-off-by: NLuca Coelho <luciano.coelho@intel.com>

Integrated 9000 devices have a bug with shadow registers
value retention.
If driver writes RBD registers while MAC is asleep the
values are stored in shadow registers to be copied whenever
MAC wakes up.
However, in 9000 devices a MAC wakeup is not triggered
and when the bus powers down due to inactivity the shadow
values and dirty bits are lost.
Turn on the chicken-bits that cause MAC wakeup for RX-related
values as well when the device is in D0.
When the device is in low power mode turn the RX wakeup chicken
bits off since driver is idle and this W/A is not needed.
Remove previous W/A which was ineffective.
Signed-off-by: NSara Sharon <sara.sharon@intel.com>
Signed-off-by: NLuca Coelho <luciano.coelho@intel.com>

When initializing RX we grab NIC access for every read and
write. This is redundant - we can just grab access once.
Signed-off-by: NSara Sharon <sara.sharon@intel.com>
Signed-off-by: NLuca Coelho <luciano.coelho@intel.com>

The RX queues have a shadow register for the write pointer
that enables updates without grabbing NIC access. Use them
instead of the periphery registers because accessing those
is much more expensive.
Signed-off-by: NSara Sharon <sara.sharon@intel.com>
Signed-off-by: NLuca Coelho <luciano.coelho@intel.com>

Since msleep is based on jiffies, it can sleep for a long time.
Use usleep_range() instead to shorten the maximum time.
Signed-off-by: NJohannes Berg <johannes.berg@intel.com>
Signed-off-by: NLuca Coelho <luciano.coelho@intel.com>

Newer hardware generations will take longer to be accessible again
after reset, so we need to wait longer before continuing any flow
that did a reset.

Rather than make the wait time configurable, simply extend it for
all.

Since all of these code paths can sleep, use usleep_range() rather
than mdelay().
Signed-off-by: NJohannes Berg <johannes.berg@intel.com>
Signed-off-by: NLuca Coelho <luciano.coelho@intel.com>

Since we have a lot of configuration structs (almost 70) saving
some memory in each one of them leads to an overall saving of
~2.6KiB of memory.
Signed-off-by: NJohannes Berg <johannes.berg@intel.com>
Signed-off-by: NLuca Coelho <luciano.coelho@intel.com>

CSR registers are always available even when the NIC is not awake, no
need to wake up the NIC before accessing them. This has a huge impact
when we re-enable an interrupt at the end of the ISR since waking up the
NIC can take some time.
Signed-off-by: NHaim Dreyfuss <haim.dreyfuss@intel.com>
Signed-off-by: NLuca Coelho <luciano.coelho@intel.com>

Currently there is one to one function between device id to it's ucode.
The new generation devices allows to combine different phy and mac images.
Now we have two different ucode images with the same device id.
Read RF ID to identify phy image and overwrite it if needed.
Signed-off-by: NHaim Dreyfuss <haim.dreyfuss@intel.com>
Signed-off-by: NLuca Coelho <luciano.coelho@intel.com>

If a host command was queued while in runtime suspend, it would go out
before the D0I3_END_CMD was sent.  Sometimes it works, but sometimes
it fails, and it is obviously the wrong thing to do.

To fix this, have the opmode take a reference before sending a SYNC
command and make the pcie trans wait for the runtime state to become
active before actually queueing the command.
Signed-off-by: NLuca Coelho <luciano.coelho@intel.com>

Rename 9560 to 9260.
Add new PCI ID for 9260 and change some entries from 5165 to 9260.
Also order the 9000 series.
Signed-off-by: NHaim Dreyfuss <haim.dreyfuss@intel.com>
Signed-off-by: NLuca Coelho <luciano.coelho@intel.com>

It's cleaner to always call the iwl_trans_ref/unref() functions
instead of sometimes calling the trans-specific ops directly.  This
also prepares for moving some of the code from the trans-specific ops
to the common trans code.
Signed-off-by: NLuca Coelho <luciano.coelho@intel.com>
Signed-off-by: NLuca Coelho <luciano.coelho@intel.com>

Add new 8265 series PCI IDs.
Signed-off-by: NOren Givon <oren.givon@intel.com>
Signed-off-by: NEmmanuel Grumbach <emmanuel.grumbach@intel.com>

isr_stats is written twice with the same value, remove one of the
redundant assignments to isr_stats.
Signed-off-by: NColin Ian King <colin.king@canonical.com>
Signed-off-by: NEmmanuel Grumbach <emmanuel.grumbach@intel.com>

In MSIX mode we iterate over the allocated interrupt vectors and
register them to an handler. In case of registration failure,
we free all the allocated irq.
we use the outer index mistakenly instead of the inner one.
Signed-off-by: NHaim Dreyfuss <haim.dreyfuss@intel.com>
Signed-off-by: NEmmanuel Grumbach <emmanuel.grumbach@intel.com>

Edit some of the 9560 series and 5165 series PCI IDs.
These devices do not exist yet.
Signed-off-by: NOren Givon <oren.givon@intel.com>
Signed-off-by: NEmmanuel Grumbach <emmanuel.grumbach@intel.com>

We want to request an interrupt vector for RSS queue per CPU,
one vector for fallback queue, and one for non-rx interrupts.
Future patch will make sure that no RSS traffic is directed to
fallback queue.
This will enable us to enable fast path on traffic that otherwise
would have been received on the fallback queue.
Signed-off-by: NSara Sharon <sara.sharon@intel.com>
Signed-off-by: NEmmanuel Grumbach <emmanuel.grumbach@intel.com>

Due to hardware bug, upon any shadow free-queue register write
access, a legacy RBD shadow register must be written as well.
This is required in order to trigger a copy of the shadow registers
values after MAC exits sleep state.
Specifically, the driver has to write (any value) to the legacy RBD
register each time FRBDCB is accessed.
Signed-off-by: NSara Sharon <sara.sharon@intel.com>
Signed-off-by: NEmmanuel Grumbach <emmanuel.grumbach@intel.com>

Update device id and FW serial number for 2X2 antenna devices
in 9000 generation product. These will not be available on
the market in the coming year.
Signed-off-by: NHaim Dreyfuss <haim.dreyfuss@intel.com>
Signed-off-by: NEmmanuel Grumbach <emmanuel.grumbach@intel.com>

iwlwifi / iwlmvm didn't destroy their mutexes. Fix that.
Signed-off-by: NEmmanuel Grumbach <emmanuel.grumbach@intel.com>
Signed-off-by: NEmmanuel Grumbach <emmanuel.grumbach@intel.com>

We insert padding if the MAC header's size is not a multiple of 4
to ensure that the SNAP header is DWORD aligned. When we do so, we
let the firmware know by setting a bit in Tx command
(TX_CMD_FLG_MH_PAD) which will instruct the firmware to drop those
2 bytes before sending the frame.
However, this is not needed for AMSDU as the sub frame header (14B)
complements the MAC header (26B) so that the SNAP header is DWORD
aligned without adding any pad.

Until 9000, the firmware didn't check the TX_CMD_FLG_MH_PAD bit
but rather checked the length of the MAC header itself and
assumed the entity that enqueued the frame (driver or internal
firmware code) added the pad.
Since the driver inserted the pad even for AMSDU this logic applied.
Note that the padding is a DMA optimization but it's not strictly
needed, so we could pad even if it was not needed.

However, the CSUM hardware introduced for the 9000 devices requires
to not pad AMSDU as it is not needed, and will fail if such a pad
exists.
Due to older FW not checking the padding bit but checking the mac
header size itself - we cannot do this adjustments for older
generations.
Do not align the size if it is an AMSDU and HW checksum is enabled -
which will only happen on 9000 devices and on.
Signed-off-by: NSara Sharon <sara.sharon@intel.com>
Signed-off-by: NEmmanuel Grumbach <emmanuel.grumbach@intel.com>

Bjorn pointed out that printing an error value as an
hexadecimal isn't very convenient. Change that.
Reported-by: NBjorn Helgaas <bhelgaas@google.com>
Signed-off-by: NEmmanuel Grumbach <emmanuel.grumbach@intel.com>

We don't use the refcount value anymore, all the refcounting is done
in the runtime PM usage_count value.  Remove it.
Signed-off-by: NLuca Coelho <luciano.coelho@intel.com>
Signed-off-by: NEmmanuel Grumbach <emmanuel.grumbach@intel.com>

When entering suspend the driver calls iwl_disable_interrupts() and
then iwl_pcie_disable_ict().
On resume the driver calls only iwl_pcie_reset_ict() without calling
explicitly to iwl_enable_interrupts().
This mostly works since iwl_pcie_reset_ict is calling to
iwl_enable_interrupts, but it doesn't work when there is no ict_table
in MSIx mode.
The result is that driver tries to resume but fails since it doesn't
get the RX interrupt from FW indicating that d0i3 exit was completed.
Fix it by adding an explicit call to enable interrupts.
Signed-off-by: NSara Sharon <sara.sharon@intel.com>
Signed-off-by: NEmmanuel Grumbach <emmanuel.grumbach@intel.com>

My patch resized the pool size, but neglected to resize
the global table, which is obviously wrong since the global
table maps the pool's rxb to vid one to one. This results
in a panic in 9000 devices.
Add a build bug to avoid such a case in the future.

Fixes: 7b542436 ("iwlwifi: pcie: fine tune number of rxbs")
Reported-by: NHaim Dreyfuss <haim.dreyfuss@intel.com>
Signed-off-by: NSara Sharon <sara.sharon@intel.com>
Signed-off-by: NEmmanuel Grumbach <emmanuel.grumbach@intel.com>

IWL_INFO is not an error but still printed by default.
"can't access the RSA semaphore it is write protected" seems
worrisome but it is not really a problem.

CC: <stable@vger.kernel.org> [4.1+]
Signed-off-by: NEmmanuel Grumbach <emmanuel.grumbach@intel.com>

Currently when stop flow is performed, there might be transport TX RTPM
references that are not freed in case we unmap a queue that still has
packets not reclaimed. Fix that.
Signed-off-by: NSara Sharon <sara.sharon@intel.com>
Signed-off-by: NEmmanuel Grumbach <emmanuel.grumbach@intel.com>

When trying to reach high Rx throughput of more than 500Mbps on
a device with a relatively weak CPU (Atom x5-Z8500), CPU utilization
may become a bottleneck. Analysis showed that we are looping in
iwl_pcie_rx_handle for very long periods which led to starvation
of other threads (iwl_pcie_rx_handle runs with _bh disabled).
We were handling Rx and allocating new buffers and the new buffers
were ready quickly enough to be available before we had finished
handling all the buffers available in the hardware. As a
consequence, we called iwl_pcie_rxq_restock to refill the hardware
with the new buffers, and start again handling new buffers without
exiting the function. Since we read the hardware pointer again when
we goto restart, new buffers were handled immediately instead of
exiting the function.

This patch avoids refilling RBs inside rx handling loop, unless an
emergency situation is reached. It also doesn't read the hardware
pointer again unless we are in an emergency (unlikely) case.
This significantly reduce the maximal time we spend in
iwl_pcie_rx_handle with _bh disabled.
Signed-off-by: NGregory Greenman <gregory.greenman@intel.com>
Signed-off-by: NEmmanuel Grumbach <emmanuel.grumbach@intel.com>

We kick the allocator when we have 2 RBDs that don't have
attached RBs, and the allocator allocates 8 RBs meaning
that it needs another 6 RBDs to attach the RBs to.
The design is that allocator should always have enough RBDs
to fulfill requests, so we give in advance 6 RBDs to the
allocator so that when it is kicked, it gets additional 2 RBDs
and has enough RBDs.
These RBDs were taken from the Rx queue itself, meaning
that each Rx queue didn't have the maximal number of
RBDs, but MAX - 6.
Change initial number of RBDs in the system to include both
queue size and allocator reserves.
Note the multi-queue is always 511 instead of 512 to avoid a
full queue since we cannot detect this state easily enough in
the 9000 arch.
Signed-off-by: NSara Sharon <sara.sharon@intel.com>
Signed-off-by: NEmmanuel Grumbach <emmanuel.grumbach@intel.com>

128 byte chunk size is supported only on PCIe and not
on IOSF. For now, change it back to 64 byte.
Reported-by: NOren Givon <oren.givon@intel.com>
Signed-off-by: NSara Sharon <sara.sharon@intel.com>
Signed-off-by: NEmmanuel Grumbach <emmanuel.grumbach@intel.com>

Change the code to move rxbs directly from the allocator's
list to the queue's free list. This makes the code more
readable, saves the interim array and the double loop over
the free RBs.
Signed-off-by: NSara Sharon <sara.sharon@intel.com>
Signed-off-by: NEmmanuel Grumbach <emmanuel.grumbach@intel.com>

The pci driver keeps any unbound device in active state and forbids
runtime PM.  When our driver gets probed, we take control of the
state.  When the device is released (i.e. during unbind or module
removal), we should return the state to what it was before.  To do so,
we need to forbid RTPM in the driver remove op.

Additionally, remove an unnecessary pm_runtime_disable() call, move
the initial ref_count setting to a better place and add some comments
explaining what is going on.
Signed-off-by: NLuca Coelho <luciano.coelho@intel.com>
Signed-off-by: NEmmanuel Grumbach <emmanuel.grumbach@intel.com>

In 9000 series A0 step the closed_rb_num is not wrapping around
properly. The queue is wrapping around as it should, so we can
W/A it by wrapping the closed_rb_num in the driver.
While at it, extend RX logging and add error handling of other
cases HW values may cause us to access invalid memory locations.
Add also a proper masking of vid value read from HW - this should
not have actual affect, but better to be on the safe side.
Signed-off-by: NSara Sharon <sara.sharon@intel.com>
Signed-off-by: NEmmanuel Grumbach <emmanuel.grumbach@intel.com>

The patch below introduced a variable shadowing. Fix that.

Fixes: 3955525d ("iwlwifi: pcie: buffer packets to avoid overflowing Tx queues")
Signed-off-by: NEmmanuel Grumbach <emmanuel.grumbach@intel.com>

With these ops, we can know when we are about to enter system suspend.
This allows us to exit D0i3 state before entering suspend.
Signed-off-by: NLuca Coelho <luciano.coelho@intel.com>
Signed-off-by: NEmmanuel Grumbach <emmanuel.grumbach@intel.com>

Fine tune RFH registers further:
* Set default queue explicitly
* Set RFH to drop frames exceeding RB size
* Set the maximum rx transfer size to DRAM to 128 instead of 64
Signed-off-by: NSara Sharon <sara.sharon@intel.com>
Signed-off-by: NEmmanuel Grumbach <emmanuel.grumbach@intel.com>