This is preparation to fix abort operation race (See "xhci: Fix race
related to abort operation"). To make timeout sleepable, use
delayed_work instead of timer.

[change a newly added pending timer fix to pending work -Mathias]
Signed-off-by: NOGAWA Hirofumi <hirofumi@mail.parknet.co.jp>
Signed-off-by: NMathias Nyman <mathias.nyman@linux.intel.com>
Signed-off-by: NGreg Kroah-Hartman <gregkh@linuxfoundation.org>

In command timer function, xhci_handle_command_timeout(), xhci->lock
is unlocked before call into xhci_abort_cmd_ring(). This might cause
race between the timer function and the event handler.

The xhci_abort_cmd_ring() function sets the CMD_RING_ABORT bit in the
command register and polling it until the setting takes effect. A stop
command ring event might be handled between writing the abort bit and
polling for it. The event handler will restart the command ring, which
causes the failure of polling, and we ever believed that we failed to
stop it.

As a bonus, this also fixes some issues of calling functions without
locking in xhci_handle_command_timeout().

Cc: <stable@vger.kernel.org> # 3.7+
Signed-off-by: NLu Baolu <baolu.lu@linux.intel.com>
Signed-off-by: NMathias Nyman <mathias.nyman@linux.intel.com>
Signed-off-by: NGreg Kroah-Hartman <gregkh@linuxfoundation.org>

If we get a command completion event at the same time as the command
timeout work starts on another cpu we might end up aborting the wrong
command.

If the command completion takes the xhci lock before the timeout work, it
will handle the command, pick the next command, mark it as current_cmd, and
re-queue the timeout work. When the timeout work finally gets the lock
It will start aborting the wrong command.

This case can be resolved by checking if the timeout work is pending inside
the timeout function itself. A new timeout work can only be pending if the
command completed and a new command was queued.

If there are no more commands pending then command completion will set
the current_cmd to NULL, which is already handled in the timeout work.

Cc: <stable@vger.kernel.org>
Reported-by: NBaolin Wang <baolin.wang@linaro.org>
Signed-off-by: NMathias Nyman <mathias.nyman@linux.intel.com>
Signed-off-by: NGreg Kroah-Hartman <gregkh@linuxfoundation.org>

When current command was supposed to be aborted, host will free the command
in handle_cmd_completion() function. But it might be still referenced by
xhci->current_cmd, which need to set NULL.

Cc: <stable@vger.kernel.org>
Signed-off-by: NBaolin Wang <baolin.wang@linaro.org>
Signed-off-by: NMathias Nyman <mathias.nyman@linux.intel.com>
Signed-off-by: NGreg Kroah-Hartman <gregkh@linuxfoundation.org>

handle_cmd_completion() frees a command structure which might be still
referenced by xhci->current_cmd.
This might cause problem when xhci->current_cmd is accessed after that.

A real-life case could be like this. The host takes a very long time to
respond to a command, and the command timer is fired at the same time
when the command completion event arrives. The command completion
handler frees xhci->current_cmd before the timer function can grab
xhci->lock. Afterward, timer function grabs the lock and go ahead with
checking and setting members of xhci->current_cmd.

Cc: <stable@vger.kernel.org> # v3.16+
Signed-off-by: NLu Baolu <baolu.lu@linux.intel.com>
Signed-off-by: NMathias Nyman <mathias.nyman@linux.intel.com>
Signed-off-by: NGreg Kroah-Hartman <gregkh@linuxfoundation.org>

xhci->slot_id is used for providing a way to pass slot id from the
command completion handler to the function waiting for completion.
It's shared by enumerations of all USB devices connected to an
xhci host. Hence, it's a source for possible races. Since we've
introduced command structure and the command queue to xhci driver.
It's better to move slot_id from xhci_hcd structure to xhci_command
structure. Hence the race source is removed.
Signed-off-by: NLu Baolu <baolu.lu@linux.intel.com>
Signed-off-by: NMathias Nyman <mathias.nyman@linux.intel.com>
Signed-off-by: NGreg Kroah-Hartman <gregkh@linuxfoundation.org>

define GET_EP_CTX_STATE() macro to get the endpoint state from a
pointer to a le32 enpoint context structure

No functional changes
Signed-off-by: NMathias Nyman <mathias.nyman@linux.intel.com>
Signed-off-by: NGreg Kroah-Hartman <gregkh@linuxfoundation.org>

In xhci_handle_event(), when errors are detected, driver always sets
a bit in error_bitmask (one member of the xhci private driver data).
That means users have to retrieve and decode the value of error_bitmask
in xhci private driver data if they want to know whether those erros
ever happened in xhci_handle_event(). Otherwise, those errors are just
ignored silently.

This patch cleans up this by replacing the setting of error_bitmask
with the kernel print functions, so that users can easily check and
report the errors happened in xhci_handle_event().
Signed-off-by: NLu Baolu <baolu.lu@linux.intel.com>
Signed-off-by: NMathias Nyman <mathias.nyman@linux.intel.com>
Signed-off-by: NGreg Kroah-Hartman <gregkh@linuxfoundation.org>

We want one function that gives back urb in interrupt that can
be used in both normal cases and when killing off urbs.

Add small helpers that check if td is last_td_in_urb() and
that increase the td count of a urb inc_td_cnt()
Signed-off-by: NMathias Nyman <mathias.nyman@linux.intel.com>
Signed-off-by: NGreg Kroah-Hartman <gregkh@linuxfoundation.org>

mostly refactoring and code cleanup.

while handling transfer events we used to check the return value of
various functions to see if we can give back the URB.
Turns out the only case when those return values are set are when
finish_td() notices we are completing the last td in the URB.

give back the urb directly in finish_td() instead.

Only functional change is that we now increase the event ring dequeue
pointer in sowtware after giving back the URB. This should not matter
as we are in hardware interrupt context and the time when writing the
new event ring dequeue to hardware remains the same.
Hardware is the only one that can put event TRBs on the event ring.
Signed-off-by: NMathias Nyman <mathias.nyman@linux.intel.com>
Signed-off-by: NGreg Kroah-Hartman <gregkh@linuxfoundation.org>

Move giving back the urb to a separate function
No functional changes
Signed-off-by: NMathias Nyman <mathias.nyman@linux.intel.com>
Signed-off-by: NGreg Kroah-Hartman <gregkh@linuxfoundation.org>

Fixes the following sparse warning:

drivers/usb/host/xhci-ring.c:608:6: warning:
 symbol 'xhci_unmap_td_bounce_buffer' was not declared.
 Should it be static?
Signed-off-by: NWei Yongjun <weiyj.lk@gmail.com>
Signed-off-by: NMathias Nyman <mathias.nyman@linux.intel.com>
Signed-off-by: NGreg Kroah-Hartman <gregkh@linuxfoundation.org>

No functional changes but a few debugging outputs removed
Signed-off-by: NMathias Nyman <mathias.nyman@linux.intel.com>
Signed-off-by: NGreg Kroah-Hartman <gregkh@linuxfoundation.org>

No functional changes
Signed-off-by: NMathias Nyman <mathias.nyman@linux.intel.com>
Signed-off-by: NGreg Kroah-Hartman <gregkh@linuxfoundation.org>

Renaming variables, no functional changes

Event trbs on the event ring cointain a pointer to the transfer trb
on a endpoint ring that caused the event.

The naming has been very confusing as the event trb was called "event"
and the endpoint transfer trb was called "event_trb".

rename all variables that are realated to the endpoint transfer trb
from event_* to ep_*

No functional changes
Signed-off-by: NMathias Nyman <mathias.nyman@linux.intel.com>
Signed-off-by: NGreg Kroah-Hartman <gregkh@linuxfoundation.org>

mostly refactoring, but does also change error message if
we get a bad length in the event trb.

Adds a helper function to sum trb lengths of partial TDs
Signed-off-by: NMathias Nyman <mathias.nyman@linux.intel.com>
Signed-off-by: NGreg Kroah-Hartman <gregkh@linuxfoundation.org>

Refactor pricess_ctrl_tx() to make it more readable
No functional changes
Signed-off-by: NMathias Nyman <mathias.nyman@linux.intel.com>
Signed-off-by: NGreg Kroah-Hartman <gregkh@linuxfoundation.org>

We only check for the URB_SHORT_NOT_OK urb flag to set the urb status
to -EREMOTEIO for transfers that didn't transfer all data.

This will be done in __usb_hcd_giveback() for other than isoc tranfers,
so remove the redundant check from xhci.

Also fix a couple split lines in debug messages to make them greppable.
Signed-off-by: NMathias Nyman <mathias.nyman@linux.intel.com>
Signed-off-by: NGreg Kroah-Hartman <gregkh@linuxfoundation.org>

replace the complex check for noop with a simple helper
Improve code readability

No functional changes
Signed-off-by: NMathias Nyman <mathias.nyman@linux.intel.com>
Signed-off-by: NGreg Kroah-Hartman <gregkh@linuxfoundation.org>

Use the trb_is_link() and link_trb_toggles_cycle() helpers to
make the code more readable

No functional changes
Signed-off-by: NMathias Nyman <mathias.nyman@linux.intel.com>
Signed-off-by: NGreg Kroah-Hartman <gregkh@linuxfoundation.org>

Fist function called in xhci_halt is xhci_quiesce, so no point
in calling it twice
Signed-off-by: NMathias Nyman <mathias.nyman@linux.intel.com>
Signed-off-by: NGreg Kroah-Hartman <gregkh@linuxfoundation.org>

usb_endpoint_maxp() is now returning maxpacket
correctly - iow only bits 10:0. We can finaly remove
XHCI's private GET_MAX_PACKET macro.
Signed-off-by: NFelipe Balbi <felipe.balbi@linux.intel.com>

The stop endpoint command has its own 5 second timeout timer.
If the timeout function is triggered between USB3 and USB2 host
removal it will try to call usb_hc_died(xhci_to_hcd(xhci)->primary_hcd)

the ->primary_hcd will be set to NULL at USB3 hcd removal.

Fix this by first checking if the PCI host is being removed, and
also by using only xhci_to_hcd() as it will always return the primary
hcd.

CC: <stable@vger.kernel.org>
Signed-off-by: NMathias Nyman <mathias.nyman@linux.intel.com>
Signed-off-by: NGreg Kroah-Hartman <gregkh@linuxfoundation.org>

Enqueue the first TRB even if full_len is zero.
Without this "adb install <apk>" freezes the system.
Signed-off-by: NAlban Browaeys <alban.browaeys@gmail.com>
Fixes: 86065c27 ("xhci: don't rely on precalculated value of needed trbs in the enqueue loop")
Signed-off-by: NMathias Nyman <mathias.nyman@linux.intel.com>
Signed-off-by: NGreg Kroah-Hartman <gregkh@linuxfoundation.org>

Fix "Command completion event does not match command" errors by always
handling the command ring stopped events.

The command ring stopped event is generated as a result of aborting
or stopping the command ring with a register write. It is not caused
by a command in the command queue, and thus won't have a matching command
in the comman list.

Solve it by handling the command ring stopped event before checking for a
matching command.

In most command time out cases we abort the command ring, and get
a command ring stopped event. The events command pointer will point at
the current command ring dequeue, which in most cases matches the timed
out command in the command list, and no error messages are seen.

If we instead get a command aborted event before the command ring stopped
event, the abort event will increse the command ring dequeue pointer, and
the following command ring stopped events command pointer will point at the
next, not yet queued command. This case triggered the error message
Signed-off-by: NMathias Nyman <mathias.nyman@linux.intel.com>
CC: <stable@vger.kernel.org>
Signed-off-by: NGreg Kroah-Hartman <gregkh@linuxfoundation.org>

gcc warns about what first looks like a reference to an uninitialized
variable:

drivers/usb/host/xhci-ring.c: In function 'handle_cmd_completion':
drivers/usb/host/xhci-ring.c:753:4: error: 'ep_ring' may be used uninitialized in this function [-Werror=maybe-uninitialized]
    xhci_unmap_td_bounce_buffer(xhci, ep_ring, cur_td);
    ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
drivers/usb/host/xhci-ring.c:647:20: note: 'ep_ring' was declared here
  struct xhci_ring *ep_ring;
                    ^~~~~~~

It's clear to see that the list_empty() check means it can never be
uninitialized, however it still looks wrong:

When ep->cancelled_td_list contains more than one entry, the
ep_ring variable will point to the ring that was retrieved
from the last urb, and we have to look it up again in the
second loop instead, which fixes the behavior and gets rid of the
warning too.
Signed-off-by: NArnd Bergmann <arnd@arndb.de>
Fixes: f9c589e1 ("xhci: TD-fragment, align the unsplittable case with a bounce buffer")
Acked-by: NMathias Nyman <mathias.nyman@linux.intel.com>
Signed-off-by: NGreg Kroah-Hartman <gregkh@linuxfoundation.org>

It's only used with rings that have link trbs
Signed-off-by: NMathias Nyman <mathias.nyman@linux.intel.com>
Signed-off-by: NGreg Kroah-Hartman <gregkh@linuxfoundation.org>

Only used in one place, replace with trb_is_link() helper
Signed-off-by: NMathias Nyman <mathias.nyman@linux.intel.com>
Signed-off-by: NGreg Kroah-Hartman <gregkh@linuxfoundation.org>

inc_deq() is called both for rings with link trbs and the event ring
without link trbs.
The last_trb() check in inc_deq() has a off by one error, going beyond
allocated array when checking if trb == [TRBS_PER_SEGMENT], and the whole
inc_deq() depend on this.

Rewrite the inc_deq() funciton, remove the faulty last_trb() helper, add
new last_trb_on_seg() and last_trb_on_ring() helpers
Signed-off-by: NMathias Nyman <mathias.nyman@linux.intel.com>
Signed-off-by: NGreg Kroah-Hartman <gregkh@linuxfoundation.org>

Add a new is_link_trb() function that only checks for link trbs.
We want to split generic last_trb() function which is used for both
event rings without link trbs, and endpoint and command rings with links.

This will allow us to easier check for link trbs added mid segments.
Signed-off-by: NMathias Nyman <mathias.nyman@linux.intel.com>
Signed-off-by: NGreg Kroah-Hartman <gregkh@linuxfoundation.org>

Remove the event ring related checks in inc_enq()

Host hardware is the producer of events on the event ring,
driver will not queue anything, or call inc_enq() for the
event ring.
Signed-off-by: NMathias Nyman <mathias.nyman@linux.intel.com>
Signed-off-by: NGreg Kroah-Hartman <gregkh@linuxfoundation.org>

If the last trb before a link is not packet size aligned, and is not
splittable then use a bounce buffer for that chunk of max packet size
unalignable data.

Allocate a max packet size bounce buffer for every segment of a bulk
endpoint ring at the same time as allocating the ring.
If we need to align the data before the link trb in that segment then
copy the data to the segment bounce buffer, dma map it, and enqueue it.
Once the td finishes, or is cancelled, unmap it.

For in transfers we need to first map the bounce buffer, then queue it,
after it finishes, copy the bounce buffer to the original sg list, and
finally unmap it
Signed-off-by: NMathias Nyman <mathias.nyman@linux.intel.com>
Signed-off-by: NGreg Kroah-Hartman <gregkh@linuxfoundation.org>

TD fragments section 4.11.7.1 in xhci specs have additional requirements
on how trbs in TDs must be organized.

TD fragments shall not span transfer ring segments and TD fragments must
be packet aligned. Normally we don't care about TD fragments, on TD is one
big fragment, but if a TD spans ring segments it will be treated as two
fragments, and we need to comply with the alignment requirements.

For us this means that the payload data must be packet aligned in the
last trb before a link trb.
In most mass storage bulk tranfers we are lucky as the block size aligns
nicely with packet size, and there are no issues.
However, usb network adapters using scatterlists can hit this alignment
issue, and usbtest in kernel triggers this in minutes.

This patch is a partial solution, it solves the easy case when the last
trb before the link trb contains a packet boundary.
If that is the case then just split the trb at the boundary.
If not, then just print a debug message and continue as we have always
done, hoping for the best
Signed-off-by: NMathias Nyman <mathias.nyman@linux.intel.com>
Signed-off-by: NGreg Kroah-Hartman <gregkh@linuxfoundation.org>

Queue trbs until all payload data in the urb is tranferred.

The actual number of trbs might need to change from the pre-calculated
number when the packet alignment restrictions for td fragments in
xhci 4.11.7.1 are taken into account.

Long term plan is to get rid of calculating the needed trbs in advance
all together. It's an unnecessary extra walk through the scatterlist.

This change also allows some bulk queue function simplifications
Signed-off-by: NMathias Nyman <mathias.nyman@linux.intel.com>
Signed-off-by: NGreg Kroah-Hartman <gregkh@linuxfoundation.org>

We only need to know if we are queuing the last trb for a TD when
calculating the td remainder field.
The total number of trbs left is not used.

We won't be able to trust the pre-calculated number of trbs used if we
need to align trb data by splitting or merging trbs in order to satisfy
comply with data alignment requirements in xhci specs section 4.11.7.1.
Signed-off-by: NMathias Nyman <mathias.nyman@linux.intel.com>
Signed-off-by: NGreg Kroah-Hartman <gregkh@linuxfoundation.org>

If a zero-length packet is needed after a bulk transfer, then an
additional zero length TD was prepared before enqueueing the bulk transfer
This set up the zero packet TD structure with incorrect td->start_seg
and td->first_trb pointers.

Prepare the zero packet TD after the data bulk TD is enqueued instead.
It sets these pointers correctly.

This change also simplifies unnecessary complexity related to keeping
track of the last trb when enqueuing trbs.
Signed-off-by: NMathias Nyman <mathias.nyman@linux.intel.com>
Signed-off-by: NGreg Kroah-Hartman <gregkh@linuxfoundation.org>

Tiny change, a bit more readable.
The real reason for this change is that the coming td fragment work
had several over 80 lines character lines split just because of a few
extra characters in variable names.

no functional changes
Signed-off-by: NMathias Nyman <mathias.nyman@linux.intel.com>
Signed-off-by: NGreg Kroah-Hartman <gregkh@linuxfoundation.org>

If commands timeout we mark them for abortion, then stop the command
ring, and turn the commands to no-ops and finally restart the command
ring.

If the host is working properly the no-op commands will finish and
pending completions are called.
If we notice the host is failing, driver clears the command ring and
completes, deletes and frees all pending commands.

There are two separate cases reported where host is believed to work
properly but is not. In the first case we successfully stop the ring
but no abort or stop command ring event is ever sent and host locks up.

The second case is if a host is removed, command times out and driver
believes the ring is stopped, and assumes it will be restarted, but
actually ends up timing out on the same command forever.
If one of the pending commands has the xhci->mutex held it will block
xhci_stop() in the remove codepath which otherwise would cleanup pending
commands.

Add a check that clears all pending commands in case host is removed,
or we are stuck timing out on the same command. Also restart the
command timeout timer when stopping the command ring to ensure we
recive an ring stop/abort event.

Cc: stable <stable@vger.kernel.org>
Tested-by: NJoe Lawrence <joe.lawrence@stratus.com>
Signed-off-by: NMathias Nyman <mathias.nyman@linux.intel.com>
Signed-off-by: NGreg Kroah-Hartman <gregkh@linuxfoundation.org>

Under stress occasions some TI devices might not return early when
reading the status register during the quirk invocation of xhci_irq made
by usb_hcd_pci_remove.  This means that instead of returning, we end up
handling this interruption in the middle of a shutdown.  Since
xhci->event_ring has already been freed in xhci_mem_cleanup, we end up
accessing freed memory, causing the Oops below.

commit 8c24d6d7 ("usb: xhci: stop everything on the first call to
xhci_stop") is the one that changed the instant in which we clean up the
event queue when stopping a device.  Before, we didn't call
xhci_mem_cleanup at the first time xhci_stop is executed (for the shared
HCD), instead, we only did it after the invocation for the primary HCD,
much later at the removal path.  The code flow for this oops looks like
this:

xhci_pci_remove()
	usb_remove_hcd(xhci->shared)
	        xhci_stop(xhci->shared)
 			xhci_halt()
			xhci_mem_cleanup(xhci);  // Free the event_queue
	usb_hcd_pci_remove(primary)
		xhci_irq()  // Access the event_queue if STS_EINT is set. Crash.
		xhci_stop()
			xhci_halt()
			// return early

The fix modifies xhci_stop to only cleanup the xhci data when releasing
the primary HCD.  This way, we still have the event_queue configured
when invoking xhci_irq.  We still halt the device on the first call to
xhci_stop, though.

I could reproduce this issue several times on the mainline kernel by
doing a bind-unbind stress test with a specific storage gadget attached.
I also ran the same test over-night with my patch applied and didn't
observe the issue anymore.

[  113.334124] Unable to handle kernel paging request for data at address 0x00000028
[  113.335514] Faulting instruction address: 0xd00000000d4f767c
[  113.336839] Oops: Kernel access of bad area, sig: 11 [#1]
[  113.338214] SMP NR_CPUS=1024 NUMA PowerNV

[c000000efe47ba90] c000000000720850 usb_hcd_irq+0x50/0x80
[c000000efe47bac0] c00000000073d328 usb_hcd_pci_remove+0x68/0x1f0
[c000000efe47bb00] d00000000daf0128 xhci_pci_remove+0x78/0xb0
[xhci_pci]
[c000000efe47bb30] c00000000055cf70 pci_device_remove+0x70/0x110
[c000000efe47bb70] c00000000061c6bc __device_release_driver+0xbc/0x190
[c000000efe47bba0] c00000000061c7d0 device_release_driver+0x40/0x70
[c000000efe47bbd0] c000000000619510 unbind_store+0x120/0x150
[c000000efe47bc20] c0000000006183c4 drv_attr_store+0x64/0xa0
[c000000efe47bc60] c00000000039f1d0 sysfs_kf_write+0x80/0xb0
[c000000efe47bca0] c00000000039e14c kernfs_fop_write+0x18c/0x1f0
[c000000efe47bcf0] c0000000002e962c __vfs_write+0x6c/0x190
[c000000efe47bd90] c0000000002eab40 vfs_write+0xc0/0x200
[c000000efe47bde0] c0000000002ec85c SyS_write+0x6c/0x110
[c000000efe47be30] c000000000009260 system_call+0x38/0x108
Signed-off-by: NGabriel Krisman Bertazi <krisman@linux.vnet.ibm.com>
Cc: Roger Quadros <rogerq@ti.com>
Cc: joel@jms.id.au
Cc: stable@vger.kernel.org
Reviewed-by: NRoger Quadros <rogerq@ti.com>
Cc: <stable@vger.kernel.org> #v4.3+
Tested-by: NJoel Stanley <joel@jms.id.au>
Signed-off-by: NMathias Nyman <mathias.nyman@linux.intel.com>
Signed-off-by: NGreg Kroah-Hartman <gregkh@linuxfoundation.org>

Move duplicate code from xhci_queue_intr_tx()
and xhci_queue_isoc_tx_prepare() to the check_interval() function.
Signed-off-by: NAlexandr Ivanov <alexandr.sky@gmail.com>
Signed-off-by: NMathias Nyman <mathias.nyman@linux.intel.com>
Signed-off-by: NGreg Kroah-Hartman <gregkh@linuxfoundation.org>