Else it races with xhci_setup_device

Cc: <stable@vger.kernel.org>
Signed-off-by: NRoger Quadros <rogerq@ti.com>
Signed-off-by: NMathias Nyman <mathias.nyman@linux.intel.com>
Signed-off-by: NGreg Kroah-Hartman <gregkh@linuxfoundation.org>