When device is stopped, there is no need to handle ISR.
Especially when otg switch to HOST mode.
Signed-off-by: NNeil Zhang <zhangwm@marvell.com>
Signed-off-by: NFelipe Balbi <balbi@ti.com>