Sometimes the connection fail happen on renesas_usbhs.
This patch fix it up.
Signed-off-by: NKuninori Morimoto <morimoto.kuninori@renesas.com>
Signed-off-by: NGreg Kroah-Hartman <gregkh@suse.de>