Fix the problem that enumeration of a USB device was slow.
Signed-off-by: NYoshihiro Shimoda <shimoda.yoshihiro@renesas.com>
Signed-off-by: NGreg Kroah-Hartman <gregkh@suse.de>