This patch fixes an issue that this driver doesn't remove its debugfs.

Fixes: 43ba968b ("usb: gadget: udc: renesas_usb3: add debugfs to set the b-device mode")
Cc: <stable@vger.kernel.org> # v4.14+
Signed-off-by: NYoshihiro Shimoda <yoshihiro.shimoda.uh@renesas.com>
Reviewed-by: NSimon Horman <horms+renesas@verge.net.au>
Signed-off-by: NFelipe Balbi <felipe.balbi@linux.intel.com>