Tearing down an unitialized rx channel causes a pending address hole
event to be queued. When booting linux it will report this pending
as something like "Address Hole seen by USB_OTG  at address 57fff584",
since u-boot did not handled this interrupt. Prevent that by not
tearing down the rx channel, when not receiving.
Signed-off-by: NJeroen Hofstee <jeroen@myspectrum.nl>