During development, even though board is wired
to e.g. OTG, we might want to compile host-only
or peripheral-only configurations.

Let's allow that to happen.
Signed-off-by: NFelipe Balbi <balbi@ti.com>