Drop the unused endpoints.  They should only be used when there is an
actual remote-endpoint connected.

Cc: Sekhar Nori <nsekhar@ti.com>
Signed-off-by: NKevin Hilman <khilman@baylibre.com>
Signed-off-by: NSekhar Nori <nsekhar@ti.com>