Don't leave the bridge attached to a stale driver instance when
unbinding, to allow reattachment on a rebind.
Signed-off-by: NPhilipp Zabel <p.zabel@pengutronix.de>