We could walk of a bad list otherwise when someone concurrently
unbinds stuff for fun.

I've suspected this as the root-cause behind seemingly inconsistent
state, but alas it's not.
Acked-by: NChris Wilson <chris@chris-wilson.co.uk>
Signed-off-by: NDaniel Vetter <daniel.vetter@ffwll.ch>