If we failed to set the domain, the buffer was no longer being tracked
on any list.

Cc: stable@kernel.org
Signed-off-by: NChris Wilson <chris@chris-wilson.co.uk>
Signed-off-by: NJesse Barnes <jbarnes@virtuousgeek.org>