Otherwise we will hit a list handling assertion when moving the object
to the inactive list.
Signed-off-by: NChris Wilson <chris@chris-wilson.co.uk>