This reverts commit 355a7018.

This had some bad side effects under normal operation, and should
have been dropped earlier.
Signed-off-by: NDave Airlie <airlied@redhat.com>