With refcounting it looks like you can just drop that refcount, but
that's not really the case. So make sure no one forgets.

Motivated by the unlocked call in the mmio flip code.

Cc: John Harrison <John.C.Harrison@Intel.com>
Cc: Thomas Daniel <Thomas.Daniel@intel.com>
Signed-off-by: NDaniel Vetter <daniel.vetter@intel.com>