Do not remove the device from the IOMMU while the driver is
still attached.
Signed-off-by: NJoerg Roedel <jroedel@suse.de>