This removes a possible race condition and crash and allows for further
improvement of the driver.
Signed-off-by: NAlexandre Belloni <alexandre.belloni@free-electrons.com>