Make sure we can be woken from the forced sleep that is done on errors.
Removing a card often results in -ENOMEDIUM or -EILSEQ so we previously
locked up the removal process for a second.

We could completely exit on -ENOMEDIUM, but it might be a transient
glitch so treat it like any other error.
Signed-off-by: NPierre Ossman <drzeus@drzeus.cx>