We need to count the timeout down.
Reported-by: NAxel Lin <axel.lin@gmail.com>
Signed-off-by: NMark Brown <broonie@opensource.wolfsonmicro.com>
Acked-by: NLiam Girdwood <lrg@ti.com>