Ensure that we don't spuriously trigger early.
Signed-off-by: NMark Brown <broonie@opensource.wolfsonmicro.com>
Acked-by: NLiam Girdwood <lrg@ti.com>