Since we've already got logic to special case immediate teardown of the
stream we may as well use it if the pmdown_time has been set to zero by
the application layer instead of scheduling a work item with zero delay.
Signed-off-by: NMark Brown <broonie@opensource.wolfsonmicro.com>
Acked-by: NLiam Girdwood <lrg@ti.com>