Avoid ending up with a higher frequency than requested
Signed-off-by: NMatthias Seidel <kernel@mseidel.net>
Signed-off-by: NMark Brown <broonie@kernel.org>