Avoid extra special casing by setting the cache_bypass flag when we're
not caching.
Signed-off-by: NMark Brown <broonie@opensource.wolfsonmicro.com>