We should free any hwspinlocks when we destroy the regmap, do so.
Signed-off-by: NMark Brown <broonie@kernel.org>