This simplifies the error handling and allows us to drop the whole
chv_pinctrl_remove() function.
Signed-off-by: NMika Westerberg <mika.westerberg@linux.intel.com>
Signed-off-by: NLinus Walleij <linus.walleij@linaro.org>