We had a mix of using the class device and the parent device while
allocating data through the devm_* managed functions.

Let's be more consistent and always use the parent device.
Signed-off-by: NUlf Hansson <ulf.hansson@linaro.org>