Make sure to drop the reference to the parent device taken by
class_find_device() after populating the bus.

Fixes: 3b9334ac ("mfd: vexpress: Convert custom func API to regmap")
Acked-by: NSudeep Holla <sudeep.holla@arm.com>
Signed-off-by: NJohan Hovold <johan@kernel.org>
Signed-off-by: NArnd Bergmann <arnd@arndb.de>