This simplifies error and removal paths.
Signed-off-by: NLaurent Pinchart <laurent.pinchart@ideasonboard.com>
Signed-off-by: NWolfram Sang <wsa@the-dreams.de>