Set the card owner field to prevent the module from being removed from
underneath its users.
Signed-off-by: NAxel Lin <axel.lin@ingics.com>
Signed-off-by: NMark Brown <broonie@kernel.org>