Some caches aren't released properly at module unloading time.

Cc: <stable@vger.kernel.org> [v3.4+]
Signed-off-by: NTakashi Iwai <tiwai@suse.de>