There was another instance where we were
holding pointers which could be long gone.

Fix that by caching only values pointed to
by such pointer.

Because no crash has been observed, this patch
will be sent on v3.19 merge window, instead of
-rc.
Signed-off-by: NFelipe Balbi <balbi@ti.com>