This will allow us to only allocate memory when
we actually need.
Signed-off-by: NFelipe Balbi <balbi@ti.com>