Rather than in late_init to avoid race conditions between freeing the
buffers and the initial modeset.
Reviewed-by: NChristian König <christian.koenig@amd.com>
Signed-off-by: NAlex Deucher <alexander.deucher@amd.com>