Before 3f055b59 we were doing that. However after the commit
we are leaving the files behind.
Signed-off-by: NMichal Privoznik <mprivozn@redhat.com>