We need destroy the memory pool in failure
Signed-off-by: NShaohua Li <shli@fb.com>