Set a specific error if the parameters are otherwise valid but exceed the
memory limit.
Reviewed-by: NRich Salz <rsalz@openssl.org>