The problem occurs in EVP_PKEY_sign() when using RSA with X931 padding.
It is only triggered if the RSA key size is smaller than the digest length.
So with SHA512 you can trigger the overflow with anything less than an RSA
512 bit key. I managed to trigger a 62 byte overflow when using a 16 bit RSA
key. This wasn't sufficient to cause a crash, although your mileage may
vary.

In practice RSA keys of this length are never used and X931 padding is very
rare. Even if someone did use an excessively short RSA key, the chances of
them combining that with a longer digest and X931 padding is very
small. For these reasons I do not believe there is a security implication to
this. Thanks to Kevin Wojtysiak (Int3 Solutions) and Paramjot Oberoi (Int3
Solutions) for reporting this issue.
Reviewed-by: NAndy Polyakov <appro@openssl.org>

Move EVP internals to evp_int.h, remove -Ievp hack from crypto/Makefile
Reviewed-by: NMatt Caswell <matt@openssl.org>

Reviewed-by: NTim Hudson <tjh@openssl.org>

Reviewed-by: NTim Hudson <tjh@openssl.org>

In keygen, return KEY_SIZE_TOO_SMALL not INVALID_KEYBITS.

** I also increased the minimum from 256 to 512, which is now
documented in CHANGES file. **
Reviewed-by: NMatt Caswell <matt@openssl.org>

Extend RSA ASN1 method to support CMS PSS signatures for both sign
and verify.

For signing the EVP_PKEY_CTX parameters are read and the appropriate
CMS structures set up.

For verification the CMS structures are analysed and the corresponding
parameters in the EVP_PKEY_CTX set.

Also add RSA-OAEP support.

For encrypt the EVP_PKEY_CTX parameters are used.

For decrypt the CMS structure is uses to set the appropriate EVP_PKEY_CTX
parameters.

Add OAEP ctrls to retrieve MD and label. Return errors if
an attempt is made to set or retrieve OAEP parameters when
padding mode is not OAEP.

Extend OAEP support. Generalise the OAEP padding functions to support
arbitrary digests. Extend EVP_PKEY RSA method to handle the new OAEP
padding functions and add ctrls to set the additional parameters.

Update RSA EVP_PKEY_METHOD to use the OCTET STRING form of MDC2 signature:
this will make all versions of MDC2 signature equivalent.

requests. Add new ASN1 signature initialisation function to handle this
case.

OPENSSL_NO_DEPRECATED, etc. Steve, please double-check the CMS stuff...

examples. All RFC4134 examples can not be processed.

Initial support for CMS.

Add zlib compression BIO.

Add AES key wrap implementation.

Generalize S/MIME MIME code to support CMS and/or PKCS7.

of handshake failure

2. Changes to x509_certificate_type function (crypto/x509/x509type.c) to
make it recognize GOST certificates as EVP_PKT_SIGN|EVP_PKT_EXCH
(required for s3_srvr to accept GOST client certificates).

3. Changes to EVP
	- adding of function EVP_PKEY_CTX_get0_peerkey
	- Make function EVP_PKEY_derive_set_peerkey work for context with
	  ENCRYPT operation, because we use peerkey field in the context to
	  pass non-ephemeral secret key to GOST encrypt operation.
	- added EVP_PKEY_CTRL_SET_IV control command. It is really
	  GOST-specific, but it is used in SSL code, so it has to go
	  in some header file, available during libssl compilation

4. Fix to HMAC to avoid call of OPENSSL_cleanse on undefined data

5. Include des.h if KSSL_DEBUG is defined into some libssl files, to
  make debugging output which depends on constants defined there, work
  and other KSSL_DEBUG output fixes

6. Declaration of real GOST ciphersuites, two authentication methods
   SSL_aGOST94 and SSL_aGOST2001 and one key exchange method SSL_kGOST

7. Implementation  of these methods.

8. Support for sending unsolicited serverhello extension if GOST
  ciphersuite is selected. It is require for interoperability with
  CryptoPro CSP 3.0 and 3.6 and controlled by
  SSL_OP_CRYPTOPRO_TLSEXT_BUG constant.
  This constant is added to SSL_OP_ALL, because it does nothing, if
  non-GOST ciphersuite is selected, and all implementation of GOST
  include compatibility with CryptoPro.

9. Support for CertificateVerify message without length field. It is
   another CryptoPro bug, but support is made unconditional, because it
   does no harm for draft-conforming implementation.

10. In tls1_mac extra copy of stream mac context is no more done.
  When I've written currently commited code I haven't read
  EVP_DigestSignFinal manual carefully enough and haven't noticed that
  it does an internal digest ctx copying.

This implementation was tested against
1. CryptoPro CSP 3.6 client and server
2. Cryptopro CSP 3.0 server

Fix warnings.

Send ctrls to EVP_PKEY_METHOD during signing of PKCS7 structure so
customisation is possible.

functions and EVP_MD_CTX_copy work properly.

More error checking for RSA pmeth.

value of the passed output buffer is NULL.

The old method of using EVP_PKEY_size(pkey) isn't flexible enough to cover all
cases where the output length may depend on the operation or the parameters
associated with it.

been deleted.