The set0 setters take ownership of their arguments, so the values should
be set to NULL to avoid a double-free in the cleanup block should
ssl_security(SSL_SECOP_TMP_DH) fail. Found by BoringSSL's WeakDH test.
Reviewed-by: NKurt Roeckx <kurt@openssl.org>
Reviewed-by: NRich Salz <rsalz@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/1299)

In light of potential UKS (unknown key share) attacks on some
applications, primarily browsers, despite RFC761, name checks are
by default applied with DANE-EE(3) TLSA records.  Applications for
which UKS is not a problem can optionally disable DANE-EE(3) name
checks via the new SSL_CTX_dane_set_flags() and friends.
Reviewed-by: NRich Salz <rsalz@openssl.org>

Commit aea145e3 removed some error codes that are generated
algorithmically: mapping alerts to error texts.  Found by
Andreas Karlsson.  This restores them, and adds two missing ones.
Reviewed-by: NMatt Caswell <matt@openssl.org>

The SSL_load_client_CA_file() failed to load any CAs due to an
inccorrect assumption about the return value of lh_*_insert(). The
return value when inserting into a hash is the old value of the key.

The bug was introduced in 3c82e437.
Reviewed-by: NKurt Roeckx <kurt@openssl.org>
Reviewed-by: NRich Salz <rsalz@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/1279)

We calculate the size required for the ServerKeyExchange message and then
call BUF_MEM_grow_clean() on the buffer. However we fail to take account of
2 bytes required for the signature algorithm and 2 bytes for the signature
length, i.e. we could overflow by 4 bytes. In reality this won't happen
because the buffer is pre-allocated to a large size that means it should be
big enough anyway.

Addresses an OCAP Audit issue.
Reviewed-by: NRich Salz <rsalz@openssl.org>

Reviewed-by: NAndy Polyakov <appro@openssl.org>
Reviewed-by: NKurt Roeckx <kurt@openssl.org>
Reviewed-by: NRich Salz <rsalz@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/1264)

Reviewed-by: NAndy Polyakov <appro@openssl.org>
Reviewed-by: NKurt Roeckx <kurt@openssl.org>
Reviewed-by: NRich Salz <rsalz@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/1264)

In some situations (such as when we receive a fragment of an alert)
we try to get the next packet but did not mark the current one as read,
meaning that we got the same record back again - leading to an infinite
loop.

Found using the BoringSSL test suite.
Reviewed-by: NAndy Polyakov <appro@openssl.org>

Reviewed-by: NRich Salz <rsalz@openssl.org>
Reviewed-by: NMatt Caswell <matt@openssl.org>

Reviewed-by: NKurt Roeckx <kurt@openssl.org>
Reviewed-by: NRich Salz <rsalz@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/1245)

Using RSA_PKCS1_PADDING with RSA_private_decrypt is inherently unsafe.
The API requires writing output on success and touching the error queue
on error. Thus, although the padding check itself is constant-time as of
294d1e36, and the logic after the
decryption in the SSL code is constant-time as of
adb46dbc, the API boundary in the middle
still leaks whether the padding check succeeded, giving us our
much-loved Bleichenbacher padding oracle.

Instead, PKCS#1 padding must be handled by the caller which uses
RSA_NO_PADDING, in timing-sensitive code integrated with the
Bleichenbacher mitigation. Removing PKCS#1 padding in constant time is
actually much simpler when the expected length is a constant (and if
it's not a constant, avoiding a padding oracle seems unlikely), so just
do it inline.
Signed-off-by: NKurt Roeckx <kurt@roeckx.be>
Reviewed-by: NRich Salz <rsalz@openssl.org>

GH: #1222

Reviewed-by: NKurt Roeckx <kurt@openssl.org>
Reviewed-by: NRich Salz <rsalz@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/1168)

Reviewed-by: NStephen Henson <steve@openssl.org>
Reviewed-by: NEmilia Käsper <emilia@openssl.org>

ssl_session_hash() always looks at the first 4 bytes, regardless of the length.
A client can send a session id that's shorter, and the callback could also
generate one that's shorter.  So we make sure that the rest of the buffer is
initliazed to 0 so that we always calculate the same hash.

Found by tis-interpreter, also previously reported as RT #2871
Reviewed-by: NRich Salz <rsalz@openssl.org>

MR: #2911

Fix a comment following commit c2c49969.

RT2388
Reviewed-by: NRich Salz <rsalz@openssl.org>

Some misc return value checks
Reviewed-by: NRich Salz <rsalz@openssl.org>

SSL_clear() and SSL_free() will remove a session from the cache if it is
considered "bad". However SSL_set_session() does not do this for the session
it is replacing.

SSL_clear() clears an SSL object ready for reuse. It does not clear the
session though. This means that:

  SSL_clear(s)
  SSL_set_session(s, sess);

and
  SSL_set_session(s, sess);
  SSL_clear(s);

do not do the same thing, although logically you would expect that they
would.

The failure of SSL_set_session() to remove bad sessions from the cache
should be considered a bug, so this commit fixes it.

RT#597
Reviewed-by: NRich Salz <rsalz@openssl.org>

Found by tis-interpreter
Reviewed-by: NRich Salz <rsalz@openssl.org>

GH: #1173

Reviewed-by: NKurt Roeckx <kurt@openssl.org>
Reviewed-by: NMatt Caswell <matt@openssl.org>

When session tickets are used, it's possible that SNI might swtich the
SSL_CTX on an SSL. Normally, this is not a problem, because the
initial_ctx/session_ctx are used for all session ticket/id processes.

However, when the SNI callback occurs, it's possible that the callback
may update the options in the SSL from the SSL_CTX, and this could
cause SSL_OP_NO_TICKET to be set. If this occurs, then two bad things
can happen:

1. The session ticket TLSEXT may not be written when the ticket expected
flag is set. The state machine transistions to writing the ticket, and
the client responds with an error as its not expecting a ticket.
2. When creating the session ticket, if the ticket key cb returns 0
the crypto/hmac contexts are not initialized, and the code crashes when
trying to encrypt the session ticket.

To fix 1, if the ticket TLSEXT is not written out, clear the expected
ticket flag.
To fix 2, consider a return of 0 from the ticket key cb a recoverable
error, and write a 0 length ticket and continue. The client-side code
can explicitly handle this case.

Fix these two cases, and add unit test code to validate ticket behavior.
Reviewed-by: NEmilia Käsper <emilia@openssl.org>
Reviewed-by: NRich Salz <rsalz@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/1098)

GH1098: Add X509_get_pathlen() (and a test)
GH1097:  Add SSL_is_dtls() function.

Documented.
Reviewed-by: NMatt Caswell <matt@openssl.org>

Sessions are stored on the session_ctx, which doesn't change after
SSL_set_SSL_CTX().
Reviewed-by: NRich Salz <rsalz@openssl.org>
Reviewed-by: NMatt Caswell <matt@openssl.org>

Previously if we received an empty record we just threw it away and
ignored it. Really though if we get an empty record of a different content
type to what we are expecting then that should be an error, i.e. we should
reject out of context empty records. This commit makes the necessary changes
to achieve that.

RT#4395
Reviewed-by: NAndy Polyakov <appro@openssl.org>

The number of read pipelines should be reset in the event of reuse of an
SSL object.
Reviewed-by: NAndy Polyakov <appro@openssl.org>

Reviewed-by: NRich Salz <rsalz@openssl.org>

Reviewed-by: NRich Salz <rsalz@openssl.org>

The tlsext_status_type field in SSL is used by e.g. OpenResty to determine
if the client requested the certificate status, but SSL is now opaque.
Reviewed-by: NRich Salz <rsalz@openssl.org>
Reviewed-by: NMatt Caswell <matt@openssl.org>

Reviewed-by: NRichard Levitte <levitte@openssl.org>

The ssl3_init_finished_mac() function can fail, in which case we need to
propagate the error up through the stack.

RT#3198
Reviewed-by: NRich Salz <rsalz@openssl.org>

Reviewed-by: NMatt Caswell <matt@openssl.org>
Reviewed-by: NRich Salz <rsalz@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/1135)

accompanying documentation.
Reviewed-by: NMatt Caswell <matt@openssl.org>
Reviewed-by: NRich Salz <rsalz@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/1135)

Reviewed-by: NViktor Dukhovni <viktor@openssl.org>

When setting the certificate chain, if a certificate doesn't pass
security checks, then chain may point to a freed STACK_OF(X509)
Reviewed-by: NRich Salz <rsalz@openssl.org>
Reviewed-by: NMatt Caswell <matt@openssl.org>

Windows was complaining about a unary minus operator being applied to an
unsigned type. It did seem to go on and do the right thing anyway, but the
code does look a little suspect. This fixes it.
Reviewed-by: NViktor Dukhovni <viktor@openssl.org>

The ssl3_digest_cached_records() function was failing to handle errors
that might be returned from EVP_DigestSignInit() and
EVP_DigestSignUpdate().

RT#4180
Reviewed-by: NStephen Henson <steve@openssl.org>

Add script to find unused err/reason codes
Remove unused reason codes.
Remove entries for unused functions
Reviewed-by: NMatt Caswell <matt@openssl.org>

Reviewed-by: NRich Salz <rsalz@openssl.org>

from BoringSSL 306ece31bcaaed49e0240a2e5555f8901ebb2d45
Reviewed-by: NRich Salz <rsalz@openssl.org>
Reviewed-by: NMatt Caswell <matt@openssl.org>

Rename sk_xxx to OPENSSL_sk_xxx and _STACK to OPENSSL_STACK
Rename lh_xxx API to OPENSSL_LH_xxx and LHASH_NODE to OPENSSL_LH_NODE
Make lhash stuff opaque.
Use typedefs for function pointers; makes the code simpler.
Remove CHECKED_xxx macros.
Add documentation; remove old X509-oriented doc.
Add API-compat names for entire old API
Reviewed-by: NDr. Stephen Henson <steve@openssl.org>

In the new state machine if using nbio and we get the header of a
handshake message is one record with the body in the next, with an nbio
event in the middle, then the connection was failing. This is because
s->init_num was getting reset. We should only reset it after we have
read the whole message.

RT#4394
Reviewed-by: NAndy Polyakov <appro@openssl.org>