We use an int instead. That means SSL_key_update() also should use an int.
Reviewed-by: NRich Salz <rsalz@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/2609)

Reviewed-by: NRich Salz <rsalz@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/2609)

Reviewed-by: NRich Salz <rsalz@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/2609)

Too many KeyUpdate message could be inicative of a problem (e.g. an
infinite KeyUpdate loop if the peer always responds to a KeyUpdate message
with an "update_requested" KeyUpdate response), or (conceivably) an attack.
Either way we limit the number of KeyUpdate messages we are prepared to
handle.
Reviewed-by: NRich Salz <rsalz@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/2609)

Reviewed-by: NRich Salz <rsalz@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/2609)

Reviewed-by: NRich Salz <rsalz@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/2609)

Reviewed-by: NRich Salz <rsalz@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/2609)

Reviewed-by: NRich Salz <rsalz@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/2609)

Reviewed-by: NRich Salz <rsalz@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/2609)

This just receives the message. It doesn't actually update any keys yet.
Reviewed-by: NRich Salz <rsalz@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/2609)

Reviewed-by: NRich Salz <rsalz@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/2609)

This implements the server side KeyUpdate sending capability as well.
Reviewed-by: NRich Salz <rsalz@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/2609)

For client auth call tls_choose_sigalg to select the certificate
and signature algorithm. Use the selected algorithm in
tls_construct_cert_verify.

Remove obsolete tls12_get_sigandhash.
Reviewed-by: NRich Salz <rsalz@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/2643)

Select appropriate signature algorithm and certificate for client
authentication using tls_choose_sigalg.

A lot of selection logic is very similar except not finding a
certificate is not a fatal error: we just do not present a
certificate.

For TLS 1.2 and earlier we only check the current certificate
is suitable (for compatibility with previous logic) for TLS 1.3
(where there are no compatibility issues) we support multiple
client certificates for different algorithms.
Reviewed-by: NRich Salz <rsalz@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/2643)

Following on from CVE-2017-3733, this removes the OPENSSL_assert() check
that failed and replaces it with a soft assert, and an explicit check of
value with an error return if it fails.
Reviewed-by: NRichard Levitte <levitte@openssl.org>

In 1.1.0 changing the ciphersuite during a renegotiation can result in
a crash leading to a DoS attack. In master this does not occur with TLS
(instead you get an internal error, which is still wrong but not a security
issue) - but the problem still exists in the DTLS code.

The problem is caused by changing the flag indicating whether to use ETM
or not immediately on negotiation of ETM, rather than at CCS. Therefore,
during a renegotiation, if the ETM state is changing (usually due to a
change of ciphersuite), then an error/crash will occur.

Due to the fact that there are separate CCS messages for read and write
we actually now need two flags to determine whether to use ETM or not.

CVE-2017-3733
Reviewed-by: NRichard Levitte <levitte@openssl.org>

Reviewed-by: NRich Salz <rsalz@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/2644)

Reviewed-by: NRichard Levitte <levitte@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/2637)

Reviewed-by: NMatt Caswell <matt@openssl.org>
Reviewed-by: NRich Salz <rsalz@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/2636)

Reviewed-by: NRich Salz <rsalz@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/2623)

Reviewed-by: NRich Salz <rsalz@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/2623)

Use negotiated signature algorithm and certificate index in
tls_construct_key_exchange instead of recalculating it.
Reviewed-by: NRich Salz <rsalz@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/2623)

Now the certificate and signature algorithm is set in one place we
can use it directly insetad of recalculating it. The old functions
ssl_get_server_send_pkey() and ssl_get_server_cert_index() are no
longer required.
Reviewed-by: NRich Salz <rsalz@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/2623)

Update tls_choose_sigalg to choose a signature algorithm for all
versions of TLS not just 1.3.

For TLS 1.2 we choose the highest preference signature algorithm
for the chosen ciphersuite.

For TLS 1.1 and earlier the signature algorithm is determined by
the ciphersuite alone. For RSA we use a special MD5+SHA1 signature
algorithm.
Reviewed-by: NRich Salz <rsalz@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/2623)

Reviewed-by: NRich Salz <rsalz@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/2623)

Add inline function ssl_has_cert which checks to see if a certificate and
private key for a given index are not NULL.
Reviewed-by: NRich Salz <rsalz@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/2623)

Reviewed-by: NMatt Caswell <matt@openssl.org>
Reviewed-by: NRich Salz <rsalz@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/2618)

Reviewed-by: NRich Salz <rsalz@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/2624)

Reviewed-by: NRichard Levitte <levitte@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/2341)

Reviewed-by: NRichard Levitte <levitte@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/2341)

For converting the 2 byte group id into an unsigned int.
Reviewed-by: NRichard Levitte <levitte@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/2341)

Reviewed-by: NRichard Levitte <levitte@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/2341)

Reviewed-by: NRichard Levitte <levitte@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/2341)

Reviewed-by: NRichard Levitte <levitte@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/2341)

Reviewed-by: NRichard Levitte <levitte@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/2341)

Reviewed-by: NRichard Levitte <levitte@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/2341)

Reviewed-by: NRichard Levitte <levitte@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/2341)

Reviewed-by: NRichard Levitte <levitte@openssl.org>
Reviewed-by: NMatt Caswell <matt@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/2559)

The original intent of SSL_PKEY_RSA_SIGN and SSL_PKEY_RSA_ENC was to
support two different keys for RSA signing and decrypt. However this
was never implemented and we only ever set one key and the other was
always NULL. Replace with single SSL_PKEY_RSA type.
Reviewed-by: NMatt Caswell <matt@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/2587)

Reviewed-by: NMatt Caswell <matt@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/2587)