TLS < 1.2 has fixed signature algorithms: MD5+SHA1 for RSA and SHA1 for the
others. TLS 1.2 sends a list of supported ciphers, but allows not sending
it in which case SHA1 is used. TLS 1.3 makes sending the list mandatory.

When we didn't receive a list from the client, we always used the
defaults without checking that they are allowed by the configuration.
Reviewed-by: NPaul Dale <paul.dale@oracle.com>
GH: #10784
(cherry picked from commit b0031e5dc2c8c99a6c04bc7625aa00d3d20a59a5)

Many Windows-based GOST TLS implementations are unable to extend the
list of supported SignatureAlgorithms because of lack of the necessary
callback in Windows. So for TLS 1.2 it makes sense to imply the support
of GOST algorithms in case when the GOST ciphersuites are present.

This is a backport of #10377 to 1.1.1 branch
Reviewed-by: NPaul Dale <paul.dale@oracle.com>
(Merged from https://github.com/openssl/openssl/pull/10378)

Apart from public and internal header files, there is a third type called
local header files, which are located next to source files in the source
directory. Currently, they have different suffixes like

  '*_lcl.h', '*_local.h', or '*_int.h'

This commit changes the different suffixes to '*_local.h' uniformly.
Reviewed-by: NRichard Levitte <levitte@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/9681)

Reviewed-by: NRichard Levitte <levitte@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/9847)

In commit 6aca8d1a ("Honour mandatory digest on private key in
has_usable_cert()") I added two checks for the capabilities of the
EVP_PKEY being used. One of them was wrong, as it should only be
checking the signature of the X.509 cert (by its issuer) against the
sigalgs given in a TLS v1.3 signature_algorithms_cert extension.

Remove it.
Reviewed-by: NMatt Caswell <matt@openssl.org>
Reviewed-by: NBen Kaduk <kaduk@mit.edu>
Reviewed-by: NTomas Mraz <tmraz@fedoraproject.org>
(Merged from https://github.com/openssl/openssl/pull/9705)

The function SSL_check_chain() can be used by applications to check that
a cert and chain is compatible with the negotiated parameters. This could
be useful (for example) from the certificate callback. Unfortunately this
function was applying TLSv1.2 sig algs rules and did not work correctly if
TLSv1.3 was negotiated.

We refactor tls_choose_sigalg to split it up and create a new function
find_sig_alg which can (optionally) take a certificate and key as
parameters and find an appropriate sig alg if one exists. If the cert and
key are not supplied then we try to find a cert and key from the ones we
have available that matches the shared sig algs.
Reviewed-by: NTomas Mraz <tmraz@fedoraproject.org>
(Merged from https://github.com/openssl/openssl/pull/9443)

It was only ever in cert_st because ssl_st was a public structure
and could not be modified without breaking the API.  However, both
structures are now opaque, and thus we can freely change their layout
without breaking applications.  In this case, keeping the shared
sigalgs in the SSL object prevents complications wherein they would
inadvertently get cleared during SSL_set_SSL_CTX() (e.g., as run
during a cert_cb).

Fixes #9099
Reviewed-by: NMatt Caswell <matt@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/9157)

(cherry picked from commit 29948ac80c1388cfeb0bd64539ac1fa6e0bb8990)

The previous 2 commits moved supported groups and ciphers out of the
session object to avoid race conditions. We now also move ecpointformats
for consistency. There does not seem to be a race condition with access
to this data since it is only ever set in a non-resumption handshake.
However, there is no reason for it to be in the session.
Reviewed-by: NTomas Mraz <tmraz@fedoraproject.org>
(Merged from https://github.com/openssl/openssl/pull/9176)

If the private key says it can only support one specific digest, then
don't ask it to perform a different one.

Fixes: #7348
Reviewed-by: NNicola Tuveri <nic.tuv@gmail.com>
Reviewed-by: NRichard Levitte <levitte@openssl.org>

(cherry picked from commit 2d263a4a73f852005b16359873475d48755999ad)
Reviewed-by: NMatt Caswell <matt@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/7609)

Reviewed-by: NRichard Levitte <levitte@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/7620)

(cherry picked from commit 65d2c16cbe0da8efed2f285f59930297326fb435)

    SSL_get_signature_nid()      -- local signature algorithm
    SSL_get_signature_type_nid() -- local signature algorithm key type
    SSL_get_peer_tmp_key()       -- Peer key-exchange public key
    SSL_get_tmp_key              -- local key exchange public key

Aliased pre-existing SSL_get_server_tmp_key(), which was formerly
just for clients, to SSL_get_peer_tmp_key().  Changed internal
calls to use the new name.
Reviewed-by: NMatt Caswell <matt@openssl.org>

TLSv1.3 is more restrictive about the curve used. There must be a matching
sig alg defined for that curve. Therefore if we are using some other curve
in our certificate then we should not negotiate TLSv1.3.

Fixes #7435
Reviewed-by: NViktor Dukhovni <viktor@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/7442)

(cherry picked from commit de4dc598024fd0a9c2b7a466fd5323755d369522)

Previously we indicated this as a malloc failure which isn't very
helpful.
Reviewed-by: NKurt Roeckx <kurt@roeckx.be>
(Merged from https://github.com/openssl/openssl/pull/7479)

(cherry picked from commit 680bd131b69d57e891888ab70d300176a5a16617)

If we have selected a ciphersuite using RSA key exchange then we must
not attempt to use an RSA-PSS cert for that.

Fixes #7059
Reviewed-by: NPaul Dale <paul.dale@oracle.com>
Reviewed-by: NTim Hudson <tjh@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/7099)

We need to ensure that the min-max version range we use when constructing
the ClientHello is the same range we use when we validate the version
selected by the ServerHello. Otherwise this may appear as a fallback or
downgrade.

Fixes #6964
Reviewed-by: NViktor Dukhovni <viktor@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/7013)

Using the rsa_pss_rsae_sha256 sig alg should imply that the key OID is
rsaEncryption. Similarly rsa_pss_pss_sha256 implies the key OID is
rsassaPss. However we did not check this and incorrectly tolerated a key
OID that did not match the sig alg sent by the peer.

Fixes #6611
Reviewed-by: NRich Salz <rsalz@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/6732)

Fixes #6513
Reviewed-by: NRich Salz <rsalz@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/6650)

In ssl/t1_lib.c.
Reviewed-by: NRich Salz <rsalz@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/6613)

Commit 61fb5923 broke EAP-FAST support. This fixes it.

Fixes #6395
Reviewed-by: NRich Salz <rsalz@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/6428)

Reviewed-by: NBernd Edlinger <bernd.edlinger@hotmail.de>
Reviewed-by: NMatt Caswell <matt@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/6168)

Reviewed-by: NBernd Edlinger <bernd.edlinger@hotmail.de>
Reviewed-by: NMatt Caswell <matt@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/6168)

Found by OSS-fuzz

Bug introduced in commit 61fb5923Reviewed-by: NMatt Caswell <matt@openssl.org>
GH: #6235

Don't call the decrypt ticket callback if we've already encountered a
fatal error. Do call it if we have an empty ticket present.

Change the return code to have 5 distinct returns codes and separate it
from the input status value.
Reviewed-by: NViktor Dukhovni <viktor@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/6198)

The return value from the ticket_key callback was not properly handled in
TLSv1.3, so that a ticket was *always* renewed even if the callback
requested that it should not be.

Also the ticket decrypt callback was not being called at all in TLSv1.3.
Reviewed-by: NViktor Dukhovni <viktor@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/6198)

The MAX_CURVELIST macro defines the total number of in-built SSL/TLS curves
that we support. However it has not been updated as new curves are added.

Fixes #5232
Reviewed-by: NBernd Edlinger <bernd.edlinger@hotmail.de>
(Merged from https://github.com/openssl/openssl/pull/6065)

The spec does not prohib certs form using compressed points. It only
requires that points in a key share are uncompressed. It says nothing
about point compression for certs, so we should not fail if a cert uses a
compressed point.

Fixes #5743
Reviewed-by: NRich Salz <rsalz@openssl.org>
Reviewed-by: NBernd Edlinger <bernd.edlinger@hotmail.de>
(Merged from https://github.com/openssl/openssl/pull/6055)

Almost all *alloc failures now set an error code.
Reviewed-by: NMatthias St. Pierre <Matthias.St.Pierre@ncp-e.com>
(Merged from https://github.com/openssl/openssl/pull/5842)

If a server has been configured to use an ECDSA certificate, we should
allow it regardless of whether the server's own supported groups list
includes the certificate's group.

Fixes #2033
Reviewed-by: NRich Salz <rsalz@openssl.org>
Reviewed-by: NBernd Edlinger <bernd.edlinger@hotmail.de>
(Merged from https://github.com/openssl/openssl/pull/5601)

Place the session ticket AES and HMAC keys into secure memory.
Reviewed-by: NRichard Levitte <levitte@openssl.org>
Reviewed-by: NRich Salz <rsalz@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/2351)

Conceptually in TLSv1.3 there can be multiple sessions associated with a
single connection. Each NewSessionTicket issued can be considered a
separate session. We can end up issuing multiple NewSessionTickets on a
single connection at the moment (e.g. in a post-handshake auth scenario).
Each of those issued tickets should have the new_session_cb called, it
should go into the session cache separately and it should have a unique
id associated with it (so that they can be found individually in the
cache).
Reviewed-by: NRich Salz <rsalz@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/5644)

Adds application data into the encrypted session ticket
Reviewed-by: NPaul Dale <paul.dale@oracle.com>
Reviewed-by: NMatt Caswell <matt@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/3802)

Reviewed-by: NRich Salz <rsalz@openssl.org>
Reviewed-by: NKurt Roeckx <kurt@roeckx.be>
(Merged from https://github.com/openssl/openssl/pull/5470)

This could in theory result in an overread - but due to the over allocation
of the underlying buffer does not represent a security issue.

Thanks to Fedor Indutny for reporting this issue.
Reviewed-by: NRich Salz <rsalz@openssl.org>
Reviewed-by: NBen Kaduk <kaduk@mit.edu>
(Merged from https://github.com/openssl/openssl/pull/5414)

We should always check the return code.

This fixes a coverity issue.
Reviewed-by: NTim Hudson <tjh@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/5308)

The check for a duplicate value was reading one entry past
where it was supposed to, getting an uninitialized value.
Reviewed-by: NMatt Caswell <matt@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/5172)

The new extension is like signature_algorithms, but only for the
signature *on* the certificate we will present to the peer (the
old signature_algorithms extension is still used for signatures that
we *generate*, i.e., those over TLS data structures).

We do not need to generate this extension, since we are the same
implementation as our X.509 stack and can handle the same types
of signatures, but we need to be prepared to receive it, and use the received
information when selecting what certificate to present.

There is a lot of interplay between signature_algorithms_cert and
signature_algorithms, since both affect what certificate we can
use, and thus the resulting signature algorithm used for TLS messages.
So, apply signature_algorithms_cert (if present) as a filter on what
certificates we can consider when choosing a certificate+sigalg
pair.

As part of this addition, we also remove the fallback code that let
keys of type EVP_PKEY_RSA be used to generate RSA-PSS signatures -- the
new rsa_pss_pss_* and rsa_pss_rsae_* signature schemes have pulled
the key type into what is covered by the signature algorithm, so
we should not apply this sort of compatibility workaround.
Reviewed-by: NMatt Caswell <matt@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/5068)

Our historical SSL{,_CTX}_set_sigalgs() APIs take an array of
NID pairs (hash and signature), and our parser for manually
specifying unified sigalgs (that do not necessarily correspond
to an actual signature+hash pair) was transiting via (the implementation
of) this historical API.  The TLS 1.3 draft-23 has introduced
signature schemes that have identical signature type and hash type,
differing only in the (RSA) public key OID, which prevents
the rsa_pss_pss_* schemes from being properly identified and
sent on the wire.

To fix the issue, parse sigalg strings directly into SIGALG_LOOKUP
objects, and pass around an array of uint16 wire protocol values
instead of NID pairs.  The old interface is retained for API
compatibility but will become less and less useful with time.
Reviewed-by: NMatt Caswell <matt@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/5068)

We now have a split in the signature algorithms codepoint space for
whether the certificate's key is for rsaEncryption or a PSS-specific
key, which should let us get rid of some special-casing that we
previously needed to try to coax rsaEncryption keys into performing PSS.
(This will be done in a subsequent commit.)

Send the new PSS-with-PSS-specific key first in our list, so that
we prefer the new technology to the old one.

We need to update the expected certificate type in one test,
since the "RSA-PSS+SHA256" form now corresponds to a public key
of type rsaEncryption, so we should expect the server certificate
type to be just "RSA".  If we want to get a server certificate
type of "RSA-PSS", we need to use a new signature algorithm
that cannot be represented as signature+hash, so add a test for that
as well.
Reviewed-by: NMatt Caswell <matt@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/5068)

Reviewed-by: NTim Hudson <tjh@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/5038)

Extract the RSA key using EVP_PKEY_get0.  Type is checked externally to be either EVP_PKEY_RSA_PSS or EVP_PKEY_RSA.
Reviewed-by: NRichard Levitte <levitte@openssl.org>
Reviewed-by: NRich Salz <rsalz@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/4389)