TLS < 1.2 has fixed signature algorithms: MD5+SHA1 for RSA and SHA1 for the
others. TLS 1.2 sends a list of supported ciphers, but allows not sending
it in which case SHA1 is used. TLS 1.3 makes sending the list mandatory.

When we didn't receive a list from the client, we always used the
defaults without checking that they are allowed by the configuration.
Reviewed-by: NPaul Dale <paul.dale@oracle.com>
GH: #10784

Many Windows-based GOST TLS implementations are unable to extend the
list of supported SignatureAlgorithms because of lack of the necessary
callback in Windows. So for TLS 1.2 it makes sense to imply the support
of GOST algorithms in case when the GOST ciphersuites are present.
Reviewed-by: NPaul Dale <paul.dale@oracle.com>
(Merged from https://github.com/openssl/openssl/pull/10377)

Apart from public and internal header files, there is a third type called
local header files, which are located next to source files in the source
directory. Currently, they have different suffixes like

  '*_lcl.h', '*_local.h', or '*_int.h'

This commit changes the different suffixes to '*_local.h' uniformly.
Reviewed-by: NRichard Levitte <levitte@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/9333)

In commit 2d263a4a ("Honour mandatory digest on private key in
has_usable_cert()" I added two checks for the capabilities of the
EVP_PKEY being used. One of them was wrong, as it should only be
checking the signature of the X.509 cert (by its issuer) against the
sigalgs given in a TLS v1.3 signature_algorithms_cert extension.

Remove it and provide the code comments which, if they'd been present
in the first place, would hopefully have prevented the mistake.
Reviewed-by: NTomas Mraz <tmraz@fedoraproject.org>
Reviewed-by: NMatt Caswell <matt@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/9672)

The function SSL_check_chain() can be used by applications to check that
a cert and chain is compatible with the negotiated parameters. This could
be useful (for example) from the certificate callback. Unfortunately this
function was applying TLSv1.2 sig algs rules and did not work correctly if
TLSv1.3 was negotiated.

We refactor tls_choose_sigalg to split it up and create a new function
find_sig_alg which can (optionally) take a certificate and key as
parameters and find an appropriate sig alg if one exists. If the cert and
key are not supplied then we try to find a cert and key from the ones we
have available that matches the shared sig algs.
Reviewed-by: NTomas Mraz <tmraz@fedoraproject.org>
(Merged from https://github.com/openssl/openssl/pull/9442)

Reviewed-by: NPaul Dale <paul.dale@oracle.com>
Reviewed-by: NMatt Caswell <matt@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/9323)

Some have been creeping into the source code.
Reviewed-by: NPaul Yang <yang.yang@baishancloud.com>
(Merged from https://github.com/openssl/openssl/pull/9397)

It was only ever in cert_st because ssl_st was a public structure
and could not be modified without breaking the API.  However, both
structures are now opaque, and thus we can freely change their layout
without breaking applications.  In this case, keeping the shared
sigalgs in the SSL object prevents complications wherein they would
inadvertently get cleared during SSL_set_SSL_CTX() (e.g., as run
during a cert_cb).

Fixes #9099
Reviewed-by: NMatt Caswell <matt@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/9157)

The previous 2 commits moved supported groups and ciphers out of the
session object to avoid race conditions. We now also move ecpointformats
for consistency. There does not seem to be a race condition with access
to this data since it is only ever set in a non-resumption handshake.
However, there is no reason for it to be in the session.
Reviewed-by: NTomas Mraz <tmraz@fedoraproject.org>
(Merged from https://github.com/openssl/openssl/pull/9162)

Make sure that the combination of no-ec with no-dh builds successfully.
If neither ec or dh are available then TLSv1.3 is not possible.
Reviewed-by: NRichard Levitte <levitte@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/9156)

Now that we have TLSv1.3 FFDHE support there is no reason why we should
not allow TLSv1.3 to be used in a no-ec build. This commit enables that
to happen.

It also fixes no-ec which was previously broken.
Reviewed-by: NRichard Levitte <levitte@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/9156)

Reviewed-by: NTomas Mraz <tmraz@fedoraproject.org>
Reviewed-by: NMatt Caswell <matt@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/8178)

With the removal of SSLv2, the s3 structure is always allocated, so
there is little point in having it be an allocated pointer. Collapse
the ssl3_state_st structure into ssl_st and fixup any references.

This should be faster than going through an indirection and due to
fewer allocations, but I'm not seeing any significant performance
improvement; it seems to be within the margin of error in timing.
Reviewed-by: NPaul Yang <yang.yang@baishancloud.com>
Reviewed-by: NMatt Caswell <matt@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/7888)

Use 0 if we don't want to set a minimum or maximum version
Reviewed-by: NMatt Caswell <matt@openssl.org>
GH: #7260

Reviewed-by: NMatt Caswell <matt@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/7768)

Reviewed-by: NRichard Levitte <levitte@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/7620)

    SSL_get_signature_nid()      -- local signature algorithm
    SSL_get_signature_type_nid() -- local signature algorithm key type
    SSL_get_peer_tmp_key()       -- Peer key-exchange public key
    SSL_get_tmp_key              -- local key exchange public key

Aliased pre-existing SSL_get_server_tmp_key(), which was formerly
just for clients, to SSL_get_peer_tmp_key().  Changed internal
calls to use the new name.
Reviewed-by: NMatt Caswell <matt@openssl.org>

TLSv1.3 is more restrictive about the curve used. There must be a matching
sig alg defined for that curve. Therefore if we are using some other curve
in our certificate then we should not negotiate TLSv1.3.

Fixes #7435
Reviewed-by: NViktor Dukhovni <viktor@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/7442)

Rather than relying only on mandatory default digests, add a way for
the EVP_PKEY to individually report whether each digest algorithm is
supported.
Reviewed-by: NNicola Tuveri <nic.tuv@gmail.com>
Reviewed-by: NRichard Levitte <levitte@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/7408)

If the private key says it can only support one specific digest, then
don't ask it to perform a different one.

Fixes: #7348
Reviewed-by: NNicola Tuveri <nic.tuv@gmail.com>
Reviewed-by: NRichard Levitte <levitte@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/7408)

Previously we indicated this as a malloc failure which isn't very
helpful.
Reviewed-by: NKurt Roeckx <kurt@roeckx.be>
(Merged from https://github.com/openssl/openssl/pull/7479)

If we have selected a ciphersuite using RSA key exchange then we must
not attempt to use an RSA-PSS cert for that.

Fixes #7059
Reviewed-by: NPaul Dale <paul.dale@oracle.com>
Reviewed-by: NTim Hudson <tjh@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/7099)

We need to ensure that the min-max version range we use when constructing
the ClientHello is the same range we use when we validate the version
selected by the ServerHello. Otherwise this may appear as a fallback or
downgrade.

Fixes #6964
Reviewed-by: NViktor Dukhovni <viktor@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/7013)

Using the rsa_pss_rsae_sha256 sig alg should imply that the key OID is
rsaEncryption. Similarly rsa_pss_pss_sha256 implies the key OID is
rsassaPss. However we did not check this and incorrectly tolerated a key
OID that did not match the sig alg sent by the peer.

Fixes #6611
Reviewed-by: NRich Salz <rsalz@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/6732)

Fixes #6513
Reviewed-by: NRich Salz <rsalz@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/6650)

In ssl/t1_lib.c.
Reviewed-by: NRich Salz <rsalz@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/6613)

Commit 61fb5923 broke EAP-FAST support. This fixes it.

Fixes #6395
Reviewed-by: NRich Salz <rsalz@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/6428)

Reviewed-by: NBernd Edlinger <bernd.edlinger@hotmail.de>
Reviewed-by: NMatt Caswell <matt@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/6168)

Reviewed-by: NBernd Edlinger <bernd.edlinger@hotmail.de>
Reviewed-by: NMatt Caswell <matt@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/6168)

Found by OSS-fuzz

Bug introduced in commit 61fb5923Reviewed-by: NMatt Caswell <matt@openssl.org>
GH: #6235

Don't call the decrypt ticket callback if we've already encountered a
fatal error. Do call it if we have an empty ticket present.

Change the return code to have 5 distinct returns codes and separate it
from the input status value.
Reviewed-by: NViktor Dukhovni <viktor@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/6198)

The return value from the ticket_key callback was not properly handled in
TLSv1.3, so that a ticket was *always* renewed even if the callback
requested that it should not be.

Also the ticket decrypt callback was not being called at all in TLSv1.3.
Reviewed-by: NViktor Dukhovni <viktor@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/6198)

The MAX_CURVELIST macro defines the total number of in-built SSL/TLS curves
that we support. However it has not been updated as new curves are added.

Fixes #5232
Reviewed-by: NBernd Edlinger <bernd.edlinger@hotmail.de>
(Merged from https://github.com/openssl/openssl/pull/6065)

The spec does not prohib certs form using compressed points. It only
requires that points in a key share are uncompressed. It says nothing
about point compression for certs, so we should not fail if a cert uses a
compressed point.

Fixes #5743
Reviewed-by: NRich Salz <rsalz@openssl.org>
Reviewed-by: NBernd Edlinger <bernd.edlinger@hotmail.de>
(Merged from https://github.com/openssl/openssl/pull/6055)

Almost all *alloc failures now set an error code.
Reviewed-by: NMatthias St. Pierre <Matthias.St.Pierre@ncp-e.com>
(Merged from https://github.com/openssl/openssl/pull/5842)

If a server has been configured to use an ECDSA certificate, we should
allow it regardless of whether the server's own supported groups list
includes the certificate's group.

Fixes #2033
Reviewed-by: NRich Salz <rsalz@openssl.org>
Reviewed-by: NBernd Edlinger <bernd.edlinger@hotmail.de>
(Merged from https://github.com/openssl/openssl/pull/5601)

Place the session ticket AES and HMAC keys into secure memory.
Reviewed-by: NRichard Levitte <levitte@openssl.org>
Reviewed-by: NRich Salz <rsalz@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/2351)

Conceptually in TLSv1.3 there can be multiple sessions associated with a
single connection. Each NewSessionTicket issued can be considered a
separate session. We can end up issuing multiple NewSessionTickets on a
single connection at the moment (e.g. in a post-handshake auth scenario).
Each of those issued tickets should have the new_session_cb called, it
should go into the session cache separately and it should have a unique
id associated with it (so that they can be found individually in the
cache).
Reviewed-by: NRich Salz <rsalz@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/5644)

Adds application data into the encrypted session ticket
Reviewed-by: NPaul Dale <paul.dale@oracle.com>
Reviewed-by: NMatt Caswell <matt@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/3802)

Reviewed-by: NRich Salz <rsalz@openssl.org>
Reviewed-by: NKurt Roeckx <kurt@roeckx.be>
(Merged from https://github.com/openssl/openssl/pull/5470)