Fixes #5203
Reviewed-by: NRich Salz <rsalz@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/5207)

If a server receives an unexpected ClientHello then we may or may not
accept it. Make sure all such decisions are made in the state machine
and not in the record layer. This also removes a disparity between the
TLS and the DTLS code. The TLS code was making this decision in the
record layer, while the DTLS code was making it later.

Finally it also solves a problem where a warning alert was being sent
during tls_setup_handshake() and the function was returning a failure
return code. This is problematic because it can be called from a
transition function - which we only allow fatal errors to occur in.
Reviewed-by: NRich Salz <rsalz@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/5190)

Reviewed-by: NTim Hudson <tjh@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/5197)

For proper escaping, we need the direct perl variable values, not a
make variable reference.
Reviewed-by: NTim Hudson <tjh@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/5196)

Reviewed-by: NTim Hudson <tjh@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/5196)

This message will ONLY be visible in OpenSSL 1.1.1, it will not show
in 1.1.1a or any other release or update.
Reviewed-by: NRich Salz <rsalz@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/5185)

Reviewed-by: NRich Salz <rsalz@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/5185)

Reviewed-by: NRich Salz <rsalz@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/5185)

The additional possibility is:

perl configdata.pm --options            Display the features, both
                                        enabled and disabled, and
                                        display defined macro and
                                        skipped directories where
                                        applicable.
Reviewed-by: NRich Salz <rsalz@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/5185)

Reviewed-by: NRich Salz <rsalz@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/5185)

The "make variable" information displayed by Configure was selective
and incomplete, and possibly undesirable (too verbose).

Instead, we make configdata.pm and have the user run it to get the
information they desire, and also make it possible to have it perform
a reconfiguration.

Possibilities so far:

perl configdata.pm --dump               Displays everything (i.e. the
                                        combined output from
                                        --command-line, --environment,
                                        --make-variables and
                                        --build-parameters.
perl configdata.pm --command-line       Displays the config command
                                        line.
perl configdata.pm --envirnoment        Displays the recorded
                                        environment variables.
perl configdata.pm --make-variables     Displays the configured "make
                                        variables".
perl configdata.pm --build-parameters   Displays the build file and
                                        the template files to create
                                        it.
perl configdata.pm --reconfigure        Re-runs the configuration with
                                        the recorded environment
                                        variables.

--verbose can be used to have --reconfigure be a bit more verbose.
Reviewed-by: NRich Salz <rsalz@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/5185)

It's already in opensslconf.h, which is included where this is relevant.
Reviewed-by: NRich Salz <rsalz@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/5188)

Reviewed-by: NRich Salz <rsalz@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/5188)

Thank you Beat Bolli for notifying us
Reviewed-by: NRich Salz <rsalz@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/5189)

The rehash test broke the test if run by root.  Instead, just skip the
check that requires non-root to be worth it.

Fixes #4387
Reviewed-by: NMatt Caswell <matt@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/5184)

Reviewed-by: NPaul Dale <paul.dale@oracle.com>
Reviewed-by: NMatthias St. Pierre <Matthias.St.Pierre@ncp-e.com>
(Merged from https://github.com/openssl/openssl/pull/4752)

Fixes: #5130
Reviewed-by: NRich Salz <rsalz@openssl.org>
Reviewed-by: NMatthias St. Pierre <Matthias.St.Pierre@ncp-e.com>
(Merged from https://github.com/openssl/openssl/pull/5178)

Reviewed-by: NRich Salz <rsalz@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/5181)

C++ flags got the same config target value as C flags, but then
nothing else happened while C flags get all kinds of stuff added to
them (especially when --strict-warnings is used).

Now, C++ flags get the exact same treatment as C flags.  However, this
only happens when a C++ compiler is specified, to avoid confusing
messages about added C++ flags.
Reviewed-by: NRich Salz <rsalz@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/5181)

vc_wince_info()->{defines} was left around, when it should be
vc_wince_info()->{cppflags}
Reviewed-by: NTim Hudson <tjh@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/5182)

Most of all, this change preserves casing a bit better
Reviewed-by: NRich Salz <rsalz@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/5180)

Reviewed-by: NRichard Levitte <levitte@openssl.org>
Reviewed-by: NMatthias St. Pierre <Matthias.St.Pierre@ncp-e.com>
(Merged from https://github.com/openssl/openssl/pull/5166)

Reviewed-by: NTim Hudson <tjh@openssl.org>
Reviewed-by: NRich Salz <rsalz@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/5177)

There were a small number that inherited no BASE, the now inherit
BASE_unix.
Reviewed-by: NTim Hudson <tjh@openssl.org>
Reviewed-by: NRich Salz <rsalz@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/5177)

Ideally, each config target should inherit a base to get their
platform specific defaults.  Unfortunately, that is currently not the
case, so we duplicate the Unixly defaults from the BASE_unix template
into the DEFAULT template.
Reviewed-by: NTim Hudson <tjh@openssl.org>
Reviewed-by: NRich Salz <rsalz@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/5177)

Default values belong in the DEFAULT config target template, in
Configurations/00-base-templates.conf.

This isn't a complete move, but takes care of the most blatant
examples.
Reviewed-by: NTim Hudson <tjh@openssl.org>
Reviewed-by: NRich Salz <rsalz@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/5177)

Support the following "make variables":

AR              (GNU compatible)
ARFLAGS         (GNU Compatible)
AS              (GNU Compatible)
ASFLAGS         (GNU Compatible)
CC              (GNU Compatible)
CFLAGS          (GNU Compatible)
CXX             (GNU Compatible)
CXXFLAGS        (GNU Compatible)
CPP             (GNU Compatible)
CPPFLAGS        (GNU Compatible)
CPPDEFINES      List of CPP macro definitions.  Alternative for -D
CPPINCLUDES     List of CPP inclusion directories.  Alternative for -I
HASHBANGPERL    Perl invocation to be inserted after '#!' in public
                perl scripts.
LDFLAGS         (GNU Compatible)
LDLIBS          (GNU Compatible)
RANLIB          Program to generate library archive index
RC              Program to manipulate Windows resources
RCFLAGS         Flags for $(RC)
RM              (GNU Compatible)

Setting one of these overrides the corresponding data from our config
targets.  However, flags given directly on the configuration command
line are additional, and are therefore added to the flags coming from
one of the variables above or the config target.

Fixes #2420
Reviewed-by: NTim Hudson <tjh@openssl.org>
Reviewed-by: NRich Salz <rsalz@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/5177)

C preprocessor flags get separated from C flags, which has the
advantage that we don't get loads of macro definitions and inclusion
directory specs when linking shared libraries, DSOs and programs.

This is a step to add support for "make variables" when configuring.
Reviewed-by: NTim Hudson <tjh@openssl.org>
Reviewed-by: NRich Salz <rsalz@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/5177)

Reviewed-by: NRich Salz <rsalz@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/5173)

Make the sigalg name in comments reflect one that actually exists
in the draft standard.
Reviewed-by: NMatt Caswell <matt@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/5174)

The latest TLS 1.3 draft split the RSA-PSS signature schemes into
two versions that indicate the OID of the RSA key being used.
This forced us to rename the preprocessor defines for the sigalg
values, and the ssl-trace code was not adopted to match, since
it was not enabled int the default build.

Belatedly update the ssl_sigalg_tbl in the trace code to match.
Reviewed-by: NMatt Caswell <matt@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/5174)

The check for a duplicate value was reading one entry past
where it was supposed to, getting an uninitialized value.
Reviewed-by: NMatt Caswell <matt@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/5172)

We don't need to send this extension in normal operation since
we are our own X.509 library, but add some test cases that force
the extension to be sent and exercise our code to process the extension.
Reviewed-by: NMatt Caswell <matt@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/5068)

The new extension is like signature_algorithms, but only for the
signature *on* the certificate we will present to the peer (the
old signature_algorithms extension is still used for signatures that
we *generate*, i.e., those over TLS data structures).

We do not need to generate this extension, since we are the same
implementation as our X.509 stack and can handle the same types
of signatures, but we need to be prepared to receive it, and use the received
information when selecting what certificate to present.

There is a lot of interplay between signature_algorithms_cert and
signature_algorithms, since both affect what certificate we can
use, and thus the resulting signature algorithm used for TLS messages.
So, apply signature_algorithms_cert (if present) as a filter on what
certificates we can consider when choosing a certificate+sigalg
pair.

As part of this addition, we also remove the fallback code that let
keys of type EVP_PKEY_RSA be used to generate RSA-PSS signatures -- the
new rsa_pss_pss_* and rsa_pss_rsae_* signature schemes have pulled
the key type into what is covered by the signature algorithm, so
we should not apply this sort of compatibility workaround.
Reviewed-by: NMatt Caswell <matt@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/5068)

These functions can now take both "sig+hash" strings and
algorithm-specific identifiers like "rsa_pss_pss_sha256" that
indicate a particular entry from the TLS signature algorithm
registry.

Also clarify that only the "_list" form allows for the new-style names
(the non-"list" interfaces take sig and hasn NIDs, which cannot
access all of the new-style schemes).
Reviewed-by: NMatt Caswell <matt@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/5068)

Our historical SSL{,_CTX}_set_sigalgs() APIs take an array of
NID pairs (hash and signature), and our parser for manually
specifying unified sigalgs (that do not necessarily correspond
to an actual signature+hash pair) was transiting via (the implementation
of) this historical API.  The TLS 1.3 draft-23 has introduced
signature schemes that have identical signature type and hash type,
differing only in the (RSA) public key OID, which prevents
the rsa_pss_pss_* schemes from being properly identified and
sent on the wire.

To fix the issue, parse sigalg strings directly into SIGALG_LOOKUP
objects, and pass around an array of uint16 wire protocol values
instead of NID pairs.  The old interface is retained for API
compatibility but will become less and less useful with time.
Reviewed-by: NMatt Caswell <matt@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/5068)

We now have a split in the signature algorithms codepoint space for
whether the certificate's key is for rsaEncryption or a PSS-specific
key, which should let us get rid of some special-casing that we
previously needed to try to coax rsaEncryption keys into performing PSS.
(This will be done in a subsequent commit.)

Send the new PSS-with-PSS-specific key first in our list, so that
we prefer the new technology to the old one.

We need to update the expected certificate type in one test,
since the "RSA-PSS+SHA256" form now corresponds to a public key
of type rsaEncryption, so we should expect the server certificate
type to be just "RSA".  If we want to get a server certificate
type of "RSA-PSS", we need to use a new signature algorithm
that cannot be represented as signature+hash, so add a test for that
as well.
Reviewed-by: NMatt Caswell <matt@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/5068)

Reviewed-by: NMatt Caswell <matt@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/5068)

Reviewed-by: NMatt Caswell <matt@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/5068)

Reviewed-by: NRich Salz <rsalz@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/5152)