We provide SSL_write_early() which *must* be called first on a connection
(prior to any other IO function including SSL_connect()/SSL_do_handshake()).
Also SSL_write_early_finish() which signals the end of early data.
Reviewed-by: NRich Salz <rsalz@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/2737)

Add the new enum value and case statements as appropriate.
Reviewed-by: NMatt Caswell <matt@openssl.org>
Reviewed-by: NRichard Levitte <levitte@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/2279)

The record layer was making decisions that should really be left to the
state machine around unexpected handshake messages that are received after
the initial handshake (i.e. renegotiation related messages). This commit
removes that code from the record layer and updates the state machine
accordingly. This simplifies the state machine and paves the way for
handling other messages post-handshake such as the NewSessionTicket in
TLSv1.3.
Reviewed-by: NRich Salz <rsalz@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/2259)

This flag is never set by anything so remove it.
Reviewed-by: NRich Salz <rsalz@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/2259)

In a non client-auth renegotiation where the original handshake *was*
client auth, then the server will expect the client to send a Certificate
message anyway resulting in a connection failure.
Reviewed-by: NRichard Levitte <levitte@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/1982)

In a non client-auth renegotiation where the original handshake *was*
client auth, then the client will send a Certificate message anyway
resulting in a connection failure.

Fixes #1920
Reviewed-by: NRichard Levitte <levitte@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/1982)

Reviewed-by: NRich Salz <rsalz@openssl.org>
Reviewed-by: NTim Hudson <tjh@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/1669)

Writing still to be done
Reviewed-by: NRich Salz <rsalz@openssl.org>

We now set the handshake header, and close the packet directly in the
write_state_machine. This is now possible because it is common for all
messages.
Reviewed-by: NRich Salz <rsalz@openssl.org>

Instead of initialising, finishing and cleaning up the WPACKET in every
message construction function, we should do it once in
write_state_machine().
Reviewed-by: NRich Salz <rsalz@openssl.org>

The buffer to receive messages is initialised to 16k. If a message is
received that is larger than that then the buffer is "realloc'd". This can
cause the location of the underlying buffer to change. Anything that is
referring to the old location will be referring to free'd data. In the
recent commit c1ef7c97 (master) and 4b390b6c (1.1.0) the point in the code
where the message buffer is grown was changed. However s->init_msg was not
updated to point at the new location.

CVE-2016-6309
Reviewed-by: NEmilia Käsper <emilia@openssl.org>

Change code so when switching on an enumeration, have case's for all
enumeration values.
Reviewed-by: NAndy Polyakov <appro@openssl.org>

Reviewed-by: NMatt Caswell <matt@openssl.org>

A TLS message includes 3 bytes for its length in the header for the message.
This would allow for messages up to 16Mb in length. Messages of this length
are excessive and OpenSSL includes a check to ensure that a peer is sending
reasonably sized messages in order to avoid too much memory being consumed
to service a connection. A flaw in the logic of version 1.1.0 means that
memory for the message is allocated too early, prior to the excessive
message length check. Due to way memory is allocated in OpenSSL this could
mean an attacker could force up to 21Mb to be allocated to service a
connection. This could lead to a Denial of Service through memory
exhaustion. However, the excessive message length check still takes place,
and this would cause the connection to immediately fail. Assuming that the
application calls SSL_free() on the failed conneciton in a timely manner
then the 21Mb of allocated memory will then be immediately freed again.
Therefore the excessive memory allocation will be transitory in nature.
This then means that there is only a security impact if:

1) The application does not call SSL_free() in a timely manner in the
event that the connection fails
or
2) The application is working in a constrained environment where there
is very little free memory
or
3) The attacker initiates multiple connection attempts such that there
are multiple connections in a state where memory has been allocated for
the connection; SSL_free() has not yet been called; and there is
insufficient memory to service the multiple requests.

Except in the instance of (1) above any Denial Of Service is likely to
be transitory because as soon as the connection fails the memory is
subsequently freed again in the SSL_free() call. However there is an
increased risk during this period of application crashes due to the lack
of memory - which would then mean a more serious Denial of Service.

This issue does not affect DTLS users.

Issue was reported by Shi Lei (Gear Team, Qihoo 360 Inc.).

CVE-2016-6307
Reviewed-by: NRichard Levitte <levitte@openssl.org>

Makes the logic a little bit clearer.
Reviewed-by: NAndy Polyakov <appro@openssl.org>
Reviewed-by: NRich Salz <rsalz@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/1571)

Run util/openssl-format-source on ssl/

Some comments and hand-formatted tables were fixed up
manually by disabling auto-formatting.
Reviewed-by: NRich Salz <rsalz@openssl.org>

Fix some indentation at the same time
Reviewed-by: NMatt Caswell <matt@openssl.org>
Reviewed-by: NRich Salz <rsalz@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/1292)

In TLS if the server sends a CertificateRequest and the client does not
provide one, if the server cannot continue it should send a
HandshakeFailure alert. In SSLv3 the same should happen, but instead we
were sending an UnexpectedMessage alert. This is incorrect - the message
isn't unexpected - it is valid for the client not to send one - its just
that we cannot continue without one.
Reviewed-by: NEmilia Käsper <emilia@openssl.org>

The ssl3_init_finished_mac() function can fail, in which case we need to
propagate the error up through the stack.

RT#3198
Reviewed-by: NRich Salz <rsalz@openssl.org>

In the new state machine if using nbio and we get the header of a
handshake message is one record with the body in the next, with an nbio
event in the middle, then the connection was failing. This is because
s->init_num was getting reset. We should only reset it after we have
read the whole message.

RT#4394
Reviewed-by: NAndy Polyakov <appro@openssl.org>

The write BIO for handshake messages is bufferred so that we only write
out to the network when we have a complete flight. There was some
complexity in the buffering logic so that we switched buffering on and
off at various points through out the handshake. The only real reason to
do this was historically it complicated the state machine when you wanted
to flush because you had to traverse through the "flush" state (in order
to cope with NBIO). Where we knew up front that there was only going to
be one message in the flight we switched off buffering to avoid that.

In the new state machine there is no longer a need for a flush state so
it is simpler just to have buffering on for the whole handshake. This
also gives us the added benefit that we can simply call flush after every
flight even if it only has one message in it. This means that BIO authors
can implement their own buffering strategies and not have to be aware of
the state of the SSL object (previously they would have to switch off
their own buffering during the handshake because they could not rely on
a flush being received when they really needed to write data out). This
last point addresses GitHub Issue #322.
Reviewed-by: NAndy Polyakov <appro@openssl.org>

Reviewed-by: NRichard Levitte <levitte@openssl.org>
Reviewed-by: NKurt Roeckx <kurt@openssl.org>

Signed-off-by: NRich Salz <rsalz@openssl.org>
Reviewed-by: NMatt Caswell <matt@openssl.org>

This was done by the following
        find . -name '*.[ch]' | /tmp/pl
where /tmp/pl is the following three-line script:
        print unless $. == 1 && m@/\* .*\.[ch] \*/@;
        close ARGV if eof; # Close file to reset $.

And then some hand-editing of other files.
Reviewed-by: NViktor Dukhovni <viktor@openssl.org>

The protocol selection code is now consolidated in a few consecutive
short functions in a single file and is table driven.  Protocol-specific
constraints that influence negotiation are moved into the flags
field of the method structure.  The same protocol version constraints
are now applied in all code paths.  It is now much easier to add
new protocol versions without reworking the protocol selection
logic.

In the presence of "holes" in the list of enabled client protocols
we no longer select client protocols below the hole based on a
subset of the constraints and then fail shortly after when it is
found that these don't meet the remaining constraints (suiteb, FIPS,
security level, ...).  Ideally, with the new min/max controls users
will be less likely to create "holes" in the first place.
Reviewed-by: NKurt Roeckx <kurt@openssl.org>

The SCTP code is not compiled by default. This fixes some compilation
problems in that code.
Reviewed-by: NTim Hudson <tjh@openssl.org>

Add the ossl_statem prefix to various funtions to avoid name clashes.
Reviewed-by: NTim Hudson <tjh@openssl.org>
Reviewed-by: NRichard Levitte <levitte@openssl.org>

Various enums were introduced as part of the state machine rewrite. As a
matter of style it is preferred for these to be typedefs.
Reviewed-by: NTim Hudson <tjh@openssl.org>
Reviewed-by: NRichard Levitte <levitte@openssl.org>

Clang with --strict-warnings was complaining about an uninitalised
variable. In reality it will never be used uninitialised but clang can't
figure out the logic, so just init it anyway to silence the warning.
Reviewed-by: NTim Hudson <tjh@openssl.org>
Reviewed-by: NRichard Levitte <levitte@openssl.org>

The SSL variable |in_handshake| seems misplaced. It would be better to have
it in the STATEM structure.
Reviewed-by: NTim Hudson <tjh@openssl.org>
Reviewed-by: NRichard Levitte <levitte@openssl.org>

Adding the new state machine broke the DTLSv1_listen code because
calling SSL_in_before() was erroneously returning true after DTLSv1_listen
had successfully completed. This change ensures that SSL_in_before returns
false.
Reviewed-by: NTim Hudson <tjh@openssl.org>
Reviewed-by: NRichard Levitte <levitte@openssl.org>

Remove repeated blocks of checking SSL and then SSL_CTX for the
info_callback.
Reviewed-by: NTim Hudson <tjh@openssl.org>
Reviewed-by: NRichard Levitte <levitte@openssl.org>

Reviewed-by: NTim Hudson <tjh@openssl.org>
Reviewed-by: NRichard Levitte <levitte@openssl.org>

Add an OSSL_ prefix onto STATEM
Reviewed-by: NTim Hudson <tjh@openssl.org>
Reviewed-by: NRichard Levitte <levitte@openssl.org>

SSL_state has been replaced by SSL_get_state and SSL_set_state is no longer
supported.
Reviewed-by: NTim Hudson <tjh@openssl.org>
Reviewed-by: NRichard Levitte <levitte@openssl.org>

The |no_cert_verify| should be in the state machine structure not in SSL
Reviewed-by: NTim Hudson <tjh@openssl.org>
Reviewed-by: NRichard Levitte <levitte@openssl.org>

Change various state machine functions to use the prefix ossl_statem
instead.
Reviewed-by: NTim Hudson <tjh@openssl.org>
Reviewed-by: NRichard Levitte <levitte@openssl.org>

Rename the enum HANDSHAKE_STATE to OSSL_HANDSHAKE_STATE to ensure there are
no namespace clashes, and convert it into a typedef.
Reviewed-by: NTim Hudson <tjh@openssl.org>
Reviewed-by: NRichard Levitte <levitte@openssl.org>

Fix an out of date reference to old state machine code in a comment
Reviewed-by: NTim Hudson <tjh@openssl.org>
Reviewed-by: NRichard Levitte <levitte@openssl.org>

Move some function definitions around within the state machine to make sure
they are in the correct files. Also create a statem_locl.h header for stuff
entirely local to the state machine code and move various definitions into
it.
Reviewed-by: NTim Hudson <tjh@openssl.org>
Reviewed-by: NRichard Levitte <levitte@openssl.org>