Previously when a ClientHello arrives with a valid cookie using
DTLSv1_listen() we only "peeked" at the message and left it on the
underlying fd. This works fine for single threaded applications but for
multi-threaded apps this does not work since the fd is typically reused for
the server thread, while a new fd is created and connected for the client.
By "peeking" we leave the message on the server fd, and consequently we
think we've received another valid ClientHello and so we create yet another
fd for the client, and so on until we run out of fds.

In this new approach we remove the ClientHello and buffer it in the SSL
object.

Fixes #6934
Reviewed-by: NBen Kaduk <kaduk@mit.edu>
(Merged from https://github.com/openssl/openssl/pull/7375)

Rather than using init_buf we use the record layer read and write buffers
in DTLSv1_listen(). These seem more appropriate anyway and will help with
the next commit.
Reviewed-by: NBen Kaduk <kaduk@mit.edu>
(Merged from https://github.com/openssl/openssl/pull/7375)

Reviewed-by: NRich Salz <rsalz@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/4841)

Since return is inconsistent, I removed unnecessary parentheses and
unified them.
Reviewed-by: NRich Salz <rsalz@openssl.org>
Reviewed-by: NMatt Caswell <matt@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/4541)

Reviewed-by: NMatt Caswell <matt@openssl.org>
Reviewed-by: NBernd Edlinger <bernd.edlinger@hotmail.de>
(Merged from https://github.com/openssl/openssl/pull/4011)

Move struct timeval includes into e_os.h (where the Windows ones were).
Enaure that the include is guarded canonically.

Refer #4271
Reviewed-by: NRich Salz <rsalz@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/4312)

cryptilib.h is the second.
Reviewed-by: NAndy Polyakov <appro@openssl.org>
Reviewed-by: NRichard Levitte <levitte@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/4188)

Removed e_os.h from all bar three headers (apps/apps.h crypto/bio/bio_lcl.h and
ssl/ssl_locl.h).

Added e_os.h into the files that need it now.

Directly reference internal/nelem.h when required.
Reviewed-by: NAndy Polyakov <appro@openssl.org>
Reviewed-by: NRichard Levitte <levitte@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/4188)

Remove GETPID_IS_MEANINGLESS and osslargused.

Move socket-related things to new file internal/sockets.h; this is now
only needed by four(!!!) files.  Compiles should be a bit faster.
Remove USE_SOCKETS ifdef's
Reviewed-by: NRichard Levitte <levitte@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/4209)

Provides consistent output and approach.
Reviewed-by: NTim Hudson <tjh@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/3496)

Reviewed-by: NTim Hudson <tjh@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/3496)

Reviewed-by: NRich Salz <rsalz@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/2775)

In 1.1.0 changing the ciphersuite during a renegotiation can result in
a crash leading to a DoS attack. In master this does not occur with TLS
(instead you get an internal error, which is still wrong but not a security
issue) - but the problem still exists in the DTLS code.

The problem is caused by changing the flag indicating whether to use ETM
or not immediately on negotiation of ETM, rather than at CCS. Therefore,
during a renegotiation, if the ETM state is changing (usually due to a
change of ciphersuite), then an error/crash will occur.

Due to the fact that there are separate CCS messages for read and write
we actually now need two flags to determine whether to use ETM or not.

CVE-2017-3733
Reviewed-by: NRichard Levitte <levitte@openssl.org>

Calling SSL_set_accept_state() after DTLSv1_listen() clears the state, so
SSL_accept() no longer works. In 1.0.2 calling DTLSv1_listen() would set
the accept state automatically. We should still do that.

Fixes #1989
Reviewed-by: NAndy Polyakov <appro@openssl.org>

Reviewed-by: NRich Salz <rsalz@openssl.org>
Reviewed-by: NTim Hudson <tjh@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/1669)

And use them in the DTLS code
Reviewed-by: NRich Salz <rsalz@openssl.org>

Reviewed-by: NRich Salz <rsalz@openssl.org>

Reviewed-by: NRich Salz <rsalz@openssl.org>

We add ssl_cipher_get_overhead() as an internal function, to avoid
having too much ciphersuite-specific knowledge in DTLS_get_data_mtu()
itself. It's going to need adjustment for TLSv1.3... but then again, so
is fairly much *all* of the SSL_CIPHER handling. This bit is in the noise.
Reviewed-by: NRich Salz <rsalz@openssl.org>
Reviewed-by: NMatt Caswell <matt@openssl.org>

ssl_set_handshake_header2() was only ever a temporary name while we had
to have ssl_set_handshake_header() for code that hadn't been converted to
WPACKET yet. No code remains that needed that so we can rename it.
Reviewed-by: NRich Salz <rsalz@openssl.org>

Remove the old ssl_set_handshake_header() implementations. Later we will
rename ssl_set_handshake_header2() to ssl_set_handshake_header().
Reviewed-by: NRich Salz <rsalz@openssl.org>

We actually construct a HelloVerifyRequest in two places with common code
pulled into a single function. This one commit handles both places.
Reviewed-by: NRich Salz <rsalz@openssl.org>

Reviewed-by: NRich Salz <rsalz@openssl.org>

DTLS can handle out of order record delivery. Additionally since
handshake messages can be bigger than will fit into a single packet, the
messages can be fragmented across multiple records (as with normal TLS).
That means that the messages can arrive mixed up, and we have to
reassemble them. We keep a queue of buffered messages that are "from the
future", i.e. messages we're not ready to deal with yet but have arrived
early. The messages held there may not be full yet - they could be one
or more fragments that are still in the process of being reassembled.

The code assumes that we will eventually complete the reassembly and
when that occurs the complete message is removed from the queue at the
point that we need to use it.

However, DTLS is also tolerant of packet loss. To get around that DTLS
messages can be retransmitted. If we receive a full (non-fragmented)
message from the peer after previously having received a fragment of
that message, then we ignore the message in the queue and just use the
non-fragmented version. At that point the queued message will never get
removed.

Additionally the peer could send "future" messages that we never get to
in order to complete the handshake. Each message has a sequence number
(starting from 0). We will accept a message fragment for the current
message sequence number, or for any sequence up to 10 into the future.
However if the Finished message has a sequence number of 2, anything
greater than that in the queue is just left there.

So, in those two ways we can end up with "orphaned" data in the queue
that will never get removed - except when the connection is closed. At
that point all the queues are flushed.

An attacker could seek to exploit this by filling up the queues with
lots of large messages that are never going to be used in order to
attempt a DoS by memory exhaustion.

I will assume that we are only concerned with servers here. It does not
seem reasonable to be concerned about a memory exhaustion attack on a
client. They are unlikely to process enough connections for this to be
an issue.

A "long" handshake with many messages might be 5 messages long (in the
incoming direction), e.g. ClientHello, Certificate, ClientKeyExchange,
CertificateVerify, Finished. So this would be message sequence numbers 0
to 4. Additionally we can buffer up to 10 messages in the future.
Therefore the maximum number of messages that an attacker could send
that could get orphaned would typically be 15.

The maximum size that a DTLS message is allowed to be is defined by
max_cert_list, which by default is 100k. Therefore the maximum amount of
"orphaned" memory per connection is 1500k.

Message sequence numbers get reset after the Finished message, so
renegotiation will not extend the maximum number of messages that can be
orphaned per connection.

As noted above, the queues do get cleared when the connection is closed.
Therefore in order to mount an effective attack, an attacker would have
to open many simultaneous connections.

Issue reported by Quan Luo.

CVE-2016-2179
Reviewed-by: NRichard Levitte <levitte@openssl.org>

Run util/openssl-format-source on ssl/

Some comments and hand-formatted tables were fixed up
manually by disabling auto-formatting.
Reviewed-by: NRich Salz <rsalz@openssl.org>

DTLSv1_client_method() is deprecated, but it was the only way to obtain
DTLS1_BAD_VER support. The SSL_OP_CISCO_ANYCONNECT hack doesn't work with
DTLS_client_method(), and it's relatively non-trivial to make it work without
expanding the hack into lots of places.

So deprecate SSL_OP_CISCO_ANYCONNECT with DTLSv1_client_method(), and make
it work with SSL_CTX_set_{min,max}_proto_version(DTLS1_BAD_VER) instead.
Reviewed-by: NRich Salz <rsalz@openssl.org>
Reviewed-by: NMatt Caswell <matt@openssl.org>

Fix some indentation at the same time
Reviewed-by: NMatt Caswell <matt@openssl.org>
Reviewed-by: NRich Salz <rsalz@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/1292)

Reviewed-by: NAndy Polyakov <appro@openssl.org>
Reviewed-by: NKurt Roeckx <kurt@openssl.org>
Reviewed-by: NRich Salz <rsalz@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/1264)

Reviewed-by: NKurt Roeckx <kurt@openssl.org>
Reviewed-by: NRich Salz <rsalz@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/1245)

Reviewed-by: NRichard Levitte <levitte@openssl.org>
Reviewed-by: NKurt Roeckx <kurt@openssl.org>

Based on an orignal commit by GitHub user BertramScharpf. Rebased and
updated to take account of all the updates since this was first raised.

GH PR#62
Reviewed-by: NRich Salz <rsalz@openssl.org>

Reviewed-by: NRichard Levitte <levitte@openssl.org>
Reviewed-by: NRich Salz <rsalz@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/952)

This reverts commit 620d540b.
It wasn't reviewed.
Reviewed-by: NRich Salz <rsalz@openssl.org>

Reviewed-by: NRichard Levitte <levitte@openssl.org>
Reviewed-by: NRich Salz <rsalz@openssl.org>

Misc fixes for no-sock
Reviewed-by: NRichard Levitte <levitte@openssl.org>

Reviewed-by: NRichard Levitte <levitte@openssl.org>

Reviewed-by: NViktor Dukhovni <viktor@openssl.org>

MR: #1595

Reviewed-by: NRichard Levitte <levitte@openssl.org>

To enable heartbeats for DTLS, configure with enable-heartbeats.
Heartbeats for TLS have been completely removed.

This addresses RT 3647
Reviewed-by: NRichard Levitte <levitte@openssl.org>

Adds a set of tests for the newly rewritten DTLSv1_listen function.
The test pokes various packets at the function and then checks
the return value and the data written out to ensure it is what we
would have expected.
Reviewed-by: NViktor Dukhovni <viktor@openssl.org>