Previously, a truncated DTLS fragment in
|dtls1_process_out_of_seq_message| would cause *ok to be cleared, but
the return value would still be the number of bytes read. This would
cause |dtls1_get_message| not to consider it an error and it would
continue processing as normal until the calling function noticed that
*ok was zero.

I can't see an exploit here because |dtls1_get_message| uses
|s->init_num| as the length, which will always be zero from what I can
see.
Reviewed-by: NMatt Caswell <matt@openssl.org>
Reviewed-by: NEmilia Käsper <emilia@openssl.org>

The |pqueue_insert| function can fail if one attempts to insert a
duplicate sequence number. When handling a fragment of an out of
sequence message, |dtls1_process_out_of_seq_message| would not call
|dtls1_reassemble_fragment| if the fragment's length was zero. It would
then allocate a fresh fragment and attempt to insert it, but ignore the
return value, leaking the fragment.

This allows an attacker to exhaust the memory of a DTLS peer.

Fixes CVE-2014-3507
Reviewed-by: NMatt Caswell <matt@openssl.org>
Reviewed-by: NEmilia Käsper <emilia@openssl.org>

In |dtls1_reassemble_fragment|, the value of
|msg_hdr->frag_off+frag_len| was being checked against the maximum
handshake message size, but then |msg_len| bytes were allocated for the
fragment buffer. This means that so long as the fragment was within the
allowed size, the pending handshake message could consume 16MB + 2MB
(for the reassembly bitmap). Approx 10 outstanding handshake messages
are allowed, meaning that an attacker could consume ~180MB per DTLS
connection.

In the non-fragmented path (in |dtls1_process_out_of_seq_message|), no
check was applied.

Fixes CVE-2014-3506

Wholly based on patch by Adam Langley with one minor amendment.
Reviewed-by: NEmilia Käsper <emilia@openssl.org>

Reviewed-by: NEmilia Käsper <emilia@openssl.org>

The |item| variable, in both of these cases, may contain a pointer to a
|pitem| structure within |s->d1->buffered_messages|. It was being freed
in the error case while still being in |buffered_messages|. When the
error later caused the |SSL*| to be destroyed, the item would be double
freed.

Thanks to Wah-Teh Chang for spotting that the fix in 1632ef74 was
inconsistent with the other error paths (but correct).

Fixes CVE-2014-3505
Reviewed-by: NMatt Caswell <matt@openssl.org>
Reviewed-by: NEmilia Käsper <emilia@openssl.org>

PR#3394

A buffer overrun attack can be triggered by sending invalid DTLS fragments
to an OpenSSL DTLS client or server. This is potentially exploitable to
run arbitrary code on a vulnerable client or server.

Fixed by adding consistency check for DTLS fragments.

Thanks to Jüri Aedla for reporting this issue.
(cherry picked from commit 1632ef744872edc2aa2a53d487d3e79c965a4ad3)

Unnecessary recursion when receiving a DTLS hello request can be used to
crash a DTLS client. Fixed by handling DTLS hello request without recursion.

Thanks to Imre Rad (Search-Lab Ltd.) for discovering this issue.
(cherry picked from commit d3152655d5319ce883c8e3ac4b99f8de4c59d846)

PR#3302

A missing bounds check in the handling of the TLS heartbeat extension
can be used to reveal up to 64k of memory to a connected client or
server.

Thanks for Neel Mehta of Google Security for discovering this bug and to
Adam Langley <agl@chromium.org> and Bodo Moeller <bmoeller@acm.org> for
preparing the fix (CVE-2014-0160)
(cherry picked from commit 96db9023b881d7cd9f379b0c154650d6c108e9a3)

For DTLS we might need to retransmit messages from the previous session
so keep a copy of write context in DTLS retransmission buffers instead
of replacing it after sending CCS. CVE-2013-6450.
(cherry picked from commit 34628967f1e65dc8f34e000f0f5518e21afbfc7b)

Port TLS 1.2 GCM code to DTLS. Enable use of TLS 1.2 only ciphers when in
DTLS 1.2 mode too.

Since this is always called from DTLS code it is safe to assume the header
length should be the DTLS value. This avoids the need to check the version
number and should work with any version of DTLS (not just 1.0).

Revise DTLS code. There was a *lot* of code duplication in the
DTLS code that generates records. This makes it harder to maintain and
sometimes a TLS update is omitted by accident from the DTLS code.

Specifically almost all of the record generation functions have code like
this:

some_pointer = buffer + HANDSHAKE_HEADER_LENGTH;
... Record creation stuff ...
set_handshake_header(ssl, SSL_MT_SOMETHING, message_len);

...

write_handshake_message(ssl);

Where the "Record creation stuff" is identical between SSL/TLS and DTLS or
in some cases has very minor differences.

By adding a few fields to SSL3_ENC to include the header length, some flags
and function pointers for handshake header setting and handshake writing the
code can cope with both cases.

Note: although this passes "make test" and some simple DTLS tests there may
be some minor differences in the DTLS code that have to be accounted for.

Submitted by: Robin Seggelmann <seggelmann@fh-muenster.de>

Reduce MTU after failed transmissions.

Submitted by: Robin Seggelmann <seggelmann@fh-muenster.de>

Fix padding bugs in Heartbeat support.

certificate chain instead of an X509 structure.

This makes it easier to enhance code in future and the chain
output functions have access to the CERT_PKEY structure being
used.

New function ssl_add_cert_chain which adds a certificate chain to
SSL internal BUF_MEM. Use this function in ssl3_output_cert_chain
and dtls1_output_cert_chain instead of partly duplicating code.

Submitted by: Robin Seggelmann <seggelmann@fh-muenster.de>
Reviewed by: steve

Support for TLS/DTLS heartbeats.

Submitted by: Robin Seggelmann <seggelmann@fh-muenster.de>
Reviewed by: steve

Add SCTP support for DTLS (RFC 6083).

Submitted by: Robin Seggelmann <seggelmann@fh-muenster.de>
Reviewed by: steve

Send alert instead of assertion failure for incorrectly formatted DTLS
fragments.

Submitted by: Robin Seggelmann <seggelmann@fh-muenster.de>
Reviewed by: steve

Fix DTLS bug which prevents manual MTU setting

Submitted by: Robin Seggelmann <seggelmann@fh-muenster.de>
Reviewed by: steve

Fix DTLS Retransmission Buffer Bug

Submitted by: Robin Seggelmann <seggelmann@fh-muenster.de>
Reviewed by: steve

Don't change state when answering DTLS ClientHello.

Submitted by: Robin Seggelmann <seggelmann@fh-muenster.de>
Reviewed by: steve

Fix DTLS fragment reassembly bug.

Submitted By: Robin Seggelmann <seggelmann@fh-muenster.de>

Fix bug in bitmask macros and stop warnings.

Submitted By: Robin Seggelmann <seggelmann@fh-muenster.de>

Fix various DTLS fragment reassembly bugs.

Clear error queue in a few places in SSL code where errors are expected
so they don't stay in the queue.

Submitted by: Robin Seggelmann <seggelmann@fh-muenster.de>
Approved by: steve@openssl.org

Add Renegotiation extension to DTLS, fix DTLS ClientHello processing bug.

Submitted by: Robin Seggelmann <seggelmann@fh-muenster.de>
Approved by: steve@openssl.org

DTLS Fragment size bug fix.

Submitted by: Julia Lawall <julia@diku.dk>
Approved by: steve@openssl.org

Correct BIO_ctrl error handling

Submitted by: Robin Seggelmann <seggelmann@fh-muenster.de>
Approved by: steve@openssl.org

DTLS timeout handling fix.