Make the include guards consistent by renaming them systematically according
to the naming conventions below

The public header files (in the 'include/openssl' directory) are not changed
in 1.1.1, because it is a stable release.

For the private header files files, the guard names try to match the path
specified in the include directives, with all letters converted to upper case
and '/' and '.' replaced by '_'. An extra 'OSSL_' is added as prefix.
Reviewed-by: NRichard Levitte <levitte@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/9681)

Apart from public and internal header files, there is a third type called
local header files, which are located next to source files in the source
directory. Currently, they have different suffixes like

  '*_lcl.h', '*_local.h', or '*_int.h'

This commit changes the different suffixes to '*_local.h' uniformly.
Reviewed-by: NRichard Levitte <levitte@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/9681)

Currently, there are two different directories which contain internal
header files of libcrypto which are meant to be shared internally:

While header files in 'include/internal' are intended to be shared
between libcrypto and libssl, the files in 'crypto/include/internal'
are intended to be shared inside libcrypto only.

To make things complicated, the include search path is set up in such
a way that the directive #include "internal/file.h" could refer to
a file in either of these two directoroes. This makes it necessary
in some cases to add a '_int.h' suffix to some files to resolve this
ambiguity:

  #include "internal/file.h"      # located in 'include/internal'
  #include "internal/file_int.h"  # located in 'crypto/include/internal'

This commit moves the private crypto headers from

  'crypto/include/internal'  to  'include/crypto'

As a result, the include directives become unambiguous

  #include "internal/file.h"       # located in 'include/internal'
  #include "crypto/file.h"         # located in 'include/crypto'

hence the superfluous '_int.h' suffixes can be stripped.

The files 'store_int.h' and 'store.h' need to be treated specially;
they are joined into a single file.
Reviewed-by: NRichard Levitte <levitte@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/9681)

Found by OSS-Fuzz
Reviewed-by: NRichard Levitte <levitte@openssl.org>
Reviewed-by: NPaul Dale <paul.dale@oracle.com>

GH: #9959
(cherry picked from commit a6105ef40d65b35818f2b8ae8ca9e57ca6956d1d)

Reviewed-by: NRichard Levitte <levitte@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/9889)

Reviewed-by: NDmitry Belyavskiy <beldmit@gmail.com>
Reviewed-by: NMatt Caswell <matt@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/9830)

(cherry picked from commit 6ef03ea98fac501e6d6e33bac6ad3c92ea074712)

Reviewed-by: NPaul Dale <paul.dale@oracle.com>
Reviewed-by: NMatt Caswell <matt@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/9833)

(cherry picked from commit f28bc7d386b25fb75625d0c62c6b2e6d21de0d09)

There can be data to write in output buffer and data to read that were
not yet read in the input stream.

Fixes #9866
Reviewed-by: NRichard Levitte <levitte@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/9877)

(cherry picked from commit 6beb8b39ba8e4cb005c1fcd2586ba19e17f04b95)

Reviewed-by: NMatt Caswell <matt@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/9860)

Fixes #9858
Reviewed-by: NMatt Caswell <matt@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/9860)

Reviewed-by: NRichard Levitte <levitte@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/9847)

An attack is simple, if the first CMS_recipientInfo is valid but the
second CMS_recipientInfo is chosen ciphertext. If the second
recipientInfo decodes to PKCS #1 v1.5 form plaintext, the correct
encryption key will be replaced by garbage, and the message cannot be
decoded, but if the RSA decryption fails, the correct encryption key is
used and the recipient will not notice the attack.

As a work around for this potential attack the length of the decrypted
key must be equal to the cipher default key length, in case the
certifiate is not given and all recipientInfo are tried out.

The old behaviour can be re-enabled in the CMS code by setting the
CMS_DEBUG_DECRYPT flag.
Reviewed-by: NMatt Caswell <matt@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/9777)

(cherry picked from commit 5840ed0cd1e6487d247efbc1a04136a41d7b3a37)

Due to the dynamic allocation that was added to rand_pool_add_begin
this function could now return a null pointer where it was previously
guaranteed to succeed. But the return value of this function does
not need to be checked by design.

Move rand_pool_grow from rand_pool_add_begin to rand_pool_bytes_needed.
Make an allocation error persistent to avoid falling back to less secure
or blocking entropy sources.

Fixes: a6a66e4511ee ("Make rand_pool buffers more dynamic in their sizing.")
Reviewed-by: NMatthias St. Pierre <Matthias.St.Pierre@ncp-e.com>
Reviewed-by: NPaul Dale <paul.dale@oracle.com>
(Merged from https://github.com/openssl/openssl/pull/9687)

(cherry picked from commit fa3eb248e29ca8031e6a14e8a2c6f3cd58b5450e)

There was a warning about unused variables in this config:
./config --strict-warnings --with-rand-seed=rdcpu
Reviewed-by: NMatthias St. Pierre <Matthias.St.Pierre@ncp-e.com>
Reviewed-by: NPaul Dale <paul.dale@oracle.com>
(Merged from https://github.com/openssl/openssl/pull/9687)

(cherry picked from commit e301c147a763f67dcc5ba63eb7e2ae40d83a68aa)

Since commit 7c226dfc a chained DRBG does not add additional
data anymore when reseeding from its parent. The reason is that
the size of the additional data exceeded the allowed size when
no derivation function was used.

This commit provides an alternative fix: instead of adding the
entire DRBG's complete state, we just add the DRBG's address
in memory, thereby providing some distinction between the different
DRBG instances.
Reviewed-by: NPaul Dale <paul.dale@oracle.com>
Reviewed-by: NMatt Caswell <matt@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/9802)

Provides a little extra fork-safety on UNIX systems, adding to the
fact that all DRBGs reseed automatically when the fork_id changes.
Reviewed-by: NPaul Dale <paul.dale@oracle.com>
Reviewed-by: NMatt Caswell <matt@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/9802)

When the new OpenSSL CSPRNG was introduced in version 1.1.1,
it was announced in the release notes that it would be fork-safe,
which the old CSPRNG hadn't been.

The fork-safety was implemented using a fork count, which was
incremented by a pthread_atfork handler. Initially, this handler
was enabled by default. Unfortunately, the default behaviour
had to be changed for other reasons in commit b5319bdb, so
the new OpenSSL CSPRNG failed to keep its promise.

This commit restores the fork-safety using a different approach.
It replaces the fork count by a fork id, which coincides with
the process id on UNIX-like operating systems and is zero on other
operating systems. It is used to detect when an automatic reseed
after a fork is necessary.

To prevent a future regression, it also adds a test to verify that
the child reseeds after fork.

CVE-2019-1549
Reviewed-by: NPaul Dale <paul.dale@oracle.com>
Reviewed-by: NMatt Caswell <matt@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/9802)

crypto/rand/rand_win.c(70) : error C2065: 'BCRYPT_USE_SYSTEM_PREFERRED_RNG' : undeclared identifier
Reviewed-by: NRichard Levitte <levitte@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/9827)

(cherry picked from commit d3a1128bc25ec8bf835c81821e1be68fba39ab4b)

Reviewed-by: NPaul Dale <paul.dale@oracle.com>
(Merged from https://github.com/openssl/openssl/pull/9796)

(cherry picked from commit fa01370f7dc8f0a379483bbe74de11225857e5fe)

so results were undefined.
Reviewed-by: NPaul Dale <paul.dale@oracle.com>
(Merged from https://github.com/openssl/openssl/pull/9796)

(cherry picked from commit 2b95e8efcf8b99892106070d9ac745a0a369f503)

For for G=2 and 5 DH_generate_parameters will continue to generate
the order 2q subgroup for compatibility with previous versions.

For G=3 DH_generate_parameters generates an order q subgroup, but it
will not pass the check in DH_check with previous OpenSSL versions.
Reviewed-by: NMatt Caswell <matt@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/9820)

Description
-----------

Upon `EC_GROUP_new_from_ecparameters()` check if the parameters match any
of the built-in curves. If that is the case, return a new
`EC_GROUP_new_by_curve_name()` object instead of the explicit parameters
`EC_GROUP`.

This affects all users of `EC_GROUP_new_from_ecparameters()`:
- direct calls to `EC_GROUP_new_from_ecparameters()`
- direct calls to `EC_GROUP_new_from_ecpkparameters()` with an explicit
  parameters argument
- ASN.1 parsing of explicit parameters keys (as it eventually
  ends up calling `EC_GROUP_new_from_ecpkparameters()`)

A parsed explicit parameter key will still be marked with the
`OPENSSL_EC_EXPLICIT_CURVE` ASN.1 flag on load, so, unless
programmatically forced otherwise, if the key is eventually serialized
the output will still be encoded with explicit parameters, even if
internally it is treated as a named curve `EC_GROUP`.

Before this change, creating any `EC_GROUP` object using
`EC_GROUP_new_from_ecparameters()`, yielded an object associated with
the default generic `EC_METHOD`, but this was never guaranteed in the
documentation.
After this commit, users of the library that intentionally want to
create an `EC_GROUP` object using a specific `EC_METHOD` can still
explicitly call `EC_GROUP_new(foo_method)` and then manually set the
curve parameters using `EC_GROUP_set_*()`.

Motivation
----------

This has obvious performance benefits for the built-in curves with
specialized `EC_METHOD`s and subtle but important security benefits:
- the specialized methods have better security hardening than the
  generic implementations
- optional fields in the parameter encoding, like the `cofactor`, cannot
  be leveraged by an attacker to force execution of the less secure
  code-paths for single point scalar multiplication
- in general, this leads to reducing the attack surface

Check the manuscript at https://arxiv.org/abs/1909.01785 for an in depth
analysis of the issues related to this commit.

It should be noted that `libssl` does not allow to negotiate explicit
parameters (as per RFC 8422), so it is not directly affected by the
consequences of using explicit parameters that this commit fixes.
On the other hand, we detected external applications and users in the
wild that use explicit parameters by default (and sometimes using 0 as
the cofactor value, which is technically not a valid value per the
specification, but is tolerated by parsers for wider compatibility given
that the field is optional).
These external users of `libcrypto` are exposed to these vulnerabilities
and their security will benefit from this commit.

Related commits
---------------

While this commit is beneficial for users using built-in curves and
explicit parameters encoding for serialized keys, commit
b783beeadf6b80bc431e6f3230b5d5585c87ef87 (and its equivalents for the
1.0.2, 1.1.0 and 1.1.1 stable branches) fixes the consequences of the
invalid cofactor values more in general also for other curves
(CVE-2019-1547).

The following list covers commits in `master` that are related to the
vulnerabilities presented in the manuscript motivating this commit:

- d2baf88c43 [crypto/rsa] Set the constant-time flag in multi-prime RSA too
- 311e903d84 [crypto/asn1] Fix multiple SCA vulnerabilities during RSA key validation.
- b783beeadf [crypto/ec] for ECC parameters with NULL or zero cofactor, compute it
- 724339ff44 Fix SCA vulnerability when using PVK and MSBLOB key formats

Note that the PRs that contributed the listed commits also include other
commits providing related testing and documentation, in addition to
links to PRs and commits backporting the fixes to the 1.0.2, 1.1.0 and
1.1.1 branches.

This commit includes a partial backport of
https://github.com/openssl/openssl/pull/8555
(commit 8402cd5f75f8c2f60d8bd39775b24b03dd8b3b38)
for which the main author is Shane Lontis.

Responsible Disclosure
----------------------

This and the other issues presented in https://arxiv.org/abs/1909.01785
were reported by Cesar Pereida García, Sohaib ul Hassan, Nicola Tuveri,
Iaroslav Gridin, Alejandro Cabrera Aldaya and Billy Bob Brumley from the
NISEC group at Tampere University, FINLAND.

The OpenSSL Security Team evaluated the security risk for this
vulnerability as low, and encouraged to propose fixes using public Pull
Requests.

_______________________________________________________________________________
Co-authored-by: NShane Lontis <shane.lontis@oracle.com>

(Backport from https://github.com/openssl/openssl/pull/9808)
Reviewed-by: NMatt Caswell <matt@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/9809)

This leaves VPAES and AESNI support.
The VPAES performance is comparable but BSAES is not
completely constant time. There are table lookups
using secret key data in AES_set_encrypt/decrypt_key
and in ctr mode short data uses the non-constant
time AES_encrypt function instead of bit-slicing.
Furthermore the AES_ASM is by far outperformed
by recent GCC versions.
Since BSAES calls back to AES_ASM for short
data blocks the performance on those is also
worse than the pure software implementaion.

Fixes: #9640
Reviewed-by: NRichard Levitte <levitte@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/9675)

The cofactor argument to EC_GROUP_set_generator is optional, and SCA
mitigations for ECC currently use it. So the library currently falls
back to very old SCA-vulnerable code if the cofactor is not present.

This PR allows EC_GROUP_set_generator to compute the cofactor for all
curves of cryptographic interest. Steering scalar multiplication to more
SCA-robust code.

This issue affects persisted private keys in explicit parameter form,
where the (optional) cofactor field is zero or absent.

It also affects curves not built-in to the library, but constructed
programatically with explicit parameters, then calling
EC_GROUP_set_generator with a nonsensical value (NULL, zero).

The very old scalar multiplication code is known to be vulnerable to
local uarch attacks, outside of the OpenSSL threat model. New results
suggest the code path is also vulnerable to traditional wall clock
timing attacks.

CVE-2019-1547
Reviewed-by: NMatt Caswell <matt@openssl.org>
Reviewed-by: NTomas Mraz <tmraz@fedoraproject.org>
Reviewed-by: NNicola Tuveri <nic.tuv@gmail.com>
(Merged from https://github.com/openssl/openssl/pull/9781)

Reviewed-by: NMatt Caswell <matt@openssl.org>
Reviewed-by: NBernd Edlinger <bernd.edlinger@hotmail.de>
(Merged from https://github.com/openssl/openssl/pull/9511)

(cherry picked from commit 4fe2ee3a449a8ca2886584e221f34ff0ef5de119)

Replace flip_endian() by using the little endian specific
BN_bn2lebinpad() and BN_lebin2bn().
Reviewed-by: NMatt Caswell <matt@openssl.org>
Reviewed-by: NBernd Edlinger <bernd.edlinger@hotmail.de>
(Merged from https://github.com/openssl/openssl/pull/9511)

(cherry picked from commit e0b660c27d8d97b4ad9e2098cc957de26872c0ef)

Reviewed-by: NMatt Caswell <matt@openssl.org>
Reviewed-by: NBernd Edlinger <bernd.edlinger@hotmail.de>
(Merged from https://github.com/openssl/openssl/pull/9511)

(cherry picked from commit 1b338abe3abb8c73f004c34d4b8a9272b89dfd5d)

This issue was partially addressed by commit
972c87df, which hardened its callee
BN_num_bits_word() to avoid leaking the most-significant word of its
argument via branching and memory access pattern.
The commit message also reported:
> There are a few places where BN_num_bits is called on an input where
> the bit length is also secret. This does *not* fully resolve those
> cases as we still only look at the top word.

BN_num_bits() is called directly or indirectly (e.g., through
BN_num_bytes() or BN_bn2binpad() ) in various parts of the `crypto/ec`
code, notably in all the currently supported implementations of scalar
multiplication (in the generic path through ec_scalar_mul_ladder() as
well as in dedicated methods like ecp_nistp{224,256,521}.c and
ecp_nistz256.c).

Under the right conditions, a motivated SCA attacker could retrieve the
secret bitlength of a secret nonce through this vulnerability,
potentially leading, ultimately, to recover a long-term secret key.

With this commit, exclusively for BIGNUMs that are flagged with
BN_FLG_CONSTTIME, instead of accessing only bn->top, all the limbs of
the BIGNUM are accessed up to bn->dmax and bitwise masking is used to
avoid branching.

Memory access pattern still leaks bn->dmax, the size of the lazily
allocated buffer for representing the BIGNUM, which is inevitable with
the current BIGNUM architecture: reading past bn->dmax would be an
out-of-bound read.
As such, it's the caller responsibility to ensure that bn->dmax does not
leak secret information, by explicitly expanding the internal BIGNUM
buffer to a public value sufficient to avoid any lazy reallocation
while manipulating it: this should be already done at the top level
alongside setting the BN_FLG_CONSTTIME.

Thanks to David Schrammel and Samuel Weiser for reporting this issue
through responsible disclosure.
Reviewed-by: NMatt Caswell <matt@openssl.org>
Reviewed-by: NBernd Edlinger <bernd.edlinger@hotmail.de>
(Merged from https://github.com/openssl/openssl/pull/9511)

(cherry picked from commit 8b44198b916015f77bef1befa26edb48ad8a0238)

BN_bn2bin() is not constant-time and leaks the number of bits in the
processed BIGNUM.

The specialized methods in ecp_nistp224.c, ecp_nistp256.c and
ecp_nistp521.c internally used BN_bn2bin() to convert scalars into the
internal fixed length representation.

This can leak during ECDSA/ECDH key generation or handling the nonce
while generating an ECDSA signature, when using these implementations.
The amount and risk of leaked information useful for a SCA attack
varies for each of the three curves, as it depends mainly on the
ratio between the bitlength of the curve subgroup order (governing the
size of the secret nonce/key) and the limb size for the internal BIGNUM
representation (which depends on the compilation target architecture).

To fix this, we replace BN_bn2bin() with BN_bn2binpad(), bounding the
output length to the width of the internal representation buffer: this
length is public.

Internally the final implementation of both BN_bn2binpad() and
BN_bn2bin() already has masking in place to avoid leaking bn->top
through memory access patterns.
Memory access pattern still leaks bn->dmax, the size of the lazily
allocated buffer for representing the BIGNUM, which is inevitable with
the current BIGNUM architecture: reading past bn->dmax would be an
out-of-bound read.
As such, it's the caller responsibility to ensure that bn->dmax does not
leak secret information, by explicitly expanding the internal BIGNUM
buffer to a public value sufficient to avoid any lazy reallocation
while manipulating it: this is already done at the top level alongside
setting the BN_FLG_CONSTTIME.

Finally, the internal implementation of BN_bn2binpad() indirectly calls
BN_num_bits() via BN_num_bytes(): the current implementation of
BN_num_bits() can leak information to a SCA attacker, and is addressed
in the next commit.

Thanks to David Schrammel and Samuel Weiser for reporting this issue
through responsible disclosure.
Reviewed-by: NMatt Caswell <matt@openssl.org>
Reviewed-by: NBernd Edlinger <bernd.edlinger@hotmail.de>
(Merged from https://github.com/openssl/openssl/pull/9511)

(cherry picked from commit 805315d3a20f7274195eed75b06c391dacf3b197)

Reviewed-by: NMatt Caswell <matt@openssl.org>
Reviewed-by: NNicola Tuveri <nic.tuv@gmail.com>
(Merged from https://github.com/openssl/openssl/pull/9782)

(cherry picked from commit 31ca19403d56ad71d823cf62990518dfc6905bb4)

Reviewed-by: NBernd Edlinger <bernd.edlinger@hotmail.de>
Reviewed-by: NMatt Caswell <matt@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/9779)

This commit addresses multiple side-channel vulnerabilities present
during RSA key validation.
Private key parameters are re-computed using variable-time functions.

This issue was discovered and reported by the NISEC group at TAU Finland.
Reviewed-by: NBernd Edlinger <bernd.edlinger@hotmail.de>
Reviewed-by: NMatt Caswell <matt@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/9779)

Reviewed-by: NRichard Levitte <levitte@openssl.org>
Reviewed-by: NTomas Mraz <tmraz@fedoraproject.org>
(Merged from https://github.com/openssl/openssl/pull/9639)

(cherry picked from commit c70e2ec33943d3bd46d3d9950f774307feda832b)

Fixes #9757
Reviewed-by: NRichard Levitte <levitte@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/9761)

(cherry picked from commit 41ffd2ab09d24692c71850ccd7d5ff154196fe01)

This will never be the case for 1.1.1 so removed.

Fixes: comment 1 of #9757
Reviewed-by: NMatt Caswell <matt@openssl.org>
Reviewed-by: NPaul Dale <paul.dale@oracle.com>
(Merged from https://github.com/openssl/openssl/pull/9762)

Reviewed-by: NMatthias St. Pierre <Matthias.St.Pierre@ncp-e.com>
(Merged from https://github.com/openssl/openssl/pull/9734)

(cherry picked from commit 46a9cc9451213039fd53f62733b2ccd04e853bb2)

This commit addresses a side-channel vulnerability present when
PVK and MSBLOB key formats are loaded into OpenSSL.
The public key was not computed using a constant-time exponentiation
function.

This issue was discovered and reported by the NISEC group at TAU Finland.
Reviewed-by: NNicola Tuveri <nic.tuv@gmail.com>
Reviewed-by: NBernd Edlinger <bernd.edlinger@hotmail.de>
Reviewed-by: NPaul Dale <paul.dale@oracle.com>
Reviewed-by: NMatt Caswell <matt@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/9587)

(cherry picked from commit 724339ff44235149c4e8ddae614e1dda6863e23e)

There is a problem in the rand_unix.c code when the random seed fd is greater
than or equal to FD_SETSIZE and the FDSET overruns its limit and walks the
stack.
Reviewed-by: NBernd Edlinger <bernd.edlinger@hotmail.de>
(Merged from https://github.com/openssl/openssl/pull/9686)

(cherry picked from commit e1f8584d47a499301fba781086af6885fcf21fec)

Improve handling of low entropy at start up from /dev/urandom by waiting for
a read(2) call on /dev/random to succeed.  Once one such call has succeeded,
a shared memory segment is created and persisted as an indicator to other
processes that /dev/urandom is properly seeded.

This does not fully prevent against attacks weakening the entropy source.
An attacker who has control of the machine early in its boot sequence
could create the shared memory segment preventing detection of low entropy
conditions.  However, this is no worse than the current situation.

An attacker would also be capable of removing the shared memory segment
and causing seeding to reoccur resulting in a denial of service attack.
This is partially mitigated by keeping the shared memory alive for the
duration of the process's existence.  Thus, an attacker would not only need
to have called call shmctl(2) with the IPC_RMID command but the system
must subsequently enter a state where no instances of libcrypto exist in
any process.  Even one long running process will prevent this attack.

The System V shared memory calls used here go back at least as far as
Linux kernel 2.0.  Linux kernels 4.8 and later, don't have a reliable way
to detect that /dev/urandom has been properly seeded, so a failure is raised
for this case (i.e. the getentropy(2) call has already failed).
Reviewed-by: NBernd Edlinger <bernd.edlinger@hotmail.de>
(Merged from https://github.com/openssl/openssl/pull/9595)

[manual merge]

Reviewed-by: NPaul Dale <paul.dale@oracle.com>
(Merged from https://github.com/openssl/openssl/pull/9639)

(cherry picked from commit 038b381ecf2a988eee4c7bb21074ed0603303bd1)