Apart from public and internal header files, there is a third type called
local header files, which are located next to source files in the source
directory. Currently, they have different suffixes like

  '*_lcl.h', '*_local.h', or '*_int.h'

This commit changes the different suffixes to '*_local.h' uniformly.
Reviewed-by: NRichard Levitte <levitte@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/9681)

Currently, there are two different directories which contain internal
header files of libcrypto which are meant to be shared internally:

While header files in 'include/internal' are intended to be shared
between libcrypto and libssl, the files in 'crypto/include/internal'
are intended to be shared inside libcrypto only.

To make things complicated, the include search path is set up in such
a way that the directive #include "internal/file.h" could refer to
a file in either of these two directoroes. This makes it necessary
in some cases to add a '_int.h' suffix to some files to resolve this
ambiguity:

  #include "internal/file.h"      # located in 'include/internal'
  #include "internal/file_int.h"  # located in 'crypto/include/internal'

This commit moves the private crypto headers from

  'crypto/include/internal'  to  'include/crypto'

As a result, the include directives become unambiguous

  #include "internal/file.h"       # located in 'include/internal'
  #include "crypto/file.h"         # located in 'include/crypto'

hence the superfluous '_int.h' suffixes can be stripped.

The files 'store_int.h' and 'store.h' need to be treated specially;
they are joined into a single file.
Reviewed-by: NRichard Levitte <levitte@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/9681)

There was a warning about unused variables in this config:
./config --strict-warnings --with-rand-seed=rdcpu
Reviewed-by: NMatthias St. Pierre <Matthias.St.Pierre@ncp-e.com>
Reviewed-by: NPaul Dale <paul.dale@oracle.com>
(Merged from https://github.com/openssl/openssl/pull/9687)

(cherry picked from commit e301c147a763f67dcc5ba63eb7e2ae40d83a68aa)

Provides a little extra fork-safety on UNIX systems, adding to the
fact that all DRBGs reseed automatically when the fork_id changes.
Reviewed-by: NPaul Dale <paul.dale@oracle.com>
Reviewed-by: NMatt Caswell <matt@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/9802)

Fixes #9757
Reviewed-by: NRichard Levitte <levitte@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/9761)

(cherry picked from commit 41ffd2ab09d24692c71850ccd7d5ff154196fe01)

This will never be the case for 1.1.1 so removed.

Fixes: comment 1 of #9757
Reviewed-by: NMatt Caswell <matt@openssl.org>
Reviewed-by: NPaul Dale <paul.dale@oracle.com>
(Merged from https://github.com/openssl/openssl/pull/9762)

Reviewed-by: NMatthias St. Pierre <Matthias.St.Pierre@ncp-e.com>
(Merged from https://github.com/openssl/openssl/pull/9734)

(cherry picked from commit 46a9cc9451213039fd53f62733b2ccd04e853bb2)

There is a problem in the rand_unix.c code when the random seed fd is greater
than or equal to FD_SETSIZE and the FDSET overruns its limit and walks the
stack.
Reviewed-by: NBernd Edlinger <bernd.edlinger@hotmail.de>
(Merged from https://github.com/openssl/openssl/pull/9686)

(cherry picked from commit e1f8584d47a499301fba781086af6885fcf21fec)

Improve handling of low entropy at start up from /dev/urandom by waiting for
a read(2) call on /dev/random to succeed.  Once one such call has succeeded,
a shared memory segment is created and persisted as an indicator to other
processes that /dev/urandom is properly seeded.

This does not fully prevent against attacks weakening the entropy source.
An attacker who has control of the machine early in its boot sequence
could create the shared memory segment preventing detection of low entropy
conditions.  However, this is no worse than the current situation.

An attacker would also be capable of removing the shared memory segment
and causing seeding to reoccur resulting in a denial of service attack.
This is partially mitigated by keeping the shared memory alive for the
duration of the process's existence.  Thus, an attacker would not only need
to have called call shmctl(2) with the IPC_RMID command but the system
must subsequently enter a state where no instances of libcrypto exist in
any process.  Even one long running process will prevent this attack.

The System V shared memory calls used here go back at least as far as
Linux kernel 2.0.  Linux kernels 4.8 and later, don't have a reliable way
to detect that /dev/urandom has been properly seeded, so a failure is raised
for this case (i.e. the getentropy(2) call has already failed).
Reviewed-by: NBernd Edlinger <bernd.edlinger@hotmail.de>
(Merged from https://github.com/openssl/openssl/pull/9595)

[manual merge]

Reviewed-by: NPaul Dale <paul.dale@oracle.com>
(Merged from https://github.com/openssl/openssl/pull/9639)

(cherry picked from commit 038b381ecf2a988eee4c7bb21074ed0603303bd1)

Reviewed-by: NPaul Dale <paul.dale@oracle.com>
(Merged from https://github.com/openssl/openssl/pull/9619)

(cherry picked from commit 24d932ec842bc5fdbd5e9b519cecf15a56bc74e6)

CLA: trivial
Reviewed-by: NRichard Levitte <levitte@openssl.org>
Reviewed-by: NBen Kaduk <kaduk@mit.edu>
Reviewed-by: NTomas Mraz <tmraz@fedoraproject.org>
(Merged from https://github.com/openssl/openssl/pull/9149)

The DEVRANDOM_WAIT feature added a select() call to wait for the
`/dev/random` device to become readable before reading from the
`/dev/urandom` device. It was introduced in commit 38023b87f037
in order to mitigate the fact that the `/dev/urandom` device
does not block until the initial seeding of the kernel CSPRNG
has completed, contrary to the behaviour of the `getrandom()`
system call.

It turned out that this change had negative side effects on
performance which were not acceptable. After some discussion it
was decided to revert this feature and leave it up to the OS
resp. the platform maintainer to ensure a proper initialization
during early boot time.

Fixes #9078

This partially reverts commit 38023b87f037.
Reviewed-by: NTim Hudson <tjh@openssl.org>
Reviewed-by: NViktor Dukhovni <viktor@openssl.org>

(cherry picked from commit a08714e18131b1998faa0113e5bd4024044654ac)

(Merged from https://github.com/openssl/openssl/pull/9118)

Use select to wait for /dev/random in readable state,
but do not actually read anything from /dev/random,
use /dev/urandom first.

Use linux define __NR_getrandom instead of the
glibc define SYS_getrandom, in case the kernel headers
are more current than the glibc headers.

Fixes #8215
Reviewed-by: NKurt Roeckx <kurt@roeckx.be>
(Merged from https://github.com/openssl/openssl/pull/8251)

(cherry picked from commit 38023b87f037f4b832c236dfce2a76272be08763)

Reviewed-by: NRichard Levitte <levitte@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/8347)

Reviewed-by: NMatt Caswell <matt@openssl.org>
Reviewed-by: NMatthias St. Pierre <Matthias.St.Pierre@ncp-e.com>
(Merged from https://github.com/openssl/openssl/pull/7569)

(cherry picked from commit 5c8b7b4caa0faedb69277063a7c6b3a8e56c6308)

Commit c7504aeb (pr #6432) fixed a regression for applications in
chroot environments, which compensated the fact that the new OpenSSL CSPRNG
(based on the NIST DRBG) now reseeds periodically, which the previous
one didn't. Now the reseeding could fail in the chroot environment if the
DEVRANDOM devices were not present anymore and no other entropy source
(e.g. getrandom()) was available.

The solution was to keep the file handles for the DEVRANDOM devices open
by default. In fact, the fix did more than this, it opened the DEVRANDOM
devices early and unconditionally in rand_pool_init(), which had the
unwanted side effect that the devices were opened (and kept open) even
in cases when they were not used at all, for example when the getrandom()
system call was available. Due  to a bug (issue #7419) this even happened
when the feature was disabled by the application.

This commit removes the unconditional opening of all DEVRANDOM devices.
They will now only be opened (and kept open) on first use. In particular,
if getrandom() is available, the handles will not be opened unnecessarily.

This change does not introduce a regression for applications compiled for
libcrypto 1.1.0, because the SSLEAY RNG also seeds on first use. So in the
above constellation the CSPRNG will only be properly seeded if it is happens
before the forking and chrooting.

Fixes #7419
Reviewed-by: NPaul Dale <paul.dale@oracle.com>
(Merged from https://github.com/openssl/openssl/pull/7437)

(cherry picked from commit 8cfc19716c22dac737ec8cfc5f7d085e7c37f4d8)

In commit 8bf366519661 some renamings andd typo fixes were made
while adding back the DRBG-HMAC and DRBG-HASH implementation.
Since the commit could not be backported, a lot of unnecessary
differences between master and 1.1.1 were introduced.

These differences result in tiresome merge conflicts when
cherry-picking. To minimize these merge-conflicts, this patch
ports all 'non-feature' changes of commit 8bf366519661
(e.g., renamings of private variables, fixes of typographical
errors, comment changes) manually back to 1.1.1.

The commits a83dc59afa2e (#7399) and 8817215d5c52 (#7456)
failed to cherry-pick previously to 1.1.1, with this patch
they both cherry-pick without conflicts.
Reviewed-by: NRichard Levitte <levitte@openssl.org>
Reviewed-by: NBernd Edlinger <bernd.edlinger@hotmail.de>
(Merged from https://github.com/openssl/openssl/pull/7505)

In pull request #4328 the seeding of the DRBG via RAND_add()/RAND_seed()
was implemented by buffering the data in a random pool where it is
picked up later by the rand_drbg_get_entropy() callback. This buffer
was limited to the size of 4096 bytes.

When a larger input was added via RAND_add() or RAND_seed() to the DRBG,
the reseeding failed, but the error returned by the DRBG was ignored
by the two calling functions, which both don't return an error code.
As a consequence, the data provided by the application was effectively
ignored.

This commit fixes the problem by a more efficient implementation which
does not copy the data in memory and by raising the buffer the size limit
to INT32_MAX (2 gigabytes). This is less than the NIST limit of 2^35 bits
but it was chosen intentionally to avoid platform dependent problems
like integer sizes and/or signed/unsigned conversion.

Additionally, the DRBG is now less permissive on errors: In addition to
pushing a message to the openssl error stack, it enters the error state,
which forces a reinstantiation on next call.

Thanks go to Dr. Falko Strenzke for reporting this issue to the
openssl-security mailing list. After internal discussion the issue
has been categorized as not being security relevant, because the DRBG
reseeds automatically and is fully functional even without additional
randomness provided by the application.

Fixes #7381
Reviewed-by: NPaul Dale <paul.dale@oracle.com>
(Merged from https://github.com/openssl/openssl/pull/7382)

(cherry picked from commit 3064b55134434a0b2850f07eff57120f35bb269a)

Fixes a compiler warning about an unused syscall_random()
and cleans up the OPENSSL_RAND_SEED preprocessor logic.
Reviewed-by: NMatt Caswell <matt@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/779)

(cherry picked from commit d90e128be6e584d319931c78cb8f8f68d17b6a34)

Don't discard partial reads from /dev/*random and retry instead.
Reviewed-by: NAndy Polyakov <appro@openssl.org>
Reviewed-by: NPaul Dale <paul.dale@oracle.com>
Reviewed-by: NTim Hudson <tjh@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/6990)

Fixes #6978

Don't discard partial reads from syscall_random() and retry instead.
Reviewed-by: NAndy Polyakov <appro@openssl.org>
Reviewed-by: NPaul Dale <paul.dale@oracle.com>
Reviewed-by: NTim Hudson <tjh@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/6990)

Change return value type to ssize_t and ensure that a negative value
is returned only if a corresponding errno is set.
Reviewed-by: NAndy Polyakov <appro@openssl.org>
Reviewed-by: NPaul Dale <paul.dale@oracle.com>
Reviewed-by: NTim Hudson <tjh@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/6990)

Occasionally, e.g. when compiling for elderly glibc, you end up passing
-D_GNU_SOURCE on command line, and doing so triggered warning...
Reviewed-by: NRich Salz <rsalz@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/6616)

This allows operation inside a chroot environment without having the
random device present.

A new call, RAND_keep_random_devices_open(), has been introduced that can
be used to control file descriptor use by the random seed sources. Some
seed sources maintain open file descriptors by default, which allows
such sources to operate in a chroot(2) jail without the associated device
nodes being available.
Reviewed-by: NMatt Caswell <matt@openssl.org>
Reviewed-by: NMatthias St. Pierre <Matthias.St.Pierre@ncp-e.com>
(Merged from https://github.com/openssl/openssl/pull/6432)

Unlike other ELF systems, HP-UX run-time linker fails to detect symbol
availability through weak declaration.
Reviewed-by: NRich Salz <rsalz@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/6583)

Calling the functions rand_pool_add_{additional,nonce}_data()
in crypto/rand/rand_lib.c with no implementation for djgpp/MSDOS
causees unresolved symbols when linking with djgpp.

Reported and fixed by Gisle Vanem
Reviewed-by: NRich Salz <rsalz@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/6421)

If built with no-dso, syscall_random remains "blind" to getentropy.
Since it's possible to detect symbol availability on ELF-based systems
without involving DSO module, bypass it.
Reviewed-by: NRich Salz <rsalz@openssl.org>
Reviewed-by: NKurt Roeckx <kurt@roeckx.be>
(Merged from https://github.com/openssl/openssl/pull/6436)

If built with no-dso, DSO_global_lookup leaves "unsupported" message
in error queue. Since there is a fall-back code, it's unnecessary
distraction.
Reviewed-by: NRich Salz <rsalz@openssl.org>
Reviewed-by: NKurt Roeckx <kurt@roeckx.be>
(Merged from https://github.com/openssl/openssl/pull/6436)

Reviewed-by: NAndy Polyakov <appro@openssl.org>
GH: #6405

Only Linux and FreeBSD provide getrandom(), but they both also provide
getentropy() since the same version and we already tried to call that.
Reviewed-by: NAndy Polyakov <appro@openssl.org>
GH: #6405

This will actually support most OSs, and at least adds support for
Solaris and OSX

Fixes: #6403
Reviewed-by: NAndy Polyakov <appro@openssl.org>
GH: #6405

Reviewed-by: NAndy Polyakov <appro@openssl.org>
GH: #6405

rand_pool_bytes_needed() was constructed in such a way that the
smallest acceptable entropy factor was 1 entropy bits per 8 bits of
data.  At the same time, we have a DRBG_MINMAX_FACTOR that allows
weaker source, as small as 1 bit of entropy per 128 bits of data.
The conclusion is that rand_pool_bytes_needed() needs to change to
support weaker entropy sources.  We therefore change the input of
entropy per byte to be an entropy factor instead.  This entropy factor
expresses how many bits of data it takes (on average) to get 1 bit of
entropy.
Reviewed-by: NMatthias St. Pierre <Matthias.St.Pierre@ncp-e.com>
(Merged from https://github.com/openssl/openssl/pull/6150)

Reviewed-by: NAndy Polyakov <appro@openssl.org>
GH: #6044

Reviewed-by: NDr. Matthias St. Pierre <Matthias.St.Pierre@ncp-e.com>
GH: #5910

Fixes #5849

In pull request #5503 a fallback was added which adds a random nonce of
security_strength/2 bits if no nonce callback is provided. This change raised
the entropy requirements form 256 to 384 bit, which can cause problems on some
platforms (e.g. VMS, see issue #5849).

The requirements for the nonce are given in section 8.6.7 of NIST SP 800-90Ar1:

  A nonce may be required in the construction of a seed during instantiation
  in order to provide a security cushion to block certain attacks.
  The nonce shall be either:

  a) A value with at least (security_strength/2) bits of entropy, or

  b) A value that is expected to repeat no more often than a
     (security_strength/2)-bit random string would be expected to repeat.

  Each nonce shall be unique to the cryptographic module in which instantiation
  is performed, but need not be secret. When used, the nonce shall be considered
  to be a critical security parameter.

This commit implements a nonce of type b) in order to lower the entropy
requirements during instantiation back to 256 bits.

The formulation "shall be unique to the cryptographic module" above implies
that the nonce needs to be unique among (with high probability) among all
DRBG instances in "space" and "time". We try to achieve this goal by creating a
nonce of the following form

    nonce = app-specific-data || high-resolution-utc-timestamp || counter

Where || denotes concatenation. The application specific data can be something
like the process or group id of the application. A utc timestamp is used because
it increases monotonically, provided the system time is synchronized. This approach
may not be perfect yet for a FIPS evaluation, but it should be good enough for the
moment.

This commit also harmonizes the implementation of the get_nonce() and the
get_additional_data() callbacks and moves the platform specific parts from
rand_lib.c into rand_unix.c, rand_win.c, and rand_vms.c.
Reviewed-by: NRichard Levitte <levitte@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/5920)

When these two functions returned zero, it could mean:

1. that an error occured.  In their case, the error is an overflow of
   the pool, i.e. the correct response from the caller would be to
   stop trying to fill the pool.
2. that there isn't enought entropy acquired yet, i.e. the correct
   response from the caller would be to try and add more entropy to
   the pool.

Because of this ambiguity, the returned zero turns out to be useless.
This change makes the returned value more consistent.  1 means the
addition of new entropy was successful, 0 means it wasn't.  To know if
the pool has been filled enough, the caller will have to call some
other function, such as rand_pool_entropy_available().

Fixes #5846
Reviewed-by: NMatthias St. Pierre <Matthias.St.Pierre@ncp-e.com>
(Merged from https://github.com/openssl/openssl/pull/5876)

Fixes #4403

This commit moves the internal header file "internal/rand.h" to
<openssl/rand_drbg.h>, making the RAND_DRBG API public.
The RAND_POOL API remains private, its function prototypes were
moved to "internal/rand_int.h" and converted to lowercase.

Documentation for the new API is work in progress on GitHub #5461.
Reviewed-by: NRichard Levitte <levitte@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/5462)

Reviewed-by: NRichard Levitte <levitte@openssl.org>