that needed test_main now works using the same infrastructure as tests that used
register_tests.

This meant:
* renaming register_tests to setup_tests and giving it a success/failure return.
* renaming the init_test function to setup_test_framework.
* renaming the finish_test function to pulldown_test_framework.
* adding a user provided global_init function that runs before the test frame
    work is initialised.  It returns a failure indication that stops the stest.
* adding helper functions that permit tests to access their command line args.
* spliting the BIO initialisation and finalisation out from the test setup and
    teardown.
* hiding some of the now test internal functions.
* fix the comments in testutil.h
Reviewed-by: NRichard Levitte <levitte@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/3953)

$(SHLIB_MAJOR).$(SHLIB_MINOR) is really a synonym for
$(SHLIB_VERSION_NUMBER), and is therefore an added complexity,
so better to use $(SHLIB_VERSION_NUMBER) directly.  SHLIB_MAJOR and
SHLIB_MINOR are now unused, but are kept around purely as information
in case someone relies on their existence.

At the same time, add support for custom shared library extensions
with the three new Makefile variables SHLIB_EXT, SHLIB_EXT_SIMPLE and
SHLIB_EXT_IMPORT.  By default, they hold the variants of shared
library extensions we support.  On mingw and cygwin, SHLIB_EXT_IMPORT
is defined; on all other Unix platforms, it's empty.

An example to get shared libraries with a slightly different SOVER name:

    $ make SHLIB_EXT='.$(SHLIB_VERSION_NUMBER).so'

Fixes #3902
Reviewed-by: NAndy Polyakov <appro@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/3964)

Reviewed-by: NRichard Levitte <levitte@openssl.org>
Reviewed-by: NRich Salz <rsalz@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/1334)

Print the parameters for scrypt PBKDF when used in PKCS#12 files.
Reviewed-by: NRichard Levitte <levitte@openssl.org>
Reviewed-by: NRich Salz <rsalz@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/1334)

The info printing for PBES2 already uses space after type and commas to
separate items in the line so use the same format for MAC info too.
Reviewed-by: NRichard Levitte <levitte@openssl.org>
Reviewed-by: NRich Salz <rsalz@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/1334)

Since scrypt PBKDF can be used both in PKCS#5 and PKCS#12 files,
do share the code between them.
Reviewed-by: NRichard Levitte <levitte@openssl.org>
Reviewed-by: NRich Salz <rsalz@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/1334)

report additional information about the MAC used over the
PKCS#12 structure: size of mac and salt as well as the
hash algorithm used for creating it
Reviewed-by: NRichard Levitte <levitte@openssl.org>
Reviewed-by: NRich Salz <rsalz@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/1334)

since when we get to this point, other information was already
printed, we should insert some whitespace between already
printed data and this "unsupported parameters" error message
Reviewed-by: NRichard Levitte <levitte@openssl.org>
Reviewed-by: NRich Salz <rsalz@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/1334)

the man pages have lines that contain nothing but whitespace,
clean it up by removing that whitespace
Reviewed-by: NRichard Levitte <levitte@openssl.org>
Reviewed-by: NRich Salz <rsalz@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/1334)

Reviewed-by: NRichard Levitte <levitte@openssl.org>
Reviewed-by: NAndy Polyakov <appro@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/3998)

In 'crypto/rand/ossl_rand.c', a call to
'ASYNC_unblock_pause()' is missing in an error case.

CLA: trivial
Reviewed-by: NRich Salz <rsalz@openssl.org>
Reviewed-by: NBen Kaduk <kaduk@mit.edu>
(Merged from https://github.com/openssl/openssl/pull/4020)

OpenSSL already has the feature of SSL_MODE_RELEASE_BUFFERS that can
be set to release the read or write buffers when data has finished
reading or writing. OpenSSL will automatically re-allocate the buffers
as needed. This can be quite aggressive in terms of memory allocation.

This provides a manual mechanism. SSL_free_buffers() will free
the data buffers if there's no pending data. SSL_alloc_buffers()
will realloc them; but this function is not strictly necessary, as it's
still done automatically in the state machine.
Reviewed-by: NRichard Levitte <levitte@openssl.org>
Reviewed-by: NRich Salz <rsalz@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/2240)

In function wait_for_async(), allocated async fds is freed if
`SSL_get_all_async_fds` fails, but later `fds` is used. Interestingly,
it is not freed when everything succeeds.

Rewrite the FD set loop to make it more readable and to not modify the allocated
pointer so it can be freed.
Reviewed-by: NAndy Polyakov <appro@openssl.org>
Reviewed-by: NPaul Dale <paul.dale@oracle.com>
(Merged from https://github.com/openssl/openssl/pull/3992)

Reviewed-by: NPaul Dale <paul.dale@oracle.com>
Reviewed-by: NRichard Levitte <levitte@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/3943)

Reviewed-by: NPaul Dale <paul.dale@oracle.com>
Reviewed-by: NRichard Levitte <levitte@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/3943)

Reviewed-by: NPaul Dale <paul.dale@oracle.com>
Reviewed-by: NRichard Levitte <levitte@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/3943)

Thanks to David Benjamin for spotting this!
Reviewed-by: NRich Salz <rsalz@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/4009)

Reviewed-by: NKurt Roeckx <kurt@roeckx.be>
(Merged from https://github.com/openssl/openssl/pull/3898)

Reviewed-by: NKurt Roeckx <kurt@roeckx.be>
(Merged from https://github.com/openssl/openssl/pull/3898)

Improvement is result of combination of data layout ideas from
Keccak Code Package and initial version of this module.

Hardware used for benchmarking courtesy of Atos, experiments run by
Romain Dolbeau <romain.dolbeau@atos.net>. Kudos!
Reviewed-by: NBernd Edlinger <bernd.edlinger@hotmail.de>
Reviewed-by: NRich Salz <rsalz@openssl.org>

Reviewed-by: NAndy Polyakov <appro@openssl.org>
Reviewed-by: NRich Salz <rsalz@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/4000)

CLA: trivial
Reviewed-by: NKurt Roeckx <kurt@roeckx.be>
Reviewed-by: NRich Salz <rsalz@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/3934)

A previous change inavertently removed a silencing '@'
Reviewed-by: NTim Hudson <tjh@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/4003)

There's a case when the environment variable OPENSSL_CONF is
useless...  when cross compiling for mingw and your wine environment
has an environment variable OPENSSL_CONF.  The latter will override
anything that's given when starting wine and there make the use of
that environment variable useless in our tests.

Therefore, we should not trust it, and use explicit '-config' options
instead.
Reviewed-by: NRich Salz <rsalz@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/3994)

Makefile.shared was designed to figure out static library names,
shared library names, library version compatibility, import library
names and the like on its own.  This was a design for pre-1.1.0
OpenSSL because the main Makefile didn't have all that knowledge.

With 1.1.0, the situation isn't the same, a lot more knowledge is
included in the main Makefile, and while Makefile.shared did things
right most of the time (there are some corner cases, such as the
choice of .sl or .so as DSO extension on some HPUX versions), there's
still an inherent fragility when one has to keep an eye on
Makefile.shared to make sure it produces what the main Makefile
produces.

This change simplifies Makefile.shared by removing all its
"intelligence" and have it depend entirely on the input from the main
Makefile instead.  That way, all the naming is driven from
configuration data.
Reviewed-by: NRich Salz <rsalz@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/3983)

Using Zeller's congruence to fill the day of week field,
Also populate the day of year field.

Add unit test to cover a number of cases.
Reviewed-by: NRich Salz <rsalz@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/3999)

Based on discussion in PR #3566. Reduce duplicated code in original
asn1_utctime_to_tm and asn1_generalizedtime_to_tm, and introduce a new
internal function asn1_time_to_tm. This function also checks if the days
in the input time string is valid or not for the corresponding month.

Test cases are also added.
Reviewed-by: NAndy Polyakov <appro@openssl.org>
Reviewed-by: NPaul Dale <paul.dale@oracle.com>
(Merged from https://github.com/openssl/openssl/pull/3905)

Reviewed-by: NPaul Dale <paul.dale@oracle.com>
Reviewed-by: NMatt Caswell <matt@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/3893)

`args_verify()` and `opt_reset()` are declared in `apps/apps.h`, but they are
not referenced anywhere. So can we remove them from `apps.h`?
Reviewed-by: NKurt Roeckx <kurt@roeckx.be>
Reviewed-by: NRich Salz <rsalz@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/3995)

"Note" part is based on PR #3566
Reviewed-by: NBen Kaduk <kaduk@mit.edu>
Reviewed-by: NRich Salz <rsalz@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/3895)

Changes the EC_KEY_METHOD_get_* family to not need a EC_KEY_METHOD* as
its first parameter, but a const EC_KEY_METHOD*, which is entirely
sufficient.
Reviewed-by: NKurt Roeckx <kurt@roeckx.be>
Reviewed-by: NPaul Dale <paul.dale@oracle.com>
GH: #3985

Add a new config param to specify how the CSPRNG should be seeded.
Illegal values or nonsensical combinations (e.g., anything other
than "os" on VMS or HP VOS etc) result in build failures.
Add RDSEED support.
Add RDTSC but leave it disabled for now pending more investigation.

Refactor and reorganization all seeding files (rand_unix/win/vms) so
that they are simpler.

Only require 128 bits of seeding material.

Many document improvements, including why to not use RAND_add() and the
limitations around using load_file/write_file.
Document RAND_poll().

Cleanup Windows RAND_poll and return correct status

More completely initialize the default DRBG.
Reviewed-by: NPaul Dale <paul.dale@oracle.com>
(Merged from https://github.com/openssl/openssl/pull/3965)

Hardware used for benchmarking courtesy of Atos, experiments run by
Romain Dolbeau <romain.dolbeau@atos.net>. Kudos!
Reviewed-by: NRich Salz <rsalz@openssl.org>

"Optimize" is in quotes because it's rather a "salvage operation"
for now. Idea is to identify processor capability flags that
drive Knights Landing to suboptimial code paths and mask them.
Two flags were identified, XSAVE and ADCX/ADOX. Former affects
choice of AES-NI code path specific for Silvermont (Knights Landing
is of Silvermont "ancestry"). And 64-bit ADCX/ADOX instructions are
effectively mishandled at decode time. In both cases we are looking
at ~2x improvement.

AVX-512 results cover even Skylake-X :-)

Hardware used for benchmarking courtesy of Atos, experiments run by
Romain Dolbeau <romain.dolbeau@atos.net>. Kudos!
Reviewed-by: NRich Salz <rsalz@openssl.org>

Fixes: issue #3747

make SSL_CIPHER_standard_name globally available and introduce a new
function OPENSSL_cipher_name.

A new option '-convert' is also added to 'openssl ciphers' app.

Documentation and test cases are added.
Reviewed-by: NRichard Levitte <levitte@openssl.org>
Reviewed-by: NPaul Dale <paul.dale@oracle.com>
(Merged from https://github.com/openssl/openssl/pull/3859)

Looking at
http://nvlpubs.nist.gov/nistpubs/SpecialPublications/NIST.SP.800-90Ar1.pdf
we see that in the CTR_DRBG_Update() algorithm (internal page number 51),
the provided input data is (after truncation to seedlen) xor-d with the
key and V vector (of length keylen and blocklen respectively).  The comment
in ctr_XOR notes that xor-ing with 0 is the identity function, so we can
just ignore the case when the provided input is shorter than seedlen.

The code in ctr_XOR() then proceeds to xor the key with the input, up
to the amount of input present, and computes the remaining input that
could be used to xor with the V vector, before accessing a full 16-byte
stretch of the input vector and ignoring the calculated length.  The correct
behavior is to respect the supplied input length and only xor the
indicated number of bytes.
Reviewed-by: NRich Salz <rsalz@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/3971)

Reviewed-by: NRich Salz <rsalz@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/3971)

Reviewed-by: NRich Salz <rsalz@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/3971)

Replacement fix for #3975
Reviewed-by: NRichard Levitte <levitte@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/3979)

Reviewed-by: NRich Salz <rsalz@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/3974)