Because some operating systems have executable extensions, typically
".exe", we need to append it when looking for files in test() and
app() (or rather, their subroutines).
Reviewed-by: NRich Salz <rsalz@openssl.org>

It displays the command's exit code as well.
Reviewed-by: NEmilia Käsper <emilia@openssl.org>

If the command file that app(), test(), perlapp(9 and perltest() are
looking for doesn't exist in the build tree, look for it in the source
tree as well.
Reviewed-by: NEmilia Käsper <emilia@openssl.org>

The intention with that option is to allow extra flags to the perl
interpreter itself.
Reviewed-by: NEmilia Käsper <emilia@openssl.org>

$? in perl gets the status value from wait(2), which is a word with
the exit code in the upper half and the number of a raised signal in
the lower half.  OpenSSL::Test::run() ignored the signal half up until
now.

With this change, we recalculate an exit code the same way the Unix
shells do, using this formula:

    ($? & 0x7f) ? ($? & 0x7f)|0x80 : ($? >> 8);
Reviewed-by: NRich Salz <rsalz@openssl.org>
Reviewed-by: NEmilia Käsper <emilia@openssl.org>

It was unexpected that OpenSSL::Test::setup() should be called twice
by the same recipe.  However, that may happen if a recipe combines
OpenSSL::Test and OpenSSL::Test::Simple, which can be a sensible thing
to do.  Therefore, we now allow it.
Reviewed-by: NRich Salz <rsalz@openssl.org>

The logging that was performed in OpenSSL::Test was initially set up
as a means not to let messages that test programs write to STDERR get
displayed when a test isn't running in verbose mode.  However, the way
it was implemented, it meant that those messages were never displayed,
and you had to look in a test log.  This also meant that output to
STDERR and output to STDOUT got broken apart, which isn't optimal.

So, we remove the whole test log file implementation, and instead,
we're sending STDERR to the null device unless one of these conditions
apply:

- the test recipe already redirects stderr.  Just let it.
- the environment variable HARNESS_ACTIVE is undefined, meaning the
  recipe is run directly as a perl script instead of being harnessed
  by Test::Harness
- the environment variable HARNESS_VERBOSE is set.

Getting a full log of the tests now becomes as simple as this:

    HARNESS_VERBOSE=yes make test 2>&1 | tee tests.log
Reviewed-by: NRich Salz <rsalz@openssl.org>

To be able to run tests when we've built in a directory other than
the source tree, the testing framework needs a few adjustments.

test/testlib/OpenSSL/Test.pm needs to know where it can find
shlib_wrap.sh, and a number of other tests need to be told a different
place to find engines than what they may be able to figure out on
their own.  Relying to $TOP is not enough, $SRCTOP and $BLDTOP can be
used as an alternative.

As part of this change, top_file and top_dir are removed and
srctop_file, bldtop_file, srctop_dir and bldtop_dir take their place.
Reviewed-by: NBen Laurie <ben@openssl.org>

Just like File::Path::make_path, File::Path::remove_tree didn't show
up before File::Path 2.06 / perl v5.10.1, so we prefer the legacy
function here as well.
Reviewed-by: NAndy Polyakov <appro@openssl.org>

Reviewed-by: NRich Salz <rsalz@openssl.org>

$EXE_SHELL should only be used with out own programs, not with
surrounding programs such as the perl interpreter.
Reviewed-by: NMatt Caswell <matt@openssl.org>

Since we're building some of our perl scripts and the result might not
end up in apps/ (*), we may need to treat them like the compile
programs we use for testing.

This introduces perlapp() and perltest(), which behave like app() and
test(), but will add the perl executable in the command line.

-----

(*) For example, with a mk1mf build, the result will end up in $(BIN_D)
Reviewed-by: NMatt Caswell <matt@openssl.org>

On Unixly platforms, this doesn't matter.  On VMS, it does.
Reviewed-by: NRich Salz <rsalz@openssl.org>

On VMS, the command MCR will assume SYS$SYSTEM: when the first
argument lacks a directory spec.  So for programs in the current
directory, we add [] to tell MCR it is in the current directory.
It's the same as having ./ at the start of a program on Unix so the
shell doesn't start looking along $PATH.
Reviewed-by: NRich Salz <rsalz@openssl.org>

Reviewed-by: NRichard Levitte <levitte@openssl.org>

(part by Alessandro Ghedini)
Reviewed-by: NRichard Levitte <levitte@openssl.org>

If a test recipe does something like this:

    indir "foo.$$" => sub {
        chmod 0500, File::Spec->curdir();
        ok(run(app(["something"])));
    }

we get a problem, because we were storing the temporary stderr file in
the current directory at all times (so while inside the 'indir', we
would attemp to store it in "foo.$$").

So, change our ways to always store that temporary file in the exact
same location, defined by the environment variable RESULT_D, or
failing that TEST_D, or failing that $TOP/test.
Reviewed-by: NRich Salz <rsalz@openssl.org>

Be careful when shifting in a function argument, you end up changing
the caller's value.  Instead, when it is an array, make a shallow copy
and shift in that instead.
Reviewed-by: NRich Salz <rsalz@openssl.org>

For OpenSSL::Test, it meant rearranging the code to better suite the
structure of the documentation.
Reviewed-by: NRich Salz <rsalz@openssl.org>

It became tedious as well as error prone to have all recipes use
Test::More as well as OpenSSL::Test.  The easier way is to make
OpenSSL::Test an extension of Test::More, thereby having all version
checks as well as future checks firmly there.  Additionally, that
allows us to extend existing Test::More functions if the need would
arise.
Reviewed-by: NRich Salz <rsalz@openssl.org>

When the environment variable STOPTEST is defined (with any value other
than the empty string), the test machinery in OpenSSL::Test goes into a
different mode that will stop all testing at the end of a failing recipe.
Reviewed-by: NRich Salz <rsalz@openssl.org>

Reviewed-by: NRich Salz <rsalz@openssl.org>

The idea with this perl based testing framework is to make use of
what's delivered with perl and exists on all sorts of platforms.

The choice came to using Test::More and Test::Harness, as that seems
to be the most widely spread foundation, even if perl is aged.

The main runner of the show is run_tests.pl.  As it currently stands,
it's designed to run from inside Makefile, but it's absolutely
possible to run it from the command line as well, like so:

	cd test
	OPENSSL_SRCDIR=.. perl run_tests.pl

The tester scripts themselves are stored in the subdirectory recipes/,
and initially, we have two such scripts, recipes/00-check_testalltests.t
and recipes/00-check_testexes.t.  recipes/00-check_testalltests.t will
pick out the dependencies of "alltests" in test/Makefile, and check if
it can find recipes with corresponding names.  recipes/00-check_testexes.t
does something similar, but bases it on existing compiled test binaries.
They make it easy to figure out what's to be added, and will be
removed when this effort is finished.

Individual recipes can be run as well, of course, as they are perl
scripts in themselves.  For example, you can run only
recipes/00-check_testexes.t like so:

	cd test
	OPENSSL_SRCDIR=.. perl recipes/00-check_testexes.t

To make coding easier, there's a routine library OpenSSL::Test, which
is reachable in a perl script like so:

	use lib 'testlib';
	use OpenSSL::Test;
Reviewed-by: NRich Salz <rsalz@openssl.org>