From 2f4759d5ce5817ae179d7ce7b1ba0ef1678b3db5 Mon Sep 17 00:00:00 2001 From: Tom Lane Date: Mon, 6 Aug 2001 22:55:00 +0000 Subject: [PATCH] Doesn't seem to be much point in keeping this README up to date anymore, since it's completely redundant with regress.sgml. I think we agreed to remove it awhile back, actually, but no one got around to doing it. --- src/test/regress/README | 270 ---------------------------------------- 1 file changed, 270 deletions(-) delete mode 100644 src/test/regress/README diff --git a/src/test/regress/README b/src/test/regress/README deleted file mode 100644 index 57186442b0..0000000000 --- a/src/test/regress/README +++ /dev/null @@ -1,270 +0,0 @@ -REGRESSION TESTS - -The regression tests are a comprehensive set of tests for the SQL -implementation in PostgreSQL. They test standard SQL operations as -well as the extended capabilities of PostgreSQL. The test suite was -originally developed by Jolly Chen and Andrew Yu, and was extensively -revised and repackaged by Marc Fournier and Thomas Lockhart. From -PostgreSQL 6.1 onward the regression tests are current for every -official release. - -The regression test can be run against an already installed and -running server, or using a temporary installation within the build -tree. Furthermore, there is a "parallel" and a "sequential" mode for -running the tests. The sequential method runs each test script in -turn, whereas the parallel method starts up multiple server processes -to run groups of tests in parallel. Parallel testing gives confidence -that interprocess communication and locking are working correctly. For -historical reasons, the sequential test is usually run against an -existing installation and the parallel method "stand-alone", but there -are technical reasons for this. - -To run the regression tests after building but before installation, -type - -$ gmake check - -in the top-level directory. (Or you can change to src/test/regress and -run the command there.) This will first build several auxiliary files, -such as platform-dependent "expected" files and some sample -user-defined trigger functions, and then run the test driver -script. At the end you should see something like - -====================== - All 76 tests passed. -====================== - -or otherwise a note about what tests failed. See the section called -Test Evaluation below for more. - - Note: Because this test method runs a temporary server, it will - not work when you are the root user (the server will not start as - root). If you already did the build as root, you do not have to - start all over. Instead, make the regression test directory - writable by some other user, log in as that user, and restart the - tests. For example, - - root# chmod -R a+w src/test/regress - root# su - joeuser - joeuser$ gmake check - - (The only possible "security risk" here is that other users might - be able to alter the regression test results behind your back. Use - common sense when managing user permissions.) - - Alternatively, run the tests after installation. - - Tip: On some systems, the default Bourne-compatible shell - (/bin/sh) gets confused when it has to manage too many child - processes in parallel. This may cause the parallel test run to - lock up or fail. In such cases, specify a different - Bourne-compatible shell on the command line, for example: - - $ gmake SHELL=/bin/ksh check - -To run the tests after installation, initialize a data area and start -the server, then type - -$ gmake installcheck - -The tests will expect to contact the server at the local host and the -default port number, unless directed otherwise by PGHOST and PGPORT -environment variables. - -Test Evaluation - -Some properly installed and fully functional PostgreSQL installations -can "fail" some of these regression tests due to platform-specific -artifacts such as varying floating point representation and time zone -support. The tests are currently evaluated using a simple diff -comparison against the outputs generated on a reference system, so the -results are sensitive to small system differences. When a test is -reported as "failed", always examine the differences between expected -and actual results; you may well find that the differences are not -significant. Nonetheless, we still strive to maintain accurate reference -files across all supported platforms, so it can be expected that all -tests pass. - -The actual outputs of the regression tests are in files in the -src/test/regress/results directory. The test script uses diff to -compare each output file against the reference outputs stored in the -src/test/regress/expected directory. Any differences are saved for -your inspection in src/test/regress/regression.diffs. (Or you can run -diff yourself, if you prefer.) - -Error message differences - -Some of the regression tests involve intentional invalid input -values. Error messages can come from either the PostgreSQL code or -from the host platform system routines. In the latter case, the -messages may vary between platforms, but should reflect similar -information. These differences in messages will result in a "failed" -regression test which can be validated by inspection. - -Locale differences - -The tests expect to run in plain "C" locale. This should not cause any -problems when you run the tests against a temporary installation, since -the regression test driver takes care to start the server in C locale. -However, if you run the tests against an already-installed server that -is using non-C locale settings, you may see differences caused by -varying rules for string sort order, formatting of numeric and monetary -values, and so forth. - -In some locales the resulting differences are small and easily checked by -inspection. However, in a locale that changes the rules for formatting -of numeric values (typically by swapping the usage of commas and -decimal points), entry of some data values will fail, resulting in -extensive differences later in the tests where the missing data values -are supposed to be used. - -Date and time differences - -Most of the date and time results are dependent on the time zone -environment. The reference files are generated for time zone PST8PDT -(Berkeley, California) and there will be apparent failures if the -tests are not run with that time zone setting. The regression test -driver sets environment variable PGTZ to PST8PDT to ensure proper -results. However, your system must provide library support for the -PST8PDT time zone, or the time zone-dependent tests will fail. To -verify that your machine does have this support, type the following: - -$ env TZ=PST8PDT date - -The command above should have returned the current system time in the -PST8PDT time zone. If the PST8PDT database is not available, then your -system may have returned the time in GMT. If the PST8PDT time zone is -not available, you can set the time zone rules explicitly: - -PGTZ='PST8PDT7,M04.01.0,M10.05.03'; export PGTZ - -There appear to be some systems which do not accept the recommended -syntax for explicitly setting the local time zone rules; you may need -to use a different PGTZ setting on such machines. - -Some systems using older time zone libraries fail to apply -daylight-savings corrections to dates before 1970, causing pre-1970 -PDT times to be displayed in PST instead. This will result in -localized differences in the test results. - -Some of the queries in the "timestamp" test will fail if you run the -test on the day of a daylight-savings time changeover, or the day -before or after one. These queries assume that the intervals between -midnight yesterday, midnight today and midnight tomorrow are exactly -twenty-four hours -- which is wrong if daylight-savings time went into -or out of effect meanwhile. - -Floating point differences - -Some of the tests involve computing 64-bit (double precision) numbers -from table columns. Differences in results involving mathematical -functions of double precision columns have been observed. The float8 -and geometry tests are particularly prone to small differences across -platforms, or even with different compiler optimization options. Human -eyeball comparison is needed to determine the real significance of -these differences which are usually 10 places to the right of the -decimal point. - -Some systems signal errors from pow() and exp() differently from the -mechanism expected by the current PostgreSQL code. - -Polygon differences - -Several of the tests involve operations on geographic data about the -Oakland/Berkeley, CA street map. The map data is expressed as polygons -whose vertices are represented as pairs of double precision numbers -(decimal latitude and longitude). Initially, some tables are created -and loaded with geographic data, then some views are created which -join two tables using the polygon intersection operator (##), then a -select is done on the view. - -When comparing the results from different platforms, differences occur -in the 2nd or 3rd place to the right of the decimal point. The SQL -statements where these problems occur are the following: - -SELECT * from street; -SELECT * from iexit; - -Tuple ordering differences - -You might see differences in which the same tuples are output in a -different order than what appears in the expected file. In most cases -this is not, strictly speaking, a bug. Most of the regression test -scripts are not so pedantic as to use an ORDER BY for every single -SELECT, and so their result tuple orderings are not well-defined -according to the letter of the SQL spec. In practice, since we are -looking at the same queries being executed on the same data by the same -software, we usually get the same result ordering on all platforms, and -so the lack of ORDER BY isn't a problem. Some queries do exhibit -cross-platform ordering differences, however. (Ordering differences -can also be triggered by non-C locale settings.) - -Therefore, if you see an ordering difference, it's not something to -worry about, unless the query does have an ORDER BY that your result -is violating. But please report it anyway, so that we can add an -ORDER BY to that particular query and thereby eliminate the bogus -"failure" in future releases. - -You might wonder why we don't ORDER all the regress test SELECTs to -get rid of this issue once and for all. The reason is that that would -make the regression tests less useful, not more, since they'd tend -to exercise query plan types that produce ordered results to the -exclusion of those that don't. - -The "random" test - -There is at least one case in the "random" test script that is -intended to produce random results. This causes random to fail the -regression test once in a while (perhaps once in every five to ten -trials). Typing - -diff results/random.out expected/random.out - -should produce only one or a few lines of differences. You need not -worry unless the random test always fails in repeated attempts. (On -the other hand, if the random test is never reported to fail even in -many trials of the regress tests, you probably should worry.) - -Platform-specific comparison files - -Since some of the tests inherently produce platform-specific results, -we have provided a way to supply platform-specific result comparison -files. Frequently, the same variation applies to multiple platforms; -rather than supplying a separate comparison file for every platform, -there is a mapping file that defines which comparison file to use. So, -to eliminate bogus test "failures" for a particular platform, you must -choose or make a variant result file, and then add a line to the -mapping file, which is "resultmap". - -Each line in the mapping file is of the form - - testname/platformpattern=comparisonfilename - -The test name is just the name of the particular regression test -module. The platform pattern is a pattern in the style of expr(1) -(that is, a regular expression with an implicit ^ anchor at the start). -It is matched against the platform name as printed by config.guess -followed by ":gcc" or ":cc", depending on whether you use the GNU compiler -or the system's native compiler (on systems where there is a difference). -The comparison file name is the name of the substitute result comparison -file. - -For example: the int2 regress test includes a deliberate entry of a -value that is too large to fit in int2. The specific error message -that is produced is platform-dependent; our reference platform emits - - ERROR: pg_atoi: error reading "100000": Numerical result out of range - -but a fair number of other Unix platforms emit - - ERROR: pg_atoi: error reading "100000": Result too large - -Therefore, we provide a variant comparison file, int2-too-large.out, -that includes this spelling of the error message. To silence the bogus -"failure" message on HPPA platforms, resultmap includes - - int2/hppa=int2-too-large - -which will trigger on any machine for which config.guess's output -begins with 'hppa'. Other lines in resultmap select the variant -comparison file for other platforms where it's appropriate. -- GitLab