• J
    run 'make check's rules (and even subtests) in parallel · e3ee98b3
    Jim Meyering 提交于
    Summary:
    When GNU parallel is available, "make check" tests are now run in parallel.
    When /dev/shm is usable, we tell those tests to create temporary files therein.
    Now, the longest-running single test, db_test, (which is composed of hundreds of sub-tests)
    is no longer run sequentially: instead, each of its sub-tests is run independently, and can
    be parallelized along with all other tests. To make that process easier, this change
    creates a temporary directory, "t/", in which it puts a small script for each of those
    subtests. The output from each parallel-run test is now saved in t/log-TEST_NAME.
    
    When GNU parallel is not available, we run the tests in sequence, just as before.
    If GNU parallel is available and you don't like the default of running one subtest
    per core, you can invoke "make J=1 check" to run only one test at a time.
    Beware: this will take a long time, and it starts with the two longest-running tests, so you
    will wait for a long time before seeing any results. Instead, if you want to use fewer resources
    but still see useful progress, try "make J=60% check". That will attempt to ensure that 60% of
    the cores are occupied by test runs.
    
    To watch progress of individual tests (duration, success (PASS-or-FAIL), name), run "make watch-log"
    in the same directory from another window.  That will start with something like this:
    
    and when complete should show numbers/names like this:
    
      Every 0.1s: sort -k7,7nr -k4,4gr LOG|perl -n -e '@a=split("\t",$_,-1); $t=$a[8]; $t =~ s,^\./,,;' -e '$t =~ s, >.*,,; chomp $t;' -e '$t =~ /.*--gtest_filter=...  Wed Apr  1 10:51:42 2015
    
      152.221 PASS t/DBTest.FileCreationRandomFailure
      109.280 PASS t/DBTest.EncodeDecompressedBlockSizeTest
       82.315 PASS reduce_levels_test
       77.812 PASS t/DBTest.CompactionFilterWithValueChange
       73.236 PASS backupable_db_test
       63.428 PASS deletefile_test
       57.248 PASS table_test
       55.665 PASS prefix_test
       49.816 PASS t/DBTest.RateLimitingTest
      ...
    
    Test Plan:
    Timings (measured so as to exclude compile and link times):
    With this change, all tests complete in 2m40s on a system for which nproc prints 32.
    Prior to this this change, "make check" would take 24.5 minutes on that same system.
    
    Here are durations (in seconds) of the longest-running subtests:
    
    152.435 PASS t/DBTest.FileCreationRandomFailure
    107.070 PASS t/DBTest.EncodeDecompressedBlockSizeTest
     81.391 PASS ./reduce_levels_test
     71.587 PASS ./backupable_db_test
     61.746 PASS ./deletefile_test
     57.960 PASS ./table_test
     55.230 PASS ./prefix_test
     54.060 PASS t/DBTest.CompactionFilterWithValueChange
     48.873 PASS t/DBTest.RateLimitingTest
     47.569 PASS ./fault_injection_test
     46.593 PASS t/DBTest.Randomized
     42.662 PASS t/DBTest.CompactionFilter
     31.793 PASS t/DBTest.SparseMerge
     30.612 PASS t/DBTest.CompactionFilterV2
     25.891 PASS t/DBTest.GroupCommitTest
     23.863 PASS t/DBTest.DynamicLevelMaxBytesBase
     22.976 PASS ./rate_limiter_test
     18.942 PASS t/DBTest.OptimizeFiltersForHits
     16.851 PASS ./env_test
     15.399 PASS t/DBTest.CompactionFilterV2WithValueChange
     14.827 PASS t/DBTest.CompactionFilterV2NULLPrefix
    
    Reviewers: igor, sdong, rven, yhchiang, igor.sugak
    
    Reviewed By: igor.sugak
    
    Subscribers: dhruba
    
    Differential Revision: https://reviews.facebook.net/D35379
    e3ee98b3
Makefile 31.7 KB