• D
    Allow having different compression algorithms on different levels. · 321dfdc3
    Dhruba Borthakur 提交于
    Summary:
    The leveldb API is enhanced to support different compression algorithms at
    different levels.
    
    This adds the option min_level_to_compress to db_bench that specifies
    the minimum level for which compression should be done when
    compression is enabled. This can be used to disable compression for levels
    0 and 1 which are likely to suffer from stalls because of the CPU load
    for memtable flushes and (L0,L1) compaction.  Level 0 is special as it
    gets frequent memtable flushes. Level 1 is special as it frequently
    gets all:all file compactions between it and level 0. But all other levels
    could be the same. For any level N where N > 1, the rate of sequential
    IO for that level should be the same. The last level is the
    exception because it might not be full and because files from it are
    not read to compact with the next larger level.
    
    The same amount of time will be spent doing compaction at any
    level N excluding N=0, 1 or the last level. By this standard all
    of those levels should use the same compression. The difference is that
    the loss (using more disk space) from a faster compression algorithm
    is less significant for N=2 than for N=3. So we might be willing to
    trade disk space for faster write rates with no compression
    for L0 and L1, snappy for L2, zlib for L3. Using a faster compression
    algorithm for the mid levels also allows us to reclaim some cpu
    without trading off much loss in disk space overhead.
    
    Also note that little is to be gained by compressing levels 0 and 1. For
    a 4-level tree they account for 10% of the data. For a 5-level tree they
    account for 1% of the data.
    
    With compression enabled:
    * memtable flush rate is ~18MB/second
    * (L0,L1) compaction rate is ~30MB/second
    
    With compression enabled but min_level_to_compress=2
    * memtable flush rate is ~320MB/second
    * (L0,L1) compaction rate is ~560MB/second
    
    This practicaly takes the same code from https://reviews.facebook.net/D6225
    but makes the leveldb api more general purpose with a few additional
    lines of code.
    
    Test Plan: make check
    
    Differential Revision: https://reviews.facebook.net/D6261
    321dfdc3
db_bench.cc 43.1 KB