• F
    Implement full filter for block based table. · 0af157f9
    Feng Zhu 提交于
    Summary:
    1. Make filter_block.h a base class. Derive block_based_filter_block and full_filter_block. The previous one is the traditional filter block. The full_filter_block is newly added. It would generate a filter block that contain all the keys in SST file.
    
    2. When querying a key, table would first check if full_filter is available. If not, it would go to the exact data block and check using block_based filter.
    
    3. User could choose to use full_filter or tradional(block_based_filter). They would be stored in SST file with different meta index name. "filter.filter_policy" or "full_filter.filter_policy". Then, Table reader is able to know the fllter block type.
    
    4. Some optimizations have been done for full_filter_block, thus it requires a different interface compared to the original one in filter_policy.h.
    
    5. Actual implementation of filter bits coding/decoding is placed in util/bloom_impl.cc
    
    Benchmark: base commit 1d23b5c4
    Command:
    db_bench --db=/dev/shm/rocksdb --num_levels=6 --key_size=20 --prefix_size=20 --keys_per_prefix=0 --value_size=100 --write_buffer_size=134217728 --max_write_buffer_number=2 --target_file_size_base=33554432 --max_bytes_for_level_base=1073741824 --verify_checksum=false --max_background_compactions=4 --use_plain_table=0 --memtablerep=prefix_hash --open_files=-1 --mmap_read=1 --mmap_write=0 --bloom_bits=10 --bloom_locality=1 --memtable_bloom_bits=500000 --compression_type=lz4 --num=393216000 --use_hash_search=1 --block_size=1024 --block_restart_interval=16 --use_existing_db=1 --threads=1 --benchmarks=readrandom —disable_auto_compactions=1
    Read QPS increase for about 30% from 2230002 to 2991411.
    
    Test Plan:
    make all check
    valgrind db_test
    db_stress --use_block_based_filter = 0
    ./auto_sanity_test.sh
    
    Reviewers: igor, yhchiang, ljin, sdong
    
    Reviewed By: sdong
    
    Subscribers: dhruba, leveldb
    
    Differential Revision: https://reviews.facebook.net/D20979
    0af157f9
Makefile 23.4 KB