• S
    FIFO Compaction with TTL · 1cd45cd1
    Sagar Vemuri 提交于
    Summary:
    Introducing FIFO compactions with TTL.
    
    FIFO compaction is based on size only which makes it tricky to enable in production as use cases can have organic growth. A user requested an option to drop files based on the time of their creation instead of the total size.
    
    To address that request:
    - Added a new TTL option to FIFO compaction options.
    - Updated FIFO compaction score to take TTL into consideration.
    - Added a new table property, creation_time, to keep track of when the SST file is created.
    - Creation_time is set as below:
      - On Flush: Set to the time of flush.
      - On Compaction: Set to the max creation_time of all the files involved in the compaction.
      - On Repair and Recovery: Set to the time of repair/recovery.
      - Old files created prior to this code change will have a creation_time of 0.
    - FIFO compaction with TTL is enabled when ttl > 0. All files older than ttl will be deleted during compaction. i.e. `if (file.creation_time < (current_time - ttl)) then delete(file)`. This will enable cases where you might want to delete all files older than, say, 1 day.
    - FIFO compaction will fall back to the prior way of deleting files based on size if:
      - the creation_time of all files involved in compaction is 0.
      - the total size (of all SST files combined) does not drop below `compaction_options_fifo.max_table_files_size` even if the files older than ttl are deleted.
    
    This feature is not supported if max_open_files != -1 or with table formats other than Block-based.
    
    **Test Plan:**
    Added tests.
    
    **Benchmark results:**
    Base: FIFO with max size: 100MB ::
    ```
    svemuri@dev15905 ~/rocksdb (fifo-compaction) $ TEST_TMPDIR=/dev/shm ./db_bench --benchmarks=readwhilewriting --num=5000000 --threads=16 --compaction_style=2 --fifo_compaction_max_table_files_size_mb=100
    
    readwhilewriting :       1.924 micros/op 519858 ops/sec;   13.6 MB/s (1176277 of 5000000 found)
    ```
    
    With TTL (a low one for testing) ::
    ```
    svemuri@dev15905 ~/rocksdb (fifo-compaction) $ TEST_TMPDIR=/dev/shm ./db_bench --benchmarks=readwhilewriting --num=5000000 --threads=16 --compaction_style=2 --fifo_compaction_max_table_files_size_mb=100 --fifo_compaction_ttl=20
    
    readwhilewriting :       1.902 micros/op 525817 ops/sec;   13.7 MB/s (1185057 of 5000000 found)
    ```
    Example Log lines:
    ```
    2017/06/26-15:17:24.609249 7fd5a45ff700 (Original Log Time 2017/06/26-15:17:24.609177) [db/compaction_picker.cc:1471] [default] FIFO compaction: picking file 40 with creation time 1498515423 for deletion
    2017/06/26-15:17:24.609255 7fd5a45ff700 (Original Log Time 2017/06/26-15:17:24.609234) [db/db_impl_compaction_flush.cc:1541] [default] Deleted 1 files
    ...
    2017/06/26-15:17:25.553185 7fd5a61a5800 [DEBUG] [db/db_impl_files.cc:309] [JOB 0] Delete /dev/shm/dbbench/000040.sst type=2 #40 -- OK
    2017/06/26-15:17:25.553205 7fd5a61a5800 EVENT_LOG_v1 {"time_micros": 1498515445553199, "job": 0, "event": "table_file_deletion", "file_number": 40}
    ```
    
    SST Files remaining in the dbbench dir, after db_bench execution completed:
    ```
    svemuri@dev15905 ~/rocksdb (fifo-compaction)  $ ls -l /dev/shm//dbbench/*.sst
    -rw-r--r--. 1 svemuri users 30749887 Jun 26 15:17 /dev/shm//dbbench/000042.sst
    -rw-r--r--. 1 svemuri users 30768779 Jun 26 15:17 /dev/shm//dbbench/000044.sst
    -rw-r--r--. 1 svemuri users 30757481 Jun 26 15:17 /dev/shm//dbbench/000046.sst
    ```
    Closes https://github.com/facebook/rocksdb/pull/2480
    
    Differential Revision: D5305116
    
    Pulled By: sagar0
    
    fbshipit-source-id: 3e5cfcf5dd07ed2211b5b37492eb235b45139174
    1cd45cd1
可在Tags中查看这些版本中当前仓库的状态.
HISTORY.md 45.5 KB