Skip to content

R
rocksdb

kvdb / rocksdb
12 个月 前同步成功

通知 0
Star 0
Fork 0
R
rocksdb

体验新版 GitCode，发现更多精彩内容 >>

切换分支/标签
  1. 26 8月, 2014 3 次提交
  3. 23 8月, 2014 1 次提交
    • I
      Fix concurrency issue in CompactionPicker · 42ea7952
      由 Igor Canadi 提交于 
      Summary:
I am currently working on a project that uses RocksDB. While debugging some perf issues, I came up across interesting compaction concurrency issue. Namely, I had 15 idle threads and a good comapction to do, but CompactionPicker returned "Compaction nothing to do". Here's how Internal stats looked:

    2014/08/22-08:08:04.551982 7fc7fc3f5700 ------- DUMPING STATS -------
    2014/08/22-08:08:04.552000 7fc7fc3f5700
    ** Compaction Stats [default] **
    Level   Files   Size(MB) Score Read(GB)  Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) RW-Amp W-Amp Rd(MB/s) Wr(MB/s)  Rn(cnt) Rnp1(cnt) Wnp1(cnt) Wnew(cnt)  Comp(sec) Comp(cnt) Avg(sec) Stall(sec) Stall(cnt) Avg(ms)
    ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
      L0     7/5        353   1.0      0.0     0.0      0.0       2.3      2.3    0.0   0.0      0.0      9.4        0         0         0         0        247        46    5.359       8.53          1 8526.25
      L1     2/2         86   1.3      2.6     1.9      0.7       2.6      1.9    2.7   1.3     24.3     24.0       39        19        71        52        109        11    9.938       0.00          0    0.00
      L2    26/0        833   1.3      5.7     1.7      4.0       5.2      1.2    6.3   3.0     15.6     14.2       47       112       147        35        373        44    8.468       0.00          0    0.00
      L3    12/0        505   0.1      0.0     0.0      0.0       0.0      0.0    0.0   0.0      0.0      0.0        0         0         0         0          0         0    0.000       0.00          0    0.00
     Sum    47/7       1778   0.0      8.3     3.6      4.6      10.0      5.4    8.1   4.4     11.6     14.1       86       131       218        87        728       101    7.212       8.53          1 8526.25
     Int     0/0          0   0.0      2.4     0.8      1.6       2.7      1.2   11.5   6.1     12.0     13.6       20        43        63        20        203        23    8.845       0.00          0    0.00
    Flush(GB): accumulative 2.266, interval 0.444
    Stalls(secs): 0.000 level0_slowdown, 0.000 level0_numfiles, 8.526 memtable_compaction, 0.000 leveln_slowdown_soft, 0.000 leveln_slowdown_hard
    Stalls(count): 0 level0_slowdown, 0 level0_numfiles, 1 memtable_compaction, 0 leveln_slowdown_soft, 0 leveln_slowdown_hard

    ** DB Stats **
    Uptime(secs): 336.8 total, 60.4 interval
    Cumulative writes: 61584000 writes, 6480589 batches, 9.5 writes per batch, 1.39 GB user ingest
    Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, 0.00 GB written
    Interval writes: 11235257 writes, 1175050 batches, 9.6 writes per batch, 259.9 MB user ingest
    Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, 0.00 MB written

To see what happened, go here: https://github.com/facebook/rocksdb/blob/47b452cfcf9b1487d41f886a98bc0d6f95587e90/db/compaction_picker.cc#L430
* The for loop started with level 1, because it has the worst score.
* PickCompactionBySize on L429 returned nullptr because all files were being compacted
* ExpandWhileOverlapping(c) returned true (because that's what it does when it gets nullptr!?)
* for loop break-ed, never trying compactions for level 2 :( :(

This bug was present at least since January. I have no idea how we didn't find this sooner.

Test Plan:
Unit testing compaction picker is hard. I tested this by running my service and observing L0->L1 and L2->L3 compactions in parallel. However, for long-term, I opened the task #4968469. @yhchiang is currently refactoring CompactionPicker, hopefully the new version will be unit-testable ;)

Here's how my compactions look like after the patch:

    2014/08/22-08:50:02.166699 7f3400ffb700 ------- DUMPING STATS -------
    2014/08/22-08:50:02.166722 7f3400ffb700
    ** Compaction Stats [default] **
    Level   Files   Size(MB) Score Read(GB)  Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) RW-Amp W-Amp Rd(MB/s) Wr(MB/s)  Rn(cnt) Rnp1(cnt) Wnp1(cnt) Wnew(cnt)  Comp(sec) Comp(cnt) Avg(sec) Stall(sec) Stall(cnt) Avg(ms)
    ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
      L0     8/5        404   1.5      0.0     0.0      0.0       4.3      4.3    0.0   0.0      0.0      9.6        0         0         0         0        463        88    5.260       0.00          0    0.00
      L1     2/2         60   0.9      4.8     3.9      0.8       4.7      3.9    2.4   1.2     23.9     23.6       80        23       131       108        204        19   10.747       0.00          0    0.00
      L2    23/3        697   1.0     11.6     3.5      8.1      10.9      2.8    6.4   3.1     17.7     16.6       95       242       317        75        669        92    7.268       0.00          0    0.00
      L3    58/14      2207   0.3      6.2     1.6      4.6       5.9      1.3    7.4   3.6     14.6     13.9       43       121       159        38        436        36   12.106       0.00          0    0.00
     Sum    91/24      3368   0.0     22.5     9.1     13.5      25.8     12.4   11.2   6.0     13.0     14.9      218       386       607       221       1772       235    7.538       0.00          0    0.00
     Int     0/0          0   0.0      3.2     0.9      2.3       3.6      1.3   15.3   8.0     12.4     13.7       24        66        89        23        266        27    9.838       0.00          0    0.00
    Flush(GB): accumulative 4.336, interval 0.444
    Stalls(secs): 0.000 level0_slowdown, 0.000 level0_numfiles, 0.000 memtable_compaction, 0.000 leveln_slowdown_soft, 0.000 leveln_slowdown_hard
    Stalls(count): 0 level0_slowdown, 0 level0_numfiles, 0 memtable_compaction, 0 leveln_slowdown_soft, 0 leveln_slowdown_hard

    ** DB Stats **
    Uptime(secs): 577.7 total, 60.1 interval
    Cumulative writes: 116960736 writes, 11966220 batches, 9.8 writes per batch, 2.64 GB user ingest
    Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, 0.00 GB written
    Interval writes: 11643735 writes, 1206136 batches, 9.7 writes per batch, 269.2 MB user ingest
    Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, 0.00 MB written

Yay for concurrent L0->L1 and L2->L3 compactions!

Reviewers: sdong, yhchiang, ljin

Reviewed By: yhchiang

Subscribers: yhchiang, leveldb

Differential Revision: https://reviews.facebook.net/D22305
      42ea7952
  5. 21 8月, 2014 4 次提交
  7. 19 8月, 2014 4 次提交
    • R
      Adding Column Family support in db_bench. · 162b8151
      由 Radheshyam Balasundaram 提交于 
      Summary:
Adding num_column_families flag. Adding support for column families in DoWrite and ReadRandom methods.
[Igor, please let me know if this approach sounds good. I shall add it to other methods too.]

Test Plan: Ran fillseq on 1M keys and 10 Column families and ran readrandom.

Reviewers: sdong, yhchiang, igor, ljin

Reviewed By: ljin

Subscribers: leveldb

Differential Revision: https://reviews.facebook.net/D21387
      162b8151
    • S
      WriteBatchWithIndex: a wrapper of WriteBatch, with a searchable index · 28b5c760
      由 sdong 提交于 
      Summary:
Add WriteBatchWithIndex so that a user can query data out of a WriteBatch, to support MongoDB's read-its-own-write.

WriteBatchWithIndex uses a skiplist to store the binary index. The index stores the offset of the entry in the write batch. When searching for a key, the key for the entry is read by read the entry from the write batch from the offset.

Define a new iterator class for querying data out of WriteBatchWithIndex. A user can create an iterator of the write batch for one column family, seek to a key and keep calling Next() to see next entries.

I will add more unit tests if people are OK about this API.

Test Plan:
make all check
Add unit tests.

Reviewers: yhchiang, igor, MarkCallaghan, ljin

Reviewed By: ljin

Subscribers: dhruba, leveldb, xjin

Differential Revision: https://reviews.facebook.net/D21381
      28b5c760
    • R
      Adding Cuckoo Table SST option to db_bench · 36e759d1
      由 Radheshyam Balasundaram 提交于 
      Summary: Adding flags to use cuckoo table SST in db_bench.cc

Test Plan: Ran benchmark with fillseq and readrandom

Reviewers: sdong, ljin

Reviewed By: ljin

Subscribers: leveldb

Differential Revision: https://reviews.facebook.net/D21729
      36e759d1
    • I
      Fix valgrind error in c_test · a6fd14c8
      由 Igor Canadi 提交于
      a6fd14c8
  9. 16 8月, 2014 1 次提交
  11. 15 8月, 2014 1 次提交
    • S
      Support purging logs from separate log directory · 58b0f9d8
      由 sdong 提交于 
      Summary:
1. Support purging info logs from a separate paths from DB path. Refactor the codes of generating info log prefixes so that it can be called when generating new files and scanning log directory.
2. Fix the bug of not scanning multiple DB paths (should only impact multiple DB paths)

Test Plan:
Add unit test for generating and parsing info log files
Add end-to-end test in db_test

Reviewers: yhchiang, ljin

Reviewed By: ljin

Subscribers: leveldb, igor, dhruba

Differential Revision: https://reviews.facebook.net/D21801
      58b0f9d8
  13. 14 8月, 2014 4 次提交
  15. 13 8月, 2014 7 次提交
  17. 12 8月, 2014 4 次提交
    • S
      Flush only one column family · 06a52bda
      由 Stanislau Hlebik 提交于 
      Summary:
Currently DBImpl::Flush() triggers flushes in all column families.
Instead we need to trigger just the column family specified.

Test Plan: make all check

Reviewers: igor, ljin, yhchiang, sdong

Reviewed By: sdong

Subscribers: leveldb

Differential Revision: https://reviews.facebook.net/D20841
      06a52bda
    • R
      Integrating Cuckoo Hash SST Table format into RocksDB · 9674c11d
      由 Radheshyam Balasundaram 提交于 
      Summary:
Contains the following changes:
- Implementation of cuckoo_table_factory
- Adding cuckoo table into AdaptiveTableFactory
- Adding cuckoo_table_db_test, similar to lines of plain_table_db_test
- Minor fixes to Reader: When a key is found in the table, return the key found instead of the search key.
- Minor fixes to Builder: Add table properties that are required by Version::UpdateTemporaryStats() during Get operation. Don't define curr_node as a reference variable as the memory locations may get reassigned during tree.push_back operation, leading to invalid memory access.

Test Plan:
cuckoo_table_reader_test --enable_perf
cuckoo_table_builder_test
cuckoo_table_db_test
make check all
make valgrind_check
make asan_check

Reviewers: sdong, igor, yhchiang, ljin

Reviewed By: ljin

Subscribers: leveldb

Differential Revision: https://reviews.facebook.net/D21219
      9674c11d
    • F
      check prefix_size when using hash search in db_bench · d3f2ec69
      由 Feng Zhu 提交于 
      Summary:
1. Check prefix_size when enable use_hash_search in db_bench
2. Remove include/statistics.h in db_bench

Test Plan: ./db_bench --use_hash_search=1

Reviewers: ljin, yhchiang, igor, sdong

Reviewed By: sdong

Subscribers: leveldb

Differential Revision: https://reviews.facebook.net/D21375
      d3f2ec69
    • M
      Changes to support unity build: · 93e6b5e9
      由 miguelportilla 提交于 
      * Script for building the unity.cc file via Makefile
* Unity executable Makefile target for testing builds
* Source code changes to fix compilation of unity build
      93e6b5e9
  19. 09 8月, 2014 4 次提交
  21. 08 8月, 2014 2 次提交
  23. 07 8月, 2014 5 次提交