- 26 8月, 2014 3 次提交
-
-
由 Lei Jin 提交于
ReadOptions.total_order_seek to allow total order seek for block-based table when hash index is enabled Summary: as title Test Plan: table_test Reviewers: igor, yhchiang, sdong Reviewed By: sdong Subscribers: leveldb Differential Revision: https://reviews.facebook.net/D22239
-
由 Lei Jin 提交于
Summary: Add a virtual function in table factory that will print table options Test Plan: make release Reviewers: igor, yhchiang, sdong Reviewed By: sdong Subscribers: leveldb Differential Revision: https://reviews.facebook.net/D22149
-
由 Lei Jin 提交于
Summary: I will move compression related options in a separate diff since this diff is already pretty lengthy. I guess I will also need to change JNI accordingly :( Test Plan: make all check Reviewers: yhchiang, igor, sdong Reviewed By: igor Subscribers: leveldb Differential Revision: https://reviews.facebook.net/D21915
-
- 23 8月, 2014 1 次提交
-
-
由 Igor Canadi 提交于
Summary: I am currently working on a project that uses RocksDB. While debugging some perf issues, I came up across interesting compaction concurrency issue. Namely, I had 15 idle threads and a good comapction to do, but CompactionPicker returned "Compaction nothing to do". Here's how Internal stats looked: 2014/08/22-08:08:04.551982 7fc7fc3f5700 ------- DUMPING STATS ------- 2014/08/22-08:08:04.552000 7fc7fc3f5700 ** Compaction Stats [default] ** Level Files Size(MB) Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) RW-Amp W-Amp Rd(MB/s) Wr(MB/s) Rn(cnt) Rnp1(cnt) Wnp1(cnt) Wnew(cnt) Comp(sec) Comp(cnt) Avg(sec) Stall(sec) Stall(cnt) Avg(ms) ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ L0 7/5 353 1.0 0.0 0.0 0.0 2.3 2.3 0.0 0.0 0.0 9.4 0 0 0 0 247 46 5.359 8.53 1 8526.25 L1 2/2 86 1.3 2.6 1.9 0.7 2.6 1.9 2.7 1.3 24.3 24.0 39 19 71 52 109 11 9.938 0.00 0 0.00 L2 26/0 833 1.3 5.7 1.7 4.0 5.2 1.2 6.3 3.0 15.6 14.2 47 112 147 35 373 44 8.468 0.00 0 0.00 L3 12/0 505 0.1 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 0 0 0 0 0.000 0.00 0 0.00 Sum 47/7 1778 0.0 8.3 3.6 4.6 10.0 5.4 8.1 4.4 11.6 14.1 86 131 218 87 728 101 7.212 8.53 1 8526.25 Int 0/0 0 0.0 2.4 0.8 1.6 2.7 1.2 11.5 6.1 12.0 13.6 20 43 63 20 203 23 8.845 0.00 0 0.00 Flush(GB): accumulative 2.266, interval 0.444 Stalls(secs): 0.000 level0_slowdown, 0.000 level0_numfiles, 8.526 memtable_compaction, 0.000 leveln_slowdown_soft, 0.000 leveln_slowdown_hard Stalls(count): 0 level0_slowdown, 0 level0_numfiles, 1 memtable_compaction, 0 leveln_slowdown_soft, 0 leveln_slowdown_hard ** DB Stats ** Uptime(secs): 336.8 total, 60.4 interval Cumulative writes: 61584000 writes, 6480589 batches, 9.5 writes per batch, 1.39 GB user ingest Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, 0.00 GB written Interval writes: 11235257 writes, 1175050 batches, 9.6 writes per batch, 259.9 MB user ingest Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, 0.00 MB written To see what happened, go here: https://github.com/facebook/rocksdb/blob/47b452cfcf9b1487d41f886a98bc0d6f95587e90/db/compaction_picker.cc#L430 * The for loop started with level 1, because it has the worst score. * PickCompactionBySize on L429 returned nullptr because all files were being compacted * ExpandWhileOverlapping(c) returned true (because that's what it does when it gets nullptr!?) * for loop break-ed, never trying compactions for level 2 :( :( This bug was present at least since January. I have no idea how we didn't find this sooner. Test Plan: Unit testing compaction picker is hard. I tested this by running my service and observing L0->L1 and L2->L3 compactions in parallel. However, for long-term, I opened the task #4968469. @yhchiang is currently refactoring CompactionPicker, hopefully the new version will be unit-testable ;) Here's how my compactions look like after the patch: 2014/08/22-08:50:02.166699 7f3400ffb700 ------- DUMPING STATS ------- 2014/08/22-08:50:02.166722 7f3400ffb700 ** Compaction Stats [default] ** Level Files Size(MB) Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) RW-Amp W-Amp Rd(MB/s) Wr(MB/s) Rn(cnt) Rnp1(cnt) Wnp1(cnt) Wnew(cnt) Comp(sec) Comp(cnt) Avg(sec) Stall(sec) Stall(cnt) Avg(ms) ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ L0 8/5 404 1.5 0.0 0.0 0.0 4.3 4.3 0.0 0.0 0.0 9.6 0 0 0 0 463 88 5.260 0.00 0 0.00 L1 2/2 60 0.9 4.8 3.9 0.8 4.7 3.9 2.4 1.2 23.9 23.6 80 23 131 108 204 19 10.747 0.00 0 0.00 L2 23/3 697 1.0 11.6 3.5 8.1 10.9 2.8 6.4 3.1 17.7 16.6 95 242 317 75 669 92 7.268 0.00 0 0.00 L3 58/14 2207 0.3 6.2 1.6 4.6 5.9 1.3 7.4 3.6 14.6 13.9 43 121 159 38 436 36 12.106 0.00 0 0.00 Sum 91/24 3368 0.0 22.5 9.1 13.5 25.8 12.4 11.2 6.0 13.0 14.9 218 386 607 221 1772 235 7.538 0.00 0 0.00 Int 0/0 0 0.0 3.2 0.9 2.3 3.6 1.3 15.3 8.0 12.4 13.7 24 66 89 23 266 27 9.838 0.00 0 0.00 Flush(GB): accumulative 4.336, interval 0.444 Stalls(secs): 0.000 level0_slowdown, 0.000 level0_numfiles, 0.000 memtable_compaction, 0.000 leveln_slowdown_soft, 0.000 leveln_slowdown_hard Stalls(count): 0 level0_slowdown, 0 level0_numfiles, 0 memtable_compaction, 0 leveln_slowdown_soft, 0 leveln_slowdown_hard ** DB Stats ** Uptime(secs): 577.7 total, 60.1 interval Cumulative writes: 116960736 writes, 11966220 batches, 9.8 writes per batch, 2.64 GB user ingest Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, 0.00 GB written Interval writes: 11643735 writes, 1206136 batches, 9.7 writes per batch, 269.2 MB user ingest Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, 0.00 MB written Yay for concurrent L0->L1 and L2->L3 compactions! Reviewers: sdong, yhchiang, ljin Reviewed By: yhchiang Subscribers: yhchiang, leveldb Differential Revision: https://reviews.facebook.net/D22305
-
- 21 8月, 2014 4 次提交
-
-
由 Yueh-Hsuan Chiang 提交于
Summary: Fix the error of c_test.c Test Plan: make c_test ./c_test
-
由 Yueh-Hsuan Chiang 提交于
Summary: Add missing implementaiton of SanitizeDBOptions in simple_table_db_test.cc Test Plan: make simple_table_db_test.cc
-
由 Yueh-Hsuan Chiang 提交于
Summary: Currently, PlainTable must use mmap_reads. When PlainTable is used but allow_mmap_reads is not set, rocksdb will fail in flush. This diff improve Options sanitization and add MmapReadRequired() to TableFactory. Test Plan: export ROCKSDB_TESTS=PlainTableOptionsSanitizeTest make db_test -j32 ./db_test Reviewers: sdong, ljin Reviewed By: ljin Subscribers: you, leveldb Differential Revision: https://reviews.facebook.net/D21939
-
由 sdong 提交于
Summary: A previous change triggered a change by mistake: DestroyDB() will keep info logs under DB directory. Revert the unintended change. Test Plan: Add a unit test case to verify it. Reviewers: ljin, yhchiang, igor Reviewed By: igor Subscribers: leveldb Differential Revision: https://reviews.facebook.net/D22209
-
- 19 8月, 2014 4 次提交
-
-
由 Radheshyam Balasundaram 提交于
Summary: Adding num_column_families flag. Adding support for column families in DoWrite and ReadRandom methods. [Igor, please let me know if this approach sounds good. I shall add it to other methods too.] Test Plan: Ran fillseq on 1M keys and 10 Column families and ran readrandom. Reviewers: sdong, yhchiang, igor, ljin Reviewed By: ljin Subscribers: leveldb Differential Revision: https://reviews.facebook.net/D21387
-
由 sdong 提交于
Summary: Add WriteBatchWithIndex so that a user can query data out of a WriteBatch, to support MongoDB's read-its-own-write. WriteBatchWithIndex uses a skiplist to store the binary index. The index stores the offset of the entry in the write batch. When searching for a key, the key for the entry is read by read the entry from the write batch from the offset. Define a new iterator class for querying data out of WriteBatchWithIndex. A user can create an iterator of the write batch for one column family, seek to a key and keep calling Next() to see next entries. I will add more unit tests if people are OK about this API. Test Plan: make all check Add unit tests. Reviewers: yhchiang, igor, MarkCallaghan, ljin Reviewed By: ljin Subscribers: dhruba, leveldb, xjin Differential Revision: https://reviews.facebook.net/D21381
-
由 Radheshyam Balasundaram 提交于
Summary: Adding flags to use cuckoo table SST in db_bench.cc Test Plan: Ran benchmark with fillseq and readrandom Reviewers: sdong, ljin Reviewed By: ljin Subscribers: leveldb Differential Revision: https://reviews.facebook.net/D21729
-
由 Igor Canadi 提交于
-
- 16 8月, 2014 1 次提交
-
-
由 Yueh-Hsuan Chiang 提交于
Summary: Avoid retrying to read property block from a table when it does not exist in updating stats for compensating deletion entries. In addition, ReadTableProperties() now returns Status::NotFound instead of Status::Corruption when table properties does not exist in the file. Test Plan: make db_test -j32 export ROCKSDB_TESTS=CompactionDeleteionTrigger ./db_test Reviewers: ljin, sdong Reviewed By: sdong Subscribers: leveldb Differential Revision: https://reviews.facebook.net/D21867
-
- 15 8月, 2014 1 次提交
-
-
由 sdong 提交于
Summary: 1. Support purging info logs from a separate paths from DB path. Refactor the codes of generating info log prefixes so that it can be called when generating new files and scanning log directory. 2. Fix the bug of not scanning multiple DB paths (should only impact multiple DB paths) Test Plan: Add unit test for generating and parsing info log files Add end-to-end test in db_test Reviewers: yhchiang, ljin Reviewed By: ljin Subscribers: leveldb, igor, dhruba Differential Revision: https://reviews.facebook.net/D21801
-
- 14 8月, 2014 4 次提交
-
-
由 Lei Jin 提交于
Summary: This is a linux-specific system call. Test Plan: ran db_bench Reviewers: igor, yhchiang, sdong Reviewed By: sdong Subscribers: haobo, leveldb Differential Revision: https://reviews.facebook.net/D21183
-
由 Lei Jin 提交于
Summary: as title Test Plan: make release Reviewers: sdong, yhchiang Reviewed By: yhchiang Subscribers: leveldb Differential Revision: https://reviews.facebook.net/D21717
-
由 Feng Zhu 提交于
Summary: 1. write db MANIFEST, CURRENT, IDENTITY, sst files, log files to log before open Test Plan: run db and check LOG file Reviewers: ljin, yhchiang, igor, dhruba, sdong Reviewed By: sdong Subscribers: leveldb Differential Revision: https://reviews.facebook.net/D21459
-
由 Stanislau Hlebik 提交于
Summary: Replace unnecessary malloc with stack allocation Test Plan: make all check Reviewers: sdong Reviewed By: sdong Subscribers: leveldb Differential Revision: https://reviews.facebook.net/D21771
-
- 13 8月, 2014 7 次提交
-
-
由 Yueh-Hsuan Chiang 提交于
Summary: Fixed compile errors (signed / unsigned comparison) in cuckoo_table_db_test on Mac Test Plan: make cuckoo_table_db_test
-
由 Yueh-Hsuan Chiang 提交于
Summary: Fixed a signed-unsigned comparison error in db_test Test Plan: make db_test
-
由 Lei Jin 提交于
Summary: as title Test Plan: make all check ran db_bench and saw seek stats at the end Reviewers: yhchiang, igor, sdong Reviewed By: sdong Subscribers: leveldb Differential Revision: https://reviews.facebook.net/D21651
-
由 Lei Jin 提交于
Summary: as title Test Plan: make all check Reviewers: igor, yhchiang, sdong Reviewed By: sdong Subscribers: leveldb Differential Revision: https://reviews.facebook.net/D21201
-
由 Spencer Kimball 提交于
Previously, the prefix extractor was being supplied with the RocksDB key instead of a parsed user key. This makes correct interpretation by calling application fragile or impossible.
-
由 Stanislau Hlebik 提交于
Summary: Small change: replace mutex_.Lock/mutex_.Unlock() with scope guard Test Plan: make all check Reviewers: igor, sdong Reviewed By: sdong Subscribers: leveldb Differential Revision: https://reviews.facebook.net/D21609
- 12 8月, 2014 4 次提交
-
-
由 Stanislau Hlebik 提交于
Summary: Currently DBImpl::Flush() triggers flushes in all column families. Instead we need to trigger just the column family specified. Test Plan: make all check Reviewers: igor, ljin, yhchiang, sdong Reviewed By: sdong Subscribers: leveldb Differential Revision: https://reviews.facebook.net/D20841
-
由 Radheshyam Balasundaram 提交于
Summary: Contains the following changes: - Implementation of cuckoo_table_factory - Adding cuckoo table into AdaptiveTableFactory - Adding cuckoo_table_db_test, similar to lines of plain_table_db_test - Minor fixes to Reader: When a key is found in the table, return the key found instead of the search key. - Minor fixes to Builder: Add table properties that are required by Version::UpdateTemporaryStats() during Get operation. Don't define curr_node as a reference variable as the memory locations may get reassigned during tree.push_back operation, leading to invalid memory access. Test Plan: cuckoo_table_reader_test --enable_perf cuckoo_table_builder_test cuckoo_table_db_test make check all make valgrind_check make asan_check Reviewers: sdong, igor, yhchiang, ljin Reviewed By: ljin Subscribers: leveldb Differential Revision: https://reviews.facebook.net/D21219
-
由 Feng Zhu 提交于
Summary: 1. Check prefix_size when enable use_hash_search in db_bench 2. Remove include/statistics.h in db_bench Test Plan: ./db_bench --use_hash_search=1 Reviewers: ljin, yhchiang, igor, sdong Reviewed By: sdong Subscribers: leveldb Differential Revision: https://reviews.facebook.net/D21375
-
由 miguelportilla 提交于
* Script for building the unity.cc file via Makefile * Unity executable Makefile target for testing builds * Source code changes to fix compilation of unity build
-
- 09 8月, 2014 4 次提交
-
-
由 sdong 提交于
Summary: In FindObsoleteFiles(), we don't scan db_log_dir. Add it. Test Plan: make all check Reviewers: ljin, igor, yhchiang Reviewed By: yhchiang Subscribers: leveldb, yhchiang Differential Revision: https://reviews.facebook.net/D21429
-
由 sdong 提交于
Summary: If there is an outstanding compaction scheduled but at the time a manual compaction is triggered, the manual compaction will preempt. In the end of the manual compaction, we should try to schedule compactions to make sure those preempted ones are not skipped. Test Plan: make all check Reviewers: yhchiang, ljin Reviewed By: ljin Subscribers: leveldb, dhruba, igor Differential Revision: https://reviews.facebook.net/D21321
-
由 Igor Canadi 提交于
Summary: Travis build was failing a lot. For example see https://travis-ci.org/facebook/rocksdb/builds/31425845 This fixes it. Also, please don't put any code after SignalAll :) Test Plan: no more SIGSEGV Reviewers: yhchiang, sdong, ljin Reviewed By: ljin Subscribers: dhruba, leveldb Differential Revision: https://reviews.facebook.net/D21417
-
由 Stanislau Hlebik 提交于
Summary: Fix old issue with DBTest.Randomized with BlockBasedTableWithWholeKeyHashIndex + added printing in DBTest.Randomized. Test Plan: make all check Reviewers: zagfox, igor, ljin, yhchiang, sdong Reviewed By: sdong Subscribers: leveldb Differential Revision: https://reviews.facebook.net/D21003
-
- 08 8月, 2014 2 次提交
-
-
由 Igor Canadi 提交于
-
由 Igor Canadi 提交于
-
- 07 8月, 2014 5 次提交
-
-
由 sdong 提交于
Summary: A format fixing Test Plan: N/A Reviewers: ljin Reviewed By: ljin Subscribers: leveldb Differential Revision: https://reviews.facebook.net/D21255
-
由 sdong 提交于
Summary: Initialize un-initialized parameters Test Plan: run the failed test (c_test) Reviewers: yhchiang, ljin Reviewed By: ljin Subscribers: leveldb Differential Revision: https://reviews.facebook.net/D21249
-
由 Spencer Kimball 提交于
create_compaction_filter_v2_.
-
由 Spencer Kimball 提交于
Test Plan: make c_test && ./c_test Some fixes after merge.
-
由 sdong 提交于
Summary: Add a DB Property "rocksdb.estimate-table-readers-mem" to return estimated memory usage by all loaded table readers, other than allocated from block cache. Refactor the property codes to allow getting property from a version, with DB mutex not acquired. Test Plan: Add several checks of this new property in existing codes for various cases. Reviewers: yhchiang, ljin Reviewed By: ljin Subscribers: xjin, igor, leveldb Differential Revision: https://reviews.facebook.net/D20733
-