- 12 9月, 2014 3 次提交
-
-
由 Igor Canadi 提交于
Summary: If bg_error_ is set, that means that we mark DB read only. However, current behavior still continues the flushes and compactions, even though bg_error_ is set. On the other hand, if bg_error_ is set, we will return Status::OK() from CompactRange(), although the compaction didn't actually succeed. This is clearly not desired behavior. I found this when I was debugging t5132159, although I'm pretty sure these aren't related. Also, when we're shutting down, it's dangerous to exit RunManualCompaction(), since that will destruct ManualCompaction object. Background compaction job might still hold a reference to manual_compaction_ and this will lead to undefined behavior. I changed the behavior so that we only exit RunManualCompaction when manual compaction job is marked done. Test Plan: make check Reviewers: sdong, ljin, yhchiang Reviewed By: yhchiang Subscribers: leveldb Differential Revision: https://reviews.facebook.net/D23223
-
由 Igor Canadi 提交于
Summary: Get valgrind to stop complaining about uninitialized value Test Plan: valgrind not complaining anymore Reviewers: sdong, yhchiang, ljin Reviewed By: ljin Subscribers: leveldb Differential Revision: https://reviews.facebook.net/D23289
-
由 Igor Canadi 提交于
Summary: The test makes sure that we don't call flush too often. For that, it's ok to check if we have less than 10 table files. Otherwise, the test is flaky because it's hard to estimate number of entries in the memtable before it gets flushed (any ideas?) Test Plan: Still works, but hopefully less flaky. Reviewers: ljin, sdong, yhchiang Reviewed by: yhchiang Subscribers: leveldb Differential Revision: https://reviews.facebook.net/D23241
-
- 11 9月, 2014 4 次提交
-
-
由 Igor Canadi 提交于
Summary: When memtable is full it calls the registered callback. That callback then registers column family as needing the flush. Every write checks if there are some column families that need to be flushed. This completely eliminates the need for MakeRoomForWrite() function and simplifies our Write code-path. There is some complexity with the concurrency when the column family is dropped. I made it a bit less complex by dropping the column family from the write thread in https://reviews.facebook.net/D22965. Let me know if you want to discuss this. Test Plan: make check works. I'll also run db_stress with creating and dropping column families for a while. Reviewers: yhchiang, sdong, ljin Reviewed By: ljin Subscribers: leveldb Differential Revision: https://reviews.facebook.net/D23067
-
由 Igor Canadi 提交于
Summary: See t5106397. Also, few more changes: 1. in unit tests, the assumption is that writes will be dropped when there is no space left on device. I changed the wording around it. 2. InvalidArgument() errors are only when user-provided arguments are invalid. When the file is corrupted, we need to return Status::Corruption Test Plan: make check Reviewers: sdong, ljin Reviewed By: ljin Subscribers: leveldb Differential Revision: https://reviews.facebook.net/D23145
-
由 Lei Jin 提交于
Summary: as title Test Plan: make rocksdbjava Reviewers: sdong, yhchiang, igor Reviewed By: igor Subscribers: leveldb Differential Revision: https://reviews.facebook.net/D23193
-
由 Feng Zhu 提交于
Summary: print qps in summary Test Plan: ./cache_bench Reviewers: yhchiang, ljin, sdong, igor Reviewed By: igor Subscribers: leveldb Differential Revision: https://reviews.facebook.net/D23079
-
- 10 9月, 2014 6 次提交
-
-
由 Igor Canadi 提交于
-
由 Jonah Cohen 提交于
Summary: Correct some comments and typos in RocksDB. Test Plan: Inspection Reviewers: sdong, igor Reviewed By: igor Differential Revision: https://reviews.facebook.net/D23133
-
由 Xiaozheng Tie 提交于
Summary: Added the following statistics to BackupableDB: 1. Number of successful and failed backups in class BackupStatistics 2. Time taken to do a backup 3. Number of files in a backup 1 is implemented in the BackupStatistics class 2 and 3 are added in the BackupMeta and BackupInfo class Test Plan: 1 can be tested using BackupStatistics::ToString(), 2 and 3 can be tested in the BackupInfo class Reviewers: sdong, igor2, ljin, igor Reviewed By: igor Differential Revision: https://reviews.facebook.net/D22785
-
由 Igor Canadi 提交于
Summary: In column family's SanitizeOptions() [1], we make sure that min_write_buffer_number_to_merge is normal value. However, this test depended on the fact that setting min_write_buffer_number_to_merge to be bigger than max_write_buffer_number will cause a deadlock. I'm not sure how it worked before. This diff fixes it by scheduling sleeping background task, which will actually block any attempts of flushing. [1] https://github.com/facebook/rocksdb/blob/master/db/column_family.cc#L104 Test Plan: the test works now Reviewers: yhchiang, sdong, ljin Reviewed By: ljin Subscribers: leveldb Differential Revision: https://reviews.facebook.net/D23103
-
由 sdong 提交于
Summary: To follow the coding convention and make sure when passing reference as a parameter it is also const, pass MergeContext as a pointer to mem tables. Test Plan: make all check Reviewers: ljin, igor Reviewed By: igor Subscribers: leveldb, dhruba, yhchiang Differential Revision: https://reviews.facebook.net/D23085
-
由 Stanislau Hlebik 提交于
Summary: Avoid creating unnecessary sst files while db opening Test Plan: make all check Reviewers: sdong, igor Reviewed By: igor Subscribers: zagfox, yhchiang, ljin, leveldb Differential Revision: https://reviews.facebook.net/D20661
-
- 09 9月, 2014 12 次提交
-
-
由 Igor Canadi 提交于
Summary: I abandoned https://reviews.facebook.net/D18789, but I wrote a good unit test there, so let's check it in. :) Test Plan: this is test Reviewers: sdong, yhchiang, ljin Reviewed By: ljin Subscribers: leveldb Differential Revision: https://reviews.facebook.net/D22827
-
由 Igor Canadi 提交于
Summary: The compilers we use treat char as signed. However, this is not guarantee of C standard and some compilers (for ARM platform for example), treat char as unsigned. Code that assumes that char is either signed or unsigned is wrong. This change explicitly casts the char to signed version. This will not break any of our use cases on x86, which, I believe are all of them. In case somebody out there is using RocksDB on ARM AND using bloom filters, they're going to have a bad time. However, it is very unlikely that this is the case. Test Plan: sanity test with previous commit (with new sanity test) Reviewers: yhchiang, ljin, sdong Reviewed By: ljin Subscribers: dhruba, leveldb Differential Revision: https://reviews.facebook.net/D22767
-
由 Lei Jin 提交于
Summary: removed reference to options in WriteBatch and DBImpl::Get() Test Plan: make all check Reviewers: yhchiang, igor, sdong Reviewed By: sdong Subscribers: leveldb Differential Revision: https://reviews.facebook.net/D23049
-
由 Igor Canadi 提交于
Summary: 1. add db statistics 2. write out batch every millionth write Test Plan: unit tests Reviewers: ljin, sdong, yinwang Reviewed By: yinwang Differential Revision: https://reviews.facebook.net/D22755
-
由 Lei Jin 提交于
Summary: as title Test Plan: make release Reviewers: igor Differential Revision: https://reviews.facebook.net/D23073
-
由 Lei Jin 提交于
Summary: as title Test Plan: make release Reviewers: sdong, yhchiang, igor Reviewed By: igor Subscribers: leveldb Differential Revision: https://reviews.facebook.net/D23007
-
由 Igor Canadi 提交于
Summary: ... Test Plan: Can't repro the test failure, but let's see what jenkins says Reviewers: zagfox, sdong, ljin Reviewed By: sdong, ljin Subscribers: leveldb Differential Revision: https://reviews.facebook.net/D23061
-
由 Lei Jin 提交于
Summary: all shared_ptrs are in immutable_options now. This will also make options assignment a little cheaper Test Plan: make release Reviewers: sdong, yhchiang, igor Reviewed By: igor Subscribers: leveldb Differential Revision: https://reviews.facebook.net/D23001
-
由 Lei Jin 提交于
Summary: I found it is almost impossible to get rid of this function in a single batch. I will take a step by step approach Test Plan: make release Reviewers: sdong, yhchiang, igor Reviewed By: igor Subscribers: leveldb Differential Revision: https://reviews.facebook.net/D22995
-
由 sdong 提交于
Summary: When we have multiple column families, users can issue Flush() on every column families to make sure everything is flushes, even if some of them might be empty. By skipping the waiting for empty cases, it can be greatly speed up. Still wait for people's comments before writing unit tests for it. Test Plan: Will write a unit test to make sure it is correct. Reviewers: ljin, yhchiang, igor Reviewed By: igor Subscribers: leveldb, dhruba Differential Revision: https://reviews.facebook.net/D22953
-
由 Igor Canadi 提交于
Summary: Introducing WriteController, which is a source of truth about per-DB write delays. Let's define an DB epoch as a period where there are no flushes and compactions (i.e. new epoch is started when flush or compaction finishes). Each epoch can either: * proceed with all writes without delay * delay all writes by fixed time * stop all writes The three modes are recomputed at each epoch change (flush, compaction), rather than on every write (which is currently the case). When we have a lot of column families, our current pull behavior adds a big overhead, since we need to loop over every column family for every write. With new push model, overhead on Write code-path is minimal. This is just the start. Next step is to also take care of stalls introduced by slow memtable flushes. The final goal is to eliminate function MakeRoomForWrite(), which currently needs to be called for every column family by every write. Test Plan: make check for now. I'll add some unit tests later. Also, perf test. Reviewers: dhruba, yhchiang, MarkCallaghan, sdong, ljin Reviewed By: ljin Subscribers: leveldb Differential Revision: https://reviews.facebook.net/D22791
-
由 Feng Zhu 提交于
Summary: 1. Make filter_block.h a base class. Derive block_based_filter_block and full_filter_block. The previous one is the traditional filter block. The full_filter_block is newly added. It would generate a filter block that contain all the keys in SST file. 2. When querying a key, table would first check if full_filter is available. If not, it would go to the exact data block and check using block_based filter. 3. User could choose to use full_filter or tradional(block_based_filter). They would be stored in SST file with different meta index name. "filter.filter_policy" or "full_filter.filter_policy". Then, Table reader is able to know the fllter block type. 4. Some optimizations have been done for full_filter_block, thus it requires a different interface compared to the original one in filter_policy.h. 5. Actual implementation of filter bits coding/decoding is placed in util/bloom_impl.cc Benchmark: base commit 1d23b5c4 Command: db_bench --db=/dev/shm/rocksdb --num_levels=6 --key_size=20 --prefix_size=20 --keys_per_prefix=0 --value_size=100 --write_buffer_size=134217728 --max_write_buffer_number=2 --target_file_size_base=33554432 --max_bytes_for_level_base=1073741824 --verify_checksum=false --max_background_compactions=4 --use_plain_table=0 --memtablerep=prefix_hash --open_files=-1 --mmap_read=1 --mmap_write=0 --bloom_bits=10 --bloom_locality=1 --memtable_bloom_bits=500000 --compression_type=lz4 --num=393216000 --use_hash_search=1 --block_size=1024 --block_restart_interval=16 --use_existing_db=1 --threads=1 --benchmarks=readrandom —disable_auto_compactions=1 Read QPS increase for about 30% from 2230002 to 2991411. Test Plan: make all check valgrind db_test db_stress --use_block_based_filter = 0 ./auto_sanity_test.sh Reviewers: igor, yhchiang, ljin, sdong Reviewed By: sdong Subscribers: dhruba, leveldb Differential Revision: https://reviews.facebook.net/D20979
-
- 08 9月, 2014 5 次提交
-
-
由 Igor Canadi 提交于
-
由 Igor Canadi 提交于
fix comments
-
由 wankai 提交于
-
由 Igor Canadi 提交于
replace hard-coded number with named variable
-
由 wankai 提交于
-
- 07 9月, 2014 2 次提交
-
-
由 Igor Canadi 提交于
Cleanups from static analysis
-
由 Igor Canadi 提交于
typo improvement
-
- 06 9月, 2014 8 次提交
-
-
由 wankai 提交于
-
由 wankai 提交于
-
由 Nik Bougalis 提交于
-
由 Nik Bougalis 提交于
-
由 Nik Bougalis 提交于
-
由 Nik Bougalis 提交于
-
由 Nik Bougalis 提交于
-
由 Feng Zhu 提交于
Summary: 1. A benchmark for cache Test Plan: ./cache_bench Reviewers: yhchiang, dhruba, sdong, igor, ljin Reviewed By: ljin Subscribers: leveldb Differential Revision: https://reviews.facebook.net/D22809
-