- 08 7月, 2014 8 次提交
-
-
由 Igor Canadi 提交于
C API: Add support for compaction filter factories (v1)
-
由 Evan Shaw 提交于
Also refactored the compaction filter tests to share some code and ensure that options were getting reset so future test results aren't confused.
-
由 Evan Shaw 提交于
-
由 Evan Shaw 提交于
-
由 Evan Shaw 提交于
-
由 Radheshyam Balasundaram 提交于
Summary: Changes: - Adding numa_aware flag to db_bench.cc - Using numa.h library to bind memory and cpu of threads to a fixed NUMA node Result: There seems to be no significant change in the micros/op time with numa_aware enabled. I also tried this with other implementations, including a combination of pthread_setaffinity_np, sched_setaffinity and set_mempolicy methods. It'd be great if someone could point out where I'm going wrong and if we can achieve a better micors/op. Test Plan: Ran db_bench tests using following command: ./db_bench --db=/mnt/tmp --num_levels=6 --key_size=20 --prefix_size=20 --keys_per_prefix=0 --value_size=100 --block_size=4096 --cache_size=17179869184 --cache_numshardbits=6 --compression_type=none --compression_ratio=1 --min_level_to_compress=-1 --disable_seek_compaction=1 --hard_rate_limit=2 --write_buffer_size=134217728 --max_write_buffer_number=2 --level0_file_num_compaction_trigger=8 --target_file_size_base=134217728 --max_bytes_for_level_base=1073741824 --disable_wal=0 --wal_dir=/mnt/tmp --sync=0 --disable_data_sync=1 --verify_checksum=1 --delete_obsolete_files_period_micros=314572800 --max_grandparent_overlap_factor=10 --max_background_compactions=4 --max_background_flushes=0 --level0_slowdown_writes_trigger=16 --level0_stop_writes_trigger=24 --statistics=0 --stats_per_interval=0 --stats_interval=1048576 --histogram=0 --use_plain_table=1 --open_files=-1 --mmap_read=1 --mmap_write=0 --memtablerep=prefix_hash --bloom_bits=10 --bloom_locality=1 --perf_level=0 --duration=300 --benchmarks=readwhilewriting --use_existing_db=1 --num=157286400 --threads=24 --writes_per_second=10240 --numa_aware=[False/True] The tests were run in private devserver with 24 cores and the db was prepopulated using filluniquerandom test. The tests resulted in 0.145 us/op with numa_aware=False and 0.161 us/op with numa_aware=True. Reviewers: sdong, yhchiang, ljin, igor Reviewed By: ljin, igor Subscribers: igor, leveldb Differential Revision: https://reviews.facebook.net/D19353
-
由 Igor Canadi 提交于
C API: support constructing write batch from serialized representation
-
由 Igor Canadi 提交于
C API: column family support
-
- 07 7月, 2014 1 次提交
-
-
由 Reed Allman 提交于
-
- 06 7月, 2014 1 次提交
-
-
由 Evan Shaw 提交于
-
- 04 7月, 2014 8 次提交
-
-
由 Yueh-Hsuan Chiang 提交于
Summary: SimpleWriteTimeoutTest has two parts: 1) insert two large key/values to make memtable full and expect both of them are successful; 2) insert another key / value and expect it to be timed-out. Previously we also set a timeout in the first step, but this might sometimes cause false alarm. This diff makes the first two writes run without timeout setting. Test Plan: export ROCKSDB_TESTS=Time make db_test Reviewers: sdong, ljin Reviewed By: ljin Subscribers: leveldb Differential Revision: https://reviews.facebook.net/D19461
-
由 Lei Jin 提交于
Summary: ReadInternalKey() will assign correct value anyway. Initialize it to true to suppress compiler error reported https://github.com/facebook/rocksdb/issues/186 Test Plan: I cannot reproduce it but this is obvious Reviewers: sdong, yhchiang Reviewed By: yhchiang Subscribers: leveldb Differential Revision: https://reviews.facebook.net/D19467
-
由 Yueh-Hsuan Chiang 提交于
Summary: Removed a variable that is only used in assertion check. Test Plan: make release Reviewers: ljin, sdong Reviewed By: sdong Subscribers: leveldb Differential Revision: https://reviews.facebook.net/D19455
-
由 Yueh-Hsuan Chiang 提交于
Summary: This diff allows the I/O stats about Flush and Compaction to be reported in a more accurate way. Instead of measuring the size of a file, it measure I/O cost in per read / write basis. Test Plan: make all check Reviewers: sdong, igor, ljin Reviewed By: ljin Subscribers: leveldb Differential Revision: https://reviews.facebook.net/D19383
-
由 Yueh-Hsuan Chiang 提交于
Summary: Update HISTORY.md to include TimeOut write API and compaction update. Test Plan: n/a Reviewers: ljin, sdong Subscribers: leveldb Differential Revision: https://reviews.facebook.net/D19449
-
由 Yueh-Hsuan Chiang 提交于
Summary: This diff adds timeout_hint_us to WriteOptions. If it's non-zero, then 1) writes associated with this options MAY be aborted when it has been waiting for longer than the specified time. If an abortion happens, associated writes will return Status::TimeOut. 2) the stall time of the associated write caused by flush or compaction will be limited by timeout_hint_us. The default value of timeout_hint_us is 0 (i.e., OFF.) The statistics of timeout writes will be recorded in WRITE_TIMEDOUT. Test Plan: export ROCKSDB_TESTS=WriteTimeoutAndDelayTest make db_test ./db_test Reviewers: igor, ljin, haobo, sdong Reviewed By: sdong Subscribers: dhruba, leveldb Differential Revision: https://reviews.facebook.net/D18837
-
由 Igor Canadi 提交于
-
由 Yueh-Hsuan Chiang 提交于
Summary: Add TimedWait() API to CondVar, which will be used in the future to support TimedOut Write API and Rate limiter. Test Plan: make db_test -j32 Reviewers: sdong, ljin Reviewed By: ljin Subscribers: leveldb Differential Revision: https://reviews.facebook.net/D19431
-
- 03 7月, 2014 2 次提交
-
-
由 sdong 提交于
Summary: In this patch, we allow RocksDB to support multiple DB paths internally. No user interface is supported yet so this patch is silent to users. Test Plan: make all check Reviewers: igor, haobo, ljin, yhchiang Reviewed By: yhchiang Subscribers: dhruba, leveldb Differential Revision: https://reviews.facebook.net/D18921
-
由 Igor Canadi 提交于
Summary: Before this diff, we're deciding enable_compression in CompactionPicker and then we're deciding final compression type in DBImpl. This is kind of confusing. After the diff, the final compression type will be decided in CompactionPicker. The reason for this is that I want CompactFiles() to specify output compression type, so that people can mix and match compression styles in their compaction algorithms. This diff makes it much easier to do that. Test Plan: make check Reviewers: dhruba, haobo, sdong, yhchiang, ljin Reviewed By: ljin Subscribers: leveldb Differential Revision: https://reviews.facebook.net/D19137
-
- 02 7月, 2014 6 次提交
-
-
由 Igor Canadi 提交于
Summary: https://www.facebook.com/groups/rocksdb.dev/permalink/590438347721350/ Test Plan: compiles Reviewers: sdong, ljin, yhchiang Reviewed By: yhchiang Subscribers: leveldb Differential Revision: https://reviews.facebook.net/D19197
-
由 sdong 提交于
Commit 6634844d by sdong Two small fixes in db_test Summary: Two fixes: (1) WalDir to pick a directory under TmpDir to allow two tests running in parallel without impacting each other (2) kBlockBasedTableWithWholeKeyHashIndex is disabled by mistake (I assume). Enable it. Test Plan: ./db_test Reviewers: yhchiang, ljin Reviewed By: ljin Subscribers: nkg-, igor, dhruba, haobo, leveldb Differential Revision: https://reviews.facebook.net/D19389
-
由 sdong 提交于
Summary: In this patch, we enhance HashLinkList memtable to reduce performance outliers when a bucket contains too many entries. We switch to skip list for this case to enable binary search. Add threshold_use_skiplist parameter to determine when a bucket needs to switch to skip list. The new data structure is documented in comments in the codes. Test Plan: make all check set threshold_use_skiplist in several tests Reviewers: yhchiang, haobo, ljin Reviewed By: yhchiang, ljin Subscribers: nkg-, xjin, dhruba, yhchiang, leveldb Differential Revision: https://reviews.facebook.net/D19299
-
由 sdong 提交于
Summary: Two fixes: (1) WalDir to pick a directory under TmpDir to allow two tests running in parallel without impacting each other (2) kBlockBasedTableWithWholeKeyHashIndex is disabled by mistake (I assume). Enable it. Test Plan: ./db_test Reviewers: yhchiang, ljin Reviewed By: ljin Subscribers: nkg-, igor, dhruba, haobo, leveldb Differential Revision: https://reviews.facebook.net/D19389
-
由 Feng Zhu 提交于
Summary: Now that the arena is used to allocate space for hashskiplist's bucket. The bucket size need to be set small enough to avoid "should_flush_" failure in memtable's assertion. Test Plan: make all check Reviewers: sdong Reviewed By: sdong Subscribers: igor Differential Revision: https://reviews.facebook.net/D19371
-
- 01 7月, 2014 3 次提交
-
-
由 Igor Canadi 提交于
-
由 Igor Canadi 提交于
Summary: files_by_size_ is sorted by time in case of universal compaction. However, Version::files_ is also sorted by time. So no need for files_by_size_ Test Plan: 1) make check with the change 2) make check with `assert(last_index == c->input_version_->files_[level].size() - 1);` in compaction picker Reviewers: dhruba, haobo, yhchiang, sdong, ljin Reviewed By: ljin Subscribers: leveldb Differential Revision: https://reviews.facebook.net/D19125
-
由 Feng Zhu 提交于
Summary: Bloomfilter and hashskiplist's buckets_ allocated by memtable's arena DynamicBloom: pass arena via constructor, allocate space in SetTotalBits HashSkipListRep: allocate space of buckets_ using arena. do not delete it in deconstructor because arena would take care of it. Several test files are changed. Test Plan: make all check Reviewers: ljin, haobo, yhchiang, sdong Reviewed By: sdong Subscribers: igor, dhruba Differential Revision: https://reviews.facebook.net/D19335
-
- 30 6月, 2014 2 次提交
-
-
由 Yueh-Hsuan Chiang 提交于
Summary: Correct the library loading for zlib in RocksJava: zlib should be loaded by loadLibrary("z") instead of loadLibrary("zlib"). Test Plan: make rocksdbjava cd java make db_bench ./jdb_bench.sh --compression_type=zlib Reviewers: sdong, ljin, ankgup87 Reviewed By: ankgup87 Subscribers: leveldb Differential Revision: https://reviews.facebook.net/D19341
-
由 Yueh-Hsuan Chiang 提交于
[Java] Merge pull request #183 --- Add statistics collector utility Summary: 1. Add statistics collector which collects statistics periodically at a period specified by caller. 2. Added unit-test for StatsCollector make rocksdbjava make test Reviewers: haobo, yhchiang, sdong, dhruba, rsumbaly, zzbennett, swapnilghike Reviewed By: yhchiang CC: leveldb Differential Revision: https://reviews.facebook.net/19215
-
- 29 6月, 2014 2 次提交
-
-
由 Ankit Gupta 提交于
-
-
- 28 6月, 2014 1 次提交
-
-
由 sdong 提交于
Summary: Code cleaning up, since we are already using __STDC_FORMAT_MACROS in printing uint64_t, change other places. Only logging is changed. Test Plan: make all check Reviewers: ljin Reviewed By: ljin Subscribers: dhruba, yhchiang, haobo, leveldb Differential Revision: https://reviews.facebook.net/D19113
-
- 27 6月, 2014 1 次提交
-
-
由 Stanislau Hlebik 提交于
Summary: Task 4580155. Some conditions in DBImpl::MakeRoomForWrite can be cached in ColumnFamilyData, because theirs value can be changed only during compaction, adding new memtable and/or add recalculation of compaction score. These conditions are: cfd->imm()->size() == cfd->options()->max_write_buffer_number - 1 cfd->current()->NumLevelFiles(0) >= cfd->options()->level0_stop_writes_trigger cfd->options()->soft_rate_limit > 0.0 && (score = cfd->current()->MaxCompactionScore()) > cfd->options()->soft_rate_limit cfd->options()->hard_rate_limit > 1.0 && (score = cfd->current()->MaxCompactionScore()) > cfd->options()->hard_rate_limit P.S. As it's my first diff, Siying suggested to add everybody as a reviewers for this diff. Sorry, if I forgot someone or add someone by mistake. Test Plan: make all check Reviewers: haobo, xjin, dhruba, yhchiang, zagfox, ljin, sdong Reviewed By: sdong Subscribers: leveldb Differential Revision: https://reviews.facebook.net/D19311
-
- 26 6月, 2014 3 次提交
-
-
由 Yueh-Hsuan Chiang 提交于
Summary: Fixed the following warning: util/options.cc: In constructor ‘rocksdb::ColumnFamilyOptions::ColumnFamilyOptions(const rocksdb::Options&)’: util/options.cc:157:58: error: comparison between signed and unsigned integer expressions [-Werror=sign-compare] if (max_bytes_for_level_multiplier_additional.size() < num_levels) { ^ Test Plan: make all check Reviewers: haobo, sdong, ljin Reviewed By: ljin Subscribers: leveldb Differential Revision: https://reviews.facebook.net/D19293
-
由 sdong 提交于
Summary: It seems to me that when ever function MemTableRep::GetIterator(const Slice& slice) is used, we can use MemTableRep::GetDynamicPrefixIterator() instead. Just delete it to simplify the codes. Test Plan: make all check Reviewers: yhchiang, ljin Reviewed By: ljin Subscribers: xjin, dhruba, haobo, leveldb Differential Revision: https://reviews.facebook.net/D19281
-
由 Yueh-Hsuan Chiang 提交于
Summary: Currently, when num_levels has been changed to > 7, internally it will not resize max_bytes_for_level_multiplier_additional. As a result, max_bytes_for_level_multiplier_additional.size() will be smaller than num_levels, which causes heap-buffer-overflow. Test Plan: make all check Reviewers: haobo, sdong, ljin Reviewed By: ljin Subscribers: leveldb Differential Revision: https://reviews.facebook.net/D19275
-
- 25 6月, 2014 2 次提交
-
-
由 Yueh-Hsuan Chiang 提交于
Summary: Move stats related member variables of FileMetaData to the bottom to improve cache locality of normal DB operations. Test Plan: make Reviewers: haobo, ljin, sdong Reviewed By: sdong Subscribers: leveldb Differential Revision: https://reviews.facebook.net/D19287
-
由 Yueh-Hsuan Chiang 提交于
Summary: This diff allows compaction to reclaim storage more effectively. In the current design, compactions are mainly triggered based on the file sizes. However, since deletion entries does not have value, files which have many deletion entries are less likely to be compacted. As a result, it may took a while to make deletion entries to be compacted. This diff address issue by compensating the size of deletion entries during compaction process: the size of each deletion entry in the compaction process is augmented by 2x average value size. The diff applies to both leveled and universal compacitons. Test Plan: develop CompactionDeletionTrigger make db_test ./db_test Reviewers: haobo, igor, ljin, sdong Reviewed By: sdong Subscribers: leveldb Differential Revision: https://reviews.facebook.net/D19029
-