- 23 10月, 2018 1 次提交
-
-
由 Siying Dong 提交于
Summary: Level compaction usually performs poorly when the writes so heavy that the level targets can't be guaranteed. With this improvement, we improve level_compaction_dynamic_level_bytes = true so that in the write heavy cases, the level multiplier can be slightly adjusted based on the size of L0. We keep the behavior the same if number of L0 files is under 2X compaction trigger and the total size is less than options.max_bytes_for_level_base, so that unless write is so heavy that compaction cannot keep up, the behavior doesn't change. Pull Request resolved: https://github.com/facebook/rocksdb/pull/4338 Differential Revision: D9636782 Pulled By: siying fbshipit-source-id: e27fc17a7c29c84b00064cc17536a01dacef7595
-
- 22 10月, 2018 1 次提交
-
-
由 Yi Wu 提交于
Summary: When `MockTimeEnv` is used in test to mock time methods, we cannot use `CondVar::TimedWait` because it is using real time, not the mocked time for wait timeout. On Mac the method can return immediately without awaking other waiting threads, if the real time is larger than `wait_until` (which is a mocked time). When that happen, the `wait()` method will fall into an infinite loop. Pull Request resolved: https://github.com/facebook/rocksdb/pull/4560 Differential Revision: D10472851 Pulled By: yiwu-arbug fbshipit-source-id: 898902546ace7db7ac509337dd8677a527209d19
-
- 20 10月, 2018 3 次提交
-
-
由 Simon Grätzer 提交于
Summary: On MacOS with clang the compilation of _tools/db_bench_tool.cc_ always fails because the format used in a `fprintf` call has the wrong type. This PR should hopefully fix this issue ``` tools/db_bench_tool.cc:4233:61: error: format specifies type 'unsigned long long' but the argument has type 'size_t' (aka 'unsigned long') ``` Pull Request resolved: https://github.com/facebook/rocksdb/pull/4533 Differential Revision: D10471657 Pulled By: maysamyabandeh fbshipit-source-id: f20f5f3756d3571b586c895c845d0d4d1e34a398
-
由 Siying Dong 提交于
Summary: WriteBatchWithIndex's SeekForPrev() has a bug that we internally place the position just before the seek key rather than after. This makes the iterator to miss the result that is the same as the seek key. Fix it by position the iterator equal or smaller. Pull Request resolved: https://github.com/facebook/rocksdb/pull/4559 Differential Revision: D10468534 Pulled By: siying fbshipit-source-id: 2fb371ae809c561b60a1c11cef71e1c66fea1f19
-
由 Yanqin Jin 提交于
Summary: Current `log::Reader` does not perform retry after encountering `EOF`. In the future, we need the log reader to be able to retry tailing the log even after `EOF`. Current implementation is simple. It does not provide more advanced retry policies. Will address this in the future. Pull Request resolved: https://github.com/facebook/rocksdb/pull/4394 Differential Revision: D9926508 Pulled By: riversand963 fbshipit-source-id: d86d145792a41bd64a72f642a2a08c7b7b5201e1
-
- 19 10月, 2018 4 次提交
-
-
由 Abhishek Madan 提交于
Summary: The new flag allows tombstones to be generated after enough keys have been written to the database, which makes it easier to ensure that tombstones cover a lot of keys. Pull Request resolved: https://github.com/facebook/rocksdb/pull/4538 Differential Revision: D10455685 Pulled By: abhimadan fbshipit-source-id: f25d5421745a353c830dea12b79784e852056551
-
由 Maysam Yabandeh 提交于
Summary: We have already disabled it on Travis since it has been too flaky. The same problem arises in Appveyor as well. Pull Request resolved: https://github.com/facebook/rocksdb/pull/4536 Differential Revision: D10452240 Pulled By: maysamyabandeh fbshipit-source-id: 728f4ecddf780097159dc0a0737d460eb5ce4f09
-
由 Philip Jameson 提交于
Reviewed By: andrewjcg Differential Revision: D10409082 fbshipit-source-id: a1432270f79c2baf2e52e3351b5f481c7398c58d
-
由 Huachao Huang 提交于
Summary: When HAVE_SSE42 is true, it should always add "-msse4.2 -mpclmul" no matter if FORCE_SSE42 is true or not. Pull Request resolved: https://github.com/facebook/rocksdb/pull/4490 Differential Revision: D10384256 Pulled By: yiwu-arbug fbshipit-source-id: c82bc988f017981a0f84ddc136f36e2366c3ea8a
-
- 18 10月, 2018 3 次提交
-
-
由 Jigar Bhati 提交于
Summary: Allow rocks java to explicitly create WriteBufferManager by plumbing it to the native code through JNI. Pull Request resolved: https://github.com/facebook/rocksdb/pull/4492 Differential Revision: D10428506 Pulled By: sagar0 fbshipit-source-id: cd9dd8c2ef745a0303416b44e2080547bdcca1fd
-
由 Abhishek Madan 提交于
Summary: When there are no range deletions, flush and compaction perform a binary search on an effectively empty map every time they call ShouldDelete. This PR lazily initializes each stripe map entry so that the binary search can be elided in these cases. After this PR, the total amount of time spent in compactions is 52.541331s, and the total amount of time spent in flush is 5.532608s, the former of which is a significant improvement from the results after #4495. Pull Request resolved: https://github.com/facebook/rocksdb/pull/4497 Differential Revision: D10428610 Pulled By: abhimadan fbshipit-source-id: 6f7e1ce3698fac3ef86d1197955e6b72e0931a0f
-
由 Zhongyi Xie 提交于
Summary: Current implementation of perf context is level agnostic. Making it hard to do performance evaluation for the LSM tree. This PR adds `PerfContextByLevel` to decompose the counters by level. This will be helpful when analyzing point and range query performance as well as tuning bloom filter Also replaced __thread with thread_local keyword for perf_context Pull Request resolved: https://github.com/facebook/rocksdb/pull/4226 Differential Revision: D10369509 Pulled By: miasantreble fbshipit-source-id: f1ced4e0de5fcebdb7f9cff36164516bc6382d82
-
- 16 10月, 2018 6 次提交
-
-
由 anand1976 提交于
Summary: When a CompactRange() call for a level is truncated before the end key is reached, because it exceeds max_compaction_bytes, we need to properly set the compaction_end parameter to indicate the stop key. The next CompactRange will use that as the begin key. We set it to the smallest key of the next file in the level after expanding inputs to get a clean cut. Previously, we were setting it before expanding inputs. So we could end up recompacting some files. In a pathological case, where a single key has many entries spanning all the files in the level (possibly due to merge operands without a partial merge operator, thus resulting in compaction output identical to the input), this would result in an endless loop over the same set of files. Pull Request resolved: https://github.com/facebook/rocksdb/pull/4496 Differential Revision: D10395026 Pulled By: anand1976 fbshipit-source-id: f0c2f89fee29b4b3be53b6467b53abba8e9146a9
-
由 Yanqin Jin 提交于
Summary: Leverage existing `FlushJob` to implement atomic flush of multiple column families. This PR depends on other PRs and is a subset of #3752 . This PR itself is not sufficient in fulfilling atomic flush. Pull Request resolved: https://github.com/facebook/rocksdb/pull/4262 Differential Revision: D9283109 Pulled By: riversand963 fbshipit-source-id: 65401f913e4160b0a61c0be6cd02adc15dad28ed
-
由 Andrew Kryczka 提交于
Summary: `CompactionIterator::snapshots_` is ordered by ascending seqnum, just like `DBImpl`'s linked list of snapshots from which it was copied. This PR exploits this ordering to make `findEarliestVisibleSnapshot` do binary search rather than linear scan. This can make flush/compaction significantly faster when many snapshots exist since that function is called on every single key. Pull Request resolved: https://github.com/facebook/rocksdb/pull/4495 Differential Revision: D10386470 Pulled By: ajkr fbshipit-source-id: 29734991631227b6b7b677e156ac567690118a8b
-
由 Maysam Yabandeh 提交于
Summary: WritePrepared is declared production ready (overdue update) and the benchmark results are also reported. Pull Request resolved: https://github.com/facebook/rocksdb/pull/4494 Differential Revision: D10385336 Pulled By: maysamyabandeh fbshipit-source-id: 662672ddfa286aa46af544f505b4d4b7a882d408
-
由 Yanqin Jin 提交于
Summary: Using const string& can avoid one extra string copy. This PR addresses a recent comment made by siying on #3933. Pull Request resolved: https://github.com/facebook/rocksdb/pull/4491 Differential Revision: D10381211 Pulled By: riversand963 fbshipit-source-id: 27fc2d65d84bc7cd07833c77cdc47f06dcfaeb31
-
由 Yi Wu 提交于
Summary: Set the macro if default allocator is jemalloc. It doesn't handle the case when allocator is specified, e.g. ``` cpp_binary( name="xxx" allocator="jemalloc", # or "malloc" or something else deps=["//rocksdb:rocksdb"], ) ``` Pull Request resolved: https://github.com/facebook/rocksdb/pull/4489 Differential Revision: D10363683 Pulled By: yiwu-arbug fbshipit-source-id: 5da490336a8e78e0feb0900c29e8036e7ec6f12b
-
- 13 10月, 2018 4 次提交
-
-
由 Yanqin Jin 提交于
Summary: We would like to collect file-system-level statistics including file name, offset, length, return code, latency, etc., which requires to add callbacks to intercept file IO function calls when RocksDB is running. To collect file-system-level statistics, users can inherit the class `EventListener`, as in `TestFileOperationListener `. Note that `TestFileOperationListener::ShouldBeNotifiedOnFileIO()` returns true. Pull Request resolved: https://github.com/facebook/rocksdb/pull/3933 Differential Revision: D10219571 Pulled By: riversand963 fbshipit-source-id: 7acc577a2d31097766a27adb6f78eaf8b1e8ff15
-
由 John Calcote 提交于
Summary: Closes https://github.com/facebook/rocksdb/issues/4447 Pull Request resolved: https://github.com/facebook/rocksdb/pull/4448 Differential Revision: D10351852 Pulled By: ajkr fbshipit-source-id: 18287b5190ae0b8153ce425da9a0bdfe1af88c34
-
由 Yi Wu 提交于
Summary: The "je_" prefix of jemalloc APIs presents only when the macro `JEMALLOC_NO_RENAME` from jemalloc.h presents. With the patch I'm also adding -DROCKSDB_JEMALLOC flag in buck TARGETS. Pull Request resolved: https://github.com/facebook/rocksdb/pull/4488 Differential Revision: D10355971 Pulled By: yiwu-arbug fbshipit-source-id: 03a2d69790a44ac89219c7525763fa937a63d95a
-
由 Chinmay Kamat 提交于
Summary: This commit adds code to acquire lock on the DB LOCK file before starting the repair process. This will prevent multiple processes from performing repair on the same DB simultaneously. Fixes repair_test to work with this change. Pull Request resolved: https://github.com/facebook/rocksdb/pull/4435 Differential Revision: D10361499 Pulled By: riversand963 fbshipit-source-id: 3c512c48b7193d383b2279ccecabdb660ac1cf22
-
- 12 10月, 2018 3 次提交
-
-
由 Wilfried Goesgens 提交于
Summary: The default behaviour of rocksdb is to use the `*A(` windows API functions. These accept filenames in the currently configured system encoding, be it Latin 1, utf8 or whatever. If the Application intends to completely work with utf8 strings internally, converting these to that codepage properly isn't even always possible. Thus this patch adds a switch to use the `*W(` functions, which accept UTF-16 filenames, and uses C++11 features to translate the UTF8 containing std::string to an UTF16 containing std::wstring. This feature is a compile time options, that can be enabled by setting `WITH_WINDOWS_UTF8_FILENAMES` to true. Pull Request resolved: https://github.com/facebook/rocksdb/pull/4469 Differential Revision: D10356011 Pulled By: yiwu-arbug fbshipit-source-id: 27b6ae9171f209085894cdf80069e8a896642044
-
由 Abhishek Madan 提交于
Summary: Using `./range_del_aggregator_bench --use_collapsed=false --num_range_tombstones=5000 --num_runs=1000`, here are the results before and after this change: Before: ``` ========================= Results: ========================= AddTombstones: 1822.61 us ShouldDelete (first): 94.5286 us ``` After: ``` ========================= Results: ========================= AddTombstones: 199.26 us ShouldDelete (first): 38.9344 us ``` Pull Request resolved: https://github.com/facebook/rocksdb/pull/4487 Differential Revision: D10347288 Pulled By: abhimadan fbshipit-source-id: d44efe3a166d583acfdc3ec1199e0892f34dbfb7
-
由 zpalmtree 提交于
Summary: Closes https://github.com/facebook/rocksdb/issues/4462 I'm not sure if you'll be happy with `std::random_device{}`, perhaps you would want to use your rand instance instead. I didn't test to see if your rand instance supports the requirements that `std::shuffle` takes. Pull Request resolved: https://github.com/facebook/rocksdb/pull/4482 Differential Revision: D10325133 Pulled By: yiwu-arbug fbshipit-source-id: 47b7adaf4bb2b8d64cf090ea6b1b48ef53180581
-
- 11 10月, 2018 9 次提交
-
-
由 Young Tack Jin 提交于
Summary: "Write (GB)" of $9 rather than "Rnp1 (GB)" of $8 Pull Request resolved: https://github.com/facebook/rocksdb/pull/4442 Differential Revision: D10318193 Pulled By: yiwu-arbug fbshipit-source-id: 03a7ef1938d9332e06fb3fd8490ca212f61fac6b
-
由 UncP 提交于
Summary: in `WriteThread::LaunchParallelMemTableWriters`, there is ` write_group->running.store(write_group->size); ` https://github.com/facebook/rocksdb/blob/master/db/write_thread.cc#L510 Pull Request resolved: https://github.com/facebook/rocksdb/pull/4450 Differential Revision: D10201900 Pulled By: yiwu-arbug fbshipit-source-id: 96c8fbbba5aff7ba8a6ceb3117a2bd7cc9b2f34b
-
由 Simon Grätzer 提交于
Summary: Wrong I overwrite `WriteBatch::Handler::Continue` to return _false_ at some point, I always get the `Status::Corruption` error. I don't think this check is used correctly here: The counter in `found` cannot reflect all entries in the WriteBatch when we exit the loop early. Pull Request resolved: https://github.com/facebook/rocksdb/pull/4478 Differential Revision: D10317416 Pulled By: yiwu-arbug fbshipit-source-id: cccae3382805035f9b3239b66682b5fcbba6bb61
-
由 Andrew Kryczka 提交于
Summary: Even during `DBIter::Prev()`, there is a case where we need to use `RangeDelPositioningMode::kForwardTraversal`. In particular, when we hit too many internal keys for a single user key, we use seek to find the newest internal key. If it's a merge operand, we then scan forwards, collecting the merge operands. This forward scan should be using `RangeDelPositioningMode::kForwardTraversal`. Pull Request resolved: https://github.com/facebook/rocksdb/pull/4481 Differential Revision: D10319507 Pulled By: ajkr fbshipit-source-id: b5ce7352461f3a7696b28a5136ae0076f2bde51f
-
由 Peter Pei 提交于
Summary: fix #4288 Add `OnCompactionBegin` support to `rocksdb::EventListener`. Currently, we only have these three callbacks: - OnFlushBegin - OnFlushCompleted - OnCompactionCompleted As paolococchi requested in #4288 , and ajkr agreed, we should also support `OnCompactionBegin`. This PR is a try to implement the support of `OnCompactionBegin`. Hope it is useful to you. Pull Request resolved: https://github.com/facebook/rocksdb/pull/4431 Differential Revision: D10055515 Pulled By: yiwu-arbug fbshipit-source-id: 39c0f95f8e9ff1c7ca3a10787502a17f258d2334
-
由 Yi Wu 提交于
Summary: Fix security warning from github: https://nvd.nist.gov/vuln/detail/CVE-2018-17567 Pull Request resolved: https://github.com/facebook/rocksdb/pull/4480 Reviewed By: gfosco Differential Revision: D10316443 Pulled By: yiwu-arbug fbshipit-source-id: 34555c6f5774d779734b664b9baa70bd4209175f
-
由 Andrew Kryczka 提交于
Summary: I wrote a couple tests using the public API to expose/prevent the bugs we talked. In particular, - When files have overlapping endpoints and a range tombstone spans them, ensure the largest key does not reappear to readers. This was happening due to a bug that skipped writing range tombstones to an output file when their begin key exactly matched the file's largest key. - When a tombstone spans multiple atomic compaction units, ensure newer keys do not disappear by being compacted beneath it. This happened due to a range tombstone appearing untruncated to readers when it spanned files with overlapping endpoints, even if it extended into files without overlapping endpoints (i.e., different atomic compaction units). Pull Request resolved: https://github.com/facebook/rocksdb/pull/4476 Differential Revision: D10286001 Pulled By: ajkr fbshipit-source-id: bb5ca51d0f90812fb37bfe1d01aec93f7eda55aa
-
由 Abhishek Madan 提交于
Summary: Follow-up to https://github.com/facebook/rocksdb/pull/4432. Pull Request resolved: https://github.com/facebook/rocksdb/pull/4479 Differential Revision: D10304151 Pulled By: abhimadan fbshipit-source-id: 3608b95c324702ca26791f95cb26dae1d49efbe7
-
由 Zhichao Cao 提交于
Summary: If the query types being analyzed do not appear in the trace, the current trace_analyzer will use 0 as the begin time, which create the time duration from 1970/01/01 to the now time. It will waste huge memory. Fixed by adding the trace_create_time to limit the duration. Pull Request resolved: https://github.com/facebook/rocksdb/pull/4473 Differential Revision: D10246204 Pulled By: zhichao-cao fbshipit-source-id: 42850b080b2e62f586fe73afd7737c2246d1a8c8
-
- 10 10月, 2018 6 次提交
-
-
由 Anand Ananthabhotla 提交于
Summary: There is a bug when the write queue leader is blocked on a write delay/stop, and the queue has writers with WriteOptions::no_slowdown set to true. They are not woken up until the write stall is cleared. The fix introduces a dummy writer inserted at the tail to indicate a write stall and prevent further inserts into the queue, and a condition variable that writers who can tolerate slowdown wait on before adding themselves to the queue. The leader calls WriteThread::BeginWriteStall() to add the dummy writer and then walk the queue to fail any writers with no_slowdown set. Once the stall clears, the leader calls WriteThread::EndWriteStall() to remove the dummy writer and signal the condition variable. Pull Request resolved: https://github.com/facebook/rocksdb/pull/4475 Differential Revision: D10285827 Pulled By: anand1976 fbshipit-source-id: 747465e5e7f07a829b1fb0bc1afcd7b93f4ab1a9
-
由 jsteemann 提交于
Summary: this avoids a few copies of std::string and other structs in the context of range-based for loops. instead of copying the values for each iteration, use a const reference to avoid copying. Pull Request resolved: https://github.com/facebook/rocksdb/pull/4459 Differential Revision: D10282045 Pulled By: sagar0 fbshipit-source-id: 5012e910dca279abd2be847e1fb432d96274edfb
-
由 moozzyk 提交于
Summary: Fixes: #4401 Pull Request resolved: https://github.com/facebook/rocksdb/pull/4444 Differential Revision: D10282120 Pulled By: sagar0 fbshipit-source-id: d9ddcc1b132208ae7f806fa2106add6fec1baa11
-
由 jsteemann 提交于
Summary: Fixes a typo in error messages returned by Iterator::GetProperty(...) Pull Request resolved: https://github.com/facebook/rocksdb/pull/4457 Differential Revision: D10281965 Pulled By: sagar0 fbshipit-source-id: 1cd3c665f467ef06cdfd9f482692e6f8568f3d22
-
由 Jiri Appl 提交于
Summary: The original logic was assuming that the only architectures that the code would build for on Windows were x86 and x64. This change will enable building for arm32 on Windows as well. Pull Request resolved: https://github.com/facebook/rocksdb/pull/4349 Differential Revision: D10280887 Pulled By: sagar0 fbshipit-source-id: 9ca0bede25505d22e13acf916d38aeeaaf5d981a
-
由 Abhishek Madan 提交于
Summary: To more accurately truncate range tombstones at SST boundaries, we now represent them in RangeDelAggregator using InternalKeys, which are end-key-exclusive as they were before this change. During compaction, "atomic compaction unit boundaries" (the range of keys contained in neighbouring and overlaping SSTs) are propagated down to RangeDelAggregator to truncate range tombstones at those boundariies instead. See https://github.com/facebook/rocksdb/pull/4432#discussion_r221072219 and https://github.com/facebook/rocksdb/pull/4432#discussion_r221138683 for motivating examples. Pull Request resolved: https://github.com/facebook/rocksdb/pull/4432 Differential Revision: D10263952 Pulled By: abhimadan fbshipit-source-id: 2fe85ff8a02b3a6a2de2edfe708012797a7bd579
-