...
 
Commits (2)
    https://gitcode.net/kvdb/rocksdb/-/commit/6fd663a22f199656e3cec051c44d60672e87650e Avoid shifting component too large error in FileTtlBooster (#11673) 2023-08-06T17:44:29-07:00 Changyu Bi changyubi@meta.com Summary: When `num_levels` &gt; 65, we may be shifting more than 63 bits in FileTtlBooster. This can give errors like: `runtime error: shift exponent 98 is too large for 64-bit type 'uint64_t' (aka 'unsigned long')`. This PR makes a quick fix for this issue by taking a min in the shifting component. This issue should be rare since it requires a user using a large `num_levels`. I'll follow up with a more complex fix if needed. Pull Request resolved: <a href="https://github.com/facebook/rocksdb/pull/11673" rel="nofollow noreferrer noopener" target="_blank">https://github.com/facebook/rocksdb/pull/11673</a> Test Plan: * Add a unit test that produce the above error before this PR. Need to compile it with ubsan: `COMPILE_WITH_UBSAN=1 OPT="-fsanitize-blacklist=.circleci/ubsan_suppression_list.txt" ROCKSDB_DISABLE_ALIGNED_NEW=1 USE_CLANG=1 make V=1 -j32 compaction_picker_test` Reviewed By: hx235 Differential Revision: D48074386 Pulled By: cbi42 fbshipit-source-id: 25e59df7e93f20e0793cffb941de70ac815d9392 https://gitcode.net/kvdb/rocksdb/-/commit/89a3958bccdc9af519af8eda9270f4b51d1257b6 include last bug fix into 8.5.0 2023-08-06T17:47:41-07:00 Andrew Kryczka andrew.kryczka2@gmail.com
......@@ -15,6 +15,7 @@
### Bug Fixes
* Fix use_after_free bug in async_io MultiReads when underlying FS enabled kFSBuffer. kFSBuffer is when underlying FS pass their own buffer instead of using RocksDB scratch in FSReadRequest. Right now it's an experimental feature.
* Fix a bug in FileTTLBooster that can cause users with a large number of levels (more than 65) to see errors like "runtime error: shift exponent .. is too large.." (#11673).
## 8.4.0 (06/26/2023)
### New Features
......
......@@ -1968,6 +1968,15 @@ TEST_F(CompactionPickerTest, OverlappingUserKeys11) {
ASSERT_EQ(7U, compaction->input(1, 0)->fd.GetNumber());
}
TEST_F(CompactionPickerTest, FileTtlBoosterLargeNumLevels) {
const uint64_t kCurrentTime = 1000000;
FileTtlBooster booster(kCurrentTime, /*ttl=*/2048,
/*num_non_empty_levels=*/100, /*level=*/1);
FileMetaData meta;
meta.oldest_ancester_time = kCurrentTime - 1023;
ASSERT_EQ(1, booster.GetBoostScore(&meta));
}
TEST_F(CompactionPickerTest, FileTtlBooster) {
// Set TTL to 2048
// TTL boosting for all levels starts at 1024,
......
......@@ -53,8 +53,10 @@ class FileTtlBooster {
enabled_ = true;
uint64_t all_boost_start_age = ttl / 2;
uint64_t all_boost_age_range = (ttl / 32) * 31 - all_boost_start_age;
// TODO(cbi): more reasonable algorithm that gives different values
// when num_non_empty_levels - level - 1 > 63.
uint64_t boost_age_range =
all_boost_age_range >> (num_non_empty_levels - level - 1);
all_boost_age_range >> std::min(63, num_non_empty_levels - level - 1);
boost_age_start_ = all_boost_start_age + boost_age_range;
const uint64_t kBoostRatio = 16;
// prevent 0 value to avoid divide 0 error.
......