- 07 8月, 2023 2 次提交
-
-
由 Andrew Kryczka 提交于
-
由 Changyu Bi 提交于
Summary: When `num_levels` > 65, we may be shifting more than 63 bits in FileTtlBooster. This can give errors like: `runtime error: shift exponent 98 is too large for 64-bit type 'uint64_t' (aka 'unsigned long')`. This PR makes a quick fix for this issue by taking a min in the shifting component. This issue should be rare since it requires a user using a large `num_levels`. I'll follow up with a more complex fix if needed. Pull Request resolved: https://github.com/facebook/rocksdb/pull/11673 Test Plan: * Add a unit test that produce the above error before this PR. Need to compile it with ubsan: `COMPILE_WITH_UBSAN=1 OPT="-fsanitize-blacklist=.circleci/ubsan_suppression_list.txt" ROCKSDB_DISABLE_ALIGNED_NEW=1 USE_CLANG=1 make V=1 -j32 compaction_picker_test` Reviewed By: hx235 Differential Revision: D48074386 Pulled By: cbi42 fbshipit-source-id: 25e59df7e93f20e0793cffb941de70ac815d9392
-
- 28 7月, 2023 2 次提交
-
-
由 Andrew Kryczka 提交于
-
由 akankshamahajan 提交于
Summary: Fix use_after_free bug in async_io MultiReads when underlying FS enabled kFSBuffer. kFSBuffer is when underlying FS pass their own buffer instead of using RocksDB scratch in FSReadRequest Since it's an experimental feature, added a hack for now to fix the bug. Planning to make public API change to remove const from the callback as it doesn't make sense to use const. Pull Request resolved: https://github.com/facebook/rocksdb/pull/11645 Test Plan: tested locally Reviewed By: ltamasi Differential Revision: D47819907 Pulled By: akankshamahajan15 fbshipit-source-id: 1faf5ef795bf27e2b3a60960374d91274931df8d
-
- 26 7月, 2023 1 次提交
-
-
由 Andrew Kryczka 提交于
-
- 22 7月, 2023 2 次提交
-
-
由 zhangyuxiang.ax 提交于
Summary: Plaintable will miss properties. It should have some behavior like blockbasedtable. Here is a unit test for reproduce this bug. ``` #include <gflags/gflags.h> #include "rocksdb/db.h" #include "rocksdb/options.h" #include "rocksdb/table.h" #include "rocksdb/slice_transform.h" #include <iostream> #include <thread> #include <csignal> const std::string kKey = "key"; DEFINE_bool(use_plaintable, true, "use plain table"); DEFINE_string(db_path, "/dev/shm/test_zyx_path", "db_path"); rocksdb::DB* db = nullptr; class NoopTransform : public rocksdb::SliceTransform { public: explicit NoopTransform() { } virtual const char* Name() const override { return "rocksdb.Noop"; } virtual rocksdb::Slice Transform(const rocksdb::Slice& src) const override { return src; } virtual bool InDomain(const rocksdb::Slice& src) const override { return true; } virtual bool InRange(const rocksdb::Slice& dst) const override { return true; } virtual bool SameResultWhenAppended(const rocksdb::Slice& prefix) const override { return false; } }; class TestPropertiesCollector : public ::rocksdb::TablePropertiesCollector { public: explicit TestPropertiesCollector() { } private: ::rocksdb::Status AddUserKey(const ::rocksdb::Slice& key, const ::rocksdb::Slice& value, ::rocksdb::EntryType type, ::rocksdb::SequenceNumber seq, uint64_t file_size) override { count++; return ::rocksdb::Status::OK(); } ::rocksdb::Status Finish(::rocksdb::UserCollectedProperties* properties) override { properties->insert({kKey, std::to_string(count)}); return ::rocksdb::Status::OK(); } ::rocksdb::UserCollectedProperties GetReadableProperties() const override { ::rocksdb::UserCollectedProperties properties; properties.insert({kKey, std::to_string(count)}); return properties; } const char* Name() const override { return "TestPropertiesCollector"; } int count = 0; }; class TestTablePropertiesCollectorFactory : public ::rocksdb::TablePropertiesCollectorFactory { public: explicit TestTablePropertiesCollectorFactory() { } private: ::rocksdb::TablePropertiesCollector* CreateTablePropertiesCollector( ::rocksdb::TablePropertiesCollectorFactory::Context context) override { return new TestPropertiesCollector(); } const char* Name() const override { return "test.TablePropertiesCollectorFactory"; } }; class TestFlushListener : rocksdb::EventListener { public: const char* Name() const override { return "TestFlushListener"; } void OnFlushCompleted(rocksdb::DB* /*db*/, const rocksdb::FlushJobInfo& flush_job_info) override { if (flush_job_info.table_properties.user_collected_properties.find(kKey) == flush_job_info.table_properties.user_collected_properties.end()) { std::cerr << "OnFlushCompleted: properties not found" << std::endl; return; } std::cerr << "OnFlushCompleted: properties found " << flush_job_info.table_properties.user_collected_properties.at(kKey) << std::endl; } explicit TestFlushListener() { } }; int main(int argc, char* argv[]) { gflags::ParseCommandLineFlags(&argc, &argv, true); rocksdb::DBOptions rocksdb_options; std::shared_ptr<rocksdb::EventListener> flush_offset; rocksdb_options.create_if_missing = true; rocksdb_options.create_missing_column_families = true; std::shared_ptr<::rocksdb::TablePropertiesCollectorFactory> properties_collector( new TestTablePropertiesCollectorFactory()); rocksdb::ColumnFamilyOptions cfoptions; cfoptions.table_properties_collector_factories.emplace_back(properties_collector); std::shared_ptr<rocksdb::EventListener> test_cleaner; test_cleaner.reset((rocksdb::EventListener*)new TestFlushListener()); rocksdb_options.listeners.emplace_back(test_cleaner); std::vector<rocksdb::ColumnFamilyDescriptor> cf_desc_; cf_desc_.emplace_back(rocksdb::kDefaultColumnFamilyName, cfoptions); std::vector<rocksdb::ColumnFamilyHandle*> cfhs; cfoptions.prefix_extractor.reset(new NoopTransform()); if (FLAGS_use_plaintable) { cfoptions.table_factory.reset(rocksdb::NewPlainTableFactory()); std::cerr << "use plaintable" << std::endl; } else { cfoptions.table_factory.reset(rocksdb::NewBlockBasedTableFactory()); std::cerr << "use blockbasedtable" << std::endl; } auto s = rocksdb::DB::Open(rocksdb_options, FLAGS_db_path, cf_desc_, &cfhs, &db); if (s.ok()) { rocksdb::WriteOptions wops; wops.disableWAL = true; for (int i = 0; i < 1000000; i++) { auto status = db->Put(wops, std::to_string(i), std::string(1024, '3')); if (!status.ok()) { std::cerr << "write fail " << status.getState() << std::endl; } } } else { std::cerr << "open rocksdb failed" << s.getState() << std::endl; } std::this_thread::sleep_for(std::chrono::seconds(1000)); delete db; } ``` Pull Request resolved: https://github.com/facebook/rocksdb/pull/11267 Reviewed By: jowlyzhang Differential Revision: D47689943 Pulled By: hx235 fbshipit-source-id: 585589cc48f8b26c7dd2323fc7ac4a0c3d4df6bb
-
由 Hui Xiao 提交于
Move prefetching responsibility to page cache for compaction read under non directIO usecase (#11631) Summary: **Context/Summary** As titled. The benefit of doing so is to explicitly call readahead() instead of relying page cache behavior for compaction read when we know that we most likely need readahead as compaction read is sequential read . **Test** Extended the existing UT to cover compaction read case Pull Request resolved: https://github.com/facebook/rocksdb/pull/11631 Reviewed By: ajkr Differential Revision: D47681437 Pulled By: hx235 fbshipit-source-id: 78792f64985c4dc44aa8f2a9c41ab3e8bbc0bc90
-
- 21 7月, 2023 2 次提交
-
-
由 shadowlux 提交于
Summary: Pull Request resolved: https://github.com/facebook/rocksdb/pull/11596 Reviewed By: ajkr Differential Revision: D47635614 Pulled By: jowlyzhang fbshipit-source-id: 651a06049a54d15fd4b4f010bb4b82f53ff9c9d4
-
由 Rémi Calixte 提交于
Summary: Add `rocksdb_transactiondb_get_base_db` and `rocksdb_transactiondb_close_base_db` functions to the C API modeled after `rocksdb_optimistictransactiondb_get_base_db` and `rocksdb_optimistictransactiondb_close_base_db`: https://github.com/facebook/rocksdb/blob/ca50ccc71a1ce89008b4737e74f321b8df8a3b5b/include/rocksdb/c.h#L2711-L2716 With this pair of functions, it is possible to get a `rocksdb_t *` from a `rocksdb_transactiondb_t *`. The main goal is to be able to use the approximate memory usage API, only accessible to the `rocksdb_t *` type: https://github.com/facebook/rocksdb/blob/ca50ccc71a1ce89008b4737e74f321b8df8a3b5b/include/rocksdb/c.h#L2821-L2833 Pull Request resolved: https://github.com/facebook/rocksdb/pull/11562 Reviewed By: ajkr Differential Revision: D47603343 Pulled By: jowlyzhang fbshipit-source-id: c70cf6af5834026e232fe7791634db3a396f7d5e
-
- 20 7月, 2023 7 次提交
-
-
由 ywave 提交于
Summary: Pull Request resolved: https://github.com/facebook/rocksdb/pull/11617 Test Plan: make check Reviewed By: ajkr Differential Revision: D47599209 Pulled By: jowlyzhang fbshipit-source-id: 00e96266c75128875663083a2877d27fd7392eea
-
由 shadowlux 提交于
Summary: Pull Request resolved: https://github.com/facebook/rocksdb/pull/11595 Reviewed By: ajkr Differential Revision: D47600701 Pulled By: jowlyzhang fbshipit-source-id: 22375b51c726b176e4bc502b49cf3343f45f8a0a
-
由 shuzz 提交于
Summary: improvement code by std::move and c++17 Pull Request resolved: https://github.com/facebook/rocksdb/pull/11614 Reviewed By: ajkr Differential Revision: D47599519 Pulled By: jowlyzhang fbshipit-source-id: 6b897876f4e87e94a74c53d8db2a01303d500bff
-
由 zhutao 提交于
Summary: Add path existence check in the script to avoid script running even when db_bench executable does not exist or relative path is not right. Pull Request resolved: https://github.com/facebook/rocksdb/pull/11621 Reviewed By: jowlyzhang Differential Revision: D47552590 Pulled By: ajkr fbshipit-source-id: f09ea069f69e067212b249a22ad755b76bc6063a
-
由 huangmengbin 提交于
Summary: Fixes https://github.com/facebook/rocksdb/issues/11604 Pull Request resolved: https://github.com/facebook/rocksdb/pull/11605 Reviewed By: jowlyzhang Differential Revision: D47459254 Pulled By: ajkr fbshipit-source-id: 4420e443fbf4bd01ddaa2b47285fc4445bf36246
-
由 Dan Wang 提交于
Fix the sync point SanitizeOptions::AfterChangeMaxOpenFiles which is not executed in db_compaction_test (#11583) Summary: In [db_impl_open.cc](https://github.com/facebook/rocksdb/blob/main/db/db_impl/db_impl_open.cc), the sync point `SanitizeOptions::AfterChangeMaxOpenFiles` is used to set `max_open_files` with some specified "**invalid**" value even if it has been sanitized. However, in [db_compaction_test.cc](https://github.com/facebook/rocksdb/blob/main/db/db_compaction_test.cc), `SanitizeOptions::AfterChangeMaxOpenFiles` would not be executed since `SyncPoint::EnableProcessing()` is run after `DBTestBase::Reopen()`. To enable `SanitizeOptions::AfterChangeMaxOpenFiles`, `SyncPoint::EnableProcessing()` should be put ahead of `DBTestBase::Reopen()`. Pull Request resolved: https://github.com/facebook/rocksdb/pull/11583 Test Plan: run unit tests locally as below: ``` make J=1 check [ RUN ] DBCompactionTest.LevelTtlCascadingCompactions [ OK ] DBCompactionTest.LevelTtlCascadingCompactions (85 ms) [ RUN ] DBCompactionTest.LevelPeriodicCompaction [ OK ] DBCompactionTest.LevelPeriodicCompaction (57 ms) ``` Reviewed By: jowlyzhang Differential Revision: D47311827 Pulled By: ajkr fbshipit-source-id: 99165e87a8129e404af06fdf9b4c96eca540fd23
-
由 Muhammad 提交于
Summary: This adds proper support for using rocksdb with FetchContent, without this PR the user must include the following with their own `CMakeLists.txt` file: ```cmake include_directories(./build/_deps/rocksdb-src/include) ``` Pull Request resolved: https://github.com/facebook/rocksdb/pull/11575 Reviewed By: jowlyzhang Differential Revision: D47163520 Pulled By: ajkr fbshipit-source-id: a202dcf435ecc9dd8d51c88f90e98c04814721ca
-
- 19 7月, 2023 4 次提交
-
-
由 Andrew Kryczka 提交于
Summary: An internal user wants to implement a key-aware row cache policy. For that, they need to know the components of the cache key, especially the user key component. With a specialized `RowCache` interface, we will be able to tell them the components so they won't have to make assumptions about our internal key schema. This PR prepares for the specialized `RowCache` interface by updating the migration plan of https://github.com/facebook/rocksdb/issues/11450. I added a release note for the removed APIs and didn't mention the added ones for now. Pull Request resolved: https://github.com/facebook/rocksdb/pull/11620 Reviewed By: pdillinger Differential Revision: D47536962 Pulled By: ajkr fbshipit-source-id: bbee0fc4ad67fc699a66b8f2b4ea4544dd003691
-
由 Chad Austin 提交于
Summary: Pull Request resolved: https://github.com/facebook/rocksdb/pull/11624 <queue> must be included to use std::queue. Reviewed By: pdillinger Differential Revision: D47562433 fbshipit-source-id: 7c5b19fd9e411694c782dfc0dff0231d4f92ef24
-
由 Peter Dillinger 提交于
Summary: ... ahead of planned dynamic HCC variant. This changes simplifies some logic while still enabling future code sharing between implementation variants. Detail: For complicated reasons, using a std::function parameter to `ConstApplyToEntriesRange` with a lambda argument does not play nice with templated HandleImpl. An explicit conversion to std::function would be needed for it to compile. Templating the function type is the easy work-around. Also made some functions from https://github.com/facebook/rocksdb/issues/11572 private as recommended Pull Request resolved: https://github.com/facebook/rocksdb/pull/11609 Test Plan: existing tests Reviewed By: jowlyzhang Differential Revision: D47407415 Pulled By: pdillinger fbshipit-source-id: 0f65954db16335999b78fb7d2563ec627624cef0
-
由 Changyu Bi 提交于
Summary: Extend the coverage for option `flush_verify_memtable_count`. The verification code is similar to the ones for regular flush: https://github.com/facebook/rocksdb/blob/c3c84b3397a0eaa6450340ecea3b267c0e6c1f3c/db/flush_job.cc#L956-L965 Pull Request resolved: https://github.com/facebook/rocksdb/pull/11611 Test Plan: existing tests. Reviewed By: ajkr Differential Revision: D47478893 Pulled By: cbi42 fbshipit-source-id: ca580c9dbcd6e91facf2e49210661336a79a248e
-
- 15 7月, 2023 2 次提交
-
-
由 akankshamahajan 提交于
Summary: Remove reallocation of AlignedBuffer in direct_io sync reads in RandomAccessFileReader::Read if buffer passed is already aligned. Pull Request resolved: https://github.com/facebook/rocksdb/pull/11600 Test Plan: Setup: `TEST_TMPDIR=./tmp-db/ ./db_bench -benchmarks=filluniquerandom -disable_auto_compactions=true -target_file_size_base=1048576 -write_buffer_size=1048576 -compression_type=none` Benchmark: `TEST_TMPDIR=./tmp-db/ perf record ./db_bench --cache_size=8388608 --use_existing_db=true --disable_auto_compactions=true --benchmarks=seekrandom --use_direct_reads=true -use_direct_io_for_flush_and_compaction=true -reads=1000 -seek_nexts=1 -max_auto_readahead_size=131072 -initial_auto_readahead_size=16384 -adaptive_readahead=true -num_file_reads_for_auto_readahead=0` Perf profile- Before: ``` 8.73% db_bench libc.so.6 [.] __memmove_evex_unaligned_erms 3.34% db_bench [kernel.vmlinux] [k] filemap_get_read_batch ``` After: ``` 2.50% db_bench [kernel.vmlinux] [k] filemap_get_read_batch 2.29% db_bench libc.so.6 [.] __memmove_evex_unaligned_erms ``` `make crash_test -j `with direct_io enabled completed succesfully locally. Ran few benchmarks with direct_io from seek_nexts varying between 912 to 327680 and different readahead_size parameters and it showed no regression so far. Reviewed By: ajkr Differential Revision: D47478598 Pulled By: akankshamahajan15 fbshipit-source-id: 6a48e21cb34696f5d09c22a6311a3a1cb5f9cf33
-
由 Peter Dillinger 提交于
Summary: Stacked on https://github.com/facebook/rocksdb/issues/11572 * Minimize use of std::function and lambdas to minimize chances of compiler heap-allocating closures (unnecessary stress on allocator). It appears that converting FindSlot to a template enables inlining the lambda parameters, avoiding heap allocations. * Clean up some logic with FindSlot (FIXMEs from https://github.com/facebook/rocksdb/issues/11572) * Fix handling of rare case of probing all slots, with new unit test. (Previously Insert would not roll back displacements in that case, which would kill performance if it were to happen.) * Add an -early_exit option to cache_bench for gathering memory stats before deallocation. Pull Request resolved: https://github.com/facebook/rocksdb/pull/11601 Test Plan: unit test added for probing all slots ## Seeing heap allocations Run `MALLOC_CONF="stats_print:true" ./cache_bench -cache_type=hyper_clock_cache` before https://github.com/facebook/rocksdb/issues/11572 vs. after this change. Before, we see this in the interesting bin statistics: ``` size nrequests ---- --------- 32 578460 64 24340 8192 578460 ``` And after: ``` size nrequests ---- --------- 32 (insignificant) 64 24370 8192 579130 ``` ## Performance test Build with `make USE_CLANG=1 PORTABLE=0 DEBUG_LEVEL=0 -j32 cache_bench` Run `./cache_bench -cache_type=hyper_clock_cache -ops_per_thread=5000000` in before and after configurations, simultaneously: ``` Before: Complete in 33.244 s; Rough parallel ops/sec = 2406442 After: Complete in 32.773 s; Rough parallel ops/sec = 2441019 ``` Reviewed By: jowlyzhang Differential Revision: D47375092 Pulled By: pdillinger fbshipit-source-id: 46f0f57257ddb374290a0a38c651764ea60ba410
-
- 14 7月, 2023 1 次提交
-
-
由 leipeng 提交于
Summary: We observed `CompactionOutputs::UpdateGrandparentBoundaryInfo` consumes much time for `InternalKey::DecodeFrom` and `InternalKey::~InternalKey` in flame graph. This PR omit the InternalKey object in `CompactionOutputs::UpdateGrandparentBoundaryInfo` . ![image](https://github.com/facebook/rocksdb/assets/1574991/661eaeec-2f46-46c6-a6a8-9738d6c191de) Pull Request resolved: https://github.com/facebook/rocksdb/pull/11610 Reviewed By: ajkr Differential Revision: D47426971 Pulled By: cbi42 fbshipit-source-id: f0d3a8186d778294515c0685032f5b395c4d6a62
-
- 13 7月, 2023 1 次提交
-
-
由 Peter Dillinger 提交于
Summary: Separate out some functionality that will be common to both static and dynamic HCC into BaseClockTable. Table::InsertState and GrowIfNeeded will be used by the dynamic HCC so don't make much sense right now. Pull Request resolved: https://github.com/facebook/rocksdb/pull/11572 Test Plan: existing tests. No functional changes intended. Performance test in subsequent PR https://github.com/facebook/rocksdb/issues/11601 Reviewed By: jowlyzhang Differential Revision: D47110496 Pulled By: pdillinger fbshipit-source-id: 379bd433322a42ea28c0043b41ec24956d21e7aa
-
- 11 7月, 2023 3 次提交
-
-
由 Changyu Bi 提交于
Summary: I got the following error message when an SST file is recorded in MANIFEST but is missing from the db folder. It's confusing in two ways: 1. The part about file "./074837.ldb" which RocksDB will attempt to open only after ./074837.sst is not found. 2. The last part about "No such file or directory in file ./MANIFEST-074507" sounds like `074837.ldb` is not found in manifest. ``` ldb --hex --db=. get some_key Failed: Corruption: Corruption: IO error: No such file or directory: While open a file for random read: ./074837.ldb: No such file or directory in file ./MANIFEST-074507 ``` Improving the error message a little bit: Pull Request resolved: https://github.com/facebook/rocksdb/pull/11573 Test Plan: run the same command after this PR ``` Failed: Corruption: Corruption: IO error: No such file or directory: While open a file for random read: ./074837.sst: No such file or directory The file ./MANIFEST-074507 may be corrupted. ``` Reviewed By: ajkr Differential Revision: D47192056 Pulled By: cbi42 fbshipit-source-id: 06863f376cc4455803cffb2250c41399b4c39467
-
由 weedge 提交于
Summary: `PORTABLE=1 USE_SSE=1 USE_PCLMUL=1 WITH_JEMALLOC_FLAG=1 JEMALLOC=1 make static_lib` on MacOS clang --version: Apple clang version 12.0.0 (clang-1200.0.32.29) Target: x86_64-apple-darwin22.4.0 Thread model: posix InstalledDir: /Library/Developer/CommandLineTools/usr/bin compile err like this: util/udt_util.cc:39:39: error: 'value' is unavailable: introduced in macOS 10.14 if (running_ts_sz != recorded_ts_sz.value()) { ^ /Library/Developer/CommandLineTools/usr/bin/../include/c++/v1/optional:944:33: note: 'value' has been explicitly marked unavailable here constexpr value_type const& value() const& ^ util/udt_util.cc:217:62: error: 'value' is unavailable: introduced in macOS 10.14 *new_key = StripTimestampFromUserKey(key, record_ts_sz.value()); ^ /Library/Developer/CommandLineTools/usr/bin/../include/c++/v1/optional:953:27: note: 'value' has been explicitly marked unavailable here constexpr value_type& value() & ^ 2 errors generated. make: *** [util/udt_util.o] Error 1 Pull Request resolved: https://github.com/facebook/rocksdb/pull/11574 Reviewed By: ajkr Differential Revision: D47269519 Pulled By: cbi42 fbshipit-source-id: da49d90cdf00a0af519f91c0cf7d257401eb395f
-
由 Yu Zhang 提交于
Summary: Handle file boundaries `FileMetaData.smallest`, `FileMetaData.largest` for when `persist_user_defined_timestamps` is false: 1) on the manifest write path, the original user-defined timestamps in file boundaries are stripped. This stripping is done during `VersionEdit::Encode` to limit the effect of the stripping to only the persisted version of the file boundaries. 2) on the manifest read path during DB open, a a min timestamp is padded to the file boundaries. Ideally, this padding should happen during `VersionEdit::Decode` so that all in memory file boundaries have a compatible user key format as the running user comparator. However, because the user-defined timestamp size information is not available at that time. This change is added to `VersionEditHandler::OnNonCfOperation`. Pull Request resolved: https://github.com/facebook/rocksdb/pull/11578 Test Plan: ``` make all check ./version_edit_test --gtest_filter="*EncodeDecodeNewFile4HandleFileBoundary*". ./db_with_timestamp_basic_test --gtest_filter="*HandleFileBoundariesTest*" ``` Reviewed By: pdillinger Differential Revision: D47309399 Pulled By: jowlyzhang fbshipit-source-id: 21b4d54d2089a62826b31d779094a39cb2bbbd51
-
- 08 7月, 2023 2 次提交
-
-
由 Yu Zhang 提交于
Summary: Thanks pdillinger for pointing out this test hole. The test `DBWALTestWithTimestamp.Recover` that is intended to test recovery from WAL including user-defined timestamps doesn't achieve its promised coverage. Specifically, after https://github.com/facebook/rocksdb/issues/11557, timestamps will be removed during flush, and RocksDB by default flush memtables during recovery with `avoid_flush_during_recovery` defaults to false. This test didn't fail even if all the timestamps are quickly lost due to the default flush behavior. This PR renamed test `Recover` to `RecoverAndNoFlush`, and updated it to verify timestamps are successfully recovered from WAL with some time-travel reads. `avoid_flush_during_recovery` is set to true to help do this verification. On the other hand, for test `DBWALTestWithTimestamp.RecoverAndFlush`, since flush on reopen is DB's default behavior. Setting the flags `max_write_buffer` and `arena_block_size` are not really the factors that enforces the flush, so these flags are removed. Pull Request resolved: https://github.com/facebook/rocksdb/pull/11577 Test Plan: ./db_wal_test Reviewed By: pdillinger Differential Revision: D47142892 Pulled By: jowlyzhang fbshipit-source-id: 9465e278806faa5885b541b4e32d99e698edef7d
-
由 Changyu Bi 提交于
Summary: https://github.com/facebook/rocksdb/issues/11542 added a parameter to the C API `rocksdb_options_add_compact_on_deletion_collector_factory` which causes some internal builds to fail. External users using this API would also require code change. Making the API backward compatible by restoring the old C API and add the parameter to a new C API `rocksdb_options_add_compact_on_deletion_collector_factory_del_ratio`. Also updated change log for 8.4 and will backport this change to 8.4 branch once landed. Pull Request resolved: https://github.com/facebook/rocksdb/pull/11593 Test Plan: `make c_test && ./c_test` Reviewed By: akankshamahajan15 Differential Revision: D47299555 Pulled By: cbi42 fbshipit-source-id: 517dc093ef4cf02cac2fe4af4f1af13754bbda63
-
- 06 7月, 2023 2 次提交
-
-
由 Changyu Bi 提交于
Summary: both options `ttl` and `periodic_compaction_seconds` have the same meaning for FIFO compaction, which is redundant and can be confusing to use. For example, setting TTL to 0 does not disable TTL: user needs to also set periodic_compaction_seconds to 0. Another example is that dynamically setting `periodic_compaction_seconds` (surprisingly) has no effect on TTL compaction. This is because FIFO compaction picker internally only looks at value of `ttl`. The value of `ttl` is in `SanitizeOptions()` which take into account the value of `periodic_compaction_seconds`, but dynamically setting an option does not invoke this method. This PR clarifies the usage of both options for FIFO compaction: only `ttl` should be used, `periodic_compaction_seconds` will not have any effect on FIFO compaction. Pull Request resolved: https://github.com/facebook/rocksdb/pull/11550 Test Plan: - updated existing unit test `DBOptionsTest.SanitizeFIFOPeriodicCompaction` - checked existing values of both options in feature matrix: https://fburl.com/daiquery/xxd0gs9w. All current uses cases either have `periodic_compaction_seconds = 0` or have `periodic_compaction_seconds > ttl`, so should not cause change of behavior. Reviewed By: ajkr Differential Revision: D46902959 Pulled By: cbi42 fbshipit-source-id: a9ede235b276783b4906aaec443551fa62ceff4c
-
由 Changyu Bi 提交于
Summary: `sst_dump --command=verify` did not set read_options.verify_checksum to true so it was not verifying checksum. Pull Request resolved: https://github.com/facebook/rocksdb/pull/11576 Test Plan: ran the same command on an SST file with bad checksum: ``` sst_dump --command=verify --file=...sst_file_with_bad_block_checksum Before this PR: options.env is 0x6ba048 Process ...sst_file_with_bad_block_checksum Sst file format: block-based The file is ok After this PR: options.env is 0x7f43f6690000 Process ...sst_file_with_bad_block_checksum Sst file format: block-based ... is corrupted: Corruption: block checksum mismatch: stored = 2170109798, computed = 2170097510, type = 4 ... ``` Reviewed By: ajkr Differential Revision: D47136284 Pulled By: cbi42 fbshipit-source-id: 07d68db715c00347145e5b83d649aef2c3f2acd9
-
- 04 7月, 2023 2 次提交
-
-
由 leipeng 提交于
Summary: This should be a benign bug caused by a long lived typo, this PR fix this issue. Pull Request resolved: https://github.com/facebook/rocksdb/pull/11398 Reviewed By: ajkr Differential Revision: D47163379 Pulled By: cbi42 fbshipit-source-id: 531728cae496fd7ac1371bbbd64fc103c3a90dcf
-
由 shadowlux 提交于
Summary: Pull Request resolved: https://github.com/facebook/rocksdb/pull/11580 Reviewed By: ajkr Differential Revision: D47158687 Pulled By: cbi42 fbshipit-source-id: 4841b77eee78ddcf35da6ea33da71861c5f1e773
-
- 30 6月, 2023 1 次提交
-
-
由 Yu Zhang 提交于
Summary: Logically strip the user-defined timestamp when L0 files are created during flush when `AdvancedColumnFamilyOptions.persist_user_defined_timestamps` is false. Logically stripping timestamp here means replacing the original user-defined timestamp with a mininum timestamp, which for now is hard coded to be all zeros bytes. While working on this, I caught a missing piece on the `BlockBuilder` level for this feature. The current quick path `std::min(buffer_size, last_key_size)` needs a bit tweaking to work for this feature. When user-defined timestamp is stripped during block building, on writing first entry or right after resetting, `buffer` is empty and `buffer_size` is zero as usual. However, in follow-up writes, depending on the size of the stripped user-defined timestamp, and the size of the value, what's in `buffer` can sometimes be smaller than `last_key_size`, leading `std::min(buffer_size, last_key_size)` to truncate the `last_key`. Previous test doesn't caught the bug because in those tests, the size of the stripped user-defined timestamps bytes is smaller than the length of the value. In order to avoid the conditional operation, this PR changed the original trivial `std::min` operation into an arithmetic operation. Since this is a change in a hot and performance critical path, I did the following benchmark to check no observable regression is introduced. ```TEST_TMPDIR=/dev/shm/rocksdb1 ./db_bench -benchmarks=fillseq -memtablerep=vector -allow_concurrent_memtable_write=false -num=50000000``` Compiled with DEBUG_LEVEL=0 Test vs. control runs simulaneous for better accuracy, units = ops/sec PR vs base: Round 1: 350652 vs 349055 Round 2: 365733 vs 364308 Round 3: 355681 vs 354475 Pull Request resolved: https://github.com/facebook/rocksdb/pull/11557 Test Plan: New timestamp specific test added or existing tests augmented, both are parameterized with `UserDefinedTimestampTestMode`: `UserDefinedTimestampTestMode::kNormal` -> UDT feature enabled, write / read with min timestamp `UserDefinedTimestampTestMode::kStripUserDefinedTimestamps` -> UDT feature enabled, write / read with min timestamp, set Options.persist_user_defined_timestamps to false. ``` make all check ./db_wal_test --gtest_filter="*WithTimestamp*" ./flush_job_test --gtest_filter="*WithTimestamp*" ./repair_test --gtest_filter="*WithTimestamp*" ./block_based_table_reader_test ``` Reviewed By: pdillinger Differential Revision: D47027664 Pulled By: jowlyzhang fbshipit-source-id: e729193b6334dfc63aaa736d684d907a022571f5
-
- 28 6月, 2023 6 次提交
-
-
由 Griffin Smith 提交于
Summary: Expose the remaining fields of PlainTableOptions as arguments to `rocksdb_options_set_plain_table_factory` in the C API. Pull Request resolved: https://github.com/facebook/rocksdb/pull/11442 Reviewed By: ajkr Differential Revision: D46786962 Pulled By: hx235 fbshipit-source-id: 8862083dde332bfecc5ff02f9375776ad35c11f5
-
由 Akanksha Mahajan 提交于
Summary: Add `skip_tmpdir_check` argument in crash script. If `tmp_dir` is on remote storage and exist, `isdir` will be false (checking on local storage) leading to exit. By passing `skip_tmpdir_check` with `crashtest.py`, the dir check can be skipped. Pull Request resolved: https://github.com/facebook/rocksdb/pull/11539 Test Plan: Ran locally Reviewed By: anand1976 Differential Revision: D46740456 Pulled By: akankshamahajan15 fbshipit-source-id: 8726882ef53d2c84b604c7515e84eda6d1bf797c
-
由 Jay Schmidek 提交于
Summary: Pull Request resolved: https://github.com/facebook/rocksdb/pull/9527 Reviewed By: akankshamahajan15 Differential Revision: D47007647 Pulled By: ajkr fbshipit-source-id: e13544130b2731e07fa5fa4b9a2aa5f75b548c7e
-
由 Yelso Honnr 提交于
Summary: I made some changes to add OpenBSD support. Second time doing something like this, so I apologize in advance if I'm doing something wrong (had some minor hiccups with how github worked). Fixes https://github.com/facebook/rocksdb/issues/11220 Pull Request resolved: https://github.com/facebook/rocksdb/pull/11255 Reviewed By: akankshamahajan15 Differential Revision: D46361706 Pulled By: ajkr fbshipit-source-id: 90922fa30197fe6d6f3c0e3ecca2dbb92c337277
-
由 Yingchun Lai 提交于
Summary: There are some comments on subclasses in EncryptedEnv module which are duplicate to their parent classes, it would be nice to remove the duplication and keep the consistency if the comments on parent classes updated in someday. Pull Request resolved: https://github.com/facebook/rocksdb/pull/11549 Reviewed By: akankshamahajan15 Differential Revision: D47007061 Pulled By: ajkr fbshipit-source-id: 8bfdaf9f2418a24ca951c30bb88e90ac861d9016
-
由 JUBIN CHHEDA 提交于
Summary: Infer detected a(n) [Unnecessary Copy Intermediate](https://fbinfer.com/docs/next/all-issue-types#unnecessary copy intermediate) issue. variable &my_secondary_handles is copied unnecessarily into an intermediate on line 268. To avoid the copy, try moving it by calling std::move instead or alternatively change the callee's parameter type to const &. Pull Request resolved: https://github.com/facebook/rocksdb/pull/11566 Reviewed By: akankshamahajan15 Differential Revision: D47057361 Pulled By: ajkr fbshipit-source-id: bc5d7a71638aecbf976f1a163128b489c9e87fd8
-