- 18 8月, 2020 2 次提交
-
-
由 Hans Holmberg 提交于
Summary: This pull request adds the parameter --fs_uri to db_bench and db_stress, creating a composite env combining the default env with a specified registered rocksdb file system. This makes it easier to develop and test new RocksDB FileSystems. The pull request also registers the posix file system for testing purposes. Examples: ``` $./db_bench --fs_uri=posix:// --benchmarks=fillseq $./db_stress --fs_uri=zenfs://nullb1 ``` zenfs is a RocksDB FileSystem I'm developing to add support for zoned block devices, and in that case the zoned block device is specified in the uri (a zoned null block device in the above example). Pull Request resolved: https://github.com/facebook/rocksdb/pull/6878 Reviewed By: siying Differential Revision: D23023063 Pulled By: ajkr fbshipit-source-id: 8b3fe7193ce45e683043b021779b7a4d547af247
-
由 John Goerzen 提交于
Summary: pkg-config files are quite useful for communicating to users of a library how to compile against them. This commit generates and installs a pkg-config file that can be used for both static and dynamic builds against the RocksDB library. This should make life easier for developers of client programs, language bindings, etc. Example usage: ``` g++ `pkg-config --cflags rocksdb` -o simple_example simple_example.cc `pkg-config --libs rocksdb` g++ `pkg-config --cflags --static rocksdb` -static \ -o simple_example simple_example.cc `pkg-config --libs --static rocksdb` ``` The commit also adds the generated file to .gitignore, to the uninstall target, and to clean. No additional dependencies are added to RocksDB itself, and this does not make RocksDB use pkg-config as part of its build process. Resolves https://github.com/facebook/rocksdb/issues/4452 Pull Request resolved: https://github.com/facebook/rocksdb/pull/7244 Reviewed By: siying Differential Revision: D23146153 Pulled By: ajkr fbshipit-source-id: 3045aa650d68bd5ac42d40ed709570e9584ef004
-
- 15 8月, 2020 5 次提交
-
-
由 Jay Zhuang 提交于
Summary: Have a global StatsDumpScheduler for all DB instance stats dumping, including `DumpStats()` and `PersistStats()`. Before this, there're 2 dedicate threads for every DB instance, one for DumpStats() one for PersistStats(), which could create lots of threads if there're hundreds DB instances. Pull Request resolved: https://github.com/facebook/rocksdb/pull/7223 Reviewed By: riversand963 Differential Revision: D23056737 Pulled By: jay-zhuang fbshipit-source-id: 0faa2311142a73433ebb3317361db7cbf43faeba
-
由 Yanqin Jin 提交于
Summary: If user-defined timestamp is enabled, current implementation can expose newer data to queries even if an older sequence number is specified via read_options.snapshot. This PR makes Get() respect sequence-number-based snapshot. Solution is simple. Besides using <ukey, ts, seq> to search the index for the key, we also verify that the candidate result's seq is smaller than or equal to seq. This requires passing a seq via `GetContext`, which results in the majority of code change caused by this PR. Also added a few unit tests to demonstrate standard visibility during point lookup and range scan when timestamp and snapshot are both present. Test plan (devserver): ``` make check $./db_bench --benchmarks=fillseq,readrandom -cache_size=$[64*1024*1024] ``` Result this PR: readrandom : 4.827 micros/op 207180 ops/sec; 22.9 MB/s (1000000 of 1000000 found) master: readrandom : 4.936 micros/op 202610 ops/sec; 22.4 MB/s (1000000 of 1000000 found) Pull Request resolved: https://github.com/facebook/rocksdb/pull/7227 Reviewed By: ltamasi Differential Revision: D23015242 Pulled By: riversand963 fbshipit-source-id: ea7b85a728654553ba357d2e6a207b5e40f7376a
-
由 Daniel Smith 提交于
Summary: Pull Request resolved: https://github.com/facebook/rocksdb/pull/7248 Reviewed By: ajkr Differential Revision: D23108368 Pulled By: jay-zhuang fbshipit-source-id: 41765334b0bae26bd74a4d9250b9b98ea7ac6c3f
-
由 Zitan Chen 提交于
Summary: The flaky test in the title is caused by two problems. First, there is a bug in the BackupEngine that results in skipping computing the default crc32 checksum when `share_table_files` is enabled and the table is already backed up. Second, when `RestoreDBFromBackup` fails and the backup was being restored to the DB directory, it is likely that `RestoreDBFromBackup` has cleaned up the DB directory before it fails, and therefore, files in old backups may collide with files to be backed up if `share_files_with_checksum` is not enabled. New tests that cover the above problems are added. Pull Request resolved: https://github.com/facebook/rocksdb/pull/7254 Test Plan: `./backupable_db_test` Reviewed By: pdillinger Differential Revision: D23118715 Pulled By: gg814 fbshipit-source-id: 7be8de912808944be59e93d602c7431a54c079eb
-
由 Andrew Kryczka 提交于
Summary: Manual compaction with `CompactRangeOptions::change_levels` set could refit to a level targeted by another manual compaction. If force_consistency_checks were disabled, it could be possible for overlapping files to be written at that target level. This PR prevents the possibility by calling `DisableManualCompaction()` prior to `ReFitLevel()`. It also improves the manual compaction disabling mechanism to wait for pending manual compactions to complete before returning, and support disabling from multiple threads. Fixes https://github.com/facebook/rocksdb/issues/6432. Pull Request resolved: https://github.com/facebook/rocksdb/pull/7250 Test Plan: crash test command that repro'd the bug reliably: ``` $ TEST_TMPDIR=/dev/shm python tools/db_crashtest.py blackbox --simple -target_file_size_base=524288 -write_buffer_size=1048576 -clear_column_family_one_in=0 -reopen=0 -max_key=10000000 -column_families=1 -max_background_compactions=8 -compact_range_one_in=100000 -compression_type=none -compaction_style=1 -num_levels=5 -universal_min_merge_width=4 -universal_max_merge_width=8 -level0_file_num_compaction_trigger=12 -rate_limiter_bytes_per_sec=1048576000 -universal_max_size_amplification_percent=100 --duration=3600 --interval=60 --use_direct_io_for_flush_and_compaction=0 --use_direct_reads=0 --enable_compaction_filter=0 ``` Reviewed By: ltamasi Differential Revision: D23090800 Pulled By: ajkr fbshipit-source-id: afcbcd51b42ce76789fdb907d8b9ada790709c13
-
- 14 8月, 2020 2 次提交
-
-
由 Adam Retter 提交于
Summary: Previously RocksJava limited the format_version to 4. However, the C++ API is now at 5, and this will likely increase again in future. The Java API now allows any positive integer, and an exception is raised from JNI if the format_version is out-of-bounds. Pull Request resolved: https://github.com/facebook/rocksdb/pull/7242 Reviewed By: cheng-chang Differential Revision: D23077941 Pulled By: pdillinger fbshipit-source-id: ee69f7203448acddc41c6d86b470ed987d3d366d
-
由 sdong 提交于
Summary: We see some hosts failed to build platform009 with gcc. Revert the default to be platform007 if USE_CLANG is not specified. Pull Request resolved: https://github.com/facebook/rocksdb/pull/7253 Test Plan: Build with both of USE_CLANG=1 set and not set and observe it builds successfully, and see the tool chain used. Reviewed By: jay-zhuang Differential Revision: D23110550 fbshipit-source-id: 25cb47923f7174b24debdad0cc8d90b07c4d5d09
-
- 13 8月, 2020 7 次提交
-
-
由 sdong 提交于
Summary: Upgrade tool chain to the latest. It is done mostly manually as build_tools/build_detect_platform fails to update many of them. Try to fix a new clang analyze warning with the new tool chain. Pull Request resolved: https://github.com/facebook/rocksdb/pull/7251 Test Plan: "make all", "USE_CLANG=1 make all" Reviewed By: riversand963 Differential Revision: D23091090 fbshipit-source-id: 732e5a30137837431438f85f36296406b641f975
-
由 Levi Tamasi 提交于
Summary: The patch cleans up and refactors `CompressBlock` and `CompressBlockInternal` a bit. In particular, it does the following: * It renames `CompressBlockInternal` to `CompressData` and moves it to `util/compression.h`, where other general compression-related utilities are located. This will facilitate reuse in the BlobDB write path. * The signature of the method is changed so it now takes `compression_format_version` (similarly to the compression library specific methods) instead of `format_version` (which is specific to the block based table). * `GetCompressionFormatForVersion` no longer takes `compression_type` as a parameter. This parameter was only used in a (not entirely up-to-date) assertion; also, removing it eliminates the need to ensure this precondition holds at all call sites. * Does some minor cleanup in `CompressBlock`, for instance, it is now possible to pass only one of `sampled_output_fast` and `sampled_output_slow`. Pull Request resolved: https://github.com/facebook/rocksdb/pull/7249 Test Plan: `make check` Reviewed By: riversand963 Differential Revision: D23087278 Pulled By: ltamasi fbshipit-source-id: e6316e45baed8b4e7de7c1780c90501c2a3439b3
-
由 Akanksha Mahajan 提交于
Summary: As part of the IOTracing project, this PR 1. Caches "FileSystemPtr" object(wrapper class that returns file system pointer based on tracing enabled) instead of "FileSystem" pointer. 2. FileSystemPtr object is created using FileSystem pointer and IOTracer pointer. 3. IOTracer shared_ptr is created in DBImpl and it is passed to different classes through constructor. 4. When tracing is enabled through DB::StartIOTrace, FileSystemPtr returns FileSystemTracingWrapper pointer for tracing purpose and when it is disabled underlying FileSystem pointer is returned. Pull Request resolved: https://github.com/facebook/rocksdb/pull/7180 Test Plan: make check -j64 COMPILE_WITH_TSAN=1 make check -j64 Reviewed By: anand1976 Differential Revision: D22987117 Pulled By: akankshamahajan15 fbshipit-source-id: 6073617e4c2d5bc363914f3a1f55ae3b0a58fbf1
-
由 Arkady Dyakonov 提交于
Summary: The PR fixes a Java test for Merge operator `uint64add`. The current implementation uses wrong byte order for long serialization, but fails to catch this error because the merge sum is lower than `256`. The PR makes this test case more representative (i.e. it fails with wrong byte order) and changes the byte order to little endian. Some background: RocksDB uses LittleEndian byte order for integer serialization across all platforms. `MergeTest` uses `ByteBuffer` that defaults to BigEndian byte order. This test case might probably be used as a sample of `MergeOperator` usage in Java. Pull Request resolved: https://github.com/facebook/rocksdb/pull/7243 Reviewed By: ajkr Differential Revision: D23079593 Pulled By: pdillinger fbshipit-source-id: 82e8e166901d66733e96a0116f88d0ec4761ddf1
-
由 Zitan Chen 提交于
Summary: A new option `std::shared_ptr<FileChecksumGenFactory> backup_checksum_gen_factory` is added to `BackupableDBOptions`. This allows custom checksum functions to be used for creating, verifying, or restoring backups. Tests are added. Pull Request resolved: https://github.com/facebook/rocksdb/pull/7085 Test Plan: Passed make check Reviewed By: pdillinger Differential Revision: D22390756 Pulled By: gg814 fbshipit-source-id: 3b7756ca444c2129844536b91c3ca09f53b6248f
-
由 Yanqin Jin 提交于
Summary: ``` int* value = new int; ASSERT_NE(nullptr, value); ``` `ASSERT_NE` can expand the expression such that a memory leak is reported by clang analyzer. We can remove this ASSERT_NE since we can assume the memory allocation must succeed. Otherwise a bad alloc exception will be thrown and the process will be killed anyway. Test plan (dev server): ``` USE_CLANG=1 make analyze ``` Pull Request resolved: https://github.com/facebook/rocksdb/pull/7245 Reviewed By: jay-zhuang Differential Revision: D23079641 Pulled By: riversand963 fbshipit-source-id: a6739a903f90f8715f6f1ef3e5c8a329245b8e78
-
由 Levi Tamasi 提交于
Summary: The patch updates github-pages to the latest version. Dependencies were updated using `bundle update`. Also, the deprecated option `gems` is replaced with `plugins` in the Jekyll config. Pull Request resolved: https://github.com/facebook/rocksdb/pull/7235 Test Plan: `bundle exec jekyll serve` Reviewed By: pdillinger Differential Revision: D23034419 Pulled By: ltamasi fbshipit-source-id: a3f6df1c33281bdfd33aa61c6dc92162d9b7f079
-
- 12 8月, 2020 4 次提交
-
-
由 Yanqin Jin 提交于
Summary: Timer (defined in timer.h) schedules and runs user-specified fuctions regularly. Current implementation holds the mutex while running user function, which will lead to contention and waiting. To fix, Timer::Run releases mutex before running user function, and re-acquires it afterwards. This fix will impact how we can cancel a task. If the task is running, it is not holding the mutex. The thread calling Cancel() should wait until the current task finishes. Test Plan (devserver): make check COMPILE_WITH_ASAN=1 make check Pull Request resolved: https://github.com/facebook/rocksdb/pull/7228 Reviewed By: jay-zhuang Differential Revision: D23065487 Pulled By: riversand963 fbshipit-source-id: 07cb59741f506d3eb875c8ab90f73437568d3724
-
由 Yanqin Jin 提交于
Summary: As title. Pull Request resolved: https://github.com/facebook/rocksdb/pull/7229 Test Plan: Watch for CircleCI results. Reviewed By: siying Differential Revision: D23023943 Pulled By: riversand963 fbshipit-source-id: 41a989a3ffdfd2decd309185e3b963e810419577
-
由 Peter Dillinger 提交于
Summary: We have a number of tests hanging on MacOS and windows due to mishandling of code for mock sleeps. In addition, the code was in terrible shape because the same variable (addon_time_) would sometimes refer to microseconds and sometimes to seconds. One test even assumed it was nanoseconds but was written to pass anyway. This has been cleaned up so that DB tests generally use a SpecialEnv function to mock sleep, for either some number of microseconds or seconds depending on the function called. But to call one of these, the test must first call SetMockSleep (precondition enforced with assertion), which also turns sleeps in RocksDB into mock sleeps. To also removes accounting for actual clock time, call SetTimeElapseOnlySleepOnReopen, which implies SetMockSleep (on DB re-open). This latter setting only works by applying on DB re-open, otherwise havoc can ensue if Env goes back in time with DB open. More specifics: Removed some unused test classes, and updated comments on the general problem. Fixed DBSSTTest.GetTotalSstFilesSize using a sync point callback instead of mock time. For this we have the only modification to production code, inserting a sync point callback in flush_job.cc, which is not a change to production behavior. Removed unnecessary resetting of mock times to 0 in many tests. RocksDB deals in relative time. Any behaviors relying on absolute date/time are likely a bug. (The above test DBSSTTest.GetTotalSstFilesSize was the only one clearly injecting a specific absolute time for actual testing convenience.) Just in case I misunderstood some test, I put this note in each replacement: // NOTE: Presumed unnecessary and removed: resetting mock time in env Strengthened some tests like MergeTestTime, MergeCompactionTimeTest, and FilterCompactionTimeTest in db_test.cc stats_history_test and blob_db_test are each their own beast, rather deeply dependent on MockTimeEnv. Each gets its own variant of a work-around for TimedWait in a mock time environment. (Reduces redundancy and inconsistency in stats_history_test.) Intended follow-up: Remove TimedWait from the public API of InstrumentedCondVar, and only make that accessible through Env by passing in an InstrumentedCondVar and a deadline. Then the Env implementations mocking time can fix this problem without using sync points. (Test infrastructure using sync points interferes with individual tests' control over sync points.) With that change, we can simplify/consolidate the scattered work-arounds. Pull Request resolved: https://github.com/facebook/rocksdb/pull/7101 Test Plan: make check on Linux and MacOS Reviewed By: zhichao-cao Differential Revision: D23032815 Pulled By: pdillinger fbshipit-source-id: 7f33967ada8b83011fb54e8279365c008bd6610b
-
由 Levi Tamasi 提交于
Summary: `VersionStorageInfo::AddFile` currently has a debug-mode consistency check to make sure the newly added file does not overlap with the previous one (for levels below L0). Considering that `VersionBuilder::CheckConsistency` also performs similar checks (in fact, those checks are more comprehensive and cover L0 as well), this check is redundant. The patch removes it and also cleans up `AddFile` a little. Pull Request resolved: https://github.com/facebook/rocksdb/pull/7237 Test Plan: `make check` Reviewed By: riversand963 Differential Revision: D23041937 Pulled By: ltamasi fbshipit-source-id: e00665f3b83bfd17f86c54c238800f3d77d739bd
-
- 11 8月, 2020 6 次提交
-
-
由 Andrew Kryczka 提交于
Summary: The mechanism to mark files for compaction is most commonly used in delete-triggered compaction. This PR adds an option to exercise the marking mechanism on random files created by db_stress. This PR also enables that option in db_crashtest.py on its db_stress runs at random. Pull Request resolved: https://github.com/facebook/rocksdb/pull/7231 Test Plan: - ran some minified crash tests; verified they succeed and we see `"compaction_reason": "FilesMarkedForCompaction"` regularly in the logs. ``` $ TEST_TMPDIR=/dev/shm python tools/db_crashtest.py blackbox --duration=600 --interval=30 --max_key=10000000 --write_buffer_size=1048576 --target_file_size_base=1048576 --max_bytes_for_level_base=4194304 --value_size_mult=33 $ TEST_TMPDIR=/dev/shm python tools/db_crashtest.py whitebox --duration=600 --interval=30 --max_key=1000000 --write_buffer_size=1048576 --target_file_size_base=1048576 --max_bytes_for_level_base=4194304 --value_size_mult=33 --random_kill_odd=8887 ``` Reviewed By: anand1976 Differential Revision: D23025156 Pulled By: ajkr fbshipit-source-id: a404c467ebc12afa94dae35956ea9b372f592a96
-
由 anand76 提交于
Summary: Delete triggered compaction (DTC) for universal compaction style with ```num_levels = 1``` has been disabled for sometime due to a data correctness bug. This PR re-enables it with a bug fix. A file marked for compaction can be picked, along with all L0 files after it as the compaction input. We stop adding files to the input once we encounter a file already being compacted (the original bug failed to check the compaction status of the files). Tests: Add unit tests to ```compaction_picker_test.cc``` Pull Request resolved: https://github.com/facebook/rocksdb/pull/7224 Reviewed By: ajkr Differential Revision: D23031845 Pulled By: anand1976 fbshipit-source-id: 9de3cab5f9774cede666c2c48d309a7d9b88a505
-
由 Andrew Kryczka 提交于
Summary: `-O3` is already adopted widely, so we should make it easier to configure for development/open source. This PR adds an `OPTIMIZE_LEVEL` variable that users can set to override the `-O` flag chosen in the Makefile. Pull Request resolved: https://github.com/facebook/rocksdb/pull/7202 Test Plan: built a few different ways and verified correct value is passed for `-O` flag Reviewed By: pdillinger Differential Revision: D22845291 Pulled By: ajkr fbshipit-source-id: 84471362e7d627dd606b25bf5f6a3d796817fa1c
-
由 Peter Dillinger 提交于
Summary: Trying to fix issue that caused two failures out of eight most recent master builds Pull Request resolved: https://github.com/facebook/rocksdb/pull/7234 Test Plan: Watch CircleCI Reviewed By: riversand963 Differential Revision: D23032184 Pulled By: pdillinger fbshipit-source-id: dae403f63c0e4f6ab8a3e8e49a49069a532b8f4a
-
由 Yuhong Guo 提交于
Summary: 1. `std::random_shuffle` is deprecated and now we can use `std::shuffle` ``` /rocksdb/db/prefix_test.cc:590:12: error: 'random_shuffle<std::__1::__wrap_iter<unsigned long long *> >' is deprecated [-Werror,-Wdeprecated-declarations] std::random_shuffle(prefixes.begin(), prefixes.end()); ^ /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/../include/c++/v1/algorithm:2982:1: note: 'random_shuffle<std::__1::__wrap_iter<unsigned long long *> >' has been explicitly marked deprecated here _LIBCPP_DEPRECATED_IN_CXX14 void ^ /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/../include/c++/v1/__config:1107:39: note: expanded from macro '_LIBCPP_DEPRECATED_IN_CXX14' # define _LIBCPP_DEPRECATED_IN_CXX14 _LIBCPP_DEPRECATED ^ /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/../include/c++/v1/__config:1090:48: note: expanded from macro '_LIBCPP_DEPRECATED' # define _LIBCPP_DEPRECATED __attribute__ ((deprecated)) ``` 2. `c_test` link error with `-DROCKSDB_BUILD_SHARED=OFF`: ``` [ 7%] Linking CXX executable c_test ld: library not found for -lrocksdb-shared clang: error: linker command failed with exit code 1 (use -v to see invocation) make[5]: *** [c_test] Error 1 make[4]: *** [CMakeFiles/c_test.dir/all] Error 2 make[4]: *** Waiting for unfinished jobs.... ``` Pull Request resolved: https://github.com/facebook/rocksdb/pull/7205 Reviewed By: ajkr Differential Revision: D23030641 Pulled By: pdillinger fbshipit-source-id: f270e50fc0b824ca1a0876ec5c65d33f55a72dd0
-
由 Remington Brasga 提交于
Summary: The debug is supposed to print out two keys to show the value mismatch, which was compared just a few lines above. However, the actual print-out is the same values (so they obviously won't be mismatched) Pull Request resolved: https://github.com/facebook/rocksdb/pull/6587 Reviewed By: riversand963 Differential Revision: D23025279 Pulled By: ajkr fbshipit-source-id: 4c6c35bc60b273f13c08b5464b6f690d8a5cfe41
-
- 08 8月, 2020 3 次提交
-
-
由 Yanqin Jin 提交于
Summary: As title. Pull Request resolved: https://github.com/facebook/rocksdb/pull/7226 Test Plan: ``` tools/check_format_compatible.sh ``` Reviewed By: pdillinger Differential Revision: D23006013 Pulled By: riversand963 fbshipit-source-id: 428f29ad984691183beae72d3d3e753ec9b085a7
-
由 anand76 提交于
Summary: Introduce io_timeout in ReadOptions and enabled deadline/io_timeout for Iterators. Pull Request resolved: https://github.com/facebook/rocksdb/pull/7161 Test Plan: New unit tests in db_basic_test Reviewed By: riversand963 Differential Revision: D22687352 Pulled By: anand1976 fbshipit-source-id: 67bbb0e6d7ae80b256589244468494292538c6ec
-
由 Zhichao Cao 提交于
Summary: Pointed out by https://github.com/facebook/rocksdb/issues/7197 , there is a double lock in WriteImplWALOnly. Also find another deadlock in UnorderedWriteMemtable. Move the check after switch_all_.notify_all(). Pull Request resolved: https://github.com/facebook/rocksdb/pull/7199 Test Plan: pass make check Reviewed By: anand1976 Differential Revision: D22961714 Pulled By: zhichao-cao fbshipit-source-id: 0707922dc50d28ea141a15a8cdcbd1c8993ea0d8
-
- 07 8月, 2020 5 次提交
-
-
由 mrambacher 提交于
Summary: Added 4 more tests to those which pass ASSERT_STATUS_CHECKED (cache_test, lru_cache_test, filename_test, filelock_test). Pull Request resolved: https://github.com/facebook/rocksdb/pull/7211 Reviewed By: ajkr Differential Revision: D22982858 Pulled By: zhichao-cao fbshipit-source-id: acdd071582ed6aa7447ed96c5732f10bf720d783
-
由 Yingchun Lai 提交于
Summary: A colon will be added after 'msg' automatically when invoke function Status(Code _code, const Slice& msg, const Slice& msg2), it's not needed to append a colon explicitly to 'msg'. Pull Request resolved: https://github.com/facebook/rocksdb/pull/7041 Reviewed By: ajkr Differential Revision: D22292801 fbshipit-source-id: 8f2d69065bb779d2613468bf9fc9169f32c3f1ec
-
由 jsteemann 提交于
Summary: Rename "paraniod" to "paranoid" in a few places. Pull Request resolved: https://github.com/facebook/rocksdb/pull/7163 Reviewed By: ajkr Differential Revision: D22678242 fbshipit-source-id: 28b1011a736d0a95612676f7e1b9500a70c324b4
-
由 Adam Retter 提交于
Summary: Improvements to the RocksJava release process: * Generates the Maven artifact version number as part of the release step * Also generates appropriate checksum files to speed the deploy and publish step Pull Request resolved: https://github.com/facebook/rocksdb/pull/7219 Reviewed By: ltamasi Differential Revision: D22983481 Pulled By: pdillinger fbshipit-source-id: 7b8ffaf46471cd3cda181eb830c962b317d2e688
-
由 Cheng Chang 提交于
Summary: We're going to support more locking protocols such as range lock in transaction. However, in current design, `TransactionBase` has a member `tracked_keys` which assumes that point lock (lock a single key) is used, and is used in snapshot checking (isolation protocol). When using range lock, we may use read committed instead of snapshot checking as the isolation protocol. The most significant usage scenarios of `tracked_keys` are: 1. pessimistic transaction uses it to track the locked keys, and unlock these keys when commit or rollback. 2. optimistic transaction does not lock keys upfront, it only tracks the lock intentions in tracked_keys, and do write conflict checking when commit. 3. each `SavePoint` tracks the keys that are locked since the `SavePoint`, `RollbackToSavePoint` or `PopSavePoint` relies on both the tracked keys in `SavePoint`s and `tracked_keys`. Based on these scenarios, if we can abstract out a `LockTracker` interface to hold a set of tracked locks (can be keys or key ranges), and have methods that can be composed together to implement the scenarios, then `tracked_keys` can be an internal data structure of one implementation of `LockTracker`. See `utilities/transactions/lock/lock_tracker.h` for the detailed interface design, and `utilities/transactions/lock/point_lock_tracker.cc` for the implementation. In the future, a `RangeLockTracker` can be implemented to track range locks without affecting other components. After this PR, a clean interface for lock manager should be possible, and then ideally, we can have pluggable locking protocols. Pull Request resolved: https://github.com/facebook/rocksdb/pull/7013 Test Plan: Run `transaction_test` and `optimistic_transaction_test`. Reviewed By: ajkr Differential Revision: D22163706 Pulled By: cheng-chang fbshipit-source-id: f2860577b5334e31dd2994f5bc6d7c40d502b1b4
-
- 06 8月, 2020 3 次提交
-
-
由 Cheng Chang 提交于
Summary: `WalAddition`, `WalDeletion` are defined in `wal_version.h` and used in `VersionEdit`. `WalAddition` is used to represent events of creating a new WAL (no size, just log number), or closing a WAL (with size). `WalDeletion` is used to represent events of deleting or archiving a WAL, it means the WAL is no longer alive (won't be replayed during recovery). `WalSet` is the set of alive WALs kept in `VersionSet`. 1. Why use `WalDeletion` instead of relying on `MinLogNumber` to identify outdated WALs On recovery, we can compute `MinLogNumber()` based on the log numbers kept in MANIFEST, any log with number < MinLogNumber can be ignored. So it seems that we don't need to persist `WalDeletion` to MANIFEST, since we can ignore the WALs based on MinLogNumber. But the `MinLogNumber()` is actually a lower bound, it does not exactly mean that logs starting from MinLogNumber must exist. This is because in a corner case, when a column family is empty and never flushed, its log number is set to the largest log number, but not persisted in MANIFEST. So let's say there are 2 column families, when creating the DB, the first WAL has log number 1, so it's persisted to MANIFEST for both column families. Then CF 0 is empty and never flushed, CF 1 is updated and flushed, so a new WAL with log number 2 is created and persisted to MANIFEST for CF 1. But CF 0's log number in MANIFEST is still 1. So on recovery, MinLogNumber is 1, but since log 1 only contains data for CF 1, and CF 1 is flushed, log 1 might have already been deleted from disk. We can make `MinLogNumber()` be the exactly minimum log number that must exist, by persisting the most recent log number for empty column families that are not flushed. But if there are N such column families, then every time a new WAL is created, we need to add N records to MANIFEST. In current design, a record is persisted to MANIFEST only when WAL is created, closed, or deleted/archived, so the number of WAL related records are bounded to 3x number of WALs. 2. Why keep `WalSet` in `VersionSet` instead of applying the `VersionEdit`s to `VersionStorageInfo` `VersionEdit`s are originally designed to track the addition and deletion of SST files. The SST files are related to column families, each column family has a list of `Version`s, and each `Version` keeps the set of active SST files in `VersionStorageInfo`. But WALs are a concept of DB, they are not bounded to specific column families. So logically it does not make sense to store WALs in a column family's `Version`s. Also, `Version`'s purpose is to keep reference to SST / blob files, so that they are not deleted until there is no version referencing them. But a WAL is deleted regardless of version references. So we keep the WALs in `VersionSet` for the purpose of writing out the DB state's snapshot when creating new MANIFESTs. Pull Request resolved: https://github.com/facebook/rocksdb/pull/7164 Test Plan: make version_edit_test && ./version_edit_test make wal_edit_test && ./wal_edit_test Reviewed By: ltamasi Differential Revision: D22677936 Pulled By: cheng-chang fbshipit-source-id: 5a3b6890140e572ffd79eb37e6e4c3c32361a859
-
由 Levi Tamasi 提交于
Summary: FaultInjectionTestFS::NewDirectory currently asserts that the directory creation on the target filesystem succeeds. This is actually not guaranteed since there might be a legitimate I/O error when creating the directory. The patch removes this assertion. Pull Request resolved: https://github.com/facebook/rocksdb/pull/7220 Test Plan: `make check` Reviewed By: zhichao-cao Differential Revision: D22957990 Pulled By: ltamasi fbshipit-source-id: b2e221320d8ce7235cb4897ef5936072412a25b6
-
由 sdong 提交于
Summary: IteratorIterator::IsOutOfBound() and IteratorIterator::MayBeOutOfUpperBound() are two functions that related to upper bound check. It is hard for users to reason about this complexity. Consolidate the two functions into one and assign an enum as results to improve readability. Pull Request resolved: https://github.com/facebook/rocksdb/pull/7200 Test Plan: Run all existing test. Would run crash test with atomic for a while. Reviewed By: anand1976 Differential Revision: D22833181 fbshipit-source-id: a0c724267056adbd0476bde74650e6c7226077e6
-
- 05 8月, 2020 3 次提交
-
-
由 Yanqin Jin 提交于
Summary: as title. When ReadOptions.iter_start_ts is not nullptr, DBIter::key() should return internal keys including value type. Pull Request resolved: https://github.com/facebook/rocksdb/pull/7178 Test Plan: make check Reviewed By: ltamasi Differential Revision: D22935879 Pulled By: riversand963 fbshipit-source-id: 7508d962cf11ebcfa6386d2529b4f3606b47ccfd
-
由 Akanksha Mahajan 提交于
Summary: 1. Add support to start io tracing through DB::StartIOTrace(Env*, const TraceOptions&, std::unique_ptr<TraceWriter>&&) and end tracing through DB::EndIOTrace(). This doesn't trace DB::Open. User side code: //Open DB DB::Open(options, dbname, &db); /* Start tracing */ db->StartIOTrace(env, trace_opt, std::move(trace_writer)); /* Perform Operations */ /*End tracing*/ db->EndIOTrace(); 2. Fix the build errors for Windows. Pull Request resolved: https://github.com/facebook/rocksdb/pull/7203 Test Plan: make check -j64 Reviewed By: anand1976 Differential Revision: D22901947 Pulled By: akankshamahajan15 fbshipit-source-id: e59c0b785a802168e6f1aa028d99c224a35cb30c
-
由 sdong 提交于
Summary: https://github.com/facebook/rocksdb/pull/5289 introduces a performance regression that caused an upper bound check within every BlockBasedTableIterator::Next(). This is unnecessary if we've checked the boundary key for current block and it is within upper bound. Fix the bug. Also rename the boolean to a enum so that the code is slightly better readable. The original regression was probably to fix a bug that the block upper bound check status is not reset after a new block is created. Fix it bug so that the regression can be avoided without hitting the bug. Pull Request resolved: https://github.com/facebook/rocksdb/pull/7209 Test Plan: Run all existing tests. Will run atomic black box crash test for a while. Reviewed By: anand1976 Differential Revision: D22859246 fbshipit-source-id: cbdad1f5e656c55fd8b71726d5a4f6cb53ff9140
-