- 07 11月, 2021 1 次提交
-
-
由 Dennis Maisenbacher 提交于
Summary: Otherwise a rebuild is not done if a RocksDB plugin header file is changed. Pull Request resolved: https://github.com/facebook/rocksdb/pull/9120 Test Plan: Build RocksDB with a plugin. Change a header file of the RocksDB plugin and rebuild. Signed-off-by: NDennis Maisenbacher <dennis.maisenbacher@wdc.com> Reviewed By: riversand963 Differential Revision: D32223303 Pulled By: ajkr fbshipit-source-id: 76d31b10fe915906edc181c7b6398a09b7d079ee
-
- 04 11月, 2021 1 次提交
-
-
由 Jay Zhuang 提交于
Summary: Directory fsync might be expensive on btrfs and it may not be needed. Here are 4 directory fsync cases: 1. creating a new file: dir-fsync is not needed on btrfs, as long as the new file itself is synced. 2. renaming a file: dir-fsync is not needed if the renamed file is synced. So an API `FsyncAfterFileRename(filename, ...)` is provided to sync the file on btrfs. By default, it just calls dir-fsync. 3. deleting files: dir-fsync is forced by set `IOOptions.force_dir_fsync = true` 4. renaming multiple files (like backup and checkpoint): dir-fsync is forced, the same as above. Pull Request resolved: https://github.com/facebook/rocksdb/pull/8903 Test Plan: run tests on btrfs and non btrfs Reviewed By: ajkr Differential Revision: D30885059 Pulled By: jay-zhuang fbshipit-source-id: dd2730b31580b0bcaedffc318a762d7dbf25de4a
-
- 29 10月, 2021 1 次提交
-
-
由 Peter Dillinger 提交于
Summary: This feature was not part of any common or CI build, so no surprise it broke. Now we can at least ensure compilation. I don't know how to run the test successfully (missing config file) so it is bypassed for now. Fixes https://github.com/facebook/rocksdb/issues/9078 Pull Request resolved: https://github.com/facebook/rocksdb/pull/9088 Test Plan: CI Reviewed By: mrambacher Differential Revision: D32009467 Pulled By: pdillinger fbshipit-source-id: 3e0d1e5fde7f0ece703d48a81479e1cc7392c25c
-
- 23 10月, 2021 1 次提交
-
-
由 Jonathan Albrecht 提交于
Summary: This PR adds support for building on s390x including updating travis CI. It uses the previous work in https://github.com/facebook/rocksdb/pull/6168 and adds some more changes to get all current tests (make check and jni tests) to pass. The tests were run with snappy, lz4, bzip2 and zstd all compiled in. There are a few pieces still needed to get the travis build working that I don't think I can do. adamretter is this something you could help with? 1. A prebuilt https://rocksdb-deps.s3-us-west-2.amazonaws.com/cmake/cmake-3.14.5-Linux-s390x.deb package 2. A https://hub.docker.com/r/evolvedbinary/rocksjava s390x image Not sure if there is more required for travis. Happy to help in any way I can. Pull Request resolved: https://github.com/facebook/rocksdb/pull/8962 Reviewed By: mrambacher Differential Revision: D31802198 Pulled By: pdillinger fbshipit-source-id: 683511466fa6b505f85ba5a9964a268c6151f0c2
-
- 06 10月, 2021 1 次提交
-
-
http://sourceware.org/pub/bzip2由 Yanqin Jin 提交于
Summary: Download bzip2 from `https://sourceware.org/pub/bzip2` to `http://sourceware.org/pub/bzip2` to resolve curl's ca verification error. Pull Request resolved: https://github.com/facebook/rocksdb/pull/8986 Reviewed By: akankshamahajan15 Differential Revision: D31387038 Pulled By: riversand963 fbshipit-source-id: 510fdb9530e63639cd5d20339f3f3cbf720068e9
-
- 11 9月, 2021 1 次提交
-
-
由 Peter Dillinger 提交于
Summary: It's always annoying to find a header does not include its own dependencies and only works when included after other includes. This change adds `make check-headers` which validates that each header can be included at the top of a file. Some headers are excluded e.g. because of platform or external dependencies. rocksdb_namespace.h had to be re-worked slightly to enable checking for failure to include it. (ROCKSDB_NAMESPACE is a valid namespace name.) Fixes mostly involve adding and cleaning up #includes, but for FileTraceWriter, a constructor was out-of-lined to make a forward declaration sufficient. This check is not currently run with `make check` but is added to CircleCI build-linux-unity since that one is already relatively fast. Pull Request resolved: https://github.com/facebook/rocksdb/pull/8893 Test Plan: existing tests and resolving issues detected by new check Reviewed By: mrambacher Differential Revision: D30823300 Pulled By: pdillinger fbshipit-source-id: 9fff223944994c83c105e2e6496d24845dc8e572
-
- 08 9月, 2021 1 次提交
-
-
由 Peter Dillinger 提交于
Summary: * Don't hardcode namespace rocksdb (use ROCKSDB_NAMESPACE) * Don't #include <rocksdb/...> (use double quotes) * Support putting NOCOMMIT (any case) in source code that should not be committed/pushed in current state. These will be run with `make check` and in GitHub actions Pull Request resolved: https://github.com/facebook/rocksdb/pull/8821 Test Plan: existing tests, manually try out new checks Reviewed By: zhichao-cao Differential Revision: D30791726 Pulled By: pdillinger fbshipit-source-id: 399c883f312be24d9e55c58951d4013e18429d92
-
- 25 8月, 2021 1 次提交
-
-
由 Hui Xiao 提交于
Summary: Context: To help cap various memory usage by a single limit of the block cache capacity, we charge the memory usage through inserting/releasing dummy entries in the block cache. CacheReservationManager is such a class (non thread-safe) responsible for inserting/removing dummy entries to reserve cache space for memory used by the class user. - Refactored the inner private class CacheRep of WriteBufferManager into public CacheReservationManager class for reusability such as for https://github.com/facebook/rocksdb/pull/8428 - Encapsulated implementation details of cache key generation and dummy entries insertion/release in cache reservation as discussed in https://github.com/facebook/rocksdb/pull/8506#discussion_r666550838 - Consolidated increase/decrease cache reservation into one API - UpdateCacheReservation. - Adjusted the previous dummy entry release algorithm in decreasing cache reservation to be loop-releasing dummy entries to stay symmetric to dummy entry insertion algorithm - Made the previous dummy entry release algorithm in delayed decrease mode more aggressive for better decreasing cache reservation when memory used is less likely to increase back. Previously, the algorithms only release 1 dummy entries when new_mem_used < 3/4 * cache_allocated_size_ and cache_allocated_size_ - kSizeDummyEntry > new_mem_used. Now, the algorithms loop-releases as many dummy entries as possible when new_mem_used < 3/4 * cache_allocated_size_. - Updated WriteBufferManager's test cases to adapt to changes on the release algorithm mentioned above and left comment for some test cases for clarity - Replaced the previous cache key prefix generation (utilizing object address related to the cache client) with one that utilizes Cache->NewID() to prevent cache-key collision among dummy entry clients sharing the same cache. The specific collision we are preventing happens when the object address is reused for a new cache-key prefix while the old cache-key using that same object address in its prefix still exists in the cache. This could happen due to that, under LRU cache policy, there is a possible delay in releasing a cache entry after the cache client object owning that cache entry get deallocated. In this case, the object address related to the cache client object can get reused for other client object to generate a new cache-key prefix. This prefix generation can be made obsolete after Peter's unification of all the code generating cache key, mentioned in https://github.com/facebook/rocksdb/pull/8506#discussion_r667265255 Pull Request resolved: https://github.com/facebook/rocksdb/pull/8506 Test Plan: - Passing the added unit tests cache_reservation_manager_test.cc - Passing existing and adjusted write_buffer_manager_test.cc Reviewed By: ajkr Differential Revision: D29644135 Pulled By: hx235 fbshipit-source-id: 0fc93fbfe4a40bb41be85c314f8f2bafa8b741f7
-
- 16 8月, 2021 1 次提交
-
-
由 Adam Retter 提交于
Summary: This is the `db_test2` parts of https://github.com/facebook/rocksdb/pull/7737 reworked on the latest HEAD. Pull Request resolved: https://github.com/facebook/rocksdb/pull/8640 Reviewed By: akankshamahajan15 Differential Revision: D30303684 Pulled By: mrambacher fbshipit-source-id: 263e2f82d849bde4048b60aed8b31e7deed4706a
-
- 17 7月, 2021 1 次提交
-
-
由 Jay Zhuang 提交于
Summary: Otherwise the build may report warning about missing `benchmark.h` for some targets, the error won't break the build. Pull Request resolved: https://github.com/facebook/rocksdb/pull/8523 Test Plan: `make blackbox_ubsan_crash_test` on a machine without benchmark lib installed. Reviewed By: pdillinger Differential Revision: D29682478 Pulled By: jay-zhuang fbshipit-source-id: e1261fbcda46bc6bd3cd39b7b03b7f78927d0430
-
- 13 7月, 2021 1 次提交
-
-
由 Peter Dillinger 提交于
Summary: MyRocks apparently uses valgrind to check for unreachable unfreed data, which is stricter than our valgrind checks. Internal ref: D29257815 This patch adds valgrind support to STATIC_AVOID_DESTRUCTION so that it's not reported with those stricter checks. Pull Request resolved: https://github.com/facebook/rocksdb/pull/8503 Test Plan: make valgrind_test Also, with modified VALGRIND_OPTS (see Makefile), more kinds of failures seen before than after this commit. Reviewed By: ajkr, yizhang82 Differential Revision: D29597784 Pulled By: pdillinger fbshipit-source-id: 360de157a176aec4d1be99ca20d160ecd47c0873
-
- 09 7月, 2021 1 次提交
-
-
由 Jay Zhuang 提交于
Summary: Add google benchmark for microbench. Add ribbon_bench for benchmark ribbon filter vs. other filters. Pull Request resolved: https://github.com/facebook/rocksdb/pull/8493 Test Plan: added test to CI To run the benchmark on devhost: Install benchmark: `$ sudo dnf install google-benchmark-devel` Build and run: `$ ROCKSDB_NO_FBCODE=1 DEBUG_LEVEL=0 make microbench` or with cmake: `$ mkdir build && cd build && cmake .. -DCMAKE_BUILD_TYPE=Release -DWITH_BENCHMARK=1 && make microbench` Reviewed By: pdillinger Differential Revision: D29589649 Pulled By: jay-zhuang fbshipit-source-id: 8fed13b562bef4472f161ecacec1ab6b18911dff
-
- 08 7月, 2021 1 次提交
-
-
由 Andrew Kryczka 提交于
Summary: Various tests had disabled valgrind due to it slowing down and timing out (as is the case right now) the CI runs. Where a test was disabled with no comment, I assumed slowness was the cause. For these tests that were slow under valgrind, as well as the ones identified in https://github.com/facebook/rocksdb/issues/8352, this PR moves them behind the compiler flag `-DROCKSDB_FULL_VALGRIND_RUN`. Pull Request resolved: https://github.com/facebook/rocksdb/pull/8475 Test Plan: running `make full_valgrind_test`, `make valgrind_test`, `make check`; will verify they appear working correctly Reviewed By: jay-zhuang Differential Revision: D29504843 Pulled By: ajkr fbshipit-source-id: 2aac90749cfbd30d5ce11cb29a07a1b9314eeea7
-
- 24 6月, 2021 1 次提交
-
-
由 Levi Tamasi 提交于
Summary: Follow-up to https://github.com/facebook/rocksdb/issues/8426 . The patch adds a new kind of `InternalIterator` that wraps another one and passes each key-value encountered to `BlobGarbageMeter` as inflow. This iterator will be used as an input iterator for compactions when the input SSTs reference blob files. Pull Request resolved: https://github.com/facebook/rocksdb/pull/8443 Test Plan: `make check` Reviewed By: jay-zhuang Differential Revision: D29311987 Pulled By: ltamasi fbshipit-source-id: b4493b4c0c0c2e3c2ecc33c8969a5ef02de5d9d8
-
- 22 6月, 2021 2 次提交
-
-
由 Levi Tamasi 提交于
Summary: This is part of an alternative approach to https://github.com/facebook/rocksdb/issues/8316. Unlike that approach, this one relies on key-values getting processed one by one during compaction, and does not involve persistence. Specifically, the patch adds a class `BlobGarbageMeter` that can track the number and total size of blobs in a (sub)compaction's input and output on a per-blob file basis. This information can then be used to compute the amount of additional garbage generated by the compaction for any given blob file by subtracting the "outflow" from the "inflow." Note: this patch only adds `BlobGarbageMeter` and associated unit tests. I plan to hook up this class to the input and output of `CompactionIterator` in a subsequent PR. Pull Request resolved: https://github.com/facebook/rocksdb/pull/8426 Test Plan: `make check` Reviewed By: jay-zhuang Differential Revision: D29242250 Pulled By: ltamasi fbshipit-source-id: 597e50ad556540e413a50e804ba15bc044d809bb
-
由 Andrew Kryczka 提交于
Summary: - `c_test` fails because `rocksdb_compact_range()` swallows a `Status`. - `env_test` fails because `ReadRequest`s to `MultiRead()` do not have their `Status`es checked. Pull Request resolved: https://github.com/facebook/rocksdb/pull/8430 Test Plan: `ASSERT_STATUS_CHECKED=1 make -j48 check` Reviewed By: jay-zhuang Differential Revision: D29257473 Pulled By: ajkr fbshipit-source-id: e02127f971703744be7de85f0a028e4664c79577
-
- 10 6月, 2021 1 次提交
-
-
由 Levi Tamasi 提交于
Summary: Logically, subcompactions process a key range [start, end); however, the way this is currently implemented is that the `CompactionIterator` for any given subcompaction keeps processing key-values until it actually outputs a key that is out of range, which is then discarded. Instead of doing this, the patch introduces a new type of internal iterator called `ClippingIterator` which wraps another internal iterator and "clips" its range of key-values so that any KVs returned are strictly in the [start, end) interval. This does eliminate a (minor) inefficiency by stopping processing in subcompactions exactly at the limit; however, the main motivation is related to BlobDB: namely, we need this to be able to measure the amount of garbage generated by a subcompaction precisely and prevent off-by-one errors. Pull Request resolved: https://github.com/facebook/rocksdb/pull/8327 Test Plan: `make check` Reviewed By: siying Differential Revision: D28761541 Pulled By: ltamasi fbshipit-source-id: ee0e7229f04edabbc7bed5adb51771fbdc287f69
-
- 20 5月, 2021 2 次提交
-
-
由 Jay Zhuang 提交于
Summary: Pull Request resolved: https://github.com/facebook/rocksdb/pull/8300 Reviewed By: ajkr Differential Revision: D28464726 Pulled By: jay-zhuang fbshipit-source-id: 49e9f4fb791808a6cbf39a7b1a331373f645fc5e
-
由 anand76 提交于
Summary: This PR adds a ```-secondary_cache_uri``` option to the cache_bench and db_bench tools to allow the user to specify a custom secondary cache URI. The object registry is used to create an instance of the ```SecondaryCache``` object of the type specified in the URI. The main cache_bench code is packaged into a separate library, similar to db_bench. An example invocation of db_bench with a secondary cache URI - ```db_bench --env_uri=ws://ws.flash_sandbox.vll1_2/ -db=anand/nvm_cache_2 -use_existing_db=true -benchmarks=readrandom -num=30000000 -key_size=32 -value_size=256 -use_direct_reads=true -cache_size=67108864 -cache_index_and_filter_blocks=true -secondary_cache_uri='cachelibwrapper://filename=/home/anand76/nvm_cache/cache_file;size=2147483648;regionSize=16777216;admPolicy=random;admProbability=1.0;volatileSize=8388608;bktPower=20;lockPower=12' -partition_index_and_filters=true -duration=1800``` Pull Request resolved: https://github.com/facebook/rocksdb/pull/8312 Reviewed By: zhichao-cao Differential Revision: D28544325 Pulled By: anand1976 fbshipit-source-id: 8f209b9af900c459dc42daa7a610d5f00176eeed
-
- 22 4月, 2021 1 次提交
-
-
由 Akanksha Mahajan 提交于
Summary: When WriteBufferManager is shared across DBs and column families to maintain memory usage under a limit, OOMs have been observed when flush cannot finish but writes continuously insert to memtables. In order to avoid OOMs, when memory usage goes beyond buffer_limit_ and DBs tries to write, this change will stall incoming writers until flush is completed and memory_usage drops. Design: Stall condition: When total memory usage exceeds WriteBufferManager::buffer_size_ (memory_usage() >= buffer_size_) WriterBufferManager::ShouldStall() returns true. DBImpl first block incoming/future writers by calling write_thread_.BeginWriteStall() (which adds dummy stall object to the writer's queue). Then DB is blocked on a state State::Blocked (current write doesn't go through). WBStallInterface object maintained by every DB instance is added to the queue of WriteBufferManager. If multiple DBs tries to write during this stall, they will also be blocked when check WriteBufferManager::ShouldStall() returns true. End Stall condition: When flush is finished and memory usage goes down, stall will end only if memory waiting to be flushed is less than buffer_size/2. This lower limit will give time for flush to complete and avoid continous stalling if memory usage remains close to buffer_size. WriterBufferManager::EndWriteStall() is called, which removes all instances from its queue and signal them to continue. Their state is changed to State::Running and they are unblocked. DBImpl then signal all incoming writers of that DB to continue by calling write_thread_.EndWriteStall() (which removes dummy stall object from the queue). DB instance creates WBMStallInterface which is an interface to block and signal DBs during stall. When DB needs to be blocked or signalled by WriteBufferManager, state_for_wbm_ state is changed accordingly (RUNNING or BLOCKED). Pull Request resolved: https://github.com/facebook/rocksdb/pull/7898 Test Plan: Added a new test db/db_write_buffer_manager_test.cc Reviewed By: anand1976 Differential Revision: D26093227 Pulled By: akankshamahajan15 fbshipit-source-id: 2bbd982a3fb7033f6de6153aa92a221249861aae
-
- 16 4月, 2021 1 次提交
-
-
由 mrambacher 提交于
Summary: - Fixes the makefile to do the right thing when invoking multiple targets (e.g. make shared_lib install-shared). - Fixes the building of db_stress in shared lib mode. Pull Request resolved: https://github.com/facebook/rocksdb/pull/8195 Reviewed By: pdillinger Differential Revision: D27803452 Pulled By: mrambacher fbshipit-source-id: 7c285d267770a359eb47f25855affdf58687e0e4
-
- 07 4月, 2021 1 次提交
-
-
由 Adam Retter 提交于
Summary: The previous version of ZStd doesn't build correctly with Make 3.82. Updating it resolves the issue. jay-zhuang This also needs to be cherry-picked to: 1. 6.17.fb 2. 6.18.fb 3. 6.19.fb Pull Request resolved: https://github.com/facebook/rocksdb/pull/8155 Reviewed By: riversand963 Differential Revision: D27596460 Pulled By: ajkr fbshipit-source-id: ac8492245e6273f54efcc1587346a797a91c9441
-
- 05 4月, 2021 1 次提交
-
-
由 Peter Dillinger 提交于
Summary: New tests should by default be expected to be parallelizeable and passing with ASSERT_STATUS_CHECKED. Thus, I'm changing those two lists to exclusions rather than inclusions. For the set of exclusions, I only listed things that currently failed for me when attempting not to exclude, or had some other documented reason. This marks many more tests as "parallel," which will potentially cause some failures from self-interference, but we can address those as they are discovered. Also changed CircleCI ASC test to be parallelized; the easy way to do that is to exclude building tests that don't pass ASC, which is now a small set. Pull Request resolved: https://github.com/facebook/rocksdb/pull/8146 Test Plan: Watch CI, etc. Reviewed By: riversand963 Differential Revision: D27542782 Pulled By: pdillinger fbshipit-source-id: bdd74bcd912a963ee33f3fc0d2cad2567dc7740f
-
- 31 3月, 2021 1 次提交
-
-
由 mrambacher 提交于
Summary: At least under MacOS, some things were excluded from the build (like Snappy) because the compilation flags were not passed in correctly. This PR does a few things: - Passes the EXTRA_CXX/LDFLAGS into build_detect_platform. This means that if some tool (like TBB for example) is not installed in a standard place, it could still be detected by build_detect_platform. In this case, the developer would invoke: "EXTRA_CXXFLAGS=<path to TBB include> EXTRA_LDFLAGS=<path to TBB library> make", and the build script would find the tools in the extra location. - Changes the compilation tests to use PLATFORM_CXXFLAGS. This change causes the EXTRA_FLAGS passed in to the script to be included in the compilation check. Additionally, flags set by the script itself (like --std=c++11) will be used during the checks. Validated that the make_platform.mk file generated on Linux does not change with this change. On my MacOS machine, the SNAPPY libraries are now available (they were not before as they required --std=c++11 to build). I also verified that I can build against TBB installed on my Mac by passing in the EXTRA CXX and LD FLAGS to the location in which TBB is installed. Pull Request resolved: https://github.com/facebook/rocksdb/pull/8111 Reviewed By: jay-zhuang Differential Revision: D27353516 Pulled By: mrambacher fbshipit-source-id: b6b378c96dbf678bab1479556dcbcb49c47e807d
-
- 30 3月, 2021 1 次提交
-
-
由 Adam Retter 提交于
Summary: If the platform is ppc64 and the libc is not GNU libc, then we exclude the range_tree from compilation. See https://jira.percona.com/browse/PS-7559 Pull Request resolved: https://github.com/facebook/rocksdb/pull/8070 Reviewed By: jay-zhuang Differential Revision: D27246004 Pulled By: mrambacher fbshipit-source-id: 59d8433242ce7ce608988341becb4f83312445f5
-
- 29 3月, 2021 1 次提交
-
-
由 mrambacher 提交于
Summary: Because build_version.cc is dependent on the library objects (to force a re-generation of it), the library objects would be built in order to satisfy this rule. Because there is a build_version.d file, it would need generated and included. Change the ALL_DEPS/FILES to not include build_version.cc (meaning no .d file for it, which is okay since it is generated). Also changed the rule on whether or not to generate DEP files to skip tags. Pull Request resolved: https://github.com/facebook/rocksdb/pull/8097 Reviewed By: ajkr Differential Revision: D27299815 Pulled By: mrambacher fbshipit-source-id: 1efbe8a56d062f57ae13b6c2944ad3faf775087e
-
- 23 3月, 2021 1 次提交
-
-
由 Yanqin Jin 提交于
Summary: Add some basic test for user-defined timestamp to db_stress. Currently, read with timestamp always tries to read using the current timestamp. Due to the per-key timestamp-sequence ordering constraint, we only add timestamp- related tests to the `NonBatchedOpsStressTest` since this test serializes accesses to the same key and uses a file to cross-check data correctness. The timestamp feature is not supported in a number of components, e.g. Merge, SingleDelete, DeleteRange, CompactionFilter, Readonly instance, secondary instance, SST file ingestion, transaction, etc. Therefore, db_stress should exit if user enables both timestamp and these features at the same time. The (currently) incompatible features can be found in `CheckAndSetOptionsForUserTimestamp`. This PR also fixes a bug triggered when timestamp is enabled together with `index_type=kBinarySearchWithFirstKey`. This bug fix will also be in another separate PR with more unit tests coverage. Fixing it here because I do not want to exclude the index type from crash test. Pull Request resolved: https://github.com/facebook/rocksdb/pull/8061 Test Plan: make crash_test_with_ts Reviewed By: jay-zhuang Differential Revision: D27056282 Pulled By: riversand963 fbshipit-source-id: c3e00ad1023fdb9ebbdf9601ec18270c5e2925a9
-
- 19 3月, 2021 1 次提交
-
-
由 Yanqin Jin 提交于
Summary: Pull Request resolved: https://github.com/facebook/rocksdb/pull/8077 Test Plan: Travis CI Reviewed By: zhichao-cao Differential Revision: D27178276 Pulled By: riversand963 fbshipit-source-id: 17911dcc2d5790eb396efcd7f90dea76a127cf15
-
- 16 3月, 2021 1 次提交
-
-
由 Yanqin Jin 提交于
Summary: As title. Pull Request resolved: https://github.com/facebook/rocksdb/pull/8054 Test Plan: make check Reviewed By: mrambacher Differential Revision: D27017955 Pulled By: riversand963 fbshipit-source-id: 829497d507bc89afbe982f8a8cf3555e52fd7098
-
- 10 3月, 2021 1 次提交
-
-
由 Ed rodriguez 提交于
Summary: Pull Request resolved: https://github.com/facebook/rocksdb/pull/7994 Reviewed By: jay-zhuang Differential Revision: D26904603 Pulled By: ajkr fbshipit-source-id: 0af92a51de895b40c7faaa4f0870b3f63279fe21
-
- 26 2月, 2021 1 次提交
-
-
由 Yanqin Jin 提交于
Summary: Allow applications to implement a custom compaction filter and pass it to BlobDB. The compaction filter's custom logic can operate on blobs. To do so, application needs to subclass `CompactionFilter` abstract class and implement `FilterV2()` method. Optionally, a method called `ShouldFilterBlobByKey()` can be implemented if application's custom logic rely solely on the key to make a decision without reading the blob, thus saving extra IO. Examples can be found in db/blob/db_blob_compaction_test.cc. Pull Request resolved: https://github.com/facebook/rocksdb/pull/7974 Test Plan: make check Reviewed By: ltamasi Differential Revision: D26509280 Pulled By: riversand963 fbshipit-source-id: 59f9ae5614c4359de32f4f2b16684193cc537b39
-
- 23 2月, 2021 1 次提交
-
-
由 Akanksha Mahajan 提交于
Summary: Extend VerifyFileChecksums API to verify blob files in case of use_file_checksum. Pull Request resolved: https://github.com/facebook/rocksdb/pull/7979 Test Plan: New unit test db_blob_corruption_test Reviewed By: ltamasi Differential Revision: D26534040 Pulled By: akankshamahajan15 fbshipit-source-id: 7dc5951a3df9d265ea1265e0122b43c966856ade
-
- 22 2月, 2021 1 次提交
-
-
由 mrambacher 提交于
Summary: I noticed tests frequently timing out on CircleCI when I submit a PR. I did some investigation and found the SeqAdvanceConcurrentTest suite (OneWriteQueue, TwoWriteQueues) tests were all taking a long time to complete (30 tests each taking at least 15K ms). This PR adds those test to the "slow reg" list in order to move them earlier in the execution sequence so that they are not the "long tail". For completeness, other tests that were also slow are: NumLevels/DBTestUniversalCompaction.UniversalCompactionTrivialMoveTest : 12 tests all taking 12K+ ms ReadSequentialFileTest with ReadaheadSize: 8 tests all 12K+ ms WriteUnpreparedTransactionTest.RecoveryTest : 2 tests at 22K+ ms DBBasicTest.EmptyFlush: 1 test at 35K+ ms RateLimiterTest.Rate: 1 test at 23K+ ms BackupableDBTest.ShareTableFilesWithChecksumsTransition: 1 test at 16K+ ms MulitThreadedDBTest.MultitThreaded: 78 tests at 10K+ ms TransactionStressTest.DeadlockStress: 7 tests at 11K+ ms DBBasicTestDeadline.IteratorDeadline: 3 tests at 10K+ ms No effort was made to determine why the tests were slow. Pull Request resolved: https://github.com/facebook/rocksdb/pull/7973 Reviewed By: jay-zhuang Differential Revision: D26519130 Pulled By: mrambacher fbshipit-source-id: 11555c9115acc207e45e210a7fc7f879170a3853
-
- 11 2月, 2021 1 次提交
-
-
由 Andrew Kryczka 提交于
Summary: Added support for detecting plugins linked in the "plugin/" directory and building them from our Makefile in a standardized way. See "plugin/README.md" for details. An example of a plugin that can be built in this way can be found in https://github.com/ajkr/dedupfs. There will be more to do in terms of making this process more convenient and adding support for CMake. Pull Request resolved: https://github.com/facebook/rocksdb/pull/7918 Test Plan: my own plugin (https://github.com/ajkr/dedupfs) and also heard this patch worked with ZenFS. Reviewed By: pdillinger Differential Revision: D26189969 Pulled By: ajkr fbshipit-source-id: 6624d4357d0ffbaedb42f0d12a3fcb737c78f758
-
- 10 2月, 2021 1 次提交
-
-
由 Yanqin Jin 提交于
Summary: Recent Github actions of format checking fail due to invalid location from where clang-format-diff.py is downloaded. Update the path to point to a stable, archived location. Pull Request resolved: https://github.com/facebook/rocksdb/pull/7944 Test Plan: manually check the result of Github action. Reviewed By: ltamasi Differential Revision: D26345066 Pulled By: riversand963 fbshipit-source-id: 2b1a58c2e59c2f1eb11202d321d2ea002cb0917e
-
- 02 2月, 2021 1 次提交
-
-
由 mrambacher 提交于
Summary: (Fixes a regression introduced in the build_version generation PR https://github.com/facebook/rocksdb/issues/7866 ) In the Makefile case, needed to ignore stderr on the tag (everywhere else was fine). In the CMAKE case, no GIT implies "changes" so that we use the system date rather than the empty GIT date. Pull Request resolved: https://github.com/facebook/rocksdb/pull/7916 Test Plan: Built in a tree that did not contain the ".git" directory. Validated that no errors appeared during the build process and that the build version date was not empty. Reviewed By: jay-zhuang Differential Revision: D26169203 Pulled By: mrambacher fbshipit-source-id: 3288a23b48d97efed5e5b38c9aefb3ef1153fa16
-
- 30 1月, 2021 1 次提交
-
-
由 Andrew Kryczka 提交于
Summary: This PR adds the foundation classes for key-value integrity protection and the first use case: protecting live updates from the source buffers added to `WriteBatch` through the destination buffer in `MemTable`. The width of the protection info is not yet configurable -- only eight bytes per key is supported. This PR allows users to enable protection by constructing `WriteBatch` with `protection_bytes_per_key == 8`. It does not yet expose a way for users to get integrity protection via other write APIs (e.g., `Put()`, `Merge()`, `Delete()`, etc.). The foundation classes (`ProtectionInfo.*`) embed the coverage info in their type, and provide `Protect.*()` and `Strip.*()` functions to navigate between types with different coverage. For making bytes per key configurable (for powers of two up to eight) in the future, these classes are templated on the unsigned integer type used to store the protection info. That integer contains the XOR'd result of hashes with independent seeds for all covered fields. For integer fields, the hash is computed on the raw unadjusted bytes, so the result is endian-dependent. The most significant bytes are truncated when the hash value (8 bytes) is wider than the protection integer. When `WriteBatch` is constructed with `protection_bytes_per_key == 8`, we hold a `ProtectionInfoKVOTC` (i.e., one that covers key, value, optype aka `ValueType`, timestamp, and CF ID) for each entry added to the batch. The protection info is generated from the original buffers passed by the user, as well as the original metadata generated internally. When writing to memtable, each entry is transformed to a `ProtectionInfoKVOTS` (i.e., dropping coverage of CF ID and adding coverage of sequence number), since at that point we know the sequence number, and have already selected a memtable corresponding to a particular CF. This protection info is verified once the entry is encoded in the `MemTable` buffer. Pull Request resolved: https://github.com/facebook/rocksdb/pull/7748 Test Plan: - an integration test to verify a wide variety of single-byte changes to the encoded `MemTable` buffer are caught - add to stress/crash test to verify it works in variety of configs/operations without intentional corruption - [deferred] unit tests for `ProtectionInfo.*` classes for edge cases like KV swap, `SliceParts` and `Slice` APIs are interchangeable, etc. Reviewed By: pdillinger Differential Revision: D25754492 Pulled By: ajkr fbshipit-source-id: e481bac6c03c2ab268be41359730f1ceb9964866
-
- 29 1月, 2021 1 次提交
-
-
由 mrambacher 提交于
Summary: Closes https://github.com/facebook/rocksdb/issues/7035 Changed how build_version.cc was generated: - Included the GIT tag/branch in the build_version file - Changed the "Build Date" to be: - If the GIT branch is "clean" (no changes), the date of the last git commit - If the branch is not clean, the current date - Added APIs to access the "build information", rather than accessing the strings directly. The build_version.cc file is now regenerated whenever the library objects are rebuilt. Verified that the built files remain the same size across builds on a "clean build" and the same information is reported by sst_dump --version Pull Request resolved: https://github.com/facebook/rocksdb/pull/7866 Reviewed By: pdillinger Differential Revision: D26086565 Pulled By: mrambacher fbshipit-source-id: 6fcbe47f6033989d5cf26a0ccb6dfdd9dd239d7f
-
- 14 1月, 2021 1 次提交
-
-
由 Adam Retter 提交于
Summary: Update the versions of the dependencies used for testing RocksJava. pdillinger Please can you add the following to your S3 bucket: 1. https://repo1.maven.org/maven2/junit/junit/4.13.1/junit-4.13.1.jar 2. https://repo1.maven.org/maven2/org/hamcrest/hamcrest/2.2/hamcrest-2.2.jar 3. https://repo1.maven.org/maven2/cglib/cglib/3.3.0/cglib-3.3.0.jar 4. https://repo1.maven.org/maven2/org/assertj/assertj-core/2.9.0/assertj-core-2.9.0.jar Thanks. Pull Request resolved: https://github.com/facebook/rocksdb/pull/7805 Reviewed By: akankshamahajan15 Differential Revision: D25906134 Pulled By: jay-zhuang fbshipit-source-id: 1c6c7d461a73abaff1796bb31f0ad90dcbdef1a0
-
- 08 1月, 2021 1 次提交
-
-
由 mrambacher 提交于
Summary: Fixed the following to now pass ASC checks: * `ttl_test` * `blob_db_test` * `backupable_db_test`, * `delete_scheduler_test` Pull Request resolved: https://github.com/facebook/rocksdb/pull/7834 Reviewed By: jay-zhuang Differential Revision: D25795398 Pulled By: ajkr fbshipit-source-id: a10037817deda4fc7cbb353a2e00b62ed89b6476
-