- 13 5月, 2021 1 次提交
-
-
由 Jay Zhuang 提交于
Summary: And change the cmake build on macos with GFLAGS on to cover more cases. Pull Request resolved: https://github.com/facebook/rocksdb/pull/8289 Reviewed By: zhichao-cao Differential Revision: D28372467 Pulled By: jay-zhuang fbshipit-source-id: ad7fbe523c3fb135ef5281adbaf2070ca5d0873d
-
- 22 4月, 2021 1 次提交
-
-
由 mrambacher 提交于
Summary: For some compilers/environments (e.g. Clang, riscv64), we need to link against -latomic. Check if this is a requirement and add the library to the third-party libs if it is. Pull Request resolved: https://github.com/facebook/rocksdb/pull/8183 Reviewed By: pdillinger Differential Revision: D27773564 Pulled By: mrambacher fbshipit-source-id: 68e15d823144f83fb02221c7bf5b1e43323419bf
-
- 13 4月, 2021 1 次提交
-
-
由 Yanqin Jin 提交于
Summary: Before this PR, `get_iostats_context()` will silently return a nullptr if no thread_local support is detected. This can be the result of build_detect_platform's failure to compile the simple code snippet on certain platforms, as reported in https://github.com/facebook/mysql-5.6/issues/904. To be safe, we should fail the compilation if user does not opt out IOStatsContext and ROCKSDB_SUPPORT_THREAD_LOCAL is not defined. If RocksDB relies on c++11, can we just always use thread_local? It turns out there might be performance concerns (https://github.com/facebook/rocksdb/issues/5774), which is beyond the scope of this PR. We can revisit this later. Here, we stick to the original impl. Pull Request resolved: https://github.com/facebook/rocksdb/pull/8117 Reviewed By: ajkr Differential Revision: D27356847 Pulled By: riversand963 fbshipit-source-id: f7d5776842277598d8341b955febb601946801ae
-
- 07 4月, 2021 1 次提交
-
-
由 Peter Dillinger 提交于
Summary: A current limitation of backups is that you don't know the exact database state of when the backup was taken. With this new feature, you can at least inspect the backup's DB state without restoring it by opening it as a read-only DB. Rather than add something like OpenAsReadOnlyDB to the BackupEngine API, which would inhibit opening stackable DB implementations read-only (if/when their APIs support it), we instead provide a DB name and Env that can be used to open as a read-only DB. Possible follow-up work: * Add a version of GetBackupInfo for a single backup. * Let CreateNewBackup return the BackupID of the newly-created backup. Implementation details: Refactored ChrootFileSystem to split off new base class RemapFileSystem, which allows more general remapping of files. We use this base class to implement BackupEngineImpl::RemapSharedFileSystem. To minimize API impact, I decided to just add these fields `name_for_open` and `env_for_open` to those set by GetBackupInfo when include_file_details=true. Creating the RemapSharedFileSystem adds a bit to the memory consumption, perhaps unnecessarily in some cases, but this has been mitigated by (a) only initialize the RemapSharedFileSystem lazily when GetBackupInfo with include_file_details=true is called, and (b) using the existing `shared_ptr<FileInfo>` objects to hold most of the mapping data. To enhance API safety, RemapSharedFileSystem is wrapped by new ReadOnlyFileSystem which rejects any attempts to write. This uncovered a couple of places in which DB::OpenForReadOnly would write to the filesystem, so I fixed these. Added a release note because this affects logging. Additional minor refactoring in backupable_db.cc to support the new functionality. Pull Request resolved: https://github.com/facebook/rocksdb/pull/8142 Test Plan: new test (run with ASAN and UBSAN), added to stress test and ran it for a while with amplified backup_one_in Reviewed By: ajkr Differential Revision: D27535408 Pulled By: pdillinger fbshipit-source-id: 04666d310aa0261ef6b2385c43ca793ce1dfd148
-
- 24 3月, 2021 1 次提交
-
-
由 Yanqin Jin 提交于
Summary: As title. All core db implementations should stay in db_impl. Pull Request resolved: https://github.com/facebook/rocksdb/pull/8082 Test Plan: make check Reviewed By: ajkr Differential Revision: D27211442 Pulled By: riversand963 fbshipit-source-id: e0953fde75064740e899aaff7989ff033b7f5232
-
- 16 3月, 2021 1 次提交
-
-
由 Yanqin Jin 提交于
Summary: As title. Pull Request resolved: https://github.com/facebook/rocksdb/pull/8054 Test Plan: make check Reviewed By: mrambacher Differential Revision: D27017955 Pulled By: riversand963 fbshipit-source-id: 829497d507bc89afbe982f8a8cf3555e52fd7098
-
- 10 3月, 2021 3 次提交
-
-
由 Ed rodriguez 提交于
Summary: Pull Request resolved: https://github.com/facebook/rocksdb/pull/7994 Reviewed By: jay-zhuang Differential Revision: D26904603 Pulled By: ajkr fbshipit-source-id: 0af92a51de895b40c7faaa4f0870b3f63279fe21
-
由 Peter Dillinger 提交于
Summary: Removed confusing, awkward, and undocumented internal API ReadOneLine and replaced with very simple LineFileReader. In refactoring backupable_db.cc, this has the side benefit of removing the arbitrary cap on the size of backup metadata files. Also added Status::MustCheck to make it easy to mark a Status as "must check." Using this, I can ensure that after LineFileReader::ReadLine returns false the caller checks GetStatus(). Also removed some excessive conditional compilation in status.h Pull Request resolved: https://github.com/facebook/rocksdb/pull/8026 Test Plan: added unit test, and running tests with ASSERT_STATUS_CHECKED Reviewed By: mrambacher Differential Revision: D26831687 Pulled By: pdillinger fbshipit-source-id: ef749c265a7a26bb13cd44f6f0f97db2955f6f0f
-
由 xinyuliu 提交于
Summary: Add ${ARTIFACT_SUFFIX} to benchmark tool names to enable differentiating jemalloc and non-jemalloc versions. Pull Request resolved: https://github.com/facebook/rocksdb/pull/8016 Reviewed By: jay-zhuang Differential Revision: D26907007 Pulled By: ajkr fbshipit-source-id: 78d3b3372b5454d52d5b663ea982135ea9cf7bf8
-
- 27 2月, 2021 1 次提交
-
-
由 Peter Dillinger 提交于
Summary: This change only affects non-schema-critical aspects of the production candidate Ribbon filter. Specifically, it refines choice of internal configuration parameters based on inputs. The changes are minor enough that the schema tests in bloom_test, some of which depend on this, are unaffected. There are also some minor optimizations and refactorings. This would be a schema change for "smash" Ribbon, to fix some known issues with small filters, but "smash" Ribbon is not accessible in public APIs. Unit test CompactnessAndBacktrackAndFpRate updated to test small and medium-large filters. Run with --thoroughness=100 or so for much better detection power (not appropriate for continuous regression testing). Homogenous Ribbon: This change adds internally a Ribbon filter variant we call Homogeneous Ribbon, in collaboration with Stefan Walzer. The expected "result" value for every key is zero, instead of computed from a hash. Entropy for queries not to be false positives comes from free variables ("overhead") in the solution structure, which are populated pseudorandomly. Construction is slightly faster for not tracking result values, and never fails. Instead, FP rate can jump up whenever and whereever entries are packed too tightly. For small structures, we can choose overhead to make this FP rate jump unlikely, as seen in updated unit test CompactnessAndBacktrackAndFpRate. Unlike standard Ribbon, Homogeneous Ribbon seems to scale to arbitrary number of keys when accepting an FP rate penalty for small pockets of high FP rate in the structure. For example, 64-bit ribbon with 8 solution columns and 10% allocated space overhead for slots seems to achieve about 10.5% space overhead vs. information-theoretic minimum based on its observed FP rate with expected pockets of degradation. (FP rate is close to 1/256.) If targeting a higher FP rate with fewer solution columns, Homogeneous Ribbon can be even more space efficient, because the penalty from degradation is relatively smaller. If targeting a lower FP rate, Homogeneous Ribbon is less space efficient, as more allocated overhead is needed to keep the FP rate impact of degradation relatively under control. The new OptimizeHomogAtScale tool in ribbon_test helps to find these optimal allocation overheads for different numbers of solution columns. And Ribbon widths, with 128-bit Ribbon apparently cutting space overheads in half vs. 64-bit. Other misc item specifics: * Ribbon APIs in util/ribbon_config.h now provide configuration data for not just 5% construction failure rate (95% success), but also 50% and 0.1%. * Note that the Ribbon structure does not exhibit "threshold" behavior as standard Xor filter does, so there is a roughly fixed space penalty to cut construction failure rate in half. Thus, there isn't really an "almost sure" setting. * Although we can extrapolate settings for large filters, we don't have a good formula for configuring smaller filters (< 2^17 slots or so), and efforts to summarize with a formula have failed. Thus, small data is hard-coded from updated FindOccupancy tool. * Enhances ApproximateNumEntries for public API Ribbon using more precise data (new API GetNumToAdd), thus a more accurate but not perfect reversal of CalculateSpace. (bloom_test updated to expect the greater precision) * Move EndianSwapValue from coding.h to coding_lean.h to keep Ribbon code easily transferable from RocksDB * Add some missing 'const' to member functions * Small optimization to 128-bit BitParity * Small refactoring of BandingStorage in ribbon_alg.h to support Homogeneous Ribbon * CompactnessAndBacktrackAndFpRate now has an "expand" test: on construction failure, a possible alternative to re-seeding hash functions is simply to increase the number of slots (allocated space overhead) and try again with essentially the same hash values. (Start locations will be different roundings of the same scaled hash values--because fastrange not mod.) This seems to be as effective or more effective than re-seeding, as long as we increase the number of slots (m) by roughly m += m/w where w is the Ribbon width. This way, there is effectively an expansion by one slot for each ribbon-width window in the banding. (This approach assumes that getting "bad data" from your hash function is as unlikely as it naturally should be, e.g. no adversary.) * 32-bit and 16-bit Ribbon configurations are added to ribbon_test for understanding their behavior, e.g. with FindOccupancy. They are not considered useful at this time and not tested with CompactnessAndBacktrackAndFpRate. Pull Request resolved: https://github.com/facebook/rocksdb/pull/7879 Test Plan: unit test updates included Reviewed By: jay-zhuang Differential Revision: D26371245 Pulled By: pdillinger fbshipit-source-id: da6600d90a3785b99ad17a88b2a3027710b4ea3a
-
- 26 2月, 2021 1 次提交
-
-
由 Yanqin Jin 提交于
Summary: Allow applications to implement a custom compaction filter and pass it to BlobDB. The compaction filter's custom logic can operate on blobs. To do so, application needs to subclass `CompactionFilter` abstract class and implement `FilterV2()` method. Optionally, a method called `ShouldFilterBlobByKey()` can be implemented if application's custom logic rely solely on the key to make a decision without reading the blob, thus saving extra IO. Examples can be found in db/blob/db_blob_compaction_test.cc. Pull Request resolved: https://github.com/facebook/rocksdb/pull/7974 Test Plan: make check Reviewed By: ltamasi Differential Revision: D26509280 Pulled By: riversand963 fbshipit-source-id: 59f9ae5614c4359de32f4f2b16684193cc537b39
-
- 24 2月, 2021 1 次提交
-
-
由 sherriiiliu 提交于
Summary: WITH_GFLAGS option does not work on MSVC. I checked the usage of [CMAKE_DEPENDENT_OPTION](https://cmake.org/cmake/help/latest/module/CMakeDependentOption.html). It says if the `depends` condition is not true, it will set the `option` to the value given by `force` and hides the option from the user. Therefore, `CMAKE_DEPENDENT_OPTION(WITH_GFLAGS "build with GFlags" ON "NOT MSVC;NOT MINGW" OFF)` will hide WITH_GFLAGS option from user if it is running on MSVC or MINGW and always set WITH_GFLAGS to be OFF. To expose WITH_GFLAGS option to user, I removed CMAKE_DEPENDENT_OPTION and split the logic into if-else statements Pull Request resolved: https://github.com/facebook/rocksdb/pull/7990 Reviewed By: jay-zhuang Differential Revision: D26615755 Pulled By: ajkr fbshipit-source-id: 33ca39a73423d9516510c15aaf9efb5c4072cdf9
-
- 23 2月, 2021 1 次提交
-
-
由 Akanksha Mahajan 提交于
Summary: Extend VerifyFileChecksums API to verify blob files in case of use_file_checksum. Pull Request resolved: https://github.com/facebook/rocksdb/pull/7979 Test Plan: New unit test db_blob_corruption_test Reviewed By: ltamasi Differential Revision: D26534040 Pulled By: akankshamahajan15 fbshipit-source-id: 7dc5951a3df9d265ea1265e0122b43c966856ade
-
- 11 2月, 2021 1 次提交
-
-
由 Xavier Deguillard 提交于
Summary: With M1 macs being available, it is possible that RocksDB will be built on them, without the resulting artifacts to be intended for iOS, where a non-lite RocksDB is needed. It is not clear to me why the ROCKSDB_LITE cmake option isn't used for iOS consumer, so sending this pull request as a way to foster discussion and to find a path forward to get a full RocksDB build on M1. Pull Request resolved: https://github.com/facebook/rocksdb/pull/7943 Test Plan: Applied the following patch: ``` diff --git a/fbcode/opensource/fbcode_builder/manifests/rocksdb b/fbcode/opensource/fbcode_builder/manifests/rocksdb --- a/fbcode/opensource/fbcode_builder/manifests/rocksdb +++ b/fbcode/opensource/fbcode_builder/manifests/rocksdb @@ -2,8 +2,8 @@ name = rocksdb [download] -url = https://github.com/facebook/rocksdb/archive/v6.8.1.tar.gz -sha256 = ca192a06ed3bcb9f09060add7e9d0daee1ae7a8705a3d5ecbe41867c5e2796a2 +url = https://github.com/xavierd/rocksdb/archive/master.zip +sha256 = f93f3f92df66a8401659e35398749d5910b92bd9c14b8354a35ea8852865c422 [dependencies] lz4 @@ -11,7 +11,7 @@ [build] builder = cmake -subdir = rocksdb-6.8.1 +subdir = rocksdb-master [cmake.defines] WITH_SNAPPY=ON ``` And ran `getdeps build eden` on an M1 macbook. The build used to fail at link time due to some RocksDB symbols not being found, it now fails for another reason (x86_64 Rust symbols). Reviewed By: jay-zhuang Differential Revision: D26324049 Pulled By: xavierd fbshipit-source-id: 12d86f3395709c4c323f440844e3ae65672aef2d
-
- 02 2月, 2021 1 次提交
-
-
由 mrambacher 提交于
Summary: (Fixes a regression introduced in the build_version generation PR https://github.com/facebook/rocksdb/issues/7866 ) In the Makefile case, needed to ignore stderr on the tag (everywhere else was fine). In the CMAKE case, no GIT implies "changes" so that we use the system date rather than the empty GIT date. Pull Request resolved: https://github.com/facebook/rocksdb/pull/7916 Test Plan: Built in a tree that did not contain the ".git" directory. Validated that no errors appeared during the build process and that the build version date was not empty. Reviewed By: jay-zhuang Differential Revision: D26169203 Pulled By: mrambacher fbshipit-source-id: 3288a23b48d97efed5e5b38c9aefb3ef1153fa16
-
- 30 1月, 2021 1 次提交
-
-
由 Andrew Kryczka 提交于
Summary: This PR adds the foundation classes for key-value integrity protection and the first use case: protecting live updates from the source buffers added to `WriteBatch` through the destination buffer in `MemTable`. The width of the protection info is not yet configurable -- only eight bytes per key is supported. This PR allows users to enable protection by constructing `WriteBatch` with `protection_bytes_per_key == 8`. It does not yet expose a way for users to get integrity protection via other write APIs (e.g., `Put()`, `Merge()`, `Delete()`, etc.). The foundation classes (`ProtectionInfo.*`) embed the coverage info in their type, and provide `Protect.*()` and `Strip.*()` functions to navigate between types with different coverage. For making bytes per key configurable (for powers of two up to eight) in the future, these classes are templated on the unsigned integer type used to store the protection info. That integer contains the XOR'd result of hashes with independent seeds for all covered fields. For integer fields, the hash is computed on the raw unadjusted bytes, so the result is endian-dependent. The most significant bytes are truncated when the hash value (8 bytes) is wider than the protection integer. When `WriteBatch` is constructed with `protection_bytes_per_key == 8`, we hold a `ProtectionInfoKVOTC` (i.e., one that covers key, value, optype aka `ValueType`, timestamp, and CF ID) for each entry added to the batch. The protection info is generated from the original buffers passed by the user, as well as the original metadata generated internally. When writing to memtable, each entry is transformed to a `ProtectionInfoKVOTS` (i.e., dropping coverage of CF ID and adding coverage of sequence number), since at that point we know the sequence number, and have already selected a memtable corresponding to a particular CF. This protection info is verified once the entry is encoded in the `MemTable` buffer. Pull Request resolved: https://github.com/facebook/rocksdb/pull/7748 Test Plan: - an integration test to verify a wide variety of single-byte changes to the encoded `MemTable` buffer are caught - add to stress/crash test to verify it works in variety of configs/operations without intentional corruption - [deferred] unit tests for `ProtectionInfo.*` classes for edge cases like KV swap, `SliceParts` and `Slice` APIs are interchangeable, etc. Reviewed By: pdillinger Differential Revision: D25754492 Pulled By: ajkr fbshipit-source-id: e481bac6c03c2ab268be41359730f1ceb9964866
-
- 29 1月, 2021 1 次提交
-
-
由 mrambacher 提交于
Summary: Closes https://github.com/facebook/rocksdb/issues/7035 Changed how build_version.cc was generated: - Included the GIT tag/branch in the build_version file - Changed the "Build Date" to be: - If the GIT branch is "clean" (no changes), the date of the last git commit - If the branch is not clean, the current date - Added APIs to access the "build information", rather than accessing the strings directly. The build_version.cc file is now regenerated whenever the library objects are rebuilt. Verified that the built files remain the same size across builds on a "clean build" and the same information is reported by sst_dump --version Pull Request resolved: https://github.com/facebook/rocksdb/pull/7866 Reviewed By: pdillinger Differential Revision: D26086565 Pulled By: mrambacher fbshipit-source-id: 6fcbe47f6033989d5cf26a0ccb6dfdd9dd239d7f
-
- 26 1月, 2021 1 次提交
-
-
由 mrambacher 提交于
Summary: Introduces and uses a SystemClock class to RocksDB. This class contains the time-related functions of an Env and these functions can be redirected from the Env to the SystemClock. Many of the places that used an Env (Timer, PerfStepTimer, RepeatableThread, RateLimiter, WriteController) for time-related functions have been changed to use SystemClock instead. There are likely more places that can be changed, but this is a start to show what can/should be done. Over time it would be nice to migrate most (if not all) of the uses of the time functions from the Env to the SystemClock. There are several Env classes that implement these functions. Most of these have not been converted yet to SystemClock implementations; that will come in a subsequent PR. It would be good to unify many of the Mock Timer implementations, so that they behave similarly and be tested similarly (some override Sleep, some use a MockSleep, etc). Additionally, this change will allow new methods to be introduced to the SystemClock (like https://github.com/facebook/rocksdb/issues/7101 WaitFor) in a consistent manner across a smaller number of classes. Pull Request resolved: https://github.com/facebook/rocksdb/pull/7858 Reviewed By: pdillinger Differential Revision: D26006406 Pulled By: mrambacher fbshipit-source-id: ed10a8abbdab7ff2e23d69d85bd25b3e7e899e90
-
- 22 1月, 2021 1 次提交
-
-
由 Seeker 提交于
Summary: On Unix systems, `ARTIFACT_SUFFIX` was added to the static library `librocksdb.a` but not the shared library `librocksdb.so` Pull Request resolved: https://github.com/facebook/rocksdb/pull/7755 Reviewed By: ajkr Differential Revision: D25988550 Pulled By: jay-zhuang fbshipit-source-id: 8079f26802ac937d5a75cbd6d3c0544094df1b11
-
- 21 1月, 2021 1 次提交
-
-
由 Seeker 提交于
Summary: `CMAKE_SYSTEM_PROCESSOR` defaults to `arm64` on Apple Silicon starting with CMake 3.19.2; see https://cmake.org/cmake/help/latest/variable/CMAKE_HOST_SYSTEM_PROCESSOR.html#macos-platforms Pull Request resolved: https://github.com/facebook/rocksdb/pull/7883 Reviewed By: ajkr Differential Revision: D25967902 Pulled By: jay-zhuang fbshipit-source-id: 771bcb5fe845cf5f545c226691c0c41648186de4
-
- 23 12月, 2020 1 次提交
-
-
由 Sergei Petrunia 提交于
Summary: Range Locking - an implementation based on the locktree library - Add a RangeTreeLockManager and RangeTreeLockTracker which implement range locking using the locktree library. - Point locks are handled as locks on single-point ranges. - Add a unit test: range_locking_test Pull Request resolved: https://github.com/facebook/rocksdb/pull/7506 Reviewed By: akankshamahajan15 Differential Revision: D25320703 Pulled By: cheng-chang fbshipit-source-id: f86347384b42ba2b0257d67eca0f45f806b69da7
-
- 22 12月, 2020 1 次提交
-
-
由 Jay Zhuang 提交于
Summary: `block_cache_tracer_test` and `block_cache_trace_analyzer_test` are using the same test directory. Which causes build failure if these 2 tests are running in parallel, for example: https://app.circleci.com/pipelines/github/facebook/rocksdb/5211/workflows/8639afbe-9fec-43e2-a6a4-6d47ea9cbcbe/jobs/74598/tests Pull Request resolved: https://github.com/facebook/rocksdb/pull/7783 Reviewed By: akankshamahajan15 Differential Revision: D25656762 Pulled By: jay-zhuang fbshipit-source-id: 68aa020aa5b4b3bce324315edecb4e1a60cc18e6
-
- 10 12月, 2020 1 次提交
-
-
由 Sergei Petrunia 提交于
Summary: To be used for implementing Range Locking. Pull Request resolved: https://github.com/facebook/rocksdb/pull/7753 Reviewed By: zhichao-cao Differential Revision: D25378980 Pulled By: cheng-chang fbshipit-source-id: 801a9c5cd92a84654ca2586b73e8f69001e89320
-
- 12 11月, 2020 1 次提交
-
-
由 mrambacher 提交于
Summary: The Customizable class is an extension of the Configurable class and allows instances to be created by a name/ID. Classes that extend customizable can define their Type (e.g. "TableFactory", "Cache") and a method to instantiate them (TableFactory::CreateFromString). Customizable objects can be registered with the ObjectRegistry and created dynamically. Future PRs will make more types of objects extend Customizable. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6590 Reviewed By: cheng-chang Differential Revision: D24841553 Pulled By: zhichao-cao fbshipit-source-id: d0c2132bd932e971cbfe2c908ca2e5db30c5e155
-
- 26 10月, 2020 1 次提交
-
-
由 Peter Dillinger 提交于
Summary: This is intended as the first commit toward a near-optimal alternative to static Bloom filters for SSTs. Stephan Walzer and I have agreed upon the name "Ribbon" for a PHSF based on his linear system construction in "Efficient Gauss Elimination for Near-Quadratic Matrices with One Short Random Block per Row, with Applications" ("SGauss") and my much faster "on the fly" algorithm for gaussian elimination (or for this linear system, "banding"), which can be faster than peeling while also more compact and flexible. See util/ribbon_alg.h for more detailed introduction and background. RIBBON = Rapid Incremental Boolean Banding ON-the-fly This commit just adds generic (templatized) core algorithms and a basic unit test showing some features, including the ability to construct structures within 2.5% space overhead vs. information theoretic lower bound. (Compare to cache-local Bloom filter's ~50% space overhead -> ~30% reduction anticipated.) This commit does not include the storage scheme necessary to make queries fast, especially for filter queries, nor fractional "result bits", but there is some description already and those implementations will come soon. Nor does this commit add FilterPolicy support, for use in SST files, but that will also come soon. Pull Request resolved: https://github.com/facebook/rocksdb/pull/7491 Reviewed By: jay-zhuang Differential Revision: D24517954 Pulled By: pdillinger fbshipit-source-id: 0119ee597e250d7e0edd38ada2ba50d755606fa7
-
- 20 10月, 2020 1 次提交
-
-
由 Cheng Chang 提交于
Summary: In order to be able to introduce more locking protocols, we need to abstract out the locking subsystem in TransactionDB into a set of interfaces. PR https://github.com/facebook/rocksdb/pull/7013 introduces interface `LockTracker`. This PR is a follow up to take the first step to abstract out a `LockManager` interface. Further modifications to the interface may be needed when introducing the first implementation of range lock. But the idea here is to put the range lock implementation based on range tree under the `utilities/transactions/lock/range/range_tree`. Pull Request resolved: https://github.com/facebook/rocksdb/pull/7532 Test Plan: point_lock_manager_test Reviewed By: ajkr Differential Revision: D24238731 Pulled By: cheng-chang fbshipit-source-id: 2a9458cd8b3fb008d9529dbc4d3b28c24631f463
-
- 16 10月, 2020 1 次提交
-
-
由 Levi Tamasi 提交于
Summary: The patch adds blob file support to the `Get` API by extending `Version` so that whenever a blob reference is read from a file, the blob is retrieved from the corresponding blob file and passed back to the caller. (This is assuming the blob reference is valid and the blob file is actually part of the given `Version`.) It also introduces a cache of `BlobFileReader`s called `BlobFileCache` that enables sharing `BlobFileReader`s between callers. `BlobFileCache` uses the same backing cache as `TableCache`, so `max_open_files` (if specified) limits the total number of open (table + blob) files. TODO: proactively open/cache blob files and pin the cache handles of the readers in the metadata objects similarly to what `VersionBuilder::LoadTableHandlers` does for table files. Pull Request resolved: https://github.com/facebook/rocksdb/pull/7540 Test Plan: `make check` Reviewed By: riversand963 Differential Revision: D24260219 Pulled By: ltamasi fbshipit-source-id: a8a2a4f11d3d04d6082201b52184bc4d7b0857ba
-
- 08 10月, 2020 2 次提交
-
-
由 Levi Tamasi 提交于
Summary: The patch does some cleanup in and around the legacy `BlobLogReader` class: * It renames the class to `BlobLogSequentialReader` to emphasize that it is for sequentially iterating through blobs in a blob file, as opposed to doing random point reads using `BlobIndex`es (which is `BlobFileReader`'s jurisdiction). * It removes some dead code from the old BlobDB implementation that references `BlobLogReader` (namely the method `BlobFile::OpenRandomAccessReader`). * It cleans up some `#include`s and forward declarations. * It fixes some incorrect/outdated comments related to the reader class. * It adds a few assertions to the `Read` methods of the class. Pull Request resolved: https://github.com/facebook/rocksdb/pull/7517 Test Plan: `make check` Reviewed By: riversand963 Differential Revision: D24172611 Pulled By: ltamasi fbshipit-source-id: 43e2ae1eba5c3dd30c1070cb00f217edc45bd64f
-
由 Levi Tamasi 提交于
Summary: The patch adds a class called `BlobFileReader` that can be used to retrieve blobs using the information available in blob references (e.g. blob file number, offset, and size). This will come in handy when implementing blob support for `Get`, `MultiGet`, and iterators, and also for compaction/garbage collection. When a `BlobFileReader` object is created (using the factory method `Create`), it first checks whether the specified file is potentially valid by comparing the file size against the combined size of the blob file header and footer (files smaller than the threshold are considered malformed). Then, it opens the file, and reads and verifies the header and footer. The verification involves magic number/CRC checks as well as checking for unexpected header/footer fields, e.g. incorrect column family ID or TTL blob files. Blobs can be retrieved using `GetBlob`. `GetBlob` validates the offset and compression type passed by the caller (because of the presence of the header and footer, the specified offset cannot be too close to the start/end of the file; also, the compression type has to match the one in the blob file header), and retrieves and potentially verifies and uncompresses the blob. In particular, when `ReadOptions::verify_checksums` is set, `BlobFileReader` reads the blob record header as well (as opposed to just the blob itself) and verifies the key/value size, the key itself, as well as the CRC of the blob record header and the key/value pair. In addition, the patch exposes the compression type from `BlobIndex` (both using an accessor and via `DebugString`), and adds a blob file read latency histogram to `InternalStats` that can be used with `BlobFileReader`. Pull Request resolved: https://github.com/facebook/rocksdb/pull/7461 Test Plan: `make check` Reviewed By: riversand963 Differential Revision: D23999219 Pulled By: ltamasi fbshipit-source-id: deb6b1160d251258b308d5156e2ec063c3e12e5e
-
- 02 10月, 2020 2 次提交
-
-
由 Andrew Kryczka 提交于
Summary: This PR schedules a background thread (shared across all DB instances) to flush info log every ten seconds. This improves debuggability in case of RocksDB hanging since it ensures the log messages leading up to the hang will eventually become visible in the log. The bulk of this PR is moving monitoring/stats_dump_scheduler* to db/periodic_work_scheduler* and making the corresponding name changes since now the scheduler handles info log flushing, not just stats dumping. Pull Request resolved: https://github.com/facebook/rocksdb/pull/7488 Reviewed By: riversand963 Differential Revision: D24065165 Pulled By: ajkr fbshipit-source-id: 339c47a0ff43b79fdbd055fbd9fefbb6f9d8d3b5
-
由 sdong 提交于
Summary: Introduce an new option options.check_flush_compaction_key_order, by default set to true, which checks key order of flush and compaction, and fail the operation if the order is violated. Also did minor refactor hash checking code, which consolidates the hashing logic to a vlidation class, where the key ordering logic is added. Pull Request resolved: https://github.com/facebook/rocksdb/pull/7467 Test Plan: Add unit tests to validate the check can catch reordering in flush and compaction, and can be properly disabled. Reviewed By: riversand963 Differential Revision: D24010683 fbshipit-source-id: 8dd6292d2cda8006054e9ded7cfa4bf405f0527c
-
- 26 9月, 2020 1 次提交
-
-
由 Jay Zhuang 提交于
Summary: Pull Request resolved: https://github.com/facebook/rocksdb/pull/7332 Test Plan: `cmake .. -DASSERT_STATUS_CHECKED=1`, then run tests Reviewed By: zhichao-cao Differential Revision: D23437128 Pulled By: jay-zhuang fbshipit-source-id: 66fce57ae5d126e7a8b91c16a73c81438cf3b65e
-
- 24 9月, 2020 2 次提交
-
-
由 Akanksha Mahajan 提交于
Summary: Implement a parsing tool io_tracer_parser that takes IO trace file (binary file) with command line argument --io_trace_file and output file with --output_file and dumps the IO trace records in outputfile in human readable form. Also added unit test cases that generates IO trace records and calls io_tracer_parse to parse those records. Pull Request resolved: https://github.com/facebook/rocksdb/pull/7333 Test Plan: make check -j64, Add unit test cases. Reviewed By: anand1976 Differential Revision: D23772360 Pulled By: akankshamahajan15 fbshipit-source-id: 9c20519c189362e6663352d08863326f3e496271
-
由 Peter Dillinger 提交于
Summary: (a) own copy of kMicrosInSecond (b) out-of-line sync point code Pull Request resolved: https://github.com/facebook/rocksdb/pull/7426 Test Plan: FB internal Reviewed By: ajkr Differential Revision: D23861363 Pulled By: pdillinger fbshipit-source-id: de6b1621dca2f7391c5ff72bad04a7613dc27527
-
- 15 9月, 2020 1 次提交
-
-
由 mrambacher 提交于
Summary: This PR merges the functionality of making the ColumnFamilyOptions, TableFactory, and DBOptions into Configurable into a single PR, resolving any merge conflicts Pull Request resolved: https://github.com/facebook/rocksdb/pull/5753 Reviewed By: ajkr Differential Revision: D23385030 Pulled By: zhichao-cao fbshipit-source-id: 8b977a7731556230b9b8c5a081b98e49ee4f160a
-
- 10 9月, 2020 1 次提交
-
-
由 Jay Zhuang 提交于
Summary: To fix the cmake build with third_party libs, like: `mkdir build && cd build && cmake .. -DWITH_SNAPPY=1 && make` Error: ``` Undefined symbols for architecture x86_64: "snappy::RawCompress(char const*, unsigned long, char*, unsigned long*)" ``` Pull Request resolved: https://github.com/facebook/rocksdb/pull/7351 Reviewed By: pdillinger Differential Revision: D23553705 Pulled By: jay-zhuang fbshipit-source-id: 19b45c6763c7256107583e8af4c01d370ca06128
-
- 03 9月, 2020 1 次提交
-
-
由 Daniel Smith 提交于
Summary: This PR is set up to merge into master, but it would be great to get this into a patch release if possible. Pull Request resolved: https://github.com/facebook/rocksdb/pull/7334 Reviewed By: jay-zhuang Differential Revision: D23476624 Pulled By: pdillinger fbshipit-source-id: c6cc02ce06e779e1e174ab0f4748e557d2ce7bc6
-
- 28 8月, 2020 2 次提交
-
-
由 Jay Zhuang 提交于
Summary: A new file interface `SupportPrefetch()` is added. When the user overrides it to `false`, an internal prefetch buffer will be used for readahead. Useful for non-directIO but FS doesn't have readahead support. Pull Request resolved: https://github.com/facebook/rocksdb/pull/7312 Reviewed By: anand1976 Differential Revision: D23329847 Pulled By: jay-zhuang fbshipit-source-id: 71cd4ce6f4a820840294e4e6aec111ab76175527
-
由 Levi Tamasi 提交于
Summary: The patch adds a class called `BlobFileBuilder` that can be used to build and cut blob files in background jobs (flushes/compactions). The class enforces a value size threshold (`min_blob_size`; smaller blobs will be inlined in the LSM tree itself), and supports specifying a blob file size limit (`blob_file_size`), as well as compression (`blob_compression_type`) and checksums for blob files. It also keeps track of the generated blob files and their associated `BlobFileAddition` metadata, which can be applied as part of the background job's `VersionEdit`. Pull Request resolved: https://github.com/facebook/rocksdb/pull/7306 Test Plan: `make check` Reviewed By: riversand963 Differential Revision: D23298817 Pulled By: ltamasi fbshipit-source-id: 38f35d81dab1ba81f15236240612ec173d7f21b5
-
- 18 8月, 2020 1 次提交
-
-
由 Jay Zhuang 提交于
Summary: `db_bench` is not linked with thirdparty libs in cmake, even `-DWITH_*` is specified. Pull Request resolved: https://github.com/facebook/rocksdb/pull/7264 Test Plan: `$ mkdir build; cd build; cmake .. -DWITH_SNAPPY=1; make db_bench; ./db_bench` `$ cmake .. -DWITH_SNAPPY=1 -DWITH_LZ4; make db_bench; ./db_bench -compression_type=lz4` Reviewed By: zhichao-cao Differential Revision: D23165077 Pulled By: jay-zhuang fbshipit-source-id: 9c6fead31c41664a5c75ecd6469f47402fcb7d62
-