- 18 8月, 2018 3 次提交
-
-
由 jsteemann 提交于
Summary: Transaction has had methods to deal with SavePoints already, but was missing the PopSavePoint method provided by WriteBatch and WriteBatchWithIndex. This PR adds PopSavePoint to Transaction as well. Having the method on Transaction-level too is useful for applications that repeatedly execute a sequence of operations that normally succeed, but infrequently need to get rolled back. Using SavePoints here is sensible, but as operations normally succeed the application may pile up a lot of useless SavePoints inside a Transaction, leading to slightly increased memory usage for managing the unneeded SavePoints. Pull Request resolved: https://github.com/facebook/rocksdb/pull/4256 Differential Revision: D9326932 Pulled By: yiwu-arbug fbshipit-source-id: 53a0af18a6c7e87feff8a56f1f3eab9df7f371d6
-
由 Christian Esken 提交于
Summary: Closes https://github.com/facebook/rocksdb/issues/4195 CompactRangeOptions are available the CPP API, but not in the Java API. This PR adds CompactRangeOptions to the Java API and adds an overloaded compactRange() method. See https://github.com/facebook/rocksdb/issues/4195 for the original discussion. This change supports all fields of CompactRangeOptions, including the required enum converters in the JNI portal. Significant changes: - Make CompactRangeOptions available in the compactRange() for Java. - Deprecate other compactRange() methods that have individual option params, like in the CPP code. - Migrate rocksdb_compactrange_helper() to CompactRangeOptions. - Add Java unit tests for CompactRangeOptions. Pull Request resolved: https://github.com/facebook/rocksdb/pull/4220 Differential Revision: D9380007 Pulled By: sagar0 fbshipit-source-id: 6af6c334f221427f1997b33fb24c3986b092fed6
-
由 Andrey Zagrebin 提交于
Summary: Followup for #4266. There is one more place in **get_context.cc** where **MergeOperator::ShouldMerge** should be called with reversed list of operands. Pull Request resolved: https://github.com/facebook/rocksdb/pull/4284 Differential Revision: D9380008 Pulled By: sagar0 fbshipit-source-id: 70ec26e607e5b88465e1acbdcd6c6171bd76b9f2
-
- 17 8月, 2018 4 次提交
-
-
由 Fenggang Wu 提交于
Summary: Add `--data_block_index_type` and `--data_block_hash_table_util_ratio` option to `db_bench`. `--data_block_index_type` can be either of `binary` (default) or `binary_and_hash`; `--data_block_hash_table_util_ratio` will be a double. The default value is `0.75`. Pull Request resolved: https://github.com/facebook/rocksdb/pull/4281 Differential Revision: D9361476 Pulled By: fgwu fbshipit-source-id: dc53e01acef9db81b9eec5e8a96f3bc8ed718c10
-
由 Mikhail Antonov 提交于
Summary: Pull Request resolved: https://github.com/facebook/rocksdb/pull/4275 Reviewed By: yiwu-arbug Differential Revision: D9369766 Pulled By: mikhail-antonov fbshipit-source-id: d91b64c34cc1976b324a260767fce343fa32afde
-
由 Siying Dong 提交于
Summary: Right now, `ldb idump` may have memory out of control if there is a big range of tombstones. Add an option to cut maxinum number of keys in GetAllKeyVersions(), and push down --max_num_ikeys from ldb. Pull Request resolved: https://github.com/facebook/rocksdb/pull/4271 Differential Revision: D9369149 Pulled By: siying fbshipit-source-id: 7cbb797b7d2fa16573495a7e84937456d3ff25bf
-
由 Andrey Zagrebin 提交于
Summary: This PR addresses issue #3865 and implements the following approach to fix it: - adds `MergeContext::GetOperandsDirectionForward` and `MergeContext::GetOperandsDirectionBackward` to query merge operands in a specific order - `MergeContext::GetOperands` becomes a shortcut for `MergeContext::GetOperandsDirectionForward` - pass `MergeContext::GetOperandsDirectionBackward` to `MergeOperator::ShouldMerge` and document the order Pull Request resolved: https://github.com/facebook/rocksdb/pull/4266 Differential Revision: D9360750 Pulled By: sagar0 fbshipit-source-id: 20cb73ff017760b062ecdcf4382560767086e092
-
- 16 8月, 2018 1 次提交
-
-
由 Fenggang Wu 提交于
Summary: Add hash index support to data blocks, which helps to reduce the CPU utilization of point-lookup operations. This feature is backward compatible with the data block created without the hash index. It is disabled by default unless `BlockBasedTableOptions::data_block_index_type` is set to `data_block_index_type = kDataBlockBinaryAndHash.` The DB size would be bigger with the hash index option as a hash table is added at the end of each data block. If the hash utilization ratio is 1:1, the space overhead is one byte per key. The hash table utilization ratio is adjustable using `BlockBasedTableOptions::data_block_hash_table_util_ratio`. A lower utilization ratio will improve more on the point-lookup efficiency, but take more space too. Pull Request resolved: https://github.com/facebook/rocksdb/pull/4174 Differential Revision: D8965914 Pulled By: fgwu fbshipit-source-id: 1c6bae5d1fc39c80282d8890a72e9e67bc247198
-
- 15 8月, 2018 4 次提交
-
-
由 Zhichao Cao 提交于
Summary: The wrong options are used in the trace_analyzer_test, removed. The potential loses integer precision are fixed. Pass the specified testing case, make asan_check Pull Request resolved: https://github.com/facebook/rocksdb/pull/4274 Reviewed By: yiwu-arbug Differential Revision: D9327811 Pulled By: zhichao-cao fbshipit-source-id: d62cb18d6586503a490cd323bfc1c672b68b346e
-
由 jsteemann 提交于
Summary: Fixes compilation warnings (which are turned into compilation errors by default) when compiling with g++ option `-Wsuggest-override`. Pull Request resolved: https://github.com/facebook/rocksdb/pull/4272 Differential Revision: D9322556 Pulled By: siying fbshipit-source-id: abd57a29ec8f544bee77c0bb438f31be830b7244
-
由 Anand Ananthabhotla 提交于
Summary: In the OnTableFileCreation() listener, assert on various TableProperties only when file size > 0 bytes. The listener can get called even for 0 byte SSTs which have been deleted. Pull Request resolved: https://github.com/facebook/rocksdb/pull/4273 Differential Revision: D9322738 Pulled By: anand1976 fbshipit-source-id: 17cdfb3d0da946b9a158d7328e5db1c87973956b
-
由 Maysam Yabandeh 提交于
Summary: Stress tests currently cover format_version 2 and 3. The patch adds 4 as well. Pull Request resolved: https://github.com/facebook/rocksdb/pull/4265 Differential Revision: D9323185 Pulled By: maysamyabandeh fbshipit-source-id: 54d11e41ecae09bae14cadd7313f07c9a3db5a57
-
- 14 8月, 2018 4 次提交
-
-
由 Huachao Huang 提交于
Summary: Pull Request resolved: https://github.com/facebook/rocksdb/pull/4267 Differential Revision: D9309505 Pulled By: anand1976 fbshipit-source-id: eb9fee8037f4ff24dc1cdd5cc5ef41c231a03e1f
-
由 Siying Dong 提交于
Summary: Add a unit test to check that iterators release data blocks after it has moved away from it. Verify the same for compaction input iterators. Pull Request resolved: https://github.com/facebook/rocksdb/pull/4170 Differential Revision: D8962513 Pulled By: siying fbshipit-source-id: 05a5b604d7d29887fb488f2cda7286f554a14407
-
由 Zhichao Cao 提交于
Summary: A framework of trace analyzing for RocksDB After collecting the trace by using the tool of [PR #3837](https://github.com/facebook/rocksdb/pull/3837). User can use the Trace Analyzer to interpret, analyze, and characterize the collected workload. **Input:** 1. trace file 2. Whole keys space file **Statistics:** 1. Access count of each operation (Get, Put, Delete, SingleDelete, DeleteRange, Merge) in each column family. 2. Key hotness (access count) of each one 3. Key space separation based on given prefix 4. Key size distribution 5. Value size distribution if appliable 6. Top K accessed keys 7. QPS statistics including the average QPS and peak QPS 8. Top K accessed prefix 9. The query correlation analyzing, output the number of X after Y and the corresponding average time intervals **Output:** 1. key access heat map (either in the accessed key space or whole key space) 2. trace sequence file (interpret the raw trace file to line base text file for future use) 3. Time serial (The key space ID and its access time) 4. Key access count distritbution 5. Key size distribution 6. Value size distribution (in each intervals) 7. whole key space separation by the prefix 8. Accessed key space separation by the prefix 9. QPS of each operation and each column family 10. Top K QPS and their accessed prefix range **Test:** 1. Added the unit test of analyzing Get, Put, Delete, SingleDelete, DeleteRange, Merge 2. Generated the trace and analyze the trace **Implemented but not tested (due to the limitation of trace_replay):** 1. Analyzing Iterator, supporting Seek() and SeekForPrev() analyzing 2. Analyzing the number of Key found by Get **Future Work:** 1. Support execution time analyzing of each requests 2. Support cache hit situation and block read situation of Get Pull Request resolved: https://github.com/facebook/rocksdb/pull/4091 Differential Revision: D9256157 Pulled By: zhichao-cao fbshipit-source-id: f0ceacb7eedbc43a3eee6e85b76087d7832a8fe6
-
由 Yanqin Jin 提交于
Summary: Due to 4ea56b1b, we should also remove the assersion in stress test. This removal can be temporary, and we can add it back once we figure out the reason for the 0-byte SSTs. Pull Request resolved: https://github.com/facebook/rocksdb/pull/4268 Differential Revision: D9297186 Pulled By: riversand963 fbshipit-source-id: cebba9a68f42e815f8cf24471176d2cfdf962f63
-
- 12 8月, 2018 1 次提交
-
-
由 Anand Ananthabhotla 提交于
Summary: Revert this change. Not generating the OnTableFileCreated() notification for a 0 byte SST on flush breaks the assumption that every OnTableFileCreationStarted() notification is followed by a corresponding OnTableFileCreated(). Pull Request resolved: https://github.com/facebook/rocksdb/pull/4263 Differential Revision: D9285623 Pulled By: anand1976 fbshipit-source-id: 808c3dcd498b4b4f4ed4be947a29a24b2296aa8d
-
- 11 8月, 2018 4 次提交
-
-
由 Zhichao Cao 提交于
Summary: In the current trace_and replay, Get an WriteBatch are traced. This pull request track down the Seek() and SeekForPrev() to the trace file. <target_key, timestamp, column_family_id> are write to the file. Replay of Iterator is not supported in the current implementation. Tested with trace_analyzer. Pull Request resolved: https://github.com/facebook/rocksdb/pull/4228 Differential Revision: D9201381 Pulled By: zhichao-cao fbshipit-source-id: 6f9cc9cb3c20260af741bee065ec35c5c96354ab
-
由 Zhichao Cao 提交于
Summary: The pair of ROCKSDB_LITE condition inclusion is redundant, it is already inside the #ifndef ROCKSDB_LITE. Remove them to void confusion. Tested by make asan_check. Pull Request resolved: https://github.com/facebook/rocksdb/pull/4254 Differential Revision: D9281652 Pulled By: zhichao-cao fbshipit-source-id: 06bf7641ede71391f21f6a3fe37fbd13f0e2a43a
-
由 Maysam Yabandeh 提交于
Summary: After refactoring in https://github.com/facebook/rocksdb/pull/4158 the properties block is written after the index block. This breaks the existing logic in estimating the index size in partitioned indexes. The patch fixes that by using the accurate index block size, which is available since by the time we write the properties block, the index block is already written. The patch also fixes an issue in estimating the partition size with format_version=3 which was resulting into partitions smaller than the configured metadata_block_size. Pull Request resolved: https://github.com/facebook/rocksdb/pull/4259 Differential Revision: D9274454 Pulled By: maysamyabandeh fbshipit-source-id: c82d045505cca3e7ed1a44ee1eaa26e4f25a4272
-
由 Maysam Yabandeh 提交于
Summary: Fix the compile error in "make unity_test" caused by #3983. Pull Request resolved: https://github.com/facebook/rocksdb/pull/4257 Differential Revision: D9271740 Pulled By: maysamyabandeh fbshipit-source-id: 94e56d1675bf8bdc0e94439467eb4f40dd107517
-
- 10 8月, 2018 10 次提交
-
-
由 Yanqin Jin 提交于
Summary: TSAN fails due to comparison between signed int and unsigned long. Fix it by static_casting. Pull Request resolved: https://github.com/facebook/rocksdb/pull/4250 Differential Revision: D9256535 Pulled By: riversand963 fbshipit-source-id: c6bad23ff70c6d0ec58e2e85c401ce0ad45de609
-
由 Maysam Yabandeh 提交于
Summary: Given that index value is a BlockHandle, which is basically an <offset, size> pair we can apply delta encoding on the values. The first value at each index restart interval encoded the full BlockHandle but the rest encode only the size. Refer to IndexBlockIter::DecodeCurrentValue for the detail of the encoding. This reduces the index size which helps using the block cache more efficiently. The feature is enabled with using format_version 4. The feature comes with a bit of cpu overhead which should be paid back by the higher cache hits due to smaller index block size. Results with sysbench read-only using 4k blocks and using 16 index restart interval: Format 2: 19585 rocksdb read-only range=100 Format 3: 19569 rocksdb read-only range=100 Format 4: 19352 rocksdb read-only range=100 Pull Request resolved: https://github.com/facebook/rocksdb/pull/3983 Differential Revision: D8361343 Pulled By: maysamyabandeh fbshipit-source-id: f882ee082322acac32b0072e2bdbb0b5f854e651
-
由 Andrew Kryczka 提交于
Summary: I noticed we were building against zstd 1.3.0 which is missing optimizations that our fbcode customers have (they're on zstd 1.3.5). Ran `./build_tools/update_dependencies.sh` to catch us up. Omitted the changes it made for gcc-4.8 since it's broken. Pull Request resolved: https://github.com/facebook/rocksdb/pull/4244 Differential Revision: D9230111 Pulled By: ajkr fbshipit-source-id: 3e8ec1d8a961f98ec77c8c6580bde4caacf2d437
-
由 Anand Ananthabhotla 提交于
Summary: The error_filter.py script parses the output of the "Build and run" stage of continuous tests to check for errors. It is currently only detecting compile errors and not link errors. This change fixes that. Pull Request resolved: https://github.com/facebook/rocksdb/pull/4247 Differential Revision: D9233735 Pulled By: anand1976 fbshipit-source-id: 16e5a04950891cd9aba5cb3efcb6abc2a2e0d5ae
-
由 Georgios Bitzes 提交于
Summary: This change allows using std::string_view objects directly in the DB API: db->Get(some_string_view_object, ...); The conversion from std::string_view to rocksdb::Slice is done automatically, thanks to the added constructor. I'm stopping short of adding an implicit conversion operator from rocksdb::Slice to std::string_view, as I don't think that's a good idea for PinnableSlices. Pull Request resolved: https://github.com/facebook/rocksdb/pull/4242 Differential Revision: D9224134 Pulled By: anand1976 fbshipit-source-id: f50aad04dd0b01737907c0fb88d495c83a81f4e4
-
由 Dmitri Smirnov 提交于
Summary: Although delete scheduler implementation allows for the interface not to be supported, the delete_scheduler_test does not allow for that. Address compiler warnings Make sst_dump_test use test directory structure as the current execution directory may not be writiable. Pull Request resolved: https://github.com/facebook/rocksdb/pull/4221 Differential Revision: D9210152 Pulled By: siying fbshipit-source-id: 381a74511e969ecb8089d5c4b4df87dc30c8df63
-
由 Yanqin Jin 提交于
Summary: We add two subcommands `write_extern_sst` and `ingest_extern_sst` to ldb. This PR avoids changing existing code because we hope to cherry-pick to earlier releases to support compatibility check for external SST file ingestion. Pull Request resolved: https://github.com/facebook/rocksdb/pull/4205 Differential Revision: D9112711 Pulled By: riversand963 fbshipit-source-id: 7cae88380d4de86da8440230e87eca66755648e4
-
由 Andrew Kryczka 提交于
Summary: TSAN requires the code is built with -fPIC. This PR links against a libzstd built with -fPIC when necessary, which enables ZSTD compression to be used in TSAN builds. Pull Request resolved: https://github.com/facebook/rocksdb/pull/4249 Differential Revision: D9244746 Pulled By: ajkr fbshipit-source-id: 8c6a8fadd6c8643b2077afcbc3626779e1d73b63
-
由 Zhongyi Xie 提交于
Summary: In the current code, `error_msg` is pointing to the inner buffer of a temporary std::string object. When `error_msg` is used to construct the error message, that array is already released. This PR will fix this bug by copying the string to a local variable. Fixes https://github.com/facebook/rocksdb/issues/4239 Pull Request resolved: https://github.com/facebook/rocksdb/pull/4240 Differential Revision: D9204334 Pulled By: miasantreble fbshipit-source-id: 0ac599e166ae0a4ec413e32d8b8853d7c5fba878
-
由 Andrew Kryczka 提交于
Summary: db_bench's previous default compression level (-1) was not the default compression level in all libraries. In particular, in ZSTD negative values are valid compression levels, while ZSTD's default compression level is three. This PR changes db_bench's default to be RocksDB's library-independent default compression level (see #3895). I also changed a couple other flags to get their default values from an options object directly rather than hardcoding. Pull Request resolved: https://github.com/facebook/rocksdb/pull/4248 Differential Revision: D9235140 Pulled By: ajkr fbshipit-source-id: be4e0722d59fa1968832183db36d1d20fcf11e5b
-
- 09 8月, 2018 2 次提交
-
-
由 Maysam Yabandeh 提交于
Summary: If jemalloc is disabled or the API is incorrectly referenced (jemalloc api on windows have a prefix je_) memory usage is incorrectly reported for all block sizes. This is because sizeof(char) is always 1. sizeof() is calculated at compile time and *(char*) is char. The patch uses the size of the slice to fix that. Fixes #4245 Pull Request resolved: https://github.com/facebook/rocksdb/pull/4246 Differential Revision: D9233958 Pulled By: maysamyabandeh fbshipit-source-id: 9646933b24504e2814c7379f06a31148829c6b4e
-
由 Anand Ananthabhotla 提交于
Summary: Fix the nested quotes for CRASH_TEST_EXT_ARGS, as the generated json could not be parsed by the sandcastle job. Pull Request resolved: https://github.com/facebook/rocksdb/pull/4243 Differential Revision: D9228429 Pulled By: anand1976 fbshipit-source-id: 3c2bcac34870e377949d8a79c55e33b8363b25dd
-
- 08 8月, 2018 2 次提交
-
-
由 Jean-Marc Le Roux 提交于
Summary: sysmacros.h should be included in OS_ANDROID build as well otherwise the compile would complain: error: use of undeclared identifier 'major'. Fixes https://github.com/facebook/rocksdb/issues/4231 Pull Request resolved: https://github.com/facebook/rocksdb/pull/4232 Differential Revision: D9217350 Pulled By: maysamyabandeh fbshipit-source-id: 21f4b62dbbda3163120ac0b38b95d95d35d67dce
-
由 Maysam Yabandeh 提交于
Summary: The test has become complicated over the years and hard to reason about the corner cases that makes the test flaky. The patch simplifies the test and also fixes some probable synchronization issues. Pull Request resolved: https://github.com/facebook/rocksdb/pull/4235 Differential Revision: D9187995 Pulled By: maysamyabandeh fbshipit-source-id: 53c7b060f14367e5a9e361014578c26debfe3d27
-
- 07 8月, 2018 5 次提交
-
-
由 Huachao Huang 提交于
Summary: So that we can act accordingly on blob index entries Pull Request resolved: https://github.com/facebook/rocksdb/pull/4233 Differential Revision: D9190205 Pulled By: yiwu-arbug fbshipit-source-id: e5b84d5b41e44fa7a76762f1f7b0305369bb3a0c
-
由 Yi Wu 提交于
Summary: Add API to allow fetching expiration of a key with `Get()`. Pull Request resolved: https://github.com/facebook/rocksdb/pull/4227 Differential Revision: D9169897 Pulled By: yiwu-arbug fbshipit-source-id: 2a6f216c493dc75731ddcef1daa689b517fab31b
-
由 Yi Wu 提交于
Summary: There are two issues with `VisibleToActiveSnapshot`: 1. If there are no snapshots, `oldest_snapshot` will be 0 and `VisibleToActiveSnapshot` will always return true. Since the method is used to decide whether it is safe to delete obsolete files, obsolete file won't be able to delete in this case. 2. The `auto` keyword of `auto snapshots = db_impl_->snapshots()` translate to a copy of `const SnapshotList` instead of a reference. Since copy constructor of `SnapshotList` is not defined, using the copy may yield unexpected result. Issue 2 actually hide issue 1 from being catch by tests. During test `snapshots.empty()` can return false while it should actually be empty, and `snapshots.oldest()` return an invalid address, making `oldest_snapshot` being some random large number. The issue was originally reported by BlobDB early adopter at Kuaishou. Pull Request resolved: https://github.com/facebook/rocksdb/pull/4236 Differential Revision: D9188706 Pulled By: yiwu-arbug fbshipit-source-id: a0f2624b927cf9bf28c1bb534784fee5d106f5ea
-
由 Andrew Kryczka 提交于
Summary: - Add `--compression_max_dict_bytes` and `--compression_zstd_max_train_bytes` flags to stress test - Randomly enable/disable the above flags in crash test - Set `--compression_type=zstd` in FB-specific crash test runs Pull Request resolved: https://github.com/facebook/rocksdb/pull/4234 Differential Revision: D9187207 Pulled By: ajkr fbshipit-source-id: 8d78cf8d8e1165f2cd1c32e069b73726b5bc1fd2
-
由 Yi Wu 提交于
Summary: Cleanup TTLExtractor interface. The original purpose of it is to allow our users keep using existing `Write()` interface but allow it to accept TTL via `TTLExtractor`. However the interface is confusing. Will replace it with something like `WriteWithTTL(batch, ttl)` in the future. Pull Request resolved: https://github.com/facebook/rocksdb/pull/4229 Differential Revision: D9174390 Pulled By: yiwu-arbug fbshipit-source-id: 68201703d784408b851336ab4dd9b84188245b2d
-