- 03 10月, 2018 3 次提交
-
-
由 Igor Canadi 提交于
Summary: This is a conceptually simple change, but it touches many files to pass the allocator through function calls. We introduce CacheAllocator, which can be used by clients to configure custom allocator for cache blocks. Our motivation is to hook this up with folly's `JemallocNodumpAllocator` (https://github.com/facebook/folly/blob/f43ce6d6866b7b994b3019df561109afae050ebc/folly/experimental/JemallocNodumpAllocator.h), but there are many other possible use cases. Additionally, this commit cleans up memory allocation in `util/compression.h`, making sure that all allocations are wrapped in a unique_ptr as soon as possible. Pull Request resolved: https://github.com/facebook/rocksdb/pull/4437 Differential Revision: D10132814 Pulled By: yiwu-arbug fbshipit-source-id: be1343a4b69f6048df127939fea9bbc96969f564
-
由 Yanqin Jin 提交于
Summary: Before running CompactFilesTest.SentinelCompressionType, we should check whether zlib and snappy are supported. CompactFilesTest.SentinelCompressionType is a newly added test. Compilation and linking with different options, e.g. COMPILE_WITH_TSAN, COMPILE_WITH_ASAN, etc. lead to generation of different binaries. On the one hand, it's not clear why zlib or snappy is present under ASAN, but not under TSAN. On the other hand, changing the compilation flags for TSAN or ASAN seems a bigger change worth much more attention. To unblock the cont-runs, I suggest that we simply add these two checks at the beginning of the test, as we did for GeneralTableTest.ApproximateOffsetOfCompressed in table/table_test.cc. Future actions include invesigating the absence of zlib and snappy when compiling with TSAN, i.e. COMPILE_WITH_TSAN=1, if necessary. Pull Request resolved: https://github.com/facebook/rocksdb/pull/4443 Differential Revision: D10140935 Pulled By: riversand963 fbshipit-source-id: 62f96d1e685386accd2ef0b98f6f754d3fd67b3e
-
由 Jakub Cech 提交于
Summary: Adding IOTA Foundation to USERS.MD Pull Request resolved: https://github.com/facebook/rocksdb/pull/4436 Differential Revision: D10108142 Pulled By: sagar0 fbshipit-source-id: 948dc9f7169cec5c113ae347f1af765a41355aae
-
- 02 10月, 2018 2 次提交
-
-
由 Gihwan Oh 提交于
Summary: Add newline for readability Pull Request resolved: https://github.com/facebook/rocksdb/pull/4434 Differential Revision: D10127684 Pulled By: riversand963 fbshipit-source-id: 39f3ed7eaea655b6ff83474bc9f7616c6ad59107
-
由 Yanqin Jin 提交于
Summary: In DBCompactionTestWithParam::ManualLevelCompactionOutputPathId, there is a race condition between `DBTestBase::GetSstFileCount` and `DBImpl::PurgeObsoleteFiles`. The following graph explains why. ``` Timeline db_compact_test_t bg_flush_t bg_compact_t | [initiate bg flush and | start waiting] | flush | DeleteObsoleteFiles | [waken up by bg_flush_t which | signaled in DeleteObsoleteFiles] | | [initiate compaction and | start waiting] | | [compact, | set manual.done to true] | [signal at the end of | BackgroundCallFlush] | | [waken up by bg_flush_t | which signaled before | returning from | BackgroundCallFlush] | | Check manual.done is true | | GetSstFileCount <-- race condition --> PurgeObsoleteFiles V ``` Pull Request resolved: https://github.com/facebook/rocksdb/pull/4440 Differential Revision: D10122628 Pulled By: riversand963 fbshipit-source-id: 3ede73c39fee6ad804dc6ac1ed84759c7e63977f
-
- 01 10月, 2018 1 次提交
-
-
由 Andrew Kryczka 提交于
Summary: Previously `CompactFiles` with `CompressionType::kDisableCompressionOption` caused program to crash on assertion failure. This PR fixes the crash by adding support for that setting. Now, that setting will cause RocksDB to choose compression according to the column family's options. Pull Request resolved: https://github.com/facebook/rocksdb/pull/4438 Differential Revision: D10115761 Pulled By: ajkr fbshipit-source-id: a553c6fa76fa5b6f73b0d165d95640da6f454122
-
- 28 9月, 2018 2 次提交
-
-
由 Yi Wu 提交于
Summary: Introduce `RepeatableThread` utility to run task periodically in a separate thread. It is basically the same as the the same class in fbcode, and in addition provide a helper method to let tests mock time and trigger execution one at a time. We can use this class to replace `TimerQueue` in #4382 and `BlobDB`. Pull Request resolved: https://github.com/facebook/rocksdb/pull/4423 Differential Revision: D10020932 Pulled By: yiwu-arbug fbshipit-source-id: 3616bef108c39a33c92eedb1256de424b7c04087
-
由 JiYou 提交于
Summary: `FindFile()` and `FindFileInRange()` actually works as the same of `std::lower_bound()`. Use `std::lower_bound()` to reduce the repeated code. - change `FindFile()` and `FindFileInRange()` to use `std::lower_bound()` Signed-off-by: NJiYou <jiyou09@gmail.com> Pull Request resolved: https://github.com/facebook/rocksdb/pull/4372 Differential Revision: D9919677 Pulled By: ajkr fbshipit-source-id: f74aaa30e2f80e410e299c5a5bca4eaf2a7a26de
-
- 27 9月, 2018 3 次提交
-
-
由 Sagar Vemuri 提交于
Summary: The assert in PosixEnv::FileExists is currently based on the return value of `access` syscall. Instead it should be based on errno. Initially I wanted to remove this assert as [`access`](https://linux.die.net/man/2/access) can error out in a few other cases (like EROFS). But on thinking more it feels like the assert is doing the right thing ... its good to crash on EROFS, EFAULT, EINVAL, and other major filesystem related problems so that the user is immediately aware of the problems while testing. (I think it might be ok to crash on EIO as well, but there might be a specific reason why it was decided not to crash for EIO, and I don't have that context. So letting the letting the assert checks remain as is for now). Pull Request resolved: https://github.com/facebook/rocksdb/pull/4427 Differential Revision: D10037200 Pulled By: sagar0 fbshipit-source-id: 5cc96116a2e53cef701f444a8b5290576f311e51
-
由 Andrew Kryczka 提交于
Summary: I guess we didn't update this script when `--allow_concurrent_memtable_write` became true by default. Fixes #4413. Pull Request resolved: https://github.com/facebook/rocksdb/pull/4428 Differential Revision: D10036452 Pulled By: ajkr fbshipit-source-id: f464be0642bd096d9040f82cdc3eae614a902183
-
由 Yi Wu 提交于
Summary: Improve log handling when avoid_flush_during_recovery=true. 1. restore total_log_size_ after recovery, by summing up existing log sizes. Fixes #4253. 2. truncate the last existing log, since this log can contain preallocated space and it will be a waste to keep the space. It avoids a crash loop of user application cause a lot of log with non-trivial size being created and ultimately take up all disk space. Pull Request resolved: https://github.com/facebook/rocksdb/pull/4405 Differential Revision: D9953933 Pulled By: yiwu-arbug fbshipit-source-id: 967780fee8acec7f358b6eb65190fb4684f82e56
-
- 26 9月, 2018 2 次提交
-
-
由 Nikhil Benesch 提交于
Summary: The CollapsedRangeDelMap was entirely mishandling tombstones at the same sequence number when the tombstones did not have identical start and end keys. Such tombstones are common since 90fc4069, which causes tombstones to be split during compactions. For example, if the tombstone [a, c) @ 1 lies across a compaction boundary at b, it will be split into [a, b) @ 1 and [b, c) @ 1. Without this patch, the collapsed range deletion map would look like this: a -> 1 b -> 1 c -> 0 Notice how the b -> 1 entry is redundant. When the tombstones overlap, the problem is even worse. Consider tombstones [a, c) @ 1 and [b, d) @ 1, which produces this map without this patch: a -> 1 b -> 1 c -> 0 d -> 0 This map is corrupt, as a map can never contain adjacent sentinel (zero) entries. When the iterator advances from b to c, it will notice that c is a sentinel enty and skip to d--but d is also a sentinel entry! Asking what tombstone this iterator points to will trigger an assertion, as it is not pointing to a valid tombstone. /cc ajkr Pull Request resolved: https://github.com/facebook/rocksdb/pull/4424 Differential Revision: D10039248 Pulled By: abhimadan fbshipit-source-id: 6d737c1e88d60e80cf27286726627ba44463e7f4
-
由 Yi Wu 提交于
Summary: Update template of TARGETS file according to recent changes in #4371 , #4363 and https://github.com/facebook/rocksdb/commit/dbf44c314b4adf3276afc1ca797b88944ca3162c. Pull Request resolved: https://github.com/facebook/rocksdb/pull/4426 Differential Revision: D10025053 Pulled By: yiwu-arbug fbshipit-source-id: e6a0a702bfd401fc1af240ee446f5690f0bcd85d
-
- 22 9月, 2018 1 次提交
-
-
由 Abhishek Madan 提交于
Summary: Improve time measurements for AddTombstones to only include the call and not the VectorIterator setup. Also add a new add_tombstones_per_run flag to call AddTombstones multiple times per aggregator, which will help simulate more realistic workloads. Pull Request resolved: https://github.com/facebook/rocksdb/pull/4395 Differential Revision: D9996811 Pulled By: abhimadan fbshipit-source-id: 5865a95c323fbd9b3606493013664b4890fe5a02
-
- 21 9月, 2018 2 次提交
-
-
由 Yi Wu 提交于
Summary: Fix IO error on read not being handle and crashing the DB. With the fix we properly return the error. Pull Request resolved: https://github.com/facebook/rocksdb/pull/4410 Differential Revision: D9979246 Pulled By: yiwu-arbug fbshipit-source-id: 111a85675067a29c03cb60e9a34103f4ff636694
-
由 Anand Ananthabhotla 提交于
Summary: Make the CompactOnDeletionCollectorFactory class public, and provide methods to update the window size and deletion trigger params. These will take effect on subsequent created SST files. Pull Request resolved: https://github.com/facebook/rocksdb/pull/4403 Differential Revision: D9976857 Pulled By: anand1976 fbshipit-source-id: 31dbf0511c12fa2bb9b2a7ba620079e0ee09cf48
-
- 20 9月, 2018 3 次提交
-
-
由 Chen, You 提交于
Summary: Add Java API `getAggregatedLongProperty(final String property)` Pull Request resolved: https://github.com/facebook/rocksdb/pull/4379 Differential Revision: D9921463 Pulled By: sagar0 fbshipit-source-id: a02512e1b2aff4765a10b77de9a7bf7b1909d954
-
由 Abhishek Madan 提交于
Summary: If range tombstones are generated every few writes, the KeyGenerator's limit is now extended to account for the additional Next() calls. This is primarily important for `filluniquerandom` benchmarks that enforce the call limit. Pull Request resolved: https://github.com/facebook/rocksdb/pull/4404 Differential Revision: D9949326 Pulled By: abhimadan fbshipit-source-id: 0bdfeb2cad2098dc0b8b029236dab5e4bef25e38
-
由 Zhongyi Xie 提交于
Summary: This will fix the broken asan crash test: > Traceback (most recent call last): File "tools/db_crashtest.py", line 384, in <module> main() File "tools/db_crashtest.py", line 368, in main parser.add_argument("--" + k, type=type(v() if callable(v) else v)) File "tools/db_crashtest.py", line 59, in <lambda> "index_block_restart_interval": lambda: random.choice(1, 16), TypeError: choice() takes exactly 2 arguments (3 given) Pull Request resolved: https://github.com/facebook/rocksdb/pull/4397 Differential Revision: D9933041 Pulled By: miasantreble fbshipit-source-id: 10998e5bc6b6a5cea3e4088b18465affc246e639
-
- 19 9月, 2018 4 次提交
-
-
由 Maysam Yabandeh 提交于
Summary: The default for index_block_restart_interval is 1 but some use 16 in production. The patch extends crash test to test both values. Pull Request resolved: https://github.com/facebook/rocksdb/pull/4383 Differential Revision: D9887304 Pulled By: maysamyabandeh fbshipit-source-id: a8d00fea974a79ad563f9f4d9d7b069e9f746a8f
-
由 Fosco Marotto 提交于
Summary: Per #4387 this should address the validation error with the link tag. This is a quick fix, a future iteration could significantly upgrade the jekyll integration. Pull Request resolved: https://github.com/facebook/rocksdb/pull/4392 Differential Revision: D9923643 Pulled By: gfosco fbshipit-source-id: e7ed478e55c907add8319290326540e6e44fc0d6
-
由 Andrew Kryczka 提交于
Summary: Add a unit test for range collapsing when non-default comparator is used. This exposes the bug fixed in #4386. Pull Request resolved: https://github.com/facebook/rocksdb/pull/4388 Differential Revision: D9918252 Pulled By: ajkr fbshipit-source-id: 99501b96b251eab41791a7e33b27055ee36c5c39
-
由 jsteemann 提交于
Summary: The Comparator passed to CollapsedRangeDelMap was not used for operator less of the std::map `rep_` object contained in CollapsedRangeDelMap. So the map was always sorted using the default ByteWiseComparator, which seems wrong. Passing the specified Comparator through for usage in that map object fixes actual problems we were seeing with RangeDelete operations that do not delete keys as expected when using a custom Comparator. I found that the tests in current master crash when I run them locally, both with and without my patch, at the very same location. I therefore don't know if the patch breaks something else, but it seems to fix RangeDeletion issues in our product that uses RocksDB. Pull Request resolved: https://github.com/facebook/rocksdb/pull/4386 Differential Revision: D9916506 Pulled By: ajkr fbshipit-source-id: 27bff8c775831f089dde8c5289df7343d88b2d66
-
- 18 9月, 2018 5 次提交
-
-
由 Maysam Yabandeh 提交于
Summary: Value delta encoding in format_version 4 requires the differences between the size of two consecutive handles to be sent to BlockBuilder::Add. This applies not only to indexes on blocks but also the indexes on indexes and filters in partitioned indexes and filters respectively. The patch fixes a bug where the partitioned filters would encode the entire size of the handle rather than the difference of the size with the last size. Pull Request resolved: https://github.com/facebook/rocksdb/pull/4381 Differential Revision: D9879505 Pulled By: maysamyabandeh fbshipit-source-id: 27a22e49b482b927fbd5629dc310c46d63d4b6d1
-
由 Abhishek Madan 提交于
Summary: To measure the results of upcoming DeleteRange v2 work, this commit adds simple benchmarks for RangeDelAggregator. It measures the average time for AddTombstones and ShouldDelete calls. Using this to compare the results before #4014 and on the latest master (using the default arguments) produces the following results: Before #4014: ``` ======================= Results: ======================= AddTombstones: 1356.28 us ShouldDelete: 0.401732 us ``` Latest master: ``` ======================= Results: ======================= AddTombstones: 740.82 us ShouldDelete: 0.383271 us ``` Pull Request resolved: https://github.com/facebook/rocksdb/pull/4363 Differential Revision: D9881676 Pulled By: abhimadan fbshipit-source-id: 793e7d61aa4b9d47eb917bbcc03f08695b5e5442
-
由 Anand Ananthabhotla 提交于
Summary: 1. Add override keyword to overridden virtual functions in EventListener 2. Fix a memory corruption that can happen during DB shutdown when in read-only mode due to a background write error 3. Fix uninitialized buffers in error_handler_test.cc that cause valgrind to complain Pull Request resolved: https://github.com/facebook/rocksdb/pull/4375 Differential Revision: D9875779 Pulled By: anand1976 fbshipit-source-id: 022ede1edc01a9f7e21ecf4c61ef7d46545d0640
-
由 Andrew Kryczka 提交于
Summary: - Made stress test call `Flush()` periodically according to `--flush_one_in` flag. - Enabled by default in crash test. Pull Request resolved: https://github.com/facebook/rocksdb/pull/4368 Differential Revision: D9838593 Pulled By: ajkr fbshipit-source-id: fe5a6e49b36e5ea752acc3aa8be364f8ef34d9cc
-
由 Sagar Vemuri 提交于
Summary: This is a follow up to #4370. The earlier comment is not correct. Thanks to ajkr for pointing this out. Pull Request resolved: https://github.com/facebook/rocksdb/pull/4380 Differential Revision: D9874667 Pulled By: sagar0 fbshipit-source-id: f4e092d86b29c715258210b770643d367e38caae
-
- 16 9月, 2018 2 次提交
-
-
由 Anand Ananthabhotla 提交于
Summary: Including tools/trace_analyzer_tool.cc in rocksdb_lib was causing conflicts in dependent binaries due to duplicate gflag (other_prefix). Pull Request resolved: https://github.com/facebook/rocksdb/pull/4371 Differential Revision: D9846953 Pulled By: anand1976 fbshipit-source-id: 80b4aa36ab8428b8f6dceb896c45532684102709
-
由 Anand Ananthabhotla 提交于
Summary: This commit implements automatic recovery from a Status::NoSpace() error during background operations such as write callback, flush and compaction. The broad design is as follows - 1. Compaction errors are treated as soft errors and don't put the database in read-only mode. A compaction is delayed until enough free disk space is available to accomodate the compaction outputs, which is estimated based on the input size. This means that users can continue to write, and we rely on the WriteController to delay or stop writes if the compaction debt becomes too high due to persistent low disk space condition 2. Errors during write callback and flush are treated as hard errors, i.e the database is put in read-only mode and goes back to read-write only fater certain recovery actions are taken. 3. Both types of recovery rely on the SstFileManagerImpl to poll for sufficient disk space. We assume that there is a 1-1 mapping between an SFM and the underlying OS storage container. For cases where multiple DBs are hosted on a single storage container, the user is expected to allocate a single SFM instance and use the same one for all the DBs. If no SFM is specified by the user, DBImpl::Open() will allocate one, but this will be one per DB and each DB will recover independently. The recovery implemented by SFM is as follows - a) On the first occurance of an out of space error during compaction, subsequent compactions will be delayed until the disk free space check indicates enough available space. The required space is computed as the sum of input sizes. b) The free space check requirement will be removed once the amount of free space is greater than the size reserved by in progress compactions when the first error occured c) If the out of space error is a hard error, a background thread in SFM will poll for sufficient headroom before triggering the recovery of the database and putting it in write-only mode. The headroom is calculated as the sum of the write_buffer_size of all the DB instances associated with the SFM 4. EventListener callbacks will be called at the start and completion of automatic recovery. Users can disable the auto recov ery in the start callback, and later initiate it manually by calling DB::Resume() Todo: 1. More extensive testing 2. Add disk full condition to db_stress (follow-on PR) Pull Request resolved: https://github.com/facebook/rocksdb/pull/4164 Differential Revision: D9846378 Pulled By: anand1976 fbshipit-source-id: 80ea875dbd7f00205e19c82215ff6e37da10da4a
-
- 15 9月, 2018 6 次提交
-
-
由 Sagar Vemuri 提交于
Summary: AddressSanitizer: heap-use-after-free in std::__atomic_base<bool>::load(std::memory_order) const ==1798517==ABORTING ``` Pull Request resolved: https://github.com/facebook/rocksdb/pull/4370 Differential Revision: D9844146 Pulled By: sagar0 fbshipit-source-id: 18a2970b1d504b4f6c8fb04857f26e0f32124dd1
-
由 Dmitri Smirnov 提交于
Summary: Pull Request resolved: https://github.com/facebook/rocksdb/pull/4369 Differential Revision: D9844200 Pulled By: sagar0 fbshipit-source-id: 0d9f5f73b28234eaac55d3551ce4e2dc177af138
-
由 JiYou 提交于
Summary: Because `base_files` and `added_files` both are sorted, using a merge operation to these two sorted arrays is more effective. The complexity is reduced to linear time. - optmize the merge complexity. - move the `NDEBUG` of sorted `added_files` out of merge process. Signed-off-by: NJiYou <jiyou09@gmail.com> Pull Request resolved: https://github.com/facebook/rocksdb/pull/4366 Differential Revision: D9833592 Pulled By: ajkr fbshipit-source-id: dd32b67ebdca4c20e5e9546ab8082cecefe99fd0
-
由 Yanqin Jin 提交于
Summary: Pull Request resolved: https://github.com/facebook/rocksdb/pull/4361 Differential Revision: D9803723 Pulled By: riversand963 fbshipit-source-id: 5a0d4cd3e57fd195571dcd5822895ee00547fa6a
-
由 Yanqin Jin 提交于
Summary: Use `static_cast<type>(var)` instead of `(type)var`. Pull Request resolved: https://github.com/facebook/rocksdb/pull/4367 Differential Revision: D9833391 Pulled By: riversand963 fbshipit-source-id: 3d33fc2c290e7e0f3d1d45b256a881d1bc5a7df2
-
由 Constantin Belyaev 提交于
Summary: Now port/win_env.cc do check error for cross device link creation. Fixes #4364 Pull Request resolved: https://github.com/facebook/rocksdb/pull/4365 Differential Revision: D9833144 Pulled By: ajkr fbshipit-source-id: be7555e510f4b8d2196d843841606a6cfada7644
-
- 14 9月, 2018 3 次提交
-
-
由 Andrew Kryczka 提交于
Summary: The code is dead in RocksDB as `log::Reader::initial_offset_` is always zero. We should delete it so we don't have to maintain it like in #4359. Pull Request resolved: https://github.com/facebook/rocksdb/pull/4362 Differential Revision: D9817829 Pulled By: ajkr fbshipit-source-id: 474a2c679e5bd273b40608f3a5332931d9eefe6d
-
由 kckjn97 提交于
Summary: corrected the mistyped message. Pull Request resolved: https://github.com/facebook/rocksdb/pull/4341 Differential Revision: D9816571 Pulled By: ajkr fbshipit-source-id: 1df0424e981a01470a638a37b925c4133d59a48b
-
由 Vitaly Isaev 提交于
Summary: Please consider this small PR providing access to the `MemoryUsage::GetApproximateMemoryUsageByType` function in plain C API. Actually I'm working on Go application and now trying to investigate the reasons of high memory consumption (#4313). Go [wrappers](https://github.com/tecbot/gorocksdb) are built on the top of Rocksdb C API. According to the #706, `MemoryUsage::GetApproximateMemoryUsageByType` is considered as the best option to get database internal memory usage stats, but it wasn't supported in C API yet. Pull Request resolved: https://github.com/facebook/rocksdb/pull/4340 Differential Revision: D9655135 Pulled By: ajkr fbshipit-source-id: a3d2f3f47c143ae75862fbcca2f571ea1b49e14a
-
- 13 9月, 2018 1 次提交
-
-
由 Maysam Yabandeh 提交于
Summary: With #3983 the size of IndexBlockIter was increased. This had resulted in a regression on P50 latencies in one of our benchmarks. The patch reduces IndexBlockIter size be eliminating active_comparator_ field from the class. Pull Request resolved: https://github.com/facebook/rocksdb/pull/4358 Differential Revision: D9781737 Pulled By: maysamyabandeh fbshipit-source-id: 71e2b28d90ff0813db9e04b737ae73e185583c52
-