- 02 7月, 2021 1 次提交
-
-
由 Baptiste Lemaire 提交于
Summary: Implement an experimental feature called "MemPurge", which consists in purging "garbage" bytes out of a memtable and reuse the memtable struct instead of making it immutable and eventually flushing its content to storage. The prototype is by default deactivated and is not intended for use. It is intended for correctness and validation testing. At the moment, the "MemPurge" feature can be switched on by using the `options.experimental_allow_mempurge` flag. For this early stage, when the allow_mempurge flag is set to `true`, all the flush operations will be rerouted to perform a MemPurge. This is a temporary design decision that will give us the time to explore meaningful heuristics to use MemPurge at the right time for relevant workloads . Moreover, the current MemPurge operation only supports `Puts`, `Deletes`, `DeleteRange` operations, and handles `Iterators` as well as `CompactionFilter`s that are invoked at flush time . Three unit tests are added to `db_flush_test.cc` to test if MemPurge works correctly (and checks that the previously mentioned operations are fully supported thoroughly tested). One noticeable design decision is the timing of the MemPurge operation in the memtable workflow: for this prototype, the mempurge happens when the memtable is switched (and usually made immutable). This is an inefficient process because it implies that the entirety of the MemPurge operation happens while holding the db_mutex. Future commits will make the MemPurge operation a background task (akin to the regular flush operation) and aim at drastically enhancing the performance of this operation. The MemPurge is also not fully "WAL-compatible" yet, but when the WAL is full, or when the regular MemPurge operation fails (or when the purged memtable still needs to be flushed), a regular flush operation takes place. Later commits will also correct these behaviors. Pull Request resolved: https://github.com/facebook/rocksdb/pull/8454 Reviewed By: anand1976 Differential Revision: D29433971 Pulled By: bjlemaire fbshipit-source-id: 6af48213554e35048a7e03816955100a80a26dc5
-
- 18 5月, 2021 1 次提交
-
-
由 Stanislav Tkach 提交于
Summary: Pull Request resolved: https://github.com/facebook/rocksdb/pull/8302 Reviewed By: jay-zhuang Differential Revision: D28499262 Pulled By: ajkr fbshipit-source-id: 7b17b79af871d874dfca76db9bca0d640a6cd854
-
- 28 4月, 2021 2 次提交
-
-
由 Duarte Nunes 提交于
Summary: The C client is missing the`manual_wal_flush` option and the `flush_wal` API. Pull Request resolved: https://github.com/facebook/rocksdb/pull/8226 Reviewed By: ajkr Differential Revision: D28000869 Pulled By: jay-zhuang fbshipit-source-id: ed44937e7e7e75bc0dfa870a14147fbeef0c38f8
-
由 Sahir Hoda 提交于
Summary: New C API rocksdb_options_add_compact_on_deletion_collector_factory to expose NewCompactOnDeletionCollectorFactory Pull Request resolved: https://github.com/facebook/rocksdb/pull/8233 Reviewed By: mrambacher Differential Revision: D28018381 Pulled By: anand1976 fbshipit-source-id: 674c9ed902c91ff0d9f09e7a60c5f37b907604c6
-
- 23 4月, 2021 1 次提交
-
-
由 Sahir Hoda 提交于
Summary: Add new C APIs to create the JemallocNodumpAllocator and set it on a Cache object. `make test` passes with and without `DISABLE_JEMALLOC=1`. Pull Request resolved: https://github.com/facebook/rocksdb/pull/8178 Reviewed By: jay-zhuang Differential Revision: D27944631 Pulled By: ajkr fbshipit-source-id: 2531729aa285a8985c58f22f093c4d53029c4a7b
-
- 16 4月, 2021 1 次提交
-
-
由 mrambacher 提交于
Summary: Added the Blob option settings from the AdvancedColmnFamilyOptions to the C API. There are no tests for getting/setting options in the C API currently, hence no specific test plans. Should there be a some? Pull Request resolved: https://github.com/facebook/rocksdb/pull/8148 Reviewed By: ltamasi Differential Revision: D27568495 Pulled By: mrambacher fbshipit-source-id: 3a52b784467ea2c4bc58be5f75c5d41f0a5c55d6
-
- 10 4月, 2021 1 次提交
-
-
由 Sahir Hoda 提交于
Summary: Pull Request resolved: https://github.com/facebook/rocksdb/pull/8160 Reviewed By: riversand963 Differential Revision: D27672474 Pulled By: ajkr fbshipit-source-id: fdbbc3398f0b1d4cef6b68636e5caf369c34b3a7
-
- 27 3月, 2021 1 次提交
-
-
由 anand76 提交于
Summary: The snapshot structure returned by rocksdb_transaction_get_snapshot is supposed to be freed by calling rocksdb_free(), so allocate using malloc rather than new. Fixes https://github.com/facebook/rocksdb/issues/6112 Pull Request resolved: https://github.com/facebook/rocksdb/pull/8114 Reviewed By: akankshamahajan15 Differential Revision: D27362923 Pulled By: anand1976 fbshipit-source-id: e93a8b1ffe26dafbe22529907f72b796ae971214
-
- 20 3月, 2021 2 次提交
-
-
由 storagezhang 提交于
Summary: Pull Request resolved: https://github.com/facebook/rocksdb/pull/8068 Reviewed By: zhichao-cao Differential Revision: D27147685 Pulled By: riversand963 fbshipit-source-id: 5428b1c0142ecae17c977fba31a6d49b52983d1c
-
由 storagezhang 提交于
Summary: switch may not cover all branch in `db/c.cc`: ```c++ void rocksdb_options_set_access_hint_on_compaction_start( rocksdb_options_t* opt, int v) { switch(v) { case 0: opt->rep.access_hint_on_compaction_start = ROCKSDB_NAMESPACE::Options::NONE; break; case 1: opt->rep.access_hint_on_compaction_start = ROCKSDB_NAMESPACE::Options::NORMAL; break; case 2: opt->rep.access_hint_on_compaction_start = ROCKSDB_NAMESPACE::Options::SEQUENTIAL; break; case 3: opt->rep.access_hint_on_compaction_start = ROCKSDB_NAMESPACE::Options::WILLNEED; break; } } ``` Pull Request resolved: https://github.com/facebook/rocksdb/pull/8065 Reviewed By: riversand963 Differential Revision: D27102892 Pulled By: zhichao-cao fbshipit-source-id: ad1d20d192712878e61597311ba75b55df0066d7
-
- 20 2月, 2021 1 次提交
-
-
由 Andrew Kryczka 提交于
Summary: For dictionary compression, we need to collect some representative samples of the data to be compressed, which we use to either generate or train (when `CompressionOptions::zstd_max_train_bytes > 0`) a dictionary. Previously, the strategy was to buffer all the data blocks during flush, and up to the target file size during compaction. That strategy allowed us to randomly pick samples from as wide a range as possible that'd be guaranteed to land in a single output file. However, some users try to make huge files in memory-constrained environments, where this strategy can cause OOM. This PR introduces an option, `CompressionOptions::max_dict_buffer_bytes`, that limits how much data blocks are buffered before we switch to unbuffered mode (which means creating the per-SST dictionary, writing out the buffered data, and compressing/writing new blocks as soon as they are built). It is not strict as we currently buffer more than just data blocks -- also keys are buffered. But it does make a step towards giving users predictable memory usage. Related changes include: - Changed sampling for dictionary compression to select unique data blocks when there is limited availability of data blocks - Made use of `BlockBuilder::SwapAndReset()` to save an allocation+memcpy when buffering data blocks for building a dictionary - Changed `ParseBoolean()` to accept an input containing characters after the boolean. This is necessary since, with this PR, a value for `CompressionOptions::enabled` is no longer necessarily the final component in the `CompressionOptions` string. Pull Request resolved: https://github.com/facebook/rocksdb/pull/7970 Test Plan: - updated `CompressionOptions` unit tests to verify limit is respected (to the extent expected in the current implementation) in various scenarios of flush/compaction to bottommost/non-bottommost level - looked at jemalloc heap profiles right before and after switching to unbuffered mode during flush/compaction. Verified memory usage in buffering is proportional to the limit set. Reviewed By: pdillinger Differential Revision: D26467994 Pulled By: ajkr fbshipit-source-id: 3da4ef9fba59974e4ef40e40c01611002c861465
-
- 05 2月, 2021 1 次提交
-
-
由 Stanislav Tkach 提交于
Summary: Pull Request resolved: https://github.com/facebook/rocksdb/pull/7914 Reviewed By: jay-zhuang Differential Revision: D26184409 Pulled By: ajkr fbshipit-source-id: 8e30faac5223ec80c22e2b617af67775322065d8
-
- 07 1月, 2021 1 次提交
-
-
由 Adam Retter 提交于
Summary: Third batch of adding more tests to ASSERT_STATUS_CHECKED. * db_compaction_filter_test * db_compaction_test * db_dynamic_level_test * db_inplace_update_test * db_sst_test * db_tailing_iter_test * db_io_failure_test Also update GetApproximateSizes APIs to all return Status. Pull Request resolved: https://github.com/facebook/rocksdb/pull/7715 Reviewed By: jay-zhuang Differential Revision: D25806896 Pulled By: pdillinger fbshipit-source-id: 6cb9d62ba5a756c645812754c596ad3995d7c262
-
- 29 10月, 2020 1 次提交
-
-
由 Yanqin Jin 提交于
Summary: This is a PR generated **semi-automatically** by an internal tool to remove unused includes and `using` statements. Pull Request resolved: https://github.com/facebook/rocksdb/pull/7604 Test Plan: make check Reviewed By: ajkr Differential Revision: D24579392 Pulled By: riversand963 fbshipit-source-id: c4bfa6c6b08da1de186690d37eb73d8fff45aecd
-
- 17 10月, 2020 1 次提交
-
-
由 Stanislav Tkach 提交于
Summary: Pull Request resolved: https://github.com/facebook/rocksdb/pull/7501 Reviewed By: ltamasi Differential Revision: D24344109 Pulled By: pdillinger fbshipit-source-id: d9a2b1b1cc8c8d8a96f13b8ae6814380caa10c96
-
- 15 10月, 2020 1 次提交
-
-
由 Stanislav Tkach 提交于
Summary: Pull Request resolved: https://github.com/facebook/rocksdb/pull/7550 Reviewed By: jay-zhuang Differential Revision: D24315343 Pulled By: ajkr fbshipit-source-id: fc7855b630a50c00dcb940241942295932732f39
-
- 18 9月, 2020 1 次提交
-
-
由 Adam Retter 提交于
Summary: Expose from C++ API to Java API. Pull Request resolved: https://github.com/facebook/rocksdb/pull/7046 Reviewed By: riversand963 Differential Revision: D23726297 Pulled By: pdillinger fbshipit-source-id: fc66bf626ce6fe9797e7d021ac849eacab91bf6d
-
- 10 9月, 2020 1 次提交
-
-
由 Stanislav Tkach 提交于
Summary: Pull Request resolved: https://github.com/facebook/rocksdb/pull/7321 Reviewed By: ajkr Differential Revision: D23590160 fbshipit-source-id: 35d106e732ac37f674222759cdb1dbb31e005ca7
-
- 04 9月, 2020 1 次提交
-
-
Summary: This PR creates `rocksdb_open_column_families_with_ttl` which allows C API users to open a DBWithTLL with column families. Pull Request resolved: https://github.com/facebook/rocksdb/pull/7314 Reviewed By: cheng-chang Differential Revision: D23430287 Pulled By: ajkr fbshipit-source-id: 307aa21d170d1402653263a91f6f832ef76afba0
-
- 21 8月, 2020 1 次提交
-
-
由 Stanislav Tkach 提交于
Summary: Pull Request resolved: https://github.com/facebook/rocksdb/pull/7289 Reviewed By: akankshamahajan15 Differential Revision: D23252520 Pulled By: ajkr fbshipit-source-id: 85cea485a6dcaa1c67c32a83eb49a1b623966609
-
- 30 7月, 2020 1 次提交
-
-
由 codingsh 提交于
Summary: - https://github.com/rust-rocksdb/rust-rocksdb/pull/448 Pull Request resolved: https://github.com/facebook/rocksdb/pull/7191 Reviewed By: riversand963 Differential Revision: D22809066 Pulled By: ajkr fbshipit-source-id: 036939f9a28cacc3f677c318d1aed97fe5f4f85e
-
- 29 7月, 2020 1 次提交
-
-
由 codingsh 提交于
Summary: fixed - https://github.com/rust-rocksdb/rust-rocksdb/issues/447 - https://github.com/rust-rocksdb/rust-rocksdb/pull/448 Pull Request resolved: https://github.com/facebook/rocksdb/pull/7168 Reviewed By: cheng-chang Differential Revision: D22736013 Pulled By: ajkr fbshipit-source-id: fdd784aa75d26a367b9108b05ffdd94a2ae117d3
-
- 11 7月, 2020 1 次提交
-
-
由 Stanislav Tkach 提交于
Summary: Along with https://github.com/facebook/rocksdb/issues/6925 and https://github.com/facebook/rocksdb/issues/6998, this should add getters for all Options fields except several ones with non-trivial interface (for example rocksdb_options_set_min_level_to_compress). Pull Request resolved: https://github.com/facebook/rocksdb/pull/7094 Reviewed By: riversand963 Differential Revision: D22479800 Pulled By: pdillinger fbshipit-source-id: d14f305e12cfe268d07e0fe229d55cef299c792a
-
- 09 7月, 2020 1 次提交
-
-
由 rafael-aero 提交于
Summary: Pull Request resolved: https://github.com/facebook/rocksdb/pull/7092 Reviewed By: riversand963 Differential Revision: D22412323 Pulled By: ajkr fbshipit-source-id: 3fc1c63bb19a8cd2c0ae620800c28f199a7f494b
-
- 30 6月, 2020 1 次提交
-
-
由 Stanislav Tkach 提交于
Summary: Pull Request resolved: https://github.com/facebook/rocksdb/pull/7021 Reviewed By: ajkr Differential Revision: D22246297 Pulled By: pdillinger fbshipit-source-id: 81dfd0a49e4d5ce0c9f00772c17cca425757ea24
-
- 26 6月, 2020 1 次提交
-
-
由 Stanislav Tkach 提交于
Summary: Pull Request resolved: https://github.com/facebook/rocksdb/pull/6998 Reviewed By: ajkr Differential Revision: D22211700 Pulled By: pdillinger fbshipit-source-id: 1141c20527dee5e13205059bf8e83927063c4c1e
-
- 04 6月, 2020 2 次提交
-
-
由 Stanislav Tkach 提交于
Summary: Additionally I have extended the incomplete test added in the https://github.com/facebook/rocksdb/issues/6880. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6925 Reviewed By: ajkr Differential Revision: D21869788 Pulled By: pdillinger fbshipit-source-id: e9db80f259c57ca1bdcbc2c66cb938cb1ac26e48
-
由 Anatoly Zhmur 提交于
Summary: Added setting of zstd_max_train_bytes compression option parameter to c interop. rocksdb_options_set_bottommost_compression_options was using bool parameter and thus not exported, updated it to unsigned char and added to c.h as well. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6796 Reviewed By: cheng-chang Differential Revision: D21611471 Pulled By: ajkr fbshipit-source-id: caaaf153de934837ad9af283c7f8c025ff0b0cf5
-
- 03 6月, 2020 1 次提交
-
-
由 Stanislav Tkach 提交于
Summary: Pull Request resolved: https://github.com/facebook/rocksdb/pull/6880 Reviewed By: ajkr Differential Revision: D21842752 Pulled By: pdillinger fbshipit-source-id: eda326f551ddd9cb397681544b9e9799ea614e52
-
- 13 5月, 2020 1 次提交
-
-
由 Stanislav Tkach 提交于
Summary: Pull Request resolved: https://github.com/facebook/rocksdb/pull/6832 Reviewed By: cheng-chang Differential Revision: D21498186 Pulled By: riversand963 fbshipit-source-id: 66bb0d7c06af2bf0df3c6a09b61bca2fb81f2dd3
-
- 08 4月, 2020 1 次提交
-
-
由 Kirill Abrosimov 提交于
Summary: Few functions from options added to C-api Pull Request resolved: https://github.com/facebook/rocksdb/pull/5630 Reviewed By: anand1976 Differential Revision: D20896731 Pulled By: ltamasi fbshipit-source-id: e4215a58b3c2429ec44e3f0d0381cbf86700fb14
-
- 22 2月, 2020 1 次提交
-
-
由 Zaiyang Li 提交于
Summary: Adding a C API function to set `row_cache` on `rocksdb_options_t` as this functionality is missing. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6442 Differential Revision: D20036813 Pulled By: riversand963 fbshipit-source-id: c1fa95ea343345fbc1e57961d0d048e0e79be373
-
- 21 2月, 2020 1 次提交
-
-
由 sdong 提交于
Summary: When dynamically linking two binaries together, different builds of RocksDB from two sources might cause errors. To provide a tool for user to solve the problem, the RocksDB namespace is changed to a flag which can be overridden in build time. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6433 Test Plan: Build release, all and jtest. Try to build with ROCKSDB_NAMESPACE with another flag. Differential Revision: D19977691 fbshipit-source-id: aa7f2d0972e1c31d75339ac48478f34f6cfcfb3e
-
- 04 2月, 2020 1 次提交
-
-
由 Mike Kolupaev 提交于
Summary: When paranoid_checks is on, DBImpl::CheckConsistency() iterates over all sst files and calls Env::GetFileSize() for each of them. As far as I could understand, this is pretty arbitrary and doesn't affect correctness - if filesystem doesn't corrupt fsynced files, the file sizes will always match; if it does, it may as well corrupt contents as well as sizes, and rocksdb doesn't check contents on open. If there are thousands of sst files, getting all their sizes takes a while. If, on top of that, Env is overridden to use some remote storage instead of local filesystem, it can be *really* slow and overload the remote storage service. This PR adds an option to not do GetFileSize(); instead it does GetChildren() for parent directory to check that all the expected sst files are at least present, but doesn't check their sizes. We can't just disable paranoid_checks instead because paranoid_checks do a few other important things: make the DB read-only on write errors, print error messages on read errors, etc. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6353 Test Plan: ran the added sanity check unit test. Will try it out in a LogDevice test cluster where the GetFileSize() calls are causing a lot of trouble. Differential Revision: D19656425 Pulled By: al13n321 fbshipit-source-id: c2c421b367633033760d1f56747bad206d1fbf82
-
- 31 1月, 2020 1 次提交
-
-
由 Yanqin Jin 提交于
Summary: Unit test names, together with other components, are used to create log files during some internal testing. Overly long names cause infra failure due to file names being too long. Look for internal tests. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6352 Differential Revision: D19649307 Pulled By: riversand963 fbshipit-source-id: 6f29de096e33c0eaa87d9c8702f810eda50059e7
-
- 18 1月, 2020 1 次提交
-
-
由 Matt Bell 提交于
Summary: This PR adds a `rocksdb_options_set_atomic_flush` function to the C API. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6307 Differential Revision: D19451313 Pulled By: ltamasi fbshipit-source-id: 750495642ef55b1ea7e13477f85c38cd6574849c
-
- 11 1月, 2020 1 次提交
-
-
由 Qinfan Wu 提交于
Summary: This makes it easier to call the functions from Rust as otherwise they require mutable types. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6283 Differential Revision: D19349991 Pulled By: wqfish fbshipit-source-id: e8da7a75efe8cd97757baef8ca844a054f2519b4
-
- 08 1月, 2020 1 次提交
-
-
由 Yanqin Jin 提交于
Summary: Fix an error message when CURRENT is not found. Test plan (dev server) ``` make check ``` Pull Request resolved: https://github.com/facebook/rocksdb/pull/6264 Differential Revision: D19300699 Pulled By: riversand963 fbshipit-source-id: 303fa206386a125960ecca1dbdeff07422690caf
-
- 07 1月, 2020 1 次提交
-
-
由 Qinfan Wu 提交于
Summary: It seems that the C-API doesn't expose the range delete functionality at the moment, so add the API. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6259 Differential Revision: D19290320 Pulled By: pdillinger fbshipit-source-id: 3f403a4c3446d2042d55f1ece7cdc9c040f40c27
-
- 10 12月, 2019 1 次提交
-
-
由 sdong 提交于
Summary: Formatter somehow complains some recent lines changed. Apply them to make the formatter happy. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6138 Test Plan: See CI passes. Differential Revision: D18895950 fbshipit-source-id: 7d1696cf3e3a682bc10a30cdca748a23c6565255
-