- 16 7月, 2021 5 次提交
-
-
由 zaorangyang 提交于
Summary: …e consistent with the member variables's Pull Request resolved: https://github.com/facebook/rocksdb/pull/8510 Reviewed By: ajkr Differential Revision: D29654067 Pulled By: mrambacher fbshipit-source-id: 908baaddfb20c266db7c5aca6a87971393d62ee6
-
由 Baptiste Lemaire 提交于
Summary: In this PR, `mempurge` is made compatible with the Write Ahead Log: in case of recovery, the DB is now capable of recovering the data that was "mempurged" and kept in the `imm()` list of immutable memtables. The twist was to add a uint64_t to the `memtable` struct to store the number of the earliest log file containing entries from the `memtable`. When a `Flush` operation is replaced with a `MemPurge`, the `VersionEdit` (which usually contains the new min log file number to pick up for recovery and the level 0 file path of the newly created SST file) is no longer appended to the manifest log, and every time the `deleteWal` method is called, a check is made on the list of immutable memtables. This PR also includes a unit test that verifies that no data is lost upon Reopening of the database when the mempurge feature is activated. This extensive unit test includes two column families, with valid data contained in the imm() at time of "crash"/reopening (recovery). Pull Request resolved: https://github.com/facebook/rocksdb/pull/8528 Reviewed By: pdillinger Differential Revision: D29701097 Pulled By: bjlemaire fbshipit-source-id: 072a900fb6ccc1edcf5eef6caf88f3060238edf9
-
由 longlijian 提交于
Summary: For more detail can reference the https://github.com/facebook/rocksdb/issues/6433 (https://github.com/facebook/rocksdb/pull/6433) Pull Request resolved: https://github.com/facebook/rocksdb/pull/8531 Reviewed By: siying Differential Revision: D29717057 Pulled By: ajkr fbshipit-source-id: 3ccad9501e5612590e54a7cf8c447118f323c7f4
-
由 Peter Dillinger 提交于
Summary: Some bits are mutated and read while holding a lock, other immutable bits (esp. secondary cache compatibility) can be read by arbitrary threads without holding a lock. AFAIK, this doesn't cause an issue on any architecture we care about, because you will get some legitimate version of the value that includes the initialization, as long as synchronization guarantees the initialization happens before the read. I've only seen this in https://github.com/facebook/rocksdb/issues/8538 so far, but it should be fixed regardless. Otherwise, we'll surely get these false reports again some time. Pull Request resolved: https://github.com/facebook/rocksdb/pull/8539 Test Plan: some local TSAN test runs and in CircleCI Reviewed By: zhichao-cao Differential Revision: D29720262 Pulled By: pdillinger fbshipit-source-id: 365fd7e565577c648815161f71b339bcb5ce12d5
-
由 Jay Zhuang 提交于
Summary: Some cmake and test configuration are set in pre-steps enviroment variables. Add the missing steps. Pull Request resolved: https://github.com/facebook/rocksdb/pull/8524 Test Plan: CI pass Reviewed By: siying Differential Revision: D29682731 Pulled By: jay-zhuang fbshipit-source-id: afda1acf6a7b76989db450442b0b27f387388b9d
-
- 15 7月, 2021 1 次提交
-
-
由 longlijian 提交于
Summary: The removed function in this PR, just only have declared and dose not have any reference used. Pull Request resolved: https://github.com/facebook/rocksdb/pull/8508 Reviewed By: mrambacher Differential Revision: D29649033 Pulled By: jay-zhuang fbshipit-source-id: df98143b73d6c184a2a60c9f7ea2548a065ee35d
-
- 14 7月, 2021 2 次提交
-
-
由 hongrubb 提交于
Summary: This PR is for https://github.com/facebook/rocksdb/issues/8453 We need to update `s = biter.status();` when `biter.status().IsIncomplete()` is true. By doing this, can fix the problem in issue. Besides, we still need to update `db_statistics` in `get_context.ReportCounters()` before return back. Pull Request resolved: https://github.com/facebook/rocksdb/pull/8485 Reviewed By: jay-zhuang Differential Revision: D29604835 Pulled By: ajkr fbshipit-source-id: c7f2f1cd058223ce1b507ec05d57cf264b9c9710
-
由 sherriiiliu 提交于
Summary: Fixed a few MSVC (VCToolsVersion=14.0) build errors and warnings * `DEFINE_string` is a macro and VC compiler complains that it cannot put [ifdef-inside-define](https://stackoverflow.com/questions/5586429/ifdef-inside-define) * `sleep()` is not a recognizable function. Use `FLAGS_env->SleepForMicroseconds` instead * Define precise type in comparison to avoid mismatch warning Pull Request resolved: https://github.com/facebook/rocksdb/pull/8519 Reviewed By: jay-zhuang Differential Revision: D29683086 fbshipit-source-id: 8c80941472089f8daba84ae29597e75e603850e4
-
- 13 7月, 2021 7 次提交
-
-
由 dependabot[bot] 提交于
Summary: Bumps [addressable](https://github.com/sporkmonger/addressable) from 2.7.0 to 2.8.0. <details> <summary>Changelog</summary> <p><em>Sourced from <a href="https://github.com/sporkmonger/addressable/blob/main/CHANGELOG.md">addressable's changelog</a>.</em></p> <blockquote> <h1>Addressable 2.8.0</h1> <ul> <li>fixes ReDoS vulnerability in Addressable::Template#match</li> <li>no longer replaces <code>+</code> with spaces in queries for non-http(s) schemes</li> <li>fixed encoding ipv6 literals</li> <li>the <code>:compacted</code> flag for <code>normalized_query</code> now dedupes parameters</li> <li>fix broken <code>escape_component</code> alias</li> <li>dropping support for Ruby 2.0 and 2.1</li> <li>adding Ruby 3.0 compatibility for development tasks</li> <li>drop support for <code>rack-mount</code> and remove Addressable::Template#generate</li> <li>performance improvements</li> <li>switch CI/CD to GitHub Actions</li> </ul> </blockquote> </details> <details> <summary>Commits</summary> <ul> <li><a href="https://github.com/sporkmonger/addressable/commit/6469a232c0f1892809ff66737370c765d574e16c"><code>6469a23</code></a> Updating gemspec again</li> <li><a href="https://github.com/sporkmonger/addressable/commit/24336385de0261571b3adaad0431459edb420c79"><code>2433638</code></a> Merge branch 'main' of github.com:sporkmonger/addressable into main</li> <li><a href="https://github.com/sporkmonger/addressable/commit/e9c76b889789c75d7073c17b0ab557635d3f6704"><code>e9c76b8</code></a> Merge pull request <a href="https://github-redirect.dependabot.com/sporkmonger/addressable/issues/378">https://github.com/facebook/rocksdb/issues/378</a> from ashmaroli/flat-map</li> <li><a href="https://github.com/sporkmonger/addressable/commit/56c5cf7ece9223ff4240e07078cc26d3adbbbd30"><code>56c5cf7</code></a> Update the gemspec</li> <li><a href="https://github.com/sporkmonger/addressable/commit/c1fed1ca0a44c448e74d761fd44ed94869199807"><code>c1fed1c</code></a> Require a non-vulnerable rake</li> <li><a href="https://github.com/sporkmonger/addressable/commit/0d8a3127e35886ce9284810a7f2438bff6b43cbc"><code>0d8a312</code></a> Adding note about ReDoS vulnerability</li> <li><a href="https://github.com/sporkmonger/addressable/commit/89c76130ce255c601f642a018cb5fb5a80e679a7"><code>89c7613</code></a> Merge branch 'template-regexp' into main</li> <li><a href="https://github.com/sporkmonger/addressable/commit/cf8884f815c96b646c796f707bf768cf6eb65543"><code>cf8884f</code></a> Note about alias fix</li> <li><a href="https://github.com/sporkmonger/addressable/commit/bb03f7112e8e478240a0f96e1cc7428159b41586"><code>bb03f71</code></a> Merge pull request <a href="https://github-redirect.dependabot.com/sporkmonger/addressable/issues/371">https://github.com/facebook/rocksdb/issues/371</a> from charleystran/add_missing_encode_component_doc_entry</li> <li><a href="https://github.com/sporkmonger/addressable/commit/6d1d8094a66cbf932ecf69db6850bc9edaf86de0"><code>6d1d809</code></a> Adding note about :compacted normalization</li> <li>Additional commits viewable in <a href="https://github.com/sporkmonger/addressable/compare/addressable-2.7.0...addressable-2.8.0">compare view</a></li> </ul> </details> <br /> [![Dependabot compatibility score](https://dependabot-badges.githubapp.com/badges/compatibility_score?dependency-name=addressable&package-manager=bundler&previous-version=2.7.0&new-version=2.8.0)](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores) Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting `dependabot rebase`. [//]: # (dependabot-automerge-start) [//]: # (dependabot-automerge-end) --- <details> <summary>Dependabot commands and options</summary> <br /> You can trigger Dependabot actions by commenting on this PR: - `dependabot rebase` will rebase this PR - `dependabot recreate` will recreate this PR, overwriting any edits that have been made to it - `dependabot merge` will merge this PR after your CI passes on it - `dependabot squash and merge` will squash and merge this PR after your CI passes on it - `dependabot cancel merge` will cancel a previously requested merge and block automerging - `dependabot reopen` will reopen this PR if it is closed - `dependabot close` will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually - `dependabot ignore this major version` will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself) - `dependabot ignore this minor version` will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself) - `dependabot ignore this dependency` will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself) - `dependabot use these labels` will set the current labels as the default for future PRs for this repo and language - `dependabot use these reviewers` will set the current reviewers as the default for future PRs for this repo and language - `dependabot use these assignees` will set the current assignees as the default for future PRs for this repo and language - `dependabot use this milestone` will set the current milestone as the default for future PRs for this repo and language You can disable automated security fix PRs for this repo from the [Security Alerts page](https://github.com/facebook/rocksdb/network/alerts). </details> Pull Request resolved: https://github.com/facebook/rocksdb/pull/8515 Reviewed By: jay-zhuang Differential Revision: D29668988 Pulled By: ajkr fbshipit-source-id: c4b7abd4a879a7b562cb8ba745088dba6644f503
-
由 Peter Dillinger 提交于
Summary: MyRocks apparently uses valgrind to check for unreachable unfreed data, which is stricter than our valgrind checks. Internal ref: D29257815 This patch adds valgrind support to STATIC_AVOID_DESTRUCTION so that it's not reported with those stricter checks. Pull Request resolved: https://github.com/facebook/rocksdb/pull/8503 Test Plan: make valgrind_test Also, with modified VALGRIND_OPTS (see Makefile), more kinds of failures seen before than after this commit. Reviewed By: ajkr, yizhang82 Differential Revision: D29597784 Pulled By: pdillinger fbshipit-source-id: 360de157a176aec4d1be99ca20d160ecd47c0873
-
由 mrambacher 提交于
Summary: 1. Fix printing of stats when there are no writes (wamp=0). Previously had a div0 error 2. Added multireadrandom command as a valid target 3. Added ability to pass additional command line options to db_bench. Now can say things like benchmark.sh readrandom --mmap_read and the option will be passed to db_bench. Pull Request resolved: https://github.com/facebook/rocksdb/pull/8346 Reviewed By: zhichao-cao Differential Revision: D29500436 Pulled By: mrambacher fbshipit-source-id: 54e90708aae9133be3a903e35efdf8f8abbd86fa
-
由 bjlemaire 提交于
Summary: The MemPurge output status can either be an Abort if the mempurge is aborted due to the new_mem memtable reaching more than the target capacity (currently 60%), or for other reasons. As a result, in the log, we want to differentiate between an abort status, which in this PR only leads to a ROCKS_LOG_INFO, and any other status, which in this PR leads to a ROCKS_LOG_WARN. Pull Request resolved: https://github.com/facebook/rocksdb/pull/8514 Reviewed By: pdillinger Differential Revision: D29662446 Pulled By: bjlemaire fbshipit-source-id: c9bec8e238ebc7ecb14fbbddf580e6887e281c16
-
由 Dmitry Vorobev 提交于
Summary: When db is open as secondary, there are basically 2 step process: 1) Collect column families from wal log 2) Apply changes to Memtable In case primary db is TransactionDB instance, wal log will contain some additional data, like noop, etc. ColumnFamilyCollector doesn't implement methods to handle these, so it fails to open a wal log written by TransactionDB. (Everything works fine with standard DB::Open). Memtable recovery process knows how to handle such wal logs, so only missing piece seems to be ColumnFamilyCollector. Pull Request resolved: https://github.com/facebook/rocksdb/pull/8456 Reviewed By: ajkr Differential Revision: D29455945 Pulled By: mrambacher fbshipit-source-id: 5b29560fcbc008e17e95d0dc4b07558f3d63e26f
-
由 Myth 提交于
Summary: A very simple change :) Pull Request resolved: https://github.com/facebook/rocksdb/pull/8483 Reviewed By: ajkr Differential Revision: D29558904 Pulled By: mrambacher fbshipit-source-id: bbe68c20c861103726cb6231ca3fb8fbe1e5a546
-
由 mrambacher 提交于
Summary: Add ability to treat FlushBlockPolicyFactory as a Customizable. Pull Request resolved: https://github.com/facebook/rocksdb/pull/8432 Reviewed By: zhichao-cao Differential Revision: D29558941 Pulled By: mrambacher fbshipit-source-id: 4a791af941ea4a65fc2f1fdfb1d7a95f42ca6774
-
- 12 7月, 2021 2 次提交
-
-
由 Adam Retter 提交于
Summary: Pull Request resolved: https://github.com/facebook/rocksdb/pull/8513 Reviewed By: jay-zhuang Differential Revision: D29654066 Pulled By: mrambacher fbshipit-source-id: b8f492fe21edd37fe1f1c5a4a0e9153f58bbf3e2
-
由 anand76 提交于
Summary: In ```DBImpl::WriteImpl()```, we call ```PreprocessWrite()``` which, among other things, checks the BG error and returns it set. This return status is later on passed to ```WriteStatusCheck()```, which calls ```SetBGError()```. This results in a spurious call, and info logs, on every user write request. We should avoid passing the ```PreprocessWrite()``` return status to ```WriteStatusCheck()```, as the former would have called ```SetBGError()``` already if it encountered any new errors, such as error when creating a new WAL file. Pull Request resolved: https://github.com/facebook/rocksdb/pull/8511 Test Plan: Run existing tests Reviewed By: zhichao-cao Differential Revision: D29639917 Pulled By: anand1976 fbshipit-source-id: 19234163969e1645dbeb273712aaf5cd9ea2b182
-
- 10 7月, 2021 2 次提交
-
-
由 Baptiste Lemaire 提交于
Summary: In https://github.com/facebook/rocksdb/issues/8454, I introduced a new process baptized `MemPurge` (memtable garbage collection). This new PR is built upon this past mempurge prototype. In this PR, I made the `mempurge` process a background task, which provides superior performance since the mempurge process does not cling on the db_mutex anymore, and addresses severe restrictions from the past iteration (including a scenario where the past mempurge was failling, when a memtable was mempurged but was still referred to by an iterator/snapshot/...). Now the mempurge process ressembles an in-memory compaction process: the stack of immutable memtables is filtered out, and the useful payload is used to populate an output memtable. If the output memtable is filled at more than 60% capacity (arbitrary heuristic) the mempurge process is aborted and a regular flush process takes place, else the output memtable is kept in the immutable memtable stack. Note that adding this output memtable to the `imm()` memtable stack does not trigger another flush process, so that the flush thread can go to sleep at the end of a successful mempurge. MemPurge is activated by making the `experimental_allow_mempurge` flag `true`. When activated, the `MemPurge` process will always happen when the flush reason is `kWriteBufferFull`. The 3 unit tests confirm that this process supports `Put`, `Get`, `Delete`, `DeleteRange` operators and is compatible with `Iterators` and `CompactionFilters`. Pull Request resolved: https://github.com/facebook/rocksdb/pull/8505 Reviewed By: pdillinger Differential Revision: D29619283 Pulled By: bjlemaire fbshipit-source-id: 8a99bee76b63a8211bff1a00e0ae32360aaece95
-
由 qieqieplus 提交于
Summary: Pull Request resolved: https://github.com/facebook/rocksdb/pull/8486 Reviewed By: jay-zhuang Differential Revision: D29625501 Pulled By: pdillinger fbshipit-source-id: e6e2a455ae62a71f3a202278a751b9bba17ad03c
-
- 09 7月, 2021 3 次提交
-
-
由 Jay Zhuang 提交于
Summary: Add google benchmark for microbench. Add ribbon_bench for benchmark ribbon filter vs. other filters. Pull Request resolved: https://github.com/facebook/rocksdb/pull/8493 Test Plan: added test to CI To run the benchmark on devhost: Install benchmark: `$ sudo dnf install google-benchmark-devel` Build and run: `$ ROCKSDB_NO_FBCODE=1 DEBUG_LEVEL=0 make microbench` or with cmake: `$ mkdir build && cd build && cmake .. -DCMAKE_BUILD_TYPE=Release -DWITH_BENCHMARK=1 && make microbench` Reviewed By: pdillinger Differential Revision: D29589649 Pulled By: jay-zhuang fbshipit-source-id: 8fed13b562bef4472f161ecacec1ab6b18911dff
-
由 sdong 提交于
Summary: Add comments to options.bottommost_compression for options.num_levels=1 Pull Request resolved: https://github.com/facebook/rocksdb/pull/8415 Reviewed By: ajkr Differential Revision: D29181997 fbshipit-source-id: 5f0f49470f75d796320ecb24d5dc4ef4eb6fbe0f
-
由 longlijian 提交于
Summary: … should be uint64_t. Pull Request resolved: https://github.com/facebook/rocksdb/pull/8498 Reviewed By: jay-zhuang Differential Revision: D29605064 Pulled By: ajkr fbshipit-source-id: e431448ac9d8a37ae83679c4cc5732e29fe49de4
-
- 08 7月, 2021 2 次提交
-
-
由 sdong 提交于
Summary: … small overwritten files. If a file is overwritten with renamed and the parent path is not synced, FaultInjectionTestFS::DeleteFilesCreatedAfterLastDirSync() will delete the file. However, RocksDB relies on file renaming to be atomic no matter whether the parent directory is synced or not, and the current behavior breaks the assumption and caused some false positive: https://github.com/facebook/rocksdb/pull/8489 Since the atomic renaming is used in CURRENT files, to fix the problem, in FaultInjectionTestFS::DeleteFilesCreatedAfterLastDirSync(), we recover the state of overwritten file if the file is small. Pull Request resolved: https://github.com/facebook/rocksdb/pull/8501 Test Plan: Run stress test for a while and see it doesn't break. Reviewed By: anand1976 Differential Revision: D29594384 fbshipit-source-id: 589b5c2f0a9d2aca53752d7bdb0231efa5b3ae92
-
由 Andrew Kryczka 提交于
Summary: Various tests had disabled valgrind due to it slowing down and timing out (as is the case right now) the CI runs. Where a test was disabled with no comment, I assumed slowness was the cause. For these tests that were slow under valgrind, as well as the ones identified in https://github.com/facebook/rocksdb/issues/8352, this PR moves them behind the compiler flag `-DROCKSDB_FULL_VALGRIND_RUN`. Pull Request resolved: https://github.com/facebook/rocksdb/pull/8475 Test Plan: running `make full_valgrind_test`, `make valgrind_test`, `make check`; will verify they appear working correctly Reviewed By: jay-zhuang Differential Revision: D29504843 Pulled By: ajkr fbshipit-source-id: 2aac90749cfbd30d5ce11cb29a07a1b9314eeea7
-
- 07 7月, 2021 6 次提交
-
-
由 Baptiste Lemaire 提交于
Summary: Previously, the following command: ```USE_CLANG=1 TEST_TMPDIR=/dev/shm/rocksdb OPT=-g make -j$(nproc) analyze``` was raising an error/warning the new_mem could potentially be a `nullptr`. This error appeared due to code changes from https://github.com/facebook/rocksdb/issues/8454, including an if-statement containing "`... && new_mem != nullptr && ...`", which made the analyzer believe that past this `if`-statement, a `new_mem==nullptr` was a possible scenario. This code patch simply introduces `assert`s and removes this condition in the `if`-statement. Pull Request resolved: https://github.com/facebook/rocksdb/pull/8492 Reviewed By: jay-zhuang Differential Revision: D29571275 Pulled By: bjlemaire fbshipit-source-id: 75d72246b70ebbbae7dea11ccb5778686d8bcbea
-
由 anand76 提交于
Summary: ```TestFSWritableFile``` buffers data in ```Append``` in order to simulate unsynced data loss on crash. This is only required for buffered IO and should be disabled for direct IO. Otherwise, it causes crash tests to assert on the buffer address alignment - ```db_stress: env/io_posix.cc:1194: virtual rocksdb::IOStatus rocksdb::PosixWritableFile::Append(const rocksdb::Slice&, const rocksdb::IOOptions&, rocksdb::IODebugContext*): Assertion `IsSectorAligned(data.data(), GetRequiredBufferAlignment())' failed.```. Pull Request resolved: https://github.com/facebook/rocksdb/pull/8490 Reviewed By: zhichao-cao Differential Revision: D29565080 Pulled By: anand1976 fbshipit-source-id: 682831fd66ed3b9597caa74fc453e22dfaf9b973
-
由 anand76 提交于
Summary: Write and metadata error injection during DB open was enabled in https://github.com/facebook/rocksdb/issues/8474. This causes crash tests to fail very frequently due to another fault injection feature that deletes files created after the last dir sync during DB open. In real life, a similar failure would happen if the FS returns error on the CURRENT file rename, but the rename actually succeeded and got partially persisted (dir entry for the old CURRENT file got removed, but the entry for the new one is not persisted). Temporarily disable the fault injection feature until we figure out the likelihood of this bug happening and the proper way to fix it. Pull Request resolved: https://github.com/facebook/rocksdb/pull/8489 Test Plan: Stress test can open the DB successfully Reviewed By: siying Differential Revision: D29564516 Pulled By: anand1976 fbshipit-source-id: ffd1650715ea3c5bf7131936b0ca6fcf66f4e14e
-
由 sdong 提交于
Summary: Inject read failures in DB reopen, just as what we do for metadata writes and writes. Pull Request resolved: https://github.com/facebook/rocksdb/pull/8476 Test Plan: Some manual tests and make sure failures are triggered. Reviewed By: anand1976 Differential Revision: D29507283 fbshipit-source-id: d04da0163973447041038bd87701686a417c4e0c
-
由 Levi Tamasi 提交于
Summary: We ended up using a different approach for tracking the amount of garbage in blob files (see e.g. https://github.com/facebook/rocksdb/pull/8450), so the ability to apply only a range of table property collectors is now unnecessary. The patch reverts this part of https://github.com/facebook/rocksdb/pull/8298 while keeping the cleanup done in that PR. Pull Request resolved: https://github.com/facebook/rocksdb/pull/8465 Test Plan: `make check` Reviewed By: jay-zhuang Differential Revision: D29399921 Pulled By: ltamasi fbshipit-source-id: af64816c357d0829b9d7ba8ca1477038138f6f0a
-
由 mrambacher 提交于
Summary: Pull Request resolved: https://github.com/facebook/rocksdb/pull/8480 Reviewed By: zhichao-cao Differential Revision: D29528740 Pulled By: mrambacher fbshipit-source-id: fd0f70d15f66611c8498257a9973f7e98ca13839
-
- 02 7月, 2021 6 次提交
-
-
由 Baptiste Lemaire 提交于
Summary: Implement an experimental feature called "MemPurge", which consists in purging "garbage" bytes out of a memtable and reuse the memtable struct instead of making it immutable and eventually flushing its content to storage. The prototype is by default deactivated and is not intended for use. It is intended for correctness and validation testing. At the moment, the "MemPurge" feature can be switched on by using the `options.experimental_allow_mempurge` flag. For this early stage, when the allow_mempurge flag is set to `true`, all the flush operations will be rerouted to perform a MemPurge. This is a temporary design decision that will give us the time to explore meaningful heuristics to use MemPurge at the right time for relevant workloads . Moreover, the current MemPurge operation only supports `Puts`, `Deletes`, `DeleteRange` operations, and handles `Iterators` as well as `CompactionFilter`s that are invoked at flush time . Three unit tests are added to `db_flush_test.cc` to test if MemPurge works correctly (and checks that the previously mentioned operations are fully supported thoroughly tested). One noticeable design decision is the timing of the MemPurge operation in the memtable workflow: for this prototype, the mempurge happens when the memtable is switched (and usually made immutable). This is an inefficient process because it implies that the entirety of the MemPurge operation happens while holding the db_mutex. Future commits will make the MemPurge operation a background task (akin to the regular flush operation) and aim at drastically enhancing the performance of this operation. The MemPurge is also not fully "WAL-compatible" yet, but when the WAL is full, or when the regular MemPurge operation fails (or when the purged memtable still needs to be flushed), a regular flush operation takes place. Later commits will also correct these behaviors. Pull Request resolved: https://github.com/facebook/rocksdb/pull/8454 Reviewed By: anand1976 Differential Revision: D29433971 Pulled By: bjlemaire fbshipit-source-id: 6af48213554e35048a7e03816955100a80a26dc5
-
由 Akanksha Mahajan 提交于
Summary: Call OnCompactionCompleted API in case of DisableManualCompaction() with updated Status::Incomplete Pull Request resolved: https://github.com/facebook/rocksdb/pull/8469 Reviewed By: ajkr Differential Revision: D29475517 Pulled By: akankshamahajan15 fbshipit-source-id: a1726c5e6ee18c0b5097ea04f5e6975fbe108055
-
由 Peter (Stig) Edwards 提交于
Summary: Hello and thanks for RocksDB, This PR adds support for ```-report_open_timing true``` to ```db_bench```. It can be useful when tuning RocksDB on filesystem/env with high latencies for file level operations (create/delete/rename...) seen during ```((Optimistic)Transaction)DB::Open```. Some examples: ``` > db_bench -benchmarks updaterandom -num 1 -db /dev/shm/db_bench > db_bench -benchmarks updaterandom -num 0 -db /dev/shm/db_bench -use_existing_db true -report_open_timing true -readonly true 2>&1 | grep OpenDb OpenDb: 3.90133 milliseconds > db_bench -benchmarks updaterandom -num 0 -db /dev/shm/db_bench -use_existing_db true -report_open_timing true -use_secondary_db true 2>&1 | grep OpenDb OpenDb: 3.33414 milliseconds > db_bench -benchmarks updaterandom -num 0 -db /dev/shm/db_bench -use_existing_db true -report_open_timing true 2>&1 | grep -A1 OpenDb OpenDb: 6.05423 milliseconds > db_bench -benchmarks updaterandom -num 1 > db_bench -benchmarks updaterandom -num 0 -use_existing_db true -report_open_timing true -readonly true 2>&1 | grep OpenDb OpenDb: 4.06859 milliseconds > db_bench -benchmarks updaterandom -num 0 -use_existing_db true -report_open_timing true -use_secondary_db true 2>&1 | grep OpenDb OpenDb: 2.85794 milliseconds > db_bench -benchmarks updaterandom -num 0 -use_existing_db true -report_open_timing true 2>&1 | grep OpenDb OpenDb: 6.46376 milliseconds > db_bench -benchmarks updaterandom -num 1 -db /clustered_fs/db_bench > db_bench -benchmarks updaterandom -num 0 -db /clustered_fs/db_bench -use_existing_db true -report_open_timing true -readonly true 2>&1 | grep OpenDb OpenDb: 3.79805 milliseconds > db_bench -benchmarks updaterandom -num 0 -db /clustered_fs/db_bench -use_existing_db true -report_open_timing true -use_secondary_db true 2>&1 | grep OpenDb OpenDb: 3.00174 milliseconds > db_bench -benchmarks updaterandom -num 0 -db /clustered_fs/db_bench -use_existing_db true -report_open_timing true 2>&1 | grep OpenDb OpenDb: 24.8732 milliseconds ``` Pull Request resolved: https://github.com/facebook/rocksdb/pull/8464 Reviewed By: hx235 Differential Revision: D29398096 Pulled By: zhichao-cao fbshipit-source-id: 8f05dc3284f084612a3f30234e39e1c37548f50c
-
由 Zhichao Cao 提交于
Summary: add the injest_error_severity to control if it is a retryable IO Error or a fatal or unrecoverable error. Use a flag to indicate, if fatal error comes, the flag is set and db is stopped (but not corrupted). Pull Request resolved: https://github.com/facebook/rocksdb/pull/8479 Test Plan: run ./db_stress --reopen=0 --read_fault_one_in=1000 --write_fault_one_in=5 --disable_wal=true --write_buffer_size=3000000 -writepercent=5 -readpercent=50 --injest_error_severity=2 --column_families=1, make check Reviewed By: anand1976 Differential Revision: D29524271 Pulled By: zhichao-cao fbshipit-source-id: 1aa9fb9b5655b0adba6f5ad12005ca8c074c795b
-
由 anand76 提交于
Summary: Add a new test ```fbcode_crash_test``` to rocksdb-lego-determinator. This test allows the crash test to be run on Facebook Sandcastle infra using fbcode components. Also use the default Env in db_stress to access the expected values path as it requires a memory mapped file and may not work with custom Envs. Pull Request resolved: https://github.com/facebook/rocksdb/pull/8471 Reviewed By: ajkr Differential Revision: D29474722 Pulled By: anand1976 fbshipit-source-id: 7d086d82dd7091ae48e08cb4ace763ce3e3b87ef
-
由 mrambacher 提交于
Summary: Added mutex to fix TSAN issue Pull Request resolved: https://github.com/facebook/rocksdb/pull/8477 Reviewed By: zhichao-cao Differential Revision: D29517053 Pulled By: mrambacher fbshipit-source-id: 661ccb1f495b7d34874a79e0a3d7aea1123d6047
-
- 01 7月, 2021 3 次提交
-
-
由 sdong 提交于
Summary: Previously Stress can inject metadata write failures when reopening a DB. We extend it to file append too, in the same way. Pull Request resolved: https://github.com/facebook/rocksdb/pull/8474 Test Plan: manually run crash test with various setting and make sure the failures are triggered as expected. Reviewed By: zhichao-cao Differential Revision: D29503116 fbshipit-source-id: e73a446e80ccbd09301a579280e56ff949381fab
-
由 mrambacher 提交于
Summary: Added the Customizable::ConfigureNewObject method. The method will configure the object if options are found and invoke PrepareOptions if the flag is set properly. Added tests to test that PrepareOptions is properly called and to test if PrepareOptions fails. Pull Request resolved: https://github.com/facebook/rocksdb/pull/8468 Reviewed By: zhichao-cao Differential Revision: D29494703 Pulled By: mrambacher fbshipit-source-id: d5767dee5d7a98620ac66190262101cd0aa9d2b7
-
由 anand76 提交于
Summary: When the secondary cache lookup fails, we may still allocate a handle and charge the cache for metadata usage. If the cache is full, this can cause the usage to go over capacity. Later, when a (unrelated) handle is released, it trips up an assertion that checks that usage is less than capacity. To prevent this assertion failure, don't charge the cache for a failed secondary cache lookup. Tests: Run crash_test Pull Request resolved: https://github.com/facebook/rocksdb/pull/8470 Reviewed By: zhichao-cao Differential Revision: D29474713 Pulled By: anand1976 fbshipit-source-id: 27191969c95470a7b070d292b458efce71395bf2
-
- 30 6月, 2021 1 次提交
-
-
由 Jay Zhuang 提交于
Summary: Add statistics option on CompactionService remote side. Pull Request resolved: https://github.com/facebook/rocksdb/pull/8368 Test Plan: `make check` Reviewed By: ajkr Differential Revision: D28944427 Pulled By: jay-zhuang fbshipit-source-id: 2a19217f4a69b6e511af87eed12391860ef00c5e
-