- 17 7月, 2021 9 次提交
-
-
由 sdong 提交于
Summary: Currently, the code shows that we delete memtables immedately after it is trimmed from history. Although it should never happen as the super version still holds the memtable, which is only switched after it, it feels a good practice not to do it, but use clean it up in the standard way: put it to WriteContext and clean it after DB mutex. Pull Request resolved: https://github.com/facebook/rocksdb/pull/8530 Test Plan: Run all existing tests. Reviewed By: ajkr Differential Revision: D29703410 fbshipit-source-id: 21d8068ac6377de4b6fa7a89697195742659fde4
-
由 Jay Zhuang 提交于
Summary: Pull Request resolved: https://github.com/facebook/rocksdb/pull/8552 Reviewed By: ajkr Differential Revision: D29746828 Pulled By: jay-zhuang fbshipit-source-id: 17d564895ae9cb675d455e73626b9a6717db6279
-
由 Merlin Mao 提交于
Summary: There is an extra " in options.h (`"index block""`) Pull Request resolved: https://github.com/facebook/rocksdb/pull/8550 Test Plan: None Reviewed By: jay-zhuang Differential Revision: D29746077 Pulled By: autopear fbshipit-source-id: 2e5117296e5414b7c7440d990926bc1e567a0b4f
-
由 sdong 提交于
Summary: When DB Stress enables write failure in reopen, WAL files are also created with a wrapper writalbe file which buffers write until fsync. However, crash test currently expects all writes to WAL is persistent. This is at odd with the unsynced bytes dropped. To work it around temporarily, we disable WAL write failure for now. Pull Request resolved: https://github.com/facebook/rocksdb/pull/8548 Test Plan: Run db_stress. Manual printf to make sure only WAL files are skipped. Reviewed By: jay-zhuang Differential Revision: D29745095 fbshipit-source-id: 1879dd2c01abad7879ca243ee94570ec47c347f3
-
由 Jay Zhuang 提交于
Summary: Otherwise the build may report warning about missing `benchmark.h` for some targets, the error won't break the build. Pull Request resolved: https://github.com/facebook/rocksdb/pull/8523 Test Plan: `make blackbox_ubsan_crash_test` on a machine without benchmark lib installed. Reviewed By: pdillinger Differential Revision: D29682478 Pulled By: jay-zhuang fbshipit-source-id: e1261fbcda46bc6bd3cd39b7b03b7f78927d0430
-
由 mrambacher 提交于
Summary: Some URIs for creating instances (ala SecondaryCache) use complex URIs like (cache://name;prop=value). These URIs were treated as name-value properties. With this change, if the URI does not contain an "id=XX" setting, it will be treated as a single string value (and not an ID and map of name-value properties). Pull Request resolved: https://github.com/facebook/rocksdb/pull/8547 Reviewed By: anand1976 Differential Revision: D29741386 Pulled By: mrambacher fbshipit-source-id: 0621f62bec3a6699a7b66c7c0b5634b2856653aa
-
由 Peter Dillinger 提交于
Summary: I previously didn't notice the DB mutex was being held during block cache entry stat scans, probably because I primarily checked for read performance regressions, because they require the block cache and are traditionally latency-sensitive. This change does some refactoring to avoid holding DB mutex and to avoid triggering and waiting for a scan in GetProperty("rocksdb.cfstats"). Some tests have to be updated because now the stats collector is populated in the Cache aggressively on DB startup rather than lazily. (I hope to clean up some of this added complexity in the future.) This change also ensures proper treatment of need_out_of_mutex for non-int DB properties. Pull Request resolved: https://github.com/facebook/rocksdb/pull/8538 Test Plan: Added unit test logic that uses sync points to fail if the DB mutex is held during a scan, covering the various ways that a scan might be triggered. Performance test - the known impact to holding the DB mutex is on TransactionDB, and the easiest way to see the impact is to hack the scan code to almost always miss and take an artificially long time scanning. Here I've injected an unconditional 5s sleep at the call to ApplyToAllEntries. Before (hacked): $ TEST_TMPDIR=/dev/shm ./db_bench.base_xxx -benchmarks=randomtransaction,stats -cache_index_and_filter_blocks=1 -bloom_bits=10 -partition_index_and_filters=1 -duration=30 -stats_dump_period_sec=12 -cache_size=100000000 -statistics -transaction_db 2>&1 | egrep 'db.db.write.micros|micros/op' randomtransaction : 433.219 micros/op 2308 ops/sec; 0.1 MB/s ( transactions:78999 aborts:0) rocksdb.db.write.micros P50 : 16.135883 P95 : 36.622503 P99 : 66.036115 P100 : 5000614.000000 COUNT : 149677 SUM : 8364856 $ TEST_TMPDIR=/dev/shm ./db_bench.base_xxx -benchmarks=randomtransaction,stats -cache_index_and_filter_blocks=1 -bloom_bits=10 -partition_index_and_filters=1 -duration=30 -stats_dump_period_sec=12 -cache_size=100000000 -statistics -transaction_db 2>&1 | egrep 'db.db.write.micros|micros/op' randomtransaction : 448.802 micros/op 2228 ops/sec; 0.1 MB/s ( transactions:75999 aborts:0) rocksdb.db.write.micros P50 : 16.629221 P95 : 37.320607 P99 : 72.144341 P100 : 5000871.000000 COUNT : 143995 SUM : 13472323 Notice the 5s P100 write time. After (hacked): $ TEST_TMPDIR=/dev/shm ./db_bench.new_xxx -benchmarks=randomtransaction,stats -cache_index_and_filter_blocks=1 -bloom_bits=10 -partition_index_and_filters=1 -duration=30 -stats_dump_period_sec=12 -cache_size=100000000 -statistics -transaction_db 2>&1 | egrep 'db.db.write.micros|micros/op' randomtransaction : 303.645 micros/op 3293 ops/sec; 0.1 MB/s ( transactions:98999 aborts:0) rocksdb.db.write.micros P50 : 16.061871 P95 : 33.978834 P99 : 60.018017 P100 : 616315.000000 COUNT : 187619 SUM : 4097407 $ TEST_TMPDIR=/dev/shm ./db_bench.new_xxx -benchmarks=randomtransaction,stats -cache_index_and_filter_blocks=1 -bloom_bits=10 -partition_index_and_filters=1 -duration=30 -stats_dump_period_sec=12 -cache_size=100000000 -statistics -transaction_db 2>&1 | egrep 'db.db.write.micros|micros/op' randomtransaction : 310.383 micros/op 3221 ops/sec; 0.1 MB/s ( transactions:96999 aborts:0) rocksdb.db.write.micros P50 : 16.270026 P95 : 35.786844 P99 : 64.302878 P100 : 603088.000000 COUNT : 183819 SUM : 4095918 P100 write is now ~0.6s. Not good, but it's the same even if I completely bypass all the scanning code: $ TEST_TMPDIR=/dev/shm ./db_bench.new_skip -benchmarks=randomtransaction,stats -cache_index_and_filter_blocks=1 -bloom_bits=10 -partition_index_and_filters=1 -duration=30 -stats_dump_period_sec=12 -cache_size=100000000 -statistics -transaction_db 2>&1 | egrep 'db.db.write.micros|micros/op' randomtransaction : 311.365 micros/op 3211 ops/sec; 0.1 MB/s ( transactions:96999 aborts:0) rocksdb.db.write.micros P50 : 16.274362 P95 : 36.221184 P99 : 68.809783 P100 : 649808.000000 COUNT : 183819 SUM : 4156767 $ TEST_TMPDIR=/dev/shm ./db_bench.new_skip -benchmarks=randomtransaction,stats -cache_index_and_filter_blocks=1 -bloom_bits=10 -partition_index_and_filters=1 -duration=30 -stats_dump_period_sec=12 -cache_size=100000000 -statistics -transaction_db 2>&1 | egrep 'db.db.write.micros|micros/op' randomtransaction : 308.395 micros/op 3242 ops/sec; 0.1 MB/s ( transactions:97999 aborts:0) rocksdb.db.write.micros P50 : 16.106222 P95 : 37.202403 P99 : 67.081875 P100 : 598091.000000 COUNT : 185714 SUM : 4098832 No substantial difference. Reviewed By: siying Differential Revision: D29738847 Pulled By: pdillinger fbshipit-source-id: 1c5c155f5a1b62e4fea0fd4eeb515a8b7474027b
-
由 sdong 提交于
Summary: Right now, db_bench with seekrandom and multiple DB setup creates iterator for all DBs just to query one of them. It's different from most real workloads. Fix it by only creating iterators that will be queried. Also fix a bug that DBs are not destroyed in multi-DB mode. Pull Request resolved: https://github.com/facebook/rocksdb/pull/7818 Test Plan: Run db_bench with single/multiDB X using/not using tailing iterator with ASAN build, and validate the behavior is expected. Reviewed By: ajkr Differential Revision: D25720226 fbshipit-source-id: c2ff7ff7120e5ba64287a30b057c5d29b2cbe20b
-
由 Baptiste Lemaire 提交于
Summary: Add `experiemental_allow_mempurge` flag support for `db_stress` and `db_crashtest.py`, with a `false` default value. I succesfully tested locally both `whitebox` and `blackbox` crash tests with `experiemental_allow_mempurge` flag set as true. Pull Request resolved: https://github.com/facebook/rocksdb/pull/8545 Reviewed By: akankshamahajan15 Differential Revision: D29734513 Pulled By: bjlemaire fbshipit-source-id: 24316c0eccf6caf409e95c035f31d822c66714ae
-
- 16 7月, 2021 7 次提交
-
-
由 Mark Rambacher 提交于
Summary: Made the EncryptionProvider and BlockCipher classes inherit from Customizable. Added/fixed the CreateFromString method to these classes to create instances from builtin or registered classes. Added tests to verify that instances can be registered and retrieved as appropriate. Added the ability to configure the builtin (CTR, ROT13) classes from configurable properties. Added the appropriate tests. Pull Request resolved: https://github.com/facebook/rocksdb/pull/8354 Reviewed By: zhichao-cao Differential Revision: D29558949 Pulled By: mrambacher fbshipit-source-id: c20286b32d179777e060f51a58943e9b0cf81d04
-
由 Peter Dillinger 提交于
Summary: In https://github.com/facebook/rocksdb/issues/8539 I accidentally only checked for GCC TSAN, which is what I tested locally, while CircleCI and FB CI use clang TSAN. Related: other existing code like in stack_trace.cc only check for clang TSAN. I've now standardized these to the GCC convention in port/lang.h, so now #ifdef __SANITIZE_THREAD__ can check for any TSAN (assuming lang.h include) Pull Request resolved: https://github.com/facebook/rocksdb/pull/8543 Test Plan: Put an assert(false) in slice_test and look for the NOTE about "signal-unsafe call", both GCC and clang. Eventually, CircleCI TSAN in https://github.com/facebook/rocksdb/issues/8538 Reviewed By: zhichao-cao Differential Revision: D29728483 Pulled By: pdillinger fbshipit-source-id: 8a3b8015c2ed48078214c3ee17146a2c3f11c9f7
-
由 zaorangyang 提交于
Summary: …e consistent with the member variables's Pull Request resolved: https://github.com/facebook/rocksdb/pull/8510 Reviewed By: ajkr Differential Revision: D29654067 Pulled By: mrambacher fbshipit-source-id: 908baaddfb20c266db7c5aca6a87971393d62ee6
-
由 Baptiste Lemaire 提交于
Summary: In this PR, `mempurge` is made compatible with the Write Ahead Log: in case of recovery, the DB is now capable of recovering the data that was "mempurged" and kept in the `imm()` list of immutable memtables. The twist was to add a uint64_t to the `memtable` struct to store the number of the earliest log file containing entries from the `memtable`. When a `Flush` operation is replaced with a `MemPurge`, the `VersionEdit` (which usually contains the new min log file number to pick up for recovery and the level 0 file path of the newly created SST file) is no longer appended to the manifest log, and every time the `deleteWal` method is called, a check is made on the list of immutable memtables. This PR also includes a unit test that verifies that no data is lost upon Reopening of the database when the mempurge feature is activated. This extensive unit test includes two column families, with valid data contained in the imm() at time of "crash"/reopening (recovery). Pull Request resolved: https://github.com/facebook/rocksdb/pull/8528 Reviewed By: pdillinger Differential Revision: D29701097 Pulled By: bjlemaire fbshipit-source-id: 072a900fb6ccc1edcf5eef6caf88f3060238edf9
-
由 longlijian 提交于
Summary: For more detail can reference the https://github.com/facebook/rocksdb/issues/6433 (https://github.com/facebook/rocksdb/pull/6433) Pull Request resolved: https://github.com/facebook/rocksdb/pull/8531 Reviewed By: siying Differential Revision: D29717057 Pulled By: ajkr fbshipit-source-id: 3ccad9501e5612590e54a7cf8c447118f323c7f4
-
由 Peter Dillinger 提交于
Summary: Some bits are mutated and read while holding a lock, other immutable bits (esp. secondary cache compatibility) can be read by arbitrary threads without holding a lock. AFAIK, this doesn't cause an issue on any architecture we care about, because you will get some legitimate version of the value that includes the initialization, as long as synchronization guarantees the initialization happens before the read. I've only seen this in https://github.com/facebook/rocksdb/issues/8538 so far, but it should be fixed regardless. Otherwise, we'll surely get these false reports again some time. Pull Request resolved: https://github.com/facebook/rocksdb/pull/8539 Test Plan: some local TSAN test runs and in CircleCI Reviewed By: zhichao-cao Differential Revision: D29720262 Pulled By: pdillinger fbshipit-source-id: 365fd7e565577c648815161f71b339bcb5ce12d5
-
由 Jay Zhuang 提交于
Summary: Some cmake and test configuration are set in pre-steps enviroment variables. Add the missing steps. Pull Request resolved: https://github.com/facebook/rocksdb/pull/8524 Test Plan: CI pass Reviewed By: siying Differential Revision: D29682731 Pulled By: jay-zhuang fbshipit-source-id: afda1acf6a7b76989db450442b0b27f387388b9d
-
- 15 7月, 2021 1 次提交
-
-
由 longlijian 提交于
Summary: The removed function in this PR, just only have declared and dose not have any reference used. Pull Request resolved: https://github.com/facebook/rocksdb/pull/8508 Reviewed By: mrambacher Differential Revision: D29649033 Pulled By: jay-zhuang fbshipit-source-id: df98143b73d6c184a2a60c9f7ea2548a065ee35d
-
- 14 7月, 2021 2 次提交
-
-
由 hongrubb 提交于
Summary: This PR is for https://github.com/facebook/rocksdb/issues/8453 We need to update `s = biter.status();` when `biter.status().IsIncomplete()` is true. By doing this, can fix the problem in issue. Besides, we still need to update `db_statistics` in `get_context.ReportCounters()` before return back. Pull Request resolved: https://github.com/facebook/rocksdb/pull/8485 Reviewed By: jay-zhuang Differential Revision: D29604835 Pulled By: ajkr fbshipit-source-id: c7f2f1cd058223ce1b507ec05d57cf264b9c9710
-
由 sherriiiliu 提交于
Summary: Fixed a few MSVC (VCToolsVersion=14.0) build errors and warnings * `DEFINE_string` is a macro and VC compiler complains that it cannot put [ifdef-inside-define](https://stackoverflow.com/questions/5586429/ifdef-inside-define) * `sleep()` is not a recognizable function. Use `FLAGS_env->SleepForMicroseconds` instead * Define precise type in comparison to avoid mismatch warning Pull Request resolved: https://github.com/facebook/rocksdb/pull/8519 Reviewed By: jay-zhuang Differential Revision: D29683086 fbshipit-source-id: 8c80941472089f8daba84ae29597e75e603850e4
-
- 13 7月, 2021 7 次提交
-
-
由 dependabot[bot] 提交于
Summary: Bumps [addressable](https://github.com/sporkmonger/addressable) from 2.7.0 to 2.8.0. <details> <summary>Changelog</summary> <p><em>Sourced from <a href="https://github.com/sporkmonger/addressable/blob/main/CHANGELOG.md">addressable's changelog</a>.</em></p> <blockquote> <h1>Addressable 2.8.0</h1> <ul> <li>fixes ReDoS vulnerability in Addressable::Template#match</li> <li>no longer replaces <code>+</code> with spaces in queries for non-http(s) schemes</li> <li>fixed encoding ipv6 literals</li> <li>the <code>:compacted</code> flag for <code>normalized_query</code> now dedupes parameters</li> <li>fix broken <code>escape_component</code> alias</li> <li>dropping support for Ruby 2.0 and 2.1</li> <li>adding Ruby 3.0 compatibility for development tasks</li> <li>drop support for <code>rack-mount</code> and remove Addressable::Template#generate</li> <li>performance improvements</li> <li>switch CI/CD to GitHub Actions</li> </ul> </blockquote> </details> <details> <summary>Commits</summary> <ul> <li><a href="https://github.com/sporkmonger/addressable/commit/6469a232c0f1892809ff66737370c765d574e16c"><code>6469a23</code></a> Updating gemspec again</li> <li><a href="https://github.com/sporkmonger/addressable/commit/24336385de0261571b3adaad0431459edb420c79"><code>2433638</code></a> Merge branch 'main' of github.com:sporkmonger/addressable into main</li> <li><a href="https://github.com/sporkmonger/addressable/commit/e9c76b889789c75d7073c17b0ab557635d3f6704"><code>e9c76b8</code></a> Merge pull request <a href="https://github-redirect.dependabot.com/sporkmonger/addressable/issues/378">https://github.com/facebook/rocksdb/issues/378</a> from ashmaroli/flat-map</li> <li><a href="https://github.com/sporkmonger/addressable/commit/56c5cf7ece9223ff4240e07078cc26d3adbbbd30"><code>56c5cf7</code></a> Update the gemspec</li> <li><a href="https://github.com/sporkmonger/addressable/commit/c1fed1ca0a44c448e74d761fd44ed94869199807"><code>c1fed1c</code></a> Require a non-vulnerable rake</li> <li><a href="https://github.com/sporkmonger/addressable/commit/0d8a3127e35886ce9284810a7f2438bff6b43cbc"><code>0d8a312</code></a> Adding note about ReDoS vulnerability</li> <li><a href="https://github.com/sporkmonger/addressable/commit/89c76130ce255c601f642a018cb5fb5a80e679a7"><code>89c7613</code></a> Merge branch 'template-regexp' into main</li> <li><a href="https://github.com/sporkmonger/addressable/commit/cf8884f815c96b646c796f707bf768cf6eb65543"><code>cf8884f</code></a> Note about alias fix</li> <li><a href="https://github.com/sporkmonger/addressable/commit/bb03f7112e8e478240a0f96e1cc7428159b41586"><code>bb03f71</code></a> Merge pull request <a href="https://github-redirect.dependabot.com/sporkmonger/addressable/issues/371">https://github.com/facebook/rocksdb/issues/371</a> from charleystran/add_missing_encode_component_doc_entry</li> <li><a href="https://github.com/sporkmonger/addressable/commit/6d1d8094a66cbf932ecf69db6850bc9edaf86de0"><code>6d1d809</code></a> Adding note about :compacted normalization</li> <li>Additional commits viewable in <a href="https://github.com/sporkmonger/addressable/compare/addressable-2.7.0...addressable-2.8.0">compare view</a></li> </ul> </details> <br /> [![Dependabot compatibility score](https://dependabot-badges.githubapp.com/badges/compatibility_score?dependency-name=addressable&package-manager=bundler&previous-version=2.7.0&new-version=2.8.0)](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores) Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting `dependabot rebase`. [//]: # (dependabot-automerge-start) [//]: # (dependabot-automerge-end) --- <details> <summary>Dependabot commands and options</summary> <br /> You can trigger Dependabot actions by commenting on this PR: - `dependabot rebase` will rebase this PR - `dependabot recreate` will recreate this PR, overwriting any edits that have been made to it - `dependabot merge` will merge this PR after your CI passes on it - `dependabot squash and merge` will squash and merge this PR after your CI passes on it - `dependabot cancel merge` will cancel a previously requested merge and block automerging - `dependabot reopen` will reopen this PR if it is closed - `dependabot close` will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually - `dependabot ignore this major version` will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself) - `dependabot ignore this minor version` will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself) - `dependabot ignore this dependency` will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself) - `dependabot use these labels` will set the current labels as the default for future PRs for this repo and language - `dependabot use these reviewers` will set the current reviewers as the default for future PRs for this repo and language - `dependabot use these assignees` will set the current assignees as the default for future PRs for this repo and language - `dependabot use this milestone` will set the current milestone as the default for future PRs for this repo and language You can disable automated security fix PRs for this repo from the [Security Alerts page](https://github.com/facebook/rocksdb/network/alerts). </details> Pull Request resolved: https://github.com/facebook/rocksdb/pull/8515 Reviewed By: jay-zhuang Differential Revision: D29668988 Pulled By: ajkr fbshipit-source-id: c4b7abd4a879a7b562cb8ba745088dba6644f503
-
由 Peter Dillinger 提交于
Summary: MyRocks apparently uses valgrind to check for unreachable unfreed data, which is stricter than our valgrind checks. Internal ref: D29257815 This patch adds valgrind support to STATIC_AVOID_DESTRUCTION so that it's not reported with those stricter checks. Pull Request resolved: https://github.com/facebook/rocksdb/pull/8503 Test Plan: make valgrind_test Also, with modified VALGRIND_OPTS (see Makefile), more kinds of failures seen before than after this commit. Reviewed By: ajkr, yizhang82 Differential Revision: D29597784 Pulled By: pdillinger fbshipit-source-id: 360de157a176aec4d1be99ca20d160ecd47c0873
-
由 mrambacher 提交于
Summary: 1. Fix printing of stats when there are no writes (wamp=0). Previously had a div0 error 2. Added multireadrandom command as a valid target 3. Added ability to pass additional command line options to db_bench. Now can say things like benchmark.sh readrandom --mmap_read and the option will be passed to db_bench. Pull Request resolved: https://github.com/facebook/rocksdb/pull/8346 Reviewed By: zhichao-cao Differential Revision: D29500436 Pulled By: mrambacher fbshipit-source-id: 54e90708aae9133be3a903e35efdf8f8abbd86fa
-
由 bjlemaire 提交于
Summary: The MemPurge output status can either be an Abort if the mempurge is aborted due to the new_mem memtable reaching more than the target capacity (currently 60%), or for other reasons. As a result, in the log, we want to differentiate between an abort status, which in this PR only leads to a ROCKS_LOG_INFO, and any other status, which in this PR leads to a ROCKS_LOG_WARN. Pull Request resolved: https://github.com/facebook/rocksdb/pull/8514 Reviewed By: pdillinger Differential Revision: D29662446 Pulled By: bjlemaire fbshipit-source-id: c9bec8e238ebc7ecb14fbbddf580e6887e281c16
-
由 Dmitry Vorobev 提交于
Summary: When db is open as secondary, there are basically 2 step process: 1) Collect column families from wal log 2) Apply changes to Memtable In case primary db is TransactionDB instance, wal log will contain some additional data, like noop, etc. ColumnFamilyCollector doesn't implement methods to handle these, so it fails to open a wal log written by TransactionDB. (Everything works fine with standard DB::Open). Memtable recovery process knows how to handle such wal logs, so only missing piece seems to be ColumnFamilyCollector. Pull Request resolved: https://github.com/facebook/rocksdb/pull/8456 Reviewed By: ajkr Differential Revision: D29455945 Pulled By: mrambacher fbshipit-source-id: 5b29560fcbc008e17e95d0dc4b07558f3d63e26f
-
由 Myth 提交于
Summary: A very simple change :) Pull Request resolved: https://github.com/facebook/rocksdb/pull/8483 Reviewed By: ajkr Differential Revision: D29558904 Pulled By: mrambacher fbshipit-source-id: bbe68c20c861103726cb6231ca3fb8fbe1e5a546
-
由 mrambacher 提交于
Summary: Add ability to treat FlushBlockPolicyFactory as a Customizable. Pull Request resolved: https://github.com/facebook/rocksdb/pull/8432 Reviewed By: zhichao-cao Differential Revision: D29558941 Pulled By: mrambacher fbshipit-source-id: 4a791af941ea4a65fc2f1fdfb1d7a95f42ca6774
-
- 12 7月, 2021 2 次提交
-
-
由 Adam Retter 提交于
Summary: Pull Request resolved: https://github.com/facebook/rocksdb/pull/8513 Reviewed By: jay-zhuang Differential Revision: D29654066 Pulled By: mrambacher fbshipit-source-id: b8f492fe21edd37fe1f1c5a4a0e9153f58bbf3e2
-
由 anand76 提交于
Summary: In ```DBImpl::WriteImpl()```, we call ```PreprocessWrite()``` which, among other things, checks the BG error and returns it set. This return status is later on passed to ```WriteStatusCheck()```, which calls ```SetBGError()```. This results in a spurious call, and info logs, on every user write request. We should avoid passing the ```PreprocessWrite()``` return status to ```WriteStatusCheck()```, as the former would have called ```SetBGError()``` already if it encountered any new errors, such as error when creating a new WAL file. Pull Request resolved: https://github.com/facebook/rocksdb/pull/8511 Test Plan: Run existing tests Reviewed By: zhichao-cao Differential Revision: D29639917 Pulled By: anand1976 fbshipit-source-id: 19234163969e1645dbeb273712aaf5cd9ea2b182
-
- 10 7月, 2021 2 次提交
-
-
由 Baptiste Lemaire 提交于
Summary: In https://github.com/facebook/rocksdb/issues/8454, I introduced a new process baptized `MemPurge` (memtable garbage collection). This new PR is built upon this past mempurge prototype. In this PR, I made the `mempurge` process a background task, which provides superior performance since the mempurge process does not cling on the db_mutex anymore, and addresses severe restrictions from the past iteration (including a scenario where the past mempurge was failling, when a memtable was mempurged but was still referred to by an iterator/snapshot/...). Now the mempurge process ressembles an in-memory compaction process: the stack of immutable memtables is filtered out, and the useful payload is used to populate an output memtable. If the output memtable is filled at more than 60% capacity (arbitrary heuristic) the mempurge process is aborted and a regular flush process takes place, else the output memtable is kept in the immutable memtable stack. Note that adding this output memtable to the `imm()` memtable stack does not trigger another flush process, so that the flush thread can go to sleep at the end of a successful mempurge. MemPurge is activated by making the `experimental_allow_mempurge` flag `true`. When activated, the `MemPurge` process will always happen when the flush reason is `kWriteBufferFull`. The 3 unit tests confirm that this process supports `Put`, `Get`, `Delete`, `DeleteRange` operators and is compatible with `Iterators` and `CompactionFilters`. Pull Request resolved: https://github.com/facebook/rocksdb/pull/8505 Reviewed By: pdillinger Differential Revision: D29619283 Pulled By: bjlemaire fbshipit-source-id: 8a99bee76b63a8211bff1a00e0ae32360aaece95
-
由 qieqieplus 提交于
Summary: Pull Request resolved: https://github.com/facebook/rocksdb/pull/8486 Reviewed By: jay-zhuang Differential Revision: D29625501 Pulled By: pdillinger fbshipit-source-id: e6e2a455ae62a71f3a202278a751b9bba17ad03c
-
- 09 7月, 2021 3 次提交
-
-
由 Jay Zhuang 提交于
Summary: Add google benchmark for microbench. Add ribbon_bench for benchmark ribbon filter vs. other filters. Pull Request resolved: https://github.com/facebook/rocksdb/pull/8493 Test Plan: added test to CI To run the benchmark on devhost: Install benchmark: `$ sudo dnf install google-benchmark-devel` Build and run: `$ ROCKSDB_NO_FBCODE=1 DEBUG_LEVEL=0 make microbench` or with cmake: `$ mkdir build && cd build && cmake .. -DCMAKE_BUILD_TYPE=Release -DWITH_BENCHMARK=1 && make microbench` Reviewed By: pdillinger Differential Revision: D29589649 Pulled By: jay-zhuang fbshipit-source-id: 8fed13b562bef4472f161ecacec1ab6b18911dff
-
由 sdong 提交于
Summary: Add comments to options.bottommost_compression for options.num_levels=1 Pull Request resolved: https://github.com/facebook/rocksdb/pull/8415 Reviewed By: ajkr Differential Revision: D29181997 fbshipit-source-id: 5f0f49470f75d796320ecb24d5dc4ef4eb6fbe0f
-
由 longlijian 提交于
Summary: … should be uint64_t. Pull Request resolved: https://github.com/facebook/rocksdb/pull/8498 Reviewed By: jay-zhuang Differential Revision: D29605064 Pulled By: ajkr fbshipit-source-id: e431448ac9d8a37ae83679c4cc5732e29fe49de4
-
- 08 7月, 2021 2 次提交
-
-
由 sdong 提交于
Summary: … small overwritten files. If a file is overwritten with renamed and the parent path is not synced, FaultInjectionTestFS::DeleteFilesCreatedAfterLastDirSync() will delete the file. However, RocksDB relies on file renaming to be atomic no matter whether the parent directory is synced or not, and the current behavior breaks the assumption and caused some false positive: https://github.com/facebook/rocksdb/pull/8489 Since the atomic renaming is used in CURRENT files, to fix the problem, in FaultInjectionTestFS::DeleteFilesCreatedAfterLastDirSync(), we recover the state of overwritten file if the file is small. Pull Request resolved: https://github.com/facebook/rocksdb/pull/8501 Test Plan: Run stress test for a while and see it doesn't break. Reviewed By: anand1976 Differential Revision: D29594384 fbshipit-source-id: 589b5c2f0a9d2aca53752d7bdb0231efa5b3ae92
-
由 Andrew Kryczka 提交于
Summary: Various tests had disabled valgrind due to it slowing down and timing out (as is the case right now) the CI runs. Where a test was disabled with no comment, I assumed slowness was the cause. For these tests that were slow under valgrind, as well as the ones identified in https://github.com/facebook/rocksdb/issues/8352, this PR moves them behind the compiler flag `-DROCKSDB_FULL_VALGRIND_RUN`. Pull Request resolved: https://github.com/facebook/rocksdb/pull/8475 Test Plan: running `make full_valgrind_test`, `make valgrind_test`, `make check`; will verify they appear working correctly Reviewed By: jay-zhuang Differential Revision: D29504843 Pulled By: ajkr fbshipit-source-id: 2aac90749cfbd30d5ce11cb29a07a1b9314eeea7
-
- 07 7月, 2021 5 次提交
-
-
由 Baptiste Lemaire 提交于
Summary: Previously, the following command: ```USE_CLANG=1 TEST_TMPDIR=/dev/shm/rocksdb OPT=-g make -j$(nproc) analyze``` was raising an error/warning the new_mem could potentially be a `nullptr`. This error appeared due to code changes from https://github.com/facebook/rocksdb/issues/8454, including an if-statement containing "`... && new_mem != nullptr && ...`", which made the analyzer believe that past this `if`-statement, a `new_mem==nullptr` was a possible scenario. This code patch simply introduces `assert`s and removes this condition in the `if`-statement. Pull Request resolved: https://github.com/facebook/rocksdb/pull/8492 Reviewed By: jay-zhuang Differential Revision: D29571275 Pulled By: bjlemaire fbshipit-source-id: 75d72246b70ebbbae7dea11ccb5778686d8bcbea
-
由 anand76 提交于
Summary: ```TestFSWritableFile``` buffers data in ```Append``` in order to simulate unsynced data loss on crash. This is only required for buffered IO and should be disabled for direct IO. Otherwise, it causes crash tests to assert on the buffer address alignment - ```db_stress: env/io_posix.cc:1194: virtual rocksdb::IOStatus rocksdb::PosixWritableFile::Append(const rocksdb::Slice&, const rocksdb::IOOptions&, rocksdb::IODebugContext*): Assertion `IsSectorAligned(data.data(), GetRequiredBufferAlignment())' failed.```. Pull Request resolved: https://github.com/facebook/rocksdb/pull/8490 Reviewed By: zhichao-cao Differential Revision: D29565080 Pulled By: anand1976 fbshipit-source-id: 682831fd66ed3b9597caa74fc453e22dfaf9b973
-
由 anand76 提交于
Summary: Write and metadata error injection during DB open was enabled in https://github.com/facebook/rocksdb/issues/8474. This causes crash tests to fail very frequently due to another fault injection feature that deletes files created after the last dir sync during DB open. In real life, a similar failure would happen if the FS returns error on the CURRENT file rename, but the rename actually succeeded and got partially persisted (dir entry for the old CURRENT file got removed, but the entry for the new one is not persisted). Temporarily disable the fault injection feature until we figure out the likelihood of this bug happening and the proper way to fix it. Pull Request resolved: https://github.com/facebook/rocksdb/pull/8489 Test Plan: Stress test can open the DB successfully Reviewed By: siying Differential Revision: D29564516 Pulled By: anand1976 fbshipit-source-id: ffd1650715ea3c5bf7131936b0ca6fcf66f4e14e
-
由 sdong 提交于
Summary: Inject read failures in DB reopen, just as what we do for metadata writes and writes. Pull Request resolved: https://github.com/facebook/rocksdb/pull/8476 Test Plan: Some manual tests and make sure failures are triggered. Reviewed By: anand1976 Differential Revision: D29507283 fbshipit-source-id: d04da0163973447041038bd87701686a417c4e0c
-
由 Levi Tamasi 提交于
Summary: We ended up using a different approach for tracking the amount of garbage in blob files (see e.g. https://github.com/facebook/rocksdb/pull/8450), so the ability to apply only a range of table property collectors is now unnecessary. The patch reverts this part of https://github.com/facebook/rocksdb/pull/8298 while keeping the cleanup done in that PR. Pull Request resolved: https://github.com/facebook/rocksdb/pull/8465 Test Plan: `make check` Reviewed By: jay-zhuang Differential Revision: D29399921 Pulled By: ltamasi fbshipit-source-id: af64816c357d0829b9d7ba8ca1477038138f6f0a
-