- 24 1月, 2019 5 次提交
-
-
由 Siying Dong 提交于
Summary: compaction_pri = kMinOverlappingRatio usually provides much better write amplification than the default. https://github.com/facebook/rocksdb/pull/4907 fixes one shortcome of this option. Make it default. Pull Request resolved: https://github.com/facebook/rocksdb/pull/4911 Differential Revision: D13789262 Pulled By: siying fbshipit-source-id: d90acf8c4dede44f00d183ca4c7a210259378269
-
由 Andrew Kryczka 提交于
Summary: The pointer `get_context` was passed as the value for the boolean argument `index_key_is_full`. Luckily the pointer was always non-null so evaluated to true which is the correct value for the boolean argument. But we were missing out on batch updates to stats since we were not passing anything for the `GetContext*` argument and it defaults to `nullptr`. Pull Request resolved: https://github.com/facebook/rocksdb/pull/4913 Differential Revision: D13791449 Pulled By: ajkr fbshipit-source-id: dbe40bf406c64d34cb5298604145d18b9e0ca9be
-
由 Sagar Vemuri 提交于
Summary: Fixed clang static analyzer warning about division by 0. ``` ar: creating librocksdb_debug.a tools/db_bench_tool.cc:4650:43: warning: Division by zero int pos = static_cast<int>(rand_num % range_); ~~~~~~~~~^~~~~~~~ 1 warning generated. make: *** [analyze] Error 1 ``` This is from the new code I recently merged in ce8e88d2. Pull Request resolved: https://github.com/facebook/rocksdb/pull/4910 Differential Revision: D13788037 Pulled By: sagar0 fbshipit-source-id: f48851dca85047c19fbb1a361e25ce643aa4c7ea
-
由 Siying Dong 提交于
Summary: Right now, CompactionPri = kMinOverlappingRatio provides best write amplification, but it doesn't prioritize files with more tombstones. We combine the two good features: make kMinOverlappingRatio to boost files with lots of tombstones too. Pull Request resolved: https://github.com/facebook/rocksdb/pull/4907 Differential Revision: D13788774 Pulled By: siying fbshipit-source-id: 1991cbb495fb76c8b529de69896e38d81ed9d9b3
-
由 Remington Brasga 提交于
Summary: False-negative about path not existing. The regex is ignoring the "." in front of a path. Example: "./path/to/file" Pull Request resolved: https://github.com/facebook/rocksdb/pull/4682 Differential Revision: D13777110 Pulled By: sagar0 fbshipit-source-id: 9f8173b7581407555fdc055580732aeab37d4ade
-
- 23 1月, 2019 5 次提交
-
-
由 Zhongyi Xie 提交于
Summary: this PR address the following error: > tools/db_bench_tool.cc:4776:68: error: implicit conversion loses integer precision: 'int64_t' (aka 'long') to 'unsigned int' [-Werror,-Wshorten-64-to-32] s = db_with_cfh->db->Put(write_options_, key, gen.Generate(value_size)); Pull Request resolved: https://github.com/facebook/rocksdb/pull/4906 Differential Revision: D13780185 Pulled By: miasantreble fbshipit-source-id: 1c83a77d341099518c72f0f4a63e97ab9c4784b3
-
由 Siying Dong 提交于
Summary: Right now, deleting blob files is not rate limited, even if SstFileManger is specified. On the other hand, rate limiting blob deletion is not supported. With this change, Blob file deletion will go through SstFileManager too. Pull Request resolved: https://github.com/facebook/rocksdb/pull/4904 Differential Revision: D13772545 Pulled By: siying fbshipit-source-id: bd1b1d0beb26d5167385e00b7ecb8b94b879de84
-
由 Philip Jameson 提交于
Reviewed By: luciang Differential Revision: D13733578 fbshipit-source-id: 556c115935aa42c1da85ec0e91199b9f198fc467
-
由 Sagar Vemuri 提交于
Summary: Remove unused blob WAL filter so that users are not confused. I was initially under the impression that we have WAL Filter support in BlobDB. Pull Request resolved: https://github.com/facebook/rocksdb/pull/4896 Differential Revision: D13725709 Pulled By: sagar0 fbshipit-source-id: f997d7546e138a474036e88b957907cc714327f1
-
由 Zhichao Cao 提交于
Summary: Based on the specific workload models (key access distribution, value size distribution, and iterator scan length distribution, the QPS variation), the MixGraph benchmark generate the synthetic workload according to these distributions which can reflect the real-world workload characteristics. After user enable the tracing function, they will get the trace file. By analyzing the trace file with the trace_analyzer tool, user can generate a set of statistic data files including. The *_accessed_key_stats.txt, *-accessed_value_size_distribution.txt, *-iterator_length_distribution.txt, and *-qps_stats.txt are mainly used to fit the Matlab model fitting. After that, user can get the parameters of the workload distributions (the modeling details are described: [here](https://github.com/facebook/rocksdb/wiki/RocksDB-Trace%2C-Replay%2C-and-Analyzer)) The key access distribution follows the The two-term power model. The probability density function is: `f(x) = ax^{b}+c`. The corresponding parameters are key_dist_a, key_dist_b, and key_dist_c in db_bench For the value size distribution and iterator scan length distribution, they both follow the Generalized Pareto Distribution. The probability density function is `f(x) = (1/sigma)(1+k*(x-theta)/sigma))^{-1-1/k)`. The parameters are: value_k, value_theta, value_sigma and iter_k, iter_theta, iter_sigma. For more information about the Generalized Pareto Distribution, users can find the [wiki](https://en.wikipedia.org/wiki/Generalized_Pareto_distribution) and [Matalb page](https://www.mathworks.com/help/stats/generalized-pareto-distribution.html) As for the QPS, it follows the diurnal pattern. So Sine is a good model to fit it. `F(x) = sine_a*sin(sine_b*x + sine_c) + sine_d`. The trace_will tell you the average QPS in the print out resutls, which is sine_d. After user fit the "*-qps_stats.txt" to the Matlab model, user can get the sine_a, sine_b, and sine_c. By using the 4 parameters, user can control the QPS variation including the period, average, changes. To use the bench mark, user can indicate the following parameters as examples: ``` -benchmarks="mixgraph" -key_dist_a=0.002312 -key_dist_b=0.3467 -value_k=0.9233 -value_sigma=226.4092 -iter_k=2.517 -iter_sigma=14.236 -mix_get_ratio=0.7 -mix_put_ratio=0.25 -mix_seek_ratio=0.05 -sine_mix_rate_interval_milliseconds=500 -sine_a=15000 -sine_b=1 -sine_d=20000 ``` Pull Request resolved: https://github.com/facebook/rocksdb/pull/4788 Differential Revision: D13573940 Pulled By: sagar0 fbshipit-source-id: e184c27e07b4f1bc0b436c2be36c5090c1fb0222
-
- 20 1月, 2019 1 次提交
-
-
由 Andrew Kryczka 提交于
Summary: Mention feature introduced by #4849 in HISTORY.md. Pull Request resolved: https://github.com/facebook/rocksdb/pull/4901 Differential Revision: D13746430 Pulled By: ajkr fbshipit-source-id: f7bdea6f0522ed55428cbc521f8a9f3cd0002d4e
-
- 19 1月, 2019 3 次提交
-
-
由 Andrew Kryczka 提交于
Summary: This is essentially a re-submission of #4251 with a few improvements: - Split `CompressionDict` into two separate classes: `CompressionDict` and `UncompressionDict` - Eliminated `Init` functions. Instead do all initialization work in constructors. - Added test case for parallel DB open, which is the scenario where #4251 failed under TSAN. Pull Request resolved: https://github.com/facebook/rocksdb/pull/4849 Differential Revision: D13606039 Pulled By: ajkr fbshipit-source-id: 08c236059798c710db9cbf545fce0f371232d447
-
由 Yi Wu 提交于
Summary: Fix how CompactionIterator::findEarliestVisibleSnapshots handles released snapshot. It fixing the two scenarios: Scenario 1: key1 has two values v1 and v2. There're two snapshots s1 and s2 taken after v1 and v2 are committed. Right after compaction output v2, s1 is released. Now findEarliestVisibleSnapshot may see s1 being released, and return the next snapshot, which is s2. That's larger than v2's earliest visible snapshot, which was s1. The fix: the only place we check against last snapshot and current key snapshot is when we decide whether to compact out a value if it is hidden by a later value. In the check if we see current snapshot is even larger than last snapshot, we know last snapshot is released, and we are safe to compact out current key. Scenario 2: key1 has two values v1 and v2. there are two snapshots s1 and s2 taken after v1 and v2 are committed. During compaction before we process the key, s1 is released. When compaction process v2, snapshot checker may return kSnapshotReleased, and the earliest visible snapshot for v2 become s2. When compaction process v1, snapshot checker may return kIsInSnapshot (for WritePrepared transaction, it could be because v1 is still in commit cache). The result will become inconsistent here. The fix: remember the set of released snapshots ever reported by snapshot checker, and ignore them when finding result for findEarliestVisibleSnapshot. Pull Request resolved: https://github.com/facebook/rocksdb/pull/4890 Differential Revision: D13705538 Pulled By: maysamyabandeh fbshipit-source-id: e577f0d9ee1ff5a6035f26859e56902ecc85a5a4
-
由 Maysam Yabandeh 提交于
Summary: Here is the order of ops in a commit: 1) update commit cache 2) publish seq, 3) RemovePrepared. In case of a delayed prepared, there will be a gap between when the commit is visible to snapshots until delayed_prepared_ is cleaned up. To tell apart this case from a delayed uncommitted txn from, the commit entry of a delayed prepared is also stored in delayed_prepared_commits_, which is updated before publishing the commit. Also logic in GetSnapshotInternal that ensures that each new snapshot is always larger than max_evicted_seq_ is updated to check against the upcoming value of max_evicted_seq_ rather than its current one. This is because AdvanceMaxEvictedSeq gets the list of snapshots lower than the new max, before updating max_evicted_seq_. Pull Request resolved: https://github.com/facebook/rocksdb/pull/4894 Differential Revision: D13726988 Pulled By: maysamyabandeh fbshipit-source-id: 1e70d78061b50c944c9816bf4b6dac405ab4ccd3
-
- 18 1月, 2019 2 次提交
-
-
由 tom wang 提交于
Summary: Pull Request resolved: https://github.com/facebook/rocksdb/pull/4891 Differential Revision: D13718016 Pulled By: miasantreble fbshipit-source-id: 90bd372cff453a1c2d104c1cf49731d5dd770c14
-
由 Yanqin Jin 提交于
Summary: as title. Pull Request resolved: https://github.com/facebook/rocksdb/pull/4893 Differential Revision: D13716733 Pulled By: riversand963 fbshipit-source-id: 6811d6a99fe2094d5344f854e8939f01238b2adb
-
- 17 1月, 2019 3 次提交
-
-
由 Sagar Vemuri 提交于
Summary: Remove `garbage_collection_deletion_size_threshold` as it is not used anywhere. Pull Request resolved: https://github.com/facebook/rocksdb/pull/4888 Differential Revision: D13685982 Pulled By: sagar0 fbshipit-source-id: e08d3017b9a0c8fa99bc332b595ee4ed9db70c87
-
由 Yi Wu 提交于
Summary: Compaction iterator keep a copy of list of live snapshots at the beginning of compaction, and then query snapshot checker to verify if values of a sequence number is visible to these snapshots. However when the snapshot is released in the middle of compaction, the snapshot checker implementation (i.e. WritePreparedSnapshotChecker) may remove info with the snapshot and may report incorrect result, which lead to values being compacted out when it shouldn't. This patch conservatively keep the values if snapshot checker determines that the snapshots is released. Pull Request resolved: https://github.com/facebook/rocksdb/pull/4858 Differential Revision: D13617146 Pulled By: maysamyabandeh fbshipit-source-id: cf18a94f6f61a94bcff73c280f117b224af5fbc3
-
由 Yanqin Jin 提交于
Summary: By convention, time_t almost always stores the integral number of seconds since 00:00 hours, Jan 1, 1970 UTC, according to http://www.cplusplus.com/reference/ctime/time_t/. We surely want more precision than seconds. Pull Request resolved: https://github.com/facebook/rocksdb/pull/4868 Differential Revision: D13633046 Pulled By: riversand963 fbshipit-source-id: 4e01e23a22e8838023c51a91247a286dbf3a5396
-
- 16 1月, 2019 5 次提交
-
-
由 Yi Wu 提交于
Summary: With WritePrepared transaction, flush/compaction can contain uncommitted keys, and those keys can get committed during compaction. If a snapshot is taken before the key is committed, it should not see the key. On the other hand, compaction grab the list of snapshots at its beginning, and only consider those snapshots to dedup keys. Consider the case: ``` seq = 1: put "foo" = "bar" seq = 2: transaction T: delete "foo", prepare seq = 3: compaction start seq = 4: take snapshot S seq = 5: transaction T: commit. ... seq = N: compaction iterator reached key "foo". ``` When compaction start, the list of snapshot is empty. Compaction doesn't take snapshot S into account. When it reached "foo", transaction T is committed. Compaction may think the value "foo=bar" is not visible by any snapshot (which is wrong), and compact the value out. The fix is to explicitly take a snapshot before compaction grabbing the list of snapshots. Compaction will then has to keep keys visible to this snapshot. Pull Request resolved: https://github.com/facebook/rocksdb/pull/4883 Differential Revision: D13668775 Pulled By: maysamyabandeh fbshipit-source-id: 1cab9615f94b7d3e8522cc3d44c3a14c7d4720e4
-
由 Maysam Yabandeh 提交于
Summary: The AdvanceMaxEvictedSeq algorithm assumes that new snapshots always have sequence number larger than the last max_evicted_seq_. To enforce this assumption we make two changes: i) max is not advanced beyond the last published seq, with the exception that the evicted commit entry itself is not published yet, which is quite rare. ii) When obtaining the snapshot if the max_evicted_seq_ is not published yet, commit a dummy entry so that it waits for it to be published and also increased the latest published seq by one above the max. To test these non-realistic corner cases we create a commit cache with size 1 so that every single commit results into eviction. Pull Request resolved: https://github.com/facebook/rocksdb/pull/4886 Differential Revision: D13685270 Pulled By: maysamyabandeh fbshipit-source-id: 5461bc09c2a9b75798bfcb9853a256c81cdac0b0
-
由 Siying Dong 提交于
Summary: Right now the error mesage when options.wal_dir doesn't exist is not helpful to users. Be more specific Pull Request resolved: https://github.com/facebook/rocksdb/pull/4874 Differential Revision: D13642425 Pulled By: siying fbshipit-source-id: 9a3172ed0f799af233b0f3b2e5e35bc7ce04c7b5
-
由 Sagar Vemuri 提交于
Summary: - Corrected a comment asserting that the values "smaller" than a min_blob_size will be inlined in the base db. - Also fixed the type of ttl_range_secs while dumping blobdb options. Pull Request resolved: https://github.com/facebook/rocksdb/pull/4887 Differential Revision: D13680163 Pulled By: sagar0 fbshipit-source-id: 306c8cf2daa52210ffc334a6924ef44ffdedf887
-
由 Yi Wu 提交于
Summary: When prepared_txns_ heap is empty, SmallestUnCommittedSeq() should check delayed_prepared_ set as well. Pull Request resolved: https://github.com/facebook/rocksdb/pull/4867 Differential Revision: D13632134 Pulled By: maysamyabandeh fbshipit-source-id: b0423bb0a58dc95f1e636d5ed3f6e619df801fb7
-
- 12 1月, 2019 5 次提交
-
-
由 Yanqin Jin 提交于
Summary: If we do not do this, then reading MutableCFOptions may have a race condition with SetOptions which modifies MutableCFOptions. Also reserve space in advance for vectors to avoid reallocation changing the address of its elements. Test plan ``` $make clean && make -j32 all check $make clean && COMPILE_WITH_TSAN=1 make -j32 all check $make clean && COMPILE_WITH_ASAN=1 make -j32 all check ``` Pull Request resolved: https://github.com/facebook/rocksdb/pull/4876 Differential Revision: D13644500 Pulled By: riversand963 fbshipit-source-id: 4b8112c5c819d5a2922bb61ad1521b3d2fb2fd47
-
由 Siying Dong 提交于
Summary: LDB is frequently used to exam data copied. wal_dir in option file is not modified and it usually points to the path it copied from. The user experience will be better if when ldb sees wal_dir pointed by the option file doesn't exist, rather than fail, just ignore it. Pull Request resolved: https://github.com/facebook/rocksdb/pull/4875 Differential Revision: D13643173 Pulled By: siying fbshipit-source-id: 2e64d4ea2ec49a6794b9a706b7fc1ba901128bb8
-
由 Maysam Yabandeh 提交于
Summary: Fixes a typo that made mutex_ to remain unlocked when GetSnapshotListFromDB called from WritePreparedTxnDB. Pull Request resolved: https://github.com/facebook/rocksdb/pull/4872 Differential Revision: D13640381 Pulled By: maysamyabandeh fbshipit-source-id: 50f6600568f9092b4b43115f6ebd96e6c7388ad7
-
由 Zhongyi Xie 提交于
Summary: currently clang analyze fails with the following warning: > utilities/transactions/write_prepared_transaction_test.cc:1451:5: warning: Forming reference to null pointer ASSERT_GT(wp_db->max_evicted_seq_, 0); // max after recovery Pull Request resolved: https://github.com/facebook/rocksdb/pull/4871 Differential Revision: D13638053 Pulled By: miasantreble fbshipit-source-id: b192b0c13c411c58defc9e280b34cdfcab3fa8e3
-
由 Varadharajan 提交于
Summary: Build failing due to incorrect filename. Pull Request resolved: https://github.com/facebook/rocksdb/pull/4870 Differential Revision: D13637205 Pulled By: sagar0 fbshipit-source-id: 72da45d51b49bce32f696532ba0656ee0dc2b89f
-
- 11 1月, 2019 1 次提交
-
-
由 Siying Dong 提交于
Summary: Remove some components that we never heard people using them. Pull Request resolved: https://github.com/facebook/rocksdb/pull/4101 Differential Revision: D8825431 Pulled By: siying fbshipit-source-id: 97a12ad3cad4ab12c82741a5ba49669aaa854180
-
- 10 1月, 2019 2 次提交
-
-
由 Maysam Yabandeh 提交于
Summary: The vector returned by SnapshotList::GetAll could have duplicate entries if two separate snapshots have the same sequence number. However, when this vector is used in compaction the duplicate entires are of no use and could be safely ignored. Moreover not having duplicate entires simplifies reasoning in the compaction_iterator.cc code. For example when searching for the previous_snap we currently use the snapshot before the current one but the way the code uses that it expects it to be also less than the current snapshot, which would be simpler to read if there is no duplicate entry in the snapshot list. Pull Request resolved: https://github.com/facebook/rocksdb/pull/4860 Differential Revision: D13615502 Pulled By: maysamyabandeh fbshipit-source-id: d45bf01213ead5f39db811f951802da6fcc3332b
-
由 Yanqin Jin 提交于
Summary: as titled. Currently it's possible to create a local object of type PerfContext since it's part of public API. Then it's safe to initialize the two members to 0. If PerfContext is created as thread-local object, then all members are zero-initialized according to C++ standard. Pull Request resolved: https://github.com/facebook/rocksdb/pull/4859 Differential Revision: D13614504 Pulled By: riversand963 fbshipit-source-id: 406ff548e105a074f379ad1054d56fece5f524a0
-
- 09 1月, 2019 4 次提交
-
-
由 Yanqin Jin 提交于
Summary: Pull Request resolved: https://github.com/facebook/rocksdb/pull/4857 Differential Revision: D13602688 Pulled By: riversand963 fbshipit-source-id: 993419a6afb982a7a701ff71daebebb4b4a6b265
-
由 Maysam Yabandeh 提交于
Summary: Previously IsInSnapshot assumed that the snapshot is valid at the time that the function is called. However there are cases where that might not be valid. Example is background compactions where the compaction algorithm operates with a list of snapshots some of which might be released by the time they are being passed to IsInSnapshot. The patch make two changes to enable the caller to tell difference: i) any live snapshot below max is added to max_committed_seq_, which allows IsInSnapshot to confidently tell whether the passed snapshot is invalid if it below max, ii) extends IsInSnapshot API with a "released" variable that is set true when IsInSnapshot find no such snapshot below max and also find no other way to give a certain return value. In such cases the return value is true but the caller should also check the "released" boolean after the call. In short here is the changes in the API: i) If the snapshot is valid, no change is required. ii) If the snapshot might be invalid, a reference to "released" boolean must be passed to IsInSnapshot. ii-a) If snapshot is above max, IsInSnapshot can figure the return valid using the commit cache. ii-b) otherwise if snapshot is in old_commit_map_, IsInSnapshot can use that to tell if value was visible to the snapshot. ii-c) otherwise it sets "released" to true and returns true as well. Pull Request resolved: https://github.com/facebook/rocksdb/pull/4856 Differential Revision: D13599847 Pulled By: maysamyabandeh fbshipit-source-id: 1752be28667f886a1efec8cae5714b9b7a8f1e0f
-
由 Siying Dong 提交于
Summary: https://github.com/facebook/rocksdb/pull/3340 introduces preloading when max_open_files != -1. It doesn't preload index and filter in non-initial file loading case. This is a little bit too complicated to understand. We observed in one MyRocks use case where the filter is expected to be preloaded but is not. To simplify the use case, we simply always prefetch the index and filter. They anyway is expected to be loaded in the file verification phase anyway. Pull Request resolved: https://github.com/facebook/rocksdb/pull/4852 Differential Revision: D13595402 Pulled By: siying fbshipit-source-id: d4d8624eb3e849e20aeb990df2100502d85aff31
-
由 Maysam Yabandeh 提交于
Summary: IsInSnapshotEmptyMapTest tests that IsInSnapshot returns correct value for existing data after a recovery, where max is not zero and yet commit cache is empty. The existing test was preliminary which is improved in this patch. It also increases the db sequence after recovery so that there the snapshot immediately taken after recovery would have a sequence number different than that of max_evicted_seq. This simplifies the logic in IsInSnapshot by not having to consider the special case that an old snapshot might be equal to max_evicted_seq and yet not present in old_commit_map. Pull Request resolved: https://github.com/facebook/rocksdb/pull/4853 Differential Revision: D13595223 Pulled By: maysamyabandeh fbshipit-source-id: 77c12ca8a3f61a47479a93bef2038ff502dc3322
-
- 08 1月, 2019 4 次提交
-
-
由 Yanqin Jin 提交于
Summary: as titled. We can remove the assersion because we do not perform verification in AtomicFlushStressTest::TestCheckpoint for similar reasons to TestGet, TestPut, etc. Therefore, we override TestCheckpoint in AtomicFlushStressTest so that the assertion `rand_column_families.size() == rand_keys.size()' is removed, and we do not verify the DB in this function. Pull Request resolved: https://github.com/facebook/rocksdb/pull/4846 Differential Revision: D13583377 Pulled By: riversand963 fbshipit-source-id: 03647f3da67e27a397413fd666e3bb43003bf596
-
由 Maysam Yabandeh 提交于
Summary: The rollback algorithm in WritePrepared transactions requires reading the values before the transaction start. Currently it uses the prepare_seq -1 as the snapshot sequence number for the read. This is not correct since the passed sequence number must be for a valid snapshot. The patch fixes it by passing kMaxSequenceNumber instead. This is fine since all the writes done by the aborted transaction will be skipped during the read anyway. Pull Request resolved: https://github.com/facebook/rocksdb/pull/4851 Differential Revision: D13592773 Pulled By: maysamyabandeh fbshipit-source-id: ff1bf92ea9909d4cccb173bdff49febc0e9eb7a2
-
由 tom wang 提交于
Summary: Pull Request resolved: https://github.com/facebook/rocksdb/pull/4850 Differential Revision: D13591940 Pulled By: sagar0 fbshipit-source-id: 617794e0a41d0f4554d40871180b061e84189fc5
-
由 Philip Jameson 提交于
Reviewed By: ttsugriy Differential Revision: D13583867 fbshipit-source-id: 8f218a9ffd9807d386ba0adc966af2a9a48ac64c
-