Skip to content

R
rocksdb

kvdb / rocksdb
11 个月 前同步成功

通知 0
Star 0
Fork 0
R
rocksdb

体验新版 GitCode，发现更多精彩内容 >>

切换分支/标签
  1. 20 3月, 2019 4 次提交
  3. 19 3月, 2019 2 次提交
    • S
      Feature for sampling and reporting compressibility (#4842) · b45b1cde
      由 Shobhit Dayal 提交于 
      Summary:
This is a feature to sample data-block compressibility and and report them as stats. 1 in N (tunable) blocks is sampled for compressibility using two algorithms:
1. lz4 or snappy for fast compression
2. zstd or zlib for slow but higher compression.

The stats are reported to the caller as raw-bytes and compressed-bytes. The block continues to be compressed for storage using the specified CompressionType.

The db_bench_tool how has a command line option for specifying the sampling rate. It's default value is 0 (no sampling). To test the overhead for a certain value, users can compare the performance of db_bench_tool, varying the sampling rate. It is unlikely to have a noticeable impact for high values like 20.
Pull Request resolved: https://github.com/facebook/rocksdb/pull/4842

Differential Revision: D13629011

Pulled By: shobhitdayal

fbshipit-source-id: 14ca668bcab6499b2a1734edf848eb62a4f4fafa
      b45b1cde
    • H
      utilities: Fix build failure with -Werror=maybe-uninitialized (#5074) · 20d49da9
      由 He Zhe 提交于 
      Summary:
Initialize magic_number to zero to avoid such failure.
utilities/blob_db/blob_log_format.cc:91:3: error: 'magic_number' may be used
uninitialized in this function [-Werror=maybe-uninitialized]
   if (magic_number != kMagicNumber) {
   ^~
Signed-off-by: NHe Zhe <zhe.he@windriver.com>
Pull Request resolved: https://github.com/facebook/rocksdb/pull/5074

Differential Revision: D14505514

Pulled By: miasantreble

fbshipit-source-id: 4334462958c2b9c5a7c68c6ab24dadf94ad70902
      20d49da9
  5. 16 3月, 2019 2 次提交
    • A
      Update bg_error when log flush fails in SwitchMemtable() (#5072) · b4fa51df
      由 anand76 提交于 
      Summary:
There is a potential failure case in DBImpl::SwitchMemtable() that is not handled properly. The call to cur_log_writer->WriteBuffer() can fail due to an IO error. In that case, we need to call SetBGError() in order set the background error since the WriteBuffer() failure may result in data loss.

Also, the asserts for !new_mem and !new_log are incorrect, as those would have been allocated by the time this failure is detected.
Pull Request resolved: https://github.com/facebook/rocksdb/pull/5072

Differential Revision: D14461384

Pulled By: anand1976

fbshipit-source-id: fb59bce9d61378f37d2dfcd28c0b704b0f43c3cf
      b4fa51df
    • A
      exercise WAL recycling in crash test (#5070) · 2263f869
      由 Andrew Kryczka 提交于 
      Summary:
Since this feature affects the WAL behavior, it seems important our crash-recovery tests cover it.
Pull Request resolved: https://github.com/facebook/rocksdb/pull/5070

Differential Revision: D14470085

Pulled By: miasantreble

fbshipit-source-id: 9b9682a718a926d57d055e0a5ec867efbd2eb9c1
      2263f869
  7. 15 3月, 2019 1 次提交
    • Z
      Add the -try_process_corrupted_trace option to trace_analyzer (#5067) · dcde292c
      由 Zhichao Cao 提交于 
      Summary:
In the current trace_analyzer implementation, once the trace file has corrupted content, which can be caused by unexpected tracing operations or other reasons, trace_analyzer will print the error and stop analyzing.

By adding the -try_process_corrupted_trace option, user can try to process the corrupted trace file and get the analyzing results of the trace records from the beginning to the the first corrupted point in the trace file. Analyzing might fail even this option is enabled.
Pull Request resolved: https://github.com/facebook/rocksdb/pull/5067

Differential Revision: D14433037

Pulled By: zhichao-cao

fbshipit-source-id: d095233ba371726869af0def0cdee23b69896831
      dcde292c
  9. 13 3月, 2019 2 次提交
  11. 09 3月, 2019 5 次提交
  13. 08 3月, 2019 1 次提交
  15. 07 3月, 2019 2 次提交
    • M
      WritePrepared: handle adding prepare before max_evicted_seq_ (#5025) · 04a2631d
      由 Maysam Yabandeh 提交于 
      Summary:
The patch fixes an improbable race condition between AddPrepared from one write queue and AdvanceMaxEvictedSeq from another queue. In this scenario AddPrepared finds prepare_seq lower than max and adding to PrepareHeap as usual while AdvanceMaxEvictedSeq has finished checking PrepareHeap against the future max. Thus when AdvanceMaxEvictedSeq finishes off by updating the max_evicted_seq_, PrepareHeap ends up with a prepared_seq lower than it which breaks the PrepareHeap contract. The fix is that in AddPrepared we check against the future_max_evicted_seq_ instead, which is update before AdvanceMaxEvictedSeq acquire prepare_mutex_ and looks into PrepareHeap.
A unit test added to test for the failure scenario. The code is also refactored a bit to remove the duplicate code between AdvanceMaxEvictedSeq and AddPrepared.
Pull Request resolved: https://github.com/facebook/rocksdb/pull/5025

Differential Revision: D14249028

Pulled By: maysamyabandeh

fbshipit-source-id: 072ea56663f40359662c05fafa6ac524417b0622
      04a2631d
    • M
      WritePrepared: Add rollback batch to PreparedHeap (#5026) · 703f1375
      由 Maysam Yabandeh 提交于 
      Summary:
The patch adds the sequence number of the rollback patch to the PrepareHeap when two_write_queues is enabled. Although the current behavior is still correct, the change simplifies reasoning about the code, by having all uncommitted batches registered with the PreparedHeap.
Pull Request resolved: https://github.com/facebook/rocksdb/pull/5026

Differential Revision: D14249401

Pulled By: maysamyabandeh

fbshipit-source-id: 1e3424edee5cd14e56ee35931ad3c93ed997cd5a
      703f1375
  17. 05 3月, 2019 2 次提交
    • A
      Use `fallocate` even if hole-punching unsupported (#5023) · 186b3afa
      由 Andrew Kryczka 提交于 
      Summary:
The compiler flag `-DROCKSDB_FALLOCATE_PRESENT` was only set when
`fallocate`, `FALLOC_FL_KEEP_SIZE`, and `FALLOC_FL_PUNCH_HOLE` were all
present. However, the last of the three is not really necessary for the
primary `fallocate` use case; furthermore, it was introduced only in later
Linux kernel versions (2.6.38+).

This PR changes the flag `-DROCKSDB_FALLOCATE_PRESENT` to only require
`fallocate` and `FALLOC_FL_KEEP_SIZE` to be present. There is a separate
check for `FALLOC_FL_PUNCH_HOLE` only in the place where it is used.
Pull Request resolved: https://github.com/facebook/rocksdb/pull/5023

Differential Revision: D14248487

Pulled By: siying

fbshipit-source-id: a10ed0b902fa755988e957bd2dcec9081ec0502e
      186b3afa
    • S
      Move some RocksObject into try-with-resources in Test (#5037) · a2838006
      由 SeterKwok 提交于 
      Summary:
Fix #5008
Pull Request resolved: https://github.com/facebook/rocksdb/pull/5037

Differential Revision: D14302474

Pulled By: riversand963

fbshipit-source-id: dcd9dda5d4d6d459315692f355499a39e546d518
      a2838006
  19. 02 3月, 2019 6 次提交
  21. 01 3月, 2019 6 次提交
    • M
      Call PreReleaseCallback between WAL and memtable write (#5015) · 77ebc82b
      由 Maysam Yabandeh 提交于 
      Summary:
PreReleaseCallback meant to be called before the writes are visible to the readers. Since the sequence number is known after the WAL write, there is no reason to delay calling PreReleaseCallback to after the memtable write, which would complicates the reader's logic in presence of our memtable writes that are made visible by the other write thread.
Pull Request resolved: https://github.com/facebook/rocksdb/pull/5015

Differential Revision: D14221670

Pulled By: maysamyabandeh

fbshipit-source-id: a504dd665cf923226d7af09cc8e9c7739a25edc6
      77ebc82b
    • M
      WritePrepared: commit only from the 2nd queue (#5014) · 68a2f94d
      由 Maysam Yabandeh 提交于 
      Summary:
When two_write_queues is enabled we call ::AddPrepared only from the main queue, which writes to both WAL and memtable, and call ::AddCommitted from the 2nd queue, which writes only to WAL. This simplifies the logic by avoiding concurrency between AddPrepared and also between AddCommitted. The patch fixes one case that did not conform with the rule above. This would allow future refactoring. For example AdvaneMaxEvictedSeq, which is invoked by AddCommitted, can be simplified by assuming lack of concurrent calls to it.
Pull Request resolved: https://github.com/facebook/rocksdb/pull/5014

Differential Revision: D14210493

Pulled By: maysamyabandeh

fbshipit-source-id: 6db5ba372a294a568a14caa010576460917a4eab
      68a2f94d
    • S
      Fix DefaultEnvTest.incBackgroundThreadsIfNeeded test (#5021) · 06ea73d6
      由 Sagar Vemuri 提交于 
      Summary:
`DefaultEnvTest.incBackgroundThreadsIfNeeded` jtest should assert that the number of threads is greater than or equal to the minimum number of threads.
Pull Request resolved: https://github.com/facebook/rocksdb/pull/5021

Differential Revision: D14268311

Pulled By: sagar0

fbshipit-source-id: 01fb32b5b3ce636451d162fa1a2bbc5bd1974682
      06ea73d6
    • L
      Introduce an enum for flag types in LRUHandle (#5024) · f83eecff
      由 Levi Tamasi 提交于 
      Summary:
Replace the integers used for setting and querying the various
flags in LRUHandle with enum values to improve readability.
Pull Request resolved: https://github.com/facebook/rocksdb/pull/5024

Differential Revision: D14263429

Pulled By: ltamasi

fbshipit-source-id: b1b9ba95635265f122c2b40da73850eaac18227a
      f83eecff
    • F
      [sync fix] Add defs.bzl · b157d3d1
      由 Fosco Marotto 提交于
      b157d3d1
    • S
      Add two more StatsLevel (#5027) · 5e298f86
      由 Siying Dong 提交于 
      Summary:
Statistics cost too much CPU for some use cases. Add two stats levels
so that people can choose to skip two types of expensive stats, timers and
histograms.
Pull Request resolved: https://github.com/facebook/rocksdb/pull/5027

Differential Revision: D14252765

Pulled By: siying

fbshipit-source-id: 75ecec9eaa44c06118229df4f80c366115346592
      5e298f86
  23. 27 2月, 2019 1 次提交
    • M
      WritePrepared: optimize read path by avoiding virtual (#5018) · a661c0d2
      由 Maysam Yabandeh 提交于 
      Summary:
The read path includes a callback function, ReadCallback, which would eventually calls IsInSnapshot to figure if a particular seq is in the reading snapshot or not. This callback is virtual, which adds the cost of multiple virtual function call to each read. The first few checks in IsInSnapshot, however, are quite trivial and take care of majority of the cases. The patch moves those to a non-virtual function in the the parent class, ReadCallback, to lower the virtual callback cost.
Pull Request resolved: https://github.com/facebook/rocksdb/pull/5018

Differential Revision: D14226562

Pulled By: maysamyabandeh

fbshipit-source-id: 6feed5b34f3b082e52092c5ef143e29b49c46b44
      a661c0d2
  25. 23 2月, 2019 1 次提交
  27. 22 2月, 2019 1 次提交
  29. 21 2月, 2019 2 次提交
    • Z
      add GetStatsHistory to retrieve stats snapshots (#4748) · c4f5d0aa
      由 Zhongyi Xie 提交于 
      Summary:
This PR adds public `GetStatsHistory` API to retrieve stats history in the form of an std map. The key of the map is the timestamp in microseconds when the stats snapshot is taken, the value is another std map from stats name to stats value (stored in std string). Two DBOptions are introduced: `stats_persist_period_sec` (default 10 minutes) controls the intervals between two snapshots are taken; `max_stats_history_count` (default 10) controls the max number of history snapshots to keep in memory. RocksDB will stop collecting stats snapshots if `stats_persist_period_sec` is set to 0.

(This PR is the in-memory part of https://github.com/facebook/rocksdb/pull/4535)
Pull Request resolved: https://github.com/facebook/rocksdb/pull/4748

Differential Revision: D13961471

Pulled By: miasantreble

fbshipit-source-id: ac836d401ecb84ea92216bf9966f969dedf4ad04
      c4f5d0aa
    • F
      Update version and history for 6.0 · 48c8d844
      由 Fosco Marotto 提交于
      48c8d844
  31. 20 2月, 2019 2 次提交
    • M
      Change random seed for txn stress tests on each run (#5004) · cf98df34
      由 Maysam Yabandeh 提交于 
      Summary:
Currently the transaction stress tests use thread id as the seed. Since the thread ids are likely to be the same across multiple runs, the seed is thus going to be the same. The patch includes time in calculating the seed to help covering a very different part of state space in each run of the stress tests. To be able to reproduce the bug in case the stress tests failed, it also prints out the time that was used to calculate the seed value.
Pull Request resolved: https://github.com/facebook/rocksdb/pull/5004

Differential Revision: D14144356

Pulled By: maysamyabandeh

fbshipit-source-id: 728ed522f550fc8b4f5f9f373259c05fe9a54556
      cf98df34
    • M
      WritePrepared: Improve stress tests with slow threads (#4974) · 0f4244fe
      由 Maysam Yabandeh 提交于 
      Summary:
The transaction stress tests, stress a high concurrency scenario. In WritePrepared/WriteUnPrepared we need to also stress the scenarios where an inserting/reading transaction is very slow. This would stress the corner cases that the caching is not sufficient and other slower data structures are engaged. To emulate such cases we make use of slow inserter/verifier threads and also reduce the size of cache data structures.
Pull Request resolved: https://github.com/facebook/rocksdb/pull/4974

Differential Revision: D14143070

Pulled By: maysamyabandeh

fbshipit-source-id: 81eb674678faf9fae0f654cd60ebcc74e26aeee7
      0f4244fe