1. 27 1月, 2022 2 次提交
  2. 26 1月, 2022 5 次提交
    • A
      Fix race condition in SstFileManagerImpl error recovery code (#9435) · beb86add
      anand76 提交于
      Summary:
      There is a race in SstFileManagerImpl between the ClearError() function
      and CancelErrorRecovery(). The race can cause ClearError() to deref the
      file system pointer after it has been freed. This is likely to occur
      during process shutdown, when the order of destruction of the
      DB/Env/FileSystem and SstFileManagerImpl is not deterministic.
      
      Pull Request resolved: https://github.com/facebook/rocksdb/pull/9435
      
      Test Plan:
      Reproduce the crash in a TSAN build by introducing sleeps in the code, and verify with
      the fix.
      
      Reviewed By: siying
      
      Differential Revision: D33774696
      
      Pulled By: anand1976
      
      fbshipit-source-id: 643d3da31b8d2ee6d9b6db5d33327e0053ce3b83
      beb86add
    • A
      Remove deprecated function DB::AddFile (#9433) · 8822562d
      Akanksha Mahajan 提交于
      Summary:
      RocksDB has marked DB::AddFile() as "DEPRECATED_FUNC" for a long time, and
      it will be removed in the upcoming 7.0 release.
      
      Pull Request resolved: https://github.com/facebook/rocksdb/pull/9433
      
      Test Plan: make check -j64; CircleCI
      
      Reviewed By: riversand963
      
      Differential Revision: D33763987
      
      Pulled By: akankshamahajan15
      
      fbshipit-source-id: a3407324479bb43689e1213e4e29d53095e7579a
      8822562d
    • A
      db_stress: db_stress fails on custom filesystems. (#9352) · 2eac6bb1
      Aravind Ramesh 提交于
      Summary:
      db_stress listener service always uses default filesystem to operate,
      causing it to not recognize custom filesystem (like ZenFS plugin FS).
      Pass the env to db_stress listener with the correct filesystem
      information, so it can open the user intended filesystem.
      Signed-off-by: NAravind Ramesh <Aravind.Ramesh@wdc.com>
      
      Pull Request resolved: https://github.com/facebook/rocksdb/pull/9352
      
      Reviewed By: riversand963
      
      Differential Revision: D33776762
      
      Pulled By: pdillinger
      
      fbshipit-source-id: e79bb9a544384f80ae9dd0108241ab9c83223954
      2eac6bb1
    • J
      Make `bottommost_temperature` dynamically changeable (#9402) · 022b400c
      Jay Zhuang 提交于
      Summary:
      Make `AdvancedColumnFamilyOptions.bottommost_temperature`
      dynamically changeable with `SetOptions` API.
      
      Pull Request resolved: https://github.com/facebook/rocksdb/pull/9402
      
      Test Plan: added unittest
      
      Reviewed By: siying
      
      Differential Revision: D33674487
      
      Pulled By: jay-zhuang
      
      fbshipit-source-id: 8943768156aa6197c63850a64238a8092527d517
      022b400c
    • S
      Not try to finish index builder after errors (#9426) · 5d10a53b
      sdong 提交于
      Summary:
      Right now, when error happens in block based table reader, we still call index_builder->Finish(), this causes one assertion in one stress test:
      
      db_stress: table/block_based/index_builder.cc:202: virtual rocksdb::Status rocksdb::PartitionedIndexBuilder::Finish(rocksdb::IndexBuilder::IndexBlocks*, const rocksdb::BlockHandle&): Assertion `sub_index_builder_ == nullptr' failed.
      
      This unlikely causes any corruption as we would finally abandon the file, but the code is confusing and it is hard to understand what would happen. Changing the behavior.
      
      Pull Request resolved: https://github.com/facebook/rocksdb/pull/9426
      
      Test Plan: Run existing tests
      
      Reviewed By: pdillinger
      
      Differential Revision: D33751929
      
      fbshipit-source-id: 3c916b9444a4171010fc53df40496570bef5ae7a
      5d10a53b
  3. 25 1月, 2022 4 次提交
  4. 22 1月, 2022 6 次提交
    • A
      Update version to 6.29.0 (#9418) · e8f116de
      anand76 提交于
      Summary:
      Update version for 6.29 release
      
      Pull Request resolved: https://github.com/facebook/rocksdb/pull/9418
      
      Reviewed By: riversand963
      
      Differential Revision: D33721048
      
      Pulled By: anand1976
      
      fbshipit-source-id: e73602ee1c829c2e47ce6e181bca4db7cb663979
      e8f116de
    • S
      Remove VS2017 from Appveyor CI (#9417) · a750b8a3
      sdong 提交于
      Summary:
      It appears that VS2017 is covered in CircleCI so we don't need it in Appveyor. Also, currently Appveyor has some problem with installing VS2017.
      
      Pull Request resolved: https://github.com/facebook/rocksdb/pull/9417
      
      Test Plan: Watch Appveyor run.
      
      Reviewed By: riversand963
      
      Differential Revision: D33719364
      
      fbshipit-source-id: 7f31bf056eeaf487b372881f85d134dc0fe5832a
      a750b8a3
    • P
      Add to HISTORY and minor loose ends from #9294, #9254 (#9386) · e7ac7363
      Peter Dillinger 提交于
      Summary:
      Loose ends relate to mmap on 32-bit systems. (Testing is more
      complicated when the feature was completely disabled on 32-bit.)
      
      Pull Request resolved: https://github.com/facebook/rocksdb/pull/9386
      
      Test Plan: CI
      
      Reviewed By: ajkr
      
      Differential Revision: D33590715
      
      Pulled By: pdillinger
      
      fbshipit-source-id: f2637036a538a552200adee65b6765fce8cae27b
      e7ac7363
    • P
      Fast path for detecting unchanged prefix_extractor (#9407) · fc9d4071
      Peter Dillinger 提交于
      Summary:
      Fixes a major performance regression in 6.26, where
      extra CPU is spent in SliceTransform::AsString when reads involve
      a prefix_extractor (Get, MultiGet, Seek). Common case performance
      is now better than 6.25.
      
      This change creates a "fast path" for verifying that the current prefix
      extractor is unchanged and compatible with what was used to
      generate a table file. This fast path detects the common case by
      pointer comparison on the current prefix_extractor and a "known
      good" prefix extractor (if applicable) that is saved at the time the
      table reader is opened. The "known good" prefix extractor is saved
      as another shared_ptr copy (in an existing field, however) to ensure
      the pointer is not recycled.
      
      When the prefix_extractor has changed to a different instance but
      same compatible configuration (rare, odd), performance is still a
      regression compared to 6.25, but this is likely acceptable because
      of the oddity of such a case. The performance of incompatible
      prefix_extractor is essentially unchanged.
      
      Also fixed a minor case (ForwardIterator) where a prefix_extractor
      could be used via a raw pointer after being freed as a shared_ptr,
      if replaced via SetOptions.
      
      Pull Request resolved: https://github.com/facebook/rocksdb/pull/9407
      
      Test Plan:
      ## Performance
      Populate DB with `TEST_TMPDIR=/dev/shm/rocksdb ./db_bench -benchmarks=fillrandom -num=10000000 -disable_wal=1 -write_buffer_size=10000000 -bloom_bits=16 -compaction_style=2 -fifo_compaction_max_table_files_size_mb=10000 -fifo_compaction_allow_compaction=0 -prefix_size=12`
      
      Running head-to-head comparisons simultaneously with `TEST_TMPDIR=/dev/shm/rocksdb ./db_bench -use_existing_db -readonly -benchmarks=seekrandom -num=10000000 -duration=20 -disable_wal=1 -bloom_bits=16 -compaction_style=2 -fifo_compaction_max_table_files_size_mb=10000 -fifo_compaction_allow_compaction=0 -prefix_size=12`
      
      Below each is compared by ops/sec vs. baseline which is version 6.25 (multiple baseline runs because of variable machine load)
      
      v6.26: 4833 vs. 6698 (<- major regression!)
      v6.27: 4737 vs. 6397 (still)
      New: 6704 vs. 6461 (better than baseline in common case)
      Disabled fastpath: 4843 vs. 6389 (e.g. if prefix extractor instance changes but is still compatible)
      Changed prefix size (no usable filter) in new: 787 vs. 5927
      Changed prefix size (no usable filter) in new & baseline: 773 vs. 784
      
      Reviewed By: mrambacher
      
      Differential Revision: D33677812
      
      Pulled By: pdillinger
      
      fbshipit-source-id: 571d9711c461fb97f957378a061b7e7dbc4d6a76
      fc9d4071
    • J
      Remove pyenv installation and use deps from S3 (#9406) · 7711f8cb
      Jay Zhuang 提交于
      Summary:
      * remove pyenv installation step which is not needed (it takes 3 minutes to install for every job and fail from time to time)
      * download compression lib fail from time to time, Uploaded the libs to S3 and download from them for CI, which should be more stable.
      
      Pull Request resolved: https://github.com/facebook/rocksdb/pull/9406
      
      Test Plan: CI
      
      Reviewed By: riversand963
      
      Differential Revision: D33700158
      
      Pulled By: jay-zhuang
      
      fbshipit-source-id: be7b172d7cd059c9d7b3139fd7a34f8070460e31
      7711f8cb
    • P
      Fix flaky EventListenerTest.DisableBGCompaction (#9400) · 8064a3ac
      Peter Dillinger 提交于
      Summary:
      Wasn't able to easily reproduce error, but easy to see a race
      condition between TestFlushListener::OnFlushCompleted and
      DBTestBase::Close(), which frees CF handles before closing DB.
      
      Pull Request resolved: https://github.com/facebook/rocksdb/pull/9400
      
      Test Plan: CI etc.
      
      Reviewed By: riversand963
      
      Differential Revision: D33645134
      
      Pulled By: pdillinger
      
      fbshipit-source-id: d0ec914cc43c9e14f53da633876b95b61995138d
      8064a3ac
  5. 21 1月, 2022 2 次提交
  6. 20 1月, 2022 3 次提交
    • P
      Make some FilterPolicy deprecations more clear (#9403) · ffe1e4b8
      Peter Dillinger 提交于
      Summary:
      The old block-based filter has been deprecated for years, but
      this makes that more clear by marking the functions specific to it and
      logging a warning when the feature is used.
      
      It is deprecated because of performance. In that old design, you have to
      binary search through the full SST index before a bloom filter query, which
      is much more expensive than a bloom query itself.
      
      Pull Request resolved: https://github.com/facebook/rocksdb/pull/9403
      
      Test Plan:
      Used db_bench with and without -use_block_based_filter,
      running at the same time
      
          TEST_TMPDIR=/dev/shm/rocksdb ./db_bench -benchmarks=fillrandom,readrandom -num=10000000 -duration=20 -disable_wal=1 -write_buffer_size=10000000 -bloom_bits=16 -compaction_style=2 -fifo_compaction_max_table_files_size_mb=10000 -fifo_compaction_allow_compaction=0
      
      No significant difference in construction time but 3x slower readrandom
      with -use_block_based_filter:
      readrandom   :     100.517 micros/op 9948 ops/sec;    1.1 MB/s
      vs.
      readrandom   :      33.368 micros/op 29968 ops/sec;    3.3 MB/s
      
      Also saw deprecation message (just once) in LOG only with
      -use_block_based_filter
      
      Reviewed By: ajkr
      
      Differential Revision: D33673202
      
      Pulled By: pdillinger
      
      fbshipit-source-id: 99f6f0eff619408d9e5f7ef546954ed0be6c7a5b
      ffe1e4b8
    • A
      Add API warning for `Iterator::Refresh()` with range tombstones (#9398) · 875bfd75
      Andrew Kryczka 提交于
      Summary:
      Need this until we properly return an error or fix the combination. Reported in https://github.com/facebook/rocksdb/issues/9255.
      
      Pull Request resolved: https://github.com/facebook/rocksdb/pull/9398
      
      Reviewed By: riversand963
      
      Differential Revision: D33641396
      
      Pulled By: ajkr
      
      fbshipit-source-id: 9fe804108f7b93912f5b9c7252ac49acedc4f805
      875bfd75
    • H
      Add missing comment to RateLimiter::Request() (#9392) · f61df25c
      Hui Xiao 提交于
      Summary:
      **Context/Summary:**
      There are two `RateLimiter::Request()` in public header. One of them is missing some comment that the other one has.
      
      Pull Request resolved: https://github.com/facebook/rocksdb/pull/9392
      
      Test Plan: rely on CI test
      
      Reviewed By: pdillinger
      
      Differential Revision: D33623609
      
      Pulled By: hx235
      
      fbshipit-source-id: 42dc06308ff0bcf5ee7ef67e0b1c0172fc239b20
      f61df25c
  7. 19 1月, 2022 3 次提交
  8. 15 1月, 2022 3 次提交
  9. 13 1月, 2022 6 次提交
  10. 12 1月, 2022 1 次提交
  11. 11 1月, 2022 3 次提交
    • M
      Restore Regex support for ObjectLibrary::Register, rename new APIs to allow... · 1973fcba
      mrambacher 提交于
      Restore Regex support for ObjectLibrary::Register, rename new APIs to allow old one to be deprecated in the future (#9362)
      
      Summary:
      In order to support old-style regex function registration, restored the original "Register<T>(string, Factory)" method using regular expressions.  The PatternEntry methods were left in place but renamed to AddFactory.  The goal is to allow for the deprecation of the original regex Registry method in an upcoming release.
      
      Added modes to the PatternEntry kMatchZeroOrMore and kMatchAtLeastOne to match * or +, respectively (kMatchAtLeastOne was the original behavior).
      
      Pull Request resolved: https://github.com/facebook/rocksdb/pull/9362
      
      Reviewed By: pdillinger
      
      Differential Revision: D33432562
      
      Pulled By: mrambacher
      
      fbshipit-source-id: ed88ab3f9a2ad0d525c7bd1692873f9bb3209d02
      1973fcba
    • J
      Fix flaky SimCacheTest.SimCacheLogging (#9373) · 6bab2782
      Jay Zhuang 提交于
      Summary:
      The random string may contain the string we're checking, e.g.:
      ```
      ADD - 206FBC78E96BC4C6A2DDDDC0AD5D1ADD - 111
      ```
      Only check the line starts-with "ADD -".
      
      Pull Request resolved: https://github.com/facebook/rocksdb/pull/9373
      
      Test Plan: `gtest-parallel ./sim_cache_test --gtest_filter=SimCacheTest.SimCacheLogging -r 1000`
      
      Reviewed By: riversand963
      
      Differential Revision: D33519574
      
      Pulled By: jay-zhuang
      
      fbshipit-source-id: d0c1c9b0b489246d292e7da4133030edaa748099
      6bab2782
    • Y
      Make RocksDB codebase compatible with newer compilers like clang-12 (#9370) · 55a21052
      Yanqin Jin 提交于
      Summary:
      Pull Request resolved: https://github.com/facebook/rocksdb/pull/9370
      
      GCC and newer clang, e.g. clang-12 treat `std::unique_ptr` slightly differently.
      For the following code
      ```
      #include <iostream>
      #include <memory>
      #include <type_traits>
      
      struct A {
          std::unique_ptr<int> m1;
      };
      
      int main()
      {
          std::cout << std::boolalpha;
          std::cout << std::is_standard_layout<A>::value << '\n';
          return 0;
      }
      ```
      GCC11(C++20) (tested on https://en.cppreference.com/w/cpp/types/is_standard_layout) will print "true", while newer clang, e.g. clang-12 will print "false". This breaks the usage of `offsetof()` on structs with non-static members of type `std::unique_ptr`.
      Fixing this by replacing the builtin `offsetof` with a trick documented at https://gist.github.com/graphitemaster/494f21190bb2c63c5516.
      
      Reviewed By: jay-zhuang
      
      Differential Revision: D33420840
      
      fbshipit-source-id: 02bde281dfa28809bec787ad0f7019e85dd9c607
      55a21052
  12. 08 1月, 2022 2 次提交
    • J
      Add filename to several Corruption messages (#9239) · 255aefb6
      jsteemann 提交于
      Summary:
      This change adds the filename of the offending filen to several place that produce Status objects with code `kCorruption`.
      This is not an attempt to have every Corruption message in the codebase extended with the filename, but it is a start.
      The motivation for the change was to quickly diagnose which file is corrupted when a large database is openend and there is not option to copy it offsite for analysis, run strace or install the ldb tool.
      In the particular case in question, the error message improved from a mere
      ```
      Corruption: checksum mismatch
      ```
      to
      ```
      Corruption: checksum mismatch in file /path/to/db/engine-rocksdb/MANIFEST-000171
      ```
      
      Pull Request resolved: https://github.com/facebook/rocksdb/pull/9239
      
      Reviewed By: jay-zhuang
      
      Differential Revision: D33237742
      
      Pulled By: riversand963
      
      fbshipit-source-id: bd42559cfbf786a0a674d091671d1a2bf07bdd31
      255aefb6
    • Y
      Remove obsolete function declaration (#8724) · 3dfee770
      Youngjae Lee 提交于
      Summary:
      Function `Version::UpdateFilesByCompactionPri()` is never called and not implemented.
      
      Pull Request resolved: https://github.com/facebook/rocksdb/pull/8724
      
      Reviewed By: ajkr
      
      Differential Revision: D30643943
      
      Pulled By: riversand963
      
      fbshipit-source-id: 174b2d9a2a42e286222909a035cc74a7b5602335
      3dfee770