Skip to content

R
rocksdb

kvdb / rocksdb
12 个月 前同步成功

通知 0
Star 0
Fork 0
R
rocksdb

体验新版 GitCode，发现更多精彩内容 >>

切换分支/标签
  1. 25 1月, 2022 3 次提交
  3. 22 1月, 2022 6 次提交
    • A
      Update version to 6.29.0 (#9418) · e8f116de
      由 anand76 提交于 
      Summary:
Update version for 6.29 release

Pull Request resolved: https://github.com/facebook/rocksdb/pull/9418

Reviewed By: riversand963

Differential Revision: D33721048

Pulled By: anand1976

fbshipit-source-id: e73602ee1c829c2e47ce6e181bca4db7cb663979
      e8f116de
    • S
      Remove VS2017 from Appveyor CI (#9417) · a750b8a3
      由 sdong 提交于 
      Summary:
It appears that VS2017 is covered in CircleCI so we don't need it in Appveyor. Also, currently Appveyor has some problem with installing VS2017.

Pull Request resolved: https://github.com/facebook/rocksdb/pull/9417

Test Plan: Watch Appveyor run.

Reviewed By: riversand963

Differential Revision: D33719364

fbshipit-source-id: 7f31bf056eeaf487b372881f85d134dc0fe5832a
      a750b8a3
    • P
      Add to HISTORY and minor loose ends from #9294, #9254 (#9386) · e7ac7363
      由 Peter Dillinger 提交于 
      Summary:
Loose ends relate to mmap on 32-bit systems. (Testing is more
complicated when the feature was completely disabled on 32-bit.)

Pull Request resolved: https://github.com/facebook/rocksdb/pull/9386

Test Plan: CI

Reviewed By: ajkr

Differential Revision: D33590715

Pulled By: pdillinger

fbshipit-source-id: f2637036a538a552200adee65b6765fce8cae27b
      e7ac7363
    • P
      Fast path for detecting unchanged prefix_extractor (#9407) · fc9d4071
      由 Peter Dillinger 提交于 
      Summary:
Fixes a major performance regression in 6.26, where
extra CPU is spent in SliceTransform::AsString when reads involve
a prefix_extractor (Get, MultiGet, Seek). Common case performance
is now better than 6.25.

This change creates a "fast path" for verifying that the current prefix
extractor is unchanged and compatible with what was used to
generate a table file. This fast path detects the common case by
pointer comparison on the current prefix_extractor and a "known
good" prefix extractor (if applicable) that is saved at the time the
table reader is opened. The "known good" prefix extractor is saved
as another shared_ptr copy (in an existing field, however) to ensure
the pointer is not recycled.

When the prefix_extractor has changed to a different instance but
same compatible configuration (rare, odd), performance is still a
regression compared to 6.25, but this is likely acceptable because
of the oddity of such a case. The performance of incompatible
prefix_extractor is essentially unchanged.

Also fixed a minor case (ForwardIterator) where a prefix_extractor
could be used via a raw pointer after being freed as a shared_ptr,
if replaced via SetOptions.

Pull Request resolved: https://github.com/facebook/rocksdb/pull/9407

Test Plan:
## Performance
Populate DB with `TEST_TMPDIR=/dev/shm/rocksdb ./db_bench -benchmarks=fillrandom -num=10000000 -disable_wal=1 -write_buffer_size=10000000 -bloom_bits=16 -compaction_style=2 -fifo_compaction_max_table_files_size_mb=10000 -fifo_compaction_allow_compaction=0 -prefix_size=12`

Running head-to-head comparisons simultaneously with `TEST_TMPDIR=/dev/shm/rocksdb ./db_bench -use_existing_db -readonly -benchmarks=seekrandom -num=10000000 -duration=20 -disable_wal=1 -bloom_bits=16 -compaction_style=2 -fifo_compaction_max_table_files_size_mb=10000 -fifo_compaction_allow_compaction=0 -prefix_size=12`

Below each is compared by ops/sec vs. baseline which is version 6.25 (multiple baseline runs because of variable machine load)

v6.26: 4833 vs. 6698 (<- major regression!)
v6.27: 4737 vs. 6397 (still)
New: 6704 vs. 6461 (better than baseline in common case)
Disabled fastpath: 4843 vs. 6389 (e.g. if prefix extractor instance changes but is still compatible)
Changed prefix size (no usable filter) in new: 787 vs. 5927
Changed prefix size (no usable filter) in new & baseline: 773 vs. 784

Reviewed By: mrambacher

Differential Revision: D33677812

Pulled By: pdillinger

fbshipit-source-id: 571d9711c461fb97f957378a061b7e7dbc4d6a76
      fc9d4071
    • J
      Remove pyenv installation and use deps from S3 (#9406) · 7711f8cb
      由 Jay Zhuang 提交于 
      Summary:
* remove pyenv installation step which is not needed (it takes 3 minutes to install for every job and fail from time to time)
* download compression lib fail from time to time, Uploaded the libs to S3 and download from them for CI, which should be more stable.

Pull Request resolved: https://github.com/facebook/rocksdb/pull/9406

Test Plan: CI

Reviewed By: riversand963

Differential Revision: D33700158

Pulled By: jay-zhuang

fbshipit-source-id: be7b172d7cd059c9d7b3139fd7a34f8070460e31
      7711f8cb
    • P
      Fix flaky EventListenerTest.DisableBGCompaction (#9400) · 8064a3ac
      由 Peter Dillinger 提交于 
      Summary:
Wasn't able to easily reproduce error, but easy to see a race
condition between TestFlushListener::OnFlushCompleted and
DBTestBase::Close(), which frees CF handles before closing DB.

Pull Request resolved: https://github.com/facebook/rocksdb/pull/9400

Test Plan: CI etc.

Reviewed By: riversand963

Differential Revision: D33645134

Pulled By: pdillinger

fbshipit-source-id: d0ec914cc43c9e14f53da633876b95b61995138d
      8064a3ac
  5. 21 1月, 2022 2 次提交
  7. 20 1月, 2022 3 次提交
    • P
      Make some FilterPolicy deprecations more clear (#9403) · ffe1e4b8
      由 Peter Dillinger 提交于 
      Summary:
The old block-based filter has been deprecated for years, but
this makes that more clear by marking the functions specific to it and
logging a warning when the feature is used.

It is deprecated because of performance. In that old design, you have to
binary search through the full SST index before a bloom filter query, which
is much more expensive than a bloom query itself.

Pull Request resolved: https://github.com/facebook/rocksdb/pull/9403

Test Plan:
Used db_bench with and without -use_block_based_filter,
running at the same time

    TEST_TMPDIR=/dev/shm/rocksdb ./db_bench -benchmarks=fillrandom,readrandom -num=10000000 -duration=20 -disable_wal=1 -write_buffer_size=10000000 -bloom_bits=16 -compaction_style=2 -fifo_compaction_max_table_files_size_mb=10000 -fifo_compaction_allow_compaction=0

No significant difference in construction time but 3x slower readrandom
with -use_block_based_filter:
readrandom   :     100.517 micros/op 9948 ops/sec;    1.1 MB/s
vs.
readrandom   :      33.368 micros/op 29968 ops/sec;    3.3 MB/s

Also saw deprecation message (just once) in LOG only with
-use_block_based_filter

Reviewed By: ajkr

Differential Revision: D33673202

Pulled By: pdillinger

fbshipit-source-id: 99f6f0eff619408d9e5f7ef546954ed0be6c7a5b
      ffe1e4b8
    • A
      Add API warning for `Iterator::Refresh()` with range tombstones (#9398) · 875bfd75
      由 Andrew Kryczka 提交于 
      Summary:
Need this until we properly return an error or fix the combination. Reported in https://github.com/facebook/rocksdb/issues/9255.

Pull Request resolved: https://github.com/facebook/rocksdb/pull/9398

Reviewed By: riversand963

Differential Revision: D33641396

Pulled By: ajkr

fbshipit-source-id: 9fe804108f7b93912f5b9c7252ac49acedc4f805
      875bfd75
    • H
      Add missing comment to RateLimiter::Request() (#9392) · f61df25c
      由 Hui Xiao 提交于 
      Summary:
**Context/Summary:**
There are two `RateLimiter::Request()` in public header. One of them is missing some comment that the other one has.

Pull Request resolved: https://github.com/facebook/rocksdb/pull/9392

Test Plan: rely on CI test

Reviewed By: pdillinger

Differential Revision: D33623609

Pulled By: hx235

fbshipit-source-id: 42dc06308ff0bcf5ee7ef67e0b1c0172fc239b20
      f61df25c
  9. 19 1月, 2022 3 次提交
  11. 15 1月, 2022 3 次提交
  13. 13 1月, 2022 6 次提交
  15. 12 1月, 2022 1 次提交
  17. 11 1月, 2022 3 次提交
    • M
      Restore Regex support for ObjectLibrary::Register, rename new APIs to allow... · 1973fcba
      由 mrambacher 提交于 
      Restore Regex support for ObjectLibrary::Register, rename new APIs to allow old one to be deprecated in the future (#9362)

Summary:
In order to support old-style regex function registration, restored the original "Register<T>(string, Factory)" method using regular expressions.  The PatternEntry methods were left in place but renamed to AddFactory.  The goal is to allow for the deprecation of the original regex Registry method in an upcoming release.

Added modes to the PatternEntry kMatchZeroOrMore and kMatchAtLeastOne to match * or +, respectively (kMatchAtLeastOne was the original behavior).

Pull Request resolved: https://github.com/facebook/rocksdb/pull/9362

Reviewed By: pdillinger

Differential Revision: D33432562

Pulled By: mrambacher

fbshipit-source-id: ed88ab3f9a2ad0d525c7bd1692873f9bb3209d02
      1973fcba
    • J
      Fix flaky SimCacheTest.SimCacheLogging (#9373) · 6bab2782
      由 Jay Zhuang 提交于 
      Summary:
The random string may contain the string we're checking, e.g.:
```
ADD - 206FBC78E96BC4C6A2DDDDC0AD5D1ADD - 111
```
Only check the line starts-with "ADD -".

Pull Request resolved: https://github.com/facebook/rocksdb/pull/9373

Test Plan: `gtest-parallel ./sim_cache_test --gtest_filter=SimCacheTest.SimCacheLogging -r 1000`

Reviewed By: riversand963

Differential Revision: D33519574

Pulled By: jay-zhuang

fbshipit-source-id: d0c1c9b0b489246d292e7da4133030edaa748099
      6bab2782
    • Y
      Make RocksDB codebase compatible with newer compilers like clang-12 (#9370) · 55a21052
      由 Yanqin Jin 提交于 
      Summary:
Pull Request resolved: https://github.com/facebook/rocksdb/pull/9370

GCC and newer clang, e.g. clang-12 treat `std::unique_ptr` slightly differently.
For the following code
```
#include <iostream>
#include <memory>
#include <type_traits>

struct A {
    std::unique_ptr<int> m1;
};

int main()
{
    std::cout << std::boolalpha;
    std::cout << std::is_standard_layout<A>::value << '\n';
    return 0;
}
```
GCC11(C++20) (tested on https://en.cppreference.com/w/cpp/types/is_standard_layout) will print "true", while newer clang, e.g. clang-12 will print "false". This breaks the usage of `offsetof()` on structs with non-static members of type `std::unique_ptr`.
Fixing this by replacing the builtin `offsetof` with a trick documented at https://gist.github.com/graphitemaster/494f21190bb2c63c5516.

Reviewed By: jay-zhuang

Differential Revision: D33420840

fbshipit-source-id: 02bde281dfa28809bec787ad0f7019e85dd9c607
      55a21052
  19. 08 1月, 2022 3 次提交
    • J
      Add filename to several Corruption messages (#9239) · 255aefb6
      由 jsteemann 提交于 
      Summary:
This change adds the filename of the offending filen to several place that produce Status objects with code `kCorruption`.
This is not an attempt to have every Corruption message in the codebase extended with the filename, but it is a start.
The motivation for the change was to quickly diagnose which file is corrupted when a large database is openend and there is not option to copy it offsite for analysis, run strace or install the ldb tool.
In the particular case in question, the error message improved from a mere
```
Corruption: checksum mismatch
```
to
```
Corruption: checksum mismatch in file /path/to/db/engine-rocksdb/MANIFEST-000171
```

Pull Request resolved: https://github.com/facebook/rocksdb/pull/9239

Reviewed By: jay-zhuang

Differential Revision: D33237742

Pulled By: riversand963

fbshipit-source-id: bd42559cfbf786a0a674d091671d1a2bf07bdd31
      255aefb6
    • Y
      Remove obsolete function declaration (#8724) · 3dfee770
      由 Youngjae Lee 提交于 
      Summary:
Function `Version::UpdateFilesByCompactionPri()` is never called and not implemented.

Pull Request resolved: https://github.com/facebook/rocksdb/pull/8724

Reviewed By: ajkr

Differential Revision: D30643943

Pulled By: riversand963

fbshipit-source-id: 174b2d9a2a42e286222909a035cc74a7b5602335
      3dfee770
    • H
      Release cache reservation of hash entries of the fall-back Ribbon Filter earlier (#9345) · 9110685e
      由 Hui Xiao 提交于 
      Summary:
Note: rebase on and merge after https://github.com/facebook/rocksdb/pull/9349, as part of https://github.com/facebook/rocksdb/pull/9342
**Context:**
https://github.com/facebook/rocksdb/pull/9073 charged the hash entries' memory in block cache with `CacheReservationHandle`. However, in the edge case where Ribbon Filter falls back to Bloom Filter and swaps its hash entries to the embedded bloom filter object, the handles associated with those entries are not swapped and thus not released as soon as those entries are cleared during Bloom Filter's finish process.

Although this is a minor issue since RocksDB internal calls `FilterBitsBuilder->Reset()` right after `FilterBitsBuilder->Finish()` on the main path, which releases all the cache reservation related to both the Ribbon Filter and its embedded Bloom Filter, it still worths this fix to avoid confusion.

**Summary:**
- Swapped the `CacheReservationHandle` associated with the hash entries on Ribbon Filter's fallback

Pull Request resolved: https://github.com/facebook/rocksdb/pull/9345

Test Plan: - Added a unit test to verify the number of cache reservation after clearing hash entries, which failed before the change and now succeeds

Reviewed By: pdillinger

Differential Revision: D33377225

Pulled By: hx235

fbshipit-source-id: 7487f4c40dfb6ee7928232021f93ef2c5329cffa
      9110685e
  21. 07 1月, 2022 2 次提交
  23. 06 1月, 2022 3 次提交
    • Y
      Add checking for `DB::DestroyColumnFamilyHandle()` (#9347) · b2e53ab2
      由 Yanqin Jin 提交于 
      Summary:
Closing https://github.com/facebook/rocksdb/issues/5006

Calling `DB::DestroyColumnFamilyHandle(column_family)` with `column_family` being the return value of
`DB::DefaultColumnFamily()` will return `Status::InvalidArgument()`.

Pull Request resolved: https://github.com/facebook/rocksdb/pull/9347

Test Plan: make check

Reviewed By: akankshamahajan15

Differential Revision: D33369675

Pulled By: riversand963

fbshipit-source-id: a8266a4daddf2b7a773c2dc7f3eb9a4adfb6b6dd
      b2e53ab2
    • A
      Test correctness with WAL disabled in non-txn blackbox crash tests (#9338) · 6892f19b
      由 Andrew Kryczka 提交于 
      Summary:
Recently we added the ability to verify some prefix of operations are recovered (AKA no "hole" in the recovered data) (https://github.com/facebook/rocksdb/issues/8966). Besides testing unsynced data loss scenarios, it is also useful to test WAL disabled use cases, where unflushed writes are expected to be lost. Note RocksDB only offers the prefix-recovery guarantee to WAL-disabled use cases that use atomic flush, so crash test always enables atomic flush when WAL is disabled.

To verify WAL-disabled crash-recovery correctness globally, i.e., also in whitebox and blackbox transaction tests, it is possible but requires further changes. I added TODOs in db_crashtest.py.

Depends on https://github.com/facebook/rocksdb/issues/9305.

Pull Request resolved: https://github.com/facebook/rocksdb/pull/9338

Test Plan: Running all crash tests and many instances of blackbox. Sandcastle links are in Phabricator diff test plan.

Reviewed By: riversand963

Differential Revision: D33345333

Pulled By: ajkr

fbshipit-source-id: f56dd7d2e5a78d59301bf4fc3fedb980eb31e0ce
      6892f19b
    • A
      Recover to exact latest seqno of data committed to MANIFEST (#9305) · b860a421
      由 Andrew Kryczka 提交于 
      Summary:
The LastSequence field in the MANIFEST file is the baseline seqno for a recovered DB. Recovering WAL entries might cause the recovered DB's seqno to advance above this baseline, but the recovered DB will never use a smaller seqno.

Before this PR, we were writing the DB's seqno at the time of LogAndApply() as the LastSequence value. This works in the sense that it is a large enough baseline for the recovered DB that it'll never overwrite any records in existing SST files. At the same time, it's arbitrarily larger than what's needed. This behavior comes from LevelDB, where there was no tracking of largest seqno in an SST file.

Now we know the largest seqno of newly written SST files, so we can write an exact value in LastSequence that actually reflects the largest seqno in any file referred to by the MANIFEST. This is primarily useful for correctness testing with unsynced data loss, where the recovered DB's seqno needs to indicate what records were recovered.

Pull Request resolved: https://github.com/facebook/rocksdb/pull/9305

Test Plan:
- https://github.com/facebook/rocksdb/issues/9338 adds crash-recovery correctness testing coverage for WAL disabled use cases
- https://github.com/facebook/rocksdb/issues/9357 will extend that testing to cover file ingestion
- Added assertion at end of LogAndApply() for `VersionSet::descriptor_last_sequence_` consistency with files
- Manually tested upgrade/downgrade compatibility with a custom crash test that randomly picks between a `db_stress` built with and without this PR (for old code it must run with `-disable_wal=0`)

Reviewed By: riversand963

Differential Revision: D33182770

Pulled By: ajkr

fbshipit-source-id: 0bfafaf685f347cc8cb0e1d62e0186340a738f7d
      b860a421
  25. 05 1月, 2022 1 次提交
  27. 31 12月, 2021 1 次提交
    • Y
      Fix a bug in C-binding causing iterator to return incorrect result (#9343) · 677d2b4a
      由 Yanqin Jin 提交于 
      Summary:
Fixes https://github.com/facebook/rocksdb/issues/9339

When writing SST file, the name, computed as `prefix_extractor->GetId()` will be written to the properties block.
When the SST is opened again in the future, `CreateFromString()` will take the name as argument and try
to create a prefix extractor object. Without this fix, the C API will pass a `Wrapper` pointer to the underlying
DB's `prefix_extractor`. `Wrapper::GetId()`, in this case, will be missing the prefix length component, causing a
prefix extractor of length 0 to be silently created and used.

Pull Request resolved: https://github.com/facebook/rocksdb/pull/9343

Test Plan:
```
make c_test
./c_test
```

Reviewed By: mrambacher

Differential Revision: D33355549

Pulled By: riversand963

fbshipit-source-id: c92c3acd8be262c3bff8794b4229e42b9ee31203
      677d2b4a