Skip to content

R
rocksdb

kvdb / rocksdb
11 个月 前同步成功

通知 0
Star 0
Fork 0
R
rocksdb

体验新版 GitCode，发现更多精彩内容 >>

切换分支/标签
  1. 04 8月, 2020 4 次提交
    • L
      Move CompressionType to its own header file (#7162) · 8cb278d1
      由 Levi Tamasi 提交于 
      Summary:
The patch moves `CompressionType` to its own header file and makes sure
all other public headers include this new header directly, as opposed to
relying on transitive includes or forward declarations.

Pull Request resolved: https://github.com/facebook/rocksdb/pull/7162

Test Plan: `make check`

Reviewed By: riversand963

Differential Revision: D22676545

Pulled By: ltamasi

fbshipit-source-id: 01d7a232377a229cbbc373d0ec1bf01dc0b0ce02
      8cb278d1
    • A
      dedup ReadOptions in iterator hierarchy (#7210) · a4a4a2da
      由 Andrew Kryczka 提交于 
      Summary:
Previously, a `ReadOptions` object was stored in every `BlockBasedTableIterator`
and every `LevelIterator`. This redundancy consumes extra memory,
resulting in the `Arena` making more allocations, and iteration
observing worse cache performance.

This PR migrates callers of `NewInternalIterator()` and
`MakeInputIterator()` to provide a `ReadOptions` object guaranteed to
outlive the returned iterator. When the iterator's lifetime will be managed by the
user, this lifetime guarantee is achieved by storing the `ReadOptions`
value in `ArenaWrappedDBIter`. Then, sub-iterators of `NewInternalIterator()` and
`MakeInputIterator()` can hold a reference-to-const `ReadOptions`.

Pull Request resolved: https://github.com/facebook/rocksdb/pull/7210

Test Plan:
- `make check` under ASAN and valgrind
- benchmark: on a DB with 2 L0 files and 3 L1+ levels, this PR reduced `Arena` allocation 4792 -> 4160 bytes.

Reviewed By: anand1976

Differential Revision: D22861323

Pulled By: ajkr

fbshipit-source-id: 54aebb3e89c872eeab0f5793b4b6e42878d093ce
      a4a4a2da
    • A
      Add defaults to ReadOptions doc (#7215) · 18efd760
      由 Adam Retter 提交于 
      Summary:
Very small improvements to document the defaults.

Pull Request resolved: https://github.com/facebook/rocksdb/pull/7215

Reviewed By: zhichao-cao

Differential Revision: D22902286

fbshipit-source-id: a754d172a0d8e4c03754f6f1771d4a693d60a770
      18efd760
    • J
      Fix hang timer tests on macos (#7208) · d941b89d
      由 Jay Zhuang 提交于 
      Summary:
And re-enable disabled tests.
The issue is caused by `CondVar.TimedWait()` doesn't use `MockTimeEnv`.

Issue: https://github.com/facebook/rocksdb/issues/6698

Pull Request resolved: https://github.com/facebook/rocksdb/pull/7208

Test Plan: `./timer_test --gtest_repeat=1000`

Reviewed By: riversand963

Differential Revision: D22857855

Pulled By: jay-zhuang

fbshipit-source-id: 6d15f65f6ae58b75b76cb132815c16ad81ffd12f
      d941b89d
  3. 30 7月, 2020 4 次提交
    • A
      Compaction Read/Write Stats by Compaction Type (#7165) · 56ed601d
      由 Aaron Kabcenell 提交于 
      Summary:
Adds compaction statistics (total bytes read and written) for compactions that occur for delete-triggered, periodic, and TTL compaction reasons.

Pull Request resolved: https://github.com/facebook/rocksdb/pull/7165

Test Plan:
TTL and periodic can be checked by runnning db_bench with the options activated:

/db_bench --benchmarks="fillrandom,stats" --statistics --num=10000000 -base_background_compactions=16 -periodic_compaction_seconds=1
./db_bench --benchmarks="fillrandom,stats" --statistics --num=10000000 -base_background_compactions=16 -fifo_compaction_ttl=1

Setting the time to one second causes non-zero bytes read/written for those compaction reasons. Disabling them or setting them to times longer than the test run length causes the stats to return to zero as expected.

Delete-triggered compaction counting is tested in DBTablePropertiesTest.DeletionTriggeredCompactionMarking

Reviewed By: ajkr

Differential Revision: D22693050

Pulled By: akabcenell

fbshipit-source-id: d15cef4d94576f703015c8942d5f0d492f69401d
      56ed601d
    • C
      feat: export SetBackgroundThreads(n, Env::BOTTOM); (#7191) · 50f206ad
      由 codingsh 提交于 
      Summary:
- https://github.com/rust-rocksdb/rust-rocksdb/pull/448

Pull Request resolved: https://github.com/facebook/rocksdb/pull/7191

Reviewed By: riversand963

Differential Revision: D22809066

Pulled By: ajkr

fbshipit-source-id: 036939f9a28cacc3f677c318d1aed97fe5f4f85e
      50f206ad
    • Y
      Update HISTORY and version for 6.12 release (#7194) · a38f04ac
      由 Yanqin Jin 提交于 
      Summary: Pull Request resolved: https://github.com/facebook/rocksdb/pull/7194

Reviewed By: gg814

Differential Revision: D22810654

Pulled By: riversand963

fbshipit-source-id: 01f13089fa2b7e31b827da3e30c90e5c62c41380
      a38f04ac
    • S
      Implement NextAndGetResult() in memtable and level iterator (#7179) · 692f6a31
      由 sdong 提交于 
      Summary:
NextAndGetResult() is not implemented in memtable and is very simply implemented in level iterator. The result is that for a normal leveled iterator, performance regression will be observed for calling PrepareValue() for most iterator Next(). Mitigate the problem by implementing the function for both iterators. In level iterator, the implementation cannot be perfect as when calling file iterator's SeekToFirst() we don't have information about whether the value is prepared. Fortunately, the first key should not cause a big portion of the CPu.

Pull Request resolved: https://github.com/facebook/rocksdb/pull/7179

Test Plan: Run normal crash test for a while.

Reviewed By: anand1976

Differential Revision: D22783840

fbshipit-source-id: c19f45cdf21b756190adef97a3b66ccde3936e05
      692f6a31
  5. 29 7月, 2020 5 次提交
  7. 25 7月, 2020 6 次提交
    • C
      Fix RandomAccessFileReaderTest failures on Travis (#7173) · 69a6d0b4
      由 Cheng Chang 提交于 
      Summary:
On Travis, the old `alignment()` returned by `RandomAccessFileReaderTest` is inconsistent with the `GetRequiredBufferAlignment` returned in `RandomAccessFileReader`. This PR removes `alignment()` and consistently use `GetRequiredBufferAlignment` as page size.

Pull Request resolved: https://github.com/facebook/rocksdb/pull/7173

Test Plan:
make random_access_file_reader_test && ./random_access_file_reader_test
Watch Travis

Reviewed By: siying

Differential Revision: D22741606

Pulled By: cheng-chang

fbshipit-source-id: f28f29a7c993bbc3594ae70ecd186fa8bab9c4f2
      69a6d0b4
    • Y
      Enable a few jobs in determinator (#7174) · 1adfd729
      由 Yanqin Jin 提交于 
      Summary:
https://github.com/facebook/rocksdb/issues/7170 added a few job specs. This PR enables rocksdb-lego-determinator to support them.

Test plan (dev server)
```
$build_tools/rocksdb-lego-determinator blackbox_stress_crash
$build_tools/rocksdb-lego-determinator whitebox_stress_crash
$build_tools/rocksdb-lego-determinator blackbox_asan_crash
$build_tools/rocksdb-lego-determinator whitebox_asan_crash
$build_tools/rocksdb-lego-determinator blackbox_ubsan_crash
$build_tools/rocksdb-lego-determinator whitebox_ubsan_crash
$build_tools/rocksdb-lego-determinator blackbox_tsan_crash
$build_tools/rocksdb-lego-determinator whitebox_tsan_crash
```

Pull Request resolved: https://github.com/facebook/rocksdb/pull/7174

Reviewed By: siying

Differential Revision: D22741153

Pulled By: riversand963

fbshipit-source-id: 39b7d948f04a5b109f009b5499c1dbdc83a13c6e
      1adfd729
    • A
      Fix for flaky test BackupableDBTest.RateLimiting (#7167) · 7e37a591
      由 Akanksha Mahajan 提交于 
      Summary:
BackupableDBTest.RateLimiting test is failing due to timed out
on our test server. It might be because of nested loops run sequentially that test different type of combinations of parameters. This patch converts the test into parameterized test so that all combinations can be tested out.

Pull Request resolved: https://github.com/facebook/rocksdb/pull/7167

Test Plan: make check -j64

Reviewed By: zhichao-cao

Differential Revision: D22709531

Pulled By: akankshamahajan15

fbshipit-source-id: 95518153e87b3b5311a6c1960a191bca58898786
      7e37a591
    • J
      Remove redundant ROCKSDB_LITE check (#7172) · 0c5bb10f
      由 Jay Zhuang 提交于 
      Summary:
It's already inside of a `#ifdef ROCKSDB_LITE` block.

Pull Request resolved: https://github.com/facebook/rocksdb/pull/7172

Reviewed By: gg814

Differential Revision: D22736057

Pulled By: jay-zhuang

fbshipit-source-id: 31f4aa05aba98e2e42fa6f890fa72acf3a0f12f2
      0c5bb10f
    • T
      SST Partitioner interface that allows to split SST files (#6957) · cd4592c2
      由 Tomas Kolda 提交于 
      Summary:
SST Partitioner interface that allows to split SST files during compactions.

It basically instruct compaction to create a new file when needed. When one is using well defined prefixes and prefixed way of defining tables it is good to define also partitioning so that promotion of some SST file does not cover huge key space on next level (worst case complete space).

Pull Request resolved: https://github.com/facebook/rocksdb/pull/6957

Reviewed By: ajkr

Differential Revision: D22461239

fbshipit-source-id: 9ce07bba08b3ba89c2d45630520368f704d1316e
      cd4592c2
    • Y
      Add job specs for blackbox/whitebox stress tests (#7170) · 954ee565
      由 Yanqin Jin 提交于 
      Summary:
As title.

Pull Request resolved: https://github.com/facebook/rocksdb/pull/7170

Test Plan: Manually invoke the commands.

Reviewed By: siying

Differential Revision: D22732256

Pulled By: riversand963

fbshipit-source-id: d331e5ee84658ac079814292ff1a1eacfd14bfdf
      954ee565
  9. 24 7月, 2020 1 次提交
  11. 23 7月, 2020 6 次提交
    • C
      Update HISTORY (#7158) · 7af1fab4
      由 Cheng Chang 提交于 
      Summary:
Mention the MultiRead bug in HISTORY.

Pull Request resolved: https://github.com/facebook/rocksdb/pull/7158

Test Plan: N/A

Reviewed By: siying

Differential Revision: D22670565

Pulled By: cheng-chang

fbshipit-source-id: 16abf0192957be66511f6a08e00157bfd37b189f
      7af1fab4
    • J
      Make max_subcompactions dynamically changeable (#7159) · b0c5ecd6
      由 Jay Zhuang 提交于 
      Summary:
Make `max-subcompactions` dynamically changeable by passing the `DBOption` to Compaction.

Pull Request resolved: https://github.com/facebook/rocksdb/pull/7159

Reviewed By: siying

Differential Revision: D22671238

Pulled By: jay-zhuang

fbshipit-source-id: 311ca9f6bb606965544d8708616d358cfed5be42
      b0c5ecd6
    • L
      Sync blob files before closing them (#7160) · 0d04a843
      由 Levi Tamasi 提交于 
      Summary:
BlobDB currently syncs each blob file periodically after writing a certain amount of
data (as specified by the configuration option `BlobDBOptions::bytes_per_sync`)
and all open blob files when the base DB's memtables are flushed. With the patch,
in addition to the above, blob files are also synced right before being closed, after
the footer has been written. This will be beneficial for the new integrated blob file
write path as well.

Pull Request resolved: https://github.com/facebook/rocksdb/pull/7160

Test Plan: `make check`

Reviewed By: riversand963

Differential Revision: D22672646

Pulled By: ltamasi

fbshipit-source-id: 62b34263543a7e74abcbb7adf011daa1e699998f
      0d04a843
    • J
      Fix bug in MultiRead() coalescing introduced in 4fc21664 (#6446). (#6979) · 4a60cb20
      由 Jason Volk 提交于 
      Summary:
TryMerge() overzealously creates one huge file read request in an attempt to merge smaller disjoint requests. For example, ~30 input requests of ~100 bytes output as 1 request of 100 MiB causing alarmingly large read throughputs to be repeatedly observed by the environment.
Signed-off-by: NJason Volk <jason@zemos.net>

Pull Request resolved: https://github.com/facebook/rocksdb/pull/6979

Reviewed By: siying

Differential Revision: D22668892

Pulled By: cheng-chang

fbshipit-source-id: 7506fe9621b7f1a747dadf6b8ddb1b1a141c1937
      4a60cb20
    • C
      Clean snapshot dir before taking snapshot (#7156) · 96ce0470
      由 Cheng Chang 提交于 
      Summary:
`DBTest::SnapshotFiles` runs the tests in a `while` loop.
Currently, the snapshot directory is not cleaned up in each loop, so previous snapshot files may remain in the next loop's snapshot.
When I'm working on https://github.com/facebook/rocksdb/pull/7129, when checking the tracked WALs in MANIFEST, I find that this test always fails because it reads some unknown WAL. It turns out that the unknown WAL is left from previous loops.

Pull Request resolved: https://github.com/facebook/rocksdb/pull/7156

Test Plan: make db_test && ./db_test --gtest_filters=*SnapshotFiles

Reviewed By: siying

Differential Revision: D22668360

Pulled By: cheng-chang

fbshipit-source-id: 69d4aa3506038ba30e218e8ae966357935a99c6c
      96ce0470
    • M
      Add hash of key/value checks when paranoid_file_checks=true (#7134) · d44cbc53
      由 mrambacher 提交于 
      Summary:
When paraoid_files_checks=true, a rolling key-value hash is generated and compared to what is written to the file.  If the values do not match, the SST file is rejected.

Code put in place for the check for both flush and compaction jobs.  Corresponding test added to corruption_test.

Pull Request resolved: https://github.com/facebook/rocksdb/pull/7134

Reviewed By: cheng-chang

Differential Revision: D22646149

fbshipit-source-id: 8fde1984a1a11edd3bd82a413acffc5ea7aa683f
      d44cbc53
  13. 22 7月, 2020 2 次提交
    • H
      Use steady_clock instead of system_clock in FileOperationInfo::TimePoint (#7153) · dbc51adb
      由 Haosen Wen 提交于 
      Summary:
Issue https://github.com/facebook/rocksdb/issues/7133 reported that using `system_clock` in `FileOperationInfo::TimePoint` causes the duration of file flush operation (which can be a noop on MacOS in some scenarios) appears to be 0 and fail an assertion in listener_test. Using `steady_clock` supposedly fixed the problem.
`steady_clock` actually fits better into the use cases of `FileOperationInfo::TimePoint` as all usages care about durations but not wall clock time.

Pull Request resolved: https://github.com/facebook/rocksdb/pull/7153

Test Plan: make check.

Reviewed By: riversand963

Differential Revision: D22654136

Pulled By: roghnin

fbshipit-source-id: 5980b1080734bdae496a18071a2c2b5887c67d85
      dbc51adb
    • Z
      BackupEngine computes table checksums only once if db session ids are available (#7110) · b923dc72
      由 Zitan Chen 提交于 
      Summary:
BackupEngine requires computing table checksums twice when backing up table files to the `shared_checksum` directory.

The repeated computation can be avoided by utilizing the db session id stored as a part of the table properties.

Filenames of table files in the `shared_checksum` directory depend on the following conditions:
1. the naming scheme is `kOptionalChecksumAndDbSessionId`,
2. `db_session_id` is not empty,
3. checksum is available in the DB manifest.

If 1,2,3 are satisfied, then the filenames will be of the form `<file_number>_<checksum>_<db_session_id>.sst`.
If 1,2 are satisfied, then the filenames will be of the form `<file_number>_<db_session_id>.sst`.
In all other cases, the filenames are of the form `<file_number>_<checksum>_<size>.sst`.

Additionally, if `kOptionalChecksumAndDbSessionId` is used (and not falling back to `kChecksumAndFileSize`), the `<checksum>` appeared in the filenames is hexadecimally encoded, instead of being plain `uint32_t` value.

Pull Request resolved: https://github.com/facebook/rocksdb/pull/7110

Test Plan: backupable_db_test and manual tests.

Reviewed By: ajkr

Differential Revision: D22508992

Pulled By: gg814

fbshipit-source-id: 5669f0ea9ad5a097f69f6d87aca4abba15032389
      b923dc72
  15. 21 7月, 2020 5 次提交
    • S
      Exclude two tests in CircleCI TSAN tests (#7152) · 0f487cc3
      由 sdong 提交于 
      Summary:
Two TSAN tests occaionaly fail. Exclude them for now:

[ RUN      ] DeleteFileTest.BackgroundPurgeCFDropTest
db/deletefile_test.cc:122: Failure
Expected equality of these values:
  required_manifest
    Which is: 1
  manifest_cnt
    Which is: 2

[ RUN      ] FormatLatest/ColumnFamilyTest.FlushCloseWALFiles/0
db/column_family_test.cc:3004: Failure
Expected equality of these values:
  2
  env.num_open_wal_file_.load()
    Which is: 1

Pull Request resolved: https://github.com/facebook/rocksdb/pull/7152

Test Plan: Watch CircleCI restuls

Reviewed By: ajkr

Differential Revision: D22632285

fbshipit-source-id: 29fa348e8be917be0237c74812a8b0b04978e84e
      0f487cc3
    • S
      column_family_test: fix a data race related to sleeping task (#7150) · 1cf4731d
      由 sdong 提交于 
      Summary:
TSAN reports warning in one column_family_test:

WARNING: ThreadSanitizer: data race (pid=16352)
  Write of size 8 at 0x7ffcdf042158 by main thread:
    #0 pthread_cond_destroy <null> (column_family_test+0x471f65)
    https://github.com/facebook/rocksdb/issues/1 rocksdb::port::CondVar::~CondVar() /home/circleci/project/port/port_posix.cc:101:49 (column_family_test+0x8a627a)
    https://github.com/facebook/rocksdb/issues/2 rocksdb::test::SleepingBackgroundTask::~SleepingBackgroundTask() /home/circleci/project/./test_util/testutil.h:397:7 (column_family_test+0x54b6e2)
    https://github.com/facebook/rocksdb/issues/3 rocksdb::ColumnFamilyTest_FlushCloseWALFiles_Test::TestBody() /home/circleci/project/db/column_family_test.cc:3008:1 (column_family_test+0x54b6e2)
......
  Previous read of size 8 at 0x7ffcdf042158 by thread T2 (mutexes: write M0):
    #0 pthread_cond_broadcast <null> (column_family_test+0x471dd2)
    https://github.com/facebook/rocksdb/issues/1 rocksdb::port::CondVar::SignalAll() /home/circleci/project/port/port_posix.cc:139:28 (column_family_test+0x8a651a)
    https://github.com/facebook/rocksdb/issues/2 rocksdb::test::SleepingBackgroundTask::DoSleep() /home/circleci/project/./test_util/testutil.h:412:12 (column_family_test+0x58574b)
......

Likely, SleepingBackgroundTask::DoSleep() started to execute after the main thread has finished everything, cancelled and waited for sleeping tasks to finish. At this time, although DoSlee() will not sleep, but it also accesses the mutex, creating a data race with destructor of the test. Fix this bug by waiting for the sleeping task to start sleeping after it is scheduled.

Pull Request resolved: https://github.com/facebook/rocksdb/pull/7150

Test Plan: Run these modified tests and make sure it doesn't break.

Reviewed By: riversand963

Differential Revision: D22630716

fbshipit-source-id: cc5781cf69083685de406490438898238bdfc2d3
      1cf4731d
    • A
      minimize BlockIter comparator scope (#7149) · 643c863b
      由 Andrew Kryczka 提交于 
      Summary:
PR https://github.com/facebook/rocksdb/issues/6944 transitioned `BlockIter` from using `Comparator*` to using
concrete `UserComparatorWrapper` and `InternalKeyComparator`. However,
adding them as instance variables to `BlockIter` was not optimal.
Bloating `BlockIter` caused the `ArenaWrappedDBIter`'s arena allocator to do more heap
allocations (in certain cases) which harmed performance of `DB::NewIterator()`. This PR
pushes down the concrete comparator objects to the point of usage, which
forces them to be on the stack. As a result, the `BlockIter` is back to
its original size prior to https://github.com/facebook/rocksdb/issues/6944 (actually a bit smaller since there
were two `Comparator*` before).

Pull Request resolved: https://github.com/facebook/rocksdb/pull/7149

Test Plan:
verified our internal `DB::NewIterator()`-heavy regression
test no longer reports regression.

Reviewed By: riversand963

Differential Revision: D22623189

Pulled By: ajkr

fbshipit-source-id: f6d69accfe5de51e0bd9874a480b32b29909bab6
      643c863b
    • S
      Fix a minor data race in stats dumping threads initialization (#7151) · 98707044
      由 sdong 提交于 
      Summary:
https://github.com/facebook/rocksdb/pull/7145 creates a minor data race against the stat creation counter. Turn it to atomic.

Pull Request resolved: https://github.com/facebook/rocksdb/pull/7151

Test Plan: Run the test.

Reviewed By: ajkr

Differential Revision: D22631014

fbshipit-source-id: c6fb69ac5b9df7139795dacea5ce9fb9fd3278d7
      98707044
    • J
      Store the test results to CircleCI (#7137) · 77062cf1
      由 Jay Zhuang 提交于 
      Summary:
To have test report.

Pull Request resolved: https://github.com/facebook/rocksdb/pull/7137

Reviewed By: siying

Differential Revision: D22630798

Pulled By: jay-zhuang

fbshipit-source-id: bc07ba673c0bceed5a4829b4af2d9a74435379c7
      77062cf1
  17. 18 7月, 2020 5 次提交
  19. 17 7月, 2020 1 次提交
    • M
      Add Support for saving CompressionOptions to Options File (#6817) · ec711b23
      由 mrambacher 提交于 
      Summary:
This PR does a few things:
- The "compression_opts" and "bottom_compression_opts" can now be read/written as name/value pairs of options (instead of only a colon-separated list;
- These options can now be read/written to the Options file;
- The parallel_threads value can now be set (either in the colon or name-value format).

The compression options are now stored and treated as a OptionTypeInfo::Struct by the options system, meaning they can be read and written like the other structs.  This change allows them to be read/written easily to the options file.

Additionally, the colon-format was extended to allow support for setting parallel threads.  Tests were added to test all of the option settings via the optional parameters in the colon format.

Pull Request resolved: https://github.com/facebook/rocksdb/pull/6817

Reviewed By: ajkr

Differential Revision: D22396004

Pulled By: zhichao-cao

fbshipit-source-id: 38bcf74b7e9cd5bc2a84540fac2e9ba4f765b2c8
      ec711b23
  21. 16 7月, 2020 1 次提交