Skip to content

R
rocksdb

kvdb / rocksdb
11 个月 前同步成功

通知 0
Star 0
Fork 0
R
rocksdb

体验新版 GitCode，发现更多精彩内容 >>

切换分支/标签
  1. 04 9月, 2020 4 次提交
  3. 03 9月, 2020 2 次提交
  5. 02 9月, 2020 3 次提交
    • A
      Bound L0->Lbase fanout in dynamic leveled compaction (#7325) · 177f8bd0
      由 Andrew Kryczka 提交于 
      Summary:
L0 score is based on size target and number of files. The size target
used is `max_bytes_for_level_base`. However, the base level's size can
dynamically expand in write burst mode. In fact, it can expand so much
that L0->Lbase becomes the highest fanout in target sizes. This doesn't
make sense from an efficiency perspective, so this PR bounds the
L0->Lbase fanout to the smoothed level multiplier. The L0 scoring based
on file count remains unchanged.

Pull Request resolved: https://github.com/facebook/rocksdb/pull/7325

Test Plan:
contrived benchmark that exhibits the problem:

```
$ TEST_TMPDIR=/data/users/andrewkr/ ./db_bench -benchmarks=filluniquerandom,readrandom -write_buffer_size=1048576 -target_file_size_base=1048576 -max_bytes_for_level_base=4194304 -level0_file_num_compaction_trigger=4 -level_compaction_dynamic_level_bytes=true -compression_type=none -max_background_jobs=12 -rate_limiter_bytes_per_sec=104857600 -benchmark_write_rate_limit=10485760 -num=100000000
```

Results:

- "Burst W-Amp" is the write-amp near the end of the fillrandom benchmark
- "Total W-Amp" is the write-amp after readrandom has run a while and all levels no longer need compaction

Branch | Burst W-Amp | Total W-Amp | fillrandom (MB/s)
-- | -- | -- | --
master | 20.2 | 21.5 | 4.7
dynamic-l0-score | 12.6 | 14.1 | 7.2

Reviewed By: siying

Differential Revision: D23412935

Pulled By: ajkr

fbshipit-source-id: f91f2067188e432dd39deab02f1c56f195057a0e
      177f8bd0
    • T
      Make examples work on Windows (#7304) · 8d44d792
      由 Tomasz Posłuszny 提交于 
      Summary:
Quick fixes to examples to make it easier to get familiar with RocksDB for Windows users:

- Set proper temporary directory path on Windows for all examples (with C++17 we should start using std::filesystem)
- Fixed typo and got rid of warnings treated as errors in multi_processes_example.cc
- Get number of available cores on Windows in c_simple_example.c
- Add command to remove DB directory for Windows in compaction_filter_example.cc (print error, but carry on with example upon error, because error code is returned if there is no such directory on Windows)

Pull Request resolved: https://github.com/facebook/rocksdb/pull/7304

Reviewed By: jay-zhuang

Differential Revision: D23450900

Pulled By: pdillinger

fbshipit-source-id: 4256134deb6ae6bb267ed1bd69f814842b95f60f
      8d44d792
    • J
      Recompress blobs during GC if compression changed (#7331) · 55bf42a8
      由 Jay Zhuang 提交于 
      Summary:
Recompress blobs if compression type is changed.

Pull Request resolved: https://github.com/facebook/rocksdb/pull/7331

Test Plan: `make check`

Reviewed By: ltamasi

Differential Revision: D23437102

Pulled By: jay-zhuang

fbshipit-source-id: bb699ebdad137721d422e42e331d4de8a82a7c5f
      55bf42a8
  7. 01 9月, 2020 1 次提交
    • L
      Log info about generated blob files in BlobFileBuilder (#7324) · 792d2f90
      由 Levi Tamasi 提交于 
      Summary:
The patch adds a log message to `BlobFileBuilder` that is logged upon
generating a blob file, similarly to how we log the generation of table files
during flush and compaction. The log message contains the column family
name, job id, blob file number, and the number and total size of blobs in
the new file.

Pull Request resolved: https://github.com/facebook/rocksdb/pull/7324

Test Plan: Ran `make check` and checked the actual log messages using a custom `db_bench`.

Reviewed By: riversand963

Differential Revision: D23402229

Pulled By: ltamasi

fbshipit-source-id: ca42beb4db284b783d1eb2651f321032a45d0c5f
      792d2f90
  9. 29 8月, 2020 2 次提交
  11. 28 8月, 2020 3 次提交
    • J
      Add buffer prefetch support for non directIO usecase (#7312) · c2485f2d
      由 Jay Zhuang 提交于 
      Summary:
A new file interface `SupportPrefetch()` is added. When the user overrides it to `false`, an internal prefetch buffer will be used for readahead. Useful for non-directIO but FS doesn't have readahead support.

Pull Request resolved: https://github.com/facebook/rocksdb/pull/7312

Reviewed By: anand1976

Differential Revision: D23329847

Pulled By: jay-zhuang

fbshipit-source-id: 71cd4ce6f4a820840294e4e6aec111ab76175527
      c2485f2d
    • L
      Add a blob file builder class that can be used in background jobs (#7306) · 50439606
      由 Levi Tamasi 提交于 
      Summary:
The patch adds a class called `BlobFileBuilder` that can be used to build
and cut blob files in background jobs (flushes/compactions). The class
enforces a value size threshold (`min_blob_size`; smaller blobs will be inlined
in the LSM tree itself), and supports specifying a blob file size limit (`blob_file_size`),
as well as compression (`blob_compression_type`) and checksums for blob files.
It also keeps track of the generated blob files and their associated `BlobFileAddition`
metadata, which can be applied as part of the background job's `VersionEdit`.

Pull Request resolved: https://github.com/facebook/rocksdb/pull/7306

Test Plan: `make check`

Reviewed By: riversand963

Differential Revision: D23298817

Pulled By: ltamasi

fbshipit-source-id: 38f35d81dab1ba81f15236240612ec173d7f21b5
      50439606
    • A
      Store FSRandomAccessPtr object in RandomAccessFileReader (#7192) · 8e0df905
      由 Akanksha Mahajan 提交于 
      Summary:
Replace FSRandomAccessFile pointer with FSRandomAccessFilePtr
    object in RandomAccessFileReader.
    This new object wraps FSRandomAccessFile pointer.

    Objective: If tracing is enabled, FSRandomAccessFile Ptr returns
    FSRandomAccessFileTracingWrapper pointer that includes all necessary
    information in IORecord and calls underlying FileSystem and invokes
    IOTracer to dump that record in a binary file. If tracing is disabled
    then, underlying FileSystem pointer is returned directly.
    FSRandomAccessFilePtr wrapper class is added to bypass the FSRandomAccessFileWrapper when
    tracing is disabled.

    Test Plan: make check -j64

Pull Request resolved: https://github.com/facebook/rocksdb/pull/7192

Reviewed By: anand1976

Differential Revision: D23356867

Pulled By: akankshamahajan15

fbshipit-source-id: 48f31168166a17a7444b40be44a9a9d4a5c7182c
      8e0df905
  13. 27 8月, 2020 1 次提交
    • P
      Real fix for race in backup custom checksum checking (#7309) · 9aad24da
      由 Peter Dillinger 提交于 
      Summary:
This is a "real" fix for the issue worked around in https://github.com/facebook/rocksdb/issues/7294.
To get DB checksum info for live files, we now read the manifest file
that will become part of the checkpoint/backup. This requires a little
extra handling in taking a custom checkpoint, including only reading the
manifest file up to the size prescribed by the checkpoint.

This moves GetFileChecksumsFromManifest from backup code to
file_checksum_helper.{h,cc} and removes apparently unnecessary checking
related to column families.

Updated HISTORY.md and warned potential future users of
DB::GetLiveFilesChecksumInfo()

Pull Request resolved: https://github.com/facebook/rocksdb/pull/7309

Test Plan: updated unit test, before and after

Reviewed By: ajkr

Differential Revision: D23311994

Pulled By: pdillinger

fbshipit-source-id: 741e30a2dc1830e8208f7648fcc8c5f000d4e2d5
      9aad24da
  15. 26 8月, 2020 3 次提交
    • S
      Get() to fail with underlying failures in PartitionIndexReader::CacheDependencies() (#7297) · 722814e3
      由 sdong 提交于 
      Summary:
Right now all I/O failures under PartitionIndexReader::CacheDependencies() is swallowed. This doesn't impact correctness but we've made a decision that any I/O error in read path now should be returned to users for awareness. Return errors in those cases instead.

Pull Request resolved: https://github.com/facebook/rocksdb/pull/7297

Test Plan: Add a new unit test that ingest errors in this code path and see Get() fails. Only one I/O path is hit in PartitionIndexReader::CacheDependencies(). Several option changes are attempt but not able to got other pread paths triggered. Not sure whether other failure cases would be even possible. Would rely on continuous stress test to validate it.

Reviewed By: anand1976

Differential Revision: D23257950

fbshipit-source-id: 859dbc92fa239996e1bb378329344d3d54168c03
      722814e3
    • S
      Parameterize DBBasicTest.CompactBetweenSnapshots (#7301) · cecdd5d2
      由 sdong 提交于 
      Summary:
DBBasicTest.CompactBetweenSnapshots can time-out in some slow-I/O hosts. Parameterize it so that single test runs shorter.

Pull Request resolved: https://github.com/facebook/rocksdb/pull/7301

Test Plan: Run the test and see see different runs are of different configerations in a hacky way.

Reviewed By: ltamasi

Differential Revision: D23277733

fbshipit-source-id: 1f717b4131322d175abf9e211131fe7e9b1ef758
      cecdd5d2
    • Z
      Pass SST file checksum information through OnTableFileCreated (#7108) · d51f88c9
      由 Zhichao Cao 提交于 
      Summary:
When SST file is created, application is able to know the file information through OnTableFileCreated callback in LogAndNotifyTableFileCreationFinished. Since file checksum information can be useful for application when the SST file is created, we add file_checksum and file_checksum_func_name information to TableFileCreationInfo, which will be passed through OnTableFileCreated.

Pull Request resolved: https://github.com/facebook/rocksdb/pull/7108

Test Plan: make check, listener_test.

Reviewed By: ajkr

Differential Revision: D22470240

Pulled By: zhichao-cao

fbshipit-source-id: 92c20344d9b986eadfe3480f3769bf4add0dbaae
      d51f88c9
  17. 25 8月, 2020 6 次提交
  19. 22 8月, 2020 3 次提交
    • J
      Shutdown timer in destructor (#7292) · e500c730
      由 Jay Zhuang 提交于 
      Summary:
Make sure deleting a running timer works fine.

Pull Request resolved: https://github.com/facebook/rocksdb/pull/7292

Test Plan: unittest and an invalid benchmark command: `./db_bench --db=/tmp --use_existing_db=false --benchmarks=fred --compression_type=none`

Reviewed By: riversand963

Differential Revision: D23248500

Pulled By: jay-zhuang

fbshipit-source-id: 04111681b389a9aa23a439db4568d5ca351f1144
      e500c730
    • A
      Bug Fix for memtables not trimmed down. (#7296) · 38446126
      由 Akanksha Mahajan 提交于 
      Summary:
When a memtable is trimmed in MemTableListVersion, the memtable
is only added to delete list if it is
the last reference. However it is not the last reference as it is held
by the super version. But the super version would not be switched if the
delete list is empty. So the memtable is never destroyed and memory
usage increases beyond write_buffer_size +
max_write_buffer_size_to_maintain.

Pull Request resolved: https://github.com/facebook/rocksdb/pull/7296

Test Plan:
1.  ./db_bench -benchmarks=randomtransaction
-optimistic_transaction_db=1 -statistics -stats_interval_seconds=1
-duration=90 -num=500000 --max_write_buffer_size_to_maintain=16000000
--transaction_set_snapshot

Reviewed By: ltamasi

Differential Revision: D23267395

Pulled By: akankshamahajan15

fbshipit-source-id: 3a8d437fe9f4015f851ff84c0e29528aa946b650
      38446126
    • J
      Add test function MockTimeEnv.SleepForMicroseconds() (#7293) · 187964a0
      由 Jay Zhuang 提交于 
      Summary:
And change the internal time value from seconds to microseconds.

Pull Request resolved: https://github.com/facebook/rocksdb/pull/7293

Reviewed By: pdillinger

Differential Revision: D23253751

Pulled By: jay-zhuang

fbshipit-source-id: 36aa9376b8801b85bd10163173590a17cf4f3a3a
      187964a0
  21. 21 8月, 2020 5 次提交
  23. 20 8月, 2020 3 次提交
  25. 19 8月, 2020 3 次提交
    • L
      Add initial set of options for integrated blob write path (#7280) · b9bb59d4
      由 Levi Tamasi 提交于 
      Summary: Pull Request resolved: https://github.com/facebook/rocksdb/pull/7280

Test Plan: `make check`

Reviewed By: riversand963

Differential Revision: D23195192

Pulled By: ltamasi

fbshipit-source-id: 743b382de391963e62ba86119e9fbd0233ea3b3a
      b9bb59d4
    • A
      Store FSSequentialFilePtr object in SequenceFileReader (#7190) · cc24ac14
      由 Akanksha Mahajan 提交于 
      Summary:
This diff contains following changes:
    1. Replace `FSSequentialFile` pointer with `FSSequentialFilePtr` object that wraps `FSSequentialFile` pointer in `SequenceFileReader`.

Objective: If tracing is enabled, `FSSequentialFilePtr` returns `FSSequentialFileTracingWrapper` pointer that includes all necessary information in `IORecord` and calls underlying FileSystem and invokes `IOTracer` to dump that record in a binary file. If tracing is disabled then, underlying `FileSystem` pointer is returned directly. `FSSequentialFilePtr` wrapper class is added to bypass the `FSSequentialFileTracingWrapper` when tracing is disabled.

Pull Request resolved: https://github.com/facebook/rocksdb/pull/7190

Test Plan:
make check -j64
          COMPILE_WITH_TSAN=1 make check -j64

Reviewed By: anand1976

Differential Revision: D23059616

Pulled By: akankshamahajan15

fbshipit-source-id: 1564b94dd1297cd0fbfe2ed5c9cc3e20f7395301
      cc24ac14
    • A
      fix doc about kTolerateCorruptedTailRecords recovery (#7270) · e6e2f369
      由 Andrew Kryczka 提交于 
      Summary:
- Made it clear only one record in the tail is allowed to have a problem
- Added detail about the valid use case instead of calling it legacy behavior

Pull Request resolved: https://github.com/facebook/rocksdb/pull/7270

Reviewed By: riversand963

Differential Revision: D23169075

Pulled By: ajkr

fbshipit-source-id: 2a4b45aa8641f17efa104523fbad765012a98fb0
      e6e2f369
  27. 18 8月, 2020 1 次提交
    • P
      Fix some flaky tests in BackupableDBTest with intentional flushing (#7273) · 7d0ecab5
      由 Peter Dillinger 提交于 
      Summary:
Some tests like BackupableDBTest.FileCollision and
ShareTableFilesWithChecksumsNewNaming are intermittently failing,
probably due to unpredictable flushing with FillDB. This change
should fix the failures seen and help to prevent similar flakiness in
future tests in the file.

Pull Request resolved: https://github.com/facebook/rocksdb/pull/7273

Test Plan: make check, and with valgrind

Reviewed By: siying

Differential Revision: D23176947

Pulled By: pdillinger

fbshipit-source-id: 654b73a64db475f2b9b065ed53a889a8b9083c59
      7d0ecab5