- 30 6月, 2021 2 次提交
-
-
由 Jay Zhuang 提交于
Summary: Change the job_id for remote compaction interface, which will include both internal compaction job_id, also a sub_compaction_job_id. It is not a backward compatible change. The user needs to update interface during upgrade. (We will avoid backward incompatible change after the feature is not experimental.) Pull Request resolved: https://github.com/facebook/rocksdb/pull/8364 Reviewed By: ajkr Differential Revision: D28917301 Pulled By: jay-zhuang fbshipit-source-id: 6d72a21f652bb517ad6954d0387b496797fc4e11
-
由 mrambacher 提交于
Summary: Useful for allowing new classes to create and manage Customizable objects without using internal APIs. Pull Request resolved: https://github.com/facebook/rocksdb/pull/8301 Reviewed By: zhichao-cao Differential Revision: D29428303 Pulled By: mrambacher fbshipit-source-id: 3d33d5197cc8379fe35b54d3d169f91f0dfe7a47
-
- 29 6月, 2021 3 次提交
-
-
由 Andrew Kryczka 提交于
Summary: Original author: kraj (https://github.com/facebook/rocksdb/issues/8413) We have a hack to ensure clang's `posix_memalign()` hack works to be compatible with glibc's `posix_memalign()` declaration. Our side of the hack is irrelevant and should be omitted when not using glibc. Pull Request resolved: https://github.com/facebook/rocksdb/pull/8425 Reviewed By: mrambacher Differential Revision: D29239029 Pulled By: ajkr fbshipit-source-id: 12b900f50a4823b880a6558f25d8590dbfc0aa26
-
由 Lucian Petrut 提交于
Summary: Allow using WindowsThread with Mingw Most Mingw builds require Posix threads in order to use std::thread. As per https://github.com/facebook/rocksdb/issues/7764, this is not always the case. That being considered, we're going to improve the Mingw thread model checks. Closes: https://github.com/facebook/rocksdb/issues/7764Signed-off-by: NLucian Petrut <lpetrut@cloudbasesolutions.com> Pull Request resolved: https://github.com/facebook/rocksdb/pull/8108 Reviewed By: jay-zhuang Differential Revision: D27365778 Pulled By: mrambacher fbshipit-source-id: 2c15b1f04ae90e1e3a25a33e86ceb779224a9529
-
由 mrambacher 提交于
Summary: If a Customizable option was not mutable, it would still appear in the list of mutable options when serialized. This meant that when the immutable options were used to configure another immutable object, an "option not changeable" status would be returned. Pull Request resolved: https://github.com/facebook/rocksdb/pull/8457 Reviewed By: zhichao-cao Differential Revision: D29428298 Pulled By: mrambacher fbshipit-source-id: 3945b0b822f8e5955a7c5590fe64dfd5bc1fe6a0
-
- 28 6月, 2021 2 次提交
-
-
由 mrambacher 提交于
Summary: Added BlobMetaData to ColumnFamilyMetaData and LiveBlobMetaData and DB API GetLiveBlobMetaData to retrieve it. First pass at struct. More tests and maybe fields to come... Pull Request resolved: https://github.com/facebook/rocksdb/pull/8273 Reviewed By: ltamasi Differential Revision: D29102400 Pulled By: mrambacher fbshipit-source-id: 8a2383a4446328be6b91dced9841fdd3dfc80b73
-
由 anand76 提交于
Summary: Add a ```-secondary_cache_uri``` to db_stress to allow the user to specify a custom ```SecondaryCache``` object from the object registry. Also allow db_crashtest.py to be run with an alternate db_stress location. Together, these changes will allow us to run db_stress using FB internal components. Pull Request resolved: https://github.com/facebook/rocksdb/pull/8455 Reviewed By: zhichao-cao Differential Revision: D29371972 Pulled By: anand1976 fbshipit-source-id: dd1b1fd80ebbedc11aa63d9246ea6ae49edb77c4
-
- 26 6月, 2021 1 次提交
-
-
由 Akanksha Mahajan 提交于
Summary: Run Merge with Intergrated BlobDB in stress tests, crash tests and db_bench. Pull Request resolved: https://github.com/facebook/rocksdb/pull/8461 Test Plan: 1. python3 -u tools/db_crashtest.py --simple whitebox ---use_merge=1 --enable_blob_files=1 2. ./db_bench --benchmarks="readwhilemerging" --merge_operator=uint64add --enable_blob_files=true Reviewed By: ltamasi Differential Revision: D29394824 Pulled By: akankshamahajan15 fbshipit-source-id: 0a8e492b13129673e088fb8af3402ab678bb473a
-
- 25 6月, 2021 6 次提交
-
-
由 Zhichao Cao 提交于
Summary: In PR https://github.com/facebook/rocksdb/issues/7523 , checksum handoff is introduced in RocksDB for WAL, Manifest, and SST files. When user enable checksum handoff for a certain type of file, before the data is written to the lower layer storage system, we calculate the checksum (crc32c) of each piece of data and pass the checksum down with the data, such that data verification can be down by the lower layer storage system if it has the capability. However, it cannot cover the whole lifetime of the data in the memory and also it potentially introduces extra checksum calculation overhead. In this PR, we introduce a new interface in WritableFileWriter::Append, which allows the caller be able to pass the data and the checksum (crc32c) together. In this way, WritableFileWriter can directly use the pass-in checksum (crc32c) to generate the checksum of data being passed down to the storage system. It saves the calculation overhead and achieves higher protection coverage. When a new checksum is added with the data, we use Crc32cCombine https://github.com/facebook/rocksdb/issues/8305 to combine the existing checksum and the new checksum. To avoid the segmenting of data by rate-limiter before it is stored, rate-limiter is called enough times to accumulate enough credits for a certain write. This design only support Manifest and WAL which use log_writer in the current stage. Pull Request resolved: https://github.com/facebook/rocksdb/pull/8412 Test Plan: make check, add new testing cases. Reviewed By: anand1976 Differential Revision: D29151545 Pulled By: zhichao-cao fbshipit-source-id: 75e2278c5126cfd58393c67b1efd18dcc7a30772
-
由 Andrew Kryczka 提交于
Summary: Pull Request resolved: https://github.com/facebook/rocksdb/pull/8460 Reviewed By: jay-zhuang Differential Revision: D29381865 Pulled By: ajkr fbshipit-source-id: 47ba54c25f3cc039d72ea32e1df20875795683b3
-
由 Akanksha Mahajan 提交于
Summary: Provide support for Merge operation with base values during Compaction in IntegratedBlobDB. Pull Request resolved: https://github.com/facebook/rocksdb/pull/8445 Test Plan: Add new unit test Reviewed By: ltamasi Differential Revision: D29343949 Pulled By: akankshamahajan15 fbshipit-source-id: 844f6f02f93388a11e6e08bda7bb3a2a28e47c70
-
由 Levi Tamasi 提交于
Summary: Pull Request resolved: https://github.com/facebook/rocksdb/pull/8458 Reviewed By: zhichao-cao Differential Revision: D29378728 Pulled By: ltamasi fbshipit-source-id: d5a40b1414500f53823763be5c2bfce8db04daf8
-
由 Levi Tamasi 提交于
Summary: The patch builds on `BlobGarbageMeter` and `BlobCountingIterator` (introduced in https://github.com/facebook/rocksdb/issues/8426 and https://github.com/facebook/rocksdb/issues/8443 respectively) and ties it all together. It measures the amount of garbage generated by a compaction and logs the corresponding `BlobFileGarbage` records as part of the compaction job's `VersionEdit`. Note: in order to have accurate results, `kRemoveAndSkipUntil` for compaction filters is implemented using iteration. Pull Request resolved: https://github.com/facebook/rocksdb/pull/8450 Test Plan: Ran `make check` and the crash test script. Reviewed By: jay-zhuang Differential Revision: D29338207 Pulled By: ltamasi fbshipit-source-id: 4381c432ac215139439f6d6fb801a6c0e4d8c128
-
由 Peter (Stig) Edwards 提交于
Summary: Hello and thanks for RocksDB, Here is a PR to add file deletes, renames and ```Flush()```, ```Sync()```, ```Fsync()``` and ```Close()``` to file ops report. The reason is to help tune RocksDB options when using an env/filesystem with high latencies for file level ("metadata") operations, typically seen during ```DB::Open``` (```db_bench -num 0``` also see https://github.com/facebook/rocksdb/pull/7203 where IOTracing does not trace ```DB::Open```). Before: ``` > db_bench -benchmarks updaterandom -num 0 -report_file_operations true ... Entries: 0 ... Num files opened: 12 Num Read(): 6 Num Append(): 8 Num bytes read: 6216 Num bytes written: 6289 ``` After: ``` > db_bench -benchmarks updaterandom -num 0 -report_file_operations true ... Entries: 0 ... Num files opened: 12 Num files deleted: 3 Num files renamed: 4 Num Flush(): 10 Num Sync(): 5 Num Fsync(): 1 Num Close(): 2 Num Read(): 6 Num Append(): 8 Num bytes read: 6216 Num bytes written: 6289 ``` Before: ``` > db_bench -benchmarks updaterandom -report_file_operations true ... Entries: 1000000 ... Num files opened: 18 Num Read(): 396339 Num Append(): 1000058 Num bytes read: 892030224 Num bytes written: 187569238 ``` After: ``` > db_bench -benchmarks updaterandom -report_file_operations true ... Entries: 1000000 ... Num files opened: 18 Num files deleted: 5 Num files renamed: 4 Num Flush(): 1000068 Num Sync(): 9 Num Fsync(): 1 Num Close(): 6 Num Read(): 396339 Num Append(): 1000058 Num bytes read: 892030224 Num bytes written: 187569238 ``` Another example showing how using ```DB::OpenForReadOnly``` reduces file operations compared to ```((Optimistic)Transaction)DB::Open```: ``` > db_bench -benchmarks updaterandom -num 1 > db_bench -benchmarks updaterandom -num 0 -use_existing_db true -readonly true -report_file_operations true ... Entries: 0 ... Num files opened: 8 Num files deleted: 0 Num files renamed: 0 Num Flush(): 0 Num Sync(): 0 Num Fsync(): 0 Num Close(): 0 Num Read(): 13 Num Append(): 0 Num bytes read: 374 Num bytes written: 0 ``` ``` > db_bench -benchmarks updaterandom -num 1 > db_bench -benchmarks updaterandom -num 0 -use_existing_db true -report_file_operations true ... Entries: 0 ... Num files opened: 14 Num files deleted: 3 Num files renamed: 4 Num Flush(): 14 Num Sync(): 5 Num Fsync(): 1 Num Close(): 3 Num Read(): 11 Num Append(): 10 Num bytes read: 7291 Num bytes written: 7357 ``` Pull Request resolved: https://github.com/facebook/rocksdb/pull/8448 Reviewed By: anand1976 Differential Revision: D29333818 Pulled By: zhichao-cao fbshipit-source-id: a06a8c87f799806462319115195b3e94faf5f542
-
- 24 6月, 2021 2 次提交
-
-
由 Levi Tamasi 提交于
Summary: `VersionSet::VerifyCompactionFileConsistency` was superseded by the LSM tree consistency checks introduced in https://github.com/facebook/rocksdb/pull/6901, which are more comprehensive, more efficient, and are performed unconditionally even in release builds. Pull Request resolved: https://github.com/facebook/rocksdb/pull/8449 Test Plan: `make check` Reviewed By: ajkr Differential Revision: D29337441 Pulled By: ltamasi fbshipit-source-id: a05324f88e3400e27e6a00406c878a6276e0c9cc
-
由 Levi Tamasi 提交于
Summary: Follow-up to https://github.com/facebook/rocksdb/issues/8426 . The patch adds a new kind of `InternalIterator` that wraps another one and passes each key-value encountered to `BlobGarbageMeter` as inflow. This iterator will be used as an input iterator for compactions when the input SSTs reference blob files. Pull Request resolved: https://github.com/facebook/rocksdb/pull/8443 Test Plan: `make check` Reviewed By: jay-zhuang Differential Revision: D29311987 Pulled By: ltamasi fbshipit-source-id: b4493b4c0c0c2e3c2ecc33c8969a5ef02de5d9d8
-
- 23 6月, 2021 8 次提交
-
-
由 Baptiste Lemaire 提交于
Summary: Add an argument to ldb to dump live file names, column families, and levels, `list_live_files_metadata`. The output shows all active SST file names, sorted first by column family and then by level. For each level the SST files are sorted alphabetically. Typically, the output looks like this: ``` ./ldb --db=/tmp/test_db list_live_files_metadata Live SST Files: ===== Column Family: default ===== ---------- level 0 ---------- /tmp/test_db/000069.sst ---------- level 1 ---------- /tmp/test_db/000064.sst /tmp/test_db/000065.sst /tmp/test_db/000066.sst /tmp/test_db/000071.sst ---------- level 2 ---------- /tmp/test_db/000038.sst /tmp/test_db/000039.sst /tmp/test_db/000052.sst /tmp/test_db/000067.sst /tmp/test_db/000070.sst ------------------------------ ``` Second, a flag was added `--sort_by_filename`, to change the layout of the output. When this flag is added to the command, the output shows all active SST files sorted by name, in front of which the LSM level and the column family are mentioned. With the same example, the following command would return: ``` ./ldb --db=/tmp/test_db list_live_files_metadata --sort_by_filename Live SST Files: /tmp/test_db/000038.sst : level 2, column family 'default' /tmp/test_db/000039.sst : level 2, column family 'default' /tmp/test_db/000052.sst : level 2, column family 'default' /tmp/test_db/000064.sst : level 1, column family 'default' /tmp/test_db/000065.sst : level 1, column family 'default' /tmp/test_db/000066.sst : level 1, column family 'default' /tmp/test_db/000067.sst : level 2, column family 'default' /tmp/test_db/000069.sst : level 0, column family 'default' /tmp/test_db/000070.sst : level 2, column family 'default' /tmp/test_db/000071.sst : level 1, column family 'default' ------------------------------ ``` Thus, the user can either request to show the files by levels, or sorted by filenames. This PR includes a simple Python unit test that makes sure the file name and level printed out by this new feature matches the one found with an existing feature, `dump_live_file`. Pull Request resolved: https://github.com/facebook/rocksdb/pull/8446 Reviewed By: akankshamahajan15 Differential Revision: D29320080 Pulled By: bjlemaire fbshipit-source-id: 01fb7b5637c59010d74c80730a28d815994e7009
-
由 Levi Tamasi 提交于
Summary: This test case has been failing occasionally due to automatic compactions kicking in, resulting in GC generating additional blob files that the test did not expect. Disabling automatic compactions to get rid of this flakiness. Pull Request resolved: https://github.com/facebook/rocksdb/pull/8444 Test Plan: `gtest-parallel --repeat=1000 ./blob_db_test --gtest_filter="BlobDBTest.SnapshotAndGarbageCollection"` Reviewed By: jay-zhuang Differential Revision: D29316987 Pulled By: ltamasi fbshipit-source-id: 9815d189ed7d63890622768675a01990e3680221
-
由 Levi Tamasi 提交于
Summary: Pull Request resolved: https://github.com/facebook/rocksdb/pull/8442 Reviewed By: akankshamahajan15 Differential Revision: D29306432 Pulled By: ltamasi fbshipit-source-id: 0fe55736368ac024b2f51213b65f4b06d71d60e6
-
由 Baptiste Lemaire 提交于
Summary: At the moment, the following command : "`./ --db=mypath/ dump_file_files`" returns a series of erronous names with double slashes, ie: "`mypath//000xxx.sst`", including manifest file names with double slashes "`mypath//MANIFEST-00XXX`", whereas "`./ --db=mypath dump_file_files`" correctly returns "`mypath/000xxx.sst`" and "`mypath/MANIFEST-00XXX`". This (very short) PR simply checks if there is a need to add or remove any '`/`' character when the `db_path` and `manifest_filename`/sst `filenames` are concatenated. Pull Request resolved: https://github.com/facebook/rocksdb/pull/8439 Reviewed By: akankshamahajan15 Differential Revision: D29301349 Pulled By: bjlemaire fbshipit-source-id: 3e9e58f9749d278b654ae838fcee13ad698705a8
-
由 Jay Zhuang 提交于
Revert "Revert "Snapshot release triggered compaction without multiple tombstones (#8357)" (#8410)" (#8438) Summary: This reverts commit 25be1ed6. Pull Request resolved: https://github.com/facebook/rocksdb/pull/8438 Test Plan: Run the impacted mysql test 40 times Reviewed By: ajkr Differential Revision: D29286247 Pulled By: jay-zhuang fbshipit-source-id: d3bd056971a19a8b012d5d0295fa045c012b3c04
-
由 Andrew Kryczka 提交于
Summary: `git diff origin/6.21.fb origin/6.22.fb -- HISTORY.md` looked odd. This PR fixes it up by moving items from 6.21.0 to 6.22.0 that were never in any 6.21 release. Also mentioned the background stat collection fix under 6.22 (previously it was mentioned under 6.21 patch releases only). Pull Request resolved: https://github.com/facebook/rocksdb/pull/8441 Reviewed By: jay-zhuang Differential Revision: D29304812 Pulled By: ajkr fbshipit-source-id: 2a928a9518a1d6615321d5c2d1e22b17cbb59093
-
由 Levi Tamasi 提交于
Summary: Currently, blob file checksums are incorrectly dumped as raw bytes in the `ldb manifest_dump` output (i.e. they are not printed as hex). The patch fixes this and also updates some test cases to reflect that the checksum value field in `BlobFileAddition` and `SharedBlobFileMetaData` contains the raw checksum and not a hex string. Pull Request resolved: https://github.com/facebook/rocksdb/pull/8437 Test Plan: `make check` Tested using `ldb manifest_dump` Reviewed By: akankshamahajan15 Differential Revision: D29284170 Pulled By: ltamasi fbshipit-source-id: d11cfb3435b14cd73c8a3d3eb14fa0f9fa1d2228
-
由 Jay Zhuang 提交于
Summary: `DeleteFilesInRange()` marks deleting files to `being_compacted` before deleting, which may cause ongoing compactions report corruption exception or ASSERT for debug build. Adding the missing `ComputeCompactionScore()` when `being_compacted` is set. Pull Request resolved: https://github.com/facebook/rocksdb/pull/8434 Test Plan: Unittest Reviewed By: ajkr Differential Revision: D29276127 Pulled By: jay-zhuang fbshipit-source-id: f5b223e3c1fc6d821e100e3f3442bc70c1d50cf7
-
- 22 6月, 2021 4 次提交
-
-
由 Levi Tamasi 提交于
Summary: This is part of an alternative approach to https://github.com/facebook/rocksdb/issues/8316. Unlike that approach, this one relies on key-values getting processed one by one during compaction, and does not involve persistence. Specifically, the patch adds a class `BlobGarbageMeter` that can track the number and total size of blobs in a (sub)compaction's input and output on a per-blob file basis. This information can then be used to compute the amount of additional garbage generated by the compaction for any given blob file by subtracting the "outflow" from the "inflow." Note: this patch only adds `BlobGarbageMeter` and associated unit tests. I plan to hook up this class to the input and output of `CompactionIterator` in a subsequent PR. Pull Request resolved: https://github.com/facebook/rocksdb/pull/8426 Test Plan: `make check` Reviewed By: jay-zhuang Differential Revision: D29242250 Pulled By: ltamasi fbshipit-source-id: 597e50ad556540e413a50e804ba15bc044d809bb
-
由 anand76 提交于
Summary: Tsan complains due to a perceived race condition in accessing LRUHandle flags. One thread calls ```LRUHandle::SetHit()``` from ```LRUCacheShard::Lookup()```, while another thread calls ```LRUHandle::IsPending()``` from ```LRUCacheShard::IsReady()```. The latter call is from ```MultiGet```. It doesn't actually have to call ```IsReady``` since a null value indicates the cache handle is not ready, so its sufficient to check for a null value. Also modify ```IsReady``` to acquire the LRU shard mutex. Tests: 1. make check 2. Run tsan_crash Pull Request resolved: https://github.com/facebook/rocksdb/pull/8433 Reviewed By: zhichao-cao Differential Revision: D29278030 Pulled By: anand1976 fbshipit-source-id: 0c9fed56d12eda853e72dadebe75038361bd257f
-
由 lorinlee 提交于
Summary: Signed-off-by: Nlorinlee <lorinlee1996@gmail.com> Pull Request resolved: https://github.com/facebook/rocksdb/pull/8429 Reviewed By: zhichao-cao Differential Revision: D29256538 Pulled By: ajkr fbshipit-source-id: fa483861148d82119f4c8775d0cf5f6c71b06215
-
由 Andrew Kryczka 提交于
Summary: - `c_test` fails because `rocksdb_compact_range()` swallows a `Status`. - `env_test` fails because `ReadRequest`s to `MultiRead()` do not have their `Status`es checked. Pull Request resolved: https://github.com/facebook/rocksdb/pull/8430 Test Plan: `ASSERT_STATUS_CHECKED=1 make -j48 check` Reviewed By: jay-zhuang Differential Revision: D29257473 Pulled By: ajkr fbshipit-source-id: e02127f971703744be7de85f0a028e4664c79577
-
- 20 6月, 2021 1 次提交
-
-
由 Andrew Kryczka 提交于
Summary: Pull Request resolved: https://github.com/facebook/rocksdb/pull/8427 Reviewed By: zhichao-cao Differential Revision: D29246916 Pulled By: ajkr fbshipit-source-id: ccd44ca1a6dd5101dc37f19b8e1fe6c0e3883e0a
-
- 19 6月, 2021 4 次提交
-
-
由 mwish 提交于
Summary: Pull Request resolved: https://github.com/facebook/rocksdb/pull/8423 Reviewed By: mrambacher Differential Revision: D29232587 Pulled By: jay-zhuang fbshipit-source-id: 04d4937cf0605cbf341a920d1305369a7b8f0574
-
由 Zhichao Cao 提交于
Summary: Tracing the MultiGet information including timestamp, keys, and CF_IDs to the trace file for analyzing and replay. Pull Request resolved: https://github.com/facebook/rocksdb/pull/8421 Test Plan: make check, add test to trace_analyzer_test Reviewed By: anand1976 Differential Revision: D29221195 Pulled By: zhichao-cao fbshipit-source-id: 30c677d6c39ab31ef4bbdf7e0d1fa1fd79f295ff
-
由 Akanksha Mahajan 提交于
Summary: Since windows timeout issue has been fixed. Change the image back to stable. Pull Request resolved: https://github.com/facebook/rocksdb/pull/8424 Test Plan: Check CircleCI jobs Reviewed By: zhichao-cao Differential Revision: D29235219 Pulled By: akankshamahajan15 fbshipit-source-id: 2c111f96e216dac4ae3d7ec3b4cdd8e459575d37
-
由 anand76 提交于
Summary: Implement the ```WaitAll()``` interface in ```LRUCache``` to allow callers to issue multiple lookups in parallel and wait for all of them to complete. Modify ```MultiGet``` to use this to parallelize the secondary cache lookups in order to reduce the overall latency. A call to ```cache->Lookup()``` returns a handle that has an incomplete value (nullptr), and the caller can call ```cache->IsReady()``` to check whether the lookup is complete, and pass a vector of handles to ```WaitAll``` to wait for completion. If any of the lookups fail, ```MultiGet``` will read the block from the SST file. Another change in this PR is to rename ```SecondaryCacheHandle``` to ```SecondaryCacheResultHandle``` as it more accurately describes the return result of the secondary cache lookup, which is more like a future. Tests: 1. Add unit tests in lru_cache_test 2. Benchmark results with no secondary cache configured Master - ``` readrandom : 41.175 micros/op 388562 ops/sec; 106.7 MB/s (7277999 of 7277999 found) readrandom : 41.217 micros/op 388160 ops/sec; 106.6 MB/s (7274999 of 7274999 found) multireadrandom : 10.309 micros/op 1552082 ops/sec; (28908992 of 28908992 found) multireadrandom : 10.321 micros/op 1550218 ops/sec; (29081984 of 29081984 found) ``` This PR - ``` readrandom : 41.158 micros/op 388723 ops/sec; 106.8 MB/s (7290999 of 7290999 found) readrandom : 41.185 micros/op 388463 ops/sec; 106.7 MB/s (7287999 of 7287999 found) multireadrandom : 10.277 micros/op 1556801 ops/sec; (29346944 of 29346944 found) multireadrandom : 10.253 micros/op 1560539 ops/sec; (29274944 of 29274944 found) ``` Pull Request resolved: https://github.com/facebook/rocksdb/pull/8405 Reviewed By: zhichao-cao Differential Revision: D29190509 Pulled By: anand1976 fbshipit-source-id: 6f8eff6246712af8a297cfe22ea0d1c3b2a01bb0
-
- 18 6月, 2021 6 次提交
-
-
由 Baptiste Lemaire 提交于
Summary: **Summary**: 2 new statistics counters are added to RocksDB: `MEMTABLE_PAYLOAD_BYTES_AT_FLUSH` and `MEMTABLE_GARBAGE_BYTES_AT_FLUSH`. The former tracks how many raw bytes of useful data are present on the memtable at flush time, whereas the latter is tracks how many of these raw bytes are considered garbage, meaning that they ended up not being imported on the SSTables resulting from the flush operations. **Unit test**: run `make db_flush_test -j$(nproc); ./db_flush_test` to run the unit test. This executable includes 3 tests, that test support and correct stat calculations for workloads with inserts, deletes, and DeleteRanges. The parameters are set such that the workloads are performed on a single memtable, and a single SSTable is created as a result of the flush operation. The flush operation is manually called in the test file. The tests verify that the values of these 2 statistics counters introduced in this PR can be exactly predicted, showing that we have a full understanding of the underlying operations. **Performance testing**: `./db_bench -statistics -benchmarks=fillrandom -num=10000000` repeated 10 times. Timing done using "date" function in a bash script. _Results_: Original Rocksdb fork: mean 66.6 sec, std 1.18 sec. This feature branch: mean 67.4 sec, std 1.35 sec. Pull Request resolved: https://github.com/facebook/rocksdb/pull/8411 Reviewed By: akankshamahajan15 Differential Revision: D29150629 Pulled By: bjlemaire fbshipit-source-id: 7b3c2e86d50c6aa34fa50fd134282eacb543a5b1
-
由 Akanksha Mahajan 提交于
Summary: This PR prepopulates warm/hot data blocks which are already in memory into block cache at the time of flush. On a flush, the data block that is in memory (in memtables) get flushed to the device. If using Direct IO, additional IO is incurred to read this data back into memory again, which is avoided by enabling newly added option. Right now, this is enabled only for flush for data blocks. We plan to expand this option to cover compactions in the future and for other types of blocks. Pull Request resolved: https://github.com/facebook/rocksdb/pull/8242 Test Plan: Add new unit test Reviewed By: anand1976 Differential Revision: D28521703 Pulled By: akankshamahajan15 fbshipit-source-id: 7219d6958821cedce689a219c3963a6f1a9d5f05
-
由 聂佩轩 提交于
Summary: This commit is for enabling `DBWithTTL` to use `DeteleRange` which it cannot before. As (int32_t)Timestamp is suffixed to values in `DBWithTTL`, there is no reason that it cannot use the common used api. I added `DeleteRangeCF` in `DBWithTTLImpl::Write` so that we can use `DeteleRange` normally. When we run code like `dbWithTtl->DeleteRange(start, end)`, it executes`WriteBatchInternal::DeleteRange` internally. Intended to fix https://github.com/facebook/rocksdb/issues/7218 Pull Request resolved: https://github.com/facebook/rocksdb/pull/8384 Test Plan: added corresponded testing logic to existing unit test Reviewed By: jay-zhuang Differential Revision: D29176734 fbshipit-source-id: 6874ed979fc08e1d138149d03653e43a75f0e0e6
-
由 Peter Dillinger 提交于
Summary: Marked the Ribbon filter and optimize_filters_for_memory features as production-ready, each enabling memory savings for Bloom-like filters. Use `NewRibbonFilterPolicy` in place of `NewBloomFilterPolicy` to use Ribbon filters instead of Bloom, or `ribbonfilter` in place of `bloomfilter` in configuration string. Some small refactoring in db_stress. Removed/refactored unused code in db_bench, in part preparing for future default possibly being different from "disabled." Pull Request resolved: https://github.com/facebook/rocksdb/pull/8408 Test Plan: Lots of prior automated, ad-hoc, and "real world" testing. Updated tests for new API names. Quick db_bench test: bloom fillrandom 77730 ops/sec rocksdb.block.cache.filter.bytes.insert COUNT : 89929384 ribbon fillrandom 71492 ops/sec rocksdb.block.cache.filter.bytes.insert COUNT : 64531384 Reviewed By: mrambacher Differential Revision: D29140805 Pulled By: pdillinger fbshipit-source-id: d742c922722421678f95ad85eeb0aaebc9f5e49a
-
由 anand76 提交于
Summary: RocksDB logs a warning if WAL truncation on DB open fails. Its possible that on some file systems, truncation is not required and they would return ```Status::NotSupported()``` for ```ReopenWritableFile```. Don't log a warning in such cases. Pull Request resolved: https://github.com/facebook/rocksdb/pull/8414 Reviewed By: akankshamahajan15 Differential Revision: D29181738 Pulled By: anand1976 fbshipit-source-id: 6e01e9117e1e4c1d67daa4dcee7fa59d06e057a7
-
由 Sidi Mohamed EL AATIFI 提交于
Summary: iterateLowerBound Slice representing the lower bound Pull Request resolved: https://github.com/facebook/rocksdb/pull/8394 Reviewed By: ajkr Differential Revision: D29085721 Pulled By: jay-zhuang fbshipit-source-id: a154375879395c48e9bd3794d296e70316894056
-
- 17 6月, 2021 1 次提交
-
-
由 mrambacher 提交于
Summary: Inner method added for classes to override to return has-a relationship. CheckedCast expanded to use Inner to return has-a relationship. Future Customizable classes (Env, FileSystem, Statistics, etc) will use this feature Pull Request resolved: https://github.com/facebook/rocksdb/pull/8373 Reviewed By: pdillinger Differential Revision: D29176369 Pulled By: mrambacher fbshipit-source-id: cfb6d7702fa365ca4e40c4a50a19e3a534e5ac43
-