- 15 11月, 2018 4 次提交
-
-
由 Abhishek Madan 提交于
Summary: Rather than storing a `vector<RangeTombstone>`, we now store a `vector<RangeTombstoneStack>` and a `vector<SequenceNumber>`. A `RangeTombstoneStack` contains the start and end keys of a range tombstone fragment, and indices into the seqnum vector to indicate which sequence numbers the fragment is located at. The diagram below illustrates an example: ``` tombstones_: [a, b) [c, e) [h, k) | \ / \ / | | \ / \ / | v v v v tombstone_seqs_: [ 5 3 10 7 2 8 6 ] ``` This format allows binary searching the tombstone list to use less key comparisons, which helps in cases where there are many overlapping tombstones. Also, this format makes it easier to add DBIter-like semantics to `FragmentedRangeTombstoneIterator` in the future. Pull Request resolved: https://github.com/facebook/rocksdb/pull/4632 Differential Revision: D13053103 Pulled By: abhimadan fbshipit-source-id: e8220cc712fcf5be4d602913bb23ace8ea5f8ef0
-
由 Siying Dong 提交于
Summary: DBTest.SanitizeNumThreads Sometimes fails. The test waited for 10ms timeout and expect all threads scheduled to be executed. This can be a source of flakiness. Make a check every 1ms and up to 10s. Pull Request resolved: https://github.com/facebook/rocksdb/pull/4659 Differential Revision: D13074174 Pulled By: siying fbshipit-source-id: b1d5ff87a326a4fc9eab8d1cc307bbb940dfe70c
-
由 Maysam Yabandeh 提交于
Summary: The default implementation of GetForUpdate that receives PinnableSlice was mistakenly dropping column_family and exclusive parameters. Pull Request resolved: https://github.com/facebook/rocksdb/pull/4679 Differential Revision: D13062531 Pulled By: maysamyabandeh fbshipit-source-id: 7625d0c1ba872a5d894b58ced42147d6c8556a6f
-
由 Abhishek Madan 提交于
Summary: The new flag makes it possible to constrain iterator traversal by the upper/lower bound the iterator is expected to pass. This allows seekrandom results to be more easily comparable between DBs with and without deletions. Pull Request resolved: https://github.com/facebook/rocksdb/pull/4660 Differential Revision: D13053111 Pulled By: abhimadan fbshipit-source-id: 33e250f2e2d210b54c7726399da30a33f723c33c
-
- 14 11月, 2018 14 次提交
-
-
由 Yanqin Jin 提交于
Summary: When iterator becomes invalid, there are two possibilities. First, all data in the column family have been scanned and there is nothing more to scan. Second, an underlying error has occurred, causing `status()` to be !ok. Therefore, we need to check for both cases when `!iter->Valid()`. Pull Request resolved: https://github.com/facebook/rocksdb/pull/4648 Differential Revision: D12959601 Pulled By: riversand963 fbshipit-source-id: 49c9382c9ea9e78f2e2b6f3708f0670b822ca8dd
-
由 thisisbaozi 提交于
Summary: adamretter As you already mentioned at #1247 . Pull Request resolved: https://github.com/facebook/rocksdb/pull/2303 Differential Revision: D10209001 Pulled By: sagar0 fbshipit-source-id: bcbce004112c2edeaff116968d79c6f90aab4b6c
-
由 Zhongyi Xie 提交于
Summary: `GenSubcompactionBoundaries` calls `VersionSet::ApproximateSize` which gets BlockBasedTableReader for every file and seeks in its index block to find `key`'s offset. If the table or index block aren't in memory already, this involves I/O. This can be improved by releasing DB mutex when calling ApproximateSize. Pull Request resolved: https://github.com/facebook/rocksdb/pull/4630 Differential Revision: D13052653 Pulled By: miasantreble fbshipit-source-id: cae31d46d10d0860fa8a26b8d5154b2d17d1685f
-
由 Siying Dong 提交于
Remove two variables from BlockContents class and don't use class Block for compressed block (#4650) Summary: We carry compression type and "cachable" variables for every block in the block cache, while they take well-known values. 8-byte is wasted for each block (2-byte for useful information but it takes 8 bytes because of padding). With this change, these two variables are removed. The cachable information is only useful in the process of reading the block. We use other information to infer from it. For compressed blocks, the compression type is a part of the block content itself so we can get it from there. Some code is slightly refactored so that the cachable information can flow better. Another change is to only use class BlockContents for compressed block, and narrow the class Block to only be used for uncompressed blocks, including blocks in compressed block cache. This can make the Block class less confusing. It also saves tens of bytes for each block in compressed block cache. Pull Request resolved: https://github.com/facebook/rocksdb/pull/4650 Differential Revision: D12969070 Pulled By: siying fbshipit-source-id: 548b62724e9eb66993026429fd9c7c3acd1f95ed
-
由 Zhongyi Xie 提交于
Summary: Currently transaction iterator does not apply `ReadOptions.iterate_upper_bound` when iterating. This PR attempts to fix the problem by having `BaseDeltaIterator` enforcing the upper bound check when iterator state is changed. Pull Request resolved: https://github.com/facebook/rocksdb/pull/4656 Differential Revision: D13039257 Pulled By: miasantreble fbshipit-source-id: 909eb9f6b4597a4d80418fb139f32ec82c6ec1d1
-
由 Simon Liu 提交于
Summary: It called the autovector::push_back simply in autovector::emplace_back. This was not efficient, and then optimazed this function through the perfect forwarding. This was the src and result of the benchmark(using the google'benchmark library, the type of elem in autovector was std::string, and call emplace_back with the "char *" type): https://gist.github.com/monadbobo/93448b89a42737b08cbada81de75c5cd PS: The benchmark's result of previous PR was not accurate, and so I update the test case and result. Pull Request resolved: https://github.com/facebook/rocksdb/pull/4606 Differential Revision: D13046813 Pulled By: sagar0 fbshipit-source-id: 19cde1bcadafe899aa454b703acb35737a1cc02d
-
由 Yi Wu 提交于
Summary: Per offline discussion with siying, `MemoryAllocator` and `Cache` should be decouple. The idea is that memory allocator handles memory allocation, while cache handle cache policy. It is normal that external cache libraries pack couple the two components for better optimization. If we want to integrate with such library in the future, we can make a wrapper of the library implementing both `Cache` and `MemoryAllocator` interface. Pull Request resolved: https://github.com/facebook/rocksdb/pull/4676 Differential Revision: D13047662 Pulled By: yiwu-arbug fbshipit-source-id: cd42e246d80ab600b4de47d073f7d2db308ce6dd
-
由 Siying Dong 提交于
Summary: We used to have a bug, which caused every block to be read twice, and none of our tests caught it. Add a very simply unit test to make sure that when reading a data block, we only issue one pread against the SST file. Pull Request resolved: https://github.com/facebook/rocksdb/pull/4657 Differential Revision: D13005260 Pulled By: siying fbshipit-source-id: 03167b554ad2451192b1707415536d7d05e9026c
-
由 Yi Wu 提交于
Summary: As pointed out in #4059, we miss use string as buffer for file read. Changing to use char array instead. Closing #4059 Pull Request resolved: https://github.com/facebook/rocksdb/pull/4662 Differential Revision: D13012998 Pulled By: yiwu-arbug fbshipit-source-id: 41234ba17c0bccea65bd647e362a0e979152bd1e
-
由 QingpingWang 提交于
Summary: he ratio of num_deletions to num_entries of a level can be useful to determine if a manual compaction needs to be triggered on a level. Also refer #3980 Pull Request resolved: https://github.com/facebook/rocksdb/pull/4623 Differential Revision: D13045744 Pulled By: sagar0 fbshipit-source-id: 71f3c8e363a8ffd194ec3bb0ed0b69612231f0b3
-
由 Soli Como 提交于
Summary: Currently, `Statistics` can record tick by `recordTick()` whose second parameter is an `uint64_t`. That means tick can only increase. If we want to reduce tick, we have to work around like `RecordTick(statistics_, NO_ITERATORS, uint64_t(-1));`. That's kind of a hack. So, this PR divide `NO_ITERATORS` into two counters `NO_ITERATOR_CREATED` and `NO_ITERATOR_DELETE`, making the counters increase only. Fixes #3013 . Pull Request resolved: https://github.com/facebook/rocksdb/pull/4498 Differential Revision: D10395010 Pulled By: sagar0 fbshipit-source-id: cfb523b22a37411c794b4e9da090f1ae30293db2
-
由 Soli 提交于
Summary: Call `SyncClosedLogs()` only if there are more than one column families. Update several unit tests (in `fault_injection_test` and `db_flush_test`) correspondingly. See #3840 for more info. Pull Request resolved: https://github.com/facebook/rocksdb/pull/4460 Differential Revision: D12896377 Pulled By: riversand963 fbshipit-source-id: f49afdaec32568f12f001219a3aec1dfde3b32bf
-
由 Andrew Kryczka 提交于
Summary: Use the `DBOptions` that the backup engine already holds to figure out the right `EnvOptions` to use when reading the DB files. This means that, if a user opened a DB instance with `use_direct_reads=true`, then using `BackupEngine` to back up that DB instance will use direct I/O to read files when calculating checksums and copying. Currently the WALs and manifests would still be read using buffered I/O to prevent mixing direct I/O reads with concurrent buffered I/O writes. Pull Request resolved: https://github.com/facebook/rocksdb/pull/4640 Differential Revision: D13015268 Pulled By: ajkr fbshipit-source-id: 77006ad6f3e00ce58374ca4793b785eea0db6269
-
由 Zhongyi Xie 提交于
Summary: this PR adds two more per-level perf context counters to track * number of keys returned in Get call, break down by levels * total processing time at each level during Get call Pull Request resolved: https://github.com/facebook/rocksdb/pull/4617 Differential Revision: D12898024 Pulled By: miasantreble fbshipit-source-id: 6b84ef1c8097c0d9e97bee1a774958f56ab4a6c4
-
- 13 11月, 2018 8 次提交
-
-
由 Sagar Vemuri 提交于
Summary: Our internal CI test caught RocksDB Lite build failures. The failures are due to a new test introduced in #4665 using `SSTFileWriter` and `IngestExternalFile`, but these is not exposed under lite mode. Fixed by #ifdef'ing out the test. ``` db/db_test2.cc: In member function ‘virtual void rocksdb::DBTest2_TestCompactFiles_Test::TestBody()’: db/db_test2.cc:2907:3: error: ‘SstFileWriter’ is not a member of ‘rocksdb’ rocksdb::SstFileWriter sst_file_writer{rocksdb::EnvOptions(), options}; ^ In file included from ./util/testharness.h:15:0, from ./table/mock_table.h:23, from ./db/db_test_util.h:44, from db/db_test2.cc:13: db/db_test2.cc:2912:13: error: ‘sst_file_writer’ was not declared in this scope ASSERT_OK(sst_file_writer.Open(external_file1)); ``` Pull Request resolved: https://github.com/facebook/rocksdb/pull/4675 Differential Revision: D13035984 Pulled By: sagar0 fbshipit-source-id: c1ceac550dfac1a85eeea436693dc7dd467519a6
-
由 Sagar Vemuri 提交于
Summary: In #4652 we are setting -Os for lite builds only when LITE=1 is specified. But currently almost all the users invoke lite build via OPT="-DROCKSDB_LITE=1". So this diff tries to set LITE=1 when users already pass in -DROCKSDB_LITE=1 via the command line. Pull Request resolved: https://github.com/facebook/rocksdb/pull/4671 Differential Revision: D13033801 Pulled By: sagar0 fbshipit-source-id: e7b506cee574f9e3f42221ee6647915011c78d78
-
由 Abhishek Madan 提交于
Summary: Part of the test required that a compaction start before a manual flush, but this was not enforced by the test. In some cases, particularly when writing to tmpfs, this could lead to the compaction starting after the flush, which caused the base level to be higher than it was expected to be. Add a sync point in the test to ensure that the flush and compaction happen simultaneously. The test also had some stale comments, so those have been removed or modified, and the test has been simplified so that it no longer uses sleeps and writes uncompressed SSTs. Pull Request resolved: https://github.com/facebook/rocksdb/pull/4668 Differential Revision: D13032440 Pulled By: abhimadan fbshipit-source-id: 3f23b583a096454dafb8d8ea75678605dec80209
-
由 Fosco Marotto 提交于
Summary: Pull Request resolved: https://github.com/facebook/rocksdb/pull/4669 Differential Revision: D13031522 Pulled By: gfosco fbshipit-source-id: d09e655a9d5556594f195c5d1b786900932145ce
-
由 DorianZheng 提交于
Summary: `CompactFiles` gets `SuperVersion` before `WaitForIngestFile`, while `IngestExternalFile` may add files that overlap with `input_file_names` The timeline of execution flow is as follow: Let's say that level N has two file [1,2] and [5,6] ``` timeline user_thread1 user_thread2 t0 | CompactFiles([1, 2], [5, 6]) begin t1 | GetReferencedSuperVersion() t2 | IngestExternalFile([3,4]) to level N begin t3 | CompactFiles resume V ``` Pull Request resolved: https://github.com/facebook/rocksdb/pull/4665 Differential Revision: D13030674 Pulled By: ajkr fbshipit-source-id: 8be19477fd6e505032267a979d32f3097cc3be51
-
由 Yanqin Jin 提交于
Summary: In the past, both `DBImpl::atomic_flush_` and `DBImpl::immutable_db_options_.atomic_flush` exist. However, we fail to set `immutable_db_options_.atomic_flush`, but use `DBImpl::atomic_flush_` which is set correctly. This does not lead to incorrect behavior, but is a duplicate of information. Since `immutable_db_options_` is always there and has `atomic_flush`, we should use it as source of truth and remove `DBImpl::atomic_flush_`. Pull Request resolved: https://github.com/facebook/rocksdb/pull/4631 Differential Revision: D12928371 Pulled By: riversand963 fbshipit-source-id: f85a811959d3828aad4a3a1b05f71facf19c636d
-
由 DorianZheng 提交于
Summary: Hi, yiwu-arbug, I found that `DBImpl::GetColumnFamilyHandleUnlocked` still have data race condition, because `column_family_memtables_` has a stateful cache `current_` and `column_family_memtables_::Seek` maybe call without the protection of `mutex_` by a write thread check https://github.com/facebook/rocksdb/blob/859dbda6e3cac17416aff48f1760d01707867351/db/write_batch.cc#L1188 and https://github.com/facebook/rocksdb/blob/859dbda6e3cac17416aff48f1760d01707867351/db/write_batch.cc#L1756 and https://github.com/facebook/rocksdb/blob/859dbda6e3cac17416aff48f1760d01707867351/db/db_impl_write.cc#L318 So it's better to use `versions_->GetColumnFamilySet()->GetColumnFamily` instead. Pull Request resolved: https://github.com/facebook/rocksdb/pull/4666 Differential Revision: D13027117 Pulled By: yiwu-arbug fbshipit-source-id: 4e3778eaf8e7f7c8577bbd78129b6a5fd7ce79fb
-
由 Zhichao Cao 提交于
Summary: Changes: 1. in current version, key size distribution is printed out as the result. In this change, the result will be output to a file to make further analyze easier 2. To understand how the unique keys are accessed over time, the total unique key number of each CF of each query type in each second over time is output to a file. In this way, user could know when the unique keys are accessed frequently or accessed rarely. 3. output the total QPS of each CF to a file 4. Add the print result of total queries of each CF of each query type. Pull Request resolved: https://github.com/facebook/rocksdb/pull/4646 Differential Revision: D12968156 Pulled By: zhichao-cao fbshipit-source-id: 6c411c7ec47c7843a70929136efd71a150db0e4c
-
- 10 11月, 2018 4 次提交
-
-
由 Yi Wu 提交于
Summary: The flakyness can be reproduced with the following patch: ``` --- a/db/db_impl_compaction_flush.cc +++ b/db/db_impl_compaction_flush.cc @@ -2013,6 +2013,9 @@ void DBImpl::BackgroundCallFlush() { if (job_context.HaveSomethingToDelete()) { PurgeObsoleteFiles(job_context); } + static int f_count = 0; + printf("clean flush job context %d\n", ++f_count); + env_->SleepForMicroseconds(1000000); job_context.Clean(); mutex_.Lock(); } ``` The issue is that FlushMemtable with opt.wait=true does not wait for `OnStallConditionsChanged` being called. The event listener is triggered on `JobContext::Clean`, which happens after flush result is installed. At the time we check for stall condition after flushing memtable, the job context cleanup may not be finished. To fix the flaykyness, we use sync point to create a custom WaitForFlush that waits for context cleanup. Pull Request resolved: https://github.com/facebook/rocksdb/pull/4658 Differential Revision: D13007301 Pulled By: yiwu-arbug fbshipit-source-id: d98395ee7b0ad4c62e83e8d0e9b6028058c61712
-
由 Yi Wu 提交于
Summary: When running `make shared_lib` under fbcode, there's liblua link error: https://gist.github.com/yiwu-arbug/b796bff6b3d46d90c1ed878d983de50d This is because we link liblua.a when building shared lib. If we want to link with liblua, we need to link with liblua_pic.a instead. Fixing by simply not link with lua. Pull Request resolved: https://github.com/facebook/rocksdb/pull/4651 Differential Revision: D12964798 Pulled By: yiwu-arbug fbshipit-source-id: 18d6cee94afe20748068822b76e29ef255cdb04d
-
由 Sagar Vemuri 提交于
Summary: Updated the comments around all options which are currently dynamic. Without explicitly specifying in the comments around the options, its hard for RocksDB users to know if an option is dynamically changeable or not unless they dig into the implementation details. Pull Request resolved: https://github.com/facebook/rocksdb/pull/4653 Differential Revision: D12966735 Pulled By: sagar0 fbshipit-source-id: 1a01acbf6fe506b989e71629ce223f9803ebae27
-
由 Sagar Vemuri 提交于
Summary: Ran the following commands to recursively change all the files under RocksDB: ``` find . -type f -name "*.cc" -exec sed -i 's/ unique_ptr/ std::unique_ptr/g' {} + find . -type f -name "*.cc" -exec sed -i 's/<unique_ptr/<std::unique_ptr/g' {} + find . -type f -name "*.cc" -exec sed -i 's/ shared_ptr/ std::shared_ptr/g' {} + find . -type f -name "*.cc" -exec sed -i 's/<shared_ptr/<std::shared_ptr/g' {} + ``` Running `make format` updated some formatting on the files touched. Pull Request resolved: https://github.com/facebook/rocksdb/pull/4638 Differential Revision: D12934992 Pulled By: sagar0 fbshipit-source-id: 45a15d23c230cdd64c08f9c0243e5183934338a8
-
- 09 11月, 2018 1 次提交
-
-
由 Andrew Kryczka 提交于
Summary: We already exercised backup functionality in `db_stress` according to the `-backup_one_in` flag. This PR verifies the backup can be restored/opened and sanity checks a few keys. Changes in this PR: - Extracted existing backup-related logic to a helper function, `TestBackupRestore` - Added restore logic, which targets a hidden directory named "./.restore\<thread number\>", similar to how backups target hidden directories named "./.backup\<thread number\>". - After restore, check the existence/non-existence of a few keys. - With this PR, backup is no longer compatible with clearing column families. - Also included unrelated fixes to set `ReadOptions::total_order_seek=true` when using `-compare_full_db_state_snapshot` Pull Request resolved: https://github.com/facebook/rocksdb/pull/4655 Differential Revision: D12972496 Pulled By: ajkr fbshipit-source-id: 481a40052d9a38d1bd5c5159aa4d7c5a4b546b80
-
- 08 11月, 2018 3 次提交
-
-
由 Yi Wu 提交于
Summary: Set `-Os` for lite release build to minimize binary size. Pull Request resolved: https://github.com/facebook/rocksdb/pull/4652 Differential Revision: D12965427 Pulled By: yiwu-arbug fbshipit-source-id: c8b647642c24b3e5df6a2cd13112e452a08e8398
-
由 Zhongyi Xie 提交于
Summary: Pull Request resolved: https://github.com/facebook/rocksdb/pull/4637 Differential Revision: D12963727 Pulled By: miasantreble fbshipit-source-id: 76053501afbecc6ef388ddc56542fa0185243e3f
-
由 Peter (Stig) Edwards 提交于
Summary: Ensure delete[] and not delete is called on buffer_, as it is reset with new char[buffer_size_]. Pull Request resolved: https://github.com/facebook/rocksdb/pull/4647 Differential Revision: D12961327 Pulled By: ajkr fbshipit-source-id: c1af373b98359edfdc291caebe4e0acdfb8afdd8
-
- 07 11月, 2018 4 次提交
-
-
由 Andrew Gallagher 提交于
Summary: clang modules warns about `#include`s inside of namespaces. Pull Request resolved: https://github.com/facebook/rocksdb/pull/4629 Reviewed By: ajkr Differential Revision: D12927333 Pulled By: andrewjcg fbshipit-source-id: a9e0b069e63d8224f78b7c3be1c3acf09bb83d3f
-
由 Yanqin Jin 提交于
Summary: Include 5.16 and 5.17 in check_format_compatible.sh Pull Request resolved: https://github.com/facebook/rocksdb/pull/4634 Differential Revision: D12947140 Pulled By: riversand963 fbshipit-source-id: 79852b76d5139b2f31db59ed14cb368be01f2c32
-
由 Siying Dong 提交于
Summary: valgrind tests with 1 thread run too long. To make it shorter, black list some long tests. These are already blacklisted in parallel valgrind tests, but they are not in non-parallel mode Pull Request resolved: https://github.com/facebook/rocksdb/pull/4642 Differential Revision: D12945237 Pulled By: siying fbshipit-source-id: 04cf977d435996480fe87aa09f14b17975b74f7d
-
由 Yanqin Jin 提交于
Summary: Move the line `Add xxhash64 checksum support` to `Unreleased` section because it has not been released to 5.17. Pull Request resolved: https://github.com/facebook/rocksdb/pull/4627 Differential Revision: D12944123 Pulled By: riversand963 fbshipit-source-id: 2762857065b9e741c64ff8b6116ca62fe31891d8
-
- 06 11月, 2018 2 次提交
-
-
由 Maysam Yabandeh 提交于
Summary: When evicting an entry form the commit_cache, it is verified against the list of old snapshots to see if it overlaps with any. The list of old snapshots is split into two lists: an efficient concurrent cache and an slow vector protected by a lock. The patch fixes a bug that would stop the search in the cache if it finds any and yet would not include the larger snapshots in the slower list. An extra info log entry is also removed. The condition to trigger that although very rare is still feasible and should not spam the LOG when that happens. Fixes #4621 Pull Request resolved: https://github.com/facebook/rocksdb/pull/4639 Differential Revision: D12934989 Pulled By: maysamyabandeh fbshipit-source-id: 4e0fe8147ba292b554ae78e94c21c2ef31e03e2d
-
由 Andrew Kryczka 提交于
Summary: This property can help debug why SST files aren't being deleted. Previously we only had the property "rocksdb.is-file-deletions-enabled". However, even when that returned true, obsolete SSTs may still not be deleted due to the coarse-grained mechanism we use to prevent newly created SSTs from being accidentally deleted. That coarse-grained mechanism uses a lower bound file number for SSTs that should not be deleted, and this property exposes that lower bound. Pull Request resolved: https://github.com/facebook/rocksdb/pull/4618 Differential Revision: D12898179 Pulled By: ajkr fbshipit-source-id: fe68acc041ddbcc9276bbd48976524d95aafc776
-