- 22 7月, 2015 3 次提交
-
-
由 Mike Kolupaev 提交于
Summary: I'll just copy internal task summary here: " This sequence will cause data loss in the middle after an sync write: non-sync write key 1 flush triggered, not yet scheduled sync write key 2 system crash After rebooting, users might see key 2 but not key 1, which violates the API of sync write. This can be reproduced using unit test FaultInjectionTest::DISABLED_WriteOptionSyncTest. One way to fix it is for a sync write, if there is outstanding unsynced log files, we need to syc them too. " This diff should be considered together with the next diff D40905; in isolation this fix probably could be a little simpler. Test Plan: `make check`; added a test for that (DBTest.SyncingPreviousLogs) before noticing FaultInjectionTest.WriteOptionSyncTest (keeping both since mine asserts a bit more); both tests fail without this diff; for D40905 stacked on top of this diff, ran tests with ASAN, TSAN and valgrind Reviewers: rven, yhchiang, IslamAbdelRahman, anthony, kradhakrishnan, igor, sdong Reviewed By: sdong Subscribers: dhruba Differential Revision: https://reviews.facebook.net/D40899
-
由 Andres Notzli 提交于
Summary: Fixes T6548822. Added a new function for estimating the size of the live data as proposed in the task. The value can be accessed through the property rocksdb.estimate-live-data-size. Test Plan: There are two unit tests in version_set_test and a simple test in db_test. make version_set_test && ./version_set_test; make db_test && ./db_test gtest_filter=GetProperty Reviewers: rven, igor, yhchiang, sdong Reviewed By: sdong Subscribers: dhruba, leveldb Differential Revision: https://reviews.facebook.net/D41493
-
由 sdong 提交于
Summary: DBTest.GetPropertiesOfAllTablesTest generates four files and expects four files there, but a L0->L1 comapction can trigger to compact to one single file. Fix it by raising level 0 number of file compaction trigger Test Plan: Run it many times and see it never fails. Reviewers: kradhakrishnan, IslamAbdelRahman, yhchiang, anthony Reviewed By: anthony Subscribers: leveldb, dhruba Differential Revision: https://reviews.facebook.net/D42789
-
- 21 7月, 2015 14 次提交
-
-
由 Yueh-Hsuan Chiang 提交于
Summary: Move general compaction tests from db_test.cc to db_compaction_test.cc Test Plan: db_test db_compaction_test Reviewers: igor, sdong, IslamAbdelRahman, anthony Reviewed By: anthony Subscribers: dhruba, leveldb Differential Revision: https://reviews.facebook.net/D42651
-
由 agiardullo 提交于
Summary: Add new CheckFileExists method. Considered changing the FileExists api but didn't want to break anyone's builds. Test Plan: unit tests Reviewers: yhchiang, igor, sdong Reviewed By: sdong Subscribers: dhruba, leveldb Differential Revision: https://reviews.facebook.net/D42003
-
由 Yueh-Hsuan Chiang 提交于
Summary: Move remaining universal compaction tests from db_test.cc to db_universal_compaction_test.cc Test Plan: db_test db_universal_compaction_test Reviewers: igor, anthony, IslamAbdelRahman, sdong Reviewed By: sdong Subscribers: dhruba, leveldb Differential Revision: https://reviews.facebook.net/D42645
-
由 Yueh-Hsuan Chiang 提交于
Summary: Move in-place-update related tests from db_test.cc to db_inplace_update_test.cc Test Plan: db_test db_inplace_update_test Reviewers: igor, anthony, IslamAbdelRahman, sdong Reviewed By: sdong Subscribers: dhruba Differential Revision: https://reviews.facebook.net/D42657
-
由 Yueh-Hsuan Chiang 提交于
Summary: Fixed DBTestUniversalManualCompactionOutputPathId test by changing the expected number of files when setting up the test as flushes no-longer preempt compactions in patch https://reviews.facebook.net/D41931. Also, include db_universal_copaction_test in make all check. Test Plan: db_universal_copaction_test Reviewers: igor, sdong, IslamAbdelRahman, anthony Reviewed By: anthony Subscribers: dhruba, leveldb Differential Revision: https://reviews.facebook.net/D42639
-
由 krad 提交于
Summary: This unit test is blocking our release since it fails under certain compiler versions. The failure is due to a race in the unit test and not the core functionality. Test Plan: Run locally Reviewers: sdong CC: leveldb Task ID: #7760955 Blame Rev:
-
由 sdong 提交于
Summary: Now we allow trivial move in universal compaction. Add a parameter in db_bench Test Plan: Run db_bench with this option on and off and make sure the option is switched correctly. Reviewers: yhchiang, igor, kradhakrishnan, anthony Reviewed By: anthony Subscribers: leveldb, dhruba Differential Revision: https://reviews.facebook.net/D41427
-
由 sdong 提交于
Summary: Now the major test cases of fault_injection_test only insert keys in sorted order so compactions will be trivial move. Add a new mode to insert in non-sequential order to trigger non-trivial compactions. Test Plan: Run the test Reviewers: kradhakrishnan, anthony, IslamAbdelRahman, yhchiang, igor Reviewed By: igor Subscribers: leveldb, dhruba Differential Revision: https://reviews.facebook.net/D42435
-
由 Islam AbdelRahman 提交于
Summary: Skipping these tests in ROCKSDB_LITE since they are not supported json_document_test wal_manager_test ttl_test sst_dump_test deletefile_test compact_files_test prefix_test checkpoint_test Test Plan: json_document_test wal_manager_test ttl_test sst_dump_test deletefile_test compact_files_test prefix_test checkpoint_test Reviewers: igor, sdong, yhchiang, kradhakrishnan, anthony Reviewed By: anthony Subscribers: dhruba Differential Revision: https://reviews.facebook.net/D42573
-
由 Islam AbdelRahman 提交于
Summary: Make merge_test runnable in ROCKSDB_LITE Test Plan: merge_test Reviewers: igor, sdong, yhchiang, kradhakrishnan, anthony Reviewed By: anthony Subscribers: dhruba Differential Revision: https://reviews.facebook.net/D42579
-
由 Islam AbdelRahman 提交于
Summary: Block plain_table_db_test in ROCKSDB_LITE since plain table is not supported in ROCKSDB_LITE Test Plan: plain_table_db_test Reviewers: igor, sdong, yhchiang Reviewed By: yhchiang Subscribers: dhruba Differential Revision: https://reviews.facebook.net/D42159
-
由 Islam AbdelRahman 提交于
Summary: Add main for write_callback_test when compiled under ROCKSDB_LITE Test Plan: write_callback_test Reviewers: igor, sdong, yhchiang Reviewed By: yhchiang Subscribers: dhruba Differential Revision: https://reviews.facebook.net/D42147
-
由 Islam AbdelRahman 提交于
Summary: Cuckoo table is not supported in ROCKSDB_LITE, blocking it's tests Test Plan: cuckoo_table_builder_test cuckoo_table_db_test cuckoo_table_reader_test Reviewers: sdong, igor, yhchiang Reviewed By: yhchiang Subscribers: dhruba Differential Revision: https://reviews.facebook.net/D42141
-
由 Islam AbdelRahman 提交于
Summary: Remove universal and fifo compaction tests from ROCKSDB_LITE since they are not supported Test Plan: compaction_picker_test Reviewers: sdong, igor, yhchiang Reviewed By: yhchiang Subscribers: dhruba Differential Revision: https://reviews.facebook.net/D42129
-
- 18 7月, 2015 5 次提交
-
-
由 sdong 提交于
Move rate_limiter, write buffering, most perf context instrumentation and most random kill out of Env Summary: We want to keep Env a think layer for better portability. Less platform dependent codes should be moved out of Env. In this patch, I create a wrapper of file readers and writers, and put rate limiting, write buffering, as well as most perf context instrumentation and random kill out of Env. It will make it easier to maintain multiple Env in the future. Test Plan: Run all existing unit tests. Reviewers: anthony, kradhakrishnan, IslamAbdelRahman, yhchiang, igor Reviewed By: igor Subscribers: leveldb, dhruba Differential Revision: https://reviews.facebook.net/D42321
-
由 Igor Canadi 提交于
Summary: When we first started, max_background_flushes was 0 by default and compaction thread was executing flushes (since there was no flush thread). Then, we switched the default max_background_flushes to 1. However, we still support the case where there is no flush thread and flushes are done in compaction. This is making our code a bit more complicated. By not supporting this use-case we can make our code simpler. We have a special case that when you set max_background_flushes to 0, we schedule the flush to execute on the compaction thread. Test Plan: make check (there might be some unit tests that depend on this behavior) Reviewers: IslamAbdelRahman, yhchiang, sdong Reviewed By: sdong Subscribers: dhruba, leveldb Differential Revision: https://reviews.facebook.net/D41931
-
由 Ari Ekmekji 提交于
Summary: Added a new flag --json to the ldb manifest_dump command that prints out the version edits as JSON objects for easier reading and parsing of information. Test Plan: **Sample usage: ** ``` ./ldb manifest_dump --json --path=path/to/manifest/file ``` **Sample output:** ``` {"EditNumber": 0, "Comparator": "leveldb.BytewiseComparator", "ColumnFamily": 0} {"EditNumber": 1, "LogNumber": 0, "ColumnFamily": 0} {"EditNumber": 2, "LogNumber": 4, "PrevLogNumber": 0, "NextFileNumber": 7, "LastSeq": 35356, "AddedFiles": [{"Level": 0, "FileNumber": 5, "FileSize": 1949284, "SmallestIKey": "'", "LargestIKey": "'"}], "ColumnFamily": 0} ... {"EditNumber": 13, "PrevLogNumber": 0, "NextFileNumber": 36, "LastSeq": 290994, "DeletedFiles": [{"Level": 0, "FileNumber": 17}, {"Level": 0, "FileNumber": 20}, {"Level": 0, "FileNumber": 22}, {"Level": 0, "FileNumber": 24}, {"Level": 1, "FileNumber": 13}, {"Level": 1, "FileNumber": 14}, {"Level": 1, "FileNumber": 15}, {"Level": 1, "FileNumber": 18}], "AddedFiles": [{"Level": 1, "FileNumber": 25, "FileSize": 2114340, "SmallestIKey": "'", "LargestIKey": "'"}, {"Level": 1, "FileNumber": 26, "FileSize": 2115213, "SmallestIKey": "'", "LargestIKey": "'"}, {"Level": 1, "FileNumber": 27, "FileSize": 2114807, "SmallestIKey": "'", "LargestIKey": "'"}, {"Level": 1, "FileNumber": 30, "FileSize": 2115271, "SmallestIKey": "'", "LargestIKey": "'"}, {"Level": 1, "FileNumber": 31, "FileSize": 2115165, "SmallestIKey": "'", "LargestIKey": "'"}, {"Level": 1, "FileNumber": 32, "FileSize": 2114683, "SmallestIKey": "'", "LargestIKey": "'"}, {"Level": 1, "FileNumber": 35, "FileSize": 1757512, "SmallestIKey": "'", "LargestIKey": "'"}], "ColumnFamily": 0} ... ``` Reviewers: sdong, anthony, yhchiang, igor Reviewed By: igor Subscribers: dhruba Differential Revision: https://reviews.facebook.net/D41727
-
由 Igor Canadi 提交于
Summary: It has been around for a while and it looks like it never found any uses in the wild. It's also complicating our compaction_job code quite a bit. We're deprecating it in 3.13, but will put it back in 3.14 if we actually find users that need this feature. Test Plan: make check Reviewers: noetzli, yhchiang, sdong Reviewed By: sdong Subscribers: dhruba, leveldb Differential Revision: https://reviews.facebook.net/D42405
-
由 Andres Notzli 提交于
Summary: MergeUntil was not reporting a success when merging an operand with a Value/Deletion despite the comments in MergeHelper and CompactionJob indicating otherwise. This lead to operands being written to the compaction output unnecessarily: M1 M2 M3 P M4 M5 --> (P+M1+M2+M3) M2 M3 M4 M5 (before the diff) M1 M2 M3 P M4 M5 --> (P+M1+M2+M3) M4 M5 (after the diff) In addition, the code handling Values/Deletion was basically identical. This patch unifies the code. Finally, this patch also adds testing for merge_helper. Test Plan: make && make check Reviewers: sdong, rven, yhchiang, tnovak, igor Reviewed By: igor Subscribers: tnovak, dhruba, leveldb Differential Revision: https://reviews.facebook.net/D42351
-
- 17 7月, 2015 3 次提交
-
-
由 sdong 提交于
Summary: Typo seqeuntial -> sequential in db/fault_injection_test.cc Test Plan: Build it Reviewers: ott, igor, anthony, kradhakrishnan, IslamAbdelRahman Reviewed By: IslamAbdelRahman Subscribers: leveldb, dhruba Differential Revision: https://reviews.facebook.net/D42417
-
由 sdong 提交于
Summary: Previous run may leave some SST files with higher file numbers than manifest indicates. Compaction or flush may start to run while DB::Open() is still going on. SST file garbage collection may happen interleaving with compaction or flush, and overwrite files generated by compaction of flushes after they are generated. This might cause data loss. This possibility of interleaving is recently introduced. Fix it by not allowing compaction or flush to be scheduled before DB::Open() finishes. Test Plan: Add a unit test. This verification will have a chance to fail without the fix but doesn't fix without the fix. Reviewers: kradhakrishnan, anthony, yhchiang, IslamAbdelRahman, igor Reviewed By: igor Subscribers: dhruba, leveldb Differential Revision: https://reviews.facebook.net/D42399
-
由 Andres Notzli 提交于
Summary: Fixes T7697334. Adds a simple test to check whether CompactionJob deals with corrupted keys correctly. Right now, we preserve corrupted keys. Note: depending on the type of corruption and options like comparators, CompactionJob fails. This test just checks whether corrupted keys that do not fail CompactionJob are preserved. Test Plan: `make compaction_job_test && ./compaction_job_test` -> Tests pass. Add `input->Next(); continue;` in CompactionJob::ProcessKeyValueCompaction() inside then-branch of `!ParseInternalKey(key, &ikey)` -> Tests fail. Reviewers: sdong, rven, yhchiang, igor Reviewed By: igor Subscribers: dhruba, leveldb Differential Revision: https://reviews.facebook.net/D42237
-
- 16 7月, 2015 5 次提交
-
-
由 agiardullo 提交于
Summary: Moved convenience.h out of utilities to remove a dependency on utilities in db. Test Plan: unit tests. Also compiled a link to the old location to verify the _Pragma works. Reviewers: sdong, yhchiang, igor Reviewed By: igor Subscribers: dhruba, leveldb Differential Revision: https://reviews.facebook.net/D42201
-
Summary: Trvial move in universal compaction was failing when trying to move files from levels other than 0. This was because the DeleteFile while trivially moving, was only deleting files of level 0 which caused duplication of same file in different levels. This is fixed by passing the right level as argument in the call of DeleteFile while doing trivial move. Test Plan: ./db_test ran successfully with the new test cases. Reviewers: sdong Reviewed By: sdong Subscribers: leveldb, dhruba Differential Revision: https://reviews.facebook.net/D42135
-
由 krad 提交于
Summary: gcc-4.9-glibc-2.20 complains about uninitialized variable. db/compaction_picker.cc: In member function 'bool rocksdb::CompactionPicker::IsInputNonOverlapping(rocksdb::Compaction*)': db/compaction_picker.cc:1174:17: error: 'prev.rocksdb::{anonymous}::InputFileInfo::f' may be used uninitialized in this function [-Werror=maybe-uninitialized] InputFileInfo prev, curr, next; Test Plan: pmake on local environment Reviewers: sdong igor CC: leveldb@ Task ID: # Blame Rev:
-
由 Aaron Feldman 提交于
Summary: Remove --help entry for readhot. Update read_random_exp_range flag description: The distribution is num * exp(-r), not num * exp(r). Test Plan: Run ./db_bench --help Reviewers: sdong, igor Reviewed By: igor Subscribers: dhruba Differential Revision: https://reviews.facebook.net/D42303
-
由 Andres Notzli 提交于
Summary: Before, writing key/value pairs out to files was done inside ProcessKeyValueCompaction(). To make ProcessKeyValueCompaction() more understandable, this patch moves the writing part to a separate function. This is intended to be a stepping stone for additional changes. Test Plan: make && make check Reviewers: sdong, rven, yhchiang, igor Reviewed By: igor Subscribers: dhruba, leveldb Differential Revision: https://reviews.facebook.net/D42243
-
- 15 7月, 2015 8 次提交
-
-
由 Igor Canadi 提交于
Summary: I didn't know this can work :) Test Plan: none Reviewers: sdong, yhchiang Reviewed By: yhchiang Subscribers: dhruba, leveldb Differential Revision: https://reviews.facebook.net/D42081
-
由 Andres Notzli 提交于
Summary: Fixes two minor issues in CompactionJob. CompactionJob::Run() dereferences log_buffer_ without a check, so this patch adds an assert in the constructor where log_buffer_ is assigned. compaction_job_stats_ can be null but ProcessKeyValueCompaction was dereferencing it without a check. Test Plan: make && make check Reviewers: sdong, rven, yhchiang, igor Reviewed By: igor Subscribers: dhruba, leveldb Differential Revision: https://reviews.facebook.net/D42231
-
由 Yueh-Hsuan Chiang 提交于
Summary: Move UniversalCompaction related db-tests to db_universal_compaction_test.cc Test Plan: db_test db_universal_compaction_test Reviewers: igor, sdong, IslamAbdelRahman, anthony Reviewed By: anthony Subscribers: dhruba, leveldb Differential Revision: https://reviews.facebook.net/D42225
-
由 Yueh-Hsuan Chiang 提交于
Summary: Move TailingIterator tests from db_test.cc to db_test_tailing_iterator.cc Test Plan: db_test db_test_tailing_iterator Reviewers: igor, anthony, IslamAbdelRahman, sdong Reviewed By: sdong Subscribers: dhruba Differential Revision: https://reviews.facebook.net/D42021
-
由 Yueh-Hsuan Chiang 提交于
Summary: Make TransactionLogIterator related tests from db_test.cc to db_log_iter_test.cc Test Plan: db_test db_log_iter_test Reviewers: sdong, IslamAbdelRahman, igor, anthony Reviewed By: igor Subscribers: dhruba, leveldb Differential Revision: https://reviews.facebook.net/D42045
-
由 Yueh-Hsuan Chiang 提交于
Summary: Move CompactionFilter tests in db_test.cc to db_compaction_filter_test.cc Test Plan: db_test db_compaction_filter_test Reviewers: igor, sdong, IslamAbdelRahman, anthony Reviewed By: anthony Subscribers: dhruba, leveldb Differential Revision: https://reviews.facebook.net/D42207
-
由 agiardullo 提交于
Summary: Seems like the cleanest way to resolve this is to move CompactedDBImpl into db/. CompactedDBImpl should probably live in the same place as DBImplReadonly since Opening the latter could end up instantiating the former. Both DBImplReadonly and CompactedDBImpl inherit from DBImpl access protected members of DBImpl( and the latter access friendly private methods). Test Plan: unit tests Reviewers: sdong, yhchiang, kradhakrishnan, igor Reviewed By: igor Subscribers: dhruba, leveldb Differential Revision: https://reviews.facebook.net/D42027
-
由 Islam AbdelRahman 提交于
Summary: Remove ColumnFamiliesBatchWithIndexTest from ROCKSDB_LITE since WriteBatchWithIndex is not supported Test Plan: write_batch_test Reviewers: yhchiang, sdong, igor Reviewed By: igor Subscribers: dhruba Differential Revision: https://reviews.facebook.net/D42033
-
- 14 7月, 2015 2 次提交
-
-
由 Igor Canadi 提交于
Summary: This option is guarding the feature implemented 2 and a half years ago: D8991. The feature was enabled by default back then and has been running without issues. There is no reason why any client would turn this feature off. I found no reference in fbcode. Test Plan: none Reviewers: sdong, yhchiang, anthony, dhruba Reviewed By: dhruba Subscribers: dhruba, leveldb Differential Revision: https://reviews.facebook.net/D42063
-
由 Igor Canadi 提交于
Summary: In one of our recent meetings, we discussed deprecating features that are not being actively used. One of those features, at least within Facebook, is timeout_hint. The feature is really nicely implemented, but if nobody needs it, we should remove it from our code-base (until we get a valid use-case). Some arguments: * Less code == better icache hit rate, smaller builds, simpler code * The motivation for adding timeout_hint_us was to work-around RocksDB's stall issue. However, we're currently addressing the stall issue itself (see @sdong's recent work on stall write_rate), so we should never see sharp lock-ups in the future. * Nobody is using the feature within Facebook's code-base. Googling for `timeout_hint_us` also doesn't yield any users. Test Plan: make check Reviewers: anthony, kradhakrishnan, sdong, yhchiang Reviewed By: yhchiang Subscribers: sdong, dhruba, leveldb Differential Revision: https://reviews.facebook.net/D41937
-