- 25 4月, 2015 1 次提交
-
-
由 clark.kang 提交于
-
- 24 4月, 2015 7 次提交
-
-
由 sdong 提交于
Summary: Make it build for CYGWIN. Need to define "-std=gnu++11" instead of "-std=c++11" and use some replacement functions. Test Plan: Build it and run some unit tests in CYGWIN Reviewers: yhchiang, rven, anthony, kradhakrishnan, igor Reviewed By: igor Subscribers: leveldb, dhruba Differential Revision: https://reviews.facebook.net/D37605
-
由 sdong 提交于
Summary: CompactRange for universal compaction with num_levels > 1 seems to have a bug. The unit test also has a bug so it doesn't capture the problem. Fix it. Revert the compact range to the logic equivalent to num_levels=1. Always compact all files together. It should also fix DBTest.IncreaseUniversalCompactionNumLevels. The issue was that options.write_buffer_size = 100 << 10 and options.write_buffer_size = 100 << 10 are not used in later test scenarios. So write_buffer_size of 4MB was used. The compaction trigger condition is not anymore obvious as expected. Test Plan: Run the new test and all test suites Reviewers: yhchiang, rven, kradhakrishnan, anthony, igor Reviewed By: igor Subscribers: leveldb, dhruba Differential Revision: https://reviews.facebook.net/D37551
-
由 Igor Canadi 提交于
Summary: Based on feedback from D37083. Are all of these correct? In some spaces it seems like we're doing SetMaxPossibleForUserKey() although we want the smallest possible internal key for user key. Test Plan: make check Reviewers: sdong Reviewed By: sdong Subscribers: dhruba, leveldb Differential Revision: https://reviews.facebook.net/D37341
-
由 Igor Canadi 提交于
Summary: Reading CompactionPicker I noticed this dangerous substraction of two unsigned integers. We should assert to mark this as safe. Test Plan: make check Reviewers: anthony, rven, yhchiang, sdong Reviewed By: sdong Subscribers: kradhakrishnan, dhruba, leveldb Differential Revision: https://reviews.facebook.net/D37041
-
由 Giuseppe Ottaviano 提交于
Summary: This diff implements a new `DB` method `PromoteL0` which moves all files in L0 to a given level skipping compaction, provided that the files have disjoint ranges and all levels up to the target level are empty. This method provides finer-grain control for trivial compactions, and it is useful for bulk-loading pre-sorted keys. Compared to D34797, it does not change the semantics of an existing operation, which can impact existing code. PromoteL0 is designed to work well in combination with the proposed `GetSstFileWriter`/`AddFile` interface, enabling to "design" the level structure by populating one level at a time. Such fine-grained control can be very useful for static or mostly-static databases. Test Plan: `make check` Reviewers: IslamAbdelRahman, philipp, MarkCallaghan, yhchiang, igor, sdong Reviewed By: sdong Subscribers: dhruba Differential Revision: https://reviews.facebook.net/D37107
-
由 sdong 提交于
Summary: Add more logging to help debugging issues. Test Plan: Run test suites Reviewers: yhchiang, rven, igor Reviewed By: igor Subscribers: leveldb, dhruba Differential Revision: https://reviews.facebook.net/D37401
-
由 sdong 提交于
Summary: To further distinguish the corruption cases were caused by storage media or in memory states when writing it, add a paranoid check after writing the file to iterate all the rows. Test Plan: Add a new unit test for it Reviewers: rven, igor Subscribers: leveldb, dhruba Differential Revision: https://reviews.facebook.net/D37335
-
- 23 4月, 2015 1 次提交
-
-
由 Venkatesh Radhakrishnan 提交于
Summary: A couple of times on Travis, we have had the thread status say that there were no compactions done and since we assert for it, the test failed. We now fix this by waiting till compaction started. Test Plan: run DBTEST::*PreShutdown* d=/tmp/j; rm -rf $d; seq 200 | parallel --gnu --eta 'd=/tmp/j/d-{}; mkdir -p $d; TEST_TMPDIR=$d ./db_test --gtest_filter=DBTest.PreShutdown* >& '$d'/log-{}' Reviewers: sdong, igor Reviewed By: igor Subscribers: dhruba Differential Revision: https://reviews.facebook.net/D37545
-
- 18 4月, 2015 3 次提交
-
-
由 Igor Canadi 提交于
Summary: Some Mongo+Rocks datasets in Parse's environment are not doing compactions very frequently. During the quiet period (with no IO), we'd like to schedule compactions so that our reads become faster. Also, aggressively compacting during quiet periods helps when write bursts happen. In addition, we also want to compact files that are containing deleted key ranges (like old oplog keys). All of this is currently not possible with CompactRange() because it's single-threaded and blocks all other compactions from happening. Running CompactRange() risks an issue of blocking writes because we generate too much Level 0 files before the compaction is over. Stopping writes is very dangerous because they hold transaction locks. We tried running manual compaction once on Mongo+Rocks and everything fell apart. MarkForCompaction() solves all of those problems. This is very light-weight manual compaction. It is lower priority than automatic compactions, which means it shouldn't interfere with background process keeping the LSM tree clean. However, if no automatic compactions need to be run (or we have extra background threads available), we will start compacting files that are marked for compaction. Test Plan: added a new unit test Reviewers: yhchiang, rven, MarkCallaghan, sdong Reviewed By: sdong Subscribers: yoshinorim, dhruba, leveldb Differential Revision: https://reviews.facebook.net/D37083
-
由 Jim Meyering 提交于
Summary: The usage I'm fixing here caused trouble on Fedora 21 when compiling with the current gcc version 4.9.2 20150212 (Red Hat 4.9.2-6) (GCC): db/write_controller_test.cc: In member function ‘virtual void rocksdb::WriteControllerTest_SanityTest_Test::TestBody()’: db/write_controller_test.cc:23:165: error: converting ‘false’ to pointer type for argument 1 of ‘char testing::internal::IsNullLiteralHelper(testing::internal::Secret*)’ [-Werror=conversion-null] ASSERT_EQ(false, controller.IsStopped()); ^ This change was induced mechanically via: git grep -l -E 'ASSERT_EQ\(false'|xargs perl -pi -e 's/ASSERT_EQ\(false, /ASSERT_FALSE(/' git grep -l -E 'ASSERT_EQ\(true'|xargs perl -pi -e 's/ASSERT_EQ\(true, /ASSERT_TRUE(/' Except for the three in utilities/backupable/backupable_db_test.cc for which I ended up reformatting (joining lines) in the result. As for why this problem is exhibited with that version of gcc, and none of the others I've used (from 4.8.1 through gcc-5.0.0 and newer), I suspect it's a bug in F21's gcc that has been fixed in gcc-5.0.0. Test Plan: "make" now succeed on Fedora 21 Reviewers: ljin, rven, igor.sugak, yhchiang, sdong, igor Reviewed By: igor Subscribers: dhruba Differential Revision: https://reviews.facebook.net/D37329
-
由 Igor Canadi 提交于
Summary: this is not used anywhere Test Plan: compiles Reviewers: yhchiang, sdong Reviewed By: sdong Subscribers: dhruba, leveldb Differential Revision: https://reviews.facebook.net/D37053
-
- 17 4月, 2015 1 次提交
-
-
由 Igor Canadi 提交于
Summary: If ExpandWhileOverlapping() we don't clear inputs. That's a bug introduced by my recent patch https://reviews.facebook.net/D36687. However, we have no tests covering ExpandWhileOverlapping(). I created a task t6771252 to add ExpandWhileOverlapping() tests. Test Plan: make check Reviewers: sdong, rven, yhchiang Reviewed By: yhchiang Subscribers: dhruba, leveldb Differential Revision: https://reviews.facebook.net/D37077
-
- 15 4月, 2015 4 次提交
-
-
由 sdong 提交于
Summary: D36669 introduces a bug that trivial moved data is not going to specific level but the next level, which will incorrectly be level 1 for level 0 compaciton if base level is not level 1. Fixing it by appreciating the output level Test Plan: Run all tests Reviewers: MarkCallaghan, rven, yhchiang, igor Reviewed By: igor Subscribers: leveldb, dhruba Differential Revision: https://reviews.facebook.net/D37119
-
由 sdong 提交于
Summary: Recent change of DBTest.DynamicLevelCompressionPerLevel2 has a bug that the second sync point is not enabled. Fix it. Also add an assert for that. Also, flush compression is not tracked in the test. Add it. Test Plan: Build everything Subscribers: leveldb, dhruba Differential Revision: https://reviews.facebook.net/D37101
-
由 sdong 提交于
Summary: When commiting the sync point interface change, didn't resolve the new occurance of the old interface in rebase. Fix it. Test Plan: Build and see it pass Reviewers: igor, yhchiang, rven, anthony, kradhakrishnan Subscribers: leveldb, dhruba Differential Revision: https://reviews.facebook.net/D37095
-
由 sdong 提交于
SyncPoint to allow a callback with an argument and use it to get DBTest.DynamicLevelCompressionPerLevel2 more straight-forward Summary: Allow users to give a callback function with parameter using sync point, so more complicated verification can be done in tests. Use it in DBTest.DynamicLevelCompressionPerLevel2 so that failures will be more easy to debug. Test Plan: Run all tests. Run DBTest.DynamicLevelCompressionPerLevel2 with valgrind check. Reviewers: rven, yhchiang, anthony, kradhakrishnan, igor Reviewed By: igor Subscribers: leveldb, dhruba Differential Revision: https://reviews.facebook.net/D36999
-
- 14 4月, 2015 4 次提交
-
-
由 Igor Canadi 提交于
Summary: https://reviews.facebook.net/D36963 made the debug build much faster and that triggered failures of CompactFilesOnLevelCompaction test. 3 out of 4 last tests on Jenkins failed. I'm disabling this test temporarily, since we likely know the reason why it's failing and there's already work in progress to address it -- https://reviews.facebook.net/D36225 Test Plan: none Reviewers: sdong, rven, yhchiang, meyering Reviewed By: meyering Subscribers: dhruba, leveldb Differential Revision: https://reviews.facebook.net/D36993
-
由 Igor Canadi 提交于
Summary: We should use mocked-out env for these tests to make it more realiable. Added benefit is that instead of actually sleeping for 3 seconds, we can instead pretend to sleep and just increase time counters. Test Plan: for i in `seq 100`; do ./wal_manager_test --gtest_filter=WalManagerTest.WALArchivalTtl ;done Reviewers: rven, meyering Reviewed By: meyering Subscribers: meyering, dhruba, leveldb Differential Revision: https://reviews.facebook.net/D36951
-
由 Igor Canadi 提交于
Summary: The problem is that sometimes two memtables will be compacted together into a single file. In that case, our assertion ASSERT_EQ(NumTableFilesAtLevel(0), 5); fails because same amount of data is in 4 files instead of 5. We should wait for flush so that we prevent two memtables merging into a single file. Test Plan: `for i in `seq 20`; do mrtest FIFOCompactionTest; done` -- fails at least once before. fails zero times after. Reviewers: rven Reviewed By: rven Subscribers: dhruba, leveldb Differential Revision: https://reviews.facebook.net/D36939
-
由 Igor Canadi 提交于
Summary: 1. it doesn't work 2. we're not using it In the future, if we need general benchmark framework, we should probably use https://github.com/google/benchmark Test Plan: make all Reviewers: yhchiang, rven, anthony, sdong Reviewed By: sdong Subscribers: dhruba, leveldb Differential Revision: https://reviews.facebook.net/D36777
-
- 11 4月, 2015 4 次提交
-
-
由 Igor Canadi 提交于
Summary: oops Test Plan: compiles now Reviewers: sdong, yhchiang Subscribers: dhruba, leveldb Differential Revision: https://reviews.facebook.net/D36867
-
由 Igor Canadi 提交于
Summary: The goal of this diff is to make Compaction class easier to use. This should also make new compaction algorithms easier to write (like CompactFiles from @yhchiang and dynamic leveled and multi-leveled universal from @sdong). Here are couple of things demonstrating that Compaction class is hard to use: 1. we have two constructors of Compaction class 2. there's this thing called grandparents_, but it appears to only be setup for leveled compaction and not compactfiles 3. it's easy to introduce a subtle and dangerous bug like this: D36225 4. SetupBottomMostLevel() is hard to understand and it shouldn't be. See this comment: https://github.com/facebook/rocksdb/blob/afbafeaeaebfd27a0f3e992fee8e0c57d07658fa/db/compaction.cc#L236-L241. It also made it harder for @yhchiang to write CompactFiles, as evidenced by this: https://github.com/facebook/rocksdb/blob/afbafeaeaebfd27a0f3e992fee8e0c57d07658fa/db/compaction_picker.cc#L204-L210 The problem is that we create Compaction object, which holds a lot of state, and then pass it around to some functions. After those functions are done mutating, then we call couple of functions on Compaction object, like SetupBottommostLevel() and MarkFilesBeingCompacted(). It is very hard to see what's happening with all that Compaction's state while it's travelling across different functions. If you're writing a new PickCompaction() function you need to try really hard to understand what are all the functions you need to run on Compaction object and what state you need to setup. My proposed solution is to make important parts of Compaction immutable after construction. PickCompaction() should calculate compaction inputs and then pass them onto Compaction object once they are finalized. That makes it easy to create a new compaction -- just provide all the parameters to the constructor and you're done. No need to call confusing functions after you created your object. This diff doesn't fully achieve that goal, but it comes pretty close. Here are some of the changes: * have one Compaction constructor instead of two. * inputs_ is constant after construction * MarkFilesBeingCompacted() is now private to Compaction class and automatically called on construction/destruction. * SetupBottommostLevel() is gone. Compaction figures it out on its own based on the input. * CompactionPicker's functions are not passing around Compaction object anymore. They are only passing around the state that they need. Test Plan: make check make asan_check make valgrind_check Reviewers: rven, anthony, sdong, yhchiang Reviewed By: yhchiang Subscribers: sdong, yhchiang, dhruba, leveldb Differential Revision: https://reviews.facebook.net/D36687
-
由 agiardullo 提交于
Summary: Need to remember to unref MemTableList->current() before deleting. Test Plan: ran test with valgrind Reviewers: igor Reviewed By: igor Subscribers: dhruba, leveldb Differential Revision: https://reviews.facebook.net/D36855
-
由 krad 提交于
Summary: Adding verbosity to existing comments. Test Plan: None Reviewers: sdong CC: leveldb Task ID: #6718960 Blame Rev:
-
- 10 4月, 2015 7 次提交
-
-
由 agiardullo 提交于
Summary: Test failing due to a missing directory caused by a simple bug (did not run into this on my dev box since the path already existed). We should look into deleting test::TmpDir() before each test run. Test Plan: ran test Reviewers: igor, yhchiang, meyering, sdong Reviewed By: sdong Subscribers: dhruba, leveldb Differential Revision: https://reviews.facebook.net/D36831
-
由 Yueh-Hsuan Chiang 提交于
Summary: Fixed xfunc related compile errors in ROCKSDB_LITE Now make OPT=-DROCKSDB_LITE shared_lib -j32 would work Test Plan: make clean make OPT=-DROCKSDB_LITE shared_lib -j32 make clean make OPT=-DROCKSDB_LITE static_lib -j32 Reviewers: sdong, igor Subscribers: dhruba, leveldb Differential Revision: https://reviews.facebook.net/D36825
-
由 agiardullo 提交于
Summary: Add tests for MemTableList Test Plan: run test Reviewers: yhchiang, kradhakrishnan, sdong, igor Reviewed By: igor Subscribers: dhruba, leveldb Differential Revision: https://reviews.facebook.net/D36735
-
由 Yueh-Hsuan Chiang 提交于
Summary: Fix a compile error in ROCKSDB_LITE in db/db_impl.cc related to internal_stats. Test Plan: make OPT=-DROCKSDB_LITE shared_lib Reviewers: sdong, igor Subscribers: dhruba, leveldb Differential Revision: https://reviews.facebook.net/D36819
-
由 Yueh-Hsuan Chiang 提交于
Summary: Fix a compilation error in ROCKSDB_LITE in db/internal_stats.h Other compilation errors will be fixed in a separate diff. Test Plan: make OPT=-DROCKSDB_LITE Reviewers: sdong, igor Subscribers: dhruba, leveldb Differential Revision: https://reviews.facebook.net/D36807
-
由 sdong 提交于
Summary: Add a test case: Write some keys without sync, flush, write other keys and do sync. Before flush finishes, host crashes and unsync data is dropped. Tag the new test as disabled since it is not passing. Test Plan: Run the test Reviewers: MarkCallaghan, rven, anthony, igor, kradhakrishnan Reviewed By: igor Subscribers: leveldb, dhruba Differential Revision: https://reviews.facebook.net/D36741
-
由 Mark Callaghan 提交于
Summary: This fixes two problems: 1) the env should not be created twice when use_existing_db is false 2) the env dtor should run before cachedev_fd_ is closed. Task ID: # Blame Rev: Test Plan: Revert Plan: Database Impact: Memcache Impact: Other Notes: EImportant: - begin *PUBLIC* platform impact section - Bugzilla: # - end platform impact - Reviewers: igor Reviewed By: igor Subscribers: dhruba Differential Revision: https://reviews.facebook.net/D36795
-
- 09 4月, 2015 4 次提交
-
-
由 agiardullo 提交于
Summary: Other than making some class members private, this is a documentation-only change Test Plan: unit tests Reviewers: sdong, yhchiang, igor Reviewed By: igor Subscribers: dhruba, leveldb Differential Revision: https://reviews.facebook.net/D36567
-
由 krad 提交于
Summary: I think the checksum was turned off by mistake. Test Plan: Run make check Reviewers: igor sdong chip CC: Task ID: Blame Rev:
-
由 sdong 提交于
Summary: Now EnvOptions uses unsanitized DB options. bytes_per_sync is tuned off when rate_limiter is used, but this change doesn't take effort. Test Plan: See different I/O pattern in db_bench running fillseq. Reviewers: yhchiang, kradhakrishnan, rven, anthony, igor Reviewed By: igor Subscribers: leveldb, dhruba Differential Revision: https://reviews.facebook.net/D36723
-
由 sdong 提交于
Summary: Now trivial move is only triggered when moving from level n to n+1. With dynamic level base, it is possible that file is moved from level 0 to level n, while levels from 1 to n-1 are empty. Extend trivial move to this case. Test Plan: Add a more unit test of sequential loading. Non-trivial compaction happened without the patch and now doesn't happen. Reviewers: rven, yhchiang, MarkCallaghan, igor Reviewed By: igor Subscribers: leveldb, dhruba, IslamAbdelRahman Differential Revision: https://reviews.facebook.net/D36669
-
- 08 4月, 2015 1 次提交
-
-
由 krad 提交于
Summary: Added a ASCII doodle to represent the log writer format. Test Plan: None Reviewers: sdong CC: leveldb Task ID: 6179896 Blame Rev:
-
- 07 4月, 2015 3 次提交
-
-
由 Yoshinori Matsunobu 提交于
Summary: D36447 caused build error when using COMPILE_WITH_TSAN=1. This diff fixes the error. Test Plan: jenkins Reviewers: igor, sdong Reviewed By: sdong Subscribers: dhruba Differential Revision: https://reviews.facebook.net/D36579
-
由 Yoshinori Matsunobu 提交于
Summary: There are some cases when flachcache file descriptor was already allocated (i.e. fb-MySQL). Then NewFlashcacheAwareEnv returns an error at open() because fd was already assigned. This diff adds another function to instantiate FlashcacheAwareEnv, with pre-allocated fd cachedev_fd. Test Plan: Tested with MyRocks using this function, then worked Reviewers: sdong, igor Reviewed By: igor Subscribers: dhruba, MarkCallaghan, rven Differential Revision: https://reviews.facebook.net/D36447
-
由 Igor Canadi 提交于
Summary: Now we add warnings when user configures compression and the compression is not supported. Test Plan: Configured compression to non-supported values. Observed messages in my log: 2015/03/26-12:17:57.586341 7ffb8a496840 [WARN] Compression type chosen for level 2 is not supported: LZ4. RocksDB will not compress data on level 2. 2015/03/26-12:19:10.768045 7f36f15c5840 [WARN] Compression type chosen is not supported: LZ4. RocksDB will not compress data. Reviewers: rven, sdong, yhchiang Reviewed By: sdong Subscribers: dhruba, leveldb Differential Revision: https://reviews.facebook.net/D35979
-