- 29 5月, 2015 3 次提交
-
-
由 agiardullo 提交于
Summary: For transactions, we are using the memtables to validate that there are no write conflicts. But after flushing, we don't have any memtables, and transactions could fail to commit. So we want to someone keep around some extra history to use for conflict checking. In addition, we want to provide a way to increase the size of this history if too many transactions fail to commit. After chatting with people, it seems like everyone prefers just using Memtables to store this history (instead of a separate history structure). It seems like the best place for this is abstracted inside the memtable_list. I decide to create a separate list in MemtableListVersion as using the same list complicated the flush/installalflushresults logic too much. This diff adds a new parameter to control how much memtable history to keep around after flushing. However, it sounds like people aren't too fond of adding new parameters. So I am making the default size of flushed+not-flushed memtables be set to max_write_buffers. This should not change the maximum amount of memory used, but make it more likely we're using closer the the limit. (We are now postponing deleting flushed memtables until the max_write_buffer limit is reached). So while we might use more memory on average, we are still obeying the limit set (and you could argue it's better to go ahead and use up memory now instead of waiting for a write stall to happen to test this limit). However, if people are opposed to this default behavior, we can easily set it to 0 and require this parameter be set in order to use transactions. Test Plan: Added a xfunc test to play around with setting different values of this parameter in all tests. Added testing in memtablelist_test and planning on adding more testing here. Reviewers: sdong, rven, igor Reviewed By: igor Subscribers: dhruba, leveldb Differential Revision: https://reviews.facebook.net/D37443
-
由 Yueh-Hsuan Chiang 提交于
Summary: Rename EventLoggerHelpers EventHelpers, as it's going to include all event-related helper functions instead of EventLogger only stuffs. Test Plan: make Reviewers: sdong, rven, anthony Reviewed By: anthony Subscribers: dhruba, leveldb Differential Revision: https://reviews.facebook.net/D39093
-
由 Yueh-Hsuan Chiang 提交于
Summary: Move listeners from ColumnFamilyOptions to DBOptions Test Plan: listener_test compact_files_test Reviewers: rven, anthony, sdong Reviewed By: sdong Subscribers: dhruba, leveldb Differential Revision: https://reviews.facebook.net/D39087
-
- 27 5月, 2015 4 次提交
-
-
由 Yueh-Hsuan Chiang 提交于
Summary: Compaction now boosts the size of deletion entries of a file only when the number of deletion entries is greater than the number of non-deletion entries in the file. The motivation here is that in a stable workload, the number of deletion entries should be roughly equal to the number of non-deletion entries. If we compensate the size of deletion entries in a stable workload, the deletion compensation logic might introduce unwanted effet which changes the shape of LSM tree. Test Plan: db_test --gtest_filter="*Deletion*" Reviewers: sdong, igor Reviewed By: igor Subscribers: dhruba, leveldb Differential Revision: https://reviews.facebook.net/D38703
-
由 Igor Canadi 提交于
C: add more block based table stuff, some aux slice transform/merge ops
-
由 Yueh-Hsuan Chiang 提交于
Summary: Fixed a missing "}" at the end of the generated JSON Log in EventLoggerHelpers::LogTableFileCreation. Test Plan: db_bench Reviewers: igor Reviewed By: igor Subscribers: dhruba, leveldb Differential Revision: https://reviews.facebook.net/D38919
-
由 Yueh-Hsuan Chiang 提交于
Summary: Removed an unused private variable in db_impl.h Test Plan: make db_test Reviewers: sdong, anthony, igor Reviewed By: igor Subscribers: dhruba, leveldb Differential Revision: https://reviews.facebook.net/D38925
-
- 23 5月, 2015 12 次提交
-
-
由 Reed Allman 提交于
-
由 Reed Allman 提交于
-
由 Igor Canadi 提交于
adding docker build script and dockerfile for tools
-
由 agiardullo 提交于
Summary: fix typo Test Plan: none Reviewers: tfarina, igor Reviewed By: tfarina, igor Subscribers: dhruba, leveldb Differential Revision: https://reviews.facebook.net/D37347
-
由 Aaron Schlesinger 提交于
-
由 Aaron Schlesinger 提交于
-
由 Igor Canadi 提交于
Summary: We have a bug where we don't report the last level's files as being compacted. This fixes it. Test Plan: See the fix in action here: https://phabricator.fb.com/P19845738 Reviewers: MarkCallaghan, sdong Reviewed By: sdong Subscribers: yhchiang, dhruba, leveldb Differential Revision: https://reviews.facebook.net/D38727
-
由 Yueh-Hsuan Chiang 提交于
Summary: This patch fixes the following two bugs on logging file deletion. 1. Previously, file deletion failure was only logged in INFO_LEVEL. This patch changes it to ERROR_LEVEL and does some code clean. 2. EventLogger previously will always generate the same log on table file deletion even when file deletion is not successful. Now the resulting status of file deletion will also be logged. Test Plan: make all check Reviewers: sdong, igor Reviewed By: igor Subscribers: dhruba, leveldb Differential Revision: https://reviews.facebook.net/D38817
-
由 Yueh-Hsuan Chiang 提交于
Summary: Change the log-level of DB summary and options from INFO_LEVEL to WARN_LEVEL Test Plan: Use db_bench to verify the log level. Sample output: 2015/05/22-00:20:39.778064 7fff75b41300 [WARN] RocksDB version: 3.11.0 2015/05/22-00:20:39.778095 7fff75b41300 [WARN] Git sha rocksdb_build_git_sha:7fee8775 2015/05/22-00:20:39.778099 7fff75b41300 [WARN] Compile date May 22 2015 2015/05/22-00:20:39.778101 7fff75b41300 [WARN] DB SUMMARY 2015/05/22-00:20:39.778145 7fff75b41300 [WARN] SST files in /tmp/rocksdbtest-691931916/dbbench dir, Total Num: 0, files: 2015/05/22-00:20:39.778148 7fff75b41300 [WARN] Write Ahead Log file in /tmp/rocksdbtest-691931916/dbbench: 2015/05/22-00:20:39.778150 7fff75b41300 [WARN] Options.error_if_exists: 0 2015/05/22-00:20:39.778152 7fff75b41300 [WARN] Options.create_if_missing: 1 2015/05/22-00:20:39.778153 7fff75b41300 [WARN] Options.paranoid_checks: 1 Reviewers: MarkCallaghan, igor, kradhakrishnan Reviewed By: igor Subscribers: sdong, dhruba, leveldb Differential Revision: https://reviews.facebook.net/D38835
-
由 Yueh-Hsuan Chiang 提交于
Summary: Ensure ColumnFamilyOptions.num_levels >= 2 when level compaction is used. Test Plan: Extend SanitizeOptions test in column_family_test Reviewers: sdong, rven, anthony, krishnanm86, igor Reviewed By: igor Subscribers: dhruba, leveldb Differential Revision: https://reviews.facebook.net/D38829
-
由 Yueh-Hsuan Chiang 提交于
Summary: Avoid logging under mutex in DBImpl::WriteLevel0TableForRecovery(). Test Plan: make all check Reviewers: igor, sdong Reviewed By: sdong Subscribers: dhruba, leveldb Differential Revision: https://reviews.facebook.net/D38823
-
由 Igor Canadi 提交于
Summary: Sometimes we want to run tests sequentially. J=1 gives us that option Test Plan: make J=1 check -- sequential make J=2 check -- parallel Reviewers: sdong, yhchiang, meyering Reviewed By: meyering Subscribers: dhruba, leveldb Differential Revision: https://reviews.facebook.net/D38805
-
- 22 5月, 2015 6 次提交
-
-
由 Yueh-Hsuan Chiang 提交于
Summary: Allow EventLogger to directly log from a JSONWriter. This allows the JSONWriter to be shared by EventLogger and potentially EventListener, which is an important step to integrate EventLogger and EventListener. This patch also rewrites EventLoggerHelpers::LogTableFileCreation(), which uses the new API to generate identical log. Test Plan: Run db_bench in debug mode and make sure the log is correct and no assertions fail. Reviewers: sdong, anthony, igor Reviewed By: igor Subscribers: dhruba, leveldb Differential Revision: https://reviews.facebook.net/D38709
-
由 Igor Canadi 提交于
Summary: This turns out to be pretty bad because if we prioritize L0->L1 then L1 can grow artificially large, which makes L0->L1 more and more expensive. For example: 256MB @ L0 + 256MB @ L1 --> 512MB @ L1 256MB @ L0 + 512MB @ L1 --> 768MB @ L1 256MB @ L0 + 768MB @ L1 --> 1GB @ L1 .... 256MB @ L0 + 10GB @ L1 --> 10.2GB @ L1 At some point we need to start compacting L1->L2 to speed up L0->L1. Test Plan: The performance improvement is massive for heavy write workload. This is the benchmark I ran: https://phabricator.fb.com/P19842671. Before this change, the benchmark took 47 minutes to complete. After, the benchmark finished in 2minutes. You can see full results here: https://phabricator.fb.com/P19842674 Also, we ran this diff on MongoDB on RocksDB on one replicaset. Before the change, our initial sync was so slow that it couldn't keep up with primary writes. After the change, the import finished without any issues Reviewers: dynamike, MarkCallaghan, rven, yhchiang, sdong Reviewed By: sdong Subscribers: dhruba, leveldb Differential Revision: https://reviews.facebook.net/D38637
-
由 Igor Canadi 提交于
Summary: Having stats in our LOG more often will help a lot with perf debugging. Test Plan: none Reviewers: sdong, MarkCallaghan Reviewed By: MarkCallaghan Subscribers: dhruba, leveldb Differential Revision: https://reviews.facebook.net/D38781
-
由 Yueh-Hsuan Chiang 提交于
Summary: Make DB::GetDbIdentity() be const function. Test Plan: make db_test Reviewers: igor, rven, sdong Reviewed By: sdong Subscribers: dhruba, leveldb Differential Revision: https://reviews.facebook.net/D38745
-
New features title was repeated twice. Fixed it.
-
由 Yueh-Hsuan Chiang 提交于
Add LDFLAGS to Java static library
-
- 21 5月, 2015 1 次提交
-
-
由 Yueh-Hsuan Chiang 提交于
Summary: Rename JSONWritter to JSONWriter Test Plan: make Reviewers: igor Reviewed By: igor Subscribers: dhruba, leveldb Differential Revision: https://reviews.facebook.net/D38733
-
- 20 5月, 2015 7 次提交
-
-
由 DerekSchenk 提交于
Includes the LDFLAGS so that the correct libraries will be linked. This links rt to resolve the issue https://github.com/facebook/rocksdb/issues/606.
-
由 Yueh-Hsuan Chiang 提交于
Summary: Dump db stats in WARN level Test Plan: run db_bench and verify the LOG Reviewers: igor, MarkCallaghan Subscribers: dhruba, leveldb Differential Revision: https://reviews.facebook.net/D38691
-
由 Yueh-Hsuan Chiang 提交于
Summary: Update HISTORY.md for GetThreadList() update. Test Plan: no code change Reviewers: sdong, rven, anthony, krishnanm86, igor Reviewed By: igor Subscribers: dhruba, leveldb Differential Revision: https://reviews.facebook.net/D38685
-
由 Mark Callaghan 提交于
Summary: See https://gist.github.com/mdcallag/89ebb2b8cbd331854865 for the IO stats. I added "Cumulative compaction:" and "Interval compaction:" lines. The IO rates can be confusing. Rates fro per-level stats lines, Wr(MB/s) & Rd(MB/s), are computed using the duration of the compaction job. If the job reads 10MB, writes 9MB and the job (IO & merging) takes 1 second then the rates are 10MB/s for read and 9MB/s for writes. The IO rates in the Cumulative compaction line uses the total uptime. The IO rates in the Interval compaction line uses the interval uptime. So these Cumalative & Interval compaction IO rates cannot be compared to the per-level IO rates. But both forms of the rates are useful for debugging perf. Task ID: # Blame Rev: Test Plan: run db_bench Revert Plan: Database Impact: Memcache Impact: Other Notes: EImportant: - begin *PUBLIC* platform impact section - Bugzilla: # - end platform impact - Reviewers: igor Reviewed By: igor Subscribers: dhruba Differential Revision: https://reviews.facebook.net/D38667
-
Flipped the unreleased section to 3.11
-
由 Igor Canadi 提交于
Summary: In third-party2 build we need to force git sha because we're compiling from a different git repositry. Test Plan: `FORCE_GIT_SHA=igor make` Reviewers: kradhakrishnan, sdong Reviewed By: kradhakrishnan Subscribers: dhruba, leveldb Differential Revision: https://reviews.facebook.net/D38679
-
由 Igor Canadi 提交于
Summary: Not sure why this fails on some compilers and doesn't on others. Test Plan: none Reviewers: meyering, sdong Reviewed By: sdong Subscribers: dhruba, leveldb Differential Revision: https://reviews.facebook.net/D38673
-
- 19 5月, 2015 6 次提交
-
-
由 Igor Canadi 提交于
Summary: sync_file_range is not always asyncronous and thus can block writes if we do this for WAL in the foreground thread. See more here: http://yoshinorimatsunobu.blogspot.com/2014/03/how-syncfilerange-really-works.html Some users don't want us to call sync_file_range on WALs. Some other do. Thus, I'm adding a separate option wal_bytes_per_sync to control calling sync_file_range on WAL files. bytes_per_sync will apply only to table files now. Test Plan: no more sync_file_range for WAL as evidenced by strace Reviewers: yhchiang, rven, sdong Reviewed By: sdong Subscribers: dhruba, leveldb Differential Revision: https://reviews.facebook.net/D38253
-
由 Igor Canadi 提交于
Summary: As title. I spent some time thinking about it and I don't think there should be any issue with running manual compaction and flushes in parallel Test Plan: make check works Reviewers: rven, yhchiang, sdong Reviewed By: yhchiang, sdong Subscribers: dhruba, leveldb Differential Revision: https://reviews.facebook.net/D38355
-
由 Yueh-Hsuan Chiang 提交于
Summary: Fixed the following compile errors due to some gcc does not have std::map::emplace util/thread_status_impl.cc: In static member function ‘static std::map<std::basic_string<char>, long unsigned int> rocksdb::ThreadStatus::InterpretOperationProperties(rocksdb::ThreadStatus::OperationType, const uint64_t*)’: util/thread_status_impl.cc:88:20: error: ‘class std::map<std::basic_string<char>, long unsigned int>’ has no member named ‘emplace’ util/thread_status_impl.cc:90:20: error: ‘class std::map<std::basic_string<char>, long unsigned int>’ has no member named ‘emplace’ util/thread_status_impl.cc:94:20: error: ‘class std::map<std::basic_string<char>, long unsigned int>’ has no member named ‘emplace’ util/thread_status_impl.cc:96:20: error: ‘class std::map<std::basic_string<char>, long unsigned int>’ has no member named ‘emplace’ util/thread_status_impl.cc:98:20: error: ‘class std::map<std::basic_string<char>, long unsigned int>’ has no member named ‘emplace’ util/thread_status_impl.cc:101:20: error: ‘class std::map<std::basic_string<char>, long unsigned int>’ has no member named ‘emplace’ make: *** [util/thread_status_impl.o] Error 1 Test Plan: make db_bench Reviewers: igor Subscribers: dhruba, leveldb Differential Revision: https://reviews.facebook.net/D38643
-
由 stash93 提交于
Summary: Call Flush() function instead Test Plan: make all check Reviewers: igor Reviewed By: igor Subscribers: dhruba Differential Revision: https://reviews.facebook.net/D38583
-
由 sdong 提交于
Summary: DBTest.DynamicLevelMaxBytesCompactRange needs to make sure L0 is not empty to properly cover the code paths we want to cover. However, current codes have a bug that might leave the condition not held. Improve the test to ensure it. Test Plan: Run the test in an environment that is used to fail. Also run it many times. Subscribers: leveldb, dhruba Differential Revision: https://reviews.facebook.net/D38631
-
由 sdong 提交于
Summary: CompactRange() now is much more expensive for dynamic level base size as it goes through all the levels. Skip those not used levels between level 0 an base level. Test Plan: Run all unit tests Reviewers: yhchiang, rven, anthony, kradhakrishnan, igor Reviewed By: igor Subscribers: leveldb, dhruba Differential Revision: https://reviews.facebook.net/D37125
-
- 16 5月, 2015 1 次提交
-
-
由 Yueh-Hsuan Chiang 提交于
Summary: Allow GetThreadList to report Flush properties, which includes: * job id * number of bytes that has been written since flush started. * total size of input mem-tables Test Plan: ./db_bench --threads=30 --num=1000000 --benchmarks=fillrandom --thread_status_per_interval=100 --value_size=1000 Sample output from db_bench which tracks same flush job ThreadID ThreadType cfName Operation ElapsedTime Stage State OperationProperties 140213879898240 High Pri default Flush 5789 us FlushJob::WriteLevel0Table BytesMemtables 4112835 | BytesWritten 577104 | JobID 8 | ThreadID ThreadType cfName Operation ElapsedTime Stage State OperationProperties 140213879898240 High Pri default Flush 30.634 ms FlushJob::WriteLevel0Table BytesMemtables 4112835 | BytesWritten 1734865 | JobID 8 | Reviewers: rven, igor, sdong Reviewed By: sdong Subscribers: dhruba, leveldb Differential Revision: https://reviews.facebook.net/D38505
-