- 19 8月, 2015 4 次提交
-
-
由 Yueh-Hsuan Chiang 提交于
Summary: This patch implements DBOptions deserialization and improve the current implementation of DBOptions serialization by using a static structure that stores the offset of each DBOptions member variables to perform serialization and deserialization instead of using tons of if-then-branch to determine the mapping between string and variables. Test Plan: Added test in options_test.cc Reviewers: igor, anthony, sdong, IslamAbdelRahman Reviewed By: sdong Subscribers: dhruba Differential Revision: https://reviews.facebook.net/D44097
-
由 Yueh-Hsuan Chiang 提交于
Summary: HashCuckooRep::ApproximateMemoryUsage() previously return std::numeric_limits<size_t>::max() when it cannot accept more entries. This patch makes it return a more reasonable estimation. This change is necessary in order to make GetIntProperty("rocksdb.cur-size-all-mem-tables") handles HashCuckooRep properly in diff https://reviews.facebook.net/D44229. Test Plan: db_test Reviewers: sdong, anthony, IslamAbdelRahman, igor Reviewed By: igor Subscribers: dhruba Differential Revision: https://reviews.facebook.net/D44241
-
由 Ari Ekmekji 提交于
Summary: In D43239 (https://reviews.facebook.net/D43239) there is an assertion to make sure a subcompaction's output is never empty at the end of execution. This assertion however breaks the build because some tests lead to exactly that scenario. So instead I have altered the logic to handle this case instead of just failing the assertion. The reason that it is possible for a subcompaction's output to be empty is that during a sequential execution of subcompactions, if a user aborts the compaction job then some of the later subcompactions to be executed may have yet to process any keys and therefore have yet to generate output files. This becomes very rare once the subcompactions are executed in parallel, but for now they are still sequential so the case is possible when there is an early termination, as in some of the tests. Test Plan: ./db_test ./db_compaction_test Reviewers: sdong, igor, anthony, yhchiang Reviewed By: yhchiang Subscribers: dhruba Differential Revision: https://reviews.facebook.net/D44877
-
由 Ari Ekmekji 提交于
Summary: In prepration for running multiple threads at the same time during a compaction job, this patch assigns each subcompaction its own state (instead of sharing the one global CompactionState). Each subcompaction then uses this state to update its statistics, keep track of its snapshots, etc. during the course of execution. Then at the end of all the executions the statistics are aggregated across the subcompactions so that the final result is the same as if only one larger compaction had run. Test Plan: ./db_test ./db_compaction_test ./compaction_job_test Reviewers: sdong, anthony, igor, noetzli, yhchiang Reviewed By: yhchiang Subscribers: MarkCallaghan, dhruba, leveldb Differential Revision: https://reviews.facebook.net/D43239
-
- 18 8月, 2015 1 次提交
-
-
由 Andres Notzli 提交于
Summary: While working on supporting mixing merge operators with single deletes ( https://reviews.facebook.net/D43179 ), I realized that returning and dealing with merge results can be made simpler. Submitting this as a separate diff because it is not directly related to single deletes. Before, callers of merge helper had to retrieve the merge result in one of two ways depending on whether the merge was successful or not (success = result of merge was single kTypeValue). For successful merges, the caller could query the resulting key/value pair and for unsuccessful merges, the result could be retrieved in the form of two deques of keys and values. However, with single deletes, a successful merge does not return a single key/value pair (if merge operands are merged with a single delete, we have to generate a value and keep the original single delete around to make sure that we are not accidentially producing a key overwrite). In addition, the two existing call sites of the merge helper were taking the same actions independently from whether the merge was successful or not, so this patch simplifies that. Test Plan: make clean all check Reviewers: rven, sdong, yhchiang, anthony, igor Reviewed By: igor Subscribers: dhruba, leveldb Differential Revision: https://reviews.facebook.net/D43353
-
- 15 8月, 2015 2 次提交
-
-
由 sdong 提交于
Summary: In internal stats, remember read latency histogram, if statistics is enabled. It can be retrieved from DB::GetProperty() with "rocksdb.dbstats" property, if it is enabled. Test Plan: Manually run db_bench and prints out "rocksdb.dbstats" by hand and make sure it prints out as expected Reviewers: igor, IslamAbdelRahman, rven, kradhakrishnan, anthony, yhchiang Reviewed By: yhchiang Subscribers: MarkCallaghan, leveldb, dhruba Differential Revision: https://reviews.facebook.net/D44193
-
由 Nathan Bronson 提交于
Summary: This diff allows a Writer to join the next write batch group without acquiring any locks. Waiting is performed via a per-Writer mutex, so all of the non-leader writers never need to acquire the db mutex. It is now possible to join a write batch group after the leader has been chosen but before the batch has been constructed. This diff doesn't increase parallelism, but reduces synchronization overheads. For some CPU-bound workloads (no WAL, RAM-sized working set) this can substantially reduce contention on the db mutex in a multi-threaded environment. With T=8 N=500000 in a CPU-bound scenario (see the test plan) this is good for a 33% perf win. Not all scenarios see such a win, but none show a loss. This code is slightly faster even for the single-threaded case (about 2% for the CPU-bound scenario below). Test Plan: 1. unit tests 2. COMPILE_WITH_TSAN=1 make check 3. stress high-contention scenarios with db_bench -benchmarks=fillrandom -threads=$T -batch_size=1 -memtablerep=skip_list -value_size=0 --num=$N -level0_slowdown_writes_trigger=9999 -level0_stop_writes_trigger=9999 -disable_auto_compactions --max_write_buffer_number=8 -max_background_flushes=8 --disable_wal --write_buffer_size=160000000 Reviewers: sdong, igor, rven, ljin, yhchiang Subscribers: dhruba Differential Revision: https://reviews.facebook.net/D43887
-
- 14 8月, 2015 4 次提交
-
-
由 sdong 提交于
Summary: Add options.compaction_measure_io_stats to print out / pass to listener accumulated time spent on write calls. Example outputs in info logs: 2015/08/12-16:27:59.463944 7fd428bff700 (Original Log Time 2015/08/12-16:27:59.463922) EVENT_LOG_v1 {"time_micros": 1439422079463897, "job": 6, "event": "compaction_finished", "output_level": 1, "num_output_files": 4, "total_output_size": 6900525, "num_input_records": 111483, "num_output_records": 106877, "file_write_nanos": 15663206, "file_range_sync_nanos": 649588, "file_fsync_nanos": 349614797, "file_prepare_write_nanos": 1505812, "lsm_state": [2, 4, 0, 0, 0, 0, 0]} Add two more counters in iostats_context. Also add a parameter of db_bench. Test Plan: Add a unit test. Also manually verify LOG outputs in db_bench Subscribers: leveldb, dhruba Differential Revision: https://reviews.facebook.net/D44115
-
由 Islam AbdelRahman 提交于
Summary: Change master version to 3.14 Test Plan: simple change Reviewers: sdong, yhchiang, kradhakrishnan, rven, anthony, igor Reviewed By: igor Subscribers: dhruba Differential Revision: https://reviews.facebook.net/D44187
-
由 Igor Canadi 提交于
Add a 'tools' target.
-
由 maurice barnum 提交于
My use case is to build the rocksdb static library and tools, and ideally I'd like to not spend time building the shared library and other targets that I won't use.
-
- 13 8月, 2015 2 次提交
-
-
由 sdong 提交于
Summary: Iterator has a bug: if a child iterator reaches its end, and user issues a Prev(), and just before SeekToLast() of the child iterator is called, some extra rows is added in the end, the position of iterator can be misplaced. Test Plan: Run the tests with or without valgrind Reviewers: rven, yhchiang, IslamAbdelRahman, anthony Reviewed By: anthony Subscribers: tnovak, leveldb, dhruba Differential Revision: https://reviews.facebook.net/D43671
-
由 Islam AbdelRahman 提交于
Summary: RandomRWFile is not used anywhere in out code base, this patch remove RandomRWFile Test Plan: make check -j64 USE_CLANG=1 make all -j64 OPT=-DROCKSDB_LITE make release -j64 Reviewers: sdong, yhchiang, anthony, kradhakrishnan, rven, igor Reviewed By: igor Subscribers: dhruba Differential Revision: https://reviews.facebook.net/D44091
-
- 12 8月, 2015 11 次提交
-
-
由 agiardullo 提交于
Summary: Clean up transactions to use the new RollbackToSavePoint api in WriteBatchWithIndex. Note, this diff depends on Pessimistic Transactions diff and ManagedSnapshot diff (D40869 and D43293). Test Plan: unit tests Reviewers: rven, yhchiang, kradhakrishnan, spetrunia, sdong Reviewed By: sdong Subscribers: dhruba, leveldb Differential Revision: https://reviews.facebook.net/D43371
-
由 agiardullo 提交于
Summary: Based on feedback from spetrunia, we should better differentiate error statuses for transaction failures. https://github.com/MySQLOnRocksDB/mysql-5.6/issues/86#issuecomment-124605954 Test Plan: unit tests Reviewers: rven, kradhakrishnan, spetrunia, yhchiang, sdong Reviewed By: sdong Subscribers: dhruba, leveldb Differential Revision: https://reviews.facebook.net/D43323
-
由 agiardullo 提交于
Summary: Initial implementation of Pessimistic Transactions. This diff contains the api changes discussed in D38913. This diff is pretty large, so let me know if people would prefer to meet up to discuss it. MyRocks folks: please take a look at the API in include/rocksdb/utilities/transaction[_db].h and let me know if you have any issues. Also, you'll notice a couple of TODOs in the implementation of RollbackToSavePoint(). After chatting with Siying, I'm going to send out a separate diff for an alternate implementation of this feature that implements the rollback inside of WriteBatch/WriteBatchWithIndex. We can then decide which route is preferable. Next, I'm planning on doing some perf testing and then integrating this diff into MongoRocks for further testing. Test Plan: Unit tests, db_bench parallel testing. Reviewers: igor, rven, sdong, yhchiang, yoshinorim Reviewed By: sdong Subscribers: hermanlee4, maykov, spetrunia, leveldb, dhruba Differential Revision: https://reviews.facebook.net/D40869
-
由 Islam AbdelRahman 提交于
Summary: Under certain conditions (disable compression) the compactions that are created in compaction_job_test will pass the trivial_move conditions This will cause problems since we assert that we dont run a compaction if it's a trivial move https://github.com/facebook/rocksdb/blob/master/db/compaction_job.cc#L144-L147 for example when we disable compression, compactions become a valid trivial move and the assert fails https://ci-builds.fb.com/view/rocksdb/job/rocksdb_no_compression/180/console Test Plan: compaction_job_test Reviewers: sdong, yhchiang, noetzli, igor Reviewed By: igor Subscribers: dhruba Differential Revision: https://reviews.facebook.net/D43983
-
由 agiardullo 提交于
Test Plan: untested Reviewers: sdong Reviewed By: sdong Subscribers: dhruba, leveldb Differential Revision: https://reviews.facebook.net/D44049
-
由 Yueh-Hsuan Chiang 提交于
Summary: This patch fixes the following clang-build error in util/thread_local.cc by using a cleaner macro blocker: 12:26:31 util/thread_local.cc:157:19: error: declaration shadows a static data member of 'rocksdb::ThreadLocalPtr::StaticMeta' [-Werror,-Wshadow] 12:26:31 ThreadData* tls_ = 12:26:31 ^ 12:26:31 util/thread_local.cc:19:66: note: previous declaration is here 12:26:31 __thread ThreadLocalPtr::ThreadData* ThreadLocalPtr::StaticMeta::tls_ = nullptr; 12:26:31 ^ Test Plan: db_test Reviewers: sdong, anthony, IslamAbdelRahman, igor Reviewed By: igor Subscribers: dhruba Differential Revision: https://reviews.facebook.net/D44043
-
由 Islam AbdelRahman 提交于
Summary: Add a new option that all LoadTableHandlers to use multiple threads to load files on DB Open and Recover Test Plan: make check -j64 COMPILE_WITH_TSAN=1 make check -j64 DISABLE_JEMALLOC=1 make all valgrind_check -j64 (still running) Reviewers: yhchiang, anthony, rven, kradhakrishnan, igor, sdong Reviewed By: sdong Subscribers: dhruba Differential Revision: https://reviews.facebook.net/D43755
-
由 Andres Notzli 提交于
Summary: While working on single delete support for db_bench, I realized that db_bench/db_stress contain a bunch of duplicate code related to copmression and found some typos. This patch removes duplicate code, typos and a redundant #ifndef in internal_stats.cc. Test Plan: make db_stress && make db_bench && ./db_bench --benchmarks=compress,uncompress Reviewers: yhchiang, sdong, rven, anthony, igor Reviewed By: igor Subscribers: dhruba, leveldb Differential Revision: https://reviews.facebook.net/D43965
-
由 Igor Canadi 提交于
Summary: Our linters assume that clang-format is installed at /mnt/vol/engshare/admin/scripts/clang-format and flint is installed at /home/engshare/tools/flint. This makes them fail on non-fb machines. This change will: * if clang-format is not on a specified path, it will try running generic clang-format. Linters will still fail if clang-format is not installed, but this shouldn't be a big issue, since it's pretty easy to install it. * flint will not be run if /home/engshare/tools/flint is not present Test Plan: Made a change on a mac machine. Ran `arc lint`. No failures observed. Reviewers: aekmekji, yhchiang Reviewed By: yhchiang Subscribers: dhruba, leveldb Differential Revision: https://reviews.facebook.net/D44031
-
由 Nathan Bronson 提交于
Summary: Key comparison is the single largest CPU user for CPU-bound workloads. This diff reduces the number of comparisons in two ways. The first is that it moves predecessor array gathering from FindGreaterOrEqual to FindLessThan, so that FindGreaterOrEqual can return immediately if compare_ returns 0. As part of this change I moved the sequential insertion optimization into Insert, to remove the undocumented (and smelly) requirement that prev must be equal to prev_ if it is non-null. The second optimization is that all of the search functions skip calling compare_ when moving to a lower level that has the same Next pointer. With a branching factor of 4 we would expect this to happen 1/4 of the time. On a single-threaded CPU-bound workload (-benchmarks=fillrandom -threads=1 -batch_size=1 -memtablerep=skip_list -value_size=0 --num=1600000 -level0_slowdown_writes_trigger=9999 -level0_stop_writes_trigger=9999 -disable_auto_compactions --max_write_buffer_number=8 -max_background_flushes=8 --disable_wal --write_buffer_size=160000000) on my dev server this is good for a 7% perf win. Test Plan: unit tests Reviewers: rven, ljin, yhchiang, sdong, igor Reviewed By: igor Subscribers: dhruba Differential Revision: https://reviews.facebook.net/D43233
-
由 Yueh-Hsuan Chiang 提交于
Summary: This patch provides a simplier solution to the memory leak issue identified in patch https://reviews.facebook.net/D43677, where a static function local variable can be used instead of using a global static unique_ptr. Test Plan: run db_stress on mac Reviewers: igor, sdong, anthony, IslamAbdelRahman, maykov Reviewed By: maykov Subscribers: dhruba, leveldb Differential Revision: https://reviews.facebook.net/D43995
-
- 11 8月, 2015 3 次提交
-
-
由 Igor Canadi 提交于
Summary: This was motivated by t7518166. checkCpp crashes on db_test.cc because the file is too big :( Couple of changes: * Added clang-format linter. Now we can catch all code that is not formatted correctly. * Added Howtoeven in our list of linters * Replaced cpplint with flint * Removed checkCpp lint. Nobody ownes it and it doesn't work on db_test.cc Test Plan: Made a random lint error and `arc lint`. Saw an error. Reviewers: yhchiang, kradhakrishnan, anthony, sdong Reviewed By: sdong Subscribers: dhruba, leveldb Differential Revision: https://reviews.facebook.net/D41949
-
由 Islam AbdelRahman 提交于
Summary: https://ci-builds.fb.com/view/rocksdb/job/rocksdb_clang_build/893/console Fixing clang build Test Plan: make clean USE_CLANG=1 make all -j64 Reviewers: sdong, noetzli, yhchiang, igor Reviewed By: igor Subscribers: dhruba Differential Revision: https://reviews.facebook.net/D43959
-
由 Yoshinori Matsunobu 提交于
Summary: wal_recovery_mode setting was not written to LOG. This diff adds the log message Test Plan: manually checked Reviewers: kradhakrishnan, sdong, igor Reviewed By: igor Subscribers: dhruba Differential Revision: https://reviews.facebook.net/D43953
-
- 08 8月, 2015 5 次提交
-
-
由 Andres Notzli 提交于
Summary: Changed compaction_job_test to support better/more thorough tests and added two tests. Also changed MockFileContents to order using InternalKeyComparator. Test Plan: make compaction_job_test && ./compaction_job_test; make all && make check Reviewers: sdong, rven, igor, yhchiang, anthony Reviewed By: anthony Subscribers: dhruba, leveldb Differential Revision: https://reviews.facebook.net/D42837
-
由 Islam AbdelRahman 提交于
Summary: Start measuring time spent before BackgroundEmptyTrash starts Test Plan: delete_scheduler_test Reviewers: yhchiang, sdong Reviewed By: sdong Subscribers: dhruba Differential Revision: https://reviews.facebook.net/D43857
-
由 sdong 提交于
Summary: Check in a simple auto-build manfiest so that developers can issue ad-hoc build for Windows in Appveyor. Test Plan: Run it in test branch test_appveyor and it works well. Reviewers: kradhakrishnan, rven, anthony, IslamAbdelRahman, yhchiang Reviewed By: yhchiang Subscribers: leveldb, dhruba Differential Revision: https://reviews.facebook.net/D43839
-
由 Siying Dong 提交于
Fix OSX + Windows build
-
由 Nate Rosenblum 提交于
Commit 257ee895 added a static destruction helper to avoid notional "leaks" of TLS on main thread exit. This helper fails to compile on OS X (and presumably Windows, though I haven't checked), which lacks the __thread storage class StaticMeta::tls_ member. This patch fixes the builds. Do note that the static cleanup mechanism may be somewhat brittle and atexit(3) may be a more suitable approach to releasing the main thread's TLS if it's highly desirable for this memory to not be reported "reachable" by Valgrind at exit.
-
- 07 8月, 2015 6 次提交
-
-
由 agiardullo 提交于
Summary: Implemented this simple wrapper for something else I was working on. Seemed like it makes sense to expose it instead of burying it in some random code. Test Plan: added test Reviewers: rven, kradhakrishnan, sdong, yhchiang Reviewed By: sdong Subscribers: dhruba, leveldb Differential Revision: https://reviews.facebook.net/D43293
-
由 Alexey Maykov 提交于
Summary: MyRocks valgrind run was showing memory leaks. The fixes are mostly self-explaining. There is only a single usage of ThreadLocalPtr. Potentially, we may think about replacing this use with thread_local, but it will be a bigger change. Another option to consider is using thread_local instead of __thread in ThreadLocalPtr implementation. This way, tls_ can be stored using std::unique_ptr and no destructor would be required. Test Plan: - make check - MyRocks valgrind run doesn't report leaks Reviewers: rven, sdong Reviewed By: sdong Subscribers: dhruba Differential Revision: https://reviews.facebook.net/D43677
-
由 sdong 提交于
Summary: Prepare release notes for 3.13. Test Plan: Text only Reviewers: yhchiang, IslamAbdelRahman, rven, anthony Reviewed By: anthony Subscribers: leveldb, dhruba Differential Revision: https://reviews.facebook.net/D43665
-
由 Islam AbdelRahman 提交于
Summary: Use shared_ptr instead of deleting in destructor Test Plan: DISABLE_JEMALLOC=1 make delete_scheduler_test -j64 && valgrind --error-exitcode=2 --leak-check=full ./delete_scheduler_test Reviewers: yhchiang, sdong Reviewed By: sdong Subscribers: dhruba Differential Revision: https://reviews.facebook.net/D43659
-
由 sdong 提交于
Summary: Visual Studio complains about deque<LogWriterNumber> because LogWriterNumber is non-copyable for its unique_ptr member writer. Move away from it, and do explit free. It is less safe but I can't think of a better way to unblock it. Test Plan: valgrind check test Reviewers: anthony, IslamAbdelRahman, kolmike, rven, yhchiang Reviewed By: yhchiang Subscribers: leveldb, dhruba Differential Revision: https://reviews.facebook.net/D43647
-
由 Andres Noetzli 提交于
Summary: When seeking to the last occurrence of a key with sequence number 0, db_iter ends up in an endless loop because it seeks to type kValueTypeForSeek which is larger than kTypeDeletion/kTypeValue. Added test case that triggers the behavior. Test Plan: make clean all check Reviewers: igor, rven, anthony, yhchiang, sdong Reviewed By: sdong Subscribers: dhruba, leveldb Differential Revision: https://reviews.facebook.net/D43653
-
- 06 8月, 2015 2 次提交
-
-
由 sdong 提交于
Summary: util/delete_scheduler_impl.cc is missing CMakeLists.txt, which fails Windows build. Add it. Test Plan: Build in both of Windows and Linux Reviewers: anthony, rven, yhchiang, IslamAbdelRahman Reviewed By: IslamAbdelRahman Subscribers: leveldb, dhruba Differential Revision: https://reviews.facebook.net/D43641
-
由 sdong 提交于
Summary: Provide a way for users to know more detailed ditribution of a histogram metrics. Example outputs: Manually add statement fprintf(stdout, "%s\n", dbstats->getHistogramString(SST_READ_MICROS).c_str()); Will print out something like: Count: 989151 Average: 1.7659 StdDev: 1.52 Min: 0.0000 Median: 1.2071 Max: 860.0000 Percentiles: P50: 1.21 P75: 1.70 P99: 5.12 P99.9: 13.67 P99.99: 21.70 ------------------------------------------------------ [ 0, 1 ) 390839 39.513% 39.513% ######## [ 1, 2 ) 500918 50.641% 90.154% ########## [ 2, 3 ) 79358 8.023% 98.177% ## [ 3, 4 ) 6297 0.637% 98.813% [ 4, 5 ) 1712 0.173% 98.986% [ 5, 6 ) 1134 0.115% 99.101% [ 6, 7 ) 1222 0.124% 99.224% [ 7, 8 ) 1529 0.155% 99.379% [ 8, 9 ) 1264 0.128% 99.507% [ 9, 10 ) 988 0.100% 99.607% [ 10, 12 ) 1378 0.139% 99.746% [ 12, 14 ) 1828 0.185% 99.931% [ 14, 16 ) 410 0.041% 99.972% [ 16, 18 ) 72 0.007% 99.980% [ 18, 20 ) 67 0.007% 99.986% [ 20, 25 ) 106 0.011% 99.997% [ 25, 30 ) 24 0.002% 99.999% [ 30, 35 ) 1 0.000% 100.000% [ 250, 300 ) 2 0.000% 100.000% [ 300, 350 ) 1 0.000% 100.000% [ 800, 900 ) 1 0.000% 100.000% Test Plan: Manually add a print in db_bench and make sure it prints out as expected. Will add some codes to cover the function Subscribers: leveldb, dhruba Differential Revision: https://reviews.facebook.net/D43611
-