- 12 8月, 2015 5 次提交
-
-
由 agiardullo 提交于
Summary: Based on feedback from spetrunia, we should better differentiate error statuses for transaction failures. https://github.com/MySQLOnRocksDB/mysql-5.6/issues/86#issuecomment-124605954 Test Plan: unit tests Reviewers: rven, kradhakrishnan, spetrunia, yhchiang, sdong Reviewed By: sdong Subscribers: dhruba, leveldb Differential Revision: https://reviews.facebook.net/D43323
-
由 agiardullo 提交于
Summary: Initial implementation of Pessimistic Transactions. This diff contains the api changes discussed in D38913. This diff is pretty large, so let me know if people would prefer to meet up to discuss it. MyRocks folks: please take a look at the API in include/rocksdb/utilities/transaction[_db].h and let me know if you have any issues. Also, you'll notice a couple of TODOs in the implementation of RollbackToSavePoint(). After chatting with Siying, I'm going to send out a separate diff for an alternate implementation of this feature that implements the rollback inside of WriteBatch/WriteBatchWithIndex. We can then decide which route is preferable. Next, I'm planning on doing some perf testing and then integrating this diff into MongoRocks for further testing. Test Plan: Unit tests, db_bench parallel testing. Reviewers: igor, rven, sdong, yhchiang, yoshinorim Reviewed By: sdong Subscribers: hermanlee4, maykov, spetrunia, leveldb, dhruba Differential Revision: https://reviews.facebook.net/D40869
-
由 Yueh-Hsuan Chiang 提交于
Summary: This patch fixes the following clang-build error in util/thread_local.cc by using a cleaner macro blocker: 12:26:31 util/thread_local.cc:157:19: error: declaration shadows a static data member of 'rocksdb::ThreadLocalPtr::StaticMeta' [-Werror,-Wshadow] 12:26:31 ThreadData* tls_ = 12:26:31 ^ 12:26:31 util/thread_local.cc:19:66: note: previous declaration is here 12:26:31 __thread ThreadLocalPtr::ThreadData* ThreadLocalPtr::StaticMeta::tls_ = nullptr; 12:26:31 ^ Test Plan: db_test Reviewers: sdong, anthony, IslamAbdelRahman, igor Reviewed By: igor Subscribers: dhruba Differential Revision: https://reviews.facebook.net/D44043
-
由 Islam AbdelRahman 提交于
Summary: Add a new option that all LoadTableHandlers to use multiple threads to load files on DB Open and Recover Test Plan: make check -j64 COMPILE_WITH_TSAN=1 make check -j64 DISABLE_JEMALLOC=1 make all valgrind_check -j64 (still running) Reviewers: yhchiang, anthony, rven, kradhakrishnan, igor, sdong Reviewed By: sdong Subscribers: dhruba Differential Revision: https://reviews.facebook.net/D43755
-
由 Yueh-Hsuan Chiang 提交于
Summary: This patch provides a simplier solution to the memory leak issue identified in patch https://reviews.facebook.net/D43677, where a static function local variable can be used instead of using a global static unique_ptr. Test Plan: run db_stress on mac Reviewers: igor, sdong, anthony, IslamAbdelRahman, maykov Reviewed By: maykov Subscribers: dhruba, leveldb Differential Revision: https://reviews.facebook.net/D43995
-
- 11 8月, 2015 1 次提交
-
-
由 Yoshinori Matsunobu 提交于
Summary: wal_recovery_mode setting was not written to LOG. This diff adds the log message Test Plan: manually checked Reviewers: kradhakrishnan, sdong, igor Reviewed By: igor Subscribers: dhruba Differential Revision: https://reviews.facebook.net/D43953
-
- 08 8月, 2015 2 次提交
-
-
由 Islam AbdelRahman 提交于
Summary: Start measuring time spent before BackgroundEmptyTrash starts Test Plan: delete_scheduler_test Reviewers: yhchiang, sdong Reviewed By: sdong Subscribers: dhruba Differential Revision: https://reviews.facebook.net/D43857
-
由 Nate Rosenblum 提交于
Commit 257ee895 added a static destruction helper to avoid notional "leaks" of TLS on main thread exit. This helper fails to compile on OS X (and presumably Windows, though I haven't checked), which lacks the __thread storage class StaticMeta::tls_ member. This patch fixes the builds. Do note that the static cleanup mechanism may be somewhat brittle and atexit(3) may be a more suitable approach to releasing the main thread's TLS if it's highly desirable for this memory to not be reported "reachable" by Valgrind at exit.
-
- 07 8月, 2015 2 次提交
-
-
由 Alexey Maykov 提交于
Summary: MyRocks valgrind run was showing memory leaks. The fixes are mostly self-explaining. There is only a single usage of ThreadLocalPtr. Potentially, we may think about replacing this use with thread_local, but it will be a bigger change. Another option to consider is using thread_local instead of __thread in ThreadLocalPtr implementation. This way, tls_ can be stored using std::unique_ptr and no destructor would be required. Test Plan: - make check - MyRocks valgrind run doesn't report leaks Reviewers: rven, sdong Reviewed By: sdong Subscribers: dhruba Differential Revision: https://reviews.facebook.net/D43677
-
由 Islam AbdelRahman 提交于
Summary: Use shared_ptr instead of deleting in destructor Test Plan: DISABLE_JEMALLOC=1 make delete_scheduler_test -j64 && valgrind --error-exitcode=2 --leak-check=full ./delete_scheduler_test Reviewers: yhchiang, sdong Reviewed By: sdong Subscribers: dhruba Differential Revision: https://reviews.facebook.net/D43659
-
- 06 8月, 2015 5 次提交
-
-
由 sdong 提交于
Summary: Provide a way for users to know more detailed ditribution of a histogram metrics. Example outputs: Manually add statement fprintf(stdout, "%s\n", dbstats->getHistogramString(SST_READ_MICROS).c_str()); Will print out something like: Count: 989151 Average: 1.7659 StdDev: 1.52 Min: 0.0000 Median: 1.2071 Max: 860.0000 Percentiles: P50: 1.21 P75: 1.70 P99: 5.12 P99.9: 13.67 P99.99: 21.70 ------------------------------------------------------ [ 0, 1 ) 390839 39.513% 39.513% ######## [ 1, 2 ) 500918 50.641% 90.154% ########## [ 2, 3 ) 79358 8.023% 98.177% ## [ 3, 4 ) 6297 0.637% 98.813% [ 4, 5 ) 1712 0.173% 98.986% [ 5, 6 ) 1134 0.115% 99.101% [ 6, 7 ) 1222 0.124% 99.224% [ 7, 8 ) 1529 0.155% 99.379% [ 8, 9 ) 1264 0.128% 99.507% [ 9, 10 ) 988 0.100% 99.607% [ 10, 12 ) 1378 0.139% 99.746% [ 12, 14 ) 1828 0.185% 99.931% [ 14, 16 ) 410 0.041% 99.972% [ 16, 18 ) 72 0.007% 99.980% [ 18, 20 ) 67 0.007% 99.986% [ 20, 25 ) 106 0.011% 99.997% [ 25, 30 ) 24 0.002% 99.999% [ 30, 35 ) 1 0.000% 100.000% [ 250, 300 ) 2 0.000% 100.000% [ 300, 350 ) 1 0.000% 100.000% [ 800, 900 ) 1 0.000% 100.000% Test Plan: Manually add a print in db_bench and make sure it prints out as expected. Will add some codes to cover the function Subscribers: leveldb, dhruba Differential Revision: https://reviews.facebook.net/D43611
-
由 Islam AbdelRahman 提交于
Summary: Update DeleteScheduler tests so that they verify the used penalties for waiting instead of measuring the time spent which is not reliable Test Plan: make -j64 delete_scheduler_test && ./delete_scheduler_test COMPILE_WITH_TSAN=1 make -j64 delete_scheduler_test && ./delete_scheduler_test COMPILE_WITH_ASAN=1 make -j64 delete_scheduler_test && ./delete_scheduler_test make -j64 db_test && ./db_test --gtest_filter="DBTest.RateLimitedDelete:DBTest.DeleteSchedulerMultipleDBPaths" COMPILE_WITH_TSAN=1 make -j64 db_test && ./db_test --gtest_filter="DBTest.RateLimitedDelete:DBTest.DeleteSchedulerMultipleDBPaths" COMPILE_WITH_ASAN=1 make -j64 db_test && ./db_test --gtest_filter="DBTest.RateLimitedDelete:DBTest.DeleteSchedulerMultipleDBPaths" Reviewers: yhchiang, sdong Reviewed By: sdong Subscribers: dhruba Differential Revision: https://reviews.facebook.net/D43635
-
由 sdong 提交于
Summary: Add two unit tests for SyncWAL(). One makes sure SyncWAL() doesn't block writes in the other thread. Another one makes sure SyncWAL() doesn't wait ongoing writes to finish before being executed. Create a new test file db_wal_test and move two WAL related tests from db_test to here. Test Plan: Run the new tests Reviewers: IslamAbdelRahman, rven, kradhakrishnan, kolmike, tnovak, yhchiang Reviewed By: yhchiang Subscribers: leveldb, dhruba Differential Revision: https://reviews.facebook.net/D43605
-
由 sdong 提交于
Summary: Measure read latency histogram and put in statistics. Compaction inputs are excluded from it when possible (unfortunately usually no possible as we usually take table reader from table cache. Test Plan: Run db_bench and it shows the stats, like: rocksdb.sst.read.micros statistics Percentiles :=> 50 : 1.238522 95 : 2.529740 99 : 3.912180 Reviewers: kradhakrishnan, rven, anthony, IslamAbdelRahman, MarkCallaghan, yhchiang Reviewed By: yhchiang Subscribers: leveldb, dhruba Differential Revision: https://reviews.facebook.net/D43275
-
由 Islam AbdelRahman 提交于
Summary: Fixing TSAN false positive and relaxing the conditions when we are running under TSAN Test Plan: COMPILE_WITH_TSAN=1 make -j64 delete_scheduler_test && ./delete_scheduler_test Reviewers: yhchiang, sdong Reviewed By: sdong Subscribers: dhruba Differential Revision: https://reviews.facebook.net/D43593
-
- 05 8月, 2015 7 次提交
-
-
由 Andres Notzli 提交于
Summary: While working on https://reviews.facebook.net/D43179 , I found duplicate code in the tests. This patch removes it. Test Plan: make clean all check Reviewers: igor, sdong, rven, anthony, yhchiang Reviewed By: yhchiang Subscribers: dhruba, leveldb Differential Revision: https://reviews.facebook.net/D43263
-
由 Mike Kolupaev 提交于
Summary: Subj. We really need this feature. Previous diff D40899 has most of the changes to make this possible, this diff just adds the method. Test Plan: `make check`, the new test fails without this diff; ran with ASAN, TSAN and valgrind. Reviewers: igor, rven, IslamAbdelRahman, anthony, kradhakrishnan, tnovak, yhchiang, sdong Reviewed By: sdong Subscribers: MarkCallaghan, maykov, hermanlee4, yoshinorim, tnovak, dhruba Differential Revision: https://reviews.facebook.net/D40905
-
由 Ari Ekmekji 提交于
Summary: Updated DBTest DBCompactionTest and CompactionJobStatsTest to run compaction-related tests once with subcompactions enabled and once disabled using the TEST_P test type in the Google Test suite. Test Plan: ./db_test ./db_compaction-test ./compaction_job_stats_test Reviewers: sdong, igor, anthony, yhchiang Reviewed By: yhchiang Subscribers: dhruba, leveldb Differential Revision: https://reviews.facebook.net/D43443
-
由 Islam AbdelRahman 提交于
Summary: Introduce DeleteScheduler that allow enforcing a rate limit on file deletion Instead of deleting files immediately, files are moved to trash directory and deleted in a background thread that apply sleep penalty between deletes if needed. I have updated PurgeObsoleteFiles and PurgeObsoleteWALFiles to use the delete_scheduler instead of env_->DeleteFile Test Plan: added delete_scheduler_test existing unit tests Reviewers: kradhakrishnan, anthony, rven, yhchiang, sdong Reviewed By: sdong Subscribers: dhruba Differential Revision: https://reviews.facebook.net/D43221
-
由 Yueh-Hsuan Chiang 提交于
Summary: UpdateAccumulatedStats() is used to optimize compaction decision esp. when the number of deletion entries are high, but this function can slowdown DBOpen esp. in disk environment. This patch adds DBOptions::skip_sats_update_on_db_open, which skips UpdateAccumulatedStats() in DB::Open() time when it's set to true. Test Plan: Add DBCompactionTest.SkipStatsUpdateTest Reviewers: igor, anthony, IslamAbdelRahman, sdong Reviewed By: sdong Subscribers: tnovak, dhruba, leveldb Differential Revision: https://reviews.facebook.net/D42843
-
由 Boyang Zhang 提交于
Summary: So I took a look and I used a pointer to TableBuilder. Changed it to a unique_ptr. I think this should work, but I cannot run valgrind correctly on my local machine to test it. Test Plan: Run valgrind, but it's not working locally. It says I'm executing an unrecognized instruction. Reviewers: yhchiang Subscribers: dhruba, sdong Differential Revision: https://reviews.facebook.net/D43485
-
由 sdong 提交于
Summary: In "sst_dump --show_compression_sizes", a reference of CompressionOptions is kept in TableBuilderOptions, which is destroyed later, causing a memory issue. Test Plan: Run valgrind against SSTDumpToolTest.CompressedSizes and make sure it is fixed Reviewers: IslamAbdelRahman, yhchiang, kradhakrishnan, rven Reviewed By: rven Subscribers: leveldb, dhruba Differential Revision: https://reviews.facebook.net/D43497
-
- 04 8月, 2015 2 次提交
-
-
由 Ari Ekmekji 提交于
Summary: As of now compactions involving files from Level 0 and Level 1 are single threaded because the files in L0, although sorted, are not range partitioned like the other levels. This means that during L0-L1 compaction each file from L1 needs to be merged with potentially all the files from L0. This attempt to parallelize the L0-L1 compaction assigns a thread and a corresponding iterator to each L1 file that then considers only the key range found in that L1 file and only the L0 files that have those keys (and only the specific portion of those L0 files in which those keys are found). In this way the overlap is minimized and potentially eliminated between different iterators focusing on the same files. The first step is to restructure the compaction logic to break L0-L1 compactions into multiple, smaller, sequential compactions. Eventually each of these smaller jobs will be run simultaneously. Areas to pay extra attention to are # Correct aggregation of compaction job statistics across multiple threads # Proper opening/closing of output files (make sure each thread's is unique) # Keys that span multiple L1 files # Skewed distributions of keys within L0 files Test Plan: Make and run db_test (newer version has separate compaction tests) and compaction_job_stats_test Reviewers: igor, noetzli, anthony, sdong, yhchiang Reviewed By: yhchiang Subscribers: MarkCallaghan, dhruba, leveldb Differential Revision: https://reviews.facebook.net/D42699
-
由 sdong 提交于
Summary: Now ldb dump_manifest refuses to work if there are 20 levels. Extend the limit to 64. Test Plan: Run the tool with 20 number of levels Reviewers: kradhakrishnan, anthony, IslamAbdelRahman, yhchiang Reviewed By: yhchiang Subscribers: leveldb, dhruba Differential Revision: https://reviews.facebook.net/D42879
-
- 31 7月, 2015 1 次提交
-
-
由 Andres Noetzli 提交于
Summary: sst_dump_tool contains two instances of `fprintf`s where the `format` argument is not a string literal. This prevents the code from compiling with some compilers/compiler options because of the potential security risks associated with printing non-literals. Test Plan: make all Reviewers: rven, igor, yhchiang, sdong, anthony Reviewed By: anthony Subscribers: dhruba, leveldb Differential Revision: https://reviews.facebook.net/D43305
-
- 30 7月, 2015 1 次提交
-
-
由 Boyang Zhang 提交于
Summary: Added a new feature to sst_dump_tool.cc to allow a user to see the sizes of the different compression algorithms on an .sst file. Usage: ./sst_dump --file=<filename> --show_compression_sizes ./sst_dump --file=<filename> --show_compression_sizes --set_block_size=<block_size> Note: If you do not set a block size, it will default to 16kb Test Plan: manual test and the write a unit test Reviewers: IslamAbdelRahman, anthony, yhchiang, rven, kradhakrishnan, sdong Reviewed By: sdong Subscribers: dhruba Differential Revision: https://reviews.facebook.net/D42963
-
- 29 7月, 2015 1 次提交
-
-
由 Andres Notzli 提交于
Summary: For task #7771355, we would like to log the number of corrupt keys during a compaction. This patch implements and tests the count as part of CompactionJobStats. Test Plan: make && make check Reviewers: rven, igor, yhchiang, sdong Reviewed By: sdong Subscribers: dhruba, leveldb Differential Revision: https://reviews.facebook.net/D42921
-
- 28 7月, 2015 1 次提交
-
-
Summary: Fix for universal compaction with trivial move, when the ouput level is 0. The tests where failing. Fixed by allowing normal compaction when output level is 0. Test Plan: modified test cases run successfully. Reviewers: sdong, yhchiang, IslamAbdelRahman Reviewed By: IslamAbdelRahman Subscribers: anthony, kradhakrishnan, leveldb, dhruba Differential Revision: https://reviews.facebook.net/D42933
-
- 23 7月, 2015 2 次提交
-
-
由 Dmitri Smirnov 提交于
* std::chrono does not provide enough granularity for microsecs and periodically emits duplicates * the bug is manifested in log rotation logic where we get duplicate log file names and loose previous log content * msvc does not imlement COW on std::strings adjusted the test to use refs in the loops as auto does not retain ref info * adjust auto_log rotation test with Windows specific command to remove a folder. The test previously worked because we have unix utils installed in house but this may not be the case for everyone.
-
由 Mike Kolupaev 提交于
Summary: https://reviews.facebook.net/D42321 has left PosixMmapFile in some weird state. This diff removes pending_sync_ that was now unused, fixes indentation and prevents Fsync() from calling both fsync() and fdatasync(). Test Plan: `make -j check` Reviewers: sdong Reviewed By: sdong Subscribers: dhruba Differential Revision: https://reviews.facebook.net/D42885
-
- 22 7月, 2015 2 次提交
-
-
由 sdong 提交于
Summary: Directly using TMPDIR can cause problems when running tests using parallel option. Fix them. Test Plan: Run all tests in parallel Reviewers: kradhakrishnan, yhchiang, IslamAbdelRahman, anthony Reviewed By: anthony Subscribers: leveldb, dhruba Differential Revision: https://reviews.facebook.net/D42807
-
由 Siying Dong 提交于
Summary: From other ones' investigation: "sync_file_range() behavior highly depends on kernel version and filesystem. xfs does neighbor page flushing outside of the specified ranges. For example, sync_file_range(fd, 8192, 16384) does not only trigger flushing page #3 to #4, but also flushing many more dirty pages (i.e. up to page#16)... Ranges of the sync_file_range() should be far enough from write() offset (at least 1MB)." Test Plan: make all check Reviewers: igor, rven, kradhakrishnan, yhchiang, IslamAbdelRahman, anthony Reviewed By: anthony Subscribers: yoshinorim, MarkCallaghan, sumeet, domas, dhruba, leveldb, ljin Differential Revision: https://reviews.facebook.net/D15807
-
- 21 7月, 2015 4 次提交
-
-
由 agiardullo 提交于
Summary: Add new CheckFileExists method. Considered changing the FileExists api but didn't want to break anyone's builds. Test Plan: unit tests Reviewers: yhchiang, igor, sdong Reviewed By: sdong Subscribers: dhruba, leveldb Differential Revision: https://reviews.facebook.net/D42003
-
由 Islam AbdelRahman 提交于
Summary: Make memenv_test runnable in ROCKSDB_LITE Test Plan: memenv_test Reviewers: sdong, igor, yhchiang Reviewed By: yhchiang Subscribers: dhruba Differential Revision: https://reviews.facebook.net/D42123
-
由 Islam AbdelRahman 提交于
Summary: Skipping these tests in ROCKSDB_LITE since they are not supported json_document_test wal_manager_test ttl_test sst_dump_test deletefile_test compact_files_test prefix_test checkpoint_test Test Plan: json_document_test wal_manager_test ttl_test sst_dump_test deletefile_test compact_files_test prefix_test checkpoint_test Reviewers: igor, sdong, yhchiang, kradhakrishnan, anthony Reviewed By: anthony Subscribers: dhruba Differential Revision: https://reviews.facebook.net/D42573
-
由 Islam AbdelRahman 提交于
Summary: Make mock_env_test runnable in ROCKSDB_LITE Test Plan: mock_env_test Reviewers: igor, sdong, yhchiang, kradhakrishnan, anthony Reviewed By: anthony Subscribers: dhruba Differential Revision: https://reviews.facebook.net/D42585
-
- 18 7月, 2015 4 次提交
-
-
由 sdong 提交于
Move rate_limiter, write buffering, most perf context instrumentation and most random kill out of Env Summary: We want to keep Env a think layer for better portability. Less platform dependent codes should be moved out of Env. In this patch, I create a wrapper of file readers and writers, and put rate limiting, write buffering, as well as most perf context instrumentation and random kill out of Env. It will make it easier to maintain multiple Env in the future. Test Plan: Run all existing unit tests. Reviewers: anthony, kradhakrishnan, IslamAbdelRahman, yhchiang, igor Reviewed By: igor Subscribers: leveldb, dhruba Differential Revision: https://reviews.facebook.net/D42321
-
由 sdong 提交于
Summary: CYGWIN avoided fread_unlocked in a wrong way. Fix it to the standard way. Test Plan: Run tests Reviewers: anthony, kradhakrishnan, IslamAbdelRahman, yhchiang, igor Reviewed By: igor Subscribers: leveldb, dhruba Differential Revision: https://reviews.facebook.net/D42549
-
由 Igor Canadi 提交于
Summary: When we first started, max_background_flushes was 0 by default and compaction thread was executing flushes (since there was no flush thread). Then, we switched the default max_background_flushes to 1. However, we still support the case where there is no flush thread and flushes are done in compaction. This is making our code a bit more complicated. By not supporting this use-case we can make our code simpler. We have a special case that when you set max_background_flushes to 0, we schedule the flush to execute on the compaction thread. Test Plan: make check (there might be some unit tests that depend on this behavior) Reviewers: IslamAbdelRahman, yhchiang, sdong Reviewed By: sdong Subscribers: dhruba, leveldb Differential Revision: https://reviews.facebook.net/D41931
-
由 agiardullo 提交于
Summary: ROCKSDB_WARNING is only defined if either ROCKSDB_PLATFORM_POSIX or OS_WIN is defined. This works well for building rocksdb with its own build scripts. But this won't work when an outside project(like mongodb) doesn't define ROCKSDB_PLATFORM_POSIX. This fix defines ROCKSDB_WARNING for all platforms. No idea if its defined correctly on non-posix,non-windows platforms but this is no worse that the current situation where this macro is missing on unexpected platforms. This fix should hopefully fix anyone whose build broke now that we've switched from using #warning to Pragma (to support windows). Unfortunately, while mongo-rocks compiles, it ignores the Pragma and doesn't print a warning. I have not been able to figure out a way to implement this portably on all platforms. Of course, an alternate solution would be to just get rid of ROCKSDB_WARNING and live with include file redirects indefinitely. Thoughts? Test Plan: build rocks, build mongorocks Reviewers: igor, kradhakrishnan, IslamAbdelRahman, sdong Reviewed By: sdong Subscribers: dhruba, leveldb Differential Revision: https://reviews.facebook.net/D42477
-