- 08 7月, 2015 1 次提交
-
-
由 Aaron Feldman 提交于
Summary: This addresses a test failure where an exception occured in the constructor's call to CreateDirIfMissing(). The existence of unjoined threads prevented this exception from propogating properly. See http://stackoverflow.com/questions/7381757/c-terminate-called-without-an-active-exception Test Plan: Re-run tests from task #7626266 Reviewers: sdong, anthony, igor Reviewed By: igor Subscribers: dhruba Differential Revision: https://reviews.facebook.net/D41313
-
- 07 7月, 2015 2 次提交
-
-
由 Andres Notzli 提交于
Summary: This patch adds three test cases for ExpandWhileOverlapping() to the compaction_picker_test test suite. ExpandWhileOverlapping() only has an effect if the comparison function for the internal keys allows for overlapping user keys in different SST files on the same level. Thus, this patch adds a comparator based on sequence numbers to compaction_picker_test for the new test cases. Test Plan: - make compaction_picker_test && ./compaction_picker_test -> All tests pass - Replace body of ExpandWhileOverlapping() with `return true` -> Compile and run ./compaction_picker_test as before -> New tests fail Reviewers: sdong, yhchiang, rven, anthony, IslamAbdelRahman, kradhakrishnan, igor Reviewed By: igor Subscribers: dhruba, leveldb Differential Revision: https://reviews.facebook.net/D41277
-
由 Igor Canadi 提交于
Summary: Two issues: * the input keys to the compaction don't include sequence number. * sequence number is set to max(seq_num), but it should be set to max(seq_num)+1, because the condition here is strictly-larger (i.e. we will only zero-out sequence number if the DB's sequence number is strictly greater than the key's sequence number): https://github.com/facebook/rocksdb/blob/master/db/compaction_job.cc#L830 Test Plan: make compaction_job_test && ./compaction_job_test Reviewers: sdong, lovro Reviewed By: lovro Subscribers: dhruba, leveldb Differential Revision: https://reviews.facebook.net/D41247
-
- 06 7月, 2015 1 次提交
-
-
由 lovro 提交于
Summary: While profiling compaction in our service I noticed a lot of CPU (~15% of compaction) being spent in MergingIterator and key comparison. Looking at the code I found MergingIterator was (understandably) using std::priority_queue for the multiway merge. Keys in our dataset include sequence numbers that increase with time. Adjacent keys in an L0 file are very likely to be adjacent in the full database. Consequently, compaction will often pick a chunk of rows from the same L0 file before switching to another one. It would be great to avoid the O(log K) operation per row while compacting. This diff replaces std::priority_queue with a custom binary heap implementation. It has a "replace top" operation that is cheap when the new top is the same as the old one (i.e. the priority of the top entry is decreased but it still stays on top). Test Plan: make check To test the effect on performance, I generated databases with data patterns that mimic what I describe in the summary (rows have a mostly increasing sequence number). I see a 10-15% CPU decrease for compaction (and a matching throughput improvement on tmpfs). The exact improvement depends on the number of L0 files and the amount of locality. Performance on randomly distributed keys seems on par with the old code. Reviewers: kailiu, sdong, igor Reviewed By: igor Subscribers: yoshinorim, dhruba, tnovak Differential Revision: https://reviews.facebook.net/D29133
-
- 03 7月, 2015 6 次提交
-
-
由 Ari Ekmekji 提交于
Summary: Introduced a new category in the enum InfoLogLevel in env.h. Modifed Log() in env.cc to use the Header() when the InfoLogLevel == HEADER_LEVEL. Updated tests in auto_roll_logger_test to ensure the header is handled properly in these cases. Test Plan: Augment existing tests in auto_roll_logger_test Reviewers: igor, sdong, yhchiang Reviewed By: yhchiang Subscribers: dhruba, leveldb Differential Revision: https://reviews.facebook.net/D41067
-
由 Yueh-Hsuan Chiang 提交于
Summary: Fixed endless loop in DBIter::FindPrevUserKey() Test Plan: ./db_stress --test_batches_snapshots=1 --threads=32 --write_buffer_size=4194304 --destroy_db_initially=0 --reopen=20 --readpercent=45 --prefixpercent=5 --writepercent=35 --delpercent=5 --iterpercent=10 --db=/tmp/rocksdb_crashtest_KdCI5F --max_key=100000000 --mmap_read=0 --block_size=16384 --cache_size=1048576 --open_files=500000 --verify_checksum=1 --sync=0 --progress_reports=0 --disable_wal=0 --disable_data_sync=1 --target_file_size_base=2097152 --target_file_size_multiplier=2 --max_write_buffer_number=3 --max_background_compactions=20 --max_bytes_for_level_base=10485760 --filter_deletes=0 --memtablerep=prefix_hash --prefix_size=7 --ops_per_thread=200 --kill_random_test=97 Reviewers: tnovak, igor, sdong Reviewed By: sdong Subscribers: dhruba, leveldb Differential Revision: https://reviews.facebook.net/D41085
-
由 Mike Kolupaev 提交于
Summary: This fixes the following scenario we've hit: - we reached max_total_wal_size, created a new wal and scheduled flushing all memtables corresponding to the old one, - before the last of these flushes started its column family was dropped; the last background flush call was a no-op; no one removed the old wal from alive_logs_, - hours have passed and no flushes happened even though lots of data was written; data is written to different column families, compactions are disabled; old column families are dropped before memtable grows big enough to trigger a flush; the old wal still sits in alive_logs_ preventing max_total_wal_size limit from kicking in, - a few more hours pass and we run out disk space because of one huge .log file. Test Plan: `make check`; backported the new test, checked that it fails without this diff Reviewers: igor Reviewed By: igor Subscribers: dhruba Differential Revision: https://reviews.facebook.net/D40893
-
由 Aaron Feldman 提交于
Summary: see title Test Plan: run 'make unity' Reviewers: igor Reviewed By: igor Subscribers: dhruba Differential Revision: https://reviews.facebook.net/D41079
-
由 agiardullo 提交于
Summary: About to cut release Test Plan: none Reviewers: igor Reviewed By: igor Subscribers: dhruba, leveldb Differential Revision: https://reviews.facebook.net/D41061
-
由 Aaron Feldman 提交于
Summary: Add a new field: BackupableDBOptions.max_background_copies. CreateNewBackup() and RestoreDBFromBackup() will use this number of threads to perform copies. If there is a backup rate limit, then max_background_copies must be 1. Update backupable_db_test.cc to test multi-threaded backup and restore. Update backupable_db_test.cc to test backups when the backup environment is not the same as the database environment. Test Plan: Run ./backupable_db_test Run valgrind ./backupable_db_test Run with TSAN and ASAN Reviewers: yhchiang, rven, anthony, sdong, igor Reviewed By: igor Subscribers: yhchiang, anthony, sdong, leveldb, dhruba Differential Revision: https://reviews.facebook.net/D40725
-
- 02 7月, 2015 2 次提交
-
-
由 Yueh-Hsuan Chiang 提交于
Summary: The option bottommost_level_compaction was introduced lately. This option breaks the Java API behavior. To prevent the library from doing so we set that option to a fixed value in Java. In future we are going to remove that portion and replace the hardcoded options using a more flexible way. Fixed bug introduced by WriteBatchWithIndex Patch Lately icanadi changed the behavior of WriteBatchWithIndex. See commit: 821cff11 This commit solves problems introduced by above mentioned commit. Test Plan: make rocksdbjava make jtest Reviewers: adamretter, ankgup87, yhchiang Reviewed By: yhchiang Subscribers: igor, dhruba Differential Revision: https://reviews.facebook.net/D40647
-
由 Yueh-Hsuan Chiang 提交于
Summary: Rewrite Java tests compactRangeToLevel and compactRangeToLevelColumnFamily to make them more deterministic and robust. Test Plan: make rocksdbjava make jtest Reviewers: anthony, fyrz, adamretter, igor Reviewed By: igor Subscribers: dhruba, leveldb Differential Revision: https://reviews.facebook.net/D40941
-
- 01 7月, 2015 3 次提交
-
-
由 sdong 提交于
Summary: Try to allocate LevelFileIteratorState and LevelFileNumIterator from DB iterator's arena, instead of calling malloc and free. Test Plan: valgrind check Reviewers: rven, yhchiang, anthony, kradhakrishnan, igor Reviewed By: igor Subscribers: leveldb, dhruba Differential Revision: https://reviews.facebook.net/D40929
-
由 Igor Canadi 提交于
Summary: Copy change from D37533 to gcc 4.8.1 config Test Plan: make db_bench, `ldd db_bench`, try running it Reviewers: MarkCallaghan, anthony Reviewed By: anthony Subscribers: dhruba, leveldb Differential Revision: https://reviews.facebook.net/D40845
-
由 krad 提交于
Summary: We have a race in the way test works. We avoided the race by adding the wait to the counter. I thought 1s was eternity, but that is not true in some scenarios. Increasing the timeout to 10s and adding warnings. Also, adding nosleep to avoid the case where the wakeup thread is waiting behind the sleeping thread for scheduling. Test Plan: Run make check Reviewers: siying igorcanadi CC: leveldb@ Task ID: #7312624 Blame Rev:
-
- 30 6月, 2015 5 次提交
-
-
由 Tomislav Novak 提交于
Summary: When seek target is a merge key (`kTypeMerge`), `DBIter::FindNextUserEntry()` advances the underlying iterator _past_ the current key (`saved_key_`); see `MergeValuesNewToOld()`. However, `FindPrevUserKey()` assumes that `iter_` points to an entry with the same user key as `saved_key_`. As a result, `it->Seek(key) && it->Prev()` can cause the iterator to be positioned at the _next_, instead of the previous, entry (new test, written by @lovro, reproduces the bug). This diff changes `FindPrevUserKey()` to also skip keys that are _greater_ than `saved_key_`. Test Plan: db_test Reviewers: igor, sdong Reviewed By: sdong Subscribers: leveldb, dhruba, lovro Differential Revision: https://reviews.facebook.net/D40791
-
由 Yueh-Hsuan Chiang 提交于
Summary: Make column_family_test runnable in ROCKSDB_LITE. Test Plan: column_family_test Reviewers: sdong, igor Reviewed By: igor Subscribers: dhruba, leveldb Differential Revision: https://reviews.facebook.net/D40251
-
由 krad 提交于
-
由 Igor Canadi 提交于
Summary: Based on @anthony's feedback, we want to fail early if our static linking fails. Test Plan: none Reviewers: anthony Reviewed By: anthony Subscribers: dhruba, anthony, leveldb Differential Revision: https://reviews.facebook.net/D40839
-
由 krad 提交于
Summary: Avoid falling victim to race condition. Test Plan: Run the unit test Reviewers: sdong igor CC: leveldb@ Task ID: #7312624 Blame Rev:
-
- 27 6月, 2015 4 次提交
-
-
由 Igor Canadi 提交于
Summary: Currently, when we insert something into block cache, we say that the block cache capacity decreased by the size of the block. However, size of the block might be less than the actual memory used by this object. For example, 4.5KB block will actually use 8KB of memory. So even if we configure block cache to 10GB, our actually memory usage of block cache will be 20GB! This problem showed up a lot in testing and just recently also showed up in MongoRocks production where we were using 30GB more memory than expected. This diff will fix the problem. Instead of counting the block size, we will count memory used by the block. That way, a block cache configured to be 10GB will actually use only 10GB of memory. I'm using non-portable function and I couldn't find info on portability on Google. However, it seems to work on Linux, which will cover majority of our use-cases. Test Plan: 1. fill up mongo instance with 80GB of data 2. restart mongo with block cache size configured to 10GB 3. do a table scan in mongo 4. memory usage before the diff: 12GB. memory usage after the diff: 10.5GB Reviewers: sdong, MarkCallaghan, rven, yhchiang Reviewed By: yhchiang Subscribers: dhruba, leveldb Differential Revision: https://reviews.facebook.net/D40635
-
由 Igor Canadi 提交于
Summary: It's not really nice to call user's API with garbage data in new_value. This diff makes sure that new_value is empty before calling the merge operator. Test Plan: Added assert to Merge operator in merge_test Reviewers: sdong, yhchiang Reviewed By: yhchiang Subscribers: dhruba, leveldb Differential Revision: https://reviews.facebook.net/D40773
-
由 Igor Canadi 提交于
Summary: as title Test Plan: make check Reviewers: sdong Subscribers: dhruba, leveldb Differential Revision: https://reviews.facebook.net/D40785
-
由 Igor Canadi 提交于
Summary: If we create a new temp directory for each build, scons will recompile everything because we have different parameters. Instead, let's set up a constant path to our static lib. That way we won't have to recompile. Test Plan: Run fb_compile_mongo.sh twice -- second time it didn't recompile everything Reviewers: MarkCallaghan, anthony Reviewed By: anthony Subscribers: dhruba, leveldb Differential Revision: https://reviews.facebook.net/D40707
-
- 26 6月, 2015 2 次提交
-
-
由 Venkatesh Radhakrishnan 提交于
Summary: Fixes task 7156865 where a compaction causes a hang in flush memtable if CancelAllBackgroundWork was called prior to it. Stack trace is in : https://phabricator.fb.com/P19848829 We end up waiting for a flush which will never happen because there are no background threads. Test Plan: PreShutdownFlush Reviewers: sdong, igor Reviewed By: sdong, igor Subscribers: dhruba, leveldb Differential Revision: https://reviews.facebook.net/D40617
-
Summary: #7124486: RocksDB's Iterator.SeekToLast should seek to the last key before iterate_upper_bound if presents Test Plan: ./db_iter_test run successfully with the new testcase Reviewers: rven, yhchiang, igor, anthony, kradhakrishnan, sdong Reviewed By: sdong Subscribers: leveldb, dhruba Differential Revision: https://reviews.facebook.net/D40425
-
- 25 6月, 2015 3 次提交
-
-
由 Igor Canadi 提交于
Summary: Added a script that will compile MongoRocks with the same flags as RocksDB binary. On FB infra, we can now do: cd ~/rocksdb; make static_lib cd ~/mongo; ~/rocksdb/build_tools/fb_compile_mongo.sh No need to upgrade the g++ on the devbox (like Aaron and I did) or maintain a separate script to compile (like Mark did) fb_compile_mongo.sh gets the settings from fbcode_config.sh, so it also makes it easier to upgrade the environment one day. Test Plan: Compiled mongod with new script. Also, ldd output looks good: https://phabricator.fb.com/P19891602 Reviewers: AaronFeldman, MarkCallaghan, anthony Reviewed By: anthony Subscribers: anthony, dhruba, leveldb Differential Revision: https://reviews.facebook.net/D40659
-
由 Yueh-Hsuan Chiang 提交于
Summary: Make stringappend_test runnable in ROCKSDB_LITE Test Plan: stringappend_test Reviewers: sdong, rven, anthony, kradhakrishnan, IslamAbdelRahman, igor Reviewed By: igor Subscribers: dhruba, leveldb Differential Revision: https://reviews.facebook.net/D40593
-
由 Yueh-Hsuan Chiang 提交于
Summary: BYTES_READ only count the number of logical bytes read from the DB::Get() function. It neither includes all logical bytes read nor indicates IO read bytes. This patch improves the comment for BYTES_READ. Test Plan: Only change comment. Reviewers: sdong, rven, anthony, kradhakrishnan, IslamAbdelRahman, igor Reviewed By: igor Subscribers: dhruba, leveldb Differential Revision: https://reviews.facebook.net/D40599
-
- 24 6月, 2015 5 次提交
-
-
由 Yueh-Hsuan Chiang 提交于
Summary: Block redis_test in ROCKSDB_LITE as utilities not supported in ROCKSDB_LITE. Test Plan: redis_test Reviewers: sdong, igor, rven, anthony, kradhakrishnan, IslamAbdelRahman Reviewed By: IslamAbdelRahman Subscribers: dhruba, leveldb Differential Revision: https://reviews.facebook.net/D40587
-
由 Yueh-Hsuan Chiang 提交于
Summary: Make table_properties_collector_test runnable in ROCKSDB_LITE Test Plan: table_properties_collector_test Reviewers: sdong, rven, anthony, kradhakrishnan, IslamAbdelRahman, igor Reviewed By: igor Subscribers: dhruba, leveldb Differential Revision: https://reviews.facebook.net/D40581
-
由 Yueh-Hsuan Chiang 提交于
Summary: Remove -Wl,--no-as-needed flag when making shared_lib in OSX and IOS as those environment doe not have compile option --no-as-needed ld: unknown option: --no-as-needed clang: error: linker command failed with exit code 1 (use -v to see invocation) Test Plan: make shared_lib Reviewers: meyering, igor Reviewed By: igor Subscribers: dhruba, leveldb Differential Revision: https://reviews.facebook.net/D40353
-
由 Islam AbdelRahman 提交于
Summary: Replace force_bottommost_level_compaction in CompactRangeOption with an option that allow the user to (always skip, always compact, compact if compaction filter is present) the bottommost level for level based compaction. Test Plan: make check Reviewers: sdong, yhchiang, igor Reviewed By: igor Subscribers: dhruba Differential Revision: https://reviews.facebook.net/D40527
-
由 Giuseppe Ottaviano 提交于
Summary: Implementation of a table-level row cache. It only caches point queries done through the `DB::Get` interface, queries done through the `Iterator` interface will completely skip the cache. Supports snapshots and merge operations. Test Plan: Ran `make valgrind_check commit-prereq` Reviewers: igor, philipp, sdong Reviewed By: sdong Subscribers: dhruba Differential Revision: https://reviews.facebook.net/D39849
-
- 23 6月, 2015 4 次提交
-
-
由 krad 提交于
Summary: The "one size fits all" approach with WAL recovery will only introduce inconvenience for our varied clients as we go forward. The current recovery is a bit heuristic. We introduce the following levels of consistency while replaying the WAL. 1. RecoverAfterRestart (kTolerateCorruptedTailRecords) This mocks the current recovery mode. 2. RecoverAfterCleanShutdown (kAbsoluteConsistency) This is ideal for unit test and cases where the store is shutdown cleanly. We tolerate no corruption or incomplete writes. 3. RecoverPointInTime (kPointInTimeRecovery) This is ideal when using devices with controller cache or file systems which can loose data on restart. We recover upto the point were is no corruption or incomplete write. 4. RecoverAfterDisaster (kSkipAnyCorruptRecord) This is ideal mode to recover data. We tolerate corruption and incomplete writes, and we hop over those sections that we cannot make sense of salvaging as many records as possible. Test Plan: (1) Run added unit test to cover all levels. (2) Run make check. Reviewers: leveldb, sdong, igor Subscribers: yoshinorim, dhruba Differential Revision: https://reviews.facebook.net/D38487
-
由 Islam AbdelRahman 提交于
Summary: Fixing bad merge Test Plan: make -j64 check (this is not enough to verify the fix) Reviewers: igor, sdong Reviewed By: sdong Subscribers: dhruba Differential Revision: https://reviews.facebook.net/D40521
-
由 krad 提交于
Summary: MyRocks need a mechanism to track read outliers. We need to expose this stat. Test Plan: None Reviewers: sdong CC: leveldb Task ID: #7152512 Blame Rev:
-
由 Aaron Feldman 提交于
Summary: Fix broken gflags link Test Plan: Follow the link Reviewers: igor Reviewed By: igor Subscribers: dhruba Differential Revision: https://reviews.facebook.net/D40503
-
- 20 6月, 2015 2 次提交
-
-
由 Venkatesh Radhakrishnan 提交于
Summary: Fixed a valgrind issue in checkpoint_test Test Plan: valgrind on checkpoint_test Reviewers: igor, anthony Reviewed By: anthony Subscribers: dhruba, leveldb Differential Revision: https://reviews.facebook.net/D40455
-
由 Michael Callahan 提交于
Summary: Recent checkin added ldb_test.py to the make check target but the test fails. Remove it again for now and make task. Test Plan: No more ldb_tests.py running Reviewers: igor, anthony Reviewed By: anthony Subscribers: dhruba Differential Revision: https://reviews.facebook.net/D40449
-