- 13 5月, 2015 2 次提交
-
-
由 Igor Canadi 提交于
Summary: Example output: {"time_micros": 1431463794310521, "job": 353, "event": "table_file_creation", "file_number": 387, "file_size": 86937, "table_info": {"data_size": "81801", "index_size": "9751", "filter_size": "0", "raw_key_size": "23448", "raw_average_key_size": "24.000000", "raw_value_size": "990571", "raw_average_value_size": "1013.890481", "num_data_blocks": "245", "num_entries": "977", "filter_policy_name": "", "kDeletedKeys": "0"}} Also fixed a bug where BuildTable() in recovery was passing Env::IOHigh argument into paranoid_checks_file parameter. Test Plan: make check + check out the output in the log Reviewers: sdong, rven, yhchiang Reviewed By: yhchiang Subscribers: dhruba, leveldb Differential Revision: https://reviews.facebook.net/D38343
-
由 Igor Canadi 提交于
Summary: This caused a crash of our MongoDB + RocksDB instance. PickCompactionBySize() sets its own parent_index. We never reset this parent_index when picking PickFilesMarkedForCompactionExperimental(). So we might end up doing SetupOtherInputs() with parent_index that was set by PickCompactionBySize, although we're using compaction calculated using PickFilesMarkedForCompactionExperimental. Test Plan: Added a unit test that fails with assertion on master. Reviewers: yhchiang, rven, sdong Reviewed By: sdong Subscribers: dhruba, leveldb Differential Revision: https://reviews.facebook.net/D38337
-
- 12 5月, 2015 1 次提交
-
-
由 agiardullo 提交于
Summary: Added a couple functions to WriteBatchWithIndex to make it easier to query the value of a key including reading pending writes from a batch. (This is needed for transactions). I created write_batch_with_index_internal.h to use to store an internal-only helper function since there wasn't a good place in the existing class hierarchy to store this function (and it didn't seem right to stick this function inside WriteBatchInternal::Rep). Since I needed to access the WriteBatchEntryComparator, I moved some helper classes from write_batch_with_index.cc into write_batch_with_index_internal.h/.cc. WriteBatchIndexEntry, ReadableWriteBatch, and WriteBatchEntryComparator are all unchanged (just moved to a different file(s)). Test Plan: Added new unit tests. Reviewers: rven, yhchiang, sdong, igor Reviewed By: igor Subscribers: dhruba, leveldb Differential Revision: https://reviews.facebook.net/D38037
-
- 10 5月, 2015 1 次提交
-
-
由 Igor Canadi 提交于
Summary: In new clang we need to add override to every overriden function Test Plan: none Reviewers: rven Subscribers: dhruba Differential Revision: https://reviews.facebook.net/D38259
-
- 09 5月, 2015 2 次提交
-
-
由 Igor Canadi 提交于
Summary: When reporting compaction that was started because of SuggestCompactRange() we should treat it as manual compaction. Test Plan: none Reviewers: yhchiang, rven Reviewed By: rven Subscribers: dhruba, leveldb Differential Revision: https://reviews.facebook.net/D38139
-
由 Yueh-Hsuan Chiang 提交于
Summary: Don't treat warnings as error when building rocksdbjavastatic Test Plan: make rocksdbjavastatic -j32 Reviewers: rven, fyrz, adamretter, igor Reviewed By: igor Subscribers: dhruba, leveldb Differential Revision: https://reviews.facebook.net/D38187
-
- 08 5月, 2015 3 次提交
-
-
由 Igor Canadi 提交于
Summary: Without this I get bunch of questions when I run `make clean` Test Plan: no more questions! Reviewers: rven, yhchiang, meyering, anthony Reviewed By: meyering, anthony Subscribers: meyering, dhruba, leveldb Differential Revision: https://reviews.facebook.net/D38145
-
由 Igor Canadi 提交于
Summary: In D28521 we removed GarbageCollect() from BackupEngine's constructor. The reason was that opening BackupEngine on HDFS was very slow and in most cases we didn't have any garbage. We allowed the user to call GarbageCollect() when it detects some garbage files in his backup directory. Unfortunately, this left us vulnerable to an interesting issue. Let's say we started a backup and copied files {1, 3} but the backup failed. On another host, we restore DB from backup and generate {1, 3, 5}. Since {1, 3} is already there, we will not overwrite. However, these files might be from a different database so their contents might be different. See internal task t6781803 for more info. Now, when we're copying files and we discover a file already there, we check: 1. if the file is not referenced from any backups, we overwrite the file. 2. if the file is referenced from other backups AND the checksums don't match, we fail the backup. This will only happen if user is using a single backup directory for backing up two different databases. 3. if the file is referenced from other backups AND the checksums match, it's all good. We skip the copy and go copy the next file. Test Plan: Added new test to backupable_db_test. The test fails before this patch. Reviewers: sdong, rven, yhchiang Reviewed By: yhchiang Subscribers: dhruba, leveldb Differential Revision: https://reviews.facebook.net/D37599
-
由 Igor Canadi 提交于
Summary: as title Test Plan: none Reviewers: sdong Subscribers: dhruba, leveldb Differential Revision: https://reviews.facebook.net/D38175
-
- 07 5月, 2015 2 次提交
-
-
由 Jörg Maier 提交于
Small fixes to Java benchmark
-
由 Yueh-Hsuan Chiang 提交于
Summary: Now we're able to show more details about a compaction in GetThreadList() :) This patch allows GetThreadList() to report basic compaction operation properties. Basic compaction properties include: 1. job id 2. compaction input / output level 3. compaction property flags (is_manual, is_deletion, .. etc) 4. total input bytes 5. the number of bytes has been read currently. 6. the number of bytes has been written currently. Flush operation properties will be done in a seperate diff. Test Plan: /db_bench --threads=30 --num=1000000 --benchmarks=fillrandom --thread_status_per_interval=1 Sample output of tracking same job: ThreadID ThreadType cfName Operation ElapsedTime Stage State OperationProperties 140664171987072 Low Pri default Compaction 31.357 ms CompactionJob::FinishCompactionOutputFile BaseInputLevel 1 | BytesRead 2264663 | BytesWritten 1934241 | IsDeletion 0 | IsManual 0 | IsTrivialMove 0 | JobID 277 | OutputLevel 2 | TotalInputBytes 3964158 | ThreadID ThreadType cfName Operation ElapsedTime Stage State OperationProperties 140664171987072 Low Pri default Compaction 59.440 ms CompactionJob::FinishCompactionOutputFile BaseInputLevel 1 | BytesRead 2264663 | BytesWritten 1934241 | IsDeletion 0 | IsManual 0 | IsTrivialMove 0 | JobID 277 | OutputLevel 2 | TotalInputBytes 3964158 | ThreadID ThreadType cfName Operation ElapsedTime Stage State OperationProperties 140664171987072 Low Pri default Compaction 226.375 ms CompactionJob::Install BaseInputLevel 1 | BytesRead 3958013 | BytesWritten 3621940 | IsDeletion 0 | IsManual 0 | IsTrivialMove 0 | JobID 277 | OutputLevel 2 | TotalInputBytes 3964158 | Reviewers: sdong, rven, igor Reviewed By: igor Subscribers: dhruba, leveldb Differential Revision: https://reviews.facebook.net/D37653
-
- 06 5月, 2015 5 次提交
-
-
由 Igor Canadi 提交于
Summary: Couple changes: 1. instead of SnapshotList, just take a vector of snapshots 2. don't take a separate parameter is_snapshots_supported. If there are snapshots in the list, that means they are supported. I actually think we should get rid of this notion of snapshots not being supported. 3. don't pass in mutable_cf_options as a parameter. Lifetime of mutable_cf_options is a bit tricky to maintain, so it's better to not pass it in for the whole compaction job. We only really need it when we install the compaction results. Test Plan: make check Reviewers: sdong, rven, yhchiang Reviewed By: yhchiang Subscribers: dhruba, leveldb Differential Revision: https://reviews.facebook.net/D36627
-
由 Laurent Demailly 提交于
Summary: fix crashes in stats and compaction filter for db_ttl_impl Test Plan: Ran build with lots of debugging https://reviews.facebook.net/differential/diff/194175/ Reviewers: yhchiang, igor, rven Reviewed By: igor Subscribers: rven, dhruba Differential Revision: https://reviews.facebook.net/D38001
-
由 Venkatesh Radhakrishnan 提交于
Summary: Fixes #6840824, running "make check" on centos6 hits a deadlock in column_family_test Test Plan: seq 10000 | parallel --gnu --eta 't=/dev/shm/rdb-{}; rm -rf $t; mkdir $t && export TEST_TMPDIR=$t; ./column_family_test > $t/log-{}' Made the test deterministic by narrrowing the window for the flush. Reviewers: igor, meyering Reviewed By: igor Subscribers: dhruba, leveldb Differential Revision: https://reviews.facebook.net/D38079
-
由 Igor Canadi 提交于
Summary: This will reduce errors reported by open source by 90%. Developers should have warnings break their compile. Users shouldn't. Test Plan: make dbg: g++ -g -W -Wextra -Wall -Wsign-compare -Wshadow -Wno-unused-parameter -Werror -I. -I./include -std=c++11 -DROCKSDB_PLATFORM_POSIX -DOS_LINUX -fno-builtin-memcmp -DROCKSDB_FALLOCATE_PRESENT -DSNAPPY -DGFLAGS=gflags -DZLIB -DBZIP2 -march=native -isystem ./third-party/gtest-1.7.0/fused-src -Woverloaded-virtual -Wn on-virtual-dtor -Wno-missing-field-initializers -c db/repair.cc -o db/repair.o make all: g++ -g -W -Wextra -Wall -Wsign-compare -Wshadow -Wno-unused-parameter -Werror -I. -I./include -std=c++11 -DROCKSDB_PLATFORM_POSIX -DOS_LINUX -fno-builtin-memcmp -DROCKSDB_FALLOCATE_PRESENT -DSNAPPY -DGFLAGS=gflags -DZLIB -DBZIP2 -march=native -isystem ./third-party/gtest-1.7.0/fused-src -O2 -fno-omit-frame-pointer -momit-leaf-frame-pointer -Woverloaded-virtual -Wnon-virtual-dtor -Wno-missing-field-initializers -c db/repair.cc -o db/repair.o make static_lib: g++ -g -W -Wextra -Wall -Wsign-compare -Wshadow -Wno-unused-parameter -I. -I./include -std=c++11 -DROCKSDB_PLATFORM_POSIX -DOS_LINUX -fno-builtin-memcmp -DROCKSDB_FALLOCATE_PRESENT -DSNAPPY -DGFLAGS=gflags -DZLIB -DBZIP2 -march=native -isystem ./third-party/gtest-1.7.0/fused-src -O2 -fno-omit-frame-pointer -momit-leaf-frame-pointer -DNDEBUG -Woverloaded-virtual -Wnon-virtual-dtor -Wno-missing-field-initializers -c db/repair.cc -o db/repair.o Reviewers: sdong, yhchiang, rven Reviewed By: rven Subscribers: dhruba, leveldb Differential Revision: https://reviews.facebook.net/D38031
-
由 Liangjun Feng 提交于
Summary: Optimize GetRange Function by checking the level of the files Test Plan: pass make all check Reviewers: rven, yhchiang, igor Reviewed By: igor Subscribers: dhruba Differential Revision: https://reviews.facebook.net/D37977
-
- 05 5月, 2015 2 次提交
-
-
由 Igor Canadi 提交于
Summary: Ooops :) status.ok() is acutally highly likely :) Test Plan: none Reviewers: rven, yhchiang, anthony Reviewed By: anthony Subscribers: dhruba, leveldb Differential Revision: https://reviews.facebook.net/D38043
-
由 Igor Canadi 提交于
Fix formatting
-
- 03 5月, 2015 1 次提交
-
-
由 Jim Meyering 提交于
Summary: [noticed a new warning when building with the very latest gcc] * db/memtablerep_bench.cc (FLAGS_env): Remove declaration of unused varaible, to avoid this warning/error: db/memtablerep_bench.cc:135:22: error: ‘FLAGS_env’ defined but not\ used [-Werror=unused-variable] static rocksdb::Env* FLAGS_env = rocksdb::Env::Default(); ^ Test Plan: compile Reviewers: ljin, rven, igor.sugak, yhchiang, sdong, igor Reviewed By: igor Subscribers: dhruba Differential Revision: https://reviews.facebook.net/D37983
-
- 02 5月, 2015 3 次提交
-
-
由 Mark Callaghan 提交于
Summary: See https://gist.github.com/mdcallag/b867ee051d765760be0d for a sample Task ID: # Blame Rev: Test Plan: Revert Plan: Database Impact: Memcache Impact: Other Notes: EImportant: - begin *PUBLIC* platform impact section - Bugzilla: # - end platform impact - Reviewers: igor Reviewed By: igor Subscribers: dhruba Differential Revision: https://reviews.facebook.net/D37971
-
由 Venkatesh Radhakrishnan 提交于
Summary: This diff fixes a hang reported by a Github user. https://www.facebook.com/l.php?u=https%3A%2F%2Fgithub.com%2Ffacebook%2Frocksdb%2Fissues%2F595%23issuecomment-96983273&h=9AQFYOWlo Multiple large write batches with column families cause a hang. The issue was caused by not doing flushes/compaction when the write controller was stopped. Test Plan: Create a DBTest from the user's test case Reviewers: igor Reviewed By: igor Subscribers: dhruba, leveldb Differential Revision: https://reviews.facebook.net/D37929
-
由 Mark Callaghan 提交于
Summary: The default, use one iter for the whole test, isn't good. This cost me a few hours of debugging and a few days of tessting. For readonly that isn't realistic and for read-write that keeps a lot of old sst files around. I remove the option because nothing uses it and not calling gettimeofday per loop iteration adds about 3% to QPS at 20 threads. Task ID: # Blame Rev: Test Plan: run db_bench Revert Plan: Database Impact: Memcache Impact: Other Notes: EImportant: - begin *PUBLIC* platform impact section - Bugzilla: # - end platform impact - Reviewers: igor Reviewed By: igor Subscribers: dhruba Differential Revision: https://reviews.facebook.net/D37965
-
- 01 5月, 2015 5 次提交
-
-
由 Adam Retter 提交于
Bugfix remove deprecated option use which was removed in previous commit 019ecd19
-
由 Adam Retter 提交于
-
由 Igor Canadi 提交于
Summary: fix build Test Plan: works Reviewers: kradhakrishnan Subscribers: dhruba, leveldb Differential Revision: https://reviews.facebook.net/D37911
-
由 krad 提交于
Summary: CPU profiling reveals GetApproximateSizes as a bottleneck for performance. The current implementation is sub-optimal, it scans every file in every level to compute the result. We can take advantage of the fact that all levels above 0 are sorted in the increasing order of key ranges and use binary search to locate the starting index. This can reduce the number of comparisons required to compute the result. Test Plan: We have good test coverage. Run the tests. Reviewers: sdong, igor, rven, dynamike Subscribers: dynamike, maykov, dhruba, leveldb Differential Revision: https://reviews.facebook.net/D37755
-
由 Igor Canadi 提交于
-
- 30 4月, 2015 2 次提交
-
-
由 Igor Canadi 提交于
Summary: Before the fix we also marked the bottommost level for compaction. This is wrong because then RocksDB has N+1 levels instead of N as before the compaction. Test Plan: SuggestCompactRangeTest in db_test Reviewers: yhchiang, rven Reviewed By: rven Subscribers: dhruba, leveldb Differential Revision: https://reviews.facebook.net/D37869
-
由 Igor Canadi 提交于
Summary: For very detailed explanation of what's happening read this: https://github.com/facebook/rocksdb/issues/596 Test Plan: make check + new unit test Reviewers: yhchiang, anthony, rven Reviewed By: rven Subscribers: adamretter, dhruba, leveldb Differential Revision: https://reviews.facebook.net/D37779
-
- 29 4月, 2015 1 次提交
-
-
由 Yueh-Hsuan Chiang 提交于
[RocksJava] Remove deprecated methods
-
- 28 4月, 2015 3 次提交
-
-
由 agiardullo 提交于
Summary: Remove duplicate code. If this diff looks good, I will cleanup other call sites as well. Test Plan: unit tests Reviewers: rven, yhchiang, igor Reviewed By: igor Subscribers: dhruba, leveldb Differential Revision: https://reviews.facebook.net/D37761
-
由 Mark Callaghan 提交于
Summary: This runs a benchmark for LevelDB similar to what we have in tools/run_flash_bench.sh. It requires changes to db_bench that I published in a LevelDB fork on github. Some results are at: http://smalldatum.blogspot.com/2015/04/comparing-leveldb-and-rocksdb-take-2.html Sample output: ops/sec mb/sec usec/op avg p50 Test 525 16.4 1904.5 1904.5 111.0 fillseq.v32768 75187 15.5 13.3 13.3 4.4 fillseq.v200 28328 5.8 35.3 35.3 4.7 overwrite.t1.s0 175438 0.0 5.7 5.7 4.4 readrandom.t1 28490 5.9 35.1 35.1 4.7 overwrite.t1.s0 121951 0.0 8.2 8.2 5.7 readwhilewriting.t1 Task ID: # Blame Rev: Test Plan: Revert Plan: Database Impact: Memcache Impact: Other Notes: EImportant: - begin *PUBLIC* platform impact section - Bugzilla: # - end platform impact - Reviewers: igor Reviewed By: igor Subscribers: dhruba Differential Revision: https://reviews.facebook.net/D37749
-
由 Igor Canadi 提交于
Summary: Added these events: * Recovery start, finish and also when recovery creates a file * Trivial move * Compaction start, finish and when compaction creates a file * Flush start, finish Also includes small fix to EventLogger Also added option ROCKSDB_PRINT_EVENTS_TO_STDOUT which is useful when we debug things. I've spent far too much time chasing LOG files. Still didn't get sst table properties in JSON. They are written very deeply into the stack. I'll address in separate diff. TODO: * Write specification. Let's first use this for a while and figure out what's good data to put here, too. After that we'll write spec * Write tools that parse and analyze LOGs. This can be in python or go. Good intern task. Test Plan: Ran db_bench with ROCKSDB_PRINT_EVENTS_TO_STDOUT. Here's the output: https://phabricator.fb.com/P19811976 Reviewers: sdong, yhchiang, rven, MarkCallaghan, kradhakrishnan, anthony Reviewed By: anthony Subscribers: dhruba, leveldb Differential Revision: https://reviews.facebook.net/D37521
-
- 27 4月, 2015 1 次提交
-
-
由 Aashish Pant 提交于
Test Plan: Verified that valgrind build passes for cache_test Reviewers: igor Reviewed By: igor Subscribers: dhruba Differential Revision: https://reviews.facebook.net/D37665
-
- 26 4月, 2015 2 次提交
-
-
由 Igor Canadi 提交于
Summary: Build broken by https://github.com/facebook/rocksdb/commit/6ede020dc419c1621254f26060076ee6d2c2d792 Test Plan: make all Reviewers: sdong Subscribers: dhruba, leveldb Differential Revision: https://reviews.facebook.net/D37689
-
由 Igor Canadi 提交于
fix typos
-
- 25 4月, 2015 4 次提交
-
-
由 clark.kang 提交于
-
由 Igor Canadi 提交于
Summary: Since we enabled jemalloc for open source builds, Travis looks like it's dying. Don't install jemalloc when running in travis Test Plan: none Reviewers: yhchiang Subscribers: dhruba Differential Revision: https://reviews.facebook.net/D37659
-
由 Aashish Pant 提交于
Summary: Added keyword override for SetCapacity() Test Plan: Fixes build Reviewers: igor Reviewed By: igor Subscribers: dhruba Differential Revision: https://reviews.facebook.net/D37647
-
由 Aashish Pant 提交于
Summary: When new capacity is larger than existing capacity, simply update the capacity to the new valie When new capacity is less than existing capacity, but more than the usage, simply update the capacity to new value When new capacity is less than the existing capacity and existing usage both, try to purge entries in LRU if feasible to make usage < capacity Test Plan: Created unit tests in cache_test.cc Reviewers: sdong, rven, yhchiang, igor Reviewed By: igor Subscribers: dhruba Differential Revision: https://reviews.facebook.net/D37527
-