- 03 9月, 2015 7 次提交
-
-
由 Igor Canadi 提交于
Summary: As title Test Plan: none Reviewers: yhchiang, rven, anthony, IslamAbdelRahman, kradhakrishnan, sdong Reviewed By: sdong Subscribers: dhruba, leveldb Differential Revision: https://reviews.facebook.net/D46053
-
由 Andres Noetzli 提交于
Summary: This diff is a collection of cleanups that were initially part of D43179. Additionally it adds a unified way of defining key-value maps that use a Comparator for sorting (this was previously implemented in four different places). Test Plan: make clean check all Reviewers: rven, anthony, yhchiang, sdong, igor Reviewed By: igor Subscribers: dhruba, leveldb Differential Revision: https://reviews.facebook.net/D45993
-
由 sdong 提交于
Summary: TableCache::Get() puts parameters in the wrong places so that table readers created by Get() will not have the histogram updated. Test Plan: Will write a unit test for that. Subscribers: leveldb, dhruba Differential Revision: https://reviews.facebook.net/D46035
-
由 Igor Canadi 提交于
Summary: In the first implementation of BackupEngine, LATEST_BACKUP was the commit point. The backup became committed after the write to LATEST_BACKUP completed. However, we can avoid the need for LATEST_BACKUP. Instead of write to LATEST_BACKUP, the commit point can be the rename from `meta/<backup_id>.tmp` to `meta/<backup_id>`. Once we see that there exists a file `meta/<backup_id>` (without tmp), we can assume that backup is valid. In this diff, we still write out the file LATEST_BACKUP. We need to do this so that we can maintain backward compatibility. However, the new version doesn't depend on this file anymore. We get the latest backup by `ls`-ing `meta` directory. This diff depends on D41925 Test Plan: Adjusted backupable_db_test to this new behavior Reviewers: benj, yhchiang, sdong, AaronFeldman Reviewed By: sdong Subscribers: dhruba, leveldb Differential Revision: https://reviews.facebook.net/D42069
-
由 Siying Dong 提交于
Make WinEnv::NowMicros return system time
-
由 Amit Arya 提交于
Summary: Use fallocate on LOG FILES to Test Plan: make check + ===check with strace=== [arya@devvm1441 ~/rocksdb] strace -e trace=fallocate ./ldb --db=/tmp/test_new scan fallocate(3, 01, 0, 4194304) = 0 Reviewers: sdong, anthony, IslamAbdelRahman, kradhakrishnan, lgalanis, rven, igor Reviewed By: igor Subscribers: leveldb, dhruba Differential Revision: https://reviews.facebook.net/D45969
-
由 Dmitri Smirnov 提交于
Previous change for the function https://github.com/facebook/rocksdb/commit/555ca3e7b7f06bd01dfd5e04dbb2cef5360f7917#diff-bdc04e0404c2db4fd3ac5118a63eaa4a made use of the QueryPerformanceCounter to return microseconds values that do not repeat as std::chrono::system_clock returned values that made auto_roll_logger_test fail. The interface documentation does not state that we need to return system time describing the return value as a number of microsecs since some moment in time. However, because on Linux it is implemented using gettimeofday various pieces of code (such as GenericRateLimiter) took advantage of that and make use of NowMicros() as a system timestamp. Thus the previous change broke rate_limiter_test on Windows. In addition, the interface name NowMicros() suggests that it is actually a timestamp so people use it as such. This change makes use of the new system call on Windows that returns system time with required precision. This change preserves the fix for auto_roll_logger_test and fixes rate_limiter_test. Note that DBTest.RateLimitingTest still fails due to a separately reported issue.
-
- 02 9月, 2015 2 次提交
-
-
由 Yueh-Hsuan Chiang 提交于
Summary: This patch fixed couple build issues of rocksdbjavastaticrelease. Test Plan: make rocksdbjavastaticrelease Reviewers: sdong, anthony, IslamAbdelRahman, igor Reviewed By: igor Subscribers: dhruba, leveldb Differential Revision: https://reviews.facebook.net/D45915
-
由 Tomislav Novak 提交于
Summary: I noticed that memtable iterator usually crosses the `iterate_upper_bound` threshold when tailing. Changes introduced in D43833 made `NeedToSeekImmutable` always return true in such case, even when `Seek()` only needs to rewind the memtable iterator. In a test I ran, this caused the "tailing efficiency" (ratio of calls to `Seek()` that only affect the memtable versus all seeks) to drop almost to zero. This diff attempts to fix the regression by using a different flag to indicate that `current_` is over the limit instead of resetting `valid_` in `UpdateCurrent()`. Test Plan: `DBTestTailingIterator.TailingIteratorUpperBound` Reviewers: sdong, rven Reviewed By: rven Subscribers: dhruba, march Differential Revision: https://reviews.facebook.net/D45909
-
- 01 9月, 2015 11 次提交
-
-
由 Andres Notzli 提交于
Summary: Flushes in listener_test happened to early when ROCKSDB_MALLOC_USABLE_SIZE was active (e.g. when compiling with ROCKSDB_FBCODE_BUILD_WITH_481=1) due to malloc_usable_size() reporting a better estimate (similar to https://reviews.facebook.net/D43317 ). This patch grows the write buffer size slightly to compensate for this. Test Plan: ROCKSDB_FBCODE_BUILD_WITH_481=1 make listener_test && ./listener_test Reviewers: rven, anthony, yhchiang, igor, sdong Reviewed By: igor Subscribers: dhruba, leveldb Differential Revision: https://reviews.facebook.net/D45921
-
由 Yueh-Hsuan Chiang 提交于
Summary: Fixed the following compile warning in rocksdbjava: java/rocksjni/comparatorjnicallback.cc:20:14: warning: unused variable ‘rs’ [-Wunused-variable] const jint rs = env->GetJavaVM(&m_jvm); ^ java/rocksjni/comparatorjnicallback.cc: In member function ‘JNIEnv* rocksdb::BaseComparatorJniCallback::getJniEnv() const’: java/rocksjni/comparatorjnicallback.cc:45:8: warning: unused variable ‘rs’ [-Wunused-variable] jint rs = m_jvm->AttachCurrentThread(reinterpret_cast<void **>(&env), NULL); ^ java/rocksjni/loggerjnicallback.cc: In constructor ‘rocksdb::LoggerJniCallback::LoggerJniCallback(JNIEnv*, jobject)’: java/rocksjni/loggerjnicallback.cc:19:14: warning: unused variable ‘rs’ [-Wunused-variable] const jint rs = env->GetJavaVM(&m_jvm); ^ java/rocksjni/loggerjnicallback.cc: In member function ‘JNIEnv* rocksdb::LoggerJniCallback::getJniEnv() const’: java/rocksjni/loggerjnicallback.cc:33:8: warning: unused variable ‘rs’ [-Wunused-variable] jint rs = m_jvm->AttachCurrentThread(reinterpret_cast<void **>(&env), NULL); ^ Test Plan: make rocksdbjava -j32 Reviewers: sdong, anthony, IslamAbdelRahman, igor Reviewed By: igor Subscribers: dhruba, leveldb Differential Revision: https://reviews.facebook.net/D45891
-
由 Yueh-Hsuan Chiang 提交于
Summary: Fixed the following compile warning in linux32 environment. ==> linux32: util/sst_dump_tool.cc: In member function ‘int rocksdb::SstFileReader::ShowAllCompressionSizes(size_t)’: ==> linux32: util/sst_dump_tool.cc:167:50: warning: format ‘%lu’ expects argument of type ‘long unsigned int’, but argument 3 has type ‘size_t {aka unsigned int}’ [-Wformat=] ==> linux32: fprintf(stdout, "Block Size: %lu\n", block_size); Test Plan: make sst_dump Reviewers: anthony, IslamAbdelRahman, sdong, igor Reviewed By: igor Subscribers: dhruba, leveldb Differential Revision: https://reviews.facebook.net/D45885
-
由 sdong 提交于
Summary: arena_test is failing with glibc-2.17. Make it more robust Test Plan: Run arena_test using both of glibc-2.17 and 2.2 and make sure both passes. Reviewers: yhchiang, rven, IslamAbdelRahman, kradhakrishnan, igor Reviewed By: igor Subscribers: leveldb, dhruba Differential Revision: https://reviews.facebook.net/D45879
-
由 agiardullo 提交于
Summary: Provide a way to specify a detailed static error message for a Status without incurring a memcpy. Let me know what people think of this approach. Test Plan: added simple test Reviewers: igor, yhchiang, rven, sdong Reviewed By: sdong Subscribers: dhruba, leveldb Differential Revision: https://reviews.facebook.net/D44259
-
由 agiardullo 提交于
Summary: Pessimistic Transaction expiration time checking currently causes a performace regression, Lets disable it in db_bench by default. Also, in order to be able to better tune how much contention we're simulating, added new optinos to set lock timeout and snapshot. Test Plan: run db_bench randomtranansaction Reviewers: sdong, igor, yhchiang, MarkCallaghan Reviewed By: MarkCallaghan Subscribers: dhruba, leveldb Differential Revision: https://reviews.facebook.net/D45831
-
由 Igor Canadi 提交于
remove spurious compression definitions
-
由 Ari Ekmekji 提交于
Summary: Now that the approach to parallelizing L0-L1 level-based compactions by breaking the compaction job into subcompactions is being extended to apply to universal compactions as well, the unit tests need to account for this and run the universal compaction tests with subcompactions both enabled and disabled. Test Plan: make all && make check Reviewers: sdong, igor, noetzli, anthony, yhchiang Reviewed By: yhchiang Subscribers: dhruba Differential Revision: https://reviews.facebook.net/D45657
-
由 krad 提交于
-
由 krad 提交于
Summary: This fuels commands to be executed for different sandcastle jobs. This is a nice way to separate RocksDB specific commands from Sandcastle specific job definition. Test Plan: None. Will be tested when we add Sandcastle job. Reviewers: igor, sdong Reviewed By: sdong Subscribers: dhruba, leveldb Differential Revision: https://reviews.facebook.net/D45741
-
由 sdong 提交于
Summary: malloc_usable_size() gets a better estimation of memory usage. It is already used to calculate block cache memory usage. Use it in arena too. Test Plan: Run all unit tests Reviewers: anthony, kradhakrishnan, rven, IslamAbdelRahman, yhchiang Reviewed By: yhchiang Subscribers: leveldb, dhruba Differential Revision: https://reviews.facebook.net/D43317
-
- 29 8月, 2015 5 次提交
-
-
由 Andres Noetzli 提交于
Summary: MarkLogsSynced() was doing `logs_.erase(it++);`. The standard is saying: ``` all iterators and references are invalidated, unless the erased members are at an end (front or back) of the deque (in which case only iterators and references to the erased members are invalidated) ``` Because `it` is an iterator to the first element of the container, it is invalidated, only one iteration is executed and `log.getting_synced = false;` is not being done, so `while (logs_.front().getting_synced)` in `WriteImpl()` is not terminating. Test Plan: make db_bench && ./db_bench --benchmarks=fillsync Reviewers: igor, rven, IslamAbdelRahman, anthony, kradhakrishnan, yhchiang, sdong, tnovak Reviewed By: tnovak Subscribers: kolmike, dhruba, leveldb Differential Revision: https://reviews.facebook.net/D45807
-
由 Praveen Rao 提交于
-
由 Andres Noetzli 提交于
Summary: Just realized that after D45675, part of the code in DBTest.ApproximateMemoryUsage, does not really test anything anymore, so I removed it. Test Plan: make clean all check Reviewers: rven, igor, sdong, anthony, yhchiang Reviewed By: yhchiang Subscribers: dhruba, leveldb Differential Revision: https://reviews.facebook.net/D45783
-
由 Venkatesh Radhakrishnan 提交于
Summary: The immutable memtable iterators are allocated from an arena and there is no benefit from deleting these. Also the immutable memtables themselves will continue to be in memory until the version set containing it is alive. We will not remove immutable memtable iterators over the upper bound. We now add immutable iterators to the test. Test Plan: db_tailing_iter_test.TailingIteratorTrimSeekToNext Reviewers: tnovak, sdong Reviewed By: sdong Subscribers: dhruba, leveldb Differential Revision: https://reviews.facebook.net/D45597
-
由 sdong 提交于
Summary: Add ZSTD compression type. The same way as adding LZ4. Test Plan: run all tests. Generate files in db_bench. Make sure reads succeed. But the SST files cannot be opened in older versions. Also some other adhoc tests. Reviewers: rven, anthony, IslamAbdelRahman, kradhakrishnan, igor Reviewed By: igor Subscribers: MarkCallaghan, maykov, yoshinorim, leveldb, dhruba Differential Revision: https://reviews.facebook.net/D45747
-
- 28 8月, 2015 3 次提交
-
-
由 Igor Canadi 提交于
Remove usage of C runtime API that has file handle limitation
-
由 Andres Noetzli 提交于
Summary: This patch fixes two issues in DBTest.ApproximateMemoryUsage: - It was possible that a flush happened between getting the two properties in Phase 1, resulting in different numbers for the properties and failing the assertion. This is fixed by waiting for the flush to finish before getting the properties. - There was a similar issue in Phase 2 and additionally there was an issue that rocksdb.size-all-mem-tables was not monotonically increasing because it was possible that a flush happened just after getting the properties and then another flush just before getting the properties in the next round. In this situation, the reported memory usage decreased. This is fixed by forcing a flush before getting the properties. Note: during testing, I found that kFlushesPerRound does not seem very accurate. I added a TODO for this and it would be great to get some input on what to do there. Test Plan: The first issue can be made more likely to trigger by inserting a `usleep(10000);` between the calls to GetIntProperty() in Phase 1. The second issue can be made more likely to trigger by inserting a `if (r != 0) usleep(10000);` before the calls to GetIntProperty() and a `usleep(10000);` after the calls. Then execute make db_test && ./db_test --gtest_filter=DBTest.ApproximateMemoryUsage Reviewers: rven, yhchiang, igor, sdong, anthony Reviewed By: anthony Subscribers: dhruba, leveldb Differential Revision: https://reviews.facebook.net/D45675
-
由 Siying Dong 提交于
Helper functions to support direct IO
-
- 27 8月, 2015 8 次提交
-
-
由 Javier González 提交于
Summary: This patch adds the helper functions and variables to allow a backend implementing WritableFile to support direct IO when persisting a memtable. Test Plan: Since there is no upstream implementation of WritableFile supporting direct IO, the new behavior is disabled. Tests should be provided by the backend implementing WritableFile.
-
由 Praveen Rao 提交于
-
由 Yueh-Hsuan Chiang 提交于
Summary: Add argument --show_table_properties to db_bench -show_table_properties (If true, then per-level table properties will be printed on every stats-interval when stats_interval is set and stats_per_interval is on.) type: bool default: false Test Plan: ./db_bench --show_table_properties=1 --stats_interval=100000 --stats_per_interval=1 ./db_bench --show_table_properties=1 --stats_interval=100000 --stats_per_interval=1 --num_column_families=2 Sample Output: Compaction Stats [column_family_name_000001] Level Files Size(MB) Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) Comp(cnt) Avg(sec) Stall(cnt) KeyIn KeyDrop --------------------------------------------------------------------------------------------------------------------------------------------------------------------- L0 3/0 5 0.8 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 86.3 0 17 0.021 0 0 0 L1 5/0 9 0.9 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 0.000 0 0 0 L2 9/0 16 0.2 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 0.000 0 0 0 Sum 17/0 31 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 86.3 0 17 0.021 0 0 0 Int 0/0 0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 83.9 0 2 0.022 0 0 0 Flush(GB): cumulative 0.030, interval 0.004 Stalls(count): 0 level0_slowdown, 0 level0_numfiles, 0 memtable_compaction, 0 leveln_slowdown_soft, 0 leveln_slowdown_hard Level[0]: # data blocks=2571; # entries=84813; raw key size=2035512; raw average key size=24.000000; raw value size=8481300; raw average value size=100.000000; data block size=5690119; index block size=82415; filter block size=0; (estimated) table size=5772534; filter policy name=N/A; Level[1]: # data blocks=4285; # entries=141355; raw key size=3392520; raw average key size=24.000000; raw value size=14135500; raw average value size=100.000000; data block size=9487353; index block size=137377; filter block size=0; (estimated) table size=9624730; filter policy name=N/A; Level[2]: # data blocks=7713; # entries=254439; raw key size=6106536; raw average key size=24.000000; raw value size=25443900; raw average value size=100.000000; data block size=17077893; index block size=247269; filter block size=0; (estimated) table size=17325162; filter policy name=N/A; Level[3]: # data blocks=0; # entries=0; raw key size=0; raw average key size=0.000000; raw value size=0; raw average value size=0.000000; data block size=0; index block size=0; filter block size=0; (estimated) table size=0; filter policy name=N/A; Level[4]: # data blocks=0; # entries=0; raw key size=0; raw average key size=0.000000; raw value size=0; raw average value size=0.000000; data block size=0; index block size=0; filter block size=0; (estimated) table size=0; filter policy name=N/A; Level[5]: # data blocks=0; # entries=0; raw key size=0; raw average key size=0.000000; raw value size=0; raw average value size=0.000000; data block size=0; index block size=0; filter block size=0; (estimated) table size=0; filter policy name=N/A; Level[6]: # data blocks=0; # entries=0; raw key size=0; raw average key size=0.000000; raw value size=0; raw average value size=0.000000; data block size=0; index block size=0; filter block size=0; (estimated) table size=0; filter policy name=N/A; Reviewers: anthony, IslamAbdelRahman, MarkCallaghan, sdong, igor Reviewed By: igor Subscribers: dhruba, leveldb Differential Revision: https://reviews.facebook.net/D45651
-
由 Yueh-Hsuan Chiang 提交于
Summary: This patch adds GetStringFromColumnFamilyOptions(), the inverse function of the existing GetColumnFamilyOptionsFromString(), and improves the implementation of GetColumnFamilyOptionsFromString(). Test Plan: Add a test in options_test.cc Reviewers: igor, sdong, anthony, IslamAbdelRahman Reviewed By: IslamAbdelRahman Subscribers: noetzli, dhruba, leveldb Differential Revision: https://reviews.facebook.net/D45009
-
由 Igor Canadi 提交于
Summary: ReadaheadRandomAccessFile acts as a transparent layer on top of RandomAccessFile. When a Read() request is issued, it issues a much bigger request to the OS and caches the result. When a new request comes in and we already have the data cached, it doesn't have to issue any requests to the OS. We add ReadaheadRandomAccessFile layer only when file is read during compactions. D45105 was incorrectly closed by Phabricator because I committed it to a separate branch (not master), so I'm resubmitting the diff. Test Plan: make check Reviewers: MarkCallaghan, sdong Reviewed By: sdong Subscribers: leveldb, dhruba Differential Revision: https://reviews.facebook.net/D45123
-
由 Igor Canadi 提交于
Summary: Currently, mmap returns IOError when user tries to read data past the end of the file. This diff changes the behavior. Now, we return just the bytes that we can, and report the size we returned via a Slice result. This is consistent with non-mmap behavior and also pread() system call. This diff is taken out of D45123. Test Plan: make check Reviewers: sdong Reviewed By: sdong Subscribers: dhruba, leveldb Differential Revision: https://reviews.facebook.net/D45645
-
由 sdong 提交于
DBIter to out extra keys with higher sequence numbers when changing direction from forward to backward Summary: When DBIter changes iterating direction from forward to backward, it might see some much larger keys with higher sequence ID. With this commit, these rows will be actively filtered out. It should fix existing disabled tests in db_iter_test. This may not be a perfect fix, but it introduces least impact on existing codes, in order to be safe. Test Plan: Enable existing tests and make sure they pass. Add a new test DBIterWithMergeIterTest.InnerMergeIteratorDataRace8. Also run all existing tests. Reviewers: yhchiang, rven, anthony, IslamAbdelRahman, kradhakrishnan, igor Reviewed By: igor Subscribers: leveldb, dhruba Differential Revision: https://reviews.facebook.net/D45567
-
由 Andres Noetzli 提交于
Summary: DBTest.GetProperty was failing occasionally (see task #8131266). The reason was that the test closed the database before the compaction was done. When the test reopened the database, RocksDB would schedule a compaction which in turn created table readers and lead the test to fail the assertion that rocksdb.estimate-table-readers-mem is 0. In most cases, GetIntProperty() of rocksdb.estimate-table-readers-mem happened before the compaction created the table readers, hiding the problem. This patch changes the WaitForFlushMemTable() to WaitForCompact(). WaitForFlushMemTable() is not necessary because it is already being called a couple of lines before without any insertions in-between. Test Plan: Insert `usleep(10000);` just after `Reopen(options);` on line 2333 to make the issue more likely, then run: make db_test && while ./db_test --gtest_filter=DBTest.GetProperty; do true; done Reviewers: rven, yhchiang, anthony, igor, sdong Reviewed By: sdong Subscribers: dhruba, leveldb Differential Revision: https://reviews.facebook.net/D45603
-
- 26 8月, 2015 4 次提交
-
-
由 Igor Canadi 提交于
Address noexcept and const integer lambda capture on win
-
由 Yueh-Hsuan Chiang 提交于
Summary: "rocksdb.aggregated-table-properties" and "rocksdb.aggregated-table-properties-at-level<N>" should belong to GetProperty() instead of GetIntProperty(), but the comment mistakenly classifies them to GetIntProperty(). This patch fix this comment error. Test Plan: no code change. Reviewers: sdong, anthony, IslamAbdelRahman, igor Reviewed By: igor Subscribers: dhruba Differential Revision: https://reviews.facebook.net/D45561
-
由 Dmitri Smirnov 提交于
-
由 Dmitri Smirnov 提交于
VS 2013 does not support noexcept. Complains about usage of ineteger constant within lambda requiring explicit capture.
-