- 28 5月, 2016 5 次提交
-
-
由 sdong 提交于
Summary: When rate_bytes_per_sec * refill_period_us_ overflows, the actual limited rate is very low. Handle this case so the rate will be large. Test Plan: Add a unit test for it. Reviewers: IslamAbdelRahman, andrewkr Reviewed By: andrewkr Subscribers: yiwu, lightmark, leveldb, andrewkr, dhruba Differential Revision: https://reviews.facebook.net/D58929
-
由 Andrew Kryczka 提交于
Summary: Made it consistent with posix Env, which uses pread() that returns 0 (success) when an offset is given beyond EOF. The purpose of making these Envs behave consistently is I am repurposing the in-memory Envs' tests for the basic Env tests in D58635. Test Plan: ran mock_env_test and memenv_test Reviewers: IslamAbdelRahman, lightmark, sdong Reviewed By: sdong Subscribers: andrewkr, dhruba, leveldb Differential Revision: https://reviews.facebook.net/D58845
-
由 sdong 提交于
Summary: Build failure with some compiler setting with tools/reduce_levels_test.cc:97: undefined reference to `rocksdb::LDBCommand* rocksdb::LDBCommand::InitFromCmdLineArgs<rocksdb::LDBCommand* (*)(std::string const&, std::vector<std::string, std::allocator<std::string> > const&, std::map<std::string, std::string, std::less<std::string>, std::allocator<std::pair<std::string const, std::string> > > const&, std::vector<std::string, std::allocator<std::string> > const&)>(std::vector<std::string, std::allocator<std::string> > const&, rocksdb::Options const&, rocksdb::LDBOptions const&, std::vector<rocksdb::ColumnFamilyDescriptor, std::allocator<rocksdb::ColumnFamilyDescriptor> > const*, rocksdb::LDBCommand* (*)(std::string const&, std::vector<std::string, std::allocator<std::string> > const&, std::map<std::string, std::string, std::less<std::string>, std::allocator<std::pair<std::string const, std::string> > > const&, std::vector<std::string, std::allocator<std::string> > const&))' Fix it by changing to function pointer instead Test Plan: Run all existing tests Reviewers: andrewkr, kradhakrishnan, IslamAbdelRahman Reviewed By: IslamAbdelRahman Subscribers: adsharma, lightmark, yiwu, leveldb, andrewkr, dhruba Differential Revision: https://reviews.facebook.net/D58905
-
由 Vasile Paraschiv 提交于
Summary: Add under and over limits for rocksdb::PerfLevel enum to allow us to do boundary checks before casting ints or unints to this enum. Test Plan: make all check -j32 Reviewers: sdong Reviewed By: sdong Subscribers: andrewkr, dhruba, leveldb Differential Revision: https://reviews.facebook.net/D58521
-
由 sdong 提交于
Summary: util/threadpool.cc's function name is the same as a well-known class name. It breaks unity build. Rename it. Test Plan: Run all existing test. Reviewers: yiwu, IslamAbdelRahman Reviewed By: IslamAbdelRahman Subscribers: leveldb, andrewkr, dhruba Differential Revision: https://reviews.facebook.net/D58881
-
- 27 5月, 2016 2 次提交
-
-
由 sdong 提交于
Summary: I don't think util/mutable_cf_options.h is needed inside include/rocksdb/sst_file_writer.h and it is not allowed. Remove it. Test Plan: Run all tests Reviewers: IslamAbdelRahman Reviewed By: IslamAbdelRahman Subscribers: lightmark, leveldb, andrewkr, dhruba Differential Revision: https://reviews.facebook.net/D58827
-
由 sdong 提交于
Summary: We added more table properties for each SST file, so when using 2KB SST file size, the estimated size of SST files is off by almost half, causing the LSM tree structure not as expected. Fix it by making file size 4x as previously, as well as LSM base size. Also avoid the sleeping based synchronization and turn to use sync points. Test Plan: Run paralell unit tests multiple times and make sure they always pass. Reviewers: IslamAbdelRahman, kradhakrishnan Reviewed By: kradhakrishnan Subscribers: leveldb, andrewkr, dhruba Differential Revision: https://reviews.facebook.net/D58749
-
- 25 5月, 2016 2 次提交
-
-
由 krad 提交于
-
由 Islam AbdelRahman 提交于
Summary: Fix simple issue with FLAGS_simcache_size condition Test Plan: run db_bench Reviewers: lightmark Reviewed By: lightmark Subscribers: andrewkr, dhruba Differential Revision: https://reviews.facebook.net/D58743
-
- 24 5月, 2016 6 次提交
-
-
由 Aaron Gao 提交于
Summary: add class SimCache(base class with instrumentation api) and SimLRUCache(derived class with detailed implementation) which is used as an instrumented block cache that can predict hit rate for different cache size Test Plan: Add a test case in `db_block_cache_test.cc` called `SimCacheTest` to test basic logic of SimCache. Also add option `-simcache_size` in db_bench. if set with a value other than -1, then the benchmark will use this value as the size of the simulator cache and finally output the simulation result. ``` [gzh@dev9927.prn1 ~/local/rocksdb] ./db_bench -benchmarks "fillseq,readrandom" -cache_size 1000000 -simcache_size 1000000 RocksDB: version 4.8 Date: Tue May 17 16:56:16 2016 CPU: 32 * Intel(R) Xeon(R) CPU E5-2660 0 @ 2.20GHz CPUCache: 20480 KB Keys: 16 bytes each Values: 100 bytes each (50 bytes after compression) Entries: 1000000 Prefix: 0 bytes Keys per prefix: 0 RawSize: 110.6 MB (estimated) FileSize: 62.9 MB (estimated) Write rate: 0 bytes/second Compression: Snappy Memtablerep: skip_list Perf Level: 0 WARNING: Assertions are enabled; benchmarks unnecessarily slow ------------------------------------------------ DB path: [/tmp/rocksdbtest-112628/dbbench] fillseq : 6.809 micros/op 146874 ops/sec; 16.2 MB/s DB path: [/tmp/rocksdbtest-112628/dbbench] readrandom : 6.343 micros/op 157665 ops/sec; 17.4 MB/s (1000000 of 1000000 found) SIMULATOR CACHE STATISTICS: SimCache LOOKUPs: 986559 SimCache HITs: 264760 SimCache HITRATE: 26.84% [gzh@dev9927.prn1 ~/local/rocksdb] ./db_bench -benchmarks "fillseq,readrandom" -cache_size 1000000 -simcache_size 10000000 RocksDB: version 4.8 Date: Tue May 17 16:57:10 2016 CPU: 32 * Intel(R) Xeon(R) CPU E5-2660 0 @ 2.20GHz CPUCache: 20480 KB Keys: 16 bytes each Values: 100 bytes each (50 bytes after compression) Entries: 1000000 Prefix: 0 bytes Keys per prefix: 0 RawSize: 110.6 MB (estimated) FileSize: 62.9 MB (estimated) Write rate: 0 bytes/second Compression: Snappy Memtablerep: skip_list Perf Level: 0 WARNING: Assertions are enabled; benchmarks unnecessarily slow ------------------------------------------------ DB path: [/tmp/rocksdbtest-112628/dbbench] fillseq : 5.066 micros/op 197394 ops/sec; 21.8 MB/s DB path: [/tmp/rocksdbtest-112628/dbbench] readrandom : 6.457 micros/op 154870 ops/sec; 17.1 MB/s (1000000 of 1000000 found) SIMULATOR CACHE STATISTICS: SimCache LOOKUPs: 1059764 SimCache HITs: 374501 SimCache HITRATE: 35.34% [gzh@dev9927.prn1 ~/local/rocksdb] ./db_bench -benchmarks "fillseq,readrandom" -cache_size 1000000 -simcache_size 100000000 RocksDB: version 4.8 Date: Tue May 17 16:57:32 2016 CPU: 32 * Intel(R) Xeon(R) CPU E5-2660 0 @ 2.20GHz CPUCache: 20480 KB Keys: 16 bytes each Values: 100 bytes each (50 bytes after compression) Entries: 1000000 Prefix: 0 bytes Keys per prefix: 0 RawSize: 110.6 MB (estimated) FileSize: 62.9 MB (estimated) Write rate: 0 bytes/second Compression: Snappy Memtablerep: skip_list Perf Level: 0 WARNING: Assertions are enabled; benchmarks unnecessarily slow ------------------------------------------------ DB path: [/tmp/rocksdbtest-112628/dbbench] fillseq : 5.632 micros/op 177572 ops/sec; 19.6 MB/s DB path: [/tmp/rocksdbtest-112628/dbbench] readrandom : 6.892 micros/op 145094 ops/sec; 16.1 MB/s (1000000 of 1000000 found) SIMULATOR CACHE STATISTICS: SimCache LOOKUPs: 1150767 SimCache HITs: 1034535 SimCache HITRATE: 89.90% ``` Reviewers: IslamAbdelRahman, andrewkr, sdong Reviewed By: sdong Subscribers: MarkCallaghan, andrewkr, dhruba, leveldb Differential Revision: https://reviews.facebook.net/D57999
-
由 omegaga 提交于
Summary: Update CMakeLists.txt for added db_io_failure_test. Depends on D58341 Test Plan: Run make check to see if the tests are working properly. Reviewers: sdong Reviewed By: sdong Subscribers: andrewkr, dhruba Differential Revision: https://reviews.facebook.net/D58671
-
由 krad 提交于
Summary: O_DIRECT is not available in Mac as a flag for open. The fix is to make use of fctl after the file is opened Test Plan: Run the tests on mac and Linux Reviewers: sdong Subscribers: andrewkr, dhruba, leveldb Differential Revision: https://reviews.facebook.net/D58665
-
由 Ashish Shenoy 提交于
Summary: Make CompactionOptionsFIFO a part of mutable_cf_options Test Plan: UT Reviewers: sdong Reviewed By: sdong Subscribers: andrewkr, lgalanis, dhruba Differential Revision: https://reviews.facebook.net/D58653
-
由 Shen Li 提交于
-
由 krad 提交于
Summary: This patch adds direct IO capability to RocksDB Env. The direct IO capability is required for persistent cache since NVM is best accessed as 4K direct IO. SSDs can leverage direct IO for reading. Direct IO requires the offset and size be sector size aligned, and memory to be kernel page aligned. Since neither RocksDB/Persistent read cache data layout is aligned to sector size, the code can accommodate reading unaligned IO size (or unaligned memory) at the cost of an alloc/copy. The write code path expects the size and memory to be aligned. Test Plan: Run RocksDB unit tests Reviewers: sdong Subscribers: andrewkr, dhruba, leveldb Differential Revision: https://reviews.facebook.net/D57393
-
- 23 5月, 2016 7 次提交
-
-
由 siddontang 提交于
-
由 Sage Weil 提交于
kPointInTimeRecovery is indistinguishable from kTolerateCorruptedTailRecords in recycle mode since we define the "end" of the log as the first corrupt record we encounter. kAbsoluteConsistency doesn't make sense because even a clean shutdown leaves old junk at the end of the log file. Signed-off-by: NSage Weil <sage@redhat.com>
-
由 Sage Weil 提交于
These vary only by the corruption string reported. Signed-off-by: NSage Weil <sage@redhat.com>
-
由 Sage Weil 提交于
If we are in kTolerateCorruptedTailRecords, treat these errors as the end of the log. This is particularly important for recycled logs, where we will regularly see corrupted headers (bad length or checksum) when replaying a log. If we are aligned with a block boundary or get lucky, we will land on an old header and see the log number mismatch, but more commonly we will land midway through some previous block and record and effectively see noise. These must be treated as the end of the log in order for recycling to work. This makes the LogTest.Recycle/1 test pass. We also modify a number of existing tests because the recycled log files behave fundamentally differently in that they always stop when they reach the first bad record. Signed-off-by: NSage Weil <sage@redhat.com>
-
由 Sage Weil 提交于
The behavior here needs to depend on the WAL recovery mode. No functional change in this patch. Signed-off-by: NSage Weil <sage@redhat.com>
-
由 Sage Weil 提交于
This currently fails because we do not properly map a corrupt header to the logical end of the log. Signed-off-by: NSage Weil <sage@redhat.com>
-
由 krad 提交于
Summary: Persistent read cache isn't very applicable for lite builds. Wrapping the code with #ifndef ROCKSDB_LITE .. #endif Test Plan: Run unit, lite, lite_test Reviewers: sdong Subscribers: andrewkr, dhruba, leveldb Differential Revision: https://reviews.facebook.net/D58563
-
- 21 5月, 2016 2 次提交
-
-
由 sdong 提交于
Summary: Deprecate this one option and delete code and tests that are now superfluous. Test Plan: all tests pass Reviewers: igor, yhchiang, IslamAbdelRahman Reviewed By: IslamAbdelRahman Subscribers: msalib, leveldb, andrewkr, dhruba Differential Revision: https://reviews.facebook.net/D55317
-
由 sdong 提交于
Summary: Google C++ Style writes: In particular, prefer to write lambda captures explicitly when capturing this or if the lambda will escape the current scope. Here it is the case for both. Test Plan: Run all test suites. Reviewers: andrewkr, dhruba Reviewed By: andrewkr, dhruba Subscribers: yhchiang, IslamAbdelRahman, leveldb, andrewkr, dhruba Differential Revision: https://reviews.facebook.net/D58515
-
- 20 5月, 2016 7 次提交
-
-
由 Aaron Orenstein 提交于
Summary: Reduce use of argument-dependent name lookup in RocksDB. Test Plan: 'make check' passed. Reviewers: andrewkr Reviewed By: andrewkr Subscribers: leveldb, andrewkr, dhruba Differential Revision: https://reviews.facebook.net/D58203
-
由 Dmitri Smirnov 提交于
For ease of reuse and customization as a library without wrapping. WinEnvThreads is a class for replacement. WintEnvIO is a class for reuse and behavior override. Added private virtual functions for custom override of fallocate pread for io classes.
-
由 krad 提交于
Summary: Persistent read cache relies on the accuracy of the GetUniqueIdFromFile to generate a unique key for a given block of data. Currently we don't have an implementation for Mac. This patch adds an implementation. Test Plan: Run unit tests Reviewers: sdong Subscribers: andrewkr, dhruba, leveldb Differential Revision: https://reviews.facebook.net/D58413
-
由 Islam AbdelRahman 提交于
Summary: Introduce MaxOperator a simple merge operator that return the max of all operands. This merge operand help me in benchmarking Test Plan: Add new unitttests Reviewers: sdong, andrewkr, yhchiang Reviewed By: yhchiang Subscribers: andrewkr, dhruba Differential Revision: https://reviews.facebook.net/D57873
-
由 Richard Cairns Jr 提交于
Summary: A couple of notes from the diff: - The namespace block I added at the top of table_properties_collector.cc was in reaction to an issue i was having with PutVarint64 and reusing the "val" string. I'm not sure this is the cleanest way of doing this, but abstracting this out at least results in the correct behavior. - I chose "rocksdb.merge.operands" as the property name. I am open to suggestions for better names. - The change to sst_dump_tool.cc seems a bit inelegant to me. Is there a better way to do the if-else block? Test Plan: I added a test case in table_properties_collector_test.cc. It adds two merge operands and checks to make sure that both of them are reflected by GetMergeOperands. It also checks to make sure the wasPropertyPresent bool is properly set in the method. Running both of these tests should pass: ./table_properties_collector_test ./sst_dump_test Reviewers: IslamAbdelRahman Reviewed By: IslamAbdelRahman Subscribers: andrewkr, dhruba Differential Revision: https://reviews.facebook.net/D58119
-
由 Evan Shaw 提交于
A couple "New Features" headers needed a blank line before them.
-
由 Dmitri Smirnov 提交于
Add db_test2
-
- 19 5月, 2016 3 次提交
-
-
由 omegaga 提交于
Summary: This is a part of effort to reduce the size of db_test.cc. We move the following tests to a separate file `db_io_failure_test.cc`: * DropWrites * DropWritesFlush * NoSpaceCompactRange * NonWritableFileSystem * ManifestWriteError * PutFailsParanoid Test Plan: Run `make check` to see if the tests are working properly. Reviewers: sdong, IslamAbdelRahman Reviewed By: IslamAbdelRahman Subscribers: andrewkr, dhruba Differential Revision: https://reviews.facebook.net/D58341
-
由 Islam AbdelRahman 提交于
Summary: Expose a simple function to convert CompressionType to it's corresponding option string This is for a diff @yoshinorim is working on for MyRocks Test Plan: unittest Reviewers: yhchiang, andrewkr, sdong Reviewed By: sdong Subscribers: andrewkr, dhruba, yoshinorim Differential Revision: https://reviews.facebook.net/D58215
-
由 Islam AbdelRahman 提交于
Summary: NotifyOnCompactionCompleted can unlock the mutex. That mean that we can schedule a background compaction that will start before we ReleaseCompactionFiles(). Test Plan: added unittest existing unittest Reviewers: yhchiang, sdong Reviewed By: sdong Subscribers: yoshinorim, andrewkr, dhruba Differential Revision: https://reviews.facebook.net/D58065
-
- 18 5月, 2016 5 次提交
-
-
由 Islam AbdelRahman 提交于
-
由 Reid Horuff 提交于
Summary: This tests that a prepared transaction is not lost after several crashes, restarts, and memtable flushes. Test Plan: TwoPhaseLongPrepareTest Reviewers: sdong Subscribers: hermanlee4, andrewkr, dhruba Differential Revision: https://reviews.facebook.net/D58185
-
由 Islam AbdelRahman 提交于
Summary: TransactionTest.TwoPhaseMultiThreadTest runs forever under TSAN and our CI builds time out looks like the reason is that some threads keep running and other threads dont get a chance to increment the counter Test Plan: run the test under TSAN Reviewers: sdong, horuff Reviewed By: horuff Subscribers: andrewkr, dhruba Differential Revision: https://reviews.facebook.net/D58359
-
由 krad 提交于
Summary: We expect the persistent read cache to perform at speeds upto 8 GB/s. In order to accomplish that, we need build a index mechanism which operate in the order of multiple millions per sec rate. This patch provide the basic data structure to accomplish that: (1) Hash table implementation with lock contention spread It is based on the StripedHashSet<T> implementation in The Art of multiprocessor programming by Maurice Henry & Nir Shavit (2) LRU implementation Place holder algorithm for further optimizing (3) Evictable Hash Table implementation Building block for building index data structure that evicts data like files etc TODO: (1) Figure if the sharded hash table and LRU can be used instead (2) Figure if we need to support configurable eviction algorithm for EvictableHashTable Test Plan: Run unit tests Subscribers: andrewkr, dhruba, leveldb Differential Revision: https://reviews.facebook.net/D55785
-
由 Aaron Gao 提交于
Summary: make more ColumnFamilyOptions dynamic: - compression - soft_pending_compaction_bytes_limit - hard_pending_compaction_bytes_limit - min_partial_merge_operands - report_bg_io_stats - paranoid_file_checks Test Plan: Add sanity check in `db_test.cc` for all above options except for soft_pending_compaction_bytes_limit and hard_pending_compaction_bytes_limit. All passed. Reviewers: andrewkr, sdong, IslamAbdelRahman Reviewed By: IslamAbdelRahman Subscribers: andrewkr, dhruba, leveldb Differential Revision: https://reviews.facebook.net/D57519
-
- 17 5月, 2016 1 次提交
-
-
由 Dmitri Smirnov 提交于
Implement GetUniqueIdFromFile to support new tests and the feature.
-