- 07 3月, 2013 2 次提交
-
-
由 Abhishek Kona 提交于
Summary: Store the last flushed, seq no. in db_impl. Check against it in transaction Log iterator. Do not attempt to read ahead if we do not know if the data is flushed completely. Does not work if flush is disabled. Any ideas on fixing that? * Minor change, iter->Next is called the first time automatically for * the first time. Test Plan: existing test pass. More ideas on testing this? Planning to run some stress test. Reviewers: dhruba, heyongqiang CC: leveldb Differential Revision: https://reviews.facebook.net/D9087
-
由 Dhruba Borthakur 提交于
Summary: The compaction process zeros out sequence numbers if the output is part of the bottommost level. The Slice is supposed to refer to an immutable data buffer. The merger that implements the priority queue while reading kvs as the input of a compaction run reies on this fact. The bug was that were updating the sequence number of a record in-place and that was causing suceeding invocations of the merger to return kvs in arbitrary order of sequence numbers. The fix is to copy the key to a local memory buffer before setting its seqno to 0. Test Plan: Set Options.purge_redundant_kvs_while_flush = false and then run db_stress --ops_per_thread=1000 --max_key=320 Reviewers: emayanke, sheki Reviewed By: emayanke CC: leveldb Differential Revision: https://reviews.facebook.net/D9147
-
- 06 3月, 2013 3 次提交
-
-
由 Abhishek Kona 提交于
-
由 Abhishek Kona 提交于
Test Plan: bash -n Reviewers: emayanke Differential Revision: https://reviews.facebook.net/D9129
-
由 Dhruba Borthakur 提交于
Summary: When we use -O3, the gcc 4.7.1 compiler generates 'pinsrd' which is not supported on machines with "vendor_id : AuthenticAMD". Previous release of rocksdb used -O2. Optimization -O2 was introduced at https://github.com/facebook/rocksdb/commit/772f75b3fbc5cfcf4d519114751efeae04411fa1 Test Plan: make check Reviewers: chip, heyongqiang, sheki Reviewed By: sheki CC: leveldb Differential Revision: https://reviews.facebook.net/D9093
-
- 05 3月, 2013 2 次提交
-
-
由 Zheng Shao 提交于
Summary: Add a shortcut function to make it easier for people to efficiently bulk_load data into RocksDB. Test Plan: Tried ldb with "--bulk_load" and "--bulk_load --compact" and verified the outcome. Needs to consult the team on how to test this automatically. Reviewers: sheki, dhruba, emayanke, heyongqiang Reviewed By: dhruba CC: leveldb Differential Revision: https://reviews.facebook.net/D8907
-
由 Dhruba Borthakur 提交于
Summary: TableCache->file is not used. remove it. I kept the TableAndFile structure and will clean it up in a future patch. Test Plan: make clean check Reviewers: sheki, chip Reviewed By: chip CC: leveldb Differential Revision: https://reviews.facebook.net/D9075
-
- 04 3月, 2013 2 次提交
-
-
由 Mark Callaghan 提交于
Summary: This adds the rate_delay_limit_milliseconds option to make the delay configurable in MakeRoomForWrite when the max compaction score is too high. This delay is called the Ln slowdown. This change also counts the Ln slowdown per level to make it possible to see where the stalls occur. From IO-bound performance testing, the Level N stalls occur: * with compression -> at the largest uncompressed level. This makes sense because compaction for compressed levels is much slower. When Lx is uncompressed and Lx+1 is compressed then files pile up at Lx because the (Lx,Lx+1)->Lx+1 compaction process is the first to be slowed by compression. * without compression -> at level 1 Task ID: #1832108 Blame Rev: Test Plan: run with real data, added test Revert Plan: Database Impact: Memcache Impact: Other Notes: EImportant: - begin *PUBLIC* platform impact section - Bugzilla: # - end platform impact - Reviewers: dhruba Reviewed By: dhruba Differential Revision: https://reviews.facebook.net/D9045
-
由 Dhruba Borthakur 提交于
Summary: Rocks accumulates recent writes and deletes in the in-memory memtable. When the memtable is full, it writes the contents on the memtable to a file in L0. This patch removes redundant records at the time of the flush. If there are multiple versions of the same key in the memtable, then only the most recent one is dumped into the output file. The purging of redundant records occur only if the most recent snapshot is earlier than the earliest record in the memtable. Should we switch on this feature by default or should we keep this feature turned off in the default settings? Test Plan: Added test case to db_test.cc Reviewers: sheki, vamsi, emayanke, heyongqiang Reviewed By: sheki CC: leveldb Differential Revision: https://reviews.facebook.net/D8991
-
- 02 3月, 2013 2 次提交
-
-
由 bil 提交于
Summary: 1. the default value for key size is still 16 2. enable the ability to set the key size via command line --key_size= Test Plan: build & run db_banch and pass some value via command line. verify it works correctly. Reviewers: sheki Reviewed By: sheki CC: leveldb Differential Revision: https://reviews.facebook.net/D8943
-
由 amayank 提交于
Summary: The script valgrind_test.sh runs Valgrind for all tests in the makefile including leak-checks and outputs the logs for every test in a separate file with the name "valgrind_log_<testname>". It prints the failed tests in the file "valgrind_failed_tests". All these files are created in the directory "VALGRIND_LOGS" which can be changed in the Makefile. Finally it checks the line-count for the file "valgrind_failed_tests" and returns 0 if no tests failed and 1 otherwise. Test Plan: ./valgrind_test.sh; Changed the tests to incorporte leaks and verified correctness Reviewers: dhruba, sheki, MarkCallaghan Reviewed By: sheki CC: zshao Differential Revision: https://reviews.facebook.net/D8877
-
- 01 3月, 2013 1 次提交
-
-
由 Abhishek Kona 提交于
Summary: scripted NULL to nullptr in * include/leveldb/ * db/ * table/ * util/ Test Plan: make all check Reviewers: dhruba, emayanke Reviewed By: emayanke CC: leveldb Differential Revision: https://reviews.facebook.net/D9003
-
- 27 2月, 2013 1 次提交
-
-
由 Dhruba Borthakur 提交于
Summary: There was an artifical limit of 50K files per database. This is insifficient if the database is 1 TB in size and each file is 2 MB. Test Plan: make check Reviewers: sheki, emayanke Reviewed By: emayanke CC: leveldb Differential Revision: https://reviews.facebook.net/D8919
-
- 26 2月, 2013 1 次提交
-
-
由 Abhishek Kona 提交于
Summary: Use only the counter mechanism. Do away with incNumFileOpens, incNumFileClose, incNumFileErrors s/NULL/nullptr/g in db/table_cache.cc Test Plan: make clean check Reviewers: dhruba, heyongqiang, emayanke Reviewed By: dhruba CC: leveldb Differential Revision: https://reviews.facebook.net/D8841
-
- 23 2月, 2013 2 次提交
-
-
由 Vamsi Ponnekanti 提交于
Summary: Currently the test tracks all writes in memory and uses it for verification at the end. This has 4 problems: (a) It needs mutex for each write to ensure in-memory update and leveldb update are done atomically. This slows down the benchmark. (b) Verification phase at the end is time consuming as well (c) Does not test batch writes or snapshots (d) We cannot kill the test and restart multiple times in a loop because in-memory state will be lost. I am adding a FLAGS_multi that does MultiGet/MultiPut/MultiDelete instead of get/put/delete to get/put/delete a group of related keys with same values atomically. Every get retrieves the group of keys and checks that their values are same. This does not have the above problems but the downside is that it does less amount of validation than the other approach. Test Plan: This whole this is a test! Here is a small run. I am doing larger run now. [nponnekanti@dev902 /data/users/nponnekanti/rocksdb] ./db_stress --ops_per_thread=10000 --multi=1 --ops_per_key=25 LevelDB version : 1.5 Number of threads : 32 Ops per thread : 10000 Read percentage : 10 Delete percentage : 30 Max key : 2147483648 Num times DB reopens: 10 Num keys per lock : 4 Compression : snappy ------------------------------------------------ Creating 536870912 locks 2013/02/20-16:59:32 Starting database operations Created bg thread 0x7f9ebcfff700 2013/02/20-16:59:37 Reopening database for the 1th time 2013/02/20-16:59:46 Reopening database for the 2th time 2013/02/20-16:59:57 Reopening database for the 3th time 2013/02/20-17:00:11 Reopening database for the 4th time 2013/02/20-17:00:25 Reopening database for the 5th time 2013/02/20-17:00:36 Reopening database for the 6th time 2013/02/20-17:00:47 Reopening database for the 7th time 2013/02/20-17:00:59 Reopening database for the 8th time 2013/02/20-17:01:10 Reopening database for the 9th time 2013/02/20-17:01:20 Reopening database for the 10th time 2013/02/20-17:01:31 Reopening database for the 11th time 2013/02/20-17:01:31 Starting verification Stress Test : 109.125 micros/op 22191 ops/sec : Wrote 0.00 MB (0.23 MB/sec) (59% of 32 ops) : Deleted 10 times 2013/02/20-17:01:31 Verification successful Revert Plan: OK Task ID: # Reviewers: dhruba, emayanke Reviewed By: emayanke CC: leveldb Differential Revision: https://reviews.facebook.net/D8733
-
由 Abhishek Kona 提交于
Summary: just record time consumed in compaction Test Plan: compile Reviewers: dhruba Reviewed By: dhruba CC: leveldb Differential Revision: https://reviews.facebook.net/D8781
-
- 22 2月, 2013 5 次提交
-
-
由 amayank 提交于
Summary: Added automated valgrind testing for rocksdb by adding valgrind_check in the Makefile Test Plan: make clean; make all check Reviewers: dhruba, sheki, MarkCallaghan, zshao Reviewed By: sheki CC: leveldb Differential Revision: https://reviews.facebook.net/D8787
-
由 Abhishek Kona 提交于
Summary: * Counters for bytes read and write. as a part of this diff, I want to=> * Measure compaction times. @dhruba can you point which function, should * I time to get Compaction-times. Was looking at CompactRange. Test Plan: db_test Reviewers: dhruba, emayanke CC: leveldb Differential Revision: https://reviews.facebook.net/D8763
-
由 Vamsi Ponnekanti 提交于
Summary: I had added FLAGS_numdistinct and FLAGS_deletepercent for randomwithverify but forgot to add cmdline parsing for those flags. Test Plan: [nponnekanti@dev902 /data/users/nponnekanti/rocksdb] ./db_bench --benchmarks=randomwithverify --numdistinct=500 LevelDB: version 1.5 Date: Thu Feb 21 10:34:40 2013 CPU: 24 * Intel(R) Xeon(R) CPU X5650 @ 2.67GHz CPUCache: 12288 KB Keys: 16 bytes each Values: 100 bytes each (50 bytes after compression) Entries: 1000000 RawSize: 110.6 MB (estimated) FileSize: 62.9 MB (estimated) Compression: snappy WARNING: Assertions are enabled; benchmarks unnecessarily slow ------------------------------------------------ Created bg thread 0x7fbf90bff700 randomwithverify : 4.693 micros/op 213098 ops/sec; ( get:900000 put:80000 del:20000 total:1000000 found:714556) [nponnekanti@dev902 /data/users/nponnekanti/rocksdb] ./db_bench --benchmarks=randomwithverify --deletepercent=5 LevelDB: version 1.5 Date: Thu Feb 21 10:35:03 2013 CPU: 24 * Intel(R) Xeon(R) CPU X5650 @ 2.67GHz CPUCache: 12288 KB Keys: 16 bytes each Values: 100 bytes each (50 bytes after compression) Entries: 1000000 RawSize: 110.6 MB (estimated) FileSize: 62.9 MB (estimated) Compression: snappy WARNING: Assertions are enabled; benchmarks unnecessarily slow ------------------------------------------------ Created bg thread 0x7fe14dfff700 randomwithverify : 4.883 micros/op 204798 ops/sec; ( get:900000 put:50000 del:50000 total:1000000 found:443847) [nponnekanti@dev902 /data/users/nponnekanti/rocksdb] [nponnekanti@dev902 /data/users/nponnekanti/rocksdb] ./db_bench --benchmarks=randomwithverify --deletepercent=5 --numdistinct=500 LevelDB: version 1.5 Date: Thu Feb 21 10:36:18 2013 CPU: 24 * Intel(R) Xeon(R) CPU X5650 @ 2.67GHz CPUCache: 12288 KB Keys: 16 bytes each Values: 100 bytes each (50 bytes after compression) Entries: 1000000 RawSize: 110.6 MB (estimated) FileSize: 62.9 MB (estimated) Compression: snappy WARNING: Assertions are enabled; benchmarks unnecessarily slow ------------------------------------------------ Created bg thread 0x7fc31c7ff700 randomwithverify : 4.920 micros/op 203233 ops/sec; ( get:900000 put:50000 del:50000 total:1000000 found:445522) Revert Plan: OK Task ID: # Reviewers: dhruba, emayanke Reviewed By: dhruba CC: leveldb Differential Revision: https://reviews.facebook.net/D8769
-
由 amayank 提交于
Summary: Fixed a bug in the stress-test where the correct size was not being passed to GenerateValue. This bug was there since the beginning but assertions were switched on in our code-base only recently. Added comments on the top detailing how the stress test works and how to quicken/slow it down after investigation. Test Plan: make all check. ./db_stress Reviewers: dhruba, asad Reviewed By: dhruba CC: vamsi, sheki, heyongqiang, zshao Differential Revision: https://reviews.facebook.net/D8727
-
由 Vamsi Ponnekanti 提交于
Summary: Added RandomWithVerify benchmark option. Test Plan: This whole diff is to test. [nponnekanti@dev902 /data/users/nponnekanti/rocksdb] ./db_bench --benchmarks=randomwithverify LevelDB: version 1.5 Date: Tue Feb 19 17:50:28 2013 CPU: 24 * Intel(R) Xeon(R) CPU X5650 @ 2.67GHz CPUCache: 12288 KB Keys: 16 bytes each Values: 100 bytes each (50 bytes after compression) Entries: 1000000 RawSize: 110.6 MB (estimated) FileSize: 62.9 MB (estimated) Compression: snappy WARNING: Assertions are enabled; benchmarks unnecessarily slow ------------------------------------------------ Created bg thread 0x7fa9c3fff700 randomwithverify : 5.004 micros/op 199836 ops/sec; ( get:900000 put:80000 del:20000 total:1000000 found:711992) Revert Plan: OK Task ID: # Reviewers: dhruba, emayanke Reviewed By: dhruba CC: leveldb Differential Revision: https://reviews.facebook.net/D8685
-
- 21 2月, 2013 3 次提交
-
-
由 Abhishek Kona 提交于
Summary: LDB tool to print the deleted/put keys in hex in the wal file. Test Plan: run ldb on a db to check if output was satisfactory Reviewers: dhruba Reviewed By: dhruba CC: leveldb Differential Revision: https://reviews.facebook.net/D8691
-
由 amayank 提交于
Summary: Changed the Get and Scan options with openForReadOnly mode to have access to the memtable. Changed the visibility of NewInternalIterator in db_impl from private to protected so that the derived class db_impl_read_only can call that in its NewIterator function for the scan case. The previous approach which changed the default for flush_on_destroy_ from false to true caused many problems in the unit tests due to empty sst files that it created. All unit tests pass now. Test Plan: make clean; make all check; ldb put and get and scans Reviewers: dhruba, heyongqiang, sheki Reviewed By: dhruba CC: kosievdmerwe, zshao, dilipj, kailiu Differential Revision: https://reviews.facebook.net/D8697
-
由 Abhishek Kona 提交于
Summary: * Introduce is histogram in statistics.h * stop watch to measure time. * introduce two timers as a poc. Replaced NULL with nullptr to fight some lint errors Should be useful for google. Test Plan: ran db_bench and check stats. make all check Reviewers: dhruba, heyongqiang Reviewed By: dhruba CC: leveldb Differential Revision: https://reviews.facebook.net/D8637
-
- 20 2月, 2013 1 次提交
-
-
由 Dhruba Borthakur 提交于
Summary: Cleanup README.fb Test Plan: Reviewers: CC: Task ID: # Blame Rev:
-
- 19 2月, 2013 6 次提交
-
-
由 Kai Liu 提交于
Summary: I missed InitTestDb() in one of my tess. InitTestDb() initializes the test directory, without which the test will throw IO error. This problem didn't occur before because I've already run the tests before so the test directory is already there. Test Plan: Reviewers: dhruba CC: Task ID: # Blame Rev:
-
由 Kai Liu 提交于
Summary: auto_roll_logger_test is failing because it cannot create the test dir, leading to IO error: /tmp/leveldbtest-6108/db_log_test/LOG: No such file or directory. I'll temporary remove the unit test and will revert the test this problem is solved. Test Plan: make all check Reviewers: dhruba CC: leveldb Task ID: # Blame Rev:
-
由 Dhruba Borthakur 提交于
Summary: c_test: db/filename.cc:74: std::string leveldb::DescriptorFileName(const string&,.... Test Plan: this is a failure in a unit test Differential Revision: https://reviews.facebook.net/D8667
-
由 Dhruba Borthakur 提交于
Summary: The sequence numbers in each record eat up plenty of space on storage. The optimization zeroes out sequence numbers on kvs in the Lmax layer that are earlier than the earliest snapshot. Test Plan: Unit test attached. Differential Revision: https://reviews.facebook.net/D8619
-
由 Dhruba Borthakur 提交于
Summary: Test Plan: Reviewers: CC: Task ID: # Blame Rev:
-
- 16 2月, 2013 1 次提交
-
-
由 amayank 提交于
Summary: flush_on_destroy has a default value of false and the memtable is flushed in the dbimpl-destructor only when that is set to true. Because we want the memtable to be flushed everytime that the destructor is called(db is closed) and the cases where we work with the memtable only are very less it is a good idea to give this a default value of true. Thus the put from ldb wil have its data flushed to disk in the destructor and the next Get will be able to read it when opened with OpenForReadOnly. The reason that ldb could read the latest value when the db was opened in the normal Open mode is that the Get from normal Open first reads the memtable and directly finds the latest value written there and the Get from OpenForReadOnly doesn't have access to the memtable (which is correct because all its Put/Modify) are disabled Test Plan: make all; ldb put and get and scans Reviewers: dhruba, heyongqiang, sheki Reviewed By: heyongqiang CC: kosievdmerwe, zshao, dilipj, kailiu Differential Revision: https://reviews.facebook.net/D8631
-
- 14 2月, 2013 1 次提交
-
-
由 Kai Liu 提交于
Summary: Fix the warning [-Werror=format-security] and [-Werror=unused-result]. Test Plan: enforced the Werror and run make Task ID: 2101673 Blame Rev: Reviewers: heyongqiang Differential Revision: https://reviews.facebook.net/D8553
-
- 08 2月, 2013 1 次提交
-
-
由 Chip Turner 提交于
Summary: $SUBJECT -- cosmetic fix for histograms, print P75/P99, and make sure zlib is enabled for our command line tools. Test Plan: compile, test db_bench with --compression_type=zlib Reviewers: heyongqiang Reviewed By: heyongqiang CC: adsharma, leveldb Differential Revision: https://reviews.facebook.net/D8445
-
- 05 2月, 2013 1 次提交
-
-
由 Kai Liu 提交于
Summary: * Add a SplitByTTLLogger to enable this feature. In this diff I implemented generalized AutoSplitLoggerBase class to simplify the development of such classes. * Refactor the existing AutoSplitLogger and fix several bugs. Test Plan: * Added a unit tests for different types of "auto splitable" loggers individually. * Tested the composited logger which allows the log files to be splitted by both TTL and log size. Reviewers: heyongqiang, dhruba Reviewed By: heyongqiang CC: zshao, leveldb Differential Revision: https://reviews.facebook.net/D8037
-
- 02 2月, 2013 1 次提交
-
-
由 Abhishek Kona 提交于
Summary: Just change some config. Test Plan: arc lint Reviewers: chip CC: leveldb Differential Revision: https://reviews.facebook.net/D8355
-
- 01 2月, 2013 3 次提交
-
-
由 Abhishek Kona 提交于
Summary: The existing code did not initialize a few doubles in histogram.cc. Cropped up when I wrote a unit-test. Test Plan: make all check Reviewers: chip Reviewed By: chip CC: leveldb Differential Revision: https://reviews.facebook.net/D8319
-
由 Abhishek Kona 提交于
Summary: Earlier way to record in histogram=> Linear search BucketLimit array to find the bucket and increment the counter Current way to record in histogram=> Store a HistMap statically which points the buckets of each value in the range [kFirstValue, kLastValue); In the proccess use vectors instead of array's and refactor some code to HistogramHelper class. Test Plan: run db_bench with histogram=1 and see a histogram being printed. Reviewers: dhruba, chip, heyongqiang Reviewed By: chip CC: leveldb Differential Revision: https://reviews.facebook.net/D8265
-
由 Kosie van der Merwe 提交于
Summary: Added function to `RandomAccessFile` to generate an unique ID for that file. Currently only `PosixRandomAccessFile` has this behaviour implemented and only on Linux. Changed how key is generated in `Table::BlockReader`. Added tests to check whether the unique ID is stable, unique and not a prefix of another unique ID. Added tests to see that `Table` uses the cache more efficiently. Test Plan: make check Reviewers: chip, vamsi, dhruba Reviewed By: chip CC: leveldb Differential Revision: https://reviews.facebook.net/D8145
-
- 29 1月, 2013 1 次提交
-
-
由 Chip Turner 提交于
Summary: fallocate is linux only, so let's protect it with ifdef's Test Plan: make Reviewers: sheki, dhruba Reviewed By: dhruba CC: leveldb Differential Revision: https://reviews.facebook.net/D8223
-