- 17 10月, 2012 1 次提交
-
-
由 Dhruba Borthakur 提交于
Summary: The method DeleteObsolete files is a very costly methind, especially when the number of files in a system is large. It makes a list of all live-files and then scans the directory to compute the diff. By default, this method is executed after every compaction run. This patch makes it such that DeleteObsolete files is never invoked twice within a configured period. Test Plan: run all unit tests Reviewers: heyongqiang, MarkCallaghan Reviewed By: MarkCallaghan Differential Revision: https://reviews.facebook.net/D6045
-
- 16 10月, 2012 1 次提交
-
-
由 Dhruba Borthakur 提交于
Summary: Enhance db_bench to allow setting the number of levels in a database. Test Plan: run db_bench and look at LOG Reviewers: heyongqiang, MarkCallaghan Reviewed By: MarkCallaghan CC: MarkCallaghan Differential Revision: https://reviews.facebook.net/D6027
-
- 04 10月, 2012 3 次提交
-
-
由 Dhruba Borthakur 提交于
Summary: We have seen that reading data via the pread call (instead of mmap) is much faster on Linux 2.6.x kernels. This patch makes an equivalent option to switch off mmaps for the write path as well. db_bench --mmap_write=0 will use write() instead of mmap() to write data to a file. This change is backward compatible, the default option is to continue using mmap for writing to a file. Test Plan: "make check all" Differential Revision: https://reviews.facebook.net/D5781
-
由 Mark Callaghan 提交于
Summary: The option is zero by default and in that case reporting is unchanged. By unchanged, the interval at which stats are reported is scaled after each report and newline is not issued after each report so one line is rewritten. When non-zero it specifies the constant interval (in operations) at which statistics are reported and the stats include the rate per interval. This makes it easier to determine whether QPS changes over the duration of the test. Task ID: # Blame Rev: Test Plan: run db_bench Revert Plan: Database Impact: Memcache Impact: Other Notes: EImportant: - begin *PUBLIC* platform impact section - Bugzilla: # - end platform impact - Reviewers: dhruba Reviewed By: dhruba CC: heyongqiang Differential Revision: https://reviews.facebook.net/D5817
-
由 Mark Callaghan 提交于
Summary: see above Task ID: # Blame Rev: Test Plan: run db_bench with invalid value for option Revert Plan: Database Impact: Memcache Impact: Other Notes: EImportant: - begin *PUBLIC* platform impact section - Bugzilla: # - end platform impact - Reviewers: dhruba Reviewed By: dhruba CC: heyongqiang Differential Revision: https://reviews.facebook.net/D5823
-
- 03 10月, 2012 1 次提交
-
-
由 Mark Callaghan 提交于
Summary: stdlib.h is needed for exit() --readhead --> --readahead Task ID: # Blame Rev: Test Plan: compile Revert Plan: Database Impact: Memcache Impact: Other Notes: EImportant: - begin *PUBLIC* platform impact section - Bugzilla: # - end platform impact - fix compiler warnings & errors Reviewers: dhruba Reviewed By: dhruba CC: heyongqiang Differential Revision: https://reviews.facebook.net/D5805
-
- 02 10月, 2012 1 次提交
-
-
由 Abhishek Kona 提交于
Summary: A simple CLI which calles DB->CompactRange() Can take String key's as range. Test Plan: Inserted data into a table. Waited for a minute, used compact tool on it. File modification time's changed so Compact did something on the files. Existing unit tests work. Reviewers: heyongqiang, dhruba Reviewed By: dhruba Differential Revision: https://reviews.facebook.net/D5697
-
- 29 9月, 2012 1 次提交
-
-
由 Dhruba Borthakur 提交于
Summary: In the current code, a Get() call can trigger compaction if it has to look at more than one file. This causes unnecessary compaction because looking at more than one file is a penalty only if the file is not yet in the cache. Also, th current code counts these files before the bloom filter check is applied. This patch counts a 'seek' only if the file fails the bloom filter check and has to read in data block(s) from the storage. This patch also counts a 'seek' if a file is not present in the file-cache, because opening a file means that its index blocks need to be read into cache. Test Plan: unit test attached. I will probably add one more unti tests. Reviewers: heyongqiang Reviewed By: heyongqiang CC: MarkCallaghan Differential Revision: https://reviews.facebook.net/D5709
-
- 26 9月, 2012 1 次提交
-
-
由 Dhruba Borthakur 提交于
Summary: If ReadCompaction is switched off, then it is better to not even submit background compaction jobs. I see about 3% increase in read-throughput on a pure memory database. Test Plan: run db_bench Reviewers: heyongqiang Reviewed By: heyongqiang Differential Revision: https://reviews.facebook.net/D5673
-
- 25 9月, 2012 1 次提交
-
-
由 Dhruba Borthakur 提交于
Summary: The GetLiveFiles() api lists the set of sst files and the current MANIFEST file. But the database continues to append new data to the MANIFEST file even when the application is backing it up to the backup location. This means that the database-version that is stored in the MANIFEST FILE in the backup location does not correspond to the sst files returned by GetLiveFiles. This API adds a new parameter to GetLiveFiles. This new parmeter returns the current size of the MANIFEST file. Test Plan: Unit test attached. Reviewers: heyongqiang Reviewed By: heyongqiang Differential Revision: https://reviews.facebook.net/D5631
-
- 22 9月, 2012 1 次提交
-
-
由 Dhruba Borthakur 提交于
Summary: The code was allocating 200 bytes on the stack but it writes 256 bytes into the array. x8a8ea5 std::_Rb_tree<>::erase() @ 0x7f134bee7eb0 (unknown) @ 0x8a8ea5 std::_Rb_tree<>::erase() @ 0x8a35d6 leveldb::DBImpl::CleanupCompaction() @ 0x8a7810 leveldb::DBImpl::BackgroundCompaction() @ 0x8a804d leveldb::DBImpl::BackgroundCall() @ 0x8c4eff leveldb::(anonymous namespace)::PosixEnv::BGThreadWrapper() @ 0x7f134b3c010d start_thread @ 0x7f134bf9f10d clone Test Plan: run db_bench with overwrite option Reviewers: heyongqiang Reviewed By: heyongqiang Differential Revision: https://reviews.facebook.net/D5595
-
- 19 9月, 2012 1 次提交
-
-
由 Dhruba Borthakur 提交于
Summary: Print out the compile version in the LOG. Test Plan: run dbbench and verify LOG Reviewers: heyongqiang Reviewed By: heyongqiang Differential Revision: https://reviews.facebook.net/D5529
-
- 18 9月, 2012 2 次提交
-
-
由 heyongqiang 提交于
Summary: as subject. This diff should be good for benchmarking. will send another diff to make it better in the case the seek compaction is enable. In that coming diff, will not count a seek if the bloomfilter filters. Test Plan: build Reviewers: dhruba, MarkCallaghan Reviewed By: MarkCallaghan Differential Revision: https://reviews.facebook.net/D5481
-
由 Dhruba Borthakur 提交于
Summary: A set of apis that allows an application to backup data from the leveldb database based on a set of files. Test Plan: unint test attached. more coming soon. Reviewers: heyongqiang Reviewed By: heyongqiang Differential Revision: https://reviews.facebook.net/D5439
-
- 17 9月, 2012 2 次提交
-
-
由 heyongqiang 提交于
Summary: as subject. this can be used for benchmarking. If we want it for some cases, we can do more changes to make this part of the option. Test Plan: db_test Reviewers: dhruba CC: MarkCallaghan Differential Revision: https://reviews.facebook.net/D5451
-
由 heyongqiang 提交于
Summary: as subject Test Plan: build Reviewers: dhruba Differential Revision: https://reviews.facebook.net/D5469
-
- 15 9月, 2012 2 次提交
-
-
由 Mark Callaghan 提交于
Summary: FLAGS_cache_size is a long, no need to scan %lld into a size_t for it (which generates a compiler warning) Test Plan: run db_bench Reviewers: dhruba, heyongqiang Reviewed By: heyongqiang CC: heyongqiang Differential Revision: https://reviews.facebook.net/D5427
-
由 Mark Callaghan 提交于
Summary: This adds an option to db_bench to specify the compression algorithm to use for LevelDB Test Plan: ran db_bench Reviewers: dhruba Reviewed By: dhruba Differential Revision: https://reviews.facebook.net/D5421
-
- 14 9月, 2012 2 次提交
-
-
由 Dhruba Borthakur 提交于
Summary: Ability to switch off filesystem read-aheads. This change is backward-compatible: the default setting is to allow file system read-aheads. Test Plan: run benchmarks Reviewers: heyongqiang, adsharma Reviewed By: heyongqiang Differential Revision: https://reviews.facebook.net/D5391
-
由 Dhruba Borthakur 提交于
Summary: Enable db_bench to specify block size. Test Plan: compile and run Reviewers: heyongqiang Reviewed By: heyongqiang Differential Revision: https://reviews.facebook.net/D5373
-
- 13 9月, 2012 1 次提交
-
-
由 Dhruba Borthakur 提交于
Summary: Fix compiler warnings. Use uint64_t instead of uint. Test Plan: build using -Wall Reviewers: heyongqiang Reviewed By: heyongqiang Differential Revision: https://reviews.facebook.net/D5355
-
- 07 9月, 2012 2 次提交
-
-
由 heyongqiang 提交于
Summary: added a new option db_log_dir, which points the log dir. Inside that dir, in order to make log names unique, the log file name is prefixed with the leveldb data dir absolute path. Test Plan: db_test Reviewers: dhruba Reviewed By: dhruba Differential Revision: https://reviews.facebook.net/D5205
-
由 Dhruba Borthakur 提交于
Summary: If none of reads or writes are specified by user, then pick the FLAGS_NUM as the number of iterations in the ReadRandomWriteRandom test. If either reads or writes are defined, then use their maximum. Test Plan: run benchmark Reviewers: heyongqiang Reviewed By: heyongqiang Differential Revision: https://reviews.facebook.net/D5217
-
- 05 9月, 2012 1 次提交
-
-
由 Dhruba Borthakur 提交于
Summary: This patch enables the db_bench benchmark to issue both random reads and random writes at the same time. This options can be trigged via ./db_bench --benchmarks=readrandomwriterandom The default percetage of reads is 90. One can change the percentage of reads by specifying the --readwritepercent. ./db_bench --benchmarks=readrandomwriterandom=50 This is a feature request from Jeffro asking for leveldb performance with a 90:10 read:write ratio. Test Plan: run on test machine. Reviewers: heyongqiang Reviewed By: heyongqiang Differential Revision: https://reviews.facebook.net/D5067
-
- 30 8月, 2012 2 次提交
-
-
由 Dhruba Borthakur 提交于
Summary: Clean up compiler warnings generated by -Wall option. make clean all OPT=-Wall This is a pre-requisite before making a new release. Test Plan: compile and run unit tests Reviewers: heyongqiang Reviewed By: heyongqiang Differential Revision: https://reviews.facebook.net/D5019
-
由 Dhruba Borthakur 提交于
Summary: The numbers of shards that the block cache is divided into is configurable. However, if the user specifies that he/she wants the block cache to be divided into more than 2**20 pieces, then the system will rey to allocate a huge array of that size) that could fail. It is better to limit the sharding of the block cache to an upper bound. The default sharding is 16 shards (i.e. 2**4) and the maximum is now 2 million shards (i.e. 2**20). Also, fixed a bug with the LRUCache where the numShardBits should be a private member of the LRUCache object rather than a static variable. Test Plan: run db_bench with --cache_numshardbits=64. Task ID: # Blame Rev: Reviewers: heyongqiang Reviewed By: heyongqiang Differential Revision: https://reviews.facebook.net/D5013
-
- 29 8月, 2012 5 次提交
-
-
由 heyongqiang 提交于
Summary: as subject Test Plan: db_test table_test Reviewers: dhruba
-
由 heyongqiang 提交于
Summary: as subject. ported the change from google code leveldb 1.5 Test Plan: run db_test Reviewers: dhruba Differential Revision: https://reviews.facebook.net/D4839
-
由 heyongqiang 提交于
Summary: as subject Test Plan: run filename_test Reviewers: dhruba Differential Revision: https://reviews.facebook.net/D4965
-
由 heyongqiang 提交于
Summary: as subject Test Plan:db_test Reviewers: dhruba
-
由 heyongqiang 提交于
Summary: as subject Test Plan: db_test Reviewers: dhruba Reviewed By: dhruba Differential Revision: https://reviews.facebook.net/D4929
-
- 28 8月, 2012 1 次提交
-
-
由 Dhruba Borthakur 提交于
Summary: Introduce a new method Env->Fsync() that issues fsync (instead of fdatasync). This is needed for data durability when running on ext3 filesystems. Added options to the benchmark db_bench to generate performance numbers with either fsync or fdatasync enabled. Cleaned up Makefile to build leveldb_shell only when building the thrift leveldb server. Test Plan: build and run benchmark Reviewers: heyongqiang Reviewed By: heyongqiang Differential Revision: https://reviews.facebook.net/D4911
-
- 25 8月, 2012 4 次提交
-
-
由 heyongqiang 提交于
Summary: as subject add a tool to read sst file as subject. ./sst_reader --command=check --file= ./sst_reader --command=scan --file= Test Plan: db_test run this command Reviewers: dhruba Reviewed By: dhruba Differential Revision: https://reviews.facebook.net/D4881
-
由 heyongqiang 提交于
Summary: as subject add a tool to read sst file as subject. ./sst_reader --command=check --file= ./sst_reader --command=scan --file= Test Plan: db_test run this command Reviewers: dhruba Differential Revision: https://reviews.facebook.net/D4881
-
由 heyongqiang 提交于
Summary: as subject Test Plan:db_test Reviewers: dhruba
-
由 Dhruba Borthakur 提交于
Summary: ./manifest_dump --file=/tmp/dbbench/MANIFEST-000002 Output looks like manifest_file_number 30 next_file_number 31 last_sequence 388082 log_number 28 prev_log_number 0 --- level 0 --- --- level 1 --- --- level 2 --- 5:3244155['0000000000000000' @ 1 : 1 .. '0000000000028220' @ 28221 : 1] 7:3244177['0000000000028221' @ 28222 : 1 .. '0000000000056441' @ 56442 : 1] 9:3244156['0000000000056442' @ 56443 : 1 .. '0000000000084662' @ 84663 : 1] 11:3244178['0000000000084663' @ 84664 : 1 .. '0000000000112883' @ 112884 : 1] 13:3244158['0000000000112884' @ 112885 : 1 .. '0000000000141104' @ 141105 : 1] 15:3244176['0000000000141105' @ 141106 : 1 .. '0000000000169325' @ 169326 : 1] 17:3244156['0000000000169326' @ 169327 : 1 .. '0000000000197546' @ 197547 : 1] 19:3244178['0000000000197547' @ 197548 : 1 .. '0000000000225767' @ 225768 : 1] 21:3244155['0000000000225768' @ 225769 : 1 .. '0000000000253988' @ 253989 : 1] 23:3244179['0000000000253989' @ 253990 : 1 .. '0000000000282209' @ 282210 : 1] 25:3244157['0000000000282210' @ 282211 : 1 .. '0000000000310430' @ 310431 : 1] 27:3244176['0000000000310431' @ 310432 : 1 .. '0000000000338651' @ 338652 : 1] 29:3244156['0000000000338652' @ 338653 : 1 .. '0000000000366872' @ 366873 : 1] --- level 3 --- --- level 4 --- --- level 5 --- --- level 6 --- Test Plan: run on test directory created by dbbench Reviewers: heyongqiang Reviewed By: heyongqiang CC: hustliubo Differential Revision: https://reviews.facebook.net/D4743
-
- 23 8月, 2012 2 次提交
-
-
由 Dhruba Borthakur 提交于
Summary: Log the open-options to the LOG. Use options_ instead of options because SanitizeOptions could modify the max_file_open limit. Test Plan: num db_bench Reviewers: heyongqiang Reviewed By: heyongqiang Differential Revision: https://reviews.facebook.net/D4833
-
由 heyongqiang 提交于
Summary: as subject Test Plan: manually run db_bench confirmed Reviewers: dhruba Reviewed By: dhruba Differential Revision: https://reviews.facebook.net/D4809
-
- 22 8月, 2012 2 次提交
-
-
由 heyongqiang 提交于
Summary: as subject Test Plan: manually run db_bench confirmed Reviewers: dhruba Differential Revision: https://reviews.facebook.net/D4809
-
由 Dhruba Borthakur 提交于
Summary: Fixed unit test c_test by initializing logger=NULL. Removed "atomic" from last_log_ts so that unit tests do not require C11 compiler. Anyway, last_log_ts is mostly used for logging, so it is ok if it is loosely accurate. Test Plan: run c_test Reviewers: heyongqiang Reviewed By: heyongqiang Differential Revision: https://reviews.facebook.net/D4803
-