- 20 10月, 2012 2 次提交
-
-
由 Dhruba Borthakur 提交于
Summary: Leveldb code already calculates checksums for each block. There is no need to generate checksums inside zlib. This patch switches-off checksum generation/checking in zlib library. (The Inno support for zlib uses windowsBits=14 as well.) pfabricator marks this file as binary. But here is the diff diff --git a/port/port_posix.h b/port/port_posix.h index 86a0927..db4e0b8 100644 --- a/port/port_posix.h +++ b/port/port_posix.h @@ -163,7 +163,7 @@ inline bool Snappy_Uncompress(const char* input, size_t length, } inline bool Zlib_Compress(const char* input, size_t length, - ::std::string* output, int windowBits = 15, int level = -1, + ::std::string* output, int windowBits = -14, int level = -1, int strategy = 0) { #ifdef ZLIB // The memLevel parameter specifies how much memory should be allocated for @@ -223,7 +223,7 @@ inline bool Zlib_Compress(const char* input, size_t length, } inline char* Zlib_Uncompress(const char* input_data, size_t input_length, - int* decompress_size, int windowBits = 15) { + int* decompress_size, int windowBits = -14) { #ifdef ZLIB z_stream _stream; memset(&_stream, 0, sizeof(z_stream)); Test Plan: run db_bench with zlib compression. Reviewers: heyongqiang, MarkCallaghan Reviewed By: heyongqiang Differential Revision: https://reviews.facebook.net/D6105
-
由 Dhruba Borthakur 提交于
Summary: The parameter delete_obsolete_files_period_micros controls the periodicity of deleting obsolete files. db_bench was reading in this parameter intoa local variable called 'l' but was incorrectly using another local variable called 'n' while setting it in the db.options data structure. This patch also logs the value of delete_obsolete_files_period_micros in the LOG file at db startup time. I am hoping that this will improve the overall write throughput drastically. Test Plan: run db_bench Reviewers: MarkCallaghan, heyongqiang Reviewed By: MarkCallaghan Differential Revision: https://reviews.facebook.net/D6099
-
- 18 10月, 2012 1 次提交
-
-
由 Dhruba Borthakur 提交于
Summary: fbcode build now support SSE instructions. Delete older version of the compile-helper fbcode.sh. This is subsumed by fbcode.gcc471.sh. Test Plan: run make check Reviewers: heyongqiang, MarkCallaghan Reviewed By: heyongqiang Differential Revision: https://reviews.facebook.net/D6057
-
- 17 10月, 2012 1 次提交
-
-
由 Dhruba Borthakur 提交于
Summary: The method DeleteObsolete files is a very costly methind, especially when the number of files in a system is large. It makes a list of all live-files and then scans the directory to compute the diff. By default, this method is executed after every compaction run. This patch makes it such that DeleteObsolete files is never invoked twice within a configured period. Test Plan: run all unit tests Reviewers: heyongqiang, MarkCallaghan Reviewed By: MarkCallaghan Differential Revision: https://reviews.facebook.net/D6045
-
- 16 10月, 2012 1 次提交
-
-
由 Dhruba Borthakur 提交于
Summary: Enhance db_bench to allow setting the number of levels in a database. Test Plan: run db_bench and look at LOG Reviewers: heyongqiang, MarkCallaghan Reviewed By: MarkCallaghan CC: MarkCallaghan Differential Revision: https://reviews.facebook.net/D6027
-
- 13 10月, 2012 1 次提交
-
-
由 Dhruba Borthakur 提交于
Summary: Test Plan: Reviewers: CC: Task ID: # Blame Rev:
-
- 11 10月, 2012 1 次提交
-
-
由 Asad K Awan 提交于
Summary: Created a tool that runs multiple threads that concurrently read and write to levelDB. All writes to the DB are stored in an in-memory hashtable and verified at the end of the test. All writes for a given key are serialzied. Test Plan: - Verified by writing only a few keys and logging all writes and verifying that values read and written are correct. - Verified correctness of value generator. - Ran with various parameters of number of keys, locks, and threads. Reviewers: dhruba, MarkCallaghan, heyongqiang Reviewed By: dhruba Differential Revision: https://reviews.facebook.net/D5829
-
- 06 10月, 2012 2 次提交
-
-
由 Thawan Kooburat 提交于
Summary: This implement the Java interface by using JNI Test Plan: compile test Reviewers: dhruba Reviewed By: dhruba Differential Revision: https://reviews.facebook.net/D5925
-
由 Thawan Kooburat 提交于
Summary: See the wiki below https://our.intern.facebook.com/intern/wiki/index.php/Database/leveldb/Java Test Plan: compile test Reviewers: dhruba Reviewed By: dhruba Differential Revision: https://reviews.facebook.net/D5919
-
- 04 10月, 2012 4 次提交
-
-
由 Dhruba Borthakur 提交于
Summary: Each assoc is identified by (id1, assocType). This is the rowkey. Each row has a read/write rowlock. There is statically allocated array of 2000 read/write locks. A rowkey is murmur-hashed to one of the read/write locks. assocPut and assocDelete acquires the rowlock in Write mode. The key-updates are done within the rowlock with a atomic nosync batch write to leveldb. Then the rowlock is released and a write-with-sync is done to sync leveldb transaction log. Test Plan: added unit test Reviewers: heyongqiang Reviewed By: heyongqiang Differential Revision: https://reviews.facebook.net/D5859
-
由 Dhruba Borthakur 提交于
Summary: We have seen that reading data via the pread call (instead of mmap) is much faster on Linux 2.6.x kernels. This patch makes an equivalent option to switch off mmaps for the write path as well. db_bench --mmap_write=0 will use write() instead of mmap() to write data to a file. This change is backward compatible, the default option is to continue using mmap for writing to a file. Test Plan: "make check all" Differential Revision: https://reviews.facebook.net/D5781
-
由 Mark Callaghan 提交于
Summary: The option is zero by default and in that case reporting is unchanged. By unchanged, the interval at which stats are reported is scaled after each report and newline is not issued after each report so one line is rewritten. When non-zero it specifies the constant interval (in operations) at which statistics are reported and the stats include the rate per interval. This makes it easier to determine whether QPS changes over the duration of the test. Task ID: # Blame Rev: Test Plan: run db_bench Revert Plan: Database Impact: Memcache Impact: Other Notes: EImportant: - begin *PUBLIC* platform impact section - Bugzilla: # - end platform impact - Reviewers: dhruba Reviewed By: dhruba CC: heyongqiang Differential Revision: https://reviews.facebook.net/D5817
-
由 Mark Callaghan 提交于
Summary: see above Task ID: # Blame Rev: Test Plan: run db_bench with invalid value for option Revert Plan: Database Impact: Memcache Impact: Other Notes: EImportant: - begin *PUBLIC* platform impact section - Bugzilla: # - end platform impact - Reviewers: dhruba Reviewed By: dhruba CC: heyongqiang Differential Revision: https://reviews.facebook.net/D5823
-
- 03 10月, 2012 1 次提交
-
-
由 Mark Callaghan 提交于
Summary: stdlib.h is needed for exit() --readhead --> --readahead Task ID: # Blame Rev: Test Plan: compile Revert Plan: Database Impact: Memcache Impact: Other Notes: EImportant: - begin *PUBLIC* platform impact section - Bugzilla: # - end platform impact - fix compiler warnings & errors Reviewers: dhruba Reviewed By: dhruba CC: heyongqiang Differential Revision: https://reviews.facebook.net/D5805
-
- 02 10月, 2012 3 次提交
-
-
由 Dhruba Borthakur 提交于
Summary: Implement ReadWrite locks for leveldb. These will be helpful to implement a read-modify-write operation (e.g. atomic increments). Test Plan: does not modify any existing code Reviewers: heyongqiang Reviewed By: heyongqiang CC: MarkCallaghan Differential Revision: https://reviews.facebook.net/D5787
-
由 Abhishek Kona 提交于
Summary: A simple CLI which calles DB->CompactRange() Can take String key's as range. Test Plan: Inserted data into a table. Waited for a minute, used compact tool on it. File modification time's changed so Compact did something on the files. Existing unit tests work. Reviewers: heyongqiang, dhruba Reviewed By: dhruba Differential Revision: https://reviews.facebook.net/D5697
-
由 Dhruba Borthakur 提交于
Summary: Implement assocDelete. Test Plan: unit test attached Reviewers: heyongqiang Reviewed By: heyongqiang CC: MarkCallaghan Differential Revision: https://reviews.facebook.net/D5721
-
- 30 9月, 2012 1 次提交
-
-
由 Dhruba Borthakur 提交于
Summary: Print the block cache size in the LOG. Test Plan: run db_bench and look at LOG. This is helpful while I was debugging one use-case. Reviewers: heyongqiang, MarkCallaghan Reviewed By: heyongqiang Differential Revision: https://reviews.facebook.net/D5739
-
- 29 9月, 2012 1 次提交
-
-
由 Dhruba Borthakur 提交于
Summary: In the current code, a Get() call can trigger compaction if it has to look at more than one file. This causes unnecessary compaction because looking at more than one file is a penalty only if the file is not yet in the cache. Also, th current code counts these files before the bloom filter check is applied. This patch counts a 'seek' only if the file fails the bloom filter check and has to read in data block(s) from the storage. This patch also counts a 'seek' if a file is not present in the file-cache, because opening a file means that its index blocks need to be read into cache. Test Plan: unit test attached. I will probably add one more unti tests. Reviewers: heyongqiang Reviewed By: heyongqiang CC: MarkCallaghan Differential Revision: https://reviews.facebook.net/D5709
-
- 28 9月, 2012 1 次提交
-
-
由 gjain 提交于
Summary: Create a tool to iterate through keys and dump values. Current options as follows: db_dump --start=[START_KEY] --end=[END_KEY] --max_keys=[NUM] --stats [PATH] START_KEY: First key to start at END_KEY: Key to end at (not inclusive) NUM: Maximum number of keys to dump PATH: Path to leveldb DB The --stats command line argument prints out the DB stats before dumping the keys. Test Plan: - Tested with invalid args - Tested with invalid path - Used empty DB - Used filled DB - Tried various permutations of command line options Reviewers: dhruba, heyongqiang Reviewed By: dhruba Differential Revision: https://reviews.facebook.net/D5643
-
- 26 9月, 2012 2 次提交
-
-
由 Dhruba Borthakur 提交于
Summary: Add -fPIC to the shared library builds. Needed by libleveldbjni. Test Plan: build Reviewers: heyongqiang Reviewed By: heyongqiang Differential Revision: https://reviews.facebook.net/D5667
-
由 Dhruba Borthakur 提交于
Summary: If ReadCompaction is switched off, then it is better to not even submit background compaction jobs. I see about 3% increase in read-throughput on a pure memory database. Test Plan: run db_bench Reviewers: heyongqiang Reviewed By: heyongqiang Differential Revision: https://reviews.facebook.net/D5673
-
- 25 9月, 2012 2 次提交
-
-
由 Dhruba Borthakur 提交于
Summary: Test Plan: Reviewers: CC: Task ID: # Blame Rev:
-
由 Dhruba Borthakur 提交于
Summary: The GetLiveFiles() api lists the set of sst files and the current MANIFEST file. But the database continues to append new data to the MANIFEST file even when the application is backing it up to the backup location. This means that the database-version that is stored in the MANIFEST FILE in the backup location does not correspond to the sst files returned by GetLiveFiles. This API adds a new parameter to GetLiveFiles. This new parmeter returns the current size of the MANIFEST file. Test Plan: Unit test attached. Reviewers: heyongqiang Reviewed By: heyongqiang Differential Revision: https://reviews.facebook.net/D5631
-
- 22 9月, 2012 3 次提交
-
-
由 Dhruba Borthakur 提交于
Summary: Keeping symbols in the binary increases the size of the library but makes it easier to debug. The optimization level is still -O2, so this should have no impact on performance. Test Plan: make all Reviewers: heyongqiang, MarkCallaghan Reviewed By: MarkCallaghan CC: MarkCallaghan Differential Revision: https://reviews.facebook.net/D5601
-
由 Dhruba Borthakur 提交于
Summary: Test Plan: Reviewers: CC: Task ID: # Blame Rev:
-
由 Dhruba Borthakur 提交于
Summary: The code was allocating 200 bytes on the stack but it writes 256 bytes into the array. x8a8ea5 std::_Rb_tree<>::erase() @ 0x7f134bee7eb0 (unknown) @ 0x8a8ea5 std::_Rb_tree<>::erase() @ 0x8a35d6 leveldb::DBImpl::CleanupCompaction() @ 0x8a7810 leveldb::DBImpl::BackgroundCompaction() @ 0x8a804d leveldb::DBImpl::BackgroundCall() @ 0x8c4eff leveldb::(anonymous namespace)::PosixEnv::BGThreadWrapper() @ 0x7f134b3c010d start_thread @ 0x7f134bf9f10d clone Test Plan: run db_bench with overwrite option Reviewers: heyongqiang Reviewed By: heyongqiang Differential Revision: https://reviews.facebook.net/D5595
-
- 20 9月, 2012 1 次提交
-
-
由 Dhruba Borthakur 提交于
Summary: The background threads are necessary for compaction. For slower storage, it might be necessary to have more than one compaction thread per DB. This patch allows creating a configurable number of worker threads. The default reamins at 1 (to maintain backward compatibility). Test Plan: run all unit tests. changes to db-bench coming in a separate patch. Reviewers: heyongqiang Reviewed By: heyongqiang CC: MarkCallaghan Differential Revision: https://reviews.facebook.net/D5559
-
- 19 9月, 2012 1 次提交
-
-
由 Dhruba Borthakur 提交于
Summary: Print out the compile version in the LOG. Test Plan: run dbbench and verify LOG Reviewers: heyongqiang Reviewed By: heyongqiang Differential Revision: https://reviews.facebook.net/D5529
-
- 18 9月, 2012 7 次提交
-
-
由 heyongqiang 提交于
improve comments about target_file_size_base, target_file_size_multiplier, max_bytes_for_level_base, max_bytes_for_level_multiplier Summary: Summary: as subject Test Plan: compile Reviewers: MarkCallaghan, dhruba Differential Revision: https://reviews.facebook.net/D5499
-
由 Dhruba Borthakur 提交于
Summary: Use correct version of jemalloc. Test Plan: run unit tests Reviewers: heyongqiang Reviewed By: heyongqiang Differential Revision: https://reviews.facebook.net/D5487
-
由 heyongqiang 提交于
Summary: as subject. This diff should be good for benchmarking. will send another diff to make it better in the case the seek compaction is enable. In that coming diff, will not count a seek if the bloomfilter filters. Test Plan: build Reviewers: dhruba, MarkCallaghan Reviewed By: MarkCallaghan Differential Revision: https://reviews.facebook.net/D5481
-
由 Dhruba Borthakur 提交于
Summary: Test Plan: Reviewers: CC: Task ID: # Blame Rev:
-
由 Dhruba Borthakur 提交于
Summary: Test Plan: Reviewers: CC: Task ID: # Blame Rev:
-
由 heyongqiang 提交于
Summary: as subject Test Plan: build Reviewers: dhruba Reviewed By: dhruba Differential Revision: https://reviews.facebook.net/D5475
-
由 Dhruba Borthakur 提交于
Summary: A set of apis that allows an application to backup data from the leveldb database based on a set of files. Test Plan: unint test attached. more coming soon. Reviewers: heyongqiang Reviewed By: heyongqiang Differential Revision: https://reviews.facebook.net/D5439
-
- 17 9月, 2012 2 次提交
-
-
由 heyongqiang 提交于
Summary: as subject. this can be used for benchmarking. If we want it for some cases, we can do more changes to make this part of the option. Test Plan: db_test Reviewers: dhruba CC: MarkCallaghan Differential Revision: https://reviews.facebook.net/D5451
-
由 heyongqiang 提交于
Summary: as subject Test Plan: build Reviewers: dhruba Differential Revision: https://reviews.facebook.net/D5469
-
- 15 9月, 2012 2 次提交
-
-
由 Mark Callaghan 提交于
Summary: Reads via mmap on concurrent workloads are much slower than pread. For example on a 24-core server with storage that can do 100k IOPS or more I can get no more than 10k IOPS with mmap reads and 32+ threads. Test Plan: db_bench benchmarks Reviewers: dhruba, heyongqiang Reviewed By: heyongqiang Differential Revision: https://reviews.facebook.net/D5433
-
由 Mark Callaghan 提交于
Summary: FLAGS_cache_size is a long, no need to scan %lld into a size_t for it (which generates a compiler warning) Test Plan: run db_bench Reviewers: dhruba, heyongqiang Reviewed By: heyongqiang CC: heyongqiang Differential Revision: https://reviews.facebook.net/D5427
-