1. 06 8月, 2013 1 次提交
    • D
      [RocksDB] [MergeOperator] The new Merge Interface! Uses merge sequences. · c2d7826c
      Deon Nicholas 提交于
      Summary:
      Here are the major changes to the Merge Interface. It has been expanded
      to handle cases where the MergeOperator is not associative. It does so by stacking
      up merge operations while scanning through the key history (i.e.: during Get() or
      Compaction), until a valid Put/Delete/end-of-history is encountered; it then
      applies all of the merge operations in the correct sequence starting with the
      base/sentinel value.
      
      I have also introduced an "AssociativeMerge" function which allows the user to
      take advantage of associative merge operations (such as in the case of counters).
      The implementation will always attempt to merge the operations/operands themselves
      together when they are encountered, and will resort to the "stacking" method if
      and only if the "associative-merge" fails.
      
      This implementation is conjectured to allow MergeOperator to handle the general
      case, while still providing the user with the ability to take advantage of certain
      efficiencies in their own merge-operator / data-structure.
      
      NOTE: This is a preliminary diff. This must still go through a lot of review,
      revision, and testing. Feedback welcome!
      
      Test Plan:
        -This is a preliminary diff. I have only just begun testing/debugging it.
        -I will be testing this with the existing MergeOperator use-cases and unit-tests
      (counters, string-append, and redis-lists)
        -I will be "desk-checking" and walking through the code with the help gdb.
        -I will find a way of stress-testing the new interface / implementation using
      db_bench, db_test, merge_test, and/or db_stress.
        -I will ensure that my tests cover all cases: Get-Memtable,
      Get-Immutable-Memtable, Get-from-Disk, Iterator-Range-Scan, Flush-Memtable-to-L0,
      Compaction-L0-L1, Compaction-Ln-L(n+1), Put/Delete found, Put/Delete not-found,
      end-of-history, end-of-file, etc.
        -A lot of feedback from the reviewers.
      
      Reviewers: haobo, dhruba, zshao, emayanke
      
      Reviewed By: haobo
      
      CC: leveldb
      
      Differential Revision: https://reviews.facebook.net/D11499
      c2d7826c
  2. 02 8月, 2013 1 次提交
    • M
      Expand KeyMayExist to return the proper value if it can be found in memory and... · 59d0b02f
      Mayank Agarwal 提交于
      Expand KeyMayExist to return the proper value if it can be found in memory and also check block_cache
      
      Summary: Removed KeyMayExistImpl because KeyMayExist demanded Get like semantics now. Removed no_io from memtable and imm because we need the proper value now and shouldn't just stop when we see Merge in memtable. Added checks to block_cache. Updated documentation and unit-test
      
      Test Plan: make all check;db_stress for 1 hour
      
      Reviewers: dhruba, haobo
      
      Reviewed By: dhruba
      
      CC: leveldb
      
      Differential Revision: https://reviews.facebook.net/D11853
      59d0b02f
  3. 24 7月, 2013 1 次提交
    • M
      Use KeyMayExist for WriteBatch-Deletes · bf66c10b
      Mayank Agarwal 提交于
      Summary:
      Introduced KeyMayExist checking during writebatch-delete and removed from Outer Delete API because it uses writebatch-delete.
      Added code to skip getting Table from disk if not already present in table_cache.
      Some renaming of variables.
      Introduced KeyMayExistImpl which allows checking since specified sequence number in GetImpl useful to check partially written writebatch.
      Changed KeyMayExist to not be pure virtual and provided a default implementation.
      Expanded unit-tests in db_test to check appropriately.
      Ran db_stress for 1 hour with ./db_stress --max_key=100000 --ops_per_thread=10000000 --delpercent=50 --filter_deletes=1 --statistics=1.
      
      Test Plan: db_stress;make check
      
      Reviewers: dhruba, haobo
      
      Reviewed By: dhruba
      
      CC: leveldb, xjin
      
      Differential Revision: https://reviews.facebook.net/D11745
      bf66c10b
  4. 12 7月, 2013 2 次提交
    • M
      Make rocksdb-deletes faster using bloom filter · 2a986919
      Mayank Agarwal 提交于
      Summary:
      Wrote a new function in db_impl.c-CheckKeyMayExist that calls Get but with a new parameter turned on which makes Get return false only if bloom filters can guarantee that key is not in database. Delete calls this function and if the option- deletes_use_filter is turned on and CheckKeyMayExist returns false, the delete will be dropped saving:
      1. Put of delete type
      2. Space in the db,and
      3. Compaction time
      
      Test Plan:
      make all check;
      will run db_stress and db_bench and enhance unit-test once the basic design gets approved
      
      Reviewers: dhruba, haobo, vamsi
      
      Reviewed By: haobo
      
      CC: leveldb
      
      Differential Revision: https://reviews.facebook.net/D11607
      2a986919
    • X
      Newbie code question · 8a5341ec
      Xing Jin 提交于
      Summary:
      This diff is more about my question when reading compaction codes,
      instead of a normal diff. I don't quite understand the logic here.
      
      Test Plan: I didn't do any test. If this is a bug, I will continue doing some test.
      
      Reviewers: haobo, dhruba, emayanke
      
      Reviewed By: dhruba
      
      CC: leveldb
      
      Differential Revision: https://reviews.facebook.net/D11661
      8a5341ec
  5. 18 6月, 2013 1 次提交
  6. 13 6月, 2013 1 次提交
    • H
      [RocksDB] cleanup EnvOptions · bdf10859
      Haobo Xu 提交于
      Summary:
      This diff simplifies EnvOptions by treating it as POD, similar to Options.
      - virtual functions are removed and member fields are accessed directly.
      - StorageOptions is removed.
      - Options.allow_readahead and Options.allow_readahead_compactions are deprecated.
      - Unused global variables are removed: useOsBuffer, useFsReadAhead, useMmapRead, useMmapWrite
      
      Test Plan: make check; db_stress
      
      Reviewers: dhruba
      
      CC: leveldb
      
      Differential Revision: https://reviews.facebook.net/D11175
      bdf10859
  7. 11 6月, 2013 1 次提交
    • D
      Do not submit multiple simultaneous seek-compaction requests. · e673d5d2
      Dhruba Borthakur 提交于
      Summary:
      The code was such that if multi-threaded-compactions as well
      as seek compaction are enabled then it submits multiple
      compaction request for the same range of keys. This causes
      extraneous sst-files to accumulate at various levels.
      
      Test Plan:
      I am not able to write a very good unit test for this one
      but can easily reproduce this bug with 'dbstress' with the
      following options.
      
      batch=1;maxk=100000000;ops=100000000;ro=0;fm=2;bpl=10485760;of=500000; wbn=3; mbc=20; mb=2097152; wbs=4194304; dds=1; sync=0;  t=32; bs=16384; cs=1048576; of=500000; ./db_stress --disable_seek_compaction=0 --mmap_read=0 --threads=$t --block_size=$bs --cache_size=$cs --open_files=$of --verify_checksum=1 --db=/data/mysql/leveldb/dbstress.dir --sync=$sync --disable_wal=1 --disable_data_sync=$dds --write_buffer_size=$wbs --target_file_size_base=$mb --target_file_size_multiplier=$fm --max_write_buffer_number=$wbn --max_background_compactions=$mbc --max_bytes_for_level_base=$bpl --reopen=$ro --ops_per_thread=$ops --max_key=$maxk --test_batches_snapshots=$batch
      
      Reviewers: leveldb, emayanke
      
      Reviewed By: emayanke
      
      Differential Revision: https://reviews.facebook.net/D11055
      e673d5d2
  8. 06 6月, 2013 1 次提交
  9. 31 5月, 2013 1 次提交
    • H
      [RocksDB] [Performance] Allow different posix advice to be applied to the same table file · ab8d2f6a
      Haobo Xu 提交于
      Summary:
      Current posix advice implementation ties up the access pattern hint with the creation of a file.
      It is not possible to apply different advice for different access (random get vs compaction read),
      without keeping two open files for the same table. This patch extended the RandomeAccessFile interface
      to accept new access hint at anytime. Particularly, we are able to set different access hint on the same
      table file based on when/how the file is used.
      Two options are added to set the access hint, after the file is first opened and after the file is being
      compacted.
      
      Test Plan: make check; db_stress; db_bench
      
      Reviewers: dhruba
      
      Reviewed By: dhruba
      
      CC: MarkCallaghan, leveldb
      
      Differential Revision: https://reviews.facebook.net/D10905
      ab8d2f6a
  10. 22 5月, 2013 1 次提交
    • D
      Ability to set different size fanout multipliers for every level. · d1aaaf71
      Dhruba Borthakur 提交于
      Summary:
      There is an existing field Options.max_bytes_for_level_multiplier that
      sets the multiplier for the size of each level in the database.
      
      This patch introduces the ability to set different multipliers
      for every level in the database. The size of a level is determined
      by using both max_bytes_for_level_multiplier as well as the
      per-level fanout.
      
      size of level[i] = size of level[i-1] * max_bytes_for_level_multiplier
                         * fanout[i-1]
      
      The default value of fanout is 1, so that it is backward compatible.
      
      Test Plan: make check
      
      Reviewers: haobo, emayanke
      
      Reviewed By: emayanke
      
      CC: leveldb
      
      Differential Revision: https://reviews.facebook.net/D10863
      d1aaaf71
  11. 09 5月, 2013 1 次提交
    • D
      Assertion failure for L0-L1 compactions. · a8d3aa2c
      Dhruba Borthakur 提交于
      Summary:
      For level-0 compactions, we try to find if can include more L0 files
      in the same compaction run. This causes the 'smallest' and 'largest'
      key to get extended to a larger range. But the suceeding call to
      ParentRangeInCompaction() was still using the earlier
      values of 'smallest' and 'largest',
      
      Because of this bug, a file in L1 can be part of two concurrent
      compactions: one L0-L1 compaction and the other L1-L2 compaction.
      
      This should not cause any data loss, but will cause an assertion
      failure with debug builds.
      
      Test Plan: make check
      
      Differential Revision: https://reviews.facebook.net/D10677
      a8d3aa2c
  12. 04 5月, 2013 1 次提交
    • H
      [Rocksdb] Support Merge operation in rocksdb · 05e88540
      Haobo Xu 提交于
      Summary:
      This diff introduces a new Merge operation into rocksdb.
      The purpose of this review is mostly getting feedback from the team (everyone please) on the design.
      
      Please focus on the four files under include/leveldb/, as they spell the client visible interface change.
      include/leveldb/db.h
      include/leveldb/merge_operator.h
      include/leveldb/options.h
      include/leveldb/write_batch.h
      
      Please go over local/my_test.cc carefully, as it is a concerete use case.
      
      Please also review the impelmentation files to see if the straw man implementation makes sense.
      
      Note that, the diff does pass all make check and truly supports forward iterator over db and a version
      of Get that's based on iterator.
      
      Future work:
      - Integration with compaction
      - A raw Get implementation
      
      I am working on a wiki that explains the design and implementation choices, but coding comes
      just naturally and I think it might be a good idea to share the code earlier. The code is
      heavily commented.
      
      Test Plan: run all local tests
      
      Reviewers: dhruba, heyongqiang
      
      Reviewed By: dhruba
      
      CC: leveldb, zshao, sheki, emayanke, MarkCallaghan
      
      Differential Revision: https://reviews.facebook.net/D9651
      05e88540
  13. 16 4月, 2013 1 次提交
  14. 13 4月, 2013 1 次提交
    • H
      [RocksDB] [Performance] Speed up FindObsoleteFiles · 013e9ebb
      Haobo Xu 提交于
      Summary:
      FindObsoleteFiles was slow, holding the single big lock, resulted in bad p99 behavior.
      Didn't profile anything, but several things could be improved:
      1. VersionSet::AddLiveFiles works with std::set, which is by itself slow (a tree).
         You also don't know how many dynamic allocations occur just for building up this tree.
         switched to std::vector, also added logic to pre-calculate total size and do just one allocation
      2. Don't see why env_->GetChildren() needs to be mutex proteced, moved to PurgeObsoleteFiles where
         mutex could be unlocked.
      3. switched std::set to std:unordered_set, the conversion from vector is also inside PurgeObsoleteFiles
      I have a feeling this should pretty much fix it.
      
      Test Plan: make check;  db_stress
      
      Reviewers: dhruba, heyongqiang, MarkCallaghan
      
      Reviewed By: dhruba
      
      CC: leveldb, zshao
      
      Differential Revision: https://reviews.facebook.net/D10197
      013e9ebb
  15. 21 3月, 2013 1 次提交
    • D
      Ability to configure bufferedio-reads, filesystem-readaheads and mmap-read-write per database. · ad96563b
      Dhruba Borthakur 提交于
      Summary:
      This patch allows an application to specify whether to use bufferedio,
      reads-via-mmaps and writes-via-mmaps per database. Earlier, there
      was a global static variable that was used to configure this functionality.
      
      The default setting remains the same (and is backward compatible):
       1. use bufferedio
       2. do not use mmaps for reads
       3. use mmap for writes
       4. use readaheads for reads needed for compaction
      
      I also added a parameter to db_bench to be able to explicitly specify
      whether to do readaheads for compactions or not.
      
      Test Plan: make check
      
      Reviewers: sheki, heyongqiang, MarkCallaghan
      
      Reviewed By: sheki
      
      CC: leveldb
      
      Differential Revision: https://reviews.facebook.net/D9429
      ad96563b
  16. 20 3月, 2013 1 次提交
  17. 19 3月, 2013 1 次提交
  18. 12 3月, 2013 1 次提交
    • D
      Prevent segfault because SizeUnderCompaction was called without any locks. · ebf16f57
      Dhruba Borthakur 提交于
      Summary:
      SizeBeingCompacted was called without any lock protection. This causes
      crashes, especially when running db_bench with value_size=128K.
      The fix is to compute SizeUnderCompaction while holding the mutex and
      passing in these values into the call to Finalize.
      
      (gdb) where
      #4  leveldb::VersionSet::SizeBeingCompacted (this=this@entry=0x7f0b490931c0, level=level@entry=4) at db/version_set.cc:1827
      #5  0x000000000043a3c8 in leveldb::VersionSet::Finalize (this=this@entry=0x7f0b490931c0, v=v@entry=0x7f0b3b86b480) at db/version_set.cc:1420
      #6  0x00000000004418d1 in leveldb::VersionSet::LogAndApply (this=0x7f0b490931c0, edit=0x7f0b3dc8c200, mu=0x7f0b490835b0, new_descriptor_log=<optimized out>) at db/version_set.cc:1016
      #7  0x00000000004222b2 in leveldb::DBImpl::InstallCompactionResults (this=this@entry=0x7f0b49083400, compact=compact@entry=0x7f0b2b8330f0) at db/db_impl.cc:1473
      #8  0x0000000000426027 in leveldb::DBImpl::DoCompactionWork (this=this@entry=0x7f0b49083400, compact=compact@entry=0x7f0b2b8330f0) at db/db_impl.cc:1757
      #9  0x0000000000426690 in leveldb::DBImpl::BackgroundCompaction (this=this@entry=0x7f0b49083400, madeProgress=madeProgress@entry=0x7f0b41bf2d1e, deletion_state=...) at db/db_impl.cc:1268
      #10 0x0000000000428f42 in leveldb::DBImpl::BackgroundCall (this=0x7f0b49083400) at db/db_impl.cc:1170
      #11 0x000000000045348e in BGThread (this=0x7f0b49023100) at util/env_posix.cc:941
      #12 leveldb::(anonymous namespace)::PosixEnv::BGThreadWrapper (arg=0x7f0b49023100) at util/env_posix.cc:874
      #13 0x00007f0b4a7cf10d in start_thread (arg=0x7f0b41bf3700) at pthread_create.c:301
      #14 0x00007f0b49b4b11d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:115
      
      Test Plan:
      make check
      
      I am running db_bench with a value size of 128K to see if the segfault is fixed.
      
      Reviewers: MarkCallaghan, sheki, emayanke
      
      Reviewed By: sheki
      
      CC: leveldb
      
      Differential Revision: https://reviews.facebook.net/D9279
      ebf16f57
  19. 08 3月, 2013 1 次提交
    • D
      A mechanism to detect manifest file write errors and put db in readonly mode. · 6d812b6a
      Dhruba Borthakur 提交于
      Summary:
      If there is an error while writing an edit to the manifest file, the manifest
      file is closed and reopened to check if the edit made it in. However, if the
      re-opening of the manifest is unsuccessful and options.paranoid_checks is set
      t true, then the db refuses to accept new puts, effectively putting the db
      in readonly mode.
      
      In a future diff, I would like to make the default value of paranoid_check
      to true.
      
      Test Plan: make check
      
      Reviewers: sheki
      
      Reviewed By: sheki
      
      CC: leveldb
      
      Differential Revision: https://reviews.facebook.net/D9201
      6d812b6a
  20. 04 3月, 2013 1 次提交
    • M
      Add rate_delay_limit_milliseconds · 993543d1
      Mark Callaghan 提交于
      Summary:
      This adds the rate_delay_limit_milliseconds option to make the delay
      configurable in MakeRoomForWrite when the max compaction score is too high.
      This delay is called the Ln slowdown. This change also counts the Ln slowdown
      per level to make it possible to see where the stalls occur.
      
      From IO-bound performance testing, the Level N stalls occur:
      * with compression -> at the largest uncompressed level. This makes sense
                            because compaction for compressed levels is much
                            slower. When Lx is uncompressed and Lx+1 is compressed
                            then files pile up at Lx because the (Lx,Lx+1)->Lx+1
                            compaction process is the first to be slowed by
                            compression.
      * without compression -> at level 1
      
      Task ID: #1832108
      
      Blame Rev:
      
      Test Plan:
      run with real data, added test
      
      Revert Plan:
      
      Database Impact:
      
      Memcache Impact:
      
      Other Notes:
      
      EImportant:
      
      - begin *PUBLIC* platform impact section -
      Bugzilla: #
      - end platform impact -
      
      Reviewers: dhruba
      
      Reviewed By: dhruba
      
      Differential Revision: https://reviews.facebook.net/D9045
      993543d1
  21. 01 3月, 2013 1 次提交
  22. 19 2月, 2013 1 次提交
  23. 24 1月, 2013 1 次提交
    • C
      Fix a number of object lifetime/ownership issues · 2fdf91a4
      Chip Turner 提交于
      Summary:
      Replace manual memory management with std::unique_ptr in a
      number of places; not exhaustive, but this fixes a few leaks with file
      handles as well as clarifies semantics of the ownership of file handles
      with log classes.
      
      Test Plan: db_stress, make check
      
      Reviewers: dhruba
      
      Reviewed By: dhruba
      
      CC: zshao, leveldb, heyongqiang
      
      Differential Revision: https://reviews.facebook.net/D8043
      2fdf91a4
  24. 17 1月, 2013 1 次提交
    • A
      rollover manifest file. · 7d5a4383
      Abhishek Kona 提交于
      Summary:
      Check in LogAndApply if the file size is more than the limit set in
      Options.
      Things to consider : will this be expensive?
      
      Test Plan: make all check. Inputs on a new unit test?
      
      Reviewers: dhruba
      
      Reviewed By: dhruba
      
      CC: leveldb
      
      Differential Revision: https://reviews.facebook.net/D7701
      7d5a4383
  25. 16 1月, 2013 3 次提交
    • C
      Add optional clang compile mode · a2dcd79c
      Chip Turner 提交于
      Summary:
      clang is an alternate compiler based on llvm.  It produces
      nicer error messages and finds some bugs that gcc doesn't, such as the
      size_t change in this file (which caused some write return values to be
      misinterpreted!)
      
      Clang isn't the default; to try it, do "USE_CLANG=1 make" or "export
      USE_CLANG=1" then make as normal
      
      Test Plan: "make check" and "USE_CLANG=1 make check"
      
      Reviewers: dhruba
      
      Reviewed By: dhruba
      
      Differential Revision: https://reviews.facebook.net/D7899
      a2dcd79c
    • C
      Fix broken build · 9bbcab57
      Chip Turner 提交于
      Summary: Mis-merged from HEAD, had a duplicate declaration.
      
      Test Plan: make -j32 OPT=-g
      
      Reviewers: dhruba
      
      Reviewed By: dhruba
      
      Differential Revision: https://reviews.facebook.net/D7911
      9bbcab57
    • K
      Fixed bug with seek compactions on Level 0 · 28fe86c4
      Kosie van der Merwe 提交于
      Summary: Due to how the code handled compactions in Level 0 in `PickCompaction()` it could be the case that two compactions on level 0 ran that produced tables in level 1 that overlap. However, this case seems like it would only occur on a seek compaction which is unlikely on level 0. Furthermore, level 0 and level 1 had to have a certain arrangement of files.
      
      Test Plan:
      make check
      
      Reviewers: dhruba, vamsi
      
      Reviewed By: dhruba
      
      CC: leveldb, sheki
      
      Differential Revision: https://reviews.facebook.net/D7923
      28fe86c4
  26. 15 1月, 2013 1 次提交
    • C
      Various build cleanups/improvements · c0cb289d
      Chip Turner 提交于
      Summary:
      Specific changes:
      
      1) Turn on -Werror so all warnings are errors
      2) Fix some warnings the above now complains about
      3) Add proper dependency support so changing a .h file forces a .c file
      to rebuild
      4) Automatically use fbcode gcc on any internal machine rather than
      whatever system compiler is laying around
      5) Fix jemalloc to once again be used in the builds (seemed like it
      wasn't being?)
      6) Fix issue where 'git' would fail in build_detect_version because of
      LD_LIBRARY_PATH being set in the third-party build system
      
      Test Plan:
      make, make check, make clean, touch a header file, make sure
      rebuild is expected
      
      Reviewers: dhruba
      
      Reviewed By: dhruba
      
      Differential Revision: https://reviews.facebook.net/D7887
      c0cb289d
  27. 12 1月, 2013 1 次提交
  28. 11 1月, 2013 1 次提交
  29. 03 1月, 2013 1 次提交
  30. 17 12月, 2012 1 次提交
    • Z
      manifest_dump: Add --hex=1 option · c2809753
      Zheng Shao 提交于
      Summary: Without this option, manifest_dump does not print binary keys for files in a human-readable way.
      
      Test Plan:
      ./manifest_dump --hex=1 --verbose=0 --file=/data/users/zshao/fdb_comparison/leveldb/fbobj.apprequest-0_0_original/MANIFEST-000002
      manifest_file_number 589 next_file_number 590 last_sequence 2311567 log_number 543  prev_log_number 0
      --- level 0 --- version# 0 ---
       532:1300357['0000455BABE20000' @ 2183973 : 1 .. 'FFFCA5D7ADE20000' @ 2184254 : 1]
       536:1308170['000198C75CE30000' @ 2203313 : 1 .. 'FFFCF94A79E30000' @ 2206463 : 1]
       542:1321644['0002931AA5E50000' @ 2267055 : 1 .. 'FFF77B31C5E50000' @ 2270754 : 1]
       544:1286390['000410A309E60000' @ 2278592 : 1 .. 'FFFE470A73E60000' @ 2289221 : 1]
       538:1298778['0006BCF4D8E30000' @ 2217050 : 1 .. 'FFFD77DAF7E30000' @ 2220489 : 1]
       540:1282353['00090D5356E40000' @ 2231156 : 1 .. 'FFFFF4625CE40000' @ 2231969 : 1]
      --- level 1 --- version# 0 ---
       510:2112325['000007F9C2D40000' @ 1782099 : 1 .. '146F5B67B8D80000' @ 1905458 : 1]
       511:2121742['146F8A3023D60000' @ 1824388 : 1 .. '28BC8FBB9CD40000' @ 1777993 : 1]
       512:801631['28BCD396F1DE0000' @ 2080191 : 1 .. '3082DBE9ADDB0000' @ 1989927 : 1]
      
      Reviewers: dhruba, sheki, emayanke
      
      Reviewed By: dhruba
      
      CC: leveldb
      
      Differential Revision: https://reviews.facebook.net/D7425
      c2809753
  31. 05 12月, 2012 1 次提交
  32. 29 11月, 2012 1 次提交
    • A
      Fix all the lint errors. · d29f1819
      Abhishek Kona 提交于
      Summary:
      Scripted and removed all trailing spaces and converted all tabs to
      spaces.
      
      Also fixed other lint errors.
      All lint errors from this point of time should be taken seriously.
      
      Test Plan: make all check
      
      Reviewers: dhruba
      
      Reviewed By: dhruba
      
      CC: leveldb
      
      Differential Revision: https://reviews.facebook.net/D7059
      d29f1819
  33. 27 11月, 2012 1 次提交
    • D
      Assertion failure while running with unit tests with OPT=-g · 2a396999
      Dhruba Borthakur 提交于
      Summary:
      When we expand the range of keys for a level 0 compaction, we
      need to invoke ParentFilesInCompaction() only once for the
      entire range of keys that is being compacted. We were invoking
      it for each file that was being compacted, but this triggers
      an assertion because each file's range were contiguous but
      non-overlapping.
      
      I renamed ParentFilesInCompaction to ParentRangeInCompaction
      to adequately represent that it is the range-of-keys and
      not individual files that we compact in a single compaction run.
      
      Here is the assertion that is fixed by this patch.
      db_test: db/version_set.cc:585: void leveldb::Version::ExtendOverlappingInputs(int, const leveldb::Slice&, const leveldb::Slice&, std::vector<leveldb::FileMetaData*, std::allocator<leveldb::FileMetaData*> >*, int): Assertion `user_cmp->Compare(flimit, user_begin) >= 0' failed.
      
      Test Plan: make clean check OPT=-g
      
      Reviewers: sheki
      
      Reviewed By: sheki
      
      CC: MarkCallaghan, emayanke, leveldb
      
      Differential Revision: https://reviews.facebook.net/D6963
      2a396999
  34. 22 11月, 2012 1 次提交
    • D
      Support taking a configurable number of files from the same level to compact... · 7632fdb5
      Dhruba Borthakur 提交于
      Support taking a configurable number of  files from the same level to compact in a single compaction run.
      
      Summary:
      The compaction process takes some files from LevelK and
      merges it into LevelK+1. The number of files it picks from
      LevelK was capped such a way that the total amount of
      data picked does not exceed the maxfilesize of that level.
      This essentially meant that only one file from LevelK
      is picked for a single compaction.
      
      For bulkloads, we would like to take many many file from
      LevelK and compact them using a single compaction run.
      
      This patch introduces a option called the 'source_compaction_factor'
      (similar to expanded_compaction_factor). It is a multiplier
      that is multiplied by the maxfilesize of that level to arrive
      at the limit that is used to throttle the number of source
      files from LevelK.  For bulk loads, set source_compaction_factor
      to a very high number so that multiple files from the same
      level are picked for compaction in a single compaction.
      
      The default value of source_compaction_factor is 1, so that
      we can keep backward compatibilty with existing compaction semantics.
      
      Test Plan: make clean check
      
      Reviewers: emayanke, sheki
      
      Reviewed By: emayanke
      
      CC: leveldb
      
      Differential Revision: https://reviews.facebook.net/D6867
      7632fdb5
  35. 21 11月, 2012 1 次提交
    • D
      A major bug that was not considering the compaction score of the n-1 level. · 3754f2f4
      Dhruba Borthakur 提交于
      Summary:
      The method Finalize() recomputes the compaction score of each
      level and then sorts these score from largest to smallest. The
      idea is that the level with the largest compaction score will
      be a better candidate for compaction.  There are usually very
      few levels, and a bubble sort code was used to sort these
      compaction scores. There existed a bug in the sorting code that
      skipped looking at the score for the n-1 level. This meant that
      even if the compaction score of the n-1 level is large, it will
      not be picked for compaction.
      
      This patch fixes the bug and also introduces "asserts" in the
      code to detect any possible inconsistencies caused by future bugs.
      
      This bug existed in the very first code change that introduced
      multi-threaded compaction to the leveldb code. That version of
      code was committed on Oct 19th via
      https://github.com/facebook/leveldb/commit/1ca0584345af85d2dccc434f451218119626d36e
      
      Test Plan: make clean check OPT=-g
      
      Reviewers: emayanke, sheki, MarkCallaghan
      
      Reviewed By: sheki
      
      CC: leveldb
      
      Differential Revision: https://reviews.facebook.net/D6837
      3754f2f4
  36. 20 11月, 2012 2 次提交
    • D
      Fix asserts · dde70898
      Dhruba Borthakur 提交于
      Summary:
      make check OPT=-g fails with the following assert.
      ==== Test DBTest.ApproximateSizes
      db_test: db/version_set.cc:765: void leveldb::VersionSet::Builder::CheckConsistencyForDeletes(leveldb::VersionEdit*, int, int): Assertion `found' failed.
      
      The assertion was that file #7 that was being deleted did not
      preexists, but actualy it did pre-exist as shown in the manifest
      dump shows below. The bug was that we did not check for file
      existance at the same level.
      
      *************************Edit[0] = VersionEdit {
        Comparator: leveldb.BytewiseComparator
      }
      
      *************************Edit[1] = VersionEdit {
        LogNumber: 8
        PrevLogNumber: 0
        NextFile: 9
        LastSeq: 80
        AddFile: 0 7 8005319 'key000000' @ 1 : 1 .. 'key000079' @ 80 : 1
      }
      
      *************************Edit[2] = VersionEdit {
        LogNumber: 8
        PrevLogNumber: 0
        NextFile: 13
        LastSeq: 80
        CompactPointer: 0 'key000079' @ 80 : 1
        DeleteFile: 0 7
        AddFile: 1 9 2101425 'key000000' @ 1 : 1 .. 'key000020' @ 21 : 1
        AddFile: 1 10 2101425 'key000021' @ 22 : 1 .. 'key000041' @ 42 : 1
        AddFile: 1 11 2101425 'key000042' @ 43 : 1 .. 'key000062' @ 63 : 1
        AddFile: 1 12 1701165 'key000063' @ 64 : 1 .. 'key000079' @ 80 : 1
      }
      
      Test Plan:
      
      Reviewers:
      
      CC:
      
      Task ID: #
      
      Blame Rev:
      dde70898
    • D
      Fix compilation error introduced by previous commit · 48dafb2c
      Dhruba Borthakur 提交于
      7889e094
      
      Summary:
      Fix compilation error introduced by previous commit
      7889e094
      
      Test Plan:
      make clean check
      48dafb2c