Skip to content

R
rocksdb

kvdb / rocksdb
11 个月 前同步成功

通知 0
Star 0
Fork 0
R
rocksdb

体验新版 GitCode，发现更多精彩内容 >>

切换分支/标签
  1. 01 10月, 2015 1 次提交
    • E
      New amalgamation target · 7a23e4d8
      由 Evan Shaw 提交于 
      This commit adds two new targets to the Makefile: rocksdb.cc and rocksdb.h

These files, when combined with the c.h header, are a self-contained RocksDB
source distribution called an amalgamation. (The name comes from SQLite's, which
is similar in concept.)

The main benefit of an amalgamation is that it's very easy to drop into a
new project. It also compiles faster compared to compiling individual source
files and potentially gives the compiler more opportunity to make optimizations
since it can see all functions at once.

rocksdb.cc and rocksdb.h are generated by a new script, amalgamate.py.
A detailed description of how amalgamate.py works is in a comment at the top of
the file.

There are also some small changes to existing files to enable the amalgamation:
* Use quotes for includes in unity build
* Fix an old header inclusion in util/xfunc.cc
* Move some includes outside ifdef in util/env_hdfs.cc
* Separate out tool sources in Makefile so they won't be included in unity.cc
* Unity build now produces a static library

Closes #733
      7a23e4d8
  3. 24 9月, 2015 2 次提交
    • K
      Simplifying valgrind testing · 628216fc
      由 krad 提交于 
      Summary: Our valgrind testing is buggy and cumbersome in terms of locating the
error. It originates from the fact we accumulate all output for the tests. It is
extremely hard to locate the point of error.

The communication between valgrind and the script is not sturdy. We are
experiencing bugs.

Simplifying to stop on first error.

Test Plan: Run manually

Reviewers: sdong igor

CC: leveldb@

Task ID: #6968635

Blame Rev:
      628216fc
    • A
      Remove ldb HexToString method's usage of sscanf · 4805fa0e
      由 Assaf Sela 提交于 
      Summary:
Fix hex2String performance issues by removing sscanf dependency.
Also fixed some edge case handling (odd length, bad input).

Test Plan: Created a test file which called old and new implementation, and validated results are the same. I'll paste results in the phabricator diff.

Reviewers: igor, rven, anthony, IslamAbdelRahman, kradhakrishnan, yhchiang, sdong

Reviewed By: sdong

Subscribers: thatsafunnyname, leveldb, dhruba

Differential Revision: https://reviews.facebook.net/D46785
      4805fa0e
  5. 18 9月, 2015 1 次提交
    • A
      Support for SingleDelete() · 014fd55a
      由 Andres Noetzli 提交于 
      Summary:
This patch fixes #7460559. It introduces SingleDelete as a new database
operation. This operation can be used to delete keys that were never
overwritten (no put following another put of the same key). If an overwritten
key is single deleted the behavior is undefined. Single deletion of a
non-existent key has no effect but multiple consecutive single deletions are
not allowed (see limitations).

In contrast to the conventional Delete() operation, the deletion entry is
removed along with the value when the two are lined up in a compaction. Note:
The semantics are similar to @igor's prototype that allowed to have this
behavior on the granularity of a column family (
https://reviews.facebook.net/D42093 ). This new patch, however, is more
aggressive when it comes to removing tombstones: It removes the SingleDelete
together with the value whenever there is no snapshot between them while the
older patch only did this when the sequence number of the deletion was older
than the earliest snapshot.

Most of the complex additions are in the Compaction Iterator, all other changes
should be relatively straightforward. The patch also includes basic support for
single deletions in db_stress and db_bench.

Limitations:
- Not compatible with cuckoo hash tables
- Single deletions cannot be used in combination with merges and normal
  deletions on the same key (other keys are not affected by this)
- Consecutive single deletions are currently not allowed (and older version of
  this patch supported this so it could be resurrected if needed)

Test Plan: make all check

Reviewers: yhchiang, sdong, rven, anthony, yoshinorim, igor

Reviewed By: igor

Subscribers: maykov, dhruba, leveldb

Differential Revision: https://reviews.facebook.net/D43179
      014fd55a
  7. 17 9月, 2015 1 次提交
    • Y
      Fix the verbosity issue in Java makefile · 2e8e7eb3
      由 Yueh-Hsuan Chiang 提交于 
      Summary:
When running `make rocksdbjava V=1`, it currently does
not print out the original command but something odd instead:

    mkdir -p jl/db

This patch tries to fix this issue.

Test Plan: make rocksdbjava V=1

Reviewers: igor, sdong, IslamAbdelRahman, meyering

Reviewed By: meyering

Subscribers: dhruba, leveldb

Differential Revision: https://reviews.facebook.net/D47061
      2e8e7eb3
  9. 02 9月, 2015 1 次提交
  11. 14 8月, 2015 1 次提交
    • M
      Add a 'tools' target. · 9f0dd222
      由 maurice barnum 提交于 
      My use case is to build the rocksdb static library and tools, and
ideally I'd like to not spend time building the shared library and other
targets that I won't use.
      9f0dd222
  13. 12 8月, 2015 1 次提交
    • A
      Pessimistic Transactions · c2f2cb02
      由 agiardullo 提交于 
      Summary:
Initial implementation of Pessimistic Transactions.  This diff contains the api changes discussed in D38913.  This diff is pretty large, so let me know if people would prefer to meet up to discuss it.

MyRocks folks:  please take a look at the API in include/rocksdb/utilities/transaction[_db].h and let me know if you have any issues.

Also, you'll notice a couple of TODOs in the implementation of RollbackToSavePoint().  After chatting with Siying, I'm going to send out a separate diff for an alternate implementation of this feature that implements the rollback inside of WriteBatch/WriteBatchWithIndex.  We can then decide which route is preferable.

Next, I'm planning on doing some perf testing and then integrating this diff into MongoRocks for further testing.

Test Plan: Unit tests, db_bench parallel testing.

Reviewers: igor, rven, sdong, yhchiang, yoshinorim

Reviewed By: sdong

Subscribers: hermanlee4, maykov, spetrunia, leveldb, dhruba

Differential Revision: https://reviews.facebook.net/D40869
      c2f2cb02
  15. 06 8月, 2015 3 次提交
    • S
      valgrind_check to exit on test failures · fca88f8e
      由 sdong 提交于 
      Summary: Currently, valgrind_check doesn't fail on test failures, which creates confusion. valgrind_check should fail if test fails.

Test Plan: Manually change tests to return test failure or cause memory leak and see valgrind_check has the correct behavior.

Reviewers: anthony, yhchiang, IslamAbdelRahman, igor, kradhakrishnan, rven

Reviewed By: rven

Subscribers: leveldb, dhruba

Differential Revision: https://reviews.facebook.net/D43629
      fca88f8e
    • S
      Add two unit tests for SyncWAL() · 7ccd1c80
      由 sdong 提交于 
      Summary:
Add two unit tests for SyncWAL(). One makes sure SyncWAL() doesn't block writes in the other thread. Another one makes sure SyncWAL() doesn't wait ongoing writes to finish before being executed.

Create a new test file db_wal_test and move two WAL related tests from db_test to here.

Test Plan: Run the new tests

Reviewers: IslamAbdelRahman, rven, kradhakrishnan, kolmike, tnovak, yhchiang

Reviewed By: yhchiang

Subscribers: leveldb, dhruba

Differential Revision: https://reviews.facebook.net/D43605
      7ccd1c80
    • S
      "make commit-prereq" should clean up rocksjava properly · 8ecb51a7
      由 sdong 提交于 
      Summary: "make commit-prereq" fails to clean up java, which can cause rocksjava failure.

Test Plan: Run commit-prepreq

Reviewers: IslamAbdelRahman, rven, kradhakrishnan, yhchiang

Reviewed By: yhchiang

Subscribers: leveldb, dhruba

Differential Revision: https://reviews.facebook.net/D43575
      8ecb51a7
  17. 05 8月, 2015 2 次提交
    • I
      Support delete rate limiting · c45a57b4
      由 Islam AbdelRahman 提交于 
      Summary:
Introduce DeleteScheduler that allow enforcing a rate limit on file deletion
Instead of deleting files immediately, files are moved to trash directory and deleted in a background thread that apply sleep penalty between deletes if needed.

I have updated PurgeObsoleteFiles and PurgeObsoleteWALFiles to use the delete_scheduler instead of env_->DeleteFile

Test Plan:
added delete_scheduler_test
existing unit tests

Reviewers: kradhakrishnan, anthony, rven, yhchiang, sdong

Reviewed By: sdong

Subscribers: dhruba

Differential Revision: https://reviews.facebook.net/D43221
      c45a57b4
    • S
      crash_test to cover simply cases · 1205bdbc
      由 sdong 提交于 
      Summary:
crash_test now only runs complicated options, multiple column families, prefix hash, frequently changing options, many compaction threads, etc. These options are good to cover new features but we loss coverage in most common use cases. Furthermore, by running only for multiple column families, we are not able to create LSM trees that are large enough to cover some stress cases.
Make half of crash_test runs the simply tests: single column family, default mem table, one compaction thread, no change options.

Test Plan: Run crash_test

Reviewers: rven, yhchiang, IslamAbdelRahman, kradhakrishnan

Reviewed By: kradhakrishnan

Subscribers: leveldb, dhruba

Differential Revision: https://reviews.facebook.net/D43461
      1205bdbc
  19. 04 8月, 2015 1 次提交
    • Y
      Add CompactOnDeletionCollector in utilities/table_properties_collectors. · 26894303
      由 Yueh-Hsuan Chiang 提交于 
      Summary:
This diff adds CompactOnDeletionCollector in utilities/table_properties_collectors,
which applies a sliding window to a sst file and mark this file as need-compaction
when it observe enough deletion entries within the consecutive keys covered by
the sliding window.

Test Plan: compact_on_deletion_collector_test

Reviewers: igor, anthony, IslamAbdelRahman, kradhakrishnan, yoshinorim, sdong

Reviewed By: sdong

Subscribers: maykov, dhruba

Differential Revision: https://reviews.facebook.net/D41175
      26894303
  21. 25 7月, 2015 1 次提交
    • A
      Add missing hashCode() implementation · 6a82fba7
      由 Andres Noetzli 提交于 
      Summary:
Whenever a Java class implements equals(), it has to implement hashCode(), otherwise
there might be weird behavior when inserting instances of the class in a hash map for
example. This adds two missing hashCode() implementations and extends tests to test
the hashCode() implementations.

Test Plan: make jtest

Reviewers: rven, igor, sdong, yhchiang

Reviewed By: yhchiang

Subscribers: anthony, dhruba, leveldb

Differential Revision: https://reviews.facebook.net/D43017
      6a82fba7
  23. 23 7月, 2015 1 次提交
  25. 22 7月, 2015 1 次提交
    • S
      RangeSync not to sync last 1MB of the file · 3dbf4ba2
      由 Siying Dong 提交于 
      Summary:
From other ones' investigation:

"sync_file_range() behavior highly depends on kernel version and filesystem.

xfs does neighbor page flushing outside of the specified ranges. For example, sync_file_range(fd, 8192, 16384) does not only trigger flushing page #3 to #4, but also flushing many more dirty pages (i.e. up to page#16)... Ranges of the sync_file_range() should be far enough from write() offset (at least 1MB)."

Test Plan: make all check

Reviewers: igor, rven, kradhakrishnan, yhchiang, IslamAbdelRahman, anthony

Reviewed By: anthony

Subscribers: yoshinorim, MarkCallaghan, sumeet, domas, dhruba, leveldb, ljin

Differential Revision: https://reviews.facebook.net/D15807
      3dbf4ba2
  27. 21 7月, 2015 7 次提交
  29. 18 7月, 2015 2 次提交
    • I
      Don't let flushes preempt compactions · 35ca5936
      由 Igor Canadi 提交于 
      Summary:
When we first started, max_background_flushes was 0 by default and compaction thread was executing flushes (since there was no flush thread). Then, we switched the default max_background_flushes to 1. However, we still support the case where there is no flush thread and flushes are done in compaction. This is making our code a bit more complicated. By not supporting this use-case we can make our code simpler.

We have a special case that when you set max_background_flushes to 0, we
schedule the flush to execute on the compaction thread.

Test Plan: make check (there might be some unit tests that depend on this behavior)

Reviewers: IslamAbdelRahman, yhchiang, sdong

Reviewed By: sdong

Subscribers: dhruba, leveldb

Differential Revision: https://reviews.facebook.net/D41931
      35ca5936
    • A
      Fixed and simplified merge_helper · 1d20fa9d
      由 Andres Notzli 提交于 
      Summary:
MergeUntil was not reporting a success when merging an operand with
a Value/Deletion despite the comments in MergeHelper and CompactionJob
indicating otherwise. This lead to operands being written to the compaction
output unnecessarily:

M1 M2 M3 P M4 M5 --> (P+M1+M2+M3) M2 M3 M4 M5 (before the diff)
M1 M2 M3 P M4 M5 --> (P+M1+M2+M3) M4 M5 (after the diff)

In addition, the code handling Values/Deletion was basically identical.
This patch unifies the code. Finally, this patch also adds testing for
merge_helper.

Test Plan: make && make check

Reviewers: sdong, rven, yhchiang, tnovak, igor

Reviewed By: igor

Subscribers: tnovak, dhruba, leveldb

Differential Revision: https://reviews.facebook.net/D42351
      1d20fa9d
  31. 15 7月, 2015 6 次提交
  33. 14 7月, 2015 3 次提交
  35. 24 6月, 2015 1 次提交
  37. 20 6月, 2015 3 次提交
  39. 19 6月, 2015 1 次提交
    • Y
      Remove unused target --- compactor_test · 71b438c4
      由 Yueh-Hsuan Chiang 提交于 
      Summary:
Remove compactor_test, which depends on a directory not exist
in our code base.
    make compactor_test
    GEN      util/build_version.cc
    GEN      util/build_version.cc
    make: *** No rule to make target `utilities/compaction/compactor_test.o', needed by `compactor_test'.  Stop.

Test Plan: verify the output message of make compactor_test

Reviewers: rven, anthony, kradhakrishnan, igor, IslamAbdelRahman, sdong

Reviewed By: sdong

Subscribers: dhruba, leveldb

Differential Revision: https://reviews.facebook.net/D40341
      71b438c4