Skip to content

R
rocksdb

kvdb / rocksdb
11 个月 前同步成功

通知 0
Star 0
Fork 0
R
rocksdb

体验新版 GitCode，发现更多精彩内容 >>

切换分支/标签
  1. 01 3月, 2017 1 次提交
  3. 28 2月, 2017 2 次提交
  5. 24 2月, 2017 2 次提交
  7. 13 2月, 2017 1 次提交
  9. 11 2月, 2017 1 次提交
  11. 26 1月, 2017 1 次提交
    • A
      Generalize Env registration framework · 17c11806
      由 Andrew Kryczka 提交于 
      Summary:
The Env registration framework supports registering client Envs and selecting which one to instantiate according to a text field. This enabled things like adding the -env_uri argument to db_bench, so the same binary could be reused with different Envs just by changing CLI config.

Now this problem has come up again in a non-Env context, as I want to instantiate a client Statistics implementation from db_bench, which is configured entirely via text parameters. Also, in the future we may wish to use it for deserializing client objects when loading OPTIONS file.

This diff generalizes the Env registration logic to work with arbitrary types.

- Generalized registration and instantiation code by templating them
- The entire implementation is in a header file as that's Google style guide's recommendation for template definitions
- Pattern match with std::regex_match rather than checking prefix, which was the previous behavior
- Rename functions/files to be non-Env-specific
Closes https://github.com/facebook/rocksdb/pull/1776

Differential Revision: D4421933

Pulled By: ajkr

fbshipit-source-id: 34647d1
      17c11806
  13. 21 1月, 2017 1 次提交
  15. 30 12月, 2016 1 次提交
    • M
      Delegate Cleanables · 0712d541
      由 Maysam Yabandeh 提交于 
      Summary:
Cleanable objects will perform the registered cleanups when
they are destructed. We however rather to delay this cleaning like when
we are gathering the merge operands. Current approach is to create the
Cleanable object on heap (instead of on stack) and delay deleting it.

By allowing Cleanables to delegate their cleanups to another cleanable
object we can delay the cleaning without however the need to craete the
cleanable object on heap and keeping it around. This patch applies this
technique for the cleanups of BlockIter and shows improved performance
for some in-memory benchmarks:
+1.8% for merge worklaod, +6.4% for non-merge workload when the merge
operator is specified.
https://our.intern.facebook.com/intern/tasks?t=15168163

Non-merge benchmark:
TEST_TMPDIR=/dev/shm/v100nocomp/ ./db_bench --benchmarks=fillrandom
--num=1000000 -value_size=100 -compression_type=none

Reading random with no merge operator specified:
TEST_TMPDIR=/dev/shm/v100nocomp/ ./db_bench
--benchmarks="read
Closes https://github.com/facebook/rocksdb/pull/1711

Differential Revision: D4361163

Pulled By: maysamyabandeh

fbshipit-source-id: 9801e07
      0712d541
  17. 20 12月, 2016 1 次提交
    • A
      Collapse range deletions · 50e305de
      由 Andrew Kryczka 提交于 
      Summary:
Added a tombstone-collapsing mode to RangeDelAggregator, which eliminates overlap in the TombstoneMap. In this mode, we can check whether a tombstone covers a user key using upper_bound() (i.e., binary search). However, the tradeoff is the overhead to add tombstones is now higher, so at first I've only enabled it for range scans (compaction/flush/user iterators), where we expect a high number of calls to ShouldDelete() for the same tombstones. Point queries like Get() will still use the linear scan approach.

Also in this diff I changed RangeDelAggregator's TombstoneMap to use multimap with user keys instead of map with internal keys. Callers sometimes provided ParsedInternalKey directly, from which it would've required string copying to derive an internal key Slice with which we could search the map.
Closes https://github.com/facebook/rocksdb/pull/1614

Differential Revision: D4270397

Pulled By: ajkr

fbshipit-source-id: 93092c7
      50e305de
  19. 17 12月, 2016 1 次提交
  21. 14 12月, 2016 1 次提交
  23. 08 12月, 2016 1 次提交
    • J
      Specify shell in makefile · c04f6a0b
      由 Jonathan Lee 提交于 
      Summary:
The second variable "SHELL" simply tells make explicitly which shell to use, instead of allowing it to default to "/bin/sh", which may or may not be Bash.

However, simply defining the second variable by itself causes make to throw an error concerning a circular definition, as it would be attempting to use the "shell" command while simultaneously trying to set which shell to use. Thus, the first variable "BASH_EXISTS" is defined such that make already knows about "/path/to/bash" before trying to use it to set "SHELL".

A more technically correct solution would be to edit the makefile itself to make it compatible with non-bash shells (see the original Issue discussion for details). However, as it seems very few of the people working on this project were building with non-bash shells, I figured this solution would be good enough.
Closes https://github.com/facebook/rocksdb/pull/1631

Differential Revision: D4295689

Pulled By: yiwu-arbug

fbshipit-source-id: e4f9532
      c04f6a0b
  25. 30 11月, 2016 1 次提交
  27. 19 11月, 2016 1 次提交
  29. 17 11月, 2016 1 次提交
    • Y
      Introduce Lua Extension: RocksLuaCompactionFilter · 647eafdc
      由 Yueh-Hsuan Chiang 提交于 
      Summary:
This diff includes an implementation of CompactionFilter that allows
users to write CompactionFilter in Lua.  With this ability, users can
dynamically change compaction filter logic without requiring building
the rocksdb binary and restarting the database.

To compile, WITH_LUA_PATH must be specified to the base directory
of lua.
Closes https://github.com/facebook/rocksdb/pull/1478

Differential Revision: D4150138

Pulled By: yhchiang

fbshipit-source-id: ed84222
      647eafdc
  31. 15 11月, 2016 1 次提交
  33. 14 11月, 2016 1 次提交
    • Y
      Optimize sequential insert into memtable - Part 1: Interface · 1ea79a78
      由 Yi Wu 提交于 
      Summary:
Currently our skip-list have an optimization to speedup sequential
inserts from a single stream, by remembering the last insert position.
We extend the idea to support sequential inserts from multiple streams,
and even tolerate small reordering wihtin each stream.

This PR is the interface part adding the following:
- Add `memtable_insert_prefix_extractor` to allow specifying prefix for each key.
- Add `InsertWithHint()` interface to memtable, to allow underlying
  implementation to return a hint of insert position, which can be later
  pass back to optimize inserts.
- Memtable will maintain a map from prefix to hints and pass the hint
  via `InsertWithHint()` if `memtable_insert_prefix_extractor` is non-null.
Closes https://github.com/facebook/rocksdb/pull/1419

Differential Revision: D4079367

Pulled By: yiwu-arbug

fbshipit-source-id: 3555326
      1ea79a78
  35. 01 11月, 2016 1 次提交
  37. 26 10月, 2016 1 次提交
    • K
      Makefile: generate util/build_version.cc from .in file (#1384) · 60a2bbba
      由 Kefu Chai 提交于 
      * util/build_verion.cc.in: add this file, so cmake and make can share the
  template file for generating util/build_version.cc.
* CMakeLists.txt: also, cmake v2.8.11 does not support file(GENERATE ...),
  so we are using configure_file() for creating build_version.cc.
* Makefile: use util/build_verion.cc.in for creating build_version.cc.
Signed-off-by: NKefu Chai <tchaikov@gmail.com>
      60a2bbba
  39. 22 10月, 2016 1 次提交
    • A
      Passing DISABLE_JEMALLOC=1 to valgrind_check if run locally · 0e926b84
      由 Anirban Rahut 提交于 
      Summary:
Valgrind does not work well with JEMALLOC. If you run
a simple make valgrind_check, you will see lots of issues and
crashes. When precommit runs, this is taken care of. Here we
make sure valgrind_check is passed in DISABLE_JEMALLOC=1

Test Plan: Ran local valgrind_test and noticed the difference

Reviewers: IslamAbdelRahman

Reviewed By: IslamAbdelRahman

Subscribers: andrewkr, dhruba

Differential Revision: https://reviews.facebook.net/D65379
      0e926b84
  41. 04 10月, 2016 1 次提交
  43. 24 9月, 2016 1 次提交
    • Y
      Recover same sequence id from WAL (#1350) · 4bc8c88e
      由 yiwu-arbug 提交于 
      Summary:
Revert the behavior where we don't read sequence id from WAL, but increase it as we replay the log. We still keep the behave for 2PC for now but will fix later.

This change fixes github issue 1339, where some writes come with WAL disabled and we may recover records with wrong sequence id.

Test Plan: Added unit test.

Subscribers: andrewkr, dhruba

Differential Revision: https://reviews.facebook.net/D64275
      4bc8c88e
  45. 17 9月, 2016 1 次提交
    • Y
      Fix java makefile dependencies · 41a9070f
      由 Yi Wu 提交于 
      Summary: Fix dependencies in java makefile, to avoid java build failure.

Test Plan: run "make rocksdbjava" multiple times.

Reviewers: IslamAbdelRahman, sdong, yhchiang

Reviewed By: yhchiang

Subscribers: andrewkr, dhruba, leveldb

Differential Revision: https://reviews.facebook.net/D64023
      41a9070f
  47. 08 9月, 2016 1 次提交
  49. 29 8月, 2016 1 次提交
  51. 26 8月, 2016 1 次提交
    • I
      Fix parallel valgrind (valgrind_check) · 4b3438d2
      由 Islam AbdelRahman 提交于 
      Summary:
I just realized that when we run parallel valgrind we actually don't run the parallel tests under valgrind (we run the normally)
This patch make sure that we run both parallel and non-parallel tests with valgrind

Test Plan: DISABLE_JEMALLOC=1 make valgrind_check -j64

Reviewers: andrewkr, yiwu, lightmark, sdong

Reviewed By: sdong

Subscribers: andrewkr, dhruba

Differential Revision: https://reviews.facebook.net/D62469
      4b3438d2
  53. 20 8月, 2016 1 次提交
    • Y
      LRU cache mid-point insertion · 72f8cc70
      由 Yi Wu 提交于 
      Summary:
Add mid-point insertion functionality to LRU cache. Caller of `Cache::Insert()` can set an additional parameter to make a cache entry have higher priority. The LRU cache will reserve at most `capacity * high_pri_pool_pct` bytes for high-pri cache entries. If `high_pri_pool_pct` is zero, the cache degenerates to normal LRU cache.

Context: If we are to put index and filter blocks into RocksDB block cache, index/filter block can be swap out too early. We want to add an option to RocksDB to reserve some capacity in block cache just for index/filter blocks, to mitigate the issue.

In later diffs I'll update block based table reader to use the interface to cache index/filter blocks at high priority, and expose the option to `DBOptions` and make it dynamic changeable.

Test Plan: unit test.

Reviewers: IslamAbdelRahman, sdong, lightmark

Reviewed By: lightmark

Subscribers: andrewkr, dhruba, march, leveldb

Differential Revision: https://reviews.facebook.net/D61977
      72f8cc70
  55. 11 8月, 2016 4 次提交
    • K
      Persistent Read Cache (8) Benchmark tooling · 87c91bd8
      由 krad 提交于 
      Summary:
Adding benchmark tool for persistent read cache.

TODO: Add integration to db_bench

Test Plan: Compile

Subscribers: andrewkr, dhruba, leveldb

Differential Revision: https://reviews.facebook.net/D57711
      87c91bd8
    • S
      [Proof-Of-Concept] RocksDB Blob Storage with a blob log file. · 8b79422b
      由 sdong 提交于 
      Summary:
This is a proof of concept of a RocksDB blob log file. The actual value of the Put() is appended to a blob log using normal data block format, and the handle of the block is written as the value of the key in RocksDB.

The prototype only supports Put() and Get(). It doesn't support DB restart, garbage collection, Write() call, iterator, snapshots, etc.

Test Plan: Add unit tests.

Reviewers: arahut

Reviewed By: arahut

Subscribers: kradhakrishnan, leveldb, andrewkr, dhruba

Differential Revision: https://reviews.facebook.net/D61485
      8b79422b
    • S
      read_options.background_purge_on_iterator_cleanup to cover forward iterator... · 56dd0341
      由 sdong 提交于 
      read_options.background_purge_on_iterator_cleanup to cover forward iterator and log file closing too.

Summary: With read_options.background_purge_on_iterator_cleanup=true, File deletion and closing can still happen in forward iterator, or WAL file closing. Cover those cases too.

Test Plan: I am adding unit tests.

Reviewers: andrewkr, IslamAbdelRahman, yiwu

Reviewed By: yiwu

Subscribers: leveldb, andrewkr, dhruba

Differential Revision: https://reviews.facebook.net/D61503
      56dd0341
    • A
      Remove non-gtest from parallelized tests · 1b0069ce
      由 Andrew Kryczka 提交于 
      Summary:
compact_on_deletion_collector_test does not support --gtest_list_tests
since it isn't gtest, so the full program would run for the target
gen_parallel_tests. This caused gen_parallel_tests to take 8+ minutes for tsan
and prevented compact_on_deletion_collector_test from running during check_0
since no t/run-* script could be generated.

Test Plan:
run make check, verify generating t/run-* scripts is fast and
./compact_on_deletion_collector_test is now run

Reviewers: IslamAbdelRahman, wanning, lightmark, sdong

Reviewed By: sdong

Subscribers: andrewkr, dhruba, leveldb

Differential Revision: https://reviews.facebook.net/D61695
      1b0069ce
  57. 06 8月, 2016 2 次提交
    • O
      Add time series database (resubmitted) · 44f5cc57
      由 omegaga 提交于 
      Summary: Implement a time series database that supports DateTieredCompactionStrategy. It wraps a db object and separate SST files in different column families (time windows).

Test Plan: Add `date_tiered_test`.

Reviewers: dhruba, sdong

Reviewed By: sdong

Subscribers: andrewkr, dhruba, leveldb

Differential Revision: https://reviews.facebook.net/D61653
      44f5cc57
    • S
      A utility function to help users migrate DB after options change · 7c4615cf
      由 sdong 提交于 
      Summary: Add a utility function that trigger necessary full compaction and put output to the correct level by looking at new options and old options.

Test Plan: Add unit tests for it.

Reviewers: andrewkr, igor, IslamAbdelRahman

Reviewed By: IslamAbdelRahman

Subscribers: muthu, sumeet, leveldb, andrewkr, dhruba

Differential Revision: https://reviews.facebook.net/D60783
      7c4615cf
  59. 02 8月, 2016 2 次提交
    • I
      Fix parallel tests `make check -j` · 0155c73d
      由 Islam AbdelRahman 提交于 
      Summary:
parallel tests are broken because gnu_parallel is reading deprecated options from `/etc/parallel/config`
Fix this by passing `--plain` to ignore `/etc/parallel/config`

Test Plan: make check -j64

Reviewers: kradhakrishnan, sdong, andrewkr, yiwu, arahut

Reviewed By: arahut

Subscribers: andrewkr, dhruba

Differential Revision: https://reviews.facebook.net/D61359
      0155c73d
    • O
      Experiments on column-aware encodings · d51dc96a
      由 omegaga 提交于 
      Summary:
Experiments on column-aware encodings. Supported features: 1) extract data blocks from SST file and encode with specified encodings; 2) Decode encoded data back into row format; 3) Directly extract data blocks and write in row format (without prefix encoding); 4) Get column distribution statistics for column format; 5) Dump data blocks separated by columns in human-readable format.

There is still on-going work on this diff. More refactoring is necessary.

Test Plan: Wrote tests in `column_aware_encoding_test.cc`. More tests should be added.

Reviewers: sdong

Reviewed By: sdong

Subscribers: arahut, andrewkr, dhruba

Differential Revision: https://reviews.facebook.net/D60027
      d51dc96a
  61. 30 7月, 2016 1 次提交
  63. 28 7月, 2016 1 次提交
  65. 26 7月, 2016 1 次提交