Skip to content

R
rocksdb

kvdb / rocksdb
12 个月 前同步成功

通知 0
Star 0
Fork 0
R
rocksdb

体验新版 GitCode，发现更多精彩内容 >>

切换分支/标签
  1. 14 6月, 2017 6 次提交
    • S
      Allow ignoring unknown options when loading options from a file · 89ad9f3a
      由 Sagar Vemuri 提交于 
      Summary:
Added a flag, `ignore_unknown_options`, to skip unknown options when loading an options file (using `LoadLatestOptions`/`LoadOptionsFromFile`) or while verifying options (using `CheckOptionsCompatibility`). This will help in downgrading the db to an older version.

Also added `--ignore_unknown_options` flag to ldb

**Example Use case:**
In MyRocks, if copying from newer version to older version, it is often impossible to start because of new RocksDB options that don't exist in older version, even though data format is compatible.
MyRocks uses these load and verify functions in [ha_rocksdb.cc::check_rocksdb_options_compatibility](https://github.com/facebook/mysql-5.6/blob/e004fd9f416821d043ccc8ad4a345c33ac9953f0/storage/rocksdb/ha_rocksdb.cc#L3348-L3401).

**Test Plan:**
Updated the unit tests.
`make check`

ldb:
$ ./ldb --db=/tmp/test_db --create_if_missing put a1 b1
OK

Now edit /tmp/test_db/<OPTIONS-file> and add an unknown option.

Try loading the options now, and it fails:
$ ./ldb --db=/tmp/test_db --try_load_options get a1
Failed: Invalid argument: Unrecognized option DBOptions:: abcd

Passes with the new --ignore_unknown_options flag
$ ./ldb --db=/tmp/test_db --try_load_options --ignore_unknown_options get a1
b1
Closes https://github.com/facebook/rocksdb/pull/2423

Differential Revision: D5212091

Pulled By: sagar0

fbshipit-source-id: 2ec17636feb47dc0351b53a77e5f15ef7cbf2ca7
      89ad9f3a
    • H
      fixed typo · 6b5a5dc5
      由 hyunwoo 提交于 
      Summary:
fixed typo
Closes https://github.com/facebook/rocksdb/pull/2430

Differential Revision: D5242471

Pulled By: IslamAbdelRahman

fbshipit-source-id: 832eb3a4c70221444ccd2ae63217823fec56c748
      6b5a5dc5
    • H
      fixed typo in util/dynamic_bloom.h · 0f228be3
      由 haoxiang 提交于 
      Summary:
fixed a typo in util/dynamic_bloom.h
Closes https://github.com/facebook/rocksdb/pull/2442

Differential Revision: D5242397

Pulled By: IslamAbdelRahman

fbshipit-source-id: c47fd18cc79afff6b022201a0410c0cd47626576
      0f228be3
    • A
      Call RateLimiter for compaction reads · c217e0b9
      由 Andrew Kryczka 提交于 
      Summary:
Allow users to rate limit background work based on read bytes, written bytes, or sum of read and written bytes. Support these by changing the RateLimiter API, so no additional options were needed.
Closes https://github.com/facebook/rocksdb/pull/2433

Differential Revision: D5216946

Pulled By: ajkr

fbshipit-source-id: aec57a8357dbb4bfde2003261094d786d94f724e
      c217e0b9
    • Y
      write exact sequence number for each put in write batch · 91e2aa3c
      由 Yi Wu 提交于 
      Summary:
At the beginning of write batch write, grab the latest sequence from base db and assume sequence number will increment by 1 for each put and delete, and write the exact sequence number with each put. This is assuming we are the only writer to increment sequence number (no external file ingestion, etc) and there should be no holes in the sequence number.

Also having some minor naming changes.
Closes https://github.com/facebook/rocksdb/pull/2402

Differential Revision: D5176134

Pulled By: yiwu-arbug

fbshipit-source-id: cb4712ee44478d5a2e5951213a10b72f08fe8c88
      91e2aa3c
    • M
      record index partition properties · 6f4154d6
      由 Maysam Yabandeh 提交于 
      Summary:
When Partitioning index/filter is enabled the user might need to check the index block size as well as the top-level index size via sst_dump. This patch records i) number of partitions, ii) top-level index size and make it accessible through sst_dump. The number of partitions for filters is the same as that of indexes. The top-level index for filters has a similar size to top-level index for indexes, so it is not repeated.
Closes https://github.com/facebook/rocksdb/pull/2437

Differential Revision: D5224225

Pulled By: maysamyabandeh

fbshipit-source-id: 5324598c75793523aef1bb7ee225a5475e95a9cb
      6f4154d6
  3. 13 6月, 2017 7 次提交
  5. 12 6月, 2017 2 次提交
  7. 10 6月, 2017 1 次提交
  9. 09 6月, 2017 5 次提交
  11. 07 6月, 2017 2 次提交
  13. 06 6月, 2017 7 次提交
  15. 05 6月, 2017 1 次提交
  17. 03 6月, 2017 6 次提交
    • A
      remove test dir before exit when current regression is running · 7e5fac2c
      由 Aaron Gao 提交于 
      Summary:
clean up the current test dir if the last regression test is still running.
Closes https://github.com/facebook/rocksdb/pull/2401

Differential Revision: D5177882

Pulled By: lightmark

fbshipit-source-id: 91d899fcc2bde841948eae71af8584d4bdb35468
      7e5fac2c
    • A
      using ThreadLocalPtr to hide ROCKSDB_SUPPORT_THREAD_LOCAL from public… · 7f6c02dd
      由 Aaron Gao 提交于 
      Summary:
… headers

https://github.com/facebook/rocksdb/pull/2199 should not reference RocksDB-specific macros (like ROCKSDB_SUPPORT_THREAD_LOCAL in this case) to public headers, `iostats_context.h` and `perf_context.h`. We shouldn't do that because users have to provide these compiler flags when building their binary with RocksDB.

We should hide the thread local global variable inside our implementation and just expose a function api to retrieve these variables. It may break some users for now but good for long term.

make check -j64
Closes https://github.com/facebook/rocksdb/pull/2380

Differential Revision: D5177896

Pulled By: lightmark

fbshipit-source-id: 6fcdfac57f2e2dcfe60992b7385c5403f6dcb390
      7f6c02dd
    • M
      Fix interaction between CompactionFilter::Decision::kRemoveAndSkipUnt… · 138b87ea
      由 Mike Kolupaev 提交于 
      Summary:
Fixes the following scenario:
 1. Set prefix extractor. Enable bloom filters, with `whole_key_filtering = false`. Use compaction filter that sometimes returns `kRemoveAndSkipUntil`.
 2. Do a compaction.
 3. Compaction creates an iterator with `total_order_seek = false`, calls `SeekToFirst()` on it, then repeatedly calls `Next()`.
 4. At some point compaction filter returns `kRemoveAndSkipUntil`.
 5. Compaction calls `Seek(skip_until)` on the iterator. The key that it seeks to happens to have prefix that doesn't match the bloom filter. Since `total_order_seek = false`, iterator becomes invalid, and compaction thinks that it has reached the end. The rest of the compaction input is silently discarded.

The fix is to make compaction iterator use `total_order_seek = true`.

The implementation for PlainTable is quite awkward. I've made `kRemoveAndSkipUntil` officially incompatible with PlainTable. If you try to use them together, compaction will fail, and DB will enter read-only mode (`bg_error_`). That's not a very graceful way to communicate a misconfiguration, but the alternatives don't seem worth the implementation time and complexity. To be able to check in advance that `kRemoveAndSkipUntil` is not going to be used with PlainTable, we'd need to extend the interface of either `CompactionFilter` or `InternalIterator`. It seems unlikely that anyone will ever want to use `kRemoveAndSkipUntil` with PlainTable: PlainTable probably has very few users, and `kRemoveAndSkipUntil` has only one user so far: us (logdevice).
Closes https://github.com/facebook/rocksdb/pull/2349

Differential Revision: D5110388

Pulled By: lightmark

fbshipit-source-id: ec29101a99d9dcd97db33923b87f72bce56cc17a
      138b87ea
    • S
      Improve write buffer manager (and allow the size to be tracked in block cache) · 95b0e89b
      由 Siying Dong 提交于 
      Summary:
Improve write buffer manager in several ways:
1. Size is tracked when arena block is allocated, rather than every allocation, so that it can better track actual memory usage and the tracking overhead is slightly lower.
2. We start to trigger memtable flush when 7/8 of the memory cap hits, instead of 100%, and make 100% much harder to hit.
3. Allow a cache object to be passed into buffer manager and the size allocated by memtable can be costed there. This can help users have one single memory cap across block cache and memtable.
Closes https://github.com/facebook/rocksdb/pull/2350

Differential Revision: D5110648

Pulled By: siying

fbshipit-source-id: b4238113094bf22574001e446b5d88523ba00017
      95b0e89b
    • A
      Pass CF ID to MemTableRepFactory · a4d9c025
      由 Andrew Kryczka 提交于 
      Summary:
Some users want to monitor column family activity in their custom memtable implementations. Previously there was no way to figure out with which column family a memtable is associated. This diff:

- adds an overload to MemTableRepFactory::CreateMemTableRep() that provides the CF ID. For compatibility, its default implementation calls the old overload.
- updates MemTable to create MemTableRep's using the new overload.
Closes https://github.com/facebook/rocksdb/pull/2346

Differential Revision: D5108061

Pulled By: ajkr

fbshipit-source-id: 3a1921214a348dd8ea0f54e1cab3b71c3d46d616
      a4d9c025
    • Y
      Fix DBWriteTest::ReturnSequenceNumberMultiThreaded data race · f68d88be
      由 Yi Wu 提交于 
      Summary:
rocksdb::Random is not thread-safe. Have one Random for each thread instead.
Closes https://github.com/facebook/rocksdb/pull/2400

Differential Revision: D5173919

Pulled By: yiwu-arbug

fbshipit-source-id: 1a99c7b877f3893eb22355af49e321bcad4e53e6
      f68d88be
  19. 02 6月, 2017 3 次提交