Skip to content

R
rocksdb

kvdb / rocksdb
11 个月 前同步成功

通知 0
Star 0
Fork 0
R
rocksdb

体验新版 GitCode，发现更多精彩内容 >>

切换分支/标签
  1. 13 2月, 2018 2 次提交
  3. 10 2月, 2018 5 次提交
  5. 08 2月, 2018 4 次提交
    • A
      Eliminate a memcpy for uncompressed blocks · e78715c2
      由 Andrew Kryczka 提交于 
      Summary:
`ReadBlockFromFile` uses a stack buffer to hold small data blocks before passing them to the compression library, which outputs uncompressed data in a heap buffer. In the case of `kNoCompression` there is a `memcpy` to copy from stack buffer to heap buffer.

This PR optimizes `ReadBlockFromFile` to skip the stack buffer for files whose blocks are known to be uncompressed. We determine this using the SST file property, "compression_name", if it's available.
Closes https://github.com/facebook/rocksdb/pull/3472

Differential Revision: D6920848

Pulled By: ajkr

fbshipit-source-id: 5c753e804efc178b9229ae5dbe6a4adc32031f07
      e78715c2
    • S
      Fix UBSAN Error in WritePreparedTransactionTest · a0931b31
      由 Siying Dong 提交于 
      Summary:
WritePreparedTransactionTest has the UBSAN error because the wrong order of its parent class construction. Fix it.
Closes https://github.com/facebook/rocksdb/pull/3478

Differential Revision: D6928975

Pulled By: siying

fbshipit-source-id: 13edfd5cb9cf73f1ac5ae3b6f53061d32783733d
      a0931b31
    • S
      Disable options_settable_test in UBSAN and fix UBSAN failure in blob_… · 821e0b16
      由 Siying Dong 提交于 
      Summary:
…db_test

options_settable_test won't pass UBSAN so disable it.
blob_db_test fails in UBSAN as SnapshotList doesn't initialize all the fields in dummy snapshot. Fix it. I don't understand why only blob_db_test fails though.
Closes https://github.com/facebook/rocksdb/pull/3477

Differential Revision: D6928681

Pulled By: siying

fbshipit-source-id: e31dd300fcdecdfd4f6af279a0987fd0cdec5122
      821e0b16
    • S
      Disable alignment check in UBSAN · 1336a774
      由 Siying Dong 提交于 
      Summary:
Disable alignment check in UBSAN for now. Now we can't get signals to meaningful failures. We can reenable it after we figure out how we can suppress failures in finer grain manner.
Closes https://github.com/facebook/rocksdb/pull/3473

Differential Revision: D6925971

Pulled By: siying

fbshipit-source-id: a0f1a242cde866abbc5c1eeee9ff8d1d7d582ac4
      1336a774
  7. 07 2月, 2018 4 次提交
    • M
      Add skip_cc option to TransactionDB::Write · 8feee280
      由 Maysam Yabandeh 提交于 
      Summary:
Compared to DB::Write, TransactionDB::Write has the additional overhead of creating and initializing an internal transaction object, as well as the overhead of locking/unlocking the keys. This patch extends the TransactionDB::Write with an skip_cc option to allow the users to indicate that the write batch do not conflict with others and the concurrency control and its overhead can be skipped. TransactionDB::Write by default calls DB::Write when skip_cc is set, which works for WriteCommitted WritePolicy. Any other flavor of TransactionDB that is not compatible with this default behavior (such as WritePreparedTxnDB) can extend ::Write and implement their own approach for taking into account the skip_cc optimization.
Closes https://github.com/facebook/rocksdb/pull/3457

Differential Revision: D6877318

Pulled By: maysamyabandeh

fbshipit-source-id: 56f4e21db87ff71492db4e376fb7c2b03dfeab6b
      8feee280
    • M
      Fix leak report by asan on DuplicateKeys test · 8f8eb4f1
      由 Maysam Yabandeh 提交于 
      Summary:
Deletes the transaction object at the end of the test.
Verified by:
- COMPILE_WITH_ASAN=1 make -j32 transaction_test
- ./transaction_test --gtest_filter="DBA**Duplicate*"
Closes https://github.com/facebook/rocksdb/pull/3470

Differential Revision: D6916473

Pulled By: maysamyabandeh

fbshipit-source-id: 8303df25408635d5d3ac2b25f309a3d15957c937
      8f8eb4f1
    • Y
      WritePrepared Txn: update compaction_iterator_test and db_iterator_test · 81736d8a
      由 Yi Wu 提交于 
      Summary:
Update compaction_iterator_test with write-prepared transaction DB related tests. Transaction related tests are group in CompactionIteratorWithSnapshotCheckerTest. The existing test are duplicated to make them also test with dummy SnapshotChecker that will say every key is visible to every snapshot (this is okay, we still compare sequence number to verify visibility). Merge related tests are disabled and will be revisit in another PR.

Existing db_iterator_tests are also duplicated to test with dummy read_callback that will say every key is committed.
Closes https://github.com/facebook/rocksdb/pull/3466

Differential Revision: D6909253

Pulled By: yiwu-arbug

fbshipit-source-id: 2ae4656b843a55e2e9ff8beecf21f2832f96cd25
      81736d8a
    • Z
      split RandomizedHarnessTest more ways · 2f299917
      由 Zhongyi Xie 提交于 
      Summary:
RandomizedHarnessTest enumerates different combinations of test type, compression type, restart interval, etc. For some combinations it takes very long to finish, causing the test to time out in test infrastructure.
This PR split the test input into smaller trunks in the hope that they will fit in the timeout window. Another possibility is to reduce `num_entries` of course
Closes https://github.com/facebook/rocksdb/pull/3467

Differential Revision: D6910235

Pulled By: miasantreble

fbshipit-source-id: 717246ee5d21a8a48ad82d4d9c04f9051a66f07f
      2f299917
  9. 06 2月, 2018 2 次提交
    • M
      WritePrepared Txn: Duplicate Keys, Txn Part · 88d8b2a2
      由 Maysam Yabandeh 提交于 
      Summary:
This patch takes advantage of memtable being able to detect duplicate <key,seq> and returning TryAgain to handle duplicate keys in WritePrepared Txns. Through WriteBatchWithIndex's index it detects existence of at least a duplicate key in the write batch. If duplicate key was reported, it then pays the cost of counting the number of sub-patches by iterating over the write batch and pass it to DBImpl::Write. DB will make use of the provided batch_count to assign proper sequence numbers before sending them to the WAL. When later inserting the batch to the memtable, it increases the seq each time memtbale reports a duplicate (a sub-patch in our counting) and tries again.
Closes https://github.com/facebook/rocksdb/pull/3455

Differential Revision: D6873699

Pulled By: maysamyabandeh

fbshipit-source-id: db8487526c3a5dc1ddda0ea49f0f979b26ae648d
      88d8b2a2
    • A
      Handle error return from WriteBuffer() · 4b124fb9
      由 Anand Ananthabhotla 提交于 
      Summary:
There are a couple of places where we swallow any error from
WriteBuffer() - in SwitchMemtable() and DBImpl::CloseImpl(). Propagate
the error up in those cases rather than ignoring it.
Closes https://github.com/facebook/rocksdb/pull/3404

Differential Revision: D6879954

Pulled By: anand1976

fbshipit-source-id: 2ef88b554be5286b0a8bad7384ba17a105395bdb
      4b124fb9
  11. 04 2月, 2018 1 次提交
  13. 03 2月, 2018 3 次提交
  15. 02 2月, 2018 4 次提交
    • P
      options: Fix coverity issues · 6e5b341e
      由 Prashant D 提交于 
      Summary:
options/cf_options.cc:
 77      memtable_insert_with_hint_prefix_extractor(

CID 1396208 (#1 of 1): Uninitialized scalar field (UNINIT_CTOR)
2. uninit_member: Non-static class member info_log_level is not initialized in this constructor nor in any functions that it calls.
Closes https://github.com/facebook/rocksdb/pull/3106

Differential Revision: D6874689

Pulled By: sagar0

fbshipit-source-id: b5cd2d13915fd86d87260050f9c5d117615bbe30
      6e5b341e
    • J
      crc32: suppress -Wimplicit-fallthrough warnings · e502839e
      由 Jun Wu 提交于 
      Summary:
Workaround a bunch of "implicit-fallthrough" compiler errors, like:

```
util/crc32c.cc:533:7: error: this statement may fall through [-Werror=implicit-fallthrough=]
   crc = _mm_crc32_u64(crc, *(uint64_t*)(buf + offset));
       ^
util/crc32c.cc:1016:9: note: in expansion of macro ‘CRCsinglet’
         CRCsinglet(crc0, next, -2 * 8);
         ^~~~~~~~~~
util/crc32c.cc:1017:7: note: here
       case 1:
```
Closes https://github.com/facebook/rocksdb/pull/3339

Reviewed By: sagar0

Differential Revision: D6874736

Pulled By: quark-zju

fbshipit-source-id: eec9f3bc135e12fca336928d01711006d5c3cb16
      e502839e
    • F
      Upgrade Appveyor to VS2017 · ba8aa8fd
      由 Fosco Marotto 提交于 
      Summary:
Per some discussions, this will switch our Appveyor testing to use Visual Studio 2017.
Closes https://github.com/facebook/rocksdb/pull/3445

Differential Revision: D6874918

Pulled By: gfosco

fbshipit-source-id: c5a0032ca9f37f0d3baeae35c59d850d528c3176
      ba8aa8fd
    • A
      fix ReadaheadRandomAccessFile/iterator prefetch bug · b78ed046
      由 Andrew Kryczka 提交于 
      Summary:
`ReadaheadRandomAccessFile` is used by iterators for file reads in several cases, like in compaction when `compaction_readahead_size > 0` or `use_direct_io_for_flush_and_compaction == true`, or in user iterator when `ReadOptions::readahead_size > 0`. `ReadaheadRandomAccessFile` maintains an internal buffer for readahead data. It assumes that, if the buffer's length is less than `ReadaheadRandomAccessFile::readahead_size_`, which is fixed in the constructor, then EOF has been reached so it doesn't try reading further.

Recently, d938226a started calling `RandomAccessFile::Prefetch` with various lengths: 8KB, 16KB, etc. When the `RandomAccessFile` is a `ReadaheadRandomAccessFile`, it triggers the above condition and incorrectly determines EOF. If a block is partially in the readahead buffer and EOF is incorrectly decided, the result is a truncated data block.

The problem is reproducible:

```
TEST_TMPDIR=/data/compaction_bench ./db_bench -benchmarks=fillrandom -write_buffer_size=1048576 -target_file_size_base=1048576 -block_size=18384 -use_direct_io_for_flush_and_compaction=true
...
put error: Corruption: truncated block read from /data/compaction_bench/dbbench/000014.sst offset 20245, expected 10143 bytes, got 8427
```
Closes https://github.com/facebook/rocksdb/pull/3454

Differential Revision: D6869405

Pulled By: ajkr

fbshipit-source-id: 87001c299e7600a37c0dcccbd0368e0954c929cf
      b78ed046
  17. 01 2月, 2018 6 次提交
  19. 31 1月, 2018 4 次提交
  21. 30 1月, 2018 5 次提交
    • A
      fix for checkpoint directory with trailing slash(es) · f3fe6f88
      由 Andrew Kryczka 提交于 
      Summary:
previously if `checkpoint_dir` contained a trailing slash, we'd attempt to create the `.tmp` directory under `checkpoint_dir` due to simply concatenating `checkpoint_dir + ".tmp"`. This failed because `checkpoint_dir` hadn't been created yet and our directory creation is non-recursive. This PR fixes the issue by always creating the `.tmp` directory in the same parent as `checkpoint_dir` by stripping trailing slashes before concatenating.
Closes https://github.com/facebook/rocksdb/pull/3275

Differential Revision: D6574952

Pulled By: ajkr

fbshipit-source-id: a6daa6777a901eac2460cd0140c9515f7241aefc
      f3fe6f88
    • Y
      Fix DBFlushTest::ManualFlushWithMinWriteBufferNumberToMerge dead lock · 4bdf06e7
      由 Yi Wu 提交于 
      Summary:
In the test, there can be a dead lock between background flush thread and foreground main thread as following:
* background flush thread:
  - holding db mutex, while
  - waiting on "DBImpl::FlushMemTableToOutputFile:BeforeInstallSV" sync point.
* foreground thread:
  - waiting for db mutex to write "key2"

Fixing by let background flush thread wait without holding db mutex.
Closes https://github.com/facebook/rocksdb/pull/3436

Differential Revision: D6841334

Pulled By: yiwu-arbug

fbshipit-source-id: b020768ac94e166e40953c5d09e505515a5f244d
      4bdf06e7
    • M
      Split SnapshotConcurrentAccessTest into 20 sub tests · 3073b1c5
      由 Maysam Yabandeh 提交于 
      Summary:
SnapshotConcurrentAccessTest sometimes times out when running on the test infra. This patch splits the test into smaller sub-tests to avoid the timeout. It also benefits from lower run-time of each sub-test and increases the coverage of the test. The overall run-time of each final sub-test is at most half of the original test so we should no longer see a timeout.
Closes https://github.com/facebook/rocksdb/pull/3435

Differential Revision: D6839427

Pulled By: maysamyabandeh

fbshipit-source-id: d53fdb157109e2438ca7fe447d0cf4b71f304bd8
      3073b1c5
    • S
      Tests for dynamic universal compaction options · e6605e53
      由 Sagar Vemuri 提交于 
      Summary:
Added a test for three dynamic universal compaction options, in the realm of read amplification:
- size_ratio
- min_merge_width
- max_merge_width

Also updated DynamicUniversalCompactionSizeAmplification by adding a check on compaction reason.
Found a bug in compaction reason setting while working on this PR, and fixed in #3412 .

TODO for later: Still to add tests for these options: compression_size_percent, stop_style and trivial_move.
Closes https://github.com/facebook/rocksdb/pull/3419

Differential Revision: D6822217

Pulled By: sagar0

fbshipit-source-id: 074573fca6389053cbac229891a0163f38bb56c4
      e6605e53
    • Z
      Use block cache to track memory usage when ReadOptions.fill_cache=false · 3fe09371
      由 Zhongyi Xie 提交于 
      Summary:
ReadOptions.fill_cache is set in compaction inputs and can be set by users in their queries too. It tells RocksDB not to put a data block used to block cache.

The memory used by the data block is, however, not trackable by users.

To make the system more manageable, we can cost the block to block cache while using it, and then release it after using.
Closes https://github.com/facebook/rocksdb/pull/3333

Differential Revision: D6670230

Pulled By: miasantreble

fbshipit-source-id: ab848d3ed286bd081a13ee1903de357b56cbc308
      3fe09371