Skip to content

R
rocksdb

kvdb / rocksdb
12 个月 前同步成功

通知 0
Star 0
Fork 0
R
rocksdb

体验新版 GitCode，发现更多精彩内容 >>

切换分支/标签
  1. 12 7月, 2017 4 次提交
    • A
      Improve the design and native object management of Stats in RocksJava · 000bf0af
      由 Adam Retter 提交于 
      Summary: Closes https://github.com/facebook/rocksdb/pull/2551

Differential Revision: D5399288

Pulled By: sagar0

fbshipit-source-id: dd3df2ed6cc5ae612db0998ea746cc29fccf568e
      000bf0af
    • I
      Bump version to 5.7 · 269d383d
      由 Islam AbdelRahman 提交于 
      Summary:
Bump version to 5.7
Closes https://github.com/facebook/rocksdb/pull/2566

Differential Revision: D5400043

Pulled By: IslamAbdelRahman

fbshipit-source-id: 74aae4ff143d370d7d89807e5be08a6ab827da40
      269d383d
    • Y
      Fixes db_bench with blob db · c32f2722
      由 Yi Wu 提交于 
      Summary:
* Create info log before db open to make blob db able to log to LOG file.
* Properly destroy blob db.
Closes https://github.com/facebook/rocksdb/pull/2567

Differential Revision: D5400034

Pulled By: yiwu-arbug

fbshipit-source-id: a49cfaf4b5c67d42d4cbb872bd5a9441828c17ce
      c32f2722
    • D
      db_bench_tool: fix buffer size · fcd99d27
      由 Daniel Black 提交于 
      Summary:
Found by gcc warning:

x86_64-pc-linux-gnu-g++ --version
x86_64-pc-linux-gnu-g++ (GCC) 7.1.1 20170710

tools/db_bench_tool.cc: In member function 'void rocksdb::Benchmark::RandomWithVerify(rocksdb::ThreadState*)':
tools/db_bench_tool.cc:4430:8: error: '%lu' directive output may be truncated writing between 1 and 19 bytes into a region of size between 0 and 66 [-Werror=format-truncation=]
   void RandomWithVerify(ThreadState* thread) {
        ^~~~~~~~~~~~~~~~
tools/db_bench_tool.cc:4430:8: note: directive argument in the range [0, 9223372036854775807]
tools/db_bench_tool.cc:4492:13: note: 'snprintf' output between 37 and 128 bytes into a destination of size 100
     snprintf(msg, sizeof(msg),
     ~~~~~~~~^~~~~~~~~~~~~~~~~~
              "( get:%" PRIu64 " put:%" PRIu64 " del:%" PRIu64 " total:%" \
              ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
              PRIu64 " found:%" PRIu64 ")",
              ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
              gets_done, puts_done, deletes_done, readwrites_, found);
              ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
cc1plus: all warnings being treated as errors
Makefile:1707: recipe for target 'tools/db_bench_tool.o' failed
Closes https://github.com/facebook/rocksdb/pull/2558

Differential Revision: D5398703

Pulled By: siying

fbshipit-source-id: 6ffa552bbd8b59cfc2c36289f86ff9b9acca8ca6
      fcd99d27
  3. 11 7月, 2017 3 次提交
  5. 08 7月, 2017 3 次提交
  7. 07 7月, 2017 2 次提交
  9. 06 7月, 2017 5 次提交
  11. 03 7月, 2017 1 次提交
  13. 01 7月, 2017 4 次提交
  15. 30 6月, 2017 4 次提交
    • A
      Regression test for empty dedicated range deletion file · d310e0f3
      由 Andrew Kryczka 提交于 
      Summary:
Issue: #2478
Fix: #2503

The bug happened when all of these conditions were satisfied:

- A subcompaction generates no keys
- `RangeDelAggregator::ShouldAddTombstones()` returns true because there's at least one non-obsoleted range deletion in its map
- None of the non-obsolete tombstones overlap with the subcompaction key-range

Under those conditions, we were creating a dedicated file for range deletions which was left empty, thus causing an error in VersionEdit.

I verified this test case fails before the #2503 fix and passes after.
Closes https://github.com/facebook/rocksdb/pull/2521

Differential Revision: D5352568

Pulled By: ajkr

fbshipit-source-id: f619cae39984ce9bb9b7a4e7a9ac0f2bb2ce43e9
      d310e0f3
    • M
      Add a fetch_add variation to AddDBStats · e9f91a51
      由 Maysam Yabandeh 提交于 
      Summary:
AddDBStats is in two steps of load and store, which is more efficient than fetch_add. This is however not thread-safe. Currently we have to protect concurrent access to AddDBStats with a mutex which is less efficient that fetch_add.

This patch adds the option to do fetch_add when AddDBStats. The results for my 2pc benchmark on sysbench is:
- vanilla: 68618 tps
- removing mutex on AddDBStats (unsafe): 69767 tps
- fetch_add for all AddDBStats: 69200 tps
- fetch_add only for concurrently access AddDBStats (this patch): 69579 tps
Closes https://github.com/facebook/rocksdb/pull/2505

Differential Revision: D5330656

Pulled By: maysamyabandeh

fbshipit-source-id: af64d7bee135b0e86b4fac323a4f9d9113eaa383
      e9f91a51
    • Z
      skip generating empty sst · c1b375e9
      由 zhangjinpeng1987 提交于 
      Summary:
When a compaction job output nothing, there is no necessary to generate a empty sst file which will cause `VersionEdit::EncodeTo` failed.
ref https://github.com/facebook/rocksdb/issues/2478
Closes https://github.com/facebook/rocksdb/pull/2503

Differential Revision: D5350799

Pulled By: ajkr

fbshipit-source-id: df0b4fcf3507fe1c3c435208b762e75478e00143
      c1b375e9
    • Y
      fix format compatible test · 67b417d6
      由 Yi Wu 提交于 
      Summary:
The comma "," is not a valid separator for bash arrays.
Closes https://github.com/facebook/rocksdb/pull/2516

Differential Revision: D5348101

Pulled By: yiwu-arbug

fbshipit-source-id: 8f0afdac368e21076eb7366b7df7dbaaf158cf96
      67b417d6
  17. 29 6月, 2017 4 次提交
    • S
      Bug fix: Fast CRC Support printing is not honest · afbef651
      由 Siying Dong 提交于 
      Summary:
11c5d474 introduces a bug that IsFastCrc32Supported() returns wrong result. Fix it. Also fix some FB internal scripts.
Closes https://github.com/facebook/rocksdb/pull/2513

Differential Revision: D5343802

Pulled By: yiwu-arbug

fbshipit-source-id: 057dc7ae3b262fe951413d1190ce60afc788cc05
      afbef651
    • M
      Improve Status message for block checksum mismatches · 397ab111
      由 Mike Kolupaev 提交于 
      Summary:
We've got some DBs where iterators return Status with message "Corruption: block checksum mismatch" all the time. That's not very informative. It would be much easier to investigate if the error message contained the file name - then we would know e.g. how old the corrupted file is, which would be very useful for finding the root cause. This PR adds file name, offset and other stuff to some block corruption-related status messages.

It doesn't improve all the error messages, just a few that were easy to improve. I'm mostly interested in "block checksum mismatch" and "Bad table magic number" since they're the only corruption errors that I've ever seen in the wild.
Closes https://github.com/facebook/rocksdb/pull/2507

Differential Revision: D5345702

Pulled By: al13n321

fbshipit-source-id: fc8023d43f1935ad927cef1b9c55481ab3cb1339
      397ab111
    • S
      Make "make analyze" happy · 18c63af6
      由 Siying Dong 提交于 
      Summary:
"make analyze" is reporting some errors. It's complicated to look but it seems to me that they are all false positive. Anyway, I think cleaning them up is a good idea. Some of the changes are hacky but I don't know a better way.
Closes https://github.com/facebook/rocksdb/pull/2508

Differential Revision: D5341710

Pulled By: siying

fbshipit-source-id: 6070e430e0e41a080ef441e05e8ec827d45efab6
      18c63af6
    • M
      Fix the reported asan issues · 01534db2
      由 Maysam Yabandeh 提交于 
      Summary:
This is to resolve the asan complains. In the meanwhile I am working on clarifying/revisiting the sync rules.
Closes https://github.com/facebook/rocksdb/pull/2510

Differential Revision: D5338660

Pulled By: yiwu-arbug

fbshipit-source-id: ce6f6e0826d43a2c0bfa4328a00c78f73cd6498a
      01534db2
  19. 28 6月, 2017 5 次提交
    • S
      FIFO Compaction with TTL · 1cd45cd1
      由 Sagar Vemuri 提交于 
      Summary:
Introducing FIFO compactions with TTL.

FIFO compaction is based on size only which makes it tricky to enable in production as use cases can have organic growth. A user requested an option to drop files based on the time of their creation instead of the total size.

To address that request:
- Added a new TTL option to FIFO compaction options.
- Updated FIFO compaction score to take TTL into consideration.
- Added a new table property, creation_time, to keep track of when the SST file is created.
- Creation_time is set as below:
  - On Flush: Set to the time of flush.
  - On Compaction: Set to the max creation_time of all the files involved in the compaction.
  - On Repair and Recovery: Set to the time of repair/recovery.
  - Old files created prior to this code change will have a creation_time of 0.
- FIFO compaction with TTL is enabled when ttl > 0. All files older than ttl will be deleted during compaction. i.e. `if (file.creation_time < (current_time - ttl)) then delete(file)`. This will enable cases where you might want to delete all files older than, say, 1 day.
- FIFO compaction will fall back to the prior way of deleting files based on size if:
  - the creation_time of all files involved in compaction is 0.
  - the total size (of all SST files combined) does not drop below `compaction_options_fifo.max_table_files_size` even if the files older than ttl are deleted.

This feature is not supported if max_open_files != -1 or with table formats other than Block-based.

**Test Plan:**
Added tests.

**Benchmark results:**
Base: FIFO with max size: 100MB ::
```
svemuri@dev15905 ~/rocksdb (fifo-compaction) $ TEST_TMPDIR=/dev/shm ./db_bench --benchmarks=readwhilewriting --num=5000000 --threads=16 --compaction_style=2 --fifo_compaction_max_table_files_size_mb=100

readwhilewriting :       1.924 micros/op 519858 ops/sec;   13.6 MB/s (1176277 of 5000000 found)
```

With TTL (a low one for testing) ::
```
svemuri@dev15905 ~/rocksdb (fifo-compaction) $ TEST_TMPDIR=/dev/shm ./db_bench --benchmarks=readwhilewriting --num=5000000 --threads=16 --compaction_style=2 --fifo_compaction_max_table_files_size_mb=100 --fifo_compaction_ttl=20

readwhilewriting :       1.902 micros/op 525817 ops/sec;   13.7 MB/s (1185057 of 5000000 found)
```
Example Log lines:
```
2017/06/26-15:17:24.609249 7fd5a45ff700 (Original Log Time 2017/06/26-15:17:24.609177) [db/compaction_picker.cc:1471] [default] FIFO compaction: picking file 40 with creation time 1498515423 for deletion
2017/06/26-15:17:24.609255 7fd5a45ff700 (Original Log Time 2017/06/26-15:17:24.609234) [db/db_impl_compaction_flush.cc:1541] [default] Deleted 1 files
...
2017/06/26-15:17:25.553185 7fd5a61a5800 [DEBUG] [db/db_impl_files.cc:309] [JOB 0] Delete /dev/shm/dbbench/000040.sst type=2 #40 -- OK
2017/06/26-15:17:25.553205 7fd5a61a5800 EVENT_LOG_v1 {"time_micros": 1498515445553199, "job": 0, "event": "table_file_deletion", "file_number": 40}
```

SST Files remaining in the dbbench dir, after db_bench execution completed:
```
svemuri@dev15905 ~/rocksdb (fifo-compaction)  $ ls -l /dev/shm//dbbench/*.sst
-rw-r--r--. 1 svemuri users 30749887 Jun 26 15:17 /dev/shm//dbbench/000042.sst
-rw-r--r--. 1 svemuri users 30768779 Jun 26 15:17 /dev/shm//dbbench/000044.sst
-rw-r--r--. 1 svemuri users 30757481 Jun 26 15:17 /dev/shm//dbbench/000046.sst
```
Closes https://github.com/facebook/rocksdb/pull/2480

Differential Revision: D5305116

Pulled By: sagar0

fbshipit-source-id: 3e5cfcf5dd07ed2211b5b37492eb235b45139174
      1cd45cd1
    • Y
      Fix TARGETS file tests list · 982cec22
      由 Yi Wu 提交于 
      Summary:
1. The buckifier script assume each test "foo" comes with a .cc file of the same name (i.e. foo.cc). Update cassandra tests to follow this pattern so that the buckifier script can recognize them.
2. add blob_db_test
Closes https://github.com/facebook/rocksdb/pull/2506

Differential Revision: D5331517

Pulled By: yiwu-arbug

fbshipit-source-id: 86f3eba471fc621186ab44cbd073b6162cde8e57
      982cec22
    • Y
      allow numa >= 2.0.8 · b49b3710
      由 Yi Wu 提交于 
      Summary:
Allow numa >= 2.0.8 in buck TARGET file.
Closes https://github.com/facebook/rocksdb/pull/2504

Differential Revision: D5330550

Pulled By: yiwu-arbug

fbshipit-source-id: 8ffb6167b4ad913877eac16a20a91023b31f8d41
      b49b3710
    • S
      CLANG Tidy · e517bfa2
      由 Siying Dong 提交于 
      Summary: Closes https://github.com/facebook/rocksdb/pull/2502

Differential Revision: D5326498

Pulled By: siying

fbshipit-source-id: 2f0ac6dc6ca5ddb23cecf67a278c086e52646714
      e517bfa2
    • Y
      update compatible test · dc3d2e4d
      由 Yi Wu 提交于 
      Summary:
update compatible test to include 5.5 and 5.6 branch.
Closes https://github.com/facebook/rocksdb/pull/2501

Differential Revision: D5325220

Pulled By: yiwu-arbug

fbshipit-source-id: 5f5271491e6dd2d7b2cf73a7142f38a571553bc4
      dc3d2e4d
  21. 27 6月, 2017 5 次提交
    • S
      Fix Windows build broken by 5c97a7c0 · 89468c01
      由 Siying Dong 提交于 
      Summary:
A typo conversion fails Windows build. Fix it.
Closes https://github.com/facebook/rocksdb/pull/2500

Differential Revision: D5325962

Pulled By: siying

fbshipit-source-id: 2cefdafc9afbc85f856f403af7c876b622400630
      89468c01
    • E
      Encryption at rest support · 51778612
      由 Ewout Prangsma 提交于 
      Summary:
This PR adds support for encrypting data stored by RocksDB when written to disk.

It adds an `EncryptedEnv` override of the `Env` class with matching overrides for sequential&random access files.
The encryption itself is done through a configurable `EncryptionProvider`. This class creates is asked to create `BlockAccessCipherStream` for a file. This is where the actual encryption/decryption is being done.
Currently there is a Counter mode implementation of `BlockAccessCipherStream` with a `ROT13` block cipher (NOTE the `ROT13` is for demo purposes only!!).

The Counter operation mode uses an initial counter & random initialization vector (IV).
Both are created randomly for each file and stored in a 4K (default size) block that is prefixed to that file. The `EncryptedEnv` implementation is such that clients of the `Env` class do not see this prefix (nor data, nor in filesize).
The largest part of the prefix block is also encrypted, and there is room left for implementation specific settings/values/keys in there.

To test the encryption, the `DBTestBase` class has been extended to consider a new environment variable called `ENCRYPTED_ENV`. If set, the test will setup a encrypted instance of the `Env` class to use for all tests.
Typically you would run it like this:

```
ENCRYPTED_ENV=1 make check_some
```

There is also an added test that checks that some data inserted into the database is or is not "visible" on disk. With `ENCRYPTED_ENV` active it must not find plain text strings, with `ENCRYPTED_ENV` unset, it must find the plain text strings.
Closes https://github.com/facebook/rocksdb/pull/2424

Differential Revision: D5322178

Pulled By: sdwilsh

fbshipit-source-id: 253b0a9c2c498cc98f580df7f2623cbf7678a27f
      51778612
    • S
      Trivial typo in HISTORY.md · 7061912c
      由 Siying Dong 提交于 
      Summary: Closes https://github.com/facebook/rocksdb/pull/2499

Differential Revision: D5324914

Pulled By: siying

fbshipit-source-id: c69827d4dddafa81e651180633943a3380cdd5bb
      7061912c
    • Y
      Fix jni WriteBatchThreadedTest · 2a9cd879
      由 Yi Wu 提交于 
      Summary:
WriteBatchThreadedTest is failing, at least on Mac. The problem seems to be `wb` is getting GC before we finish write. Explicitly close it seems to fix it.
Closes https://github.com/facebook/rocksdb/pull/2482

Differential Revision: D5307379

Pulled By: yiwu-arbug

fbshipit-source-id: 8ff7f8170451078c941951f5aafae83afffb7933
      2a9cd879
    • A
      revert perf_context and io_stats to __thread · 0025a364
      由 Aaron Gao 提交于 
      Summary:
https://github.com/facebook/rocksdb/pull/2380 introduces a regression by replacing __thread with ThreadLocalPtr. Revert the thread local implementation back.
Closes https://github.com/facebook/rocksdb/pull/2485

Differential Revision: D5308050

Pulled By: lightmark

fbshipit-source-id: 2676e9c22edf76e8133d3f4c50e2711e11a95480
      0025a364