Skip to content

R
rocksdb

kvdb / rocksdb
11 个月 前同步成功

通知 0
Star 0
Fork 0
R
rocksdb

体验新版 GitCode，发现更多精彩内容 >>

切换分支/标签
  1. 09 4月, 2020 3 次提交
  3. 08 4月, 2020 6 次提交
  5. 07 4月, 2020 2 次提交
  7. 05 4月, 2020 1 次提交
    • P
      Add some timestamps in CI build+test output (#6643) · a67fb4c9
      由 Peter Dillinger 提交于 
      Summary:
When Travis times out, it's hard to determine whether
the last executing thing took an excessively long time or the
sum of all the work just exceeded the time limit. This
change inserts some timestamps in the output that should
make this easier to determine.
Pull Request resolved: https://github.com/facebook/rocksdb/pull/6643

Test Plan: CI (Travis mostly)

Reviewed By: anand1976

Differential Revision: D20843901

Pulled By: pdillinger

fbshipit-source-id: e7aae5434b0c609931feddf238ce4355964488b7
      a67fb4c9
  9. 04 4月, 2020 5 次提交
    • S
      Fix clang anaylze warning caused by #6262 (#6641) · 00f8016b
      由 sdong 提交于 
      Summary:
https://github.com/facebook/rocksdb/pull/6262 causes CLANG analyze to complain. Add assertion to suppress the warning.
Pull Request resolved: https://github.com/facebook/rocksdb/pull/6641

Test Plan: Run "clang analyze" and make sure it passes.

Reviewed By: anand1976

Differential Revision: D20841722

fbshipit-source-id: 5fa6e0c5cfe7a822214c9b898a408df59d4fd2cd
      00f8016b
    • A
      fix compiler errors with -DNPERF_CONTEXT (#6642) · e60ea7fe
      由 Andrew Kryczka 提交于 
      Summary:
as titled
Pull Request resolved: https://github.com/facebook/rocksdb/pull/6642

Test Plan:
```
$ EXTRA_CXXFLAGS="-DNPERF_CONTEXT" DEBUG_LEVEL=0 make -j48 db_bench
```

Reviewed By: riversand963

Differential Revision: D20842313

Pulled By: ajkr

fbshipit-source-id: a830cad312ca681591f06749242279503b101df2
      e60ea7fe
    • M
      Move the OptionTypeMap code closer to home (#6198) · 259b6ec8
      由 mrambacher 提交于 
      Summary:
This is a predecessor to the Configurable PR.  This change moves the OptionTypeInfo maps closer to where they will be used.

When the Configurable changes are adopted, these values will become static and not associated with the OptionsHelper.
Pull Request resolved: https://github.com/facebook/rocksdb/pull/6198

Reviewed By: siying

Differential Revision: D20778108

Pulled By: zhichao-cao

fbshipit-source-id: a9f85fc73bc53503656e1958ecc1e764052fd1aa
      259b6ec8
    • P
      Revamp cache_bench to resemble a real workload (#6629) · 079e77ff
      由 Peter Dillinger 提交于 
      Summary:
I suspect LRUCache could use some optimization, and to support
such an effort, a good benchmarking tool is needed. The existing
cache_bench was heavily skewed toward insertion and lookup misses, and
did not saturate memory with other work. This change should improve
those things to better resemble a real workload.

(All below using clang compiler, for some consistency, but not
necessarily same version and settings.)

The real workload is from production MySQL on RocksDB, filtering stacks
containing "LRU", "ShardedCache" or "CacheShard."
Lookup inclusive: 66%
Insert inclusive: 17%
Release inclusive: 15%

An alternate simulated workload is MySQL running a LinkBench read test:
Lookup inclusive: 54%
Insert inclusive: 24%
Release inclusive: 21%

cache_bench default settings, prior to this change:
Lookup inclusive: 35.8%
Insert inclusive: 63.6%
Release inclusive: 0%

cache_bench after this change (intended as somewhat "tighter" workload
than average production, more like LinkBench):
Lookup inclusive: 52%
Insert inclusive: 20%
Release inclusive: 26%

And top exclusive stacks (portion of stack samples as filtered above):
Production MySQL:
LRUHandleTable::FindPointer: 25.3%
rocksdb::operator==: 15.1%  <-- Slice ==
LRUCacheShard::LRU_Remove: 13.8%
ShardedCache::Lookup: 8.9%
__pthread_mutex_lock: 7.1%
LRUCacheShard::LRU_Insert: 6.3%
MurmurHash64A: 4.8%  <-- Since upgraded to XXH3p
...

Old cache_bench:
LRUHandleTable::FindPointer: 23.6%
__pthread_mutex_lock: 15.0%
__pthread_mutex_unlock_usercnt: 11.7%
__lll_lock_wait: 8.6%
__lll_unlock_wake: 6.8%
LRUCacheShard::LRU_Insert: 6.0%
ShardedCache::Lookup: 4.4%
LRUCacheShard::LRU_Remove: 2.8%
...
rocksdb::operator==: 0.2%  <-- Slice ==
...

New cache_bench:
LRUHandleTable::FindPointer: 22.8%
__pthread_mutex_unlock_usercnt: 14.3%
rocksdb::operator==: 10.5%  <-- Slice ==
LRUCacheShard::LRU_Insert: 9.0%
__pthread_mutex_lock: 5.9%
LRUCacheShard::LRU_Remove: 5.0%
...
ShardedCache::Lookup: 2.9%
...

So there's a bit more lock contention in the benchmark than in
production, but otherwise looks similar enough to me. At least it's a
big improvement over the existing code.
Pull Request resolved: https://github.com/facebook/rocksdb/pull/6629

Test Plan: No production code changes, ran cache_bench with ASAN

Reviewed By: ltamasi

Differential Revision: D20824318

Pulled By: pdillinger

fbshipit-source-id: 6f8dc5891ead0f87edbed3a615ecd5289d9abe12
      079e77ff
    • B
      Fix msvc debug test failures (#6579) · df62cd5b
      由 Burton Li 提交于 
      Summary:
1. stats_history_test: one slice of stats history is 12526 Bytes, which is greater than original assumption.
![image](https://user-images.githubusercontent.com/17753898/77381970-5a611a80-6d3c-11ea-9d64-59d2e3c04f79.png)
2. table_test: in VerifyBlockAccessTrace function, release trace reader before delete trace file.
Pull Request resolved: https://github.com/facebook/rocksdb/pull/6579

Reviewed By: siying

Differential Revision: D20767373

Pulled By: pdillinger

fbshipit-source-id: e8647d665cbe83a3f5429639c6219b50c0912124
      df62cd5b
  11. 03 4月, 2020 6 次提交
  13. 02 4月, 2020 5 次提交
    • Y
      Add counter in perf_context to time cipher time (#6596) · 2b02ea25
      由 Yi Wu 提交于 
      Summary:
Add `encrypt_data_time` and `decrypt_data_time` perf_context counters to time encryption/decryption time when `EnvEncryption` is enabled.
Pull Request resolved: https://github.com/facebook/rocksdb/pull/6596

Test Plan: CI

Reviewed By: anand1976

Differential Revision: D20678617

fbshipit-source-id: 7b57536143aa38509cde011f704de33382169e07
      2b02ea25
    • Z
      Add pipelined & parallel compression optimization (#6262) · 03a781a9
      由 Ziyue Yang 提交于 
      Summary:
This PR adds support for pipelined & parallel compression optimization for `BlockBasedTableBuilder`. This optimization makes block building, block compression and block appending a pipeline, and uses multiple threads to accelerate block compression. Users can set `CompressionOptions::parallel_threads` greater than 1 to enable compression parallelism.
Pull Request resolved: https://github.com/facebook/rocksdb/pull/6262

Reviewed By: ajkr

Differential Revision: D20651306

fbshipit-source-id: 62125590a9c15b6d9071def9dc72589c1696a4cb
      03a781a9
    • S
      Add dependency of gtest on pthread (#6572) · 719c0f91
      由 Sylvain Oliver 提交于 
      Summary:
Compilation of rocksdb fails because -lpthread flag is needed by gtest

**Before modification** :
/usr/bin/c++   -W -Wextra -Wall -Wsign-compare -Wshadow -Wno-unused-parameter -Wno-unused-variable -Woverloaded-virtual -Wnon-virtual-dtor -Wno-missing-field-initializers -Wno-strict-aliasing -std=c++11 -march=native -Werror -fno-builtin-memcmp -g -DROCKSDB_USE_RTTI   CMakeFiles/table_reader_bench.dir/table/table_reader_bench.cc.o  -o table_reader_bench -Wl,-rpath,/develop/src/rocksdb/build librocksdb.so.6.8.0 libtestharness.a /usr/lib/x86_64-linux-gnu/libgflags.so -lpthread third-party/gtest-1.8.1/fused-src/gtest/libgtest.a

**After modification** :
/usr/bin/c++   -W -Wextra -Wall -Wsign-compare -Wshadow -Wno-unused-parameter -Wno-unused-variable -Woverloaded-virtual -Wnon-virtual-dtor -Wno-missing-field-initializers -Wno-strict-aliasing -std=c++11 -march=native -Werror -fno-builtin-memcmp -g -DROCKSDB_USE_RTTI   CMakeFiles/table_reader_bench.dir/table/table_reader_bench.cc.o  -o table_reader_bench -Wl,-rpath,/develop/src/rocksdb/build librocksdb.so.6.8.0 libtestharness.a /usr/lib/x86_64-linux-gnu/libgflags.so third-party/gtest-1.8.1/fused-src/gtest/libgtest.a -lpthread
Pull Request resolved: https://github.com/facebook/rocksdb/pull/6572

Reviewed By: anand1976

Differential Revision: D20789059

Pulled By: ajkr

fbshipit-source-id: 97329f14b9044b12c8a415da3d5f27b256ff8ff7
      719c0f91
    • S
      Fix a bug that crashes the service when write buffer manager fails to insert to block cache (#6619) · 57096ab1
      由 sdong 提交于 
      Summary:
https://github.com/facebook/rocksdb/issues/6247 reports that when write buffer manager fails to insert the dummy entry to block cache, null pointer is still stored and used to release the handle and cause corruption. Fix the bug by not releasing it with null handle.
Pull Request resolved: https://github.com/facebook/rocksdb/pull/6619

Test Plan: Add a unit test that fails without the fix.

Reviewed By: ajkr

Differential Revision: D20776769

fbshipit-source-id: 4127fbd9f295a0a3e45774746ffcd91f939f6287
      57096ab1
    • Y
      Fix clang analyze error (#6622) · b5818f87
      由 Yanqin Jin 提交于 
      Summary:
As title. https://github.com/facebook/rocksdb/issues/6612 caused clang analyze to fail with the error:
```
db/compaction/compaction_picker_fifo.cc:105:39: warning: Called C++ object pointer is null
                     cf_name.c_str(), f->fd.GetNumber(), creation_time);
                                      ^~~~~~~~~~~~~~~~~
./logging/logging.h:59:36: note: expanded from macro 'ROCKS_LOG_BUFFER'
                                 ##__VA_ARGS__)
                                   ^~~~~~~~~~~
1 warning generated.
```

Test Plan (devserver):
USE_CLANG=1 make analyze
Pull Request resolved: https://github.com/facebook/rocksdb/pull/6622

Reviewed By: ltamasi

Differential Revision: D20787407

Pulled By: riversand963

fbshipit-source-id: a5de4910cc1aa0d3481a73ec114578925bfd63f7
      b5818f87
  15. 01 4月, 2020 3 次提交
  17. 31 3月, 2020 3 次提交
  19. 30 3月, 2020 2 次提交
  21. 29 3月, 2020 1 次提交
    • C
      Be able to decrease background thread's CPU priority when creating database backup (#6602) · ee50b8d4
      由 Cheng Chang 提交于 
      Summary:
When creating a database backup, the background threads will not only consume IO resources by copying files, but also consuming CPU such as by computing checksums. During peak times, the CPU consumption by the background threads might affect online queries.

This PR makes it possible to decrease CPU priority of these threads when creating a new backup.
Pull Request resolved: https://github.com/facebook/rocksdb/pull/6602

Test Plan: make check

Reviewed By: siying, zhichao-cao

Differential Revision: D20683216

Pulled By: cheng-chang

fbshipit-source-id: 9978b9ed9488e8ce135e90ca083e5b4b7221fd84
      ee50b8d4
  23. 28 3月, 2020 3 次提交
    • L
      Call out the cache deleter related interface change in HISTORY.md (#6606) · 3a35542f
      由 Levi Tamasi 提交于 
      Summary: Pull Request resolved: https://github.com/facebook/rocksdb/pull/6606

Reviewed By: riversand963

Differential Revision: D20708411

Pulled By: ltamasi

fbshipit-source-id: c15b4ded19a4b5c84e3e4240bdcec15460806c88
      3a35542f
    • C
      Refactor IsLockExpired (#6586) · 3881a678
      由 Cheng Chang 提交于 
      Summary:
1. If expiration_time is non-positive, no need to call NowMicros, save a syscall.
2. expire_time should only be set when expired is false.
Pull Request resolved: https://github.com/facebook/rocksdb/pull/6586

Test Plan: make check

Reviewed By: lth

Differential Revision: D20673730

Pulled By: cheng-chang

fbshipit-source-id: a69e8d7b16dc6d0d00487bb1c19f0710d79482e2
      3881a678
    • Z
      Pass IOStatus to write path and set retryable IO Error as hard error in BG jobs (#6487) · 42468881
      由 Zhichao Cao 提交于 
      Summary:
In the current code base, we use Status to get and store the returned status from the call. Specifically, for IO related functions, the current Status cannot reflect the IO Error details such as error scope, error retryable attribute, and others. With the implementation of https://github.com/facebook/rocksdb/issues/5761, we have the new Wrapper for IO, which returns IOStatus instead of Status. However, the IOStatus is purged at the lower level of write path and transferred to Status.

The first job of this PR is to pass the IOStatus to the write path (flush, WAL write, and Compaction). The second job is to identify the Retryable IO Error as HardError, and set the bg_error_ as HardError. In this case, the DB Instance becomes read only. User is informed of the Status and need to take actions to deal with it (e.g., call db->Resume()).
Pull Request resolved: https://github.com/facebook/rocksdb/pull/6487

Test Plan: Added the testing case to error_handler_fs_test. Pass make asan_check

Reviewed By: anand1976

Differential Revision: D20685017

Pulled By: zhichao-cao

fbshipit-source-id: ff85f042896243abcd6ef37877834e26f36b6eb0
      42468881