Skip to content

R
rocksdb

kvdb / rocksdb
11 个月 前同步成功

通知 0
Star 0
Fork 0
R
rocksdb

体验新版 GitCode，发现更多精彩内容 >>

切换分支/标签
  1. 02 10月, 2020 4 次提交
  3. 01 10月, 2020 1 次提交
  5. 30 9月, 2020 3 次提交
  7. 29 9月, 2020 3 次提交
  9. 25 9月, 2020 1 次提交
  11. 24 9月, 2020 3 次提交
    • A
      Add IO Tracer Parser (#7333) · 98ac6b64
      由 Akanksha Mahajan 提交于 
      Summary:
Implement a parsing tool io_tracer_parser that takes IO trace file (binary file) with command line argument --io_trace_file and output file with --output_file and dumps the IO trace records in outputfile in human readable form.

Also added unit test cases that generates IO trace records and calls io_tracer_parse to parse those records.

Pull Request resolved: https://github.com/facebook/rocksdb/pull/7333

Test Plan:
make check -j64,
 Add unit test cases.

Reviewed By: anand1976

Differential Revision: D23772360

Pulled By: akankshamahajan15

fbshipit-source-id: 9c20519c189362e6663352d08863326f3e496271
      98ac6b64
    • P
      Add and fix clang -Wshift-sign-overflow (#7431) · 31d1cea4
      由 Peter Dillinger 提交于 
      Summary:
This option is apparently used by some teams within Facebook
(internal ref T75998621)

Pull Request resolved: https://github.com/facebook/rocksdb/pull/7431

Test Plan: USE_CLANG=1 make check before (fails) and after

Reviewed By: jay-zhuang

Differential Revision: D23876584

Pulled By: pdillinger

fbshipit-source-id: abb8b67a1f1aac75327944d266e284b2b6727191
      31d1cea4
    • J
      CircleCI build improvements (#7392) · 6980fff4
      由 Jay Zhuang 提交于 
      Summary:
* add slack integration
* parallelize `check_some` tests with gtest-parallel
* upgrade build image to the latest one: ubuntu-1604:202007-01
* clean up the config

Pull Request resolved: https://github.com/facebook/rocksdb/pull/7392

Reviewed By: pdillinger

Differential Revision: D23845379

Pulled By: jay-zhuang

fbshipit-source-id: f3af82dc2daeba6441f61c858e08d3936c6ccd10
      6980fff4
  13. 17 9月, 2020 1 次提交
  15. 15 9月, 2020 2 次提交
    • A
      Makefile support subset/individual valgrind tests (#7379) · d4993b9b
      由 Andrew Kryczka 提交于 
      Summary:
Introduced `valgrind_check_some`, which is analogous to the `check_some` target for non-valgrind tests. It simplifies the process for running a single valgrind test or subset of valgrind tests when trying to repro a failure.

I also added a `ROCKSDBTESTS_ONLY` parameter, which simplifies selecting a single test to run. Previously the user would have to use `ROCKSDBTESTS_START` and `ROCKSDBTESTS_END`, but it was difficult to determine the end variable since it is an exclusive endpoint and must match an actual test name.

Pull Request resolved: https://github.com/facebook/rocksdb/pull/7379

Reviewed By: pdillinger

Differential Revision: D23673608

Pulled By: ajkr

fbshipit-source-id: 87ed81f1a671d46c2dff6a701f85f1891c725b3f
      d4993b9b
    • M
      Bring the Configurable options together (#5753) · 7d472acc
      由 mrambacher 提交于 
      Summary:
This PR merges the functionality of making the ColumnFamilyOptions, TableFactory, and DBOptions into Configurable into a single PR, resolving any merge conflicts

Pull Request resolved: https://github.com/facebook/rocksdb/pull/5753

Reviewed By: ajkr

Differential Revision: D23385030

Pulled By: zhichao-cao

fbshipit-source-id: 8b977a7731556230b9b8c5a081b98e49ee4f160a
      7d472acc
  17. 10 9月, 2020 1 次提交
  19. 09 9月, 2020 1 次提交
    • P
      Fix some errors showing up in Travis builds (#7359) · 9de912de
      由 Peter Dillinger 提交于 
      Summary:
Also enables a pull request to trigger all the Travis
configurations by writing FULL_CI in the commit message. (See what I did
there?)

First issue

    make: *** No rule to make target 'jl/util/crc32c_ppc_asm.o', needed by 'rocksdbjava'.  Stop.

Second issue

    tools/db_bench_tool.cc:5514:38: error: ‘gen_exp.rocksdb::Benchmark::GenerateTwoTermExpKeys::keyrange_size_’ may be used uninitialized in this function

Pull Request resolved: https://github.com/facebook/rocksdb/pull/7359

Test Plan: CI

Reviewed By: zhichao-cao

Differential Revision: D23582132

Pulled By: pdillinger

fbshipit-source-id: 06d794673fd522ba11cf6398385387e6bd97ef89
      9de912de
  21. 05 9月, 2020 1 次提交
  23. 04 9月, 2020 1 次提交
    • P
      New bit manipulation functions and 128-bit value library (#7338) · c4d8838a
      由 Peter Dillinger 提交于 
      Summary:
These new functions and 128-bit value bit operations are
expected to be used in a forthcoming Bloom filter alternative.

No functional changes to production code, just new code only called by
unit tests, cosmetic changes to existing headers, and fix an existing
function for a yet-unused template instantiation (BitsSetToOne on
something signed and smaller than 32 bits).

Pull Request resolved: https://github.com/facebook/rocksdb/pull/7338

Test Plan:
Unit tests included. Works with and without
TEST_UINT128_COMPAT=1 to check compatibility with and without
__uint128_t. Also added that parameter to the CircleCI build
build-linux-shared_lib-alt_namespace-status_checked.

Reviewed By: jay-zhuang

Differential Revision: D23494945

Pulled By: pdillinger

fbshipit-source-id: 5c0dc419100d9df5d4d9abb153b2855d5aea39e8
      c4d8838a
  25. 29 8月, 2020 1 次提交
  27. 28 8月, 2020 2 次提交
    • J
      Add buffer prefetch support for non directIO usecase (#7312) · c2485f2d
      由 Jay Zhuang 提交于 
      Summary:
A new file interface `SupportPrefetch()` is added. When the user overrides it to `false`, an internal prefetch buffer will be used for readahead. Useful for non-directIO but FS doesn't have readahead support.

Pull Request resolved: https://github.com/facebook/rocksdb/pull/7312

Reviewed By: anand1976

Differential Revision: D23329847

Pulled By: jay-zhuang

fbshipit-source-id: 71cd4ce6f4a820840294e4e6aec111ab76175527
      c2485f2d
    • L
      Add a blob file builder class that can be used in background jobs (#7306) · 50439606
      由 Levi Tamasi 提交于 
      Summary:
The patch adds a class called `BlobFileBuilder` that can be used to build
and cut blob files in background jobs (flushes/compactions). The class
enforces a value size threshold (`min_blob_size`; smaller blobs will be inlined
in the LSM tree itself), and supports specifying a blob file size limit (`blob_file_size`),
as well as compression (`blob_compression_type`) and checksums for blob files.
It also keeps track of the generated blob files and their associated `BlobFileAddition`
metadata, which can be applied as part of the background job's `VersionEdit`.

Pull Request resolved: https://github.com/facebook/rocksdb/pull/7306

Test Plan: `make check`

Reviewed By: riversand963

Differential Revision: D23298817

Pulled By: ltamasi

fbshipit-source-id: 38f35d81dab1ba81f15236240612ec173d7f21b5
      50439606
  29. 25 8月, 2020 1 次提交
  31. 18 8月, 2020 1 次提交
    • J
      Generate and install a pkg-config file (#7244) · 59ebab65
      由 John Goerzen 提交于 
      Summary:
pkg-config files are quite useful for communicating to users of a
library how to compile against them. This commit generates and installs
a pkg-config file that can be used for both static and dynamic builds
against the RocksDB library. This should make life easier for developers
of client programs, language bindings, etc.

Example usage:

```
g++ `pkg-config --cflags rocksdb` -o simple_example simple_example.cc `pkg-config --libs rocksdb`

g++ `pkg-config --cflags --static rocksdb` -static \
   -o simple_example simple_example.cc `pkg-config --libs --static rocksdb`
```

The commit also adds the generated file to .gitignore, to the uninstall
target, and to clean.

No additional dependencies are added to RocksDB itself, and this does
not make RocksDB use pkg-config as part of its build process.

Resolves https://github.com/facebook/rocksdb/issues/4452

Pull Request resolved: https://github.com/facebook/rocksdb/pull/7244

Reviewed By: siying

Differential Revision: D23146153

Pulled By: ajkr

fbshipit-source-id: 3045aa650d68bd5ac42d40ed709570e9584ef004
      59ebab65
  33. 15 8月, 2020 1 次提交
    • J
      Introduce a global StatsDumpScheduler for stats dumping (#7223) · 69760b4d
      由 Jay Zhuang 提交于 
      Summary:
Have a global StatsDumpScheduler for all DB instance stats dumping, including `DumpStats()` and `PersistStats()`. Before this, there're 2 dedicate threads for every DB instance, one for DumpStats() one for PersistStats(), which could create lots of threads if there're hundreds DB instances.

Pull Request resolved: https://github.com/facebook/rocksdb/pull/7223

Reviewed By: riversand963

Differential Revision: D23056737

Pulled By: jay-zhuang

fbshipit-source-id: 0faa2311142a73433ebb3317361db7cbf43faeba
      69760b4d
  35. 12 8月, 2020 1 次提交
  37. 11 8月, 2020 1 次提交
    • A
      Allow optimization level to be set in Makefile (#7202) · 6e99de6d
      由 Andrew Kryczka 提交于 
      Summary:
`-O3` is already adopted widely, so we should make it easier to configure
for development/open source. This PR adds an `OPTIMIZE_LEVEL` variable
that users can set to override the `-O` flag chosen in the Makefile.

Pull Request resolved: https://github.com/facebook/rocksdb/pull/7202

Test Plan: built a few different ways and verified correct value is passed for `-O` flag

Reviewed By: pdillinger

Differential Revision: D22845291

Pulled By: ajkr

fbshipit-source-id: 84471362e7d627dd606b25bf5f6a3d796817fa1c
      6e99de6d
  39. 07 8月, 2020 2 次提交
  41. 06 8月, 2020 1 次提交
    • C
      Define WAL related classes to be used in VersionEdit and VersionSet (#7164) · cd48ecaa
      由 Cheng Chang 提交于 
      Summary:
`WalAddition`, `WalDeletion` are defined in `wal_version.h` and used in `VersionEdit`.
`WalAddition` is used to represent events of creating a new WAL (no size, just log number), or closing a WAL (with size).
`WalDeletion` is used to represent events of deleting or archiving a WAL, it means the WAL is no longer alive (won't be replayed during recovery).

`WalSet` is the set of alive WALs kept in `VersionSet`.

1. Why use `WalDeletion` instead of relying on `MinLogNumber` to identify outdated WALs

On recovery, we can compute `MinLogNumber()` based on the log numbers kept in MANIFEST, any log with number < MinLogNumber can be ignored. So it seems that we don't need to persist `WalDeletion` to MANIFEST, since we can ignore the WALs based on MinLogNumber.

But the `MinLogNumber()` is actually a lower bound, it does not exactly mean that logs starting from MinLogNumber must exist. This is because in a corner case, when a column family is empty and never flushed, its log number is set to the largest log number, but not persisted in MANIFEST. So let's say there are 2 column families, when creating the DB, the first WAL has log number 1, so it's persisted to MANIFEST for both column families. Then CF 0 is empty and never flushed, CF 1 is updated and flushed, so a new WAL with log number 2 is created and persisted to MANIFEST for CF 1. But CF 0's log number in MANIFEST is still 1. So on recovery, MinLogNumber is 1, but since log 1 only contains data for CF 1, and CF 1 is flushed, log 1 might have already been deleted from disk.

We can make `MinLogNumber()` be the exactly minimum log number that must exist, by persisting the most recent log number for empty column families that are not flushed. But if there are N such column families, then every time a new WAL is created, we need to add N records to MANIFEST.

In current design, a record is persisted to MANIFEST only when WAL is created, closed, or deleted/archived, so the number of WAL related records are bounded to 3x number of WALs.

2. Why keep `WalSet` in `VersionSet` instead of applying the `VersionEdit`s to `VersionStorageInfo`

`VersionEdit`s are originally designed to track the addition and deletion of SST files. The SST files are related to column families, each column family has a list of `Version`s, and each `Version` keeps the set of active SST files in `VersionStorageInfo`.

But WALs are a concept of DB, they are not bounded to specific column families. So logically it does not make sense to store WALs in a column family's `Version`s.
Also, `Version`'s purpose is to keep reference to SST / blob files, so that they are not deleted until there is no version referencing them. But a WAL is deleted regardless of version references.
So we keep the WALs in `VersionSet`  for the purpose of writing out the DB state's snapshot when creating new MANIFESTs.

Pull Request resolved: https://github.com/facebook/rocksdb/pull/7164

Test Plan:
make version_edit_test && ./version_edit_test
make wal_edit_test && ./wal_edit_test

Reviewed By: ltamasi

Differential Revision: D22677936

Pulled By: cheng-chang

fbshipit-source-id: 5a3b6890140e572ffd79eb37e6e4c3c32361a859
      cd48ecaa
  43. 29 7月, 2020 2 次提交
    • M
      Make env*_test work with ASSERT_STATUS_CHECKED (#7176) · d9d19074
      由 mrambacher 提交于 
      Summary:
Make (most of) the env*_test pass when ASSERT_STATUS_CHECKED is enabled.

One test that opens a database is currently disabled in this mode, as there are many errors that need revisited for DB tests and status checks.

Pull Request resolved: https://github.com/facebook/rocksdb/pull/7176

Reviewed By: cheng-chang

Differential Revision: D22799278

Pulled By: ajkr

fbshipit-source-id: 16d8a02eaeecd6df1060249b6a5811292801f2ed
      d9d19074
    • A
      Makefile support for link-time optimization (#7181) · c0c33a48
      由 Andrew Kryczka 提交于 
      Summary:
`USE_LTO=1` in `make` commands now enables LTO. The archiver (`ar`) needed
to change in this PR to use a wrapper that enables the LTO plugin.

Pull Request resolved: https://github.com/facebook/rocksdb/pull/7181

Test Plan:
build a few ways
```
$ make clean && USE_LTO=1 make -j48 db_bench
$ make clean && USE_CLANG=1 USE_LTO=1 make -j48 db_bench
$ make clean && ROCKSDB_NO_FBCODE=1 USE_LTO=1 make -j48 db_bench
```

Reviewed By: cheng-chang

Differential Revision: D22784994

Pulled By: ajkr

fbshipit-source-id: 9c45333bd49bf4615aa04c85b7c6fd3925421152
      c0c33a48
  45. 25 7月, 2020 1 次提交
  47. 21 7月, 2020 2 次提交
    • S
      Exclude two tests in CircleCI TSAN tests (#7152) · 0f487cc3
      由 sdong 提交于 
      Summary:
Two TSAN tests occaionaly fail. Exclude them for now:

[ RUN      ] DeleteFileTest.BackgroundPurgeCFDropTest
db/deletefile_test.cc:122: Failure
Expected equality of these values:
  required_manifest
    Which is: 1
  manifest_cnt
    Which is: 2

[ RUN      ] FormatLatest/ColumnFamilyTest.FlushCloseWALFiles/0
db/column_family_test.cc:3004: Failure
Expected equality of these values:
  2
  env.num_open_wal_file_.load()
    Which is: 1

Pull Request resolved: https://github.com/facebook/rocksdb/pull/7152

Test Plan: Watch CircleCI restuls

Reviewed By: ajkr

Differential Revision: D22632285

fbshipit-source-id: 29fa348e8be917be0237c74812a8b0b04978e84e
      0f487cc3
    • J
      Store the test results to CircleCI (#7137) · 77062cf1
      由 Jay Zhuang 提交于 
      Summary:
To have test report.

Pull Request resolved: https://github.com/facebook/rocksdb/pull/7137

Reviewed By: siying

Differential Revision: D22630798

Pulled By: jay-zhuang

fbshipit-source-id: bc07ba673c0bceed5a4829b4af2d9a74435379c7
      77062cf1
  49. 18 7月, 2020 1 次提交
  51. 15 7月, 2020 1 次提交