Skip to content

R
rocksdb

kvdb / rocksdb
11 个月 前同步成功

通知 0
Star 0
Fork 0
R
rocksdb

体验新版 GitCode，发现更多精彩内容 >>

切换分支/标签
  1. 30 3月, 2021 4 次提交
  3. 29 3月, 2021 4 次提交
    • M
      Fix spelling in comments in include/rocksdb/ (#8120) · 524b10bd
      由 mrambacher 提交于 
      Summary:
Ran a spell check over the comments in the include/rocksdb directory and fixed any mis-spellings.

There are still some variable names that are spelled incorrectly (like SizeApproximationOptions::include_memtabtles, SstFileMetaData::oldest_ancester_time) that were not fixed, as those would break compilation.

Pull Request resolved: https://github.com/facebook/rocksdb/pull/8120

Reviewed By: zhichao-cao

Differential Revision: D27366034

Pulled By: mrambacher

fbshipit-source-id: 6a3f3674890bb6acc751e9c5887a8fbb6adca5df
      524b10bd
    • Y
      Disable partitioned filters in ts stress test (#8127) · ae7a7956
      由 Yanqin Jin 提交于 
      Summary:
Currently, partitioned filter does not support user-defined timestamp. Disable it for now in ts stress test so that
the contrun jobs can proceed.

Pull Request resolved: https://github.com/facebook/rocksdb/pull/8127

Test Plan: make crash_test_with_ts

Reviewed By: ajkr

Differential Revision: D27388488

Pulled By: riversand963

fbshipit-source-id: 5ccff18121cb537bd82f2ac072cd25efb625c666
      ae7a7956
    • M
      Fix make tags to not rebuild all the object files (#8097) · 5841bbe3
      由 mrambacher 提交于 
      Summary:
Because build_version.cc is dependent on the library objects (to force a re-generation of it), the library objects would be built in order to satisfy this rule.  Because there is a build_version.d file, it would need generated and included.

Change the ALL_DEPS/FILES to not include build_version.cc (meaning no .d file for it, which is okay since it is generated).  Also changed the rule on whether or not to generate DEP files to skip tags.

Pull Request resolved: https://github.com/facebook/rocksdb/pull/8097

Reviewed By: ajkr

Differential Revision: D27299815

Pulled By: mrambacher

fbshipit-source-id: 1efbe8a56d062f57ae13b6c2944ad3faf775087e
      5841bbe3
    • A
      Always truncate the latest WAL file on DB Open (#8122) · 7d7f1448
      由 anand76 提交于 
      Summary:
Currently, we only truncate the latest alive WAL files when the DB is opened. If the latest WAL file is empty or was flushed during Open, its not truncated since the file will be deleted later on in the Open path. However, before deletion, a new WAL file is created, and if the process crash loops between the new WAL file creation and deletion of the old WAL file, the preallocated space will keep accumulating and eventually use up all disk space. To prevent this, always truncate the latest WAL file, even if its empty or the data was flushed.

Tests:
Add unit tests to db_wal_test

Pull Request resolved: https://github.com/facebook/rocksdb/pull/8122

Reviewed By: riversand963

Differential Revision: D27366132

Pulled By: anand1976

fbshipit-source-id: f923cc03ef033ccb32b140d36c6a63a8152f0e8e
      7d7f1448
  5. 27 3月, 2021 7 次提交
  7. 26 3月, 2021 4 次提交
    • L
      Introduce a ThreadGuard class and use it in ExternalSSTFileTest.PickedLevelBug (#8112) · 303cb23a
      由 Levi Tamasi 提交于 
      Summary:
The patch adds a resource management/RAII class called `ThreadGuard`,
which can be used to ensure that the managed thread is joined when the
`ThreadGuard` is destroyed, regardless of whether it is due to the
object going out of scope, an early return, an exception etc. This is
important because if an `std::thread` object is destroyed without having
been joined (or detached) first, the process is aborted (via
`std::terminate`).

For now, `ThreadGuard` is only used in the test case
`ExternalSSTFileTest.PickedLevelBug`; however, it could come in handy
elsewhere in the codebase as well (both in test code and "real" code).
Case in point: in the `PickedLevelBug` test case, with the earlier code we
could end up in the above situation when the following assertion (which is
before the threads are joined) is triggered:

```
ASSERT_FALSE(bg_compact_started.load());
```

Pull Request resolved: https://github.com/facebook/rocksdb/pull/8112

Test Plan:
```
make check
gtest-parallel --repeat=10000 ./external_sst_file_test --gtest_filter="*PickedLevelBug"
```

Reviewed By: riversand963

Differential Revision: D27343185

Pulled By: ltamasi

fbshipit-source-id: 2a8c3aa68bc78cc03ec0dbae909fb25c2cd15c69
      303cb23a
    • Z
      Fix flush no wal IO error bug (#8107) · af80a78b
      由 Zhichao Cao 提交于 
      Summary:
There is bug in the current code base introduced in https://github.com/facebook/rocksdb/issues/8049 , we still set the SST file write IO Error only case as hard error. Fix it by removing the logic.

Pull Request resolved: https://github.com/facebook/rocksdb/pull/8107

Test Plan: make check, error_handler_fs_test

Reviewed By: anand1976

Differential Revision: D27321422

Pulled By: zhichao-cao

fbshipit-source-id: c014afc1553ca66b655e3bbf9d0bf6eb417ccf94
      af80a78b
    • S
      Fix some typos in comments (#8066) · 711881bc
      由 storagezhang 提交于 
      Summary: Pull Request resolved: https://github.com/facebook/rocksdb/pull/8066

Reviewed By: jay-zhuang

Differential Revision: D27280799

Pulled By: mrambacher

fbshipit-source-id: 68f91f5af4ffe0a84be581961bf9366887f47702
      711881bc
    • A
      Apply `sample_for_compression` to all block-based tables (#8105) · c20a7cd6
      由 Andrew Kryczka 提交于 
      Summary:
Previously it only applied to block-based tables generated by flush. This restriction
was undocumented and blocked a new use case. Now compression sampling
applies to all block-based tables we generate when it is enabled.

Pull Request resolved: https://github.com/facebook/rocksdb/pull/8105

Test Plan: new unit test

Reviewed By: riversand963

Differential Revision: D27317275

Pulled By: ajkr

fbshipit-source-id: cd9fcc5178d6515e8cb59c6facb5ac01893cb5b0
      c20a7cd6
  9. 25 3月, 2021 3 次提交
  11. 24 3月, 2021 7 次提交
    • P
      Improve bloom_test bits_per_key flag (#8093) · da6b90ab
      由 Peter Dillinger 提交于 
      Summary:
Improved handling of -bits_per_key other than 10, but at least
the OptimizeForMemory test is simply not designed for generally handling
other settings. (ribbon_test does have a statistical framework for this
kind of testing, but it's not important to do that same for Bloom right
now.)

Closes https://github.com/facebook/rocksdb/issues/7019

Pull Request resolved: https://github.com/facebook/rocksdb/pull/8093

Test Plan: for I in `seq 1 20`; do ./bloom_test --gtest_filter=-*OptimizeForMemory* --bits_per_key=$I &> /dev/null || echo FAILED; done

Reviewed By: mrambacher

Differential Revision: D27275875

Pulled By: pdillinger

fbshipit-source-id: 7362e8ac2c41ea11f639412e4f30c8b375f04388
      da6b90ab
    • A
      Fix Race condition in db_sst_test (#8092) · 41e554da
      由 Akanksha Mahajan 提交于 
      Summary:
Fix race condition in
DBSSTTest.DBWithMaxSpaceAllowedWithBlobFiles where background flush
thread updates delete_blob_file but in test thread Flush() already
completes after getting bg_error and delete_blob_file remains false.

Pull Request resolved: https://github.com/facebook/rocksdb/pull/8092

Test Plan: Ran ASAN job few times on CircleCI

Reviewed By: riversand963

Differential Revision: D27275815

Pulled By: akankshamahajan15

fbshipit-source-id: 2939ad1671403881573bbe07c71aa474c5019130
      41e554da
    • Z
      Added append with checksum handoff API to hdfs (#8084) · 8dc6d8c7
      由 Zhichao Cao 提交于 
      Summary:
Added append with checksum handoff API to hdfs

Pull Request resolved: https://github.com/facebook/rocksdb/pull/8084

Test Plan: make check

Reviewed By: ajkr

Differential Revision: D27237823

Pulled By: zhichao-cao

fbshipit-source-id: 93b38db23b1811a6daa049afb89240089ec6f67c
      8dc6d8c7
    • Y
      Move compacted_db_impl.[c|h] to db/db_impl (#8082) · 9f7c02da
      由 Yanqin Jin 提交于 
      Summary:
As title. All core db implementations should stay in db_impl.

Pull Request resolved: https://github.com/facebook/rocksdb/pull/8082

Test Plan: make check

Reviewed By: ajkr

Differential Revision: D27211442

Pulled By: riversand963

fbshipit-source-id: e0953fde75064740e899aaff7989ff033b7f5232
      9f7c02da
    • Y
      Fix an error while running db_crashtest for non-user-ts tests (#8091) · e1aa8c16
      由 Yanqin Jin 提交于 
      Summary:
Fix the following error while running `make crash_test`
```
Traceback (most recent call last):
  File "tools/db_crashtest.py", line 705, in <module>
    main()
  File "tools/db_crashtest.py", line 696, in main
    blackbox_crash_main(args, unknown_args)
  File "tools/db_crashtest.py", line 479, in blackbox_crash_main
    + list({'db': dbname}.items())), unknown_args)
  File "tools/db_crashtest.py", line 414, in gen_cmd
    finalzied_params = finalize_and_sanitize(params)
  File "tools/db_crashtest.py", line 331, in finalize_and_sanitize
    dest_params.get("user_timestamp_size") > 0):
TypeError: '>' not supported between instances of 'NoneType' and 'int'
```

Pull Request resolved: https://github.com/facebook/rocksdb/pull/8091

Test Plan: make crash_test

Reviewed By: ltamasi

Differential Revision: D27268276

Pulled By: riversand963

fbshipit-source-id: ed2873b9587ecc51e24abc35ef2bd3d91fb1ed1b
      e1aa8c16
    • V
      Fix possible mistype in a comment (#8086) · 4a6bc47b
      由 Vlad Artamonov 提交于 
      Summary:
This is a small fix to what I think is a mistype in two comments in `DBOptionsInterface.java`. If it was not an error, feel free to close.

Pull Request resolved: https://github.com/facebook/rocksdb/pull/8086

Reviewed By: ajkr

Differential Revision: D27260488

Pulled By: mrambacher

fbshipit-source-id: 469daadaf6039d5b5187132b8e0c7c3672842f21
      4a6bc47b
    • Y
      Fix a compilation error in CircleCI vs2019 CXX20 (#8090) · 2a12b807
      由 Yanqin Jin 提交于 
      Summary:
As title.
Always specify namespace::symbol_name...
Test plan
CircleCI and other CI results

Pull Request resolved: https://github.com/facebook/rocksdb/pull/8090

Reviewed By: ltamasi

Differential Revision: D27256130

Pulled By: riversand963

fbshipit-source-id: b9b9ae2b3a8b4a16f0384292e71c6aecca93c570
      2a12b807
  13. 23 3月, 2021 3 次提交
    • Y
      Add user-defined timestamps to db_stress (#8061) · 08144bc2
      由 Yanqin Jin 提交于 
      Summary:
Add some basic test for user-defined timestamp to db_stress. Currently,
read with timestamp always tries to read using the current timestamp.
Due to the per-key timestamp-sequence ordering constraint, we only add timestamp-
related tests to the `NonBatchedOpsStressTest` since this test serializes accesses
to the same key and uses a file to cross-check data correctness.
The timestamp feature is not supported in a number of components, e.g. Merge, SingleDelete,
DeleteRange, CompactionFilter, Readonly instance, secondary instance, SST file ingestion, transaction,
etc. Therefore, db_stress should exit if user enables both timestamp and these features at the same
time. The (currently) incompatible features can be found in
`CheckAndSetOptionsForUserTimestamp`.

This PR also fixes a bug triggered when timestamp is enabled together with
`index_type=kBinarySearchWithFirstKey`. This bug fix will also be in another separate PR
with more unit tests coverage. Fixing it here because I do not want to exclude the index type
from crash test.

Pull Request resolved: https://github.com/facebook/rocksdb/pull/8061

Test Plan: make crash_test_with_ts

Reviewed By: jay-zhuang

Differential Revision: D27056282

Pulled By: riversand963

fbshipit-source-id: c3e00ad1023fdb9ebbdf9601ec18270c5e2925a9
      08144bc2
    • L
      Adjust the set of potential min_blob_size values in stress/crash tests (#8085) · 0d800dad
      由 Levi Tamasi 提交于 
      Summary:
Since our stress/crash tests by default generate values of size 8, 16, or 24,
it does not make much sense to set `min_blob_size` to 256. The patch
updates the set of potential `min_blob_size` values in the crash test
script and in `db_stress` where it might be set dynamically using
`SetOptions`.

Pull Request resolved: https://github.com/facebook/rocksdb/pull/8085

Test Plan: Ran `make check` and tried the crash test script.

Reviewed By: riversand963

Differential Revision: D27238620

Pulled By: ltamasi

fbshipit-source-id: 4a96f9944b1ed9220d3045c5ab0b34c49009aeee
      0d800dad
    • Y
      Remove duplicate code (#8079) · d6052d38
      由 Yanqin Jin 提交于 
      Summary:
The implementation of TransactionDB::WrapDB() and
TransactionDB::WrapStackableDB() are almost identical, except for the
type of the first argument `db`. This PR adds a new template function in
anonymous namespace, and calls it in the above two functions.

Pull Request resolved: https://github.com/facebook/rocksdb/pull/8079

Test Plan: make check

Reviewed By: lth

Differential Revision: D27184575

Pulled By: riversand963

fbshipit-source-id: f2855a6db3a7e897d0d611f7050ca4b696c56a7a
      d6052d38
  15. 22 3月, 2021 1 次提交
  17. 20 3月, 2021 5 次提交
    • P
      Begin forward compatibility for new backup meta schema (#8069) · 3bfd3ed2
      由 Peter Dillinger 提交于 
      Summary:
This does not add any new public APIs or published
functionality, but adds the ability to read and use (and in tests,
write) backups with a new meta file schema, based on the old schema
but not forward-compatible (before this change). The new schema enables
some capabilities not in the old:

* Explicit versioning, so that users get clean error messages the next
time we want to break forward compatibility.
* Ignoring unrecognized fields (with warning), so that new non-critical
features can be added without breaking forward compatibility.
* Rejecting future "non-ignorable" fields, so that new features critical
to some use-cases could potentially be added outside of linear schema
versions, with broken forward compatibility.
* Fields at the end of the meta file, such as for checksum of the meta
file's contents (up to that point)
* New optional 'size' field for each file, which is checked when present
* Optionally omitting 'crc32' field, so that we aren't required to have
a crc32c checksum for files to take a backup. (E.g. to support backup
via hard links and to better support file custom checksums.)

Because we do not have a JSON parser and to share code, the new schema
is simply derived from the old schema.

BackupEngine code is updated to allow missing checksums in some places,
and to make that easier, `has_checksum` and `verify_checksum_after_work`
are eliminated. Empty `checksum_hex` indicates checksum is unknown. I'm
not too afraid of regressing on data integrity, because
(a) we have pretty good test coverage of corruption detection in backups, and
(b) we are increasingly relying on the DB itself for data integrity rather than
it being an exclusive feature of backups.

Pull Request resolved: https://github.com/facebook/rocksdb/pull/8069

Test Plan:
new unit tests, added to crash test (some local run with
boosted backup probability)

Reviewed By: ajkr

Differential Revision: D27139824

Pulled By: pdillinger

fbshipit-source-id: 9e0e4decfb42bb84783d64d2d246456d97e8e8c5
      3bfd3ed2
    • S
      Remove unused variable (#8067) · c8b0842b
      由 storagezhang 提交于 
      Summary:
Remove unused variable `Slice blob_to_write` in `db/blob/blob_file_cache_test.cc`.

Pull Request resolved: https://github.com/facebook/rocksdb/pull/8067

Reviewed By: zhichao-cao

Differential Revision: D27107693

Pulled By: riversand963

fbshipit-source-id: 9bfd4d296a6a1714ad5c1fa5bb231a0c52dbd56d
      c8b0842b
    • S
      Include C++ standard library headers instead of C compatibility headers (#8068) · d9be6556
      由 storagezhang 提交于 
      Summary: Pull Request resolved: https://github.com/facebook/rocksdb/pull/8068

Reviewed By: zhichao-cao

Differential Revision: D27147685

Pulled By: riversand963

fbshipit-source-id: 5428b1c0142ecae17c977fba31a6d49b52983d1c
      d9be6556
    • S
      Add default in switch (#8065) · c7063242
      由 storagezhang 提交于 
      Summary:
switch may not cover all branch in `db/c.cc`:

```c++
void rocksdb_options_set_access_hint_on_compaction_start(
    rocksdb_options_t* opt, int v) {
  switch(v) {
    case 0:
      opt->rep.access_hint_on_compaction_start =
          ROCKSDB_NAMESPACE::Options::NONE;
      break;
    case 1:
      opt->rep.access_hint_on_compaction_start =
          ROCKSDB_NAMESPACE::Options::NORMAL;
      break;
    case 2:
      opt->rep.access_hint_on_compaction_start =
          ROCKSDB_NAMESPACE::Options::SEQUENTIAL;
      break;
    case 3:
      opt->rep.access_hint_on_compaction_start =
          ROCKSDB_NAMESPACE::Options::WILLNEED;
      break;
  }
}
```

Pull Request resolved: https://github.com/facebook/rocksdb/pull/8065

Reviewed By: riversand963

Differential Revision: D27102892

Pulled By: zhichao-cao

fbshipit-source-id: ad1d20d192712878e61597311ba75b55df0066d7
      c7063242
    • Z
      Add new Append API with DataVerificationInfo to Env WritableFile (#8071) · dd0447ae
      由 Zhichao Cao 提交于 
      Summary:
Add the new Append and PositionedAppend API to env WritableFile. User is able to benefit from the write checksum handoff API when using the legacy Env classes. FileSystem already implemented the checksum handoff API.

Pull Request resolved: https://github.com/facebook/rocksdb/pull/8071

Test Plan: make check, added new unit test.

Reviewed By: anand1976

Differential Revision: D27177043

Pulled By: zhichao-cao

fbshipit-source-id: 430c8331fc81099fa6d00f4fff703b68b9e8080e
      dd0447ae
  19. 19 3月, 2021 2 次提交