1. 10 9月, 2020 7 次提交
  2. 09 9月, 2020 7 次提交
    • P
      Fix some errors showing up in Travis builds (#7359) · 9de912de
      Peter Dillinger 提交于
      Summary:
      Also enables a pull request to trigger all the Travis
      configurations by writing FULL_CI in the commit message. (See what I did
      there?)
      
      First issue
      
          make: *** No rule to make target 'jl/util/crc32c_ppc_asm.o', needed by 'rocksdbjava'.  Stop.
      
      Second issue
      
          tools/db_bench_tool.cc:5514:38: error: ‘gen_exp.rocksdb::Benchmark::GenerateTwoTermExpKeys::keyrange_size_’ may be used uninitialized in this function
      
      Pull Request resolved: https://github.com/facebook/rocksdb/pull/7359
      
      Test Plan: CI
      
      Reviewed By: zhichao-cao
      
      Differential Revision: D23582132
      
      Pulled By: pdillinger
      
      fbshipit-source-id: 06d794673fd522ba11cf6398385387e6bd97ef89
      9de912de
    • A
      Use FSRandomRWFilePtr Object to call underlying file system. (#7198) · 0de335e0
      Akanksha Mahajan 提交于
      Summary:
      Replace FSRandomRWFile pointer with FSRandomRWFilePtr object in the rocksdb internal code.
      This new object wraps FSRandomRWFile pointer.
      
      Objective: If tracing is enabled, FSRandomRWFile object returns FSRandomRWFileTracingWrapper pointer that includes all necessary information in IORecord and calls underlying FileSystem and invokes IOTracer to dump that record in a binary file. If tracing is disabled then, underlying FileSystem pointer is returned directly.
      FSRandomRWFilePtr wrapper class is added to bypass the FSRandomRWFileWrapper when
      tracing is disabled.
      
      Pull Request resolved: https://github.com/facebook/rocksdb/pull/7198
      
      Test Plan: make check -j64
      
      Reviewed By: anand1976
      
      Differential Revision: D23421116
      
      Pulled By: akankshamahajan15
      
      fbshipit-source-id: 8a5ba0e7d9c1ba34c3a6f29829b107c5f09ab6a3
      0de335e0
    • J
      Fix compile error for old gcc-4.8 (#7358) · 8a8a01c6
      Jay Zhuang 提交于
      Summary:
      gcc-4.8 returns error when using the constructor. Not sure if it's a compiler bug/limitation or code issue:
      ```
      table/block_based/block_based_table_reader.cc:3183:67: error: use of deleted function ‘rocksdb::WritableFileStringStreamAdapter::WritableFileStringStreamAdapter(rocksdb::WritableFileStringStreamAdapter&&)’
      ```
      
      Pull Request resolved: https://github.com/facebook/rocksdb/pull/7358
      
      Reviewed By: pdillinger
      
      Differential Revision: D23577651
      
      Pulled By: jay-zhuang
      
      fbshipit-source-id: b0197e3d3538da61a6f3866410d88d2047fb9695
      8a8a01c6
    • Y
      Update HISTORY.md for PR7329 (#7355) · 8307d440
      Yanqin Jin 提交于
      Summary:
      As title.
      
      Pull Request resolved: https://github.com/facebook/rocksdb/pull/7355
      
      Reviewed By: pdillinger
      
      Differential Revision: D23566635
      
      Pulled By: riversand963
      
      fbshipit-source-id: f8d846bcff637e7617b764b7bfb9a948ea18d195
      8307d440
    • A
      Store FSWritableFilePtr object in WritableFileWriter (#7193) · b175eceb
      Akanksha Mahajan 提交于
      Summary:
      Replace FSWritableFile pointer with FSWritableFilePtr
          object in WritableFileWriter.
          This new object wraps FSWritableFile pointer.
      
          Objective: If tracing is enabled, FSWritableFile Ptr returns
          FSWritableFileTracingWrapper pointer that includes all necessary
          information in IORecord and calls underlying FileSystem and invokes
          IOTracer to dump that record in a binary file. If tracing is disabled
          then, underlying FileSystem pointer is returned directly.
          FSWritableFilePtr wrapper class is added to bypass the
          FSWritableFileWrapper when
          tracing is disabled.
      
          Test Plan: make check -j64
      
      Pull Request resolved: https://github.com/facebook/rocksdb/pull/7193
      
      Reviewed By: anand1976
      
      Differential Revision: D23355915
      
      Pulled By: akankshamahajan15
      
      fbshipit-source-id: e62a27a13c1fd77e36a6dbafc7006d969bed25cf
      b175eceb
    • P
      Fix backup/restore in stress/crash test (#7357) · 4e258d3e
      Peter Dillinger 提交于
      Summary:
      (1) Skip check on specific key if restoring an old backup
      (small minority of cases) because it can fail in those cases. (2) Remove
      an old assertion about number of column families and number of keys
      passed in, which is broken by atomic flush (cf_consistency) test. Like
      other code (for better or worse) assume a single key and iterate over
      column families. (3) Apply mock_direct_io to NewSequentialFile so that
      db_stress backup works on /dev/shm.
      
      Also add more context to output in case of backup/restore db_stress
      failure.
      
      Also a minor fix to BackupEngine to report first failure status in
      creating new backup, and drop another clue about the potential
      source of a "Backup failed" status.
      
      Reverts "Disable backup/restore stress test (https://github.com/facebook/rocksdb/issues/7350)"
      
      Pull Request resolved: https://github.com/facebook/rocksdb/pull/7357
      
      Test Plan:
      Using backup_one_in=10000,
      "USE_CLANG=1 make crash_test_with_atomic_flush" for 30+ minutes
      "USE_CLANG=1 make blackbox_crash_test" for 30+ minutes
      And with use_direct_reads with TEST_TMPDIR=/dev/shm/rocksdb
      
      Reviewed By: riversand963
      
      Differential Revision: D23567244
      
      Pulled By: pdillinger
      
      fbshipit-source-id: e77171c2e8394d173917e36898c02dead1c40b77
      4e258d3e
    • L
      Clean up SubcompactionState a bit (#7322) · 423d0511
      Levi Tamasi 提交于
      Summary:
      The patch cleans up a few things in `CompactionJob::SubcompactionState`:
      
      * Instead of using both the member initializer list and in-class initializers (and
      sometimes both at the same time for the same member), the struct now uniformly
      uses the latter to initialize integer members.
      * The default parameter value for the constructor parameter `size` is removed.
      * The explicitly deleted copy operations are removed, since they are implicitly deleted
      anyways because of the `unique_ptr` members.
      * The handwritten move operations, which did not move the member `c_iter` and
      were not declared `nothrow`, are removed. Note that with the user-declared copy
      operations gone (see the previous item), we can rely on the compiler to (correctly)
      generate these methods.
      
      Pull Request resolved: https://github.com/facebook/rocksdb/pull/7322
      
      Test Plan: `make check`
      
      Reviewed By: siying
      
      Differential Revision: D23382408
      
      Pulled By: ltamasi
      
      fbshipit-source-id: a4ae5af150161c50ff7bdc07fa145482d0150bfe
      423d0511
  3. 05 9月, 2020 3 次提交
    • Y
      Add a new stats level to exclude tickers (#7329) · ab202e8d
      Yanqin Jin 提交于
      Summary:
      Currently, application may pass a statistics object to db but later
      wants to reduce stats tracking overhead by setting stats level to
      kExceptHistogramOrTimers (the current lowest level). Tickers will still
      be incremented, causing up to 1% CPU. We can add a new lowest stats
      level `kExceptTickers` to disable ticker incrementing as well, thus
      reducing CPU cycles spent on tickers.
      
      Test Plan (devserver):
      ```
      make check
      make clean
      DEBUG_LEVEL=0 make db_bench
      ./db_bench -perf_level=1 -stats_level=0 -statistics -benchmarks=fillseq,readrandom -duration=120
      ```
      
      Measure CPU util (%) before and after change:
      CPU util by rocksdb::RecordTick: 1.1 vs (<0.1)
      
      Pull Request resolved: https://github.com/facebook/rocksdb/pull/7329
      
      Reviewed By: pdillinger
      
      Differential Revision: D23434014
      
      Pulled By: riversand963
      
      fbshipit-source-id: 72ff0f02a192ac476d4b0044b9f37fd4a22ff0d4
      ab202e8d
    • J
      Add sst_file_dumper status check (#7315) · 27aa443a
      Jay Zhuang 提交于
      Summary: Pull Request resolved: https://github.com/facebook/rocksdb/pull/7315
      
      Test Plan:
      `ASSERT_STATUS_CHECKED=1 make sst_dump_test && ./sst_dump_test`
      And manually run `./sst_dump --file=*.sst` before and after the change.
      
      Reviewed By: pdillinger
      
      Differential Revision: D23361669
      
      Pulled By: jay-zhuang
      
      fbshipit-source-id: 5bf51a2a90ee35c8c679e5f604732ec2aef5949a
      27aa443a
    • J
      Disable backup/restore stress test (#7350) · ef32f110
      Jay Zhuang 提交于
      Summary:
      Seems it's causing some tests failures.
      
      Pull Request resolved: https://github.com/facebook/rocksdb/pull/7350
      
      Reviewed By: riversand963
      
      Differential Revision: D23544109
      
      Pulled By: jay-zhuang
      
      fbshipit-source-id: 798a0ca374a20b6c2d0f29582729ff101c6a2e99
      ef32f110
  4. 04 9月, 2020 10 次提交
    • P
      Add file checksum to stress/crash test (#7343) · 06ad5dd2
      Peter Dillinger 提交于
      Summary:
      This change has the crash test randomly select from a few file
      checksum implementations, or nullptr, for DB file_checksum_gen_factory.
      For compatibility across runs on same DB, each non-null factory can
      understand all the other functions, but the default changes.
      
      Pull Request resolved: https://github.com/facebook/rocksdb/pull/7343
      
      Test Plan:
      'make blackbox_crash_test' for a while, including with some
      debug output to ensure code is being exercised.
      
      Reviewed By: zhichao-cao
      
      Differential Revision: D23494580
      
      Pulled By: pdillinger
      
      fbshipit-source-id: 73bbc7ca32c1adaf619134c0c830f12894880b8a
      06ad5dd2
    • C
      Fix wrong level args (#7346) · 3f9b7560
      Cheng Chang 提交于
      Summary:
      The level args should be output level instead of input levels.
      
      Pull Request resolved: https://github.com/facebook/rocksdb/pull/7346
      
      Test Plan: make check
      
      Reviewed By: ajkr
      
      Differential Revision: D23506373
      
      Pulled By: cheng-chang
      
      fbshipit-source-id: b2f701d44c13581c5c10c4dbebded4fcd354d641
      3f9b7560
    • P
      Fix, enable, and enhance backup/restore in db_stress (#7348) · 499c9448
      Peter Dillinger 提交于
      Summary:
      Although added to db_stress, testing of backup/restore
      was never integrated into the crash test, originally concerned about
      performance. I've enabled it now and to address the peformance concern,
      testing backup/restore is always skipped once the db exceeds a certain
      size threshold, default 100MB. This should provide sufficient
      opportunity for testing BackupEngine without bogging down everything
      else with heavier and heavier operations.
      
      Also fixed backup/restore in db_stress by making sure PurgeOldBackups
      can remove manifest files, which are normally kept around for db_stress.
      
      Added more coverage of backup options, and up to three backups being
      saved in one backup directory (in some cases).
      
      Pull Request resolved: https://github.com/facebook/rocksdb/pull/7348
      
      Test Plan:
      ran 'make blackbox_crash_test' for a while, with heightened
      probabilitly of taking backups (1/10k). Also confirmed with some debug
      output that the code is being covered, TestBackupRestore only takes
      a few seconds to complete when triggered, and even at 1/10k and ~50MB
      database, there's <,~ 1 thread testing backups at any time.
      
      Reviewed By: ajkr
      
      Differential Revision: D23510835
      
      Pulled By: pdillinger
      
      fbshipit-source-id: b6b8735591808141f81f10773ac31634cf03b6c0
      499c9448
    • A
      add `ldb unsafe_remove_sst_file` subcommand (#7335) · 57467673
      Andrew Kryczka 提交于
      Summary:
      This is adapted from https://github.com/facebook/rocksdb/issues/6678 but takes a different approach, avoiding opening a read-write DB and avoiding the `DeleteFile()` API.
      
      First, this PR refactors how options variables are initialized in `ldb` so it can be reused in a subcommand that doesn't open a DB:
      
      - Separated remaining option initialization logic out of `OpenDB()`. The new `PrepareOptions()` function initializes the full options state.
      - Fixed an old TODO about applying the subcommand CF option overrides to the proper `ColumnFamilyOptions` object.
      
      Second, this PR adds the `ldb unsafe_remove_sst_file` subcommand. It uses the `VersionSet`-level APIs to remove the file with the specified number.
      
      Pull Request resolved: https://github.com/facebook/rocksdb/pull/7335
      
      Test Plan: played with interactive python and this file removal command. Verified openability/correct results in case of multiple column families, multiple levels, etc.
      
      Reviewed By: pdillinger
      
      Differential Revision: D23454575
      
      Pulled By: ajkr
      
      fbshipit-source-id: 039b7a8cbfc42fd123dcb25821eef51d61148afe
      57467673
    • A
      add warning on `DeleteFile()` API (#7337) · 40e97b02
      Andrew Kryczka 提交于
      Summary:
      Since we can't land https://github.com/facebook/rocksdb/issues/7336 until the next major release, added a strong warning against the `DeleteFile()` API in the meantime.
      
      Pull Request resolved: https://github.com/facebook/rocksdb/pull/7337
      
      Reviewed By: pdillinger
      
      Differential Revision: D23459728
      
      Pulled By: ajkr
      
      fbshipit-source-id: 326cb9b18190386080c35c761a8736d8a877dafb
      40e97b02
    • A
      fix SstFileWriter with dictionary compression (#7323) · af54c409
      Andrew Kryczka 提交于
      Summary:
      In block-based table builder, the cut-over from buffered to unbuffered
      mode involves sampling the buffered blocks and generating a dictionary.
      There was a bug where `SstFileWriter` passed zero as the `target_file_size`
      causing the cutover to happen immediately, so there were no samples
      available for generating the dictionary.
      
      This PR changes the meaning of `target_file_size == 0` to mean buffer
      the whole file before cutting over. It also adds dictionary compression
      support to `sst_dump --command=recompress` for easy evaluation.
      
      Pull Request resolved: https://github.com/facebook/rocksdb/pull/7323
      
      Reviewed By: cheng-chang
      
      Differential Revision: D23412158
      
      Pulled By: ajkr
      
      fbshipit-source-id: 3b232050e70ef3c2ee85a4b5f6fadb139c569873
      af54c409
    • E
      Expose rocksdb_open_column_families_with_ttl C function (#7314) · 5b1ccdc1
      Eduardo Barreto Alexandre 提交于
      Summary:
      This PR creates `rocksdb_open_column_families_with_ttl` which allows C API users to open a DBWithTLL with column families.
      
      Pull Request resolved: https://github.com/facebook/rocksdb/pull/7314
      
      Reviewed By: cheng-chang
      
      Differential Revision: D23430287
      
      Pulled By: ajkr
      
      fbshipit-source-id: 307aa21d170d1402653263a91f6f832ef76afba0
      5b1ccdc1
    • H
      Avoid converting MERGES to PUTS when allow_ingest_behind is true (#7166) · d0c1a01c
      Hiep 提交于
      Summary:
      - Closes https://github.com/facebook/rocksdb/issues/6490
      - Currently MERGEs are converted to PUTs at bottom or compaction has reached the beginning of the key, this can wrongly cover a PUT future base case.
      
      Pull Request resolved: https://github.com/facebook/rocksdb/pull/7166
      
      Test Plan:
      - Automated: `make all check`
      - Manual: With `allow_ingest_behind = true`, add Merge operations to a key then run compaction. Then run ingesting external files to make sure the base case is probably compacted with existing Merges.
      
      Reviewed By: cheng-chang
      
      Differential Revision: D23325425
      
      Pulled By: ajkr
      
      fbshipit-source-id: 3eb415eb7b381b5453e45245393566153b1abb68
      d0c1a01c
    • H
      Close databases on benchmark error exits in db_bench (#7327) · 679a413f
      Hans Holmberg 提交于
      Summary:
      Delete database instances to make sure there are no loose threads
      running before exit(). This fixes segfaults seen when running
      workloads through CompositeEnvs with custom file systems.
      
      For further background on the issues arising when using CompositeEnvs, see the discussion in:
      https://github.com/facebook/rocksdb/pull/6878
      
      Pull Request resolved: https://github.com/facebook/rocksdb/pull/7327
      
      Reviewed By: cheng-chang
      
      Differential Revision: D23433244
      
      Pulled By: ajkr
      
      fbshipit-source-id: 4e19cf2067e3fe68c2a3fe1823f24b4091336bbe
      679a413f
    • P
      New bit manipulation functions and 128-bit value library (#7338) · c4d8838a
      Peter Dillinger 提交于
      Summary:
      These new functions and 128-bit value bit operations are
      expected to be used in a forthcoming Bloom filter alternative.
      
      No functional changes to production code, just new code only called by
      unit tests, cosmetic changes to existing headers, and fix an existing
      function for a yet-unused template instantiation (BitsSetToOne on
      something signed and smaller than 32 bits).
      
      Pull Request resolved: https://github.com/facebook/rocksdb/pull/7338
      
      Test Plan:
      Unit tests included. Works with and without
      TEST_UINT128_COMPAT=1 to check compatibility with and without
      __uint128_t. Also added that parameter to the CircleCI build
      build-linux-shared_lib-alt_namespace-status_checked.
      
      Reviewed By: jay-zhuang
      
      Differential Revision: D23494945
      
      Pulled By: pdillinger
      
      fbshipit-source-id: 5c0dc419100d9df5d4d9abb153b2855d5aea39e8
      c4d8838a
  5. 03 9月, 2020 2 次提交
  6. 02 9月, 2020 3 次提交
    • A
      Bound L0->Lbase fanout in dynamic leveled compaction (#7325) · 177f8bd0
      Andrew Kryczka 提交于
      Summary:
      L0 score is based on size target and number of files. The size target
      used is `max_bytes_for_level_base`. However, the base level's size can
      dynamically expand in write burst mode. In fact, it can expand so much
      that L0->Lbase becomes the highest fanout in target sizes. This doesn't
      make sense from an efficiency perspective, so this PR bounds the
      L0->Lbase fanout to the smoothed level multiplier. The L0 scoring based
      on file count remains unchanged.
      
      Pull Request resolved: https://github.com/facebook/rocksdb/pull/7325
      
      Test Plan:
      contrived benchmark that exhibits the problem:
      
      ```
      $ TEST_TMPDIR=/data/users/andrewkr/ ./db_bench -benchmarks=filluniquerandom,readrandom -write_buffer_size=1048576 -target_file_size_base=1048576 -max_bytes_for_level_base=4194304 -level0_file_num_compaction_trigger=4 -level_compaction_dynamic_level_bytes=true -compression_type=none -max_background_jobs=12 -rate_limiter_bytes_per_sec=104857600 -benchmark_write_rate_limit=10485760 -num=100000000
      ```
      
      Results:
      
      - "Burst W-Amp" is the write-amp near the end of the fillrandom benchmark
      - "Total W-Amp" is the write-amp after readrandom has run a while and all levels no longer need compaction
      
      Branch | Burst W-Amp | Total W-Amp | fillrandom (MB/s)
      -- | -- | -- | --
      master | 20.2 | 21.5 | 4.7
      dynamic-l0-score | 12.6 | 14.1 | 7.2
      
      Reviewed By: siying
      
      Differential Revision: D23412935
      
      Pulled By: ajkr
      
      fbshipit-source-id: f91f2067188e432dd39deab02f1c56f195057a0e
      177f8bd0
    • T
      Make examples work on Windows (#7304) · 8d44d792
      Tomasz Posłuszny 提交于
      Summary:
      Quick fixes to examples to make it easier to get familiar with RocksDB for Windows users:
      
      - Set proper temporary directory path on Windows for all examples (with C++17 we should start using std::filesystem)
      - Fixed typo and got rid of warnings treated as errors in multi_processes_example.cc
      - Get number of available cores on Windows in c_simple_example.c
      - Add command to remove DB directory for Windows in compaction_filter_example.cc (print error, but carry on with example upon error, because error code is returned if there is no such directory on Windows)
      
      Pull Request resolved: https://github.com/facebook/rocksdb/pull/7304
      
      Reviewed By: jay-zhuang
      
      Differential Revision: D23450900
      
      Pulled By: pdillinger
      
      fbshipit-source-id: 4256134deb6ae6bb267ed1bd69f814842b95f60f
      8d44d792
    • J
      Recompress blobs during GC if compression changed (#7331) · 55bf42a8
      Jay Zhuang 提交于
      Summary:
      Recompress blobs if compression type is changed.
      
      Pull Request resolved: https://github.com/facebook/rocksdb/pull/7331
      
      Test Plan: `make check`
      
      Reviewed By: ltamasi
      
      Differential Revision: D23437102
      
      Pulled By: jay-zhuang
      
      fbshipit-source-id: bb699ebdad137721d422e42e331d4de8a82a7c5f
      55bf42a8
  7. 01 9月, 2020 1 次提交
    • L
      Log info about generated blob files in BlobFileBuilder (#7324) · 792d2f90
      Levi Tamasi 提交于
      Summary:
      The patch adds a log message to `BlobFileBuilder` that is logged upon
      generating a blob file, similarly to how we log the generation of table files
      during flush and compaction. The log message contains the column family
      name, job id, blob file number, and the number and total size of blobs in
      the new file.
      
      Pull Request resolved: https://github.com/facebook/rocksdb/pull/7324
      
      Test Plan: Ran `make check` and checked the actual log messages using a custom `db_bench`.
      
      Reviewed By: riversand963
      
      Differential Revision: D23402229
      
      Pulled By: ltamasi
      
      fbshipit-source-id: ca42beb4db284b783d1eb2651f321032a45d0c5f
      792d2f90
  8. 29 8月, 2020 2 次提交
  9. 28 8月, 2020 3 次提交
    • J
      Add buffer prefetch support for non directIO usecase (#7312) · c2485f2d
      Jay Zhuang 提交于
      Summary:
      A new file interface `SupportPrefetch()` is added. When the user overrides it to `false`, an internal prefetch buffer will be used for readahead. Useful for non-directIO but FS doesn't have readahead support.
      
      Pull Request resolved: https://github.com/facebook/rocksdb/pull/7312
      
      Reviewed By: anand1976
      
      Differential Revision: D23329847
      
      Pulled By: jay-zhuang
      
      fbshipit-source-id: 71cd4ce6f4a820840294e4e6aec111ab76175527
      c2485f2d
    • L
      Add a blob file builder class that can be used in background jobs (#7306) · 50439606
      Levi Tamasi 提交于
      Summary:
      The patch adds a class called `BlobFileBuilder` that can be used to build
      and cut blob files in background jobs (flushes/compactions). The class
      enforces a value size threshold (`min_blob_size`; smaller blobs will be inlined
      in the LSM tree itself), and supports specifying a blob file size limit (`blob_file_size`),
      as well as compression (`blob_compression_type`) and checksums for blob files.
      It also keeps track of the generated blob files and their associated `BlobFileAddition`
      metadata, which can be applied as part of the background job's `VersionEdit`.
      
      Pull Request resolved: https://github.com/facebook/rocksdb/pull/7306
      
      Test Plan: `make check`
      
      Reviewed By: riversand963
      
      Differential Revision: D23298817
      
      Pulled By: ltamasi
      
      fbshipit-source-id: 38f35d81dab1ba81f15236240612ec173d7f21b5
      50439606
    • A
      Store FSRandomAccessPtr object in RandomAccessFileReader (#7192) · 8e0df905
      Akanksha Mahajan 提交于
      Summary:
      Replace FSRandomAccessFile pointer with FSRandomAccessFilePtr
          object in RandomAccessFileReader.
          This new object wraps FSRandomAccessFile pointer.
      
          Objective: If tracing is enabled, FSRandomAccessFile Ptr returns
          FSRandomAccessFileTracingWrapper pointer that includes all necessary
          information in IORecord and calls underlying FileSystem and invokes
          IOTracer to dump that record in a binary file. If tracing is disabled
          then, underlying FileSystem pointer is returned directly.
          FSRandomAccessFilePtr wrapper class is added to bypass the FSRandomAccessFileWrapper when
          tracing is disabled.
      
          Test Plan: make check -j64
      
      Pull Request resolved: https://github.com/facebook/rocksdb/pull/7192
      
      Reviewed By: anand1976
      
      Differential Revision: D23356867
      
      Pulled By: akankshamahajan15
      
      fbshipit-source-id: 48f31168166a17a7444b40be44a9a9d4a5c7182c
      8e0df905
  10. 27 8月, 2020 1 次提交
    • P
      Real fix for race in backup custom checksum checking (#7309) · 9aad24da
      Peter Dillinger 提交于
      Summary:
      This is a "real" fix for the issue worked around in https://github.com/facebook/rocksdb/issues/7294.
      To get DB checksum info for live files, we now read the manifest file
      that will become part of the checkpoint/backup. This requires a little
      extra handling in taking a custom checkpoint, including only reading the
      manifest file up to the size prescribed by the checkpoint.
      
      This moves GetFileChecksumsFromManifest from backup code to
      file_checksum_helper.{h,cc} and removes apparently unnecessary checking
      related to column families.
      
      Updated HISTORY.md and warned potential future users of
      DB::GetLiveFilesChecksumInfo()
      
      Pull Request resolved: https://github.com/facebook/rocksdb/pull/7309
      
      Test Plan: updated unit test, before and after
      
      Reviewed By: ajkr
      
      Differential Revision: D23311994
      
      Pulled By: pdillinger
      
      fbshipit-source-id: 741e30a2dc1830e8208f7648fcc8c5f000d4e2d5
      9aad24da
  11. 26 8月, 2020 1 次提交
    • S
      Get() to fail with underlying failures in PartitionIndexReader::CacheDependencies() (#7297) · 722814e3
      sdong 提交于
      Summary:
      Right now all I/O failures under PartitionIndexReader::CacheDependencies() is swallowed. This doesn't impact correctness but we've made a decision that any I/O error in read path now should be returned to users for awareness. Return errors in those cases instead.
      
      Pull Request resolved: https://github.com/facebook/rocksdb/pull/7297
      
      Test Plan: Add a new unit test that ingest errors in this code path and see Get() fails. Only one I/O path is hit in PartitionIndexReader::CacheDependencies(). Several option changes are attempt but not able to got other pread paths triggered. Not sure whether other failure cases would be even possible. Would rely on continuous stress test to validate it.
      
      Reviewed By: anand1976
      
      Differential Revision: D23257950
      
      fbshipit-source-id: 859dbc92fa239996e1bb378329344d3d54168c03
      722814e3