- 10 7月, 2020 3 次提交
-
-
由 Andrew Kryczka 提交于
Summary: This is a followup to https://github.com/facebook/rocksdb/issues/6646. In that PR, for simplicity I just appended a comparison against the 0th restart key in case `BinarySeek()`'s binary search landed at index 0. As a result there were `2/(N+1) + log_2(N)` key comparisons. This PR does it differently. Now we expand the binary search range by one so it also covers the case where target is at or before the restart key at index 0. As a result, it involves `log_2(N+1)` key comparisons. Pull Request resolved: https://github.com/facebook/rocksdb/pull/7068 Test Plan: ran readrandom with mostly default settings and counted key comparisons using `PerfContext`. before: `user_key_comparison_count = 28881965` after: `user_key_comparison_count = 27823245` setup command: ``` $ TEST_TMPDIR=/dev/shm/dbbench ./db_bench -benchmarks=fillrandom,compact -write_buffer_size=1048576 -target_file_size_base=1048576 -max_bytes_for_level_base=4194304 -max_background_jobs=12 -level_compaction_dynamic_level_bytes=true -num=10000000 ``` benchmark command: ``` $ TEST_TMPDIR=/dev/shm/dbbench/ ./db_bench -use_existing_db=true -benchmarks=readrandom -disable_auto_compactions=true -num=10000000 -compression_type=none -reads=1000000 -perf_level=3 ``` Reviewed By: anand1976 Differential Revision: D22357032 Pulled By: ajkr fbshipit-source-id: 8b01e9c1c2a4e9d02fc9dfe16c1cc0327f8bdf24
-
由 Yanqin Jin 提交于
Summary: As title. The goal is to shorten the execution time of several tests when they are combined together in a single TEST_F. Pull Request resolved: https://github.com/facebook/rocksdb/pull/7105 Test Plan: make db_wal_test ./db_wal_test Reviewed By: ltamasi Differential Revision: D22442705 Pulled By: riversand963 fbshipit-source-id: 0ad49b8f21fa86dcd5a4d3c9a06af313735ac217
-
由 Yanqin Jin 提交于
Summary: as title. Pull Request resolved: https://github.com/facebook/rocksdb/pull/7107 Test Plan: make ldb && ./ldb Reviewed By: pdillinger Differential Revision: D22451399 Pulled By: riversand963 fbshipit-source-id: 797645e06473bb9cf139c533877e5161281515e8
-
- 09 7月, 2020 7 次提交
-
-
由 Zitan Chen 提交于
Summary: Although PR https://github.com/facebook/rocksdb/issues/7032 fixes the construction of the `SstFileDumper` in `GetFileDbIdentities` by setting a proper `Env` of the `Options` passed in the constructor, the file path was not corrected accordingly. This actually disables backup engine to use db session ids in the file names since the `db_session_id` is always empty. Now it is fixed by setting the correct path in the construction of `SstFileDumper`. Furthermore, to preserve the Direct IO property that backup engine already has, parameter `EnvOptions` is added to `GetFileDbIdentities` and `SstFileDumper`. The `BackupUsingDirectIO` test is updated accordingly. Pull Request resolved: https://github.com/facebook/rocksdb/pull/7104 Test Plan: backupable_db_test and some manual tests. Reviewed By: ajkr Differential Revision: D22443245 Pulled By: gg814 fbshipit-source-id: 056a9bb8b82947c5e73d7c3fbb62bfe23af5e562
-
由 Akanksha Mahajan 提交于
Update Flush policy in PartitionedIndexBuilder on switching from user-key to internal-key mode (#7096) Summary: When format_version is high enough to support user-key and there are index entries for same user key that spans multiple data blocks then it changes from user-key mode to internal-key mode. But the flush policy is not reset to point to Block Builder of internal-keys. After this switch, no entries are added to user key index partition result, thus it never triggers flushing the block. Fix: 1. After adding the entry in sub_builder_index_, if there is a switch from user-key to internal-key, then flush policy is updated to point to Block Builder of internal-keys index partition. 2. Set sub_builder_index_->seperator_is_key_plus_seq_ = true if seperator_is_key_plus_seq_ is set to true so that subsequent partitions can also use internal key mode. Pull Request resolved: https://github.com/facebook/rocksdb/pull/7096 Test Plan: make check -j64 Reviewed By: ajkr Differential Revision: D22416598 Pulled By: akankshamahajan15 fbshipit-source-id: 01fc2dc07ea1b32f8fb803995ebe6e9a3fbe67ac
-
由 球状闪电 提交于
Summary: WITH_TESTS=OFF and WITH_BENCHMARK_TOOLS=ON there has errors: /bin/ld: cannot find -ltestharness Pull Request resolved: https://github.com/facebook/rocksdb/pull/7040 Reviewed By: riversand963 Differential Revision: D22447637 Pulled By: ajkr fbshipit-source-id: f63058376deb4a2e6722d63541c40caa617c331a
-
由 Peter Dillinger 提交于
Summary: Delicious copy-pasta from https://github.com/facebook/rocksdb/issues/7039 Also fixing DestroyDir to allow files to go missing while it is operating. This seems to fix failures I got with test plan reproducer. Pull Request resolved: https://github.com/facebook/rocksdb/pull/7103 Test Plan: make blackbox_crash_test_with_atomic_flush for a while with checkpoint_one_in=100 Reviewed By: siying Differential Revision: D22435315 Pulled By: pdillinger fbshipit-source-id: 0ec0538402493887aeda43ecc03f32979cb84ced
-
由 Zitan Chen 提交于
Summary: The fix in PR https://github.com/facebook/rocksdb/issues/7082 is not really successful because there is still a small chance that the test will fail. In addtion to flushing, we close the DB and then reopen before corrupting a table file in the DB. Specifically, we corrupt a table file before backup takes place as follows. * Open DB * Fill DB * Flush DB (optional, no flushing here also works) * Close DB * Reopen DB * Corrupt a table file in the DB This should make the test reliable. Pull Request resolved: https://github.com/facebook/rocksdb/pull/7102 Test Plan: `while ./backupable_db_test --gtest_filter=*TableFileCorruptedBeforeBackup*; do true; done` (kept running for an hour or so :) Reviewed By: pdillinger Differential Revision: D22432417 Pulled By: gg814 fbshipit-source-id: d407eee93ff428bb662f80cde1659fbf0149d0cd
-
由 rafael-aero 提交于
Summary: Pull Request resolved: https://github.com/facebook/rocksdb/pull/7092 Reviewed By: riversand963 Differential Revision: D22412323 Pulled By: ajkr fbshipit-source-id: 3fc1c63bb19a8cd2c0ae620800c28f199a7f494b
-
由 rockeet 提交于
Summary: Pull Request resolved: https://github.com/facebook/rocksdb/pull/7080 Reviewed By: riversand963 Differential Revision: D22412352 Pulled By: ajkr fbshipit-source-id: 1d7f4c1621040a0130245139b52c3f4d3deac865
-
- 08 7月, 2020 6 次提交
-
-
由 wenh 提交于
Summary: Currently, `EventListener` in listner.h only have callback functions for file read and write. One may favor extended callback functions for more file I/O operations like flush, sync and close. This PR tries to add those interface and have them called when appropriate throughout the code base. Pull Request resolved: https://github.com/facebook/rocksdb/pull/7055 Test Plan: Write an experimental listener with those new callback functions with log output in them; run experiments and check logs to see those functions are actually called. Default test suits `make check` should also be included. Reviewed By: riversand963 Differential Revision: D22380624 Pulled By: roghnin fbshipit-source-id: 4121491d45c2c2aae8c255e7998090559a241c6a
-
由 Andrew Kryczka 提交于
Summary: Replace `BlockIter::comparator_` and `IndexBlockIter::user_comparator_wrapper_` with a concrete `UserComparatorWrapper` and `InternalKeyComparator`. The motivation for this change was the inconvenience of not knowing the concrete type of `BlockIter::comparator_`, which prevented calling specialized internal key comparison functions to optimize comparison of keys with global seqno applied. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6944 Test Plan: benchmark setup -- single file DBs, in-memory, no compression. "normal_db" created by regular flush; "ingestion_db" created by ingesting a file. Both DBs have same contents. ``` $ TEST_TMPDIR=/dev/shm/normal_db/ ./db_bench -benchmarks=fillrandom,compact -write_buffer_size=10485760000 -disable_auto_compactions=true -compression_type=none -num=1000000 $ ./ldb write_extern_sst ./tmp.sst --db=/dev/shm/ingestion_db/dbbench/ --compression_type=no --hex --create_if_missing < <(./sst_dump --command=scan --output_hex --file=/dev/shm/normal_db/dbbench/000007.sst | awk 'began {print "0x" substr($1, 2, length($1) - 2), "==>", "0x" $5} ; /^Sst file format: block-based/ {began=1}') $ ./ldb ingest_extern_sst ./tmp.sst --db=/dev/shm/ingestion_db/dbbench/ ``` benchmark run command: ``` $ TEST_TMPDIR=/dev/shm/$DB/ ./db_bench -benchmarks=seekrandom -seek_nexts=$SEEK_NEXT -use_existing_db=true -cache_index_and_filter_blocks=false -num=1000000 -cache_size=0 -threads=1 -reads=200000000 -mmap_read=1 -verify_checksum=false ``` results: perf improved marginally for ingestion_db and did not change significantly for normal_db: SEEK_NEXT | DB | code | ops/sec | % change -- | -- | -- | -- | -- 0 | normal_db | master | 350880 | 0 | normal_db | PR6944 | 351040 | 0.0 0 | ingestion_db | master | 343255 | 0 | ingestion_db | PR6944 | 349424 | 1.8 10 | normal_db | master | 218711 | 10 | normal_db | PR6944 | 217892 | -0.4 10 | ingestion_db | master | 220334 | 10 | ingestion_db | PR6944 | 226437 | 2.8 Reviewed By: pdillinger Differential Revision: D21924676 Pulled By: ajkr fbshipit-source-id: ea4288a2eefa8112eb6c651a671c1de18c12e538
-
由 Peter Dillinger 提交于
Summary: On some platforms like MacOS, a second 'make check' can lead to /bin/rm: Argument list too long This is fixed by replacing with a 'find'. Also, using '-f' for more rm calls to avoid prompt. Pull Request resolved: https://github.com/facebook/rocksdb/pull/7095 Test Plan: 'make check' on Linux and MacOS Reviewed By: riversand963 Differential Revision: D22415808 Pulled By: pdillinger fbshipit-source-id: 0fd1ebae13739c9d81f9e813e99b062715604d6b
-
由 Peter Dillinger 提交于
Summary: by tracking and linking against runtime dependent libraries in Makefile Pull Request resolved: https://github.com/facebook/rocksdb/pull/7098 Test Plan: look for fix in CircleCI Reviewed By: riversand963 Differential Revision: D22420860 Pulled By: pdillinger fbshipit-source-id: d211d709214bf5306db68e43b7a2f18169281022
-
由 Peter Dillinger 提交于
Summary: Fix a Facebook internal build Pull Request resolved: https://github.com/facebook/rocksdb/pull/7093 Test Plan: buck build mode/opt :c_test :c_test_bin (was compilation failure, now "not found") buck build mode/dev :c_test :c_test_bin (still passes) Reviewed By: ajkr Differential Revision: D22412528 Pulled By: pdillinger fbshipit-source-id: 8e55c43dbf95386597e4cc690c41d9cbdcee03aa
-
由 Peter Dillinger 提交于
Summary: Primarily, this change adds a way to work around a bug limiting the effective output (and therefore debugability) of the Linux builds using parallel make. We would get make[1]: write error: stdout probably due to a kernel bug, apparently affecting both available ubuntu 16 machine images (maybe not affecting docker images, less horsepower). https://bugs.launchpad.net/ubuntu/+source/linux-signed/+bug/1814393 Now in the CircleCI config, make output on Ubuntu is piped through a custom 'cat' that ignores EAGAIN errors, which seems to fix the problem. Significant other changes: * Add another linux build that combines * LIB_MODE=shared, to ensure this works with compile and unit test execution * Alternative rocksdb namespace, to ensure this works (not rely on Travis) * ASSERT_STATUS_CHECKED=1, but with building all unit tests and running those expected to pass with it * Run release build with and without gflags. (Was running only without, ignore large swaths of code in a normal release build! Two regressions in this build, only with gflags, in the last week not caught by CI!) * Use gflags with unity and LITE build, as typical case. Debugability improvements: * Use V=1 to show commands being executed (thanks to EAGAIN work-around) * Print kernel version and compiler versions as part of V=1 output from Makefile Cosmetic other changes: * Put more commands on one line, for less clutter in CircleCI output pages * Remove redundant "all" in "make all check" and put make command options before targets * Change some recursive "make clean" into dependency on "clean," toward minimizing unnecessary overhead (detect platform, build version, etc.) of extra recursive makes Pull Request resolved: https://github.com/facebook/rocksdb/pull/7078 Reviewed By: siying Differential Revision: D22391647 Pulled By: pdillinger fbshipit-source-id: d446fccf5a8c568b37dc8748621c8a5c546fe135
-
- 07 7月, 2020 5 次提交
-
-
由 Levi Tamasi 提交于
Summary: Pull Request resolved: https://github.com/facebook/rocksdb/pull/7086 Test Plan: `make check` Reviewed By: zhichao-cao Differential Revision: D22395420 Pulled By: ltamasi fbshipit-source-id: 088a20097bd6b73b0c433cd79725779f97ec04f2
-
由 Peter Dillinger 提交于
Summary: Pull Request resolved: https://github.com/facebook/rocksdb/pull/7072 Reviewed By: ajkr Differential Revision: D22391641 Pulled By: pdillinger fbshipit-source-id: 0ef355576454514263ab684eb1a5c06787f3242a
-
由 Adam Retter 提交于
Summary: Closes https://github.com/facebook/rocksdb/issues/5852 Pull Request resolved: https://github.com/facebook/rocksdb/pull/7047 Reviewed By: cheng-chang Differential Revision: D22335162 Pulled By: pdillinger fbshipit-source-id: 75f3c524deccea7ebc0ad288da41f1ea81406c1c
-
由 Peter Dillinger 提交于
Summary: (a) use STRESS_LIBRARY for db_stress and make sure STRESS_LIBRARY has other stress test dependencies (as in buck build) (b) fix rpath option to be accepted on MacOS. It still doesn't fully work for me e.g. to run a LIB_MODE=shared unit test binary from another directory, as it does on Linux, but the option is now accepted, and running unit tests from current directory works for me. Also adding LIB_MODE=shared to Travis. (Later TBD where best to fit in in CircleCI.) Pull Request resolved: https://github.com/facebook/rocksdb/pull/7066 Test Plan: manual Reviewed By: cheng-chang Differential Revision: D22364068 Pulled By: pdillinger fbshipit-source-id: 6fa98a222f89f808ee786474de1100d92c1adec3
-
由 Adam Retter 提交于
Summary: Adds the function `RocksDB#rocksdbVersion()` for retrieving the RocksDB version. Pull Request resolved: https://github.com/facebook/rocksdb/pull/7083 Reviewed By: cheng-chang Differential Revision: D22391628 Pulled By: pdillinger fbshipit-source-id: e1cabcf28aa81f5ee8dcdce5c9eca6b3155a279e
-
- 04 7月, 2020 1 次提交
-
-
由 Zitan Chen 提交于
Summary: If the corruption of a table file is done before flushing, then db manifest may record the checksum for the corrupted table, which results in "matching checksums" when backup engine tries to verfiy the checksum, and causes a flaky test. Fix the issue by adding `Flush()` before trying to corrupt a table file in *db*. Pull Request resolved: https://github.com/facebook/rocksdb/pull/7082 Test Plan: `buck test` Without the fix, failed 5 of 100 tests. Suspected whether the pseudo randomness causes the issue: doubling `keys_iteration` resulted in 2 of 100 tests failed; deterministically corrupting tables file also caused 2 of 100 tests to fail. With the fix, passed 200 of 200 tests. Reviewed By: pdillinger Differential Revision: D22375421 Pulled By: gg814 fbshipit-source-id: 7304618e7520684b6087e42d0b58329c5ad18329
-
- 03 7月, 2020 6 次提交
-
-
由 Jay Zhuang 提交于
Summary: The issue is introduced by https://github.com/facebook/rocksdb/issues/7067 Pull Request resolved: https://github.com/facebook/rocksdb/pull/7077 Test Plan: `make release` Reviewed By: pdillinger Differential Revision: D22370835 Pulled By: jay-zhuang fbshipit-source-id: 44326bae07809c4518371b6a7d1f47124e24a4f3
-
由 sdong 提交于
Summary: This is to fix special logic to run tests inside FB. Buck test is broken after moving to cpp_unittest(). Move c_test back to the previous approach. Pull Request resolved: https://github.com/facebook/rocksdb/pull/7076 Test Plan: Watch the Sandcastle run Reviewed By: ajkr Differential Revision: D22370096 fbshipit-source-id: 4a464d0903f2c76ae2de3a8ad373ffc9bedec64c
-
由 Jay Zhuang 提交于
Summary: Replace `reinterpret_cast` with `static_cast_with_check` for `DBImpl` and `ColumnFamilyHandleImpl`. Pull Request resolved: https://github.com/facebook/rocksdb/pull/7067 Reviewed By: siying Differential Revision: D22361587 Pulled By: jay-zhuang fbshipit-source-id: dfe9e8f3af39c3d27cc372c55ab9ad905eb0a5a1
-
由 Zitan Chen 提交于
Summary: When table file checksums are enabled and stored in the DB manifest by using the RocksDB default crc32c checksum function, BackupEngine will calculate the crc32c checksum of the file to be copied and compare the calculated result with the one stored in the DB manifest before copying the file to the backup directory. After copying to the backup directory, BackupEngine will verify the checksum of the copied file with the one calculated before copying. This helps detect some rare corruption events such as bit-flips during the copying process. No verification with checksums in DB manifest will be performed if the table file checksum function is not the RocksDB default crc32c checksum function. In addition, If `share_table_files` and `share_files_with_checksum` are true, BackupEngine will compare the checksums computed before and after copying of the table files. Corresponding tests are added. Pull Request resolved: https://github.com/facebook/rocksdb/pull/7015 Test Plan: Passed make check Reviewed By: pdillinger Differential Revision: D22165732 Pulled By: gg814 fbshipit-source-id: ee0e8cc397c455eba64545c29380b9d9853588ec
-
由 Peter Dillinger 提交于
Summary: Even though local bisection gave me a clear signal (and still does) that reverting https://github.com/facebook/rocksdb/issues/7049 would fix the failures in MultiThreadedDBTest, https://github.com/facebook/rocksdb/issues/7022 seems to be the root cause. Reverting https://github.com/facebook/rocksdb/issues/7022 and keeping https://github.com/facebook/rocksdb/issues/7049 seems to fix the issue in local reproducer also. (Had these landed in opposite order, bisection would have found the root cause.) Pull Request resolved: https://github.com/facebook/rocksdb/pull/7071 Reviewed By: akankshamahajan15 Differential Revision: D22362857 Pulled By: pdillinger fbshipit-source-id: ed63df3d74e9d4ce1604de8fe43b216166c7a3f0
-
由 Peter Dillinger 提交于
Summary: This reverts commit 4f1534bd. This commit caused failures and deadlocks in MultiThreadedDBTest.MultiThreaded/69 and others. Pull Request resolved: https://github.com/facebook/rocksdb/pull/7070 Reviewed By: riversand963 Differential Revision: D22358778 Pulled By: pdillinger fbshipit-source-id: faf8f2cb469a7063a113921c8e9c64a9f7610dac
-
- 02 7月, 2020 6 次提交
-
-
由 sdong 提交于
Summary: After https://github.com/facebook/rocksdb/pull/7036, we still see extra DBTest that can timeout when running 10 or 20 in parallel. Expand skip-fsync mode in whole DBTest. Still preserve other tests from doing this mode to be conservative. Pull Request resolved: https://github.com/facebook/rocksdb/pull/7049 Test Plan: Run all existing files. Reviewed By: pdillinger Differential Revision: D22301700 fbshipit-source-id: f9a9e3b3b26ce640665a47cb8bff33ba0c89b565
-
由 Zitan Chen 提交于
Summary: GetFileDbIdentities requires either db_id non-null or db_session_id non-null. Passing nullptr for db_id or db_session_id in CopyOrCreateFile indicates the caller does not want to obtain the value for db_id or db_session_id. Pull Request resolved: https://github.com/facebook/rocksdb/pull/7063 Test Plan: USE_CLANG=1 make analyze backupable_db_test Reviewed By: pdillinger Differential Revision: D22338497 Pulled By: gg814 fbshipit-source-id: 2aa2dcc14d156b0f99b07d6cf3c731ee088272cd
-
由 Akanksha Mahajan 提交于
Update Flush policy in PartitionedIndexBuilder on switching from user-key to internal-key mode (#7022) Summary: When format_version is high enough to support user-key and there are index entries for same user key that spans multiple data blocks then it changes from user-key mode to internal-key mode. But the flush policy is not reset to point to Block Builder of internal-keys. After this switch, no entries are added to user key index partition result, thus it never triggers flushing the block. Fix: After adding the entry in sub_builder_index_, if there is a switch from user-key to internal-key, then flush policy is updated to point to Block Builder of internal-keys index partition. Pull Request resolved: https://github.com/facebook/rocksdb/pull/7022 Test Plan: 1. make check -j64 2. Added one unit test case Reviewed By: ajkr Differential Revision: D22197734 Pulled By: akankshamahajan15 fbshipit-source-id: d87e9e46bccab8e896ee6979d6b79c51f73d479e
-
由 Andrew Kryczka 提交于
Summary: This check is flaky because compaction could run between the `Flush()` and the `TestGetTickerCount()`, which would increase the `BLOCK_CACHE_INDEX_MISS` count beyond what the test expects. Verified by adding a `sleep(1)` between those two lines and observing the counter is too high every time. The solution is just to remove this check as it doesn't have any use anyways. The latter check of index miss is sufficient to conclude the newest L0 file (i.e., the one generated by intra-L0) does not have its index block pinned in cache. It'd be nice to simultaneously check the L0 files generated by flush do have their index blocks pinned in cache, but that's not what the line deleted in this PR was checking.. Pull Request resolved: https://github.com/facebook/rocksdb/pull/7065 Reviewed By: pdillinger Differential Revision: D22340327 Pulled By: ajkr fbshipit-source-id: e076b2c7228b7fa763dd0c0cb13828e176c1abee
-
由 Peter Dillinger 提交于
Summary: Random memtable layouts could cause random failure, reproducible with command below running for a while. Test now using deterministic behavior. Pull Request resolved: https://github.com/facebook/rocksdb/pull/7064 Test Plan: while ./db_test --gtest_filter=*SizesMemTable*; do true; done Reviewed By: siying Differential Revision: D22339442 Pulled By: pdillinger fbshipit-source-id: 8e74e5a9b5e88f7030854045a22c12cf561d5de6
-
由 Peter Dillinger 提交于
Summary: Follow-up to https://github.com/facebook/rocksdb/issues/6660. Release build had linker error. fbcode+clang+shared build was erroring on unused parameter '-nostdinc'. Fixes https://github.com/facebook/rocksdb/issues/7061 Pull Request resolved: https://github.com/facebook/rocksdb/pull/7062 Test Plan: make release, USE_CLANG=1 LIB_MODE=shared make check, etc. Reviewed By: siying Differential Revision: D22335663 Pulled By: pdillinger fbshipit-source-id: 261cd959ca1f6c273dc763a70020a535ba8e81de
-
- 01 7月, 2020 6 次提交
-
-
由 mrambacher 提交于
Summary: Change the linking of tests/tools to be against a library rather than a list of objects. This change substantially reduces the size of the objects produced. peterd clean repo size: 264M Before this change, with make all: 40G After this change, with make all: 28G With make LIB_MODE=shared all: 7.0G The list of TESTS was changed from being hard-coded to generated from the test sources variable. Note that there are some test sources that are not built as tests (though the set of tests is identical to the previous version). Added OBJ_DIR option to Makefile to allow objects to be placed in an alternative location. By default, OBJ_DIR is the same as before ("./"). This change is a precursor to being able to build/run the tests/tools linked against static libraries. Additionally, it should be possible to clean up and merge some of the rules for building tests and the like if so desired. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6660 Reviewed By: riversand963 Differential Revision: D22244463 Pulled By: pdillinger fbshipit-source-id: db9c6341d81ed62c2270374f4ede02fb9604c754
-
由 Zitan Chen 提交于
Generalize BackupEngine naming option for share_files_with_checksum SSTs and revert BackupEngine::VerifyBackup to check only file sizes by default (#7032) Summary: `bool BackupableDBOptions::new_naming_for_backup_files` is updated to `BackupTableNameOption BackupableDBOptions::share_files_with_checksum_naming`, where `BackupTableNameOption` is an `enum` type with two enumerators `kChecksumAndFileSize` and `kChecksumAndFileSize`. This opens up possibilities of extenting the current naming scheme for backup table files. By default, `BackupTableNameOption BackupableDBOptions::share_files_with_checksum_naming` is set to `kChecksumAndDbSessionId`. Revert `BackupEngine::VerifyBackup` to only check file sizes by default. Also fix the construction of the `SstFileDumper` in `GetFileDbIdentities` by setting a proper `Env` of the `Options` passed in the constructor. Pull Request resolved: https://github.com/facebook/rocksdb/pull/7032 Test Plan: make check Reviewed By: ajkr Differential Revision: D22237763 Pulled By: gg814 fbshipit-source-id: 466902a4e731babd64e30f0e82ca1aa82962e52e
-
由 Yanqin Jin 提交于
Summary: As title. Pull Request resolved: https://github.com/facebook/rocksdb/pull/7057 Test Plan: ./tools/check_format_compatible.sh Reviewed By: pdillinger Differential Revision: D22319831 Pulled By: riversand963 fbshipit-source-id: 82653a525a5296ef65a6a7a439cdd6bff88f498e
-
由 Andrew Kryczka 提交于
Summary: With mmap enabled on an uncompressed file, we were previously always doing a heap allocation to obtain the scratch buffer for `RandomAccessFileReader::Read()`. However, that allocation was unnecessary as the underlying file reader returned a pointer into its mapped memory, not the provided scratch buffer. This PR makes passes the `BlockFetcher`'s inline buffer as the scratch buffer if the data block is small enough (less than `kDefaultStackBufferSize` bytes, currently 5000). Ideally we would not pass a scratch buffer at all for an mmap read; however, the `RandomAccessFile::Read()` API guarantees such a buffer is provided, and non-standard implementations may be relying on it even when `Options::allow_mmap_reads == true`. In that case, this PR still works but introduces an extra copy from the inline buffer to a heap buffer. Pull Request resolved: https://github.com/facebook/rocksdb/pull/7043 Reviewed By: cheng-chang Differential Revision: D22320606 Pulled By: ajkr fbshipit-source-id: ad964dd23df34e07d979c6032c2dfe5454c98b52
-
由 Levi Tamasi 提交于
Summary: The earlier `VersionBuilder` code only cleaned up blob files that were marked as entirely consisting of garbage using `VersionEdits` with `BlobFileGarbage`. This covers the cases when table files go through regular compaction, where we iterate through the KVs and thus have an opportunity to calculate the amount of garbage (that is, most cases). However, it does not help when table files are simply dropped (e.g. deletion compactions or the `DeleteFile` API). To deal with such cases, the patch adds logic that cleans up all blob files at the head of the list until the first one with linked SSTs is found. (As an example, let's assume we have blob files with numbers 1..10, and the first one with any linked SSTs is number 8. This means that SSTs in the `Version` only rely on blob files with numbers >= 8, and thus 1..7 are no longer needed.) The code change itself is pretty small; however, changing the logic like this necessitated changes to some tests that have been added recently (namely to the ones that use blob files in isolation, i.e. without any table files referring to them). Some of these cases were fixed by bypassing `VersionBuilder` altogether in order to keep the tests simple (which actually makes them more proper unit tests as well), while the `VersionBuilder` unit tests were fixed by adding dummy table files to the test cases as needed. Pull Request resolved: https://github.com/facebook/rocksdb/pull/7001 Test Plan: `make check` Reviewed By: riversand963 Differential Revision: D22119474 Pulled By: ltamasi fbshipit-source-id: c6547141355667d4291d9661d6518eb741e7b54a
-
由 Yanqin Jin 提交于
Summary: as title. Pull Request resolved: https://github.com/facebook/rocksdb/pull/7059 Test Plan: ./tools/check_format_compatible.sh Reviewed By: siying Differential Revision: D22320774 Pulled By: riversand963 fbshipit-source-id: 124d13b08703d077a7aab3678e1eb639fcbcceca
-