- 02 10月, 2020 8 次提交
-
-
由 Andrew Kryczka 提交于
Summary: This PR schedules a background thread (shared across all DB instances) to flush info log every ten seconds. This improves debuggability in case of RocksDB hanging since it ensures the log messages leading up to the hang will eventually become visible in the log. The bulk of this PR is moving monitoring/stats_dump_scheduler* to db/periodic_work_scheduler* and making the corresponding name changes since now the scheduler handles info log flushing, not just stats dumping. Pull Request resolved: https://github.com/facebook/rocksdb/pull/7488 Reviewed By: riversand963 Differential Revision: D24065165 Pulled By: ajkr fbshipit-source-id: 339c47a0ff43b79fdbd055fbd9fefbb6f9d8d3b5
-
由 sdong 提交于
Summary: Add all status handling in db_properties_test so that it can pass ASSERT_STATUS_CHECKED. Pull Request resolved: https://github.com/facebook/rocksdb/pull/7490 Test Plan: Run the test with ASSERT_STATUS_CHECKED Reviewed By: jay-zhuang Differential Revision: D24065382 fbshipit-source-id: e008916155196891478c964df0226545308ca71d
-
由 Zhichao Cao 提交于
Summary: Add trace_analyzer_test to ASSERT_STATUS_CHECKED list Pull Request resolved: https://github.com/facebook/rocksdb/pull/7480 Test Plan: ASSERT_STATUS_CHECKED=1 make -j48 trace_analyzer_test Reviewed By: riversand963 Differential Revision: D24033768 Pulled By: zhichao-cao fbshipit-source-id: b415045e6fab01d6193448650772368c21c6dba6
-
由 shadowlux 提交于
Summary: Pull Request resolved: https://github.com/facebook/rocksdb/pull/7442 Reviewed By: jay-zhuang Differential Revision: D24011201 Pulled By: ajkr fbshipit-source-id: 54f24c29875f7575612a780f15a42cda918d6641
-
由 Peter Dillinger 提交于
Summary: This exposes to the listener interface whether a compaction was full or not. Also cleaned up API comment for CompactionJobInfo::stats, which is not of a nullable type. And since CompactionJob is always created with non-null CompactionJobStats, removed conditionals on it being nullptr and instead assert non-null. TODO later: update C and Java interfaces Pull Request resolved: https://github.com/facebook/rocksdb/pull/7451 Test Plan: updated existing unit tests to check new field, make check Reviewed By: ltamasi Differential Revision: D23977796 Pulled By: pdillinger fbshipit-source-id: 1ae7e26cb949631c2b2fb9e696710daf53cc378d
-
由 Levi Tamasi 提交于
Summary: `DBTest.FileCreationRandomFailure` frequently times out during our continuous test runs. (It's a case of "stress test posing as unit test.") The patch reduces the number of iterations to avoid this. Note that the lower numbers are still sufficient to trigger both flushes and compactions, so test coverage is still the same. Pull Request resolved: https://github.com/facebook/rocksdb/pull/7481 Test Plan: `make check` Reviewed By: riversand963 Differential Revision: D24034712 Pulled By: ltamasi fbshipit-source-id: 8731a9446e5a121a1041b00f0df473b9f714935a
-
由 sdong 提交于
Summary: Introduce an new option options.check_flush_compaction_key_order, by default set to true, which checks key order of flush and compaction, and fail the operation if the order is violated. Also did minor refactor hash checking code, which consolidates the hashing logic to a vlidation class, where the key ordering logic is added. Pull Request resolved: https://github.com/facebook/rocksdb/pull/7467 Test Plan: Add unit tests to validate the check can catch reordering in flush and compaction, and can be properly disabled. Reviewed By: riversand963 Differential Revision: D24010683 fbshipit-source-id: 8dd6292d2cda8006054e9ded7cfa4bf405f0527c
-
由 Koby Kahane 提交于
Summary: This PR addresses some build and functional issues on MSVC targets, as a step towards an eventual goal of having RocksDB build successfully for Windows on ARM64. Addressed issues include: - BitsSetToOne and CountTrailingZeroBits do not compile on non-x64 MSVC targets. A fallback implementation of BitsSetToOne when Intel intrinsics are not available is added, based on the C++20 `<bit>` popcount implementation in Microsoft's STL. - The implementation of FloorLog2 for MSVC targets (including x64) gives incorrect results. The unit test easily detects this, but CircleCI is currently configured to only run a specific set of tests for Windows CMake builds, so this seems to have been unnoticed. - AsmVolatilePause does not use YieldProcessor on Windows ARM64 targets, even though it is available. - When CondVar::TimedWait calls Microsoft STL's condition_variable::wait_for, it can potentially trigger a bug (just recently fixed in the upcoming VS 16.8's STL) that deadlocks various tests that wait for a timer to execute, since `Timer::Run` doesn't get a chance to execute before being blocked by the test function acquiring the mutex. - In c_test, `GetTempDir` assumes a POSIX-style temp path. - `NormalizePath` did not eliminate consecutive POSIX-style path separators on Windows, resulting in test failures in e.g., wal_manager_test. - Various other test failures. In a followup PR I hope to modify CircleCI's config.yml to invoke all RocksDB unit tests in Windows CMake builds with CTest, instead of the current use of `run_ci_db_test.ps1` which requires individual tests to be specified and is missing many of the existing tests. Notes from peterd: FloorLog2 is not yet used in production code (it's for something in progress). I also added a few more inexpensive platform-dependent tests to Windows CircleCI runs. And included facebook/folly#1461 as requested Pull Request resolved: https://github.com/facebook/rocksdb/pull/7439 Reviewed By: jay-zhuang Differential Revision: D24021563 Pulled By: pdillinger fbshipit-source-id: 0ec2027c0d6a494d8a0fe38d9667fc2f7e29f7e7
-
- 01 10月, 2020 10 次提交
-
-
由 Yanqin Jin 提交于
Summary: We would like to build a shared library with all fbcode dependencies statically linked within. This resulting .so should not drop any symbols definitions in the building process. To ensure that, we use `link_whole=True` according to https://buck.build/rule/cxx_library.html#link_whole. Since `link_whole` is `False` by default, adding a `link_whole=False` to existing libraries won't change any behavior. Pull Request resolved: https://github.com/facebook/rocksdb/pull/7466 Test Plan: build a .so and test internally. Reviewed By: pdillinger Differential Revision: D24009780 Pulled By: riversand963 fbshipit-source-id: d18804d495da7195ed72a2040e1a5de4fd336519
-
由 shadowlux 提交于
Summary: Pull Request resolved: https://github.com/facebook/rocksdb/pull/7468 Reviewed By: riversand963 Differential Revision: D24028736 Pulled By: ajkr fbshipit-source-id: bd2b4b8d069491a16373d3d2705fddf7ebfe6723
-
由 Ramkumar Vadivelu 提交于
Summary: Fixes https://github.com/facebook/rocksdb/issues/7430 Change ParseInternalKey() to return Status instead of bool. db_bench (seekrandom) based before/after results with value size of 100 bytes and 16 bytes can be found at (tests ran on an udb server): https://www.dropbox.com/s/47bwamdy5ozngph/PIK_ret_Status_results.xlsx?dl=0 ![db_bench_results](https://user-images.githubusercontent.com/62277872/94642825-2a21a800-029a-11eb-88f2-124136c83fd3.png) Pull Request resolved: https://github.com/facebook/rocksdb/pull/7457 Reviewed By: ajkr Differential Revision: D24002433 Pulled By: ramvadiv fbshipit-source-id: ac253ecf577a29044c47c3fe254a01e71404c44c
-
由 Jay Zhuang 提交于
Summary: It's a transient failure, but can be reproduce with running the test 100 times: https://app.circleci.com/pipelines/github/facebook/rocksdb/3760/workflows/de909685-f22b-45ba-a8f3-6ebb78a54e96/jobs/37039 Pull Request resolved: https://github.com/facebook/rocksdb/pull/7478 Test Plan: re-run the test 100 times Reviewed By: ajkr Differential Revision: D24035758 Pulled By: jay-zhuang fbshipit-source-id: 6b31983d5c3f7faa8d5481306098513485d0d69d
-
由 Andrew Kryczka 提交于
Summary: Do not assert the number of files after intra-L0 compaction is eligible to run since it could complete (and reduce the number of files) before the assertion executes. Pull Request resolved: https://github.com/facebook/rocksdb/pull/7477 Reviewed By: pdillinger Differential Revision: D24032049 Pulled By: ajkr fbshipit-source-id: e838ac7a24651ebd643b9e5a9d39d2e789c46929
-
由 Jay Zhuang 提交于
Summary: As the issue is fixed in official repo: https://github.com/google/gtest-parallel/pull/79 Pull Request resolved: https://github.com/facebook/rocksdb/pull/7433 Reviewed By: ajkr Differential Revision: D24023271 Pulled By: jay-zhuang fbshipit-source-id: 0babf5e4c59cd61ded5a64cf9aa2d457deeeaa47
-
由 sdong 提交于
Summary: It's important to make sure no false positive is reported when options.paranoid_file_checks is used. Add it to stress test and a place holder in crash test. It is disabled in crash test as there appears to be a bug causing false positive. Pull Request resolved: https://github.com/facebook/rocksdb/pull/7473 Test Plan: Run crash test Reviewed By: ajkr Differential Revision: D24026939 fbshipit-source-id: 89102acb45cf041776775ce44a4eef4b0f3a380c
-
由 Akanksha Mahajan 提交于
Summary: Add status check for unit tests : block_based_filter_block_test, block_fetcher_test, full_filter_block_test and partitioned_filter_block_test Pull Request resolved: https://github.com/facebook/rocksdb/pull/7464 Reviewed By: zhichao-cao Differential Revision: D24011309 Pulled By: akankshamahajan15 fbshipit-source-id: d814803f94e8bb8b811ef170d20b22d52c1a3ff2
-
由 Peter Dillinger 提交于
Summary: This has been running in production on some key workloads, so we believe it to be safe and extremely low cost. Nevertheless, I've added code to ensure that "force_consistency_checks" is mentioned in any corruption reports so that people know how to disable in case of false positive corruption reports. Pull Request resolved: https://github.com/facebook/rocksdb/pull/7446 Test Plan: make check, CI, temporary debug print new message with ./version_builder_test Reviewed By: ajkr Differential Revision: D23972101 Pulled By: pdillinger fbshipit-source-id: 9623e400f3752577c0ecf977e6d0915562cf9968
-
由 sdong 提交于
Summary: https://github.com/facebook/rocksdb/pull/7452 added an uncessary skip for status code checking. Revert it. Pull Request resolved: https://github.com/facebook/rocksdb/pull/7458 Test Plan: Watch CI to finish Reviewed By: jay-zhuang Differential Revision: D23994390 fbshipit-source-id: a2b50a6326d8073db3386bff3d32acc5a6666e9b
-
- 30 9月, 2020 8 次提交
-
-
由 Akanksha Mahajan 提交于
Summary: Add a new Option "allow_data_in_errors". When it's set by users, it allows them to opt-in to get error messages containing corrupted keys/values. Corrupt keys, values will be logged in the messages, logs, status etc. that will help users with the useful information regarding affected data. By default value is set false to prevent users data to be exposed in the messages. Pull Request resolved: https://github.com/facebook/rocksdb/pull/7420 Test Plan: 1. make check -j64 2. Add a new test case Reviewed By: ajkr Differential Revision: D23835028 Pulled By: akankshamahajan15 fbshipit-source-id: 8d2eba8fb898e79fcf1fccc07295065a75eb59b1
-
由 Jay Zhuang 提交于
Summary: Pull Request resolved: https://github.com/facebook/rocksdb/pull/7454 Reviewed By: riversand963 Differential Revision: D23981719 Pulled By: jay-zhuang fbshipit-source-id: 01073f73e54c17067b886c4a2f179b2804198399
-
由 Andrew Kryczka 提交于
Summary: Pull Request resolved: https://github.com/facebook/rocksdb/pull/7455 Reviewed By: riversand963 Differential Revision: D23985283 Pulled By: ajkr fbshipit-source-id: 5dd2be62350f6e31d13a1e7821cb848a37699c93
-
由 anand76 提交于
Summary: Pull Request resolved: https://github.com/facebook/rocksdb/pull/7450 Reviewed By: zhichao-cao Differential Revision: D23992811 Pulled By: anand1976 fbshipit-source-id: 71bd898aafce6a3add3c8cd86d9f0e0fb63860cf
-
由 Andrew Kryczka 提交于
Summary: After unclean crash, the tail of the log could look as follows due to block buffering, even when the call to `ROCKSDB_LOG_ERROR()` finished. ``` 2020/09/29-13:54:39.596710 7f67025fe700 [ERROR] [/db_impl/db_impl_compaction_flush.cc:2500] Waiting after background compaction err ``` This PR forces the flush while logging warning severity or higher to prevent that case. Pull Request resolved: https://github.com/facebook/rocksdb/pull/7462 Reviewed By: riversand963 Differential Revision: D24000154 Pulled By: ajkr fbshipit-source-id: 3bf5f1e69a62ee10e84095cebc88937a8f81b4ad
-
由 Yanqin Jin 提交于
Summary: As title Pull Request resolved: https://github.com/facebook/rocksdb/pull/7459 Test Plan: make check Reviewed By: anand1976 Differential Revision: D23999393 Pulled By: riversand963 fbshipit-source-id: 09df8e1637f4df3616c63ee314de397b35be4e4a
-
由 anand76 提交于
Summary: The assertion checks that there is no overlap in sequence numbers across levels in universal compaction. However, this assumption doesn't hold when there is a delete triggered compaction or a trivial move, as they operate on a subset of a level. Tests - make check Pull Request resolved: https://github.com/facebook/rocksdb/pull/7421 Reviewed By: ajkr Differential Revision: D23872672 Pulled By: anand1976 fbshipit-source-id: c386deab8e01a5746ca996ff1f4ebcae3b15b7d2
-
由 sdong 提交于
Summary: Add db_basic_test status check list. Some of the warnings are suppressed. It is possible that some of them are due to real bugs. Pull Request resolved: https://github.com/facebook/rocksdb/pull/7452 Test Plan: See CI tests pass. Reviewed By: zhichao-cao Differential Revision: D23979764 fbshipit-source-id: 6151570c2a9b931b0fbb3fe939a94b2bd1583cbe
-
- 29 9月, 2020 7 次提交
-
-
由 Levi Tamasi 提交于
Summary: The patch adds support for injecting errors when reading from `RandomAccessFile` using `FaultInjectionTestEnv`. (This functionality was curiously missing w/r/t `RandomAccessFile`, even though it was implemented for `RandomRWFile`.) The patch also fixes up a test case in `blob_db_test` which uses `FaultInjectionTestEnv` but has so far relied on reads from `RandomAccessFile`s succeeding even after deactivating the filesystem. Pull Request resolved: https://github.com/facebook/rocksdb/pull/7447 Test Plan: `make check` Reviewed By: zhichao-cao Differential Revision: D23971740 Pulled By: ltamasi fbshipit-source-id: 8492736cb64b1ee138c658822535f3ff4fe560c6
-
由 Yanqin Jin 提交于
Summary: Check the status for file_reader_writer_test. Pull Request resolved: https://github.com/facebook/rocksdb/pull/7449 Test Plan: ``` ASSERT_STATUS_CHECKED=1 make -j20 file_reader_writer_test ./file_reader_writer_test ``` Reviewed By: zhichao-cao Differential Revision: D23975609 Pulled By: riversand963 fbshipit-source-id: a468eb04b386967fcc0478a56e4f0a19bdf81cdf
-
由 Andrew Kryczka 提交于
Summary: Pull Request resolved: https://github.com/facebook/rocksdb/pull/7448 Reviewed By: jay-zhuang Differential Revision: D23973281 Pulled By: ajkr fbshipit-source-id: 90fe55f1db904f1a236f3f7994d0806b8d24282c
-
由 Zhichao Cao 提交于
Summary: Pull Request resolved: https://github.com/facebook/rocksdb/pull/7445 Test Plan: pass ASSERT_STATUS_CHECKED=1 make -j48 flush_job_test Reviewed By: akankshamahajan15 Differential Revision: D23969372 Pulled By: zhichao-cao fbshipit-source-id: 498ff45ef84e07ec27a8f35d0874d3371412afe9
-
由 Levi Tamasi 提交于
Summary: Pull Request resolved: https://github.com/facebook/rocksdb/pull/7444 Test Plan: `ASSERT_STATUS_CHECKED=1 make version_builder_test -j24` Reviewed By: jay-zhuang Differential Revision: D23965793 Pulled By: ltamasi fbshipit-source-id: 8beaf66548379f21146189cda699d5f6fbb35a1b
-
由 Ramkumar Vadivelu 提交于
Summary: Reset refitting_level_ flag to false in error paths in DBImpl::ReFitLevel() Pull Request resolved: https://github.com/facebook/rocksdb/pull/7403 Reviewed By: ajkr Differential Revision: D23909028 Pulled By: ramvadiv fbshipit-source-id: 521ad9aadc1b734bef9ef9119d1e1ee1fa8126e9
-
由 Peter Dillinger 提交于
Summary: A generic algorithm in progress depends on a templatized version of fastrange, so this change generalizes it and renames it to fit our style guidelines, FastRange32, FastRange64, and now FastRangeGeneric. Pull Request resolved: https://github.com/facebook/rocksdb/pull/7436 Test Plan: added a few more test cases Reviewed By: jay-zhuang Differential Revision: D23958153 Pulled By: pdillinger fbshipit-source-id: 8c3b76101653417804997e5f076623a25586f3e8
-
- 26 9月, 2020 7 次提交
-
-
由 Yanqin Jin 提交于
Summary: Re-add extra_compiler_flags when building unit tests for fbcode. Pull Request resolved: https://github.com/facebook/rocksdb/pull/7437 Test Plan: Integrate with buck and run internal tests. Reviewed By: pdillinger Differential Revision: D23943924 Pulled By: riversand963 fbshipit-source-id: b92b7ad003e06e0860c45efc5f7f9684233d0c55
-
由 Jay Zhuang 提交于
Summary: Resolve TSAN build warnings and re-enable disabled TSAN tests. Not sure if it's a compiler issue or TSAN check issue. Switching from conditional operator to if-else mitigated the problem. Pull Request resolved: https://github.com/facebook/rocksdb/pull/7386 Test Plan: run TSAN check 10 times in circleci. ``` WARNING: ThreadSanitizer: data race (pid=27735) Atomic write of size 8 at 0x7b54000005e8 by thread T32: #0 __tsan_atomic64_store <null> (db_test+0x4cee95) https://github.com/facebook/rocksdb/issues/1 std::__atomic_base<unsigned long>::store(unsigned long, std::memory_order) /usr/bin/../lib/gcc/x86_64-linux-gnu/5.4.0/../../../../include/c++/5.4.0/bits/atomic_base.h:374:2 (db_test+0x78460e) https://github.com/facebook/rocksdb/issues/2 rocksdb::VersionSet::SetLastSequence(unsigned long) /home/circleci/project/./db/version_set.h:1058:20 (db_test+0x78460e) ... Previous read of size 8 at 0x7b54000005e8 by thread T31: #0 bool rocksdb::DBImpl::MultiCFSnapshot<std::unordered_map<unsigned int, rocksdb::DBImpl::MultiGetColumnFamilyData, std::hash<unsigned int>, std::equal_to<unsigned int>, std::allocator<std::pair<unsigned int const, rocksdb::DBImpl::MultiGetColumnFamilyData> > > >(rocksdb::ReadOptions const&, rocksdb::ReadCallback*, std::function<rocksdb::DBImpl::MultiGetColumnFamilyData* (std::unordered_map<unsigned int, rocksdb::DBImpl::MultiGetColumnFamilyData, std::hash<unsigned int>, std::equal_to<unsigned int>, std::allocator<std::pair<unsigned int const, rocksdb::DBImpl::MultiGetColumnFamilyData> > >::iterator&)>&, std::unordered_map<unsigned int, rocksdb::DBImpl::MultiGetColumnFamilyData, std::hash<unsigned int>, std::equal_to<unsigned int>, std::allocator<std::pair<unsigned int const, rocksdb::DBImpl::MultiGetColumnFamilyData> > >*, unsigned long*) /home/circleci/project/db/db_impl/db_impl.cc (db_test+0x715087) ``` Reviewed By: ltamasi Differential Revision: D23725226 Pulled By: jay-zhuang fbshipit-source-id: a6d662a5ea68111246cd32ec95f3411a25f76bc6
-
由 Peter Dillinger 提交于
Summary: Possible fix for a TSAN issue reported in EnableFileDeletions. disable_delete_obsolete_files_ should only be accessed holding the db mutex, but for logging it was being accessed outside holding the mutex, now fixed. Pull Request resolved: https://github.com/facebook/rocksdb/pull/7435 Test Plan: existing tests, watch for recurrence Reviewed By: ltamasi Differential Revision: D23917578 Pulled By: pdillinger fbshipit-source-id: 8573025bca3f6fe169b24b87bbfc4ce9667b0482
-
由 Cheng Chang 提交于
Summary: Add a method `CheckWals` in `WalSet` to check the logs on disk. See `CheckWals`'s comments. This method will be used to check consistency of WALs during DB recovery. Pull Request resolved: https://github.com/facebook/rocksdb/pull/7236 Test Plan: a set of tests are added to wal_edit_test.cc. Reviewed By: riversand963 Differential Revision: D23036505 Pulled By: cheng-chang fbshipit-source-id: 5b1d6857ac173429b00f950c32c4a5b8d063a732
-
由 Jay Zhuang 提交于
Summary: Similar to PR https://github.com/facebook/rocksdb/issues/7227, add read callback to filter out rows with with higher sequence number. Pull Request resolved: https://github.com/facebook/rocksdb/pull/7404 Reviewed By: riversand963 Differential Revision: D23790762 Pulled By: jay-zhuang fbshipit-source-id: bce854307612f1a22f985ffc934da627d0a139c2
-
由 Jay Zhuang 提交于
Summary: Pull Request resolved: https://github.com/facebook/rocksdb/pull/7332 Test Plan: `cmake .. -DASSERT_STATUS_CHECKED=1`, then run tests Reviewed By: zhichao-cao Differential Revision: D23437128 Pulled By: jay-zhuang fbshipit-source-id: 66fce57ae5d126e7a8b91c16a73c81438cf3b65e
-
由 Levi Tamasi 提交于
Summary: The patch introduces a helper method in `util/compression.h` called `UncompressData` that dispatches calls to the correct uncompression method based on type, and changes `UncompressBlockContentsForCompressionType` and `Benchmark::Uncompress` in `db_bench` so they are implemented in terms of the new method. This eliminates some code duplication. (`Benchmark::Compress` is also updated to use the previously introduced `CompressData` helper.) In addition, the patch brings the implementation of `Snappy_Uncompress` into sync with the other uncompression methods by making the method compute the buffer size and allocate the buffer itself. Finally, the patch eliminates some potentially risky back-and-forth conversions between various unsigned and signed integer types by exposing the size of the allocated buffer as a `size_t` instead of an `int`. Pull Request resolved: https://github.com/facebook/rocksdb/pull/7434 Test Plan: `make check` `./db_bench -benchmarks=compress,uncompress --compression_type ...` Reviewed By: riversand963 Differential Revision: D23900011 Pulled By: ltamasi fbshipit-source-id: b25df63ceec4639889be94acb22eb53e530c54e0
-