- 21 7月, 2020 2 次提交
-
-
由 sdong 提交于
Summary: https://github.com/facebook/rocksdb/pull/7145 creates a minor data race against the stat creation counter. Turn it to atomic. Pull Request resolved: https://github.com/facebook/rocksdb/pull/7151 Test Plan: Run the test. Reviewed By: ajkr Differential Revision: D22631014 fbshipit-source-id: c6fb69ac5b9df7139795dacea5ce9fb9fd3278d7
-
由 Jay Zhuang 提交于
Summary: To have test report. Pull Request resolved: https://github.com/facebook/rocksdb/pull/7137 Reviewed By: siying Differential Revision: D22630798 Pulled By: jay-zhuang fbshipit-source-id: bc07ba673c0bceed5a4829b4af2d9a74435379c7
-
- 18 7月, 2020 5 次提交
-
-
由 Zhichao Cao 提交于
Summary: Remove the 3 testing cases that cause the time out in linux build by https://github.com/facebook/rocksdb/issues/6765 . Will fix them later. Pull Request resolved: https://github.com/facebook/rocksdb/pull/7141 Test Plan: make asan_check, buck run Reviewed By: ajkr Differential Revision: D22593831 Pulled By: zhichao-cao fbshipit-source-id: 14956c36476ecc3393f613178c22e13df843126e
-
由 sdong 提交于
Summary: In CircleCI tests, we failed to fail tests properly if parallel doesn't return an error code. It's probably would happen when unit tests fail with signals, rather than return values. Fix them. Pull Request resolved: https://github.com/facebook/rocksdb/pull/7147 Test Plan: Manually ingest a failure and see it to fail. Reviewed By: jay-zhuang Differential Revision: D22611594 fbshipit-source-id: 88a42425a41d1213d29bd2e7c80731d2bdd5644b
-
由 Stanislau Hlebik 提交于
fbshipit-source-id: 798decc90db4f13770e97cdce3c0df7d5421b2a3
-
由 Stanislau Hlebik 提交于
fbshipit-source-id: 5113fe0c527595e4227ff827253b7414abbdf7ac
-
由 Andrew Kryczka 提交于
Summary: Previously when running `db_bench` with large value for `num_multi_dbs` and enabled `Options::dump_malloc_stats`, we would see most CPU spent in jemalloc locking. After this PR that no longer shows up at the top of the profile. Pull Request resolved: https://github.com/facebook/rocksdb/pull/7145 Reviewed By: riversand963 Differential Revision: D22593031 Pulled By: ajkr fbshipit-source-id: 3b3fc91f93249c6afee53f59f34c487c3fc5add6
-
- 17 7月, 2020 1 次提交
-
-
由 mrambacher 提交于
Summary: This PR does a few things: - The "compression_opts" and "bottom_compression_opts" can now be read/written as name/value pairs of options (instead of only a colon-separated list; - These options can now be read/written to the Options file; - The parallel_threads value can now be set (either in the colon or name-value format). The compression options are now stored and treated as a OptionTypeInfo::Struct by the options system, meaning they can be read and written like the other structs. This change allows them to be read/written easily to the options file. Additionally, the colon-format was extended to allow support for setting parallel threads. Tests were added to test all of the option settings via the optional parameters in the colon format. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6817 Reviewed By: ajkr Differential Revision: D22396004 Pulled By: zhichao-cao fbshipit-source-id: 38bcf74b7e9cd5bc2a84540fac2e9ba4f765b2c8
-
- 16 7月, 2020 7 次提交
-
-
由 Levi Tamasi 提交于
Summary: Periodic syncing of blob files is handled by a lower layer, namely by `WritableFileWriter`; the `NeedsFsync` method of `BlobFile` and the `last_fsync_` member variable are actually unused and thus can be removed. See also https://github.com/facebook/rocksdb/pull/7125 . Pull Request resolved: https://github.com/facebook/rocksdb/pull/7138 Test Plan: `make check` Reviewed By: zhichao-cao Differential Revision: D22562981 Pulled By: ltamasi fbshipit-source-id: c235aad94a7c27120528c9ec270a7a5b9154e49f
-
由 Akanksha Mahajan 提交于
Summary: Add "examples" build (which build examples folder in rocksdb) in TravisCI to CircleCI. This is helpful before pull request. Pull Request resolved: https://github.com/facebook/rocksdb/pull/7136 Test Plan: Watch for CircleCI results to succeed Reviewed By: jay-zhuang Differential Revision: D22555528 Pulled By: akankshamahajan15 fbshipit-source-id: 6bca16647760d5f0131f064765fe9e88e034c578
-
由 sdong 提交于
Summary: TSAN shows warning with clang with warning similar to this: WARNING: ThreadSanitizer: data race (pid=10159) Atomic write of size 8 at 0x7b5000002890 by thread T33: #0 __tsan_atomic64_store <null> (db_test+0x4ca2b5) https://github.com/facebook/rocksdb/issues/1 std::__atomic_base<unsigned long>::store(unsigned long, std::memory_order) /usr/bin/../lib/gcc/x86_64-linux-gnu/5.4.0/../../../../include/c++/5.4.0/bits/atomic_base.h:374:2 (db_test+0x774fde) https://github.com/facebook/rocksdb/issues/2 rocksdb::VersionSet::SetLastSequence(unsigned long) /home/circleci/project/./db/version_set.h:1057:20 (db_test+0x774fde) https://github.com/facebook/rocksdb/issues/3 rocksdb::DBImpl::WriteImpl(rocksdb::WriteOptions const&, rocksdb::WriteBatch*, rocksdb::WriteCallback*, unsigned long*, unsigned long, bool, unsigned long*, unsigned long, rocksdb::PreReleaseCallback*) /home/circleci/project/db/db_impl/db_impl_write.cc:449:18 (db_test+0x774fde) ...... Previous read of size 8 at 0x7b5000002890 by thread T5 (mutexes: write M1044689462619020832): #0 rocksdb::DBImpl::ReleaseSnapshot(rocksdb::Snapshot const*) /home/circleci/project/db/db_impl/db_impl.cc (db_test+0x6f4ae7) https://github.com/facebook/rocksdb/issues/1 rocksdb::(anonymous namespace)::MTThreadBody(void*) /home/circleci/project/db/db_test.cc:2514:13 (db_test+0x56ac59) https://github.com/facebook/rocksdb/issues/2 rocksdb::(anonymous namespace)::StartThreadWrapper(void*) /home/circleci/project/env/env_posix.cc:443:3 (db_test+0x88c4cd) It is not limited to ReleaseSnapshot() and rocksdb::DBImpl::MultiCFSnapshot(). While we are not 100% sure it doesn't indicate any correctness violation, we suppress them for now to keep TSAN clean with more tests so that we can cover more bugs with CI. In the gcc runs we have been running, this warning rarely shows up. Pull Request resolved: https://github.com/facebook/rocksdb/pull/7126 Test Plan: See the mini-TSAN test to pass with reasonable run time. Reviewed By: ajkr Differential Revision: D22552375 fbshipit-source-id: ebdd3854cb3becec3403970326a1ca961db2ab00
-
由 Levi Tamasi 提交于
Summary: In both cases where `BlobDBImpl::DecompressSlice` is called, `compression_type` is already checked at the call site; thus, the check inside the method is redundant and can be turned into an assertion. Pull Request resolved: https://github.com/facebook/rocksdb/pull/7127 Test Plan: `make check` Reviewed By: zhichao-cao Differential Revision: D22533454 Pulled By: ltamasi fbshipit-source-id: ae524443fc6abe0a5fb12327a3fe761a9cd2c831
-
由 Jay Zhuang 提交于
Summary: Pull Request resolved: https://github.com/facebook/rocksdb/pull/7135 Reviewed By: siying Differential Revision: D22553594 Pulled By: jay-zhuang fbshipit-source-id: 5b40621ffad05c4fffe3ac536a11d6c87e7ef512
-
由 Zhichao Cao 提交于
Summary: In current codebase, in write path, if Retryable IO Error happens, SetBGError is called. The retryable IO Error is converted to hard error and DB is in read only mode. User or application needs to resume it. In this PR, if Retryable IO Error happens in one DB, SetBGError will create a new thread to call Resume (auto resume). otpions.max_bgerror_resume_count controls if auto resume is enabled or not (if max_bgerror_resume_count<=0, auto resume will not be enabled). options.bgerror_resume_retry_interval controls the time interval to call Resume again if the previous resume fails due to the Retryable IO Error. If non-retryable error happens during resume, auto resume will terminate. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6765 Test Plan: Added the unit test cases in error_handler_fs_test and pass make asan_check Reviewed By: anand1976 Differential Revision: D21916789 Pulled By: zhichao-cao fbshipit-source-id: acb8b5e5dc3167adfa9425a5b7fc104f6b95cb0b
-
由 yxj25245 提交于
Summary: Fix typo in ThreadData comment Pull Request resolved: https://github.com/facebook/rocksdb/pull/7131 Reviewed By: riversand963 Differential Revision: D22543135 Pulled By: jay-zhuang fbshipit-source-id: 39c9d0e8cd5a364af9a2f05fd3783e8482dea976
-
- 15 7月, 2020 6 次提交
-
-
由 Yanqin Jin 提交于
Summary: Currently, RocksDB lets compaction to go through even in case of corrupted keys, the number of which is reported in CompactionJobStats. However, RocksDB does not check this value. We should let compaction run in a stricter mode. Temporarily disable two tests that allow corrupted keys in compaction. With this PR, the two tests will assert(false) and terminate. Still need to investigate what is the recommended google-test way of doing it. Death test (EXPECT_DEATH) in gtest has warnings now. Pull Request resolved: https://github.com/facebook/rocksdb/pull/7124 Test Plan: make check Reviewed By: ajkr Differential Revision: D22530722 Pulled By: riversand963 fbshipit-source-id: 6a5a6a992028c6d4f92cb74693c92db462ae4ad6
-
由 Levi Tamasi 提交于
Summary: https://github.com/facebook/rocksdb/pull/6850, which added compaction filter support to BlobDB, reused elements of the BlobDB GC mechanism. This patch updates some log messages in this logic to account for this fact; namely, it replaces mentions of "GC" with "compaction/GC" to avoid confusion in cases when GC is not enabled. Pull Request resolved: https://github.com/facebook/rocksdb/pull/7128 Test Plan: `make check` Reviewed By: zhichao-cao Differential Revision: D22535371 Pulled By: ltamasi fbshipit-source-id: 1f14f3b02ab9983728bbca1cf680420208d9a195
-
由 Adam Retter 提交于
Summary: This fixes an issue introduced in 0c56fc4d whereby the location of Python is evaluated many times and leads to excessive logging of unknown python locations of CentOS 6. The location is now only checked once. Pull Request resolved: https://github.com/facebook/rocksdb/pull/7123 Reviewed By: zhichao-cao Differential Revision: D22532274 Pulled By: ajkr fbshipit-source-id: cade71b4b46e9a23d63ecb4dd36a4ac8ae217970
-
由 sdong 提交于
Summary: It is helpful to add some TSAN coverage before a pull request is committed. This diff adds some of them. Some slow tests are excluded for the running speed. Some are blacklisted because they show warnings. Will investigate these warnings and see whether we can fix or suppress them. Pull Request resolved: https://github.com/facebook/rocksdb/pull/7122 Test Plan: Watch CIrcleCI runs Reviewed By: riversand963 Differential Revision: D22532133 fbshipit-source-id: 81ddd02d9df19c513a12811979e8ddabae911354
-
由 Levi Tamasi 提交于
Summary: Periodic syncing of blob files is performed by `WritableFileWriter`; `bytes_per_sync_` and `next_sync_offset_` in `BlobLogWriter` are actually unused (or more precisely, only used by methods that are themselves unused). The patch removes all this dead code. Pull Request resolved: https://github.com/facebook/rocksdb/pull/7125 Test Plan: `make check` Reviewed By: riversand963 Differential Revision: D22531021 Pulled By: ltamasi fbshipit-source-id: 6b293ad5a79d3e6bf15c5c68f7aedd7ce7a15f10
-
由 Jay Zhuang 提交于
Summary: Add stress test coverage for `DB::GetProperty()`. Pull Request resolved: https://github.com/facebook/rocksdb/pull/7111 Test Plan: ``` ./db_stress -get_property_one_in=1 make crash_test ``` Reviewed By: ajkr Differential Revision: D22487906 Pulled By: jay-zhuang fbshipit-source-id: c118d95cc9b4e2fa669a06e6aa531541fa885dc5
-
- 14 7月, 2020 5 次提交
-
-
由 Yanqin Jin 提交于
Summary: During memtable lookup, an unrecognized value type should be reported as Status::Corruption. Pull Request resolved: https://github.com/facebook/rocksdb/pull/7121 Test Plan: make check Reviewed By: cheng-chang Differential Revision: D22512124 Pulled By: riversand963 fbshipit-source-id: 9b97be7d9b230c5aae9205f96054420e5ea09066
-
由 sdong 提交于
Summary: Pull Request resolved: https://github.com/facebook/rocksdb/pull/7120 Test Plan: Watch CI results. Reviewed By: cheng-chang Differential Revision: D22516995 fbshipit-source-id: ddf7c0482dfac30d9044b833852348eda987cb67
-
由 Akanksha Mahajan 提交于
Summary: 1. Add the wrapper classes FileSystemTracingWrapper, FSSequentialFileTracingWrapper, FSRandomAccessFileTracingWrapper, FSWritableFileTracingWrapper, FSRandomRWFileTracingWrapper that forward the calls to underlying storage system and then pass the file operation information to IOTracer. IOTracer dumps the record in binary format for tracing. 2. Add the wrapper classes FileSystemPtr, FSSequentialFilePtr, FSRandomAccessFilePtr, FSWritableFilePtr and FSRandomRWFilePtr that overload operator-> and return ptr to underlying storage system or Tracing wrapper class based on enabling/disabling of IO tracing. These classes are added to bypass Tracing Wrapper classes when we disable tracing. 3. Add enums in trace.h that distinguish which options need to be added for different file operations(Read, close, write etc) as part of tracing record. Pull Request resolved: https://github.com/facebook/rocksdb/pull/7002 Test Plan: make check -j64 Reviewed By: anand1976 Differential Revision: D22127897 Pulled By: akankshamahajan15 fbshipit-source-id: 74cff58ce5661c9a3832dfaa52483f3b2d8565e0
-
由 Jay Zhuang 提交于
Summary: Add circleci java build workflow. Pull Request resolved: https://github.com/facebook/rocksdb/pull/7119 Reviewed By: siying Differential Revision: D22512426 Pulled By: jay-zhuang fbshipit-source-id: 45a7445c861fee48017bae42cdb7172c8b091475
-
由 sdong 提交于
Summary: CLANG analyze is useful before pull request. Add it. Pull Request resolved: https://github.com/facebook/rocksdb/pull/7114 Test Plan: Watch the CI results to succeed. Reviewed By: riversand963 Differential Revision: D22491942 fbshipit-source-id: 9ccad91c6142fedc3d3dd491cf55054827908f36
-
- 11 7月, 2020 5 次提交
-
-
由 sdong 提交于
Summary: It is useful to run UBSAN before merging a PR. This commit adds it. We see warning for stl_tree.h, suppress the warning to make it work. Pull Request resolved: https://github.com/facebook/rocksdb/pull/7112 Test Plan: See the CI to succeed. Manually ingest a UBSAN warning and see it got reported. Reviewed By: pdillinger Differential Revision: D22490519 fbshipit-source-id: e4495a0c78a3e2dae7dbf294da79585e141cbb66
-
由 Adam Retter 提交于
Summary: Closes https://github.com/facebook/rocksdb/issues/6729 Pull Request resolved: https://github.com/facebook/rocksdb/pull/7088 Reviewed By: riversand963 Differential Revision: D22481624 Pulled By: pdillinger fbshipit-source-id: 27c0ebd4168d374ae81f3595e034150c1c97f8b8
-
由 Stanislav Tkach 提交于
Summary: Along with https://github.com/facebook/rocksdb/issues/6925 and https://github.com/facebook/rocksdb/issues/6998, this should add getters for all Options fields except several ones with non-trivial interface (for example rocksdb_options_set_min_level_to_compress). Pull Request resolved: https://github.com/facebook/rocksdb/pull/7094 Reviewed By: riversand963 Differential Revision: D22479800 Pulled By: pdillinger fbshipit-source-id: d14f305e12cfe268d07e0fe229d55cef299c792a
-
由 Adam Retter 提交于
Summary: Closes https://github.com/facebook/rocksdb/issues/7053 Pull Request resolved: https://github.com/facebook/rocksdb/pull/7090 Reviewed By: riversand963 Differential Revision: D22481199 Pulled By: pdillinger fbshipit-source-id: 287477db94d57b18bee58189135f44936f1c3ca3
-
由 wenh 提交于
Summary: There currently exist multiple `GetChildren()` calls in `DBImpl::Recover()`, which can be expensive in cases of distributed file systems. This pull request try to call `DBImpl::Recover()` of each necessary directory only _once_ and reuse the results in the places of repeated calls in current code. Pull Request resolved: https://github.com/facebook/rocksdb/pull/7044 Test Plan: Run `make check` and use the default test suite. The modified code should be semantically identical to the current code. As a proof of this solution, we may optionally deploy the system onto a (real or simulated) distributed system and expect reduced latency caused by manifest fetching. (WIP) Reviewed By: riversand963 Differential Revision: D22419925 Pulled By: roghnin fbshipit-source-id: d3774fbfbc246c5527101bc16747eb5c90919886
-
- 10 7月, 2020 5 次提交
-
-
由 Zhichao Cao 提交于
Summary: In status.cc, the assert is `assert(sizeof(msgs) > index)`; msgs is a const char* array, sizeof(msgs) is the array size*char* size, which will make the assert pass all the time. Change it to sizeof(msgs)/sizeof(char*) > index. Pull Request resolved: https://github.com/facebook/rocksdb/pull/7045 Test Plan: pass make check Reviewed By: cheng-chang Differential Revision: D22291337 Pulled By: zhichao-cao fbshipit-source-id: 4ba8ebbb8da80ace7ca6adcdb0c66726f993659d
-
由 mrambacher 提交于
Summary: Cleans up some of the dependencies on test code in the Makefile while building tools: - Moves the test::RandomString, DBBaseTest::RandomString into Random - Moves the test::RandomHumanReadableString into Random - Moves the DestroyDir method into file_utils - Moves the SetupSyncPointsToMockDirectIO into sync_point. - Moves the FaultInjection Env and FS classes under env These changes allow all of the tools to build without dependencies on test_util, thereby simplifying the build dependencies. By moving the FaultInjection code, the dependency in db_stress on different libraries for debug vs release was eliminated. Tested both release and debug builds via Make and CMake for both static and shared libraries. More work remains to clean up how the tools are built and remove some unnecessary dependencies. There is also more work that should be done to get the Makefile and CMake to align in their builds -- what is in the libraries and the sizes of the executables are different. Pull Request resolved: https://github.com/facebook/rocksdb/pull/7097 Reviewed By: riversand963 Differential Revision: D22463160 Pulled By: pdillinger fbshipit-source-id: e19462b53324ab3f0b7c72459dbc73165cc382b2
-
由 Andrew Kryczka 提交于
Summary: This is a followup to https://github.com/facebook/rocksdb/issues/6646. In that PR, for simplicity I just appended a comparison against the 0th restart key in case `BinarySeek()`'s binary search landed at index 0. As a result there were `2/(N+1) + log_2(N)` key comparisons. This PR does it differently. Now we expand the binary search range by one so it also covers the case where target is at or before the restart key at index 0. As a result, it involves `log_2(N+1)` key comparisons. Pull Request resolved: https://github.com/facebook/rocksdb/pull/7068 Test Plan: ran readrandom with mostly default settings and counted key comparisons using `PerfContext`. before: `user_key_comparison_count = 28881965` after: `user_key_comparison_count = 27823245` setup command: ``` $ TEST_TMPDIR=/dev/shm/dbbench ./db_bench -benchmarks=fillrandom,compact -write_buffer_size=1048576 -target_file_size_base=1048576 -max_bytes_for_level_base=4194304 -max_background_jobs=12 -level_compaction_dynamic_level_bytes=true -num=10000000 ``` benchmark command: ``` $ TEST_TMPDIR=/dev/shm/dbbench/ ./db_bench -use_existing_db=true -benchmarks=readrandom -disable_auto_compactions=true -num=10000000 -compression_type=none -reads=1000000 -perf_level=3 ``` Reviewed By: anand1976 Differential Revision: D22357032 Pulled By: ajkr fbshipit-source-id: 8b01e9c1c2a4e9d02fc9dfe16c1cc0327f8bdf24
-
由 Yanqin Jin 提交于
Summary: As title. The goal is to shorten the execution time of several tests when they are combined together in a single TEST_F. Pull Request resolved: https://github.com/facebook/rocksdb/pull/7105 Test Plan: make db_wal_test ./db_wal_test Reviewed By: ltamasi Differential Revision: D22442705 Pulled By: riversand963 fbshipit-source-id: 0ad49b8f21fa86dcd5a4d3c9a06af313735ac217
-
由 Yanqin Jin 提交于
Summary: as title. Pull Request resolved: https://github.com/facebook/rocksdb/pull/7107 Test Plan: make ldb && ./ldb Reviewed By: pdillinger Differential Revision: D22451399 Pulled By: riversand963 fbshipit-source-id: 797645e06473bb9cf139c533877e5161281515e8
-
- 09 7月, 2020 4 次提交
-
-
由 Zitan Chen 提交于
Summary: Although PR https://github.com/facebook/rocksdb/issues/7032 fixes the construction of the `SstFileDumper` in `GetFileDbIdentities` by setting a proper `Env` of the `Options` passed in the constructor, the file path was not corrected accordingly. This actually disables backup engine to use db session ids in the file names since the `db_session_id` is always empty. Now it is fixed by setting the correct path in the construction of `SstFileDumper`. Furthermore, to preserve the Direct IO property that backup engine already has, parameter `EnvOptions` is added to `GetFileDbIdentities` and `SstFileDumper`. The `BackupUsingDirectIO` test is updated accordingly. Pull Request resolved: https://github.com/facebook/rocksdb/pull/7104 Test Plan: backupable_db_test and some manual tests. Reviewed By: ajkr Differential Revision: D22443245 Pulled By: gg814 fbshipit-source-id: 056a9bb8b82947c5e73d7c3fbb62bfe23af5e562
-
由 Akanksha Mahajan 提交于
Update Flush policy in PartitionedIndexBuilder on switching from user-key to internal-key mode (#7096) Summary: When format_version is high enough to support user-key and there are index entries for same user key that spans multiple data blocks then it changes from user-key mode to internal-key mode. But the flush policy is not reset to point to Block Builder of internal-keys. After this switch, no entries are added to user key index partition result, thus it never triggers flushing the block. Fix: 1. After adding the entry in sub_builder_index_, if there is a switch from user-key to internal-key, then flush policy is updated to point to Block Builder of internal-keys index partition. 2. Set sub_builder_index_->seperator_is_key_plus_seq_ = true if seperator_is_key_plus_seq_ is set to true so that subsequent partitions can also use internal key mode. Pull Request resolved: https://github.com/facebook/rocksdb/pull/7096 Test Plan: make check -j64 Reviewed By: ajkr Differential Revision: D22416598 Pulled By: akankshamahajan15 fbshipit-source-id: 01fc2dc07ea1b32f8fb803995ebe6e9a3fbe67ac
-
由 球状闪电 提交于
Summary: WITH_TESTS=OFF and WITH_BENCHMARK_TOOLS=ON there has errors: /bin/ld: cannot find -ltestharness Pull Request resolved: https://github.com/facebook/rocksdb/pull/7040 Reviewed By: riversand963 Differential Revision: D22447637 Pulled By: ajkr fbshipit-source-id: f63058376deb4a2e6722d63541c40caa617c331a
-
由 Peter Dillinger 提交于
Summary: Delicious copy-pasta from https://github.com/facebook/rocksdb/issues/7039 Also fixing DestroyDir to allow files to go missing while it is operating. This seems to fix failures I got with test plan reproducer. Pull Request resolved: https://github.com/facebook/rocksdb/pull/7103 Test Plan: make blackbox_crash_test_with_atomic_flush for a while with checkpoint_one_in=100 Reviewed By: siying Differential Revision: D22435315 Pulled By: pdillinger fbshipit-source-id: 0ec0538402493887aeda43ecc03f32979cb84ced
-