- 28 4月, 2020 1 次提交
-
-
由 Peter Dillinger 提交于
Summary: Some common build variables like USE_CLANG and COMPILE_WITH_UBSAN did not work if specified as make variables, as in `make USE_CLANG=1 check` etc. rather than (in theory less hygienic) `USE_CLANG=1 make check`. This patches Makefile to export some commonly used ones to build_detect_platform so that they work. (I'm skeptical of a broad `export` in Makefile because it's hard to predict how random make variables might affect various invoked tools.) Pull Request resolved: https://github.com/facebook/rocksdb/pull/6740 Test Plan: manual / CI Reviewed By: siying Differential Revision: D21229011 Pulled By: pdillinger fbshipit-source-id: b00c69b23eb2a13105bc8d860ce2d1e61ac5a355
-
- 27 4月, 2020 1 次提交
-
-
由 Yanqin Jin 提交于
Summary: Some recent PRs added new source files or modified TARGETS file manually. During next internal release, executing the following command will revert the manual changes. Update buckifier so that the following command ``` python buckfier/buckify_rocksdb.py ``` does not change TARGETS file. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6726 Test Plan: ``` python buckifier/buckify_rocksdb.py ``` Reviewed By: siying Differential Revision: D21098930 Pulled By: riversand963 fbshipit-source-id: e884f507fefef88163363c9097a460c98f1ed850
-
- 25 4月, 2020 4 次提交
-
-
由 Cheng Chang 提交于
Summary: In crash test, the db directory might be set to /dev/shm or /tmp, in certain environments such as internal testing infrastructure, neither of these directories support direct IO, so direct IO is never enabled in crash test. This PR sets up SyncPoints in direct IO related code paths to disable O_DIRECT flag in calls to `open`, so the direct IO code paths will be executed, all direct IO related assertions will be checked, but no real direct IO request will be issued to the file system. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6727 Test Plan: export CRASH_TEST_EXT_ARGS="--use_direct_reads=1 --mmap_read=0" make -j24 crash_test Reviewed By: zhichao-cao Differential Revision: D21139250 Pulled By: cheng-chang fbshipit-source-id: db9adfe78d91aa4759835b1af91c5db7b27b62ee
-
由 Cheng Chang 提交于
Summary: In https://github.com/facebook/rocksdb/pull/6455, we modified the interface of `RandomAccessFileReader::Read` to be able to get rid of memcpy in direct IO mode. This PR applies the new interface to `BlockFetcher` when reading blocks from SST files in direct IO mode. Without this PR, in direct IO mode, when fetching and uncompressing compressed blocks, `BlockFetcher` will first copy the raw compressed block into `BlockFetcher::compressed_buf_` or `BlockFetcher::stack_buf_` inside `RandomAccessFileReader::Read` depending on the block size. then during uncompressing, it will copy the uncompressed block into `BlockFetcher::heap_buf_`. In this PR, we get rid of the first memcpy and directly uncompress the block from `direct_io_buf_` to `heap_buf_`. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6689 Test Plan: A new unit test `block_fetcher_test` is added. Reviewed By: anand1976 Differential Revision: D21006729 Pulled By: cheng-chang fbshipit-source-id: 2370b92c24075692423b81277415feb2aed5d980
-
由 Cheng Chang 提交于
Summary: In release mode, asserts are not compiled, so `r` is not used, causing compiler warnings. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6750 Test Plan: make check under release mode Reviewed By: anand1976 Differential Revision: D21220365 Pulled By: cheng-chang fbshipit-source-id: fd4afa9843d54af68c4da8660ec61549803e1167
-
由 anand76 提交于
Summary: False alarms are caused by codepaths that intentionally swallow IO errors. Tests: make crash_test Pull Request resolved: https://github.com/facebook/rocksdb/pull/6741 Reviewed By: ltamasi Differential Revision: D21181138 Pulled By: anand1976 fbshipit-source-id: 5ccfbc68eb192033488de6269e59c00f2c65ce00
-
- 24 4月, 2020 4 次提交
-
-
由 Yanqin Jin 提交于
Summary: After a successful recovery, the CURRENT file should be updated to point to the valid MANIFEST. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6746 Test Plan: make check Reviewed By: anand1976 Differential Revision: D21189876 Pulled By: riversand963 fbshipit-source-id: 7537b49988c5c425ebe9505a5cc260de351ad79b
-
由 Cheng Chang 提交于
Summary: Add assertions to check direct IO's alignment requirements in MultiRead. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6739 Test Plan: make check Reviewed By: siying Differential Revision: D21143825 Pulled By: cheng-chang fbshipit-source-id: 26f1623b062a1851080771128feac0669a61f5e9
-
由 Levi Tamasi 提交于
Summary: The patch makes a couple of small cleanups to `SharedBlobFileMetaData` and `BlobFileMetaData`: * It makes the constructors private and introduces factory methods to ensure these objects are always owned by `shared_ptr`s. Note that `SharedBlobFileMetaData` has an additional factory that takes a deleter object; we can utilize this to e.g. notify `VersionSet` when a blob file becomes obsolete (which is exactly when `SharedBlobFileMetaData` is destroyed). * It disables move operations explicitly instead of relying on them being suppressed because of a user-declared destructor. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6749 Test Plan: `make check` Reviewed By: riversand963 Differential Revision: D21206947 Pulled By: ltamasi fbshipit-source-id: 9094c14cc335b3e226f883e5a0df4f87a5cdeb95
-
由 Ibrahim Jarif 提交于
Summary: This PR fixes some typos in code comments. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6733 Reviewed By: siying Differential Revision: D21209037 Pulled By: zhichao-cao fbshipit-source-id: d9274611fab1f5e992998c8c4117b8078c4cbc69
-
- 22 4月, 2020 3 次提交
-
-
由 mrambacher 提交于
Summary: The methods in convenience.h are used to compare/convert objects to/from strings. There is a mishmash of parameters in use here with more needed in the future. This PR replaces those parameters with a single structure. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6389 Reviewed By: siying Differential Revision: D21163707 Pulled By: zhichao-cao fbshipit-source-id: f807b4cc7e2b0af3871536b69546b2604dfa81bd
-
由 anand76 提交于
Summary: Initial implementation of ReadOptions.deadline for MultiGet. If the request takes longer than the deadline, the keys not yet found will be returned with Status::TimedOut(). This implementation enforces the deadline in DBImpl, which is fairly high level. Its best effort and may not check the deadline after every key lookup, but may do so after a batch of keys. In subsequent stages, we will extend this to passing a timeout down to the FileSystem. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6710 Test Plan: Add new unit tests Reviewed By: riversand963 Differential Revision: D21149158 Pulled By: anand1976 fbshipit-source-id: 9f44eecffeb40873f5034ed59a66d21f9f88879e
-
由 Tomas Kolda 提交于
Summary: In highly concurrent requests table cache opens same file more times which lowers purpose of max_open_files. Fixes (https://github.com/facebook/rocksdb/issues/6699) Pull Request resolved: https://github.com/facebook/rocksdb/pull/6707 Reviewed By: ltamasi Differential Revision: D21044965 fbshipit-source-id: f6e91d90b60dad86e518b5147021da42460ee1d2
-
- 21 4月, 2020 5 次提交
-
-
由 Andrew Kryczka 提交于
Summary: When index block is empty or an error happens while reading it, `Invalidate()` is called rather than `Initialize()`. So `Seek()` must not refer to member variables that are only initialized in `Initialize()` until it is sure `Initialize()` has been called. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6736 Reviewed By: siying Differential Revision: D21139641 Pulled By: ajkr fbshipit-source-id: 71c58cc1adbd795dc3729dd5023bf7df1515ff32
-
由 Akanksha Mahajan 提交于
Summary: 1. Add changes so that max_background_flushes can be set dynamically. 2. Add a testcase DBOptionsTest.SetBackgroundFlushThreads which set the max_background_flushes dynamically using SetDBOptions. TestPlan: 1. make -j64 check 2. Using new testcase DBOptionsTest.SetBackgroundFlushThreads Pull Request resolved: https://github.com/facebook/rocksdb/pull/6701 Reviewed By: ajkr Differential Revision: D21028010 Pulled By: akankshamahajan15 fbshipit-source-id: 5f949e4a8fd3c32537b637947b7ee09a69cfc7c1
-
由 Peter Dillinger 提交于
Summary: Based on https://github.com/facebook/rocksdb/issues/6648 (CLA Signed), but heavily modified / extended: * Implicit capture of this via [=] deprecated in C++20, and [=,this] not standard before C++20 -> now using explicit capture lists * Implicit copy operator deprecated in gcc 9 -> add explicit '= default' definition * std::random_shuffle deprecated in C++17 and removed in C++20 -> migrated to a replacement in RocksDB random.h API * Add the ability to build with different std version though -DCMAKE_CXX_STANDARD=11/14/17/20 on the cmake command line * Minimal rebuild flag of MSVC is deprecated and is forbidden with /std:c++latest (C++20) * Added MSVC 2019 C++11 & MSVC 2019 C++20 in AppVeyor * Added GCC 9 C++11 & GCC9 C++20 in Travis Pull Request resolved: https://github.com/facebook/rocksdb/pull/6697 Test Plan: make check and CI Reviewed By: cheng-chang Differential Revision: D21020318 Pulled By: pdillinger fbshipit-source-id: 12311be5dbd8675a0e2c817f7ec50fa11c18ab91
-
由 sdong 提交于
Summary: Recently index_type kBinarySearchWithFirstKey is improved so that the API guarantee is exactly the same as other types and it is ready for wide production. We should cover it in crash tst. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6721 Test Plan: Run crash_test Reviewed By: anand1976 Differential Revision: D21099781 fbshipit-source-id: fda91eba831d9eacbb140c703e9768bb1701f935
-
由 Peter Dillinger 提交于
Summary: Pull Request resolved: https://github.com/facebook/rocksdb/pull/6734 Reviewed By: cheng-chang Differential Revision: D21134556 Pulled By: pdillinger fbshipit-source-id: 3636cc1d1333137b70031f8277458781c21631fb
-
- 18 4月, 2020 2 次提交
-
-
由 Yanqin Jin 提交于
Summary: IsDirectory() is a common API to check whether a path is a regular file or directory. POSIX: call stat() and use S_ISDIR(st_mode) Windows: PathIsDirectoryA() and PathIsDirectoryW() HDFS: FileSystem.IsDirectory() Java: File.IsDirectory() ... Pull Request resolved: https://github.com/facebook/rocksdb/pull/6711 Test Plan: make check Reviewed By: anand1976 Differential Revision: D21053520 Pulled By: riversand963 fbshipit-source-id: 680aadfd8ce982b63689190cf31b3145d5a89e27
-
由 sdong 提交于
Summary: RocksDB behavior is different while max_open_files is small or large. Add the coverage to small max_open_files. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6719 Test Plan: Run crash_test Reviewed By: pdillinger Differential Revision: D21081021 fbshipit-source-id: e3e211761a9bd25d93d19a61c1f7b62d48cf5e3c
-
- 17 4月, 2020 7 次提交
-
-
由 Nicolas Pépin-Perreault 提交于
Summary: This PR exposes the `Iterator::Refresh` method to the Java API by adding it on the `RocksIteratorInterface` interface. There are three concrete implementations: `RocksIterator`, `SstFileReaderIterator`, and `WBWIRocksIterator`. For the first two cases, the JNI side simply delegates to the underlying `Iterator::Refresh` method; in the last case, as it doesn't share an ancestor, and per the discussion in https://github.com/facebook/rocksdb/issues/3465, a `Status::NotSupported` exception is thrown. As the last PR had no activity in a while, I'm opening a new one - I'm completely fine with merging the previous PR if it gets completed before this is reviewed. Let me know if there's anything missing or anything else I can do
👍 Pull Request resolved: https://github.com/facebook/rocksdb/pull/6573 Reviewed By: cheng-chang Differential Revision: D20604666 Pulled By: pdillinger fbshipit-source-id: 4de17df1180c3b87b76cfdd77b674b81fc0563f7 -
由 Adam Retter 提交于
Summary: Adding solid support for multiple architectures was initially triggered by RocksJava users. As such I would like to keep the CI for RocksJava on all architectures, to ensure we don't break backwards compatibility. pdillinger okay let's see how long it takes to complete Travis-CI with this one... Pull Request resolved: https://github.com/facebook/rocksdb/pull/6583 Reviewed By: cheng-chang Differential Revision: D21036718 Pulled By: pdillinger fbshipit-source-id: 97afe0db2e4c575cc0284fdc1d4cc45d5deb2272
-
由 Adam Retter 提交于
Summary: Updates the version of bzip2 used for RocksJava static builds. Please, can we also get this cherry-picked to: 1. 6.7.fb 2. 6.8.fb 3. 6.9.fb Pull Request resolved: https://github.com/facebook/rocksdb/pull/6714 Reviewed By: cheng-chang Differential Revision: D21067233 Pulled By: pdillinger fbshipit-source-id: 8164b7eb99c5ca7b2021ab8c371ba9ded4cb4f7e
-
由 Andrew Kryczka 提交于
Summary: Invariant checking should use internal key comparator rather than `sstableKeyCompare()`. The latter was intended for checking whether a compaction input file's neighboring files need to be included in the same compaction. Using it for invariant checking was leading to false positives for files with overlapping endpoints. Fixes https://github.com/facebook/rocksdb/issues/6647. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6663 Test Plan: regression test Reviewed By: zhichao-cao Differential Revision: D20910466 Pulled By: ajkr fbshipit-source-id: f0b70dad7c4096fce635cab7a36f16e14f74ae3f
-
由 sdong 提交于
Summary: Options.avoid_flush_during_recovery is uncovered in crash_test. Add the coverage with a chance of 1/8, as it is a less frequently used options. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6712 Test Plan: Run crash_test and see the option can be used or not used by chance. Reviewed By: ltamasi Differential Revision: D21056566 fbshipit-source-id: c3b1521517cfc204786e6ef8c6acd7fffda64793
-
由 Yueh-Hsuan Chiang 提交于
Summary: Add env_fault_injection argument to db_stress. When enabled, FaultInjectionTestEnv will be used instead. Currently this option does not support running with other env setting. This will allow us to later manually produce error when running db_crashtest. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6687 Test Plan: make db_stress -j32 ./db_stress --env_fault_injection ./db_stress --env_fault_injection --hdfs // expect error message Reviewed By: ajkr Differential Revision: D21014683 Pulled By: yhchiang fbshipit-source-id: 0724aeac37efd57adb72a37defe6dbd3bfa8106a
-
由 Cheng Chang 提交于
Summary: Compilation fails on systems that do not support O_CLOEXEC. Fix it. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6695 Test Plan: compile without O_CLOEXEC support Reviewed By: anand1976 Differential Revision: D21011850 Pulled By: cheng-chang fbshipit-source-id: f1bf1cce2aa65c7d10b5a9613e941db30e928347
-
- 16 4月, 2020 3 次提交
-
-
由 Mike Kolupaev 提交于
Summary: Context: Index type `kBinarySearchWithFirstKey` added the ability for sst file iterator to sometimes report a key from index without reading the corresponding data block. This is useful when sst blocks are cut at some meaningful boundaries (e.g. one block per key prefix), and many seeks land between blocks (e.g. for each prefix, the ranges of keys in different sst files are nearly disjoint, so a typical seek needs to read a data block from only one file even if all files have the prefix). But this added a new error condition, which rocksdb code was really not equipped to deal with: `InternalIterator::value()` may fail with an IO error or Status::Incomplete, but it's just a method returning a Slice, with no way to report error instead. Before this PR, this type of error wasn't handled at all (an empty slice was returned), and kBinarySearchWithFirstKey implementation was considered a prototype. Now that we (LogDevice) have experimented with kBinarySearchWithFirstKey for a while and confirmed that it's really useful, this PR is adding the missing error handling. It's a pretty inconvenient situation implementation-wise. The error needs to be reported from InternalIterator when trying to access value. But there are ~700 call sites of `InternalIterator::value()`, most of which either can't hit the error condition (because the iterator is reading from memtable or from index or something) or wouldn't benefit from the deferred loading of the value (e.g. compaction iterator that reads all values anyway). Adding error handling to all these call sites would needlessly bloat the code. So instead I made the deferred value loading optional: only the call sites that may use deferred loading have to call the new method `PrepareValue()` before calling `value()`. The feature is enabled with a new bool argument `allow_unprepared_value` to a bunch of methods that create iterators (it wouldn't make sense to put it in ReadOptions because it's completely internal to iterators, with virtually no user-visible effect). Lmk if you have better ideas. Note that the deferred value loading only happens for *internal* iterators. The user-visible iterator (DBIter) always prepares the value before returning from Seek/Next/etc. We could go further and add an API to defer that value loading too, but that's most likely not useful for LogDevice, so it doesn't seem worth the complexity for now. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6621 Test Plan: make -j5 check . Will also deploy to some logdevice test clusters and look at stats. Reviewed By: siying Differential Revision: D20786930 Pulled By: al13n321 fbshipit-source-id: 6da77d918bad3780522e918f17f4d5513d3e99ee
-
由 anand76 提交于
Summary: This was causing db_crashtest.py to wrongly assume an error by parsing the output. Hopefully this will stabilize the crash tests. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6705 Test Plan: make blackbox_crash_test Reviewed By: ltamasi Differential Revision: D21043335 Pulled By: anand1976 fbshipit-source-id: 5cddd112b124d4e2ebd11724a17d4ef0f50c1cf8
-
由 sdong 提交于
Summary: Improve it in two ways: 1. tools/check_format_compatible.sh is not friendly to run outside FB environment. remove the hard-coded http proxy setting. Instead, move it to Legocastle configuration 2. Always disable warning as error, so that older build is more likely to pass. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6702 Test Plan: Run the test and make sure at least it doesn't break. Reviewed By: riversand963 Differential Revision: D21033329 fbshipit-source-id: 88b4ec1ec49547b772790050a165466bdc4a62a0
-
- 15 4月, 2020 2 次提交
-
-
由 anand76 提交于
Summary: 1. Fix a memory leak in FaultInjectionTestFS in the stack trace related code 2. Check status of all MultiGet keys before deciding whether an error was swallowed, instead of assuming an ok status for any key means an undetected error Pull Request resolved: https://github.com/facebook/rocksdb/pull/6700 Test Plan: Run db_stress with asan and fault injection Reviewed By: cheng-chang Differential Revision: D21021498 Pulled By: anand1976 fbshipit-source-id: 489191efd1ab0fa834923a1e1d57253a7a315465
-
由 Cheng Chang 提交于
Summary: The error message is causing problems in the crash tests due to the error parsing logic in db_crashtest.py. This is a follow up PR for https://github.com/facebook/rocksdb/pull/6694. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6696 Test Plan: make check Reviewed By: zhichao-cao Differential Revision: D21021875 Pulled By: cheng-chang fbshipit-source-id: 11e3f536df16941a89949ebcd2147cd8dfa3fbe0
-
- 14 4月, 2020 6 次提交
-
-
由 anand76 提交于
Summary: Log it in the info log to help in troubleshooting. It is logged as follows - ``` 2020/04/10-10:51:39.886662 7ffff7fef340 Options.table_properties_collectors: CompactOnDeletionCollector (Sliding window size = 100 Deletion trigger = 90); ``` Tests: make check Pull Request resolved: https://github.com/facebook/rocksdb/pull/6686 Reviewed By: ltamasi Differential Revision: D21002442 Pulled By: anand1976 fbshipit-source-id: 7adf0dbae7f1febcb00ce61fea5097118ede5c6a
-
由 Zhichao Cao 提交于
Summary: Add NewFileChecksumGenCrc32cFactory to file checksum public interface such that applications can use the build in crc32 checksum factory. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6688 Test Plan: pass make asan_check Reviewed By: riversand963 Differential Revision: D21006859 Pulled By: zhichao-cao fbshipit-source-id: ea8a45196a8b77c310728ab05f6cc0f49f3baef0
-
由 Ziyue Yang 提交于
Summary: BlockBasedTableBuilder::ParallelCompressionRep::first_block can be read in Flush() and written in BGWorkWriteRawBlock() concurrently. This commit fixes the issue by reading first_block out before pushing the block to compression and write. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6640 Test Plan: Run all tests concurrently with TSAN. Reviewed By: cheng-chang Differential Revision: D20851370 fbshipit-source-id: 6f039222e8319d31e15f1b45e05c106527253f72
-
由 anand76 提交于
Summary: The error message is causing problems in the crash tests due to the error parsing logic in db_crashtest.py. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6694 Reviewed By: siying Differential Revision: D20998531 Pulled By: anand1976 fbshipit-source-id: 89cb54a5f5bb664ae6d239c37559f10e14c5ea07
-
由 Andrew Kryczka 提交于
Summary: In index blocks since `format_version=3`, user keys are written rather than internal keys. When reading such blocks, the comparator is obtained via `InternalKeyComparator::user_comparator()`. That function must not return an unwrapped result as the wrapper class provides accounting logic to populate `PerfContext::user_key_comparison_count`. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6650 Test Plan: ran db_bench and verified `PerfContext::user_key_comparison_count` became larger. Reviewed By: cheng-chang Differential Revision: D20866325 Pulled By: ajkr fbshipit-source-id: ad755d46bda31157dacc5b66e532279f19ad538c
-
由 anand76 提交于
Summary: Fix the following issues - 1. Output parsing error in db_crashtest.py 2. Memory leak on exit 3. False alarm on filter block read error Pull Request resolved: https://github.com/facebook/rocksdb/pull/6693 Test Plan: asan_crash Reviewed By: cheng-chang Differential Revision: D20990399 Pulled By: anand1976 fbshipit-source-id: 178ee0dd7c69a4bc5db698379db0dedb29281699
-
- 12 4月, 2020 1 次提交
-
-
由 Yanqin Jin 提交于
Summary: Fix release build caused by variable defined but unused. Test plan (devserver) ``` make release ``` Pull Request resolved: https://github.com/facebook/rocksdb/pull/6690 Reviewed By: cheng-chang Differential Revision: D20980571 Pulled By: riversand963 fbshipit-source-id: c3f3b13f81dce4bdb19876dc2e710d5902ff8a02
-
- 11 4月, 2020 1 次提交
-
-
由 anand76 提交于
Summary: This PR implements a fault injection mechanism for injecting errors in reads in db_stress. The FaultInjectionTestFS is used for this purpose. A thread local structure is used to track the errors, so that each db_stress thread can independently enable/disable error injection and verify observed errors against expected errors. This is initially enabled only for Get and MultiGet, but can be extended to iterator as well once its proven stable. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6538 Test Plan: crash_test make check Reviewed By: riversand963 Differential Revision: D20714347 Pulled By: anand1976 fbshipit-source-id: d7598321d4a2d72bda0ced57411a337a91d87dc7
-