- 04 9月, 2020 2 次提交
-
-
由 Hans Holmberg 提交于
Summary: Delete database instances to make sure there are no loose threads running before exit(). This fixes segfaults seen when running workloads through CompositeEnvs with custom file systems. For further background on the issues arising when using CompositeEnvs, see the discussion in: https://github.com/facebook/rocksdb/pull/6878 Pull Request resolved: https://github.com/facebook/rocksdb/pull/7327 Reviewed By: cheng-chang Differential Revision: D23433244 Pulled By: ajkr fbshipit-source-id: 4e19cf2067e3fe68c2a3fe1823f24b4091336bbe
-
由 Peter Dillinger 提交于
Summary: These new functions and 128-bit value bit operations are expected to be used in a forthcoming Bloom filter alternative. No functional changes to production code, just new code only called by unit tests, cosmetic changes to existing headers, and fix an existing function for a yet-unused template instantiation (BitsSetToOne on something signed and smaller than 32 bits). Pull Request resolved: https://github.com/facebook/rocksdb/pull/7338 Test Plan: Unit tests included. Works with and without TEST_UINT128_COMPAT=1 to check compatibility with and without __uint128_t. Also added that parameter to the CircleCI build build-linux-shared_lib-alt_namespace-status_checked. Reviewed By: jay-zhuang Differential Revision: D23494945 Pulled By: pdillinger fbshipit-source-id: 5c0dc419100d9df5d4d9abb153b2855d5aea39e8
-
- 03 9月, 2020 2 次提交
-
-
由 Daniel Smith 提交于
Summary: This PR is set up to merge into master, but it would be great to get this into a patch release if possible. Pull Request resolved: https://github.com/facebook/rocksdb/pull/7334 Reviewed By: jay-zhuang Differential Revision: D23476624 Pulled By: pdillinger fbshipit-source-id: c6cc02ce06e779e1e174ab0f4748e557d2ce7bc6
-
由 Bingyi Sun 提交于
Summary: Fix typo in comment for SeekForGetImpl(). Rename "bounary" to "boundary" Pull Request resolved: https://github.com/facebook/rocksdb/pull/7328 Reviewed By: riversand963 Differential Revision: D23439748 Pulled By: zhichao-cao fbshipit-source-id: 83a34c417c71a3210ce54a090d76c4d5571313f3
-
- 02 9月, 2020 3 次提交
-
-
由 Andrew Kryczka 提交于
Summary: L0 score is based on size target and number of files. The size target used is `max_bytes_for_level_base`. However, the base level's size can dynamically expand in write burst mode. In fact, it can expand so much that L0->Lbase becomes the highest fanout in target sizes. This doesn't make sense from an efficiency perspective, so this PR bounds the L0->Lbase fanout to the smoothed level multiplier. The L0 scoring based on file count remains unchanged. Pull Request resolved: https://github.com/facebook/rocksdb/pull/7325 Test Plan: contrived benchmark that exhibits the problem: ``` $ TEST_TMPDIR=/data/users/andrewkr/ ./db_bench -benchmarks=filluniquerandom,readrandom -write_buffer_size=1048576 -target_file_size_base=1048576 -max_bytes_for_level_base=4194304 -level0_file_num_compaction_trigger=4 -level_compaction_dynamic_level_bytes=true -compression_type=none -max_background_jobs=12 -rate_limiter_bytes_per_sec=104857600 -benchmark_write_rate_limit=10485760 -num=100000000 ``` Results: - "Burst W-Amp" is the write-amp near the end of the fillrandom benchmark - "Total W-Amp" is the write-amp after readrandom has run a while and all levels no longer need compaction Branch | Burst W-Amp | Total W-Amp | fillrandom (MB/s) -- | -- | -- | -- master | 20.2 | 21.5 | 4.7 dynamic-l0-score | 12.6 | 14.1 | 7.2 Reviewed By: siying Differential Revision: D23412935 Pulled By: ajkr fbshipit-source-id: f91f2067188e432dd39deab02f1c56f195057a0e
-
由 Tomasz Posłuszny 提交于
Summary: Quick fixes to examples to make it easier to get familiar with RocksDB for Windows users: - Set proper temporary directory path on Windows for all examples (with C++17 we should start using std::filesystem) - Fixed typo and got rid of warnings treated as errors in multi_processes_example.cc - Get number of available cores on Windows in c_simple_example.c - Add command to remove DB directory for Windows in compaction_filter_example.cc (print error, but carry on with example upon error, because error code is returned if there is no such directory on Windows) Pull Request resolved: https://github.com/facebook/rocksdb/pull/7304 Reviewed By: jay-zhuang Differential Revision: D23450900 Pulled By: pdillinger fbshipit-source-id: 4256134deb6ae6bb267ed1bd69f814842b95f60f
-
由 Jay Zhuang 提交于
Summary: Recompress blobs if compression type is changed. Pull Request resolved: https://github.com/facebook/rocksdb/pull/7331 Test Plan: `make check` Reviewed By: ltamasi Differential Revision: D23437102 Pulled By: jay-zhuang fbshipit-source-id: bb699ebdad137721d422e42e331d4de8a82a7c5f
-
- 01 9月, 2020 1 次提交
-
-
由 Levi Tamasi 提交于
Summary: The patch adds a log message to `BlobFileBuilder` that is logged upon generating a blob file, similarly to how we log the generation of table files during flush and compaction. The log message contains the column family name, job id, blob file number, and the number and total size of blobs in the new file. Pull Request resolved: https://github.com/facebook/rocksdb/pull/7324 Test Plan: Ran `make check` and checked the actual log messages using a custom `db_bench`. Reviewed By: riversand963 Differential Revision: D23402229 Pulled By: ltamasi fbshipit-source-id: ca42beb4db284b783d1eb2651f321032a45d0c5f
-
- 29 8月, 2020 2 次提交
-
-
由 Akanksha Mahajan 提交于
Summary: Add a unit test case to check memory usage when max_write_buffer_size_to_maintain is set if flushed immutable memtables are trimmed timely or not. Pull Request resolved: https://github.com/facebook/rocksdb/pull/7311 Test Plan: Compared the results with before bug fix. Reviewed By: ltamasi Differential Revision: D23321702 Pulled By: akankshamahajan15 fbshipit-source-id: da04ee21137d641a07fd499a9e2749eb036fcb1e
-
由 Jonny Heggheim 提交于
Summary: Fixes https://github.com/facebook/rocksdb/issues/7185. Standard for GNU and FreeBSD. See https://www.freebsd.org/doc/en/books/porters-handbook/porting-prefix.html https://www.gnu.org/prep/standards/html_node/DESTDIR.html#DESTDIR https://www.gnu.org/prep/standards/html_node/Directory-Variables.html Pull Request resolved: https://github.com/facebook/rocksdb/pull/7187 Reviewed By: cheng-chang Differential Revision: D23333233 Pulled By: ajkr fbshipit-source-id: f704d23852c4516cf5fa00df73ff57687b2ddffb
-
- 28 8月, 2020 3 次提交
-
-
由 Jay Zhuang 提交于
Summary: A new file interface `SupportPrefetch()` is added. When the user overrides it to `false`, an internal prefetch buffer will be used for readahead. Useful for non-directIO but FS doesn't have readahead support. Pull Request resolved: https://github.com/facebook/rocksdb/pull/7312 Reviewed By: anand1976 Differential Revision: D23329847 Pulled By: jay-zhuang fbshipit-source-id: 71cd4ce6f4a820840294e4e6aec111ab76175527
-
由 Levi Tamasi 提交于
Summary: The patch adds a class called `BlobFileBuilder` that can be used to build and cut blob files in background jobs (flushes/compactions). The class enforces a value size threshold (`min_blob_size`; smaller blobs will be inlined in the LSM tree itself), and supports specifying a blob file size limit (`blob_file_size`), as well as compression (`blob_compression_type`) and checksums for blob files. It also keeps track of the generated blob files and their associated `BlobFileAddition` metadata, which can be applied as part of the background job's `VersionEdit`. Pull Request resolved: https://github.com/facebook/rocksdb/pull/7306 Test Plan: `make check` Reviewed By: riversand963 Differential Revision: D23298817 Pulled By: ltamasi fbshipit-source-id: 38f35d81dab1ba81f15236240612ec173d7f21b5
-
由 Akanksha Mahajan 提交于
Summary: Replace FSRandomAccessFile pointer with FSRandomAccessFilePtr object in RandomAccessFileReader. This new object wraps FSRandomAccessFile pointer. Objective: If tracing is enabled, FSRandomAccessFile Ptr returns FSRandomAccessFileTracingWrapper pointer that includes all necessary information in IORecord and calls underlying FileSystem and invokes IOTracer to dump that record in a binary file. If tracing is disabled then, underlying FileSystem pointer is returned directly. FSRandomAccessFilePtr wrapper class is added to bypass the FSRandomAccessFileWrapper when tracing is disabled. Test Plan: make check -j64 Pull Request resolved: https://github.com/facebook/rocksdb/pull/7192 Reviewed By: anand1976 Differential Revision: D23356867 Pulled By: akankshamahajan15 fbshipit-source-id: 48f31168166a17a7444b40be44a9a9d4a5c7182c
-
- 27 8月, 2020 1 次提交
-
-
由 Peter Dillinger 提交于
Summary: This is a "real" fix for the issue worked around in https://github.com/facebook/rocksdb/issues/7294. To get DB checksum info for live files, we now read the manifest file that will become part of the checkpoint/backup. This requires a little extra handling in taking a custom checkpoint, including only reading the manifest file up to the size prescribed by the checkpoint. This moves GetFileChecksumsFromManifest from backup code to file_checksum_helper.{h,cc} and removes apparently unnecessary checking related to column families. Updated HISTORY.md and warned potential future users of DB::GetLiveFilesChecksumInfo() Pull Request resolved: https://github.com/facebook/rocksdb/pull/7309 Test Plan: updated unit test, before and after Reviewed By: ajkr Differential Revision: D23311994 Pulled By: pdillinger fbshipit-source-id: 741e30a2dc1830e8208f7648fcc8c5f000d4e2d5
-
- 26 8月, 2020 3 次提交
-
-
由 sdong 提交于
Summary: Right now all I/O failures under PartitionIndexReader::CacheDependencies() is swallowed. This doesn't impact correctness but we've made a decision that any I/O error in read path now should be returned to users for awareness. Return errors in those cases instead. Pull Request resolved: https://github.com/facebook/rocksdb/pull/7297 Test Plan: Add a new unit test that ingest errors in this code path and see Get() fails. Only one I/O path is hit in PartitionIndexReader::CacheDependencies(). Several option changes are attempt but not able to got other pread paths triggered. Not sure whether other failure cases would be even possible. Would rely on continuous stress test to validate it. Reviewed By: anand1976 Differential Revision: D23257950 fbshipit-source-id: 859dbc92fa239996e1bb378329344d3d54168c03
-
由 sdong 提交于
Summary: DBBasicTest.CompactBetweenSnapshots can time-out in some slow-I/O hosts. Parameterize it so that single test runs shorter. Pull Request resolved: https://github.com/facebook/rocksdb/pull/7301 Test Plan: Run the test and see see different runs are of different configerations in a hacky way. Reviewed By: ltamasi Differential Revision: D23277733 fbshipit-source-id: 1f717b4131322d175abf9e211131fe7e9b1ef758
-
由 Zhichao Cao 提交于
Summary: When SST file is created, application is able to know the file information through OnTableFileCreated callback in LogAndNotifyTableFileCreationFinished. Since file checksum information can be useful for application when the SST file is created, we add file_checksum and file_checksum_func_name information to TableFileCreationInfo, which will be passed through OnTableFileCreated. Pull Request resolved: https://github.com/facebook/rocksdb/pull/7108 Test Plan: make check, listener_test. Reviewed By: ajkr Differential Revision: D22470240 Pulled By: zhichao-cao fbshipit-source-id: 92c20344d9b986eadfe3480f3769bf4add0dbaae
-
- 25 8月, 2020 6 次提交
-
-
由 Connor1996 提交于
Summary: After releasing a snapshot, it checks whether it is suitable to trigger bottom compactions. When disabling auto compactions, it may still schedule compaction when releasing a snapshot. Whereas no compaction job will be actually handled, so the state of LSM is not changed and compaction will be triggered again and again every time releasing a snapshot. Too frequent compactions lead to high CPU usage and high db_mutex lock contention which affects foreground write duration finally. Pull Request resolved: https://github.com/facebook/rocksdb/pull/7267 Test Plan: - make check - manual test Reviewed By: akankshamahajan15 Differential Revision: D23252880 Pulled By: ajkr fbshipit-source-id: 4431e071a35d9912a2a3592875db27bae521434b
-
由 mrambacher 提交于
Summary: More tests now pass. When in doubt, I added a TODO comment to check what should happen with an ignored error. Pull Request resolved: https://github.com/facebook/rocksdb/pull/7305 Reviewed By: akankshamahajan15 Differential Revision: D23301262 Pulled By: ajkr fbshipit-source-id: 5f120edc7393560aefc0633250277bbc7e8de9e6
-
由 rockeet 提交于
Summary: fix DBWithTTL::Open() param ttls: vector<int32_t> to const vector<int32_t> Pull Request resolved: https://github.com/facebook/rocksdb/pull/7196 Reviewed By: akankshamahajan15 Differential Revision: D23277772 Pulled By: ajkr fbshipit-source-id: bf69834b5c2062c7e166dab21fbfd40416c7872d
-
由 sdong 提交于
Summary: SeqAdvanceConcurrentTest sometimes runs too long on some platforms. Disable fsync to speed it up. Pull Request resolved: https://github.com/facebook/rocksdb/pull/7302 Test Plan: Run the tests and watch CI. Reviewed By: ajkr Differential Revision: D23298192 fbshipit-source-id: 2185eed4e0958c3de5e8a3f94ceed5be5945ed37
-
由 sdong 提交于
Summary: Some ExternalSSTFileTest runs very long on some places. Disable fsync in some tests to speed them up. Pull Request resolved: https://github.com/facebook/rocksdb/pull/7303 Test Plan: Run these tests. Reviewed By: riversand963 Differential Revision: D23280261 fbshipit-source-id: 0dca862e462f9e6d807f393320a1f82aa5b87e59
-
由 Akanksha Mahajan 提交于
Summary: RocksDb regression commands are exiting with error /usr/bin/ar: creating librocksdb.a /usr/bin/ld: ./cache/cache.o: relocation R_X86_64_32 against `.rodata.str1.1' can not be used when making a shared object; recompile with -fPIC Bug: It tries to link the static code into a shared lib. Fix: Added make clean before building shared_lib Pull Request resolved: https://github.com/facebook/rocksdb/pull/7300 Test Plan: make clean make -j$(nproc) static_lib make -j$(nproc) shared_lib Reviewed By: pdillinger Differential Revision: D23276842 Pulled By: akankshamahajan15 fbshipit-source-id: c2e69fa505893ad414786794fc486f3f22f059d5
-
- 22 8月, 2020 3 次提交
-
-
由 Jay Zhuang 提交于
Summary: Make sure deleting a running timer works fine. Pull Request resolved: https://github.com/facebook/rocksdb/pull/7292 Test Plan: unittest and an invalid benchmark command: `./db_bench --db=/tmp --use_existing_db=false --benchmarks=fred --compression_type=none` Reviewed By: riversand963 Differential Revision: D23248500 Pulled By: jay-zhuang fbshipit-source-id: 04111681b389a9aa23a439db4568d5ca351f1144
-
由 Akanksha Mahajan 提交于
Summary: When a memtable is trimmed in MemTableListVersion, the memtable is only added to delete list if it is the last reference. However it is not the last reference as it is held by the super version. But the super version would not be switched if the delete list is empty. So the memtable is never destroyed and memory usage increases beyond write_buffer_size + max_write_buffer_size_to_maintain. Pull Request resolved: https://github.com/facebook/rocksdb/pull/7296 Test Plan: 1. ./db_bench -benchmarks=randomtransaction -optimistic_transaction_db=1 -statistics -stats_interval_seconds=1 -duration=90 -num=500000 --max_write_buffer_size_to_maintain=16000000 --transaction_set_snapshot Reviewed By: ltamasi Differential Revision: D23267395 Pulled By: akankshamahajan15 fbshipit-source-id: 3a8d437fe9f4015f851ff84c0e29528aa946b650
-
由 Jay Zhuang 提交于
Summary: And change the internal time value from seconds to microseconds. Pull Request resolved: https://github.com/facebook/rocksdb/pull/7293 Reviewed By: pdillinger Differential Revision: D23253751 Pulled By: jay-zhuang fbshipit-source-id: 36aa9376b8801b85bd10163173590a17cf4f3a3a
-
- 21 8月, 2020 5 次提交
-
-
由 Peter Dillinger 提交于
Summary: On a read-write DB configured with DBOptions::file_checksum_gen_factory, BackupEngine::CreateNewBackup can fail intermittently, with non-OK status. This is due to a race between GetLiveFiles and GetLiveFilesChecksumInfo in creating backups. For patching 6.12 release (as this commit is intended for, except this is a forward-merged version), we can simply treat files for which we falsely failed to get checksum info as legacy files lacking checksum info. Pull Request resolved: https://github.com/facebook/rocksdb/pull/7294 Test Plan: unit test reproducer included Reviewed By: ajkr Differential Revision: D23253489 Pulled By: pdillinger fbshipit-source-id: 9e4945dad120b776ad3e753be10b962f61f28e14
-
由 mrambacher 提交于
Summary: This test uses database functionality and required more extensive work to get it to pass than the other tests. The DB functionality required for this test now passes the check. When it was unclear what the proper behavior was for unchecked status codes, a TODO was added. Pull Request resolved: https://github.com/facebook/rocksdb/pull/7283 Reviewed By: akankshamahajan15 Differential Revision: D23251497 Pulled By: ajkr fbshipit-source-id: 52b79629bdafa0a58de8ead1d1d66f141b331523
-
由 Stanislav Tkach 提交于
Summary: Pull Request resolved: https://github.com/facebook/rocksdb/pull/7289 Reviewed By: akankshamahajan15 Differential Revision: D23252520 Pulled By: ajkr fbshipit-source-id: 85cea485a6dcaa1c67c32a83eb49a1b623966609
-
由 Cheng Chang 提交于
Summary: The updates resolve comments left from https://github.com/facebook/rocksdb/pull/7164. Pull Request resolved: https://github.com/facebook/rocksdb/pull/7282 Test Plan: wal_edit_test Reviewed By: ltamasi Differential Revision: D23196824 Pulled By: cheng-chang fbshipit-source-id: 797f3fef27fc72114c2be777d9eadd3429da5301
-
由 Peter Dillinger 提交于
Summary: A recent build continued with confusing results after failing to "snap install cmake" so ensure failures in installs are fatal. Also upgrade snapd before "snap install" to hopefully avoid this error: error: cannot perform the following tasks: - Mount snap "cmake" (513) (snap "cmake" assumes unsupported features: command-chain (try to update snapd and refresh the core snap)) Pull Request resolved: https://github.com/facebook/rocksdb/pull/7239 Test Plan: watch Travis Reviewed By: akankshamahajan15 Differential Revision: D23244110 Pulled By: pdillinger fbshipit-source-id: 33dbf145f6999d0b90576cdfde484f15c5d1ac19
-
- 20 8月, 2020 3 次提交
-
-
由 Jay Zhuang 提交于
Summary: There's a potential deadlock caused by MockTimeEnv time value get to a large number, which causes TimedWait() wait forever. The test misuses the microseconds as seconds, making it more likely to happen. Pull Request resolved: https://github.com/facebook/rocksdb/pull/7277 Reviewed By: pdillinger Differential Revision: D23183873 Pulled By: jay-zhuang fbshipit-source-id: 6fc38ebd40b4125a99551204b271f91a27e70086
-
由 Will Angenent 提交于
Summary: Example of a faulty log message: Rate limiting is enabled with penalty 18203625after deleting file ... Pull Request resolved: https://github.com/facebook/rocksdb/pull/7286 Reviewed By: David132639 Differential Revision: D23215981 Pulled By: freewilll fbshipit-source-id: 8bdbbffea9f2942cc7a652f315a560d61c0f1068
-
由 Jay Zhuang 提交于
Summary: Seems it's only causing assert failure during compaction pick, but in production code, the problematic compactions are excluded at a later step. Pull Request resolved: https://github.com/facebook/rocksdb/pull/7281 Reviewed By: akankshamahajan15 Differential Revision: D23228000 Pulled By: jay-zhuang fbshipit-source-id: 2e4055aeebe0f5a2b07e299e0a2d51a1ad2e216d
-
- 19 8月, 2020 3 次提交
-
-
由 Levi Tamasi 提交于
Summary: Pull Request resolved: https://github.com/facebook/rocksdb/pull/7280 Test Plan: `make check` Reviewed By: riversand963 Differential Revision: D23195192 Pulled By: ltamasi fbshipit-source-id: 743b382de391963e62ba86119e9fbd0233ea3b3a
-
由 Akanksha Mahajan 提交于
Summary: This diff contains following changes: 1. Replace `FSSequentialFile` pointer with `FSSequentialFilePtr` object that wraps `FSSequentialFile` pointer in `SequenceFileReader`. Objective: If tracing is enabled, `FSSequentialFilePtr` returns `FSSequentialFileTracingWrapper` pointer that includes all necessary information in `IORecord` and calls underlying FileSystem and invokes `IOTracer` to dump that record in a binary file. If tracing is disabled then, underlying `FileSystem` pointer is returned directly. `FSSequentialFilePtr` wrapper class is added to bypass the `FSSequentialFileTracingWrapper` when tracing is disabled. Pull Request resolved: https://github.com/facebook/rocksdb/pull/7190 Test Plan: make check -j64 COMPILE_WITH_TSAN=1 make check -j64 Reviewed By: anand1976 Differential Revision: D23059616 Pulled By: akankshamahajan15 fbshipit-source-id: 1564b94dd1297cd0fbfe2ed5c9cc3e20f7395301
-
由 Andrew Kryczka 提交于
Summary: - Made it clear only one record in the tail is allowed to have a problem - Added detail about the valid use case instead of calling it legacy behavior Pull Request resolved: https://github.com/facebook/rocksdb/pull/7270 Reviewed By: riversand963 Differential Revision: D23169075 Pulled By: ajkr fbshipit-source-id: 2a4b45aa8641f17efa104523fbad765012a98fb0
-
- 18 8月, 2020 3 次提交
-
-
由 Peter Dillinger 提交于
Summary: Some tests like BackupableDBTest.FileCollision and ShareTableFilesWithChecksumsNewNaming are intermittently failing, probably due to unpredictable flushing with FillDB. This change should fix the failures seen and help to prevent similar flakiness in future tests in the file. Pull Request resolved: https://github.com/facebook/rocksdb/pull/7273 Test Plan: make check, and with valgrind Reviewed By: siying Differential Revision: D23176947 Pulled By: pdillinger fbshipit-source-id: 654b73a64db475f2b9b065ed53a889a8b9083c59
-
由 Jay Zhuang 提交于
Summary: `db_bench` is not linked with thirdparty libs in cmake, even `-DWITH_*` is specified. Pull Request resolved: https://github.com/facebook/rocksdb/pull/7264 Test Plan: `$ mkdir build; cd build; cmake .. -DWITH_SNAPPY=1; make db_bench; ./db_bench` `$ cmake .. -DWITH_SNAPPY=1 -DWITH_LZ4; make db_bench; ./db_bench -compression_type=lz4` Reviewed By: zhichao-cao Differential Revision: D23165077 Pulled By: jay-zhuang fbshipit-source-id: 9c6fead31c41664a5c75ecd6469f47402fcb7d62
-
由 sdong 提交于
Summary: After https://github.com/facebook/rocksdb/pull/7036, we still see extra DBTest that can timeout when running 10 or 20 in parallel. Expand skip-fsync mode in whole DBTest. Still preserve other tests from doing this mode to be conservative. This commit reinstates https://github.com/facebook/rocksdb/issues/7049, whose un-revert was lost in an automatic infrastructure mis-merge. Pull Request resolved: https://github.com/facebook/rocksdb/pull/7274 Test Plan: Run all existing files. Reviewed By: pdillinger Differential Revision: D23177444 fbshipit-source-id: 1f61690b2ac6333c3b2c87176fef6b2cba086b33
-