- 27 8月, 2020 1 次提交
-
-
由 Peter Dillinger 提交于
Summary: This is a "real" fix for the issue worked around in https://github.com/facebook/rocksdb/issues/7294. To get DB checksum info for live files, we now read the manifest file that will become part of the checkpoint/backup. This requires a little extra handling in taking a custom checkpoint, including only reading the manifest file up to the size prescribed by the checkpoint. This moves GetFileChecksumsFromManifest from backup code to file_checksum_helper.{h,cc} and removes apparently unnecessary checking related to column families. Updated HISTORY.md and warned potential future users of DB::GetLiveFilesChecksumInfo() Pull Request resolved: https://github.com/facebook/rocksdb/pull/7309 Test Plan: updated unit test, before and after Reviewed By: ajkr Differential Revision: D23311994 Pulled By: pdillinger fbshipit-source-id: 741e30a2dc1830e8208f7648fcc8c5f000d4e2d5
-
- 26 8月, 2020 3 次提交
-
-
由 sdong 提交于
Summary: Right now all I/O failures under PartitionIndexReader::CacheDependencies() is swallowed. This doesn't impact correctness but we've made a decision that any I/O error in read path now should be returned to users for awareness. Return errors in those cases instead. Pull Request resolved: https://github.com/facebook/rocksdb/pull/7297 Test Plan: Add a new unit test that ingest errors in this code path and see Get() fails. Only one I/O path is hit in PartitionIndexReader::CacheDependencies(). Several option changes are attempt but not able to got other pread paths triggered. Not sure whether other failure cases would be even possible. Would rely on continuous stress test to validate it. Reviewed By: anand1976 Differential Revision: D23257950 fbshipit-source-id: 859dbc92fa239996e1bb378329344d3d54168c03
-
由 sdong 提交于
Summary: DBBasicTest.CompactBetweenSnapshots can time-out in some slow-I/O hosts. Parameterize it so that single test runs shorter. Pull Request resolved: https://github.com/facebook/rocksdb/pull/7301 Test Plan: Run the test and see see different runs are of different configerations in a hacky way. Reviewed By: ltamasi Differential Revision: D23277733 fbshipit-source-id: 1f717b4131322d175abf9e211131fe7e9b1ef758
-
由 Zhichao Cao 提交于
Summary: When SST file is created, application is able to know the file information through OnTableFileCreated callback in LogAndNotifyTableFileCreationFinished. Since file checksum information can be useful for application when the SST file is created, we add file_checksum and file_checksum_func_name information to TableFileCreationInfo, which will be passed through OnTableFileCreated. Pull Request resolved: https://github.com/facebook/rocksdb/pull/7108 Test Plan: make check, listener_test. Reviewed By: ajkr Differential Revision: D22470240 Pulled By: zhichao-cao fbshipit-source-id: 92c20344d9b986eadfe3480f3769bf4add0dbaae
-
- 25 8月, 2020 6 次提交
-
-
由 Connor1996 提交于
Summary: After releasing a snapshot, it checks whether it is suitable to trigger bottom compactions. When disabling auto compactions, it may still schedule compaction when releasing a snapshot. Whereas no compaction job will be actually handled, so the state of LSM is not changed and compaction will be triggered again and again every time releasing a snapshot. Too frequent compactions lead to high CPU usage and high db_mutex lock contention which affects foreground write duration finally. Pull Request resolved: https://github.com/facebook/rocksdb/pull/7267 Test Plan: - make check - manual test Reviewed By: akankshamahajan15 Differential Revision: D23252880 Pulled By: ajkr fbshipit-source-id: 4431e071a35d9912a2a3592875db27bae521434b
-
由 mrambacher 提交于
Summary: More tests now pass. When in doubt, I added a TODO comment to check what should happen with an ignored error. Pull Request resolved: https://github.com/facebook/rocksdb/pull/7305 Reviewed By: akankshamahajan15 Differential Revision: D23301262 Pulled By: ajkr fbshipit-source-id: 5f120edc7393560aefc0633250277bbc7e8de9e6
-
由 rockeet 提交于
Summary: fix DBWithTTL::Open() param ttls: vector<int32_t> to const vector<int32_t> Pull Request resolved: https://github.com/facebook/rocksdb/pull/7196 Reviewed By: akankshamahajan15 Differential Revision: D23277772 Pulled By: ajkr fbshipit-source-id: bf69834b5c2062c7e166dab21fbfd40416c7872d
-
由 sdong 提交于
Summary: SeqAdvanceConcurrentTest sometimes runs too long on some platforms. Disable fsync to speed it up. Pull Request resolved: https://github.com/facebook/rocksdb/pull/7302 Test Plan: Run the tests and watch CI. Reviewed By: ajkr Differential Revision: D23298192 fbshipit-source-id: 2185eed4e0958c3de5e8a3f94ceed5be5945ed37
-
由 sdong 提交于
Summary: Some ExternalSSTFileTest runs very long on some places. Disable fsync in some tests to speed them up. Pull Request resolved: https://github.com/facebook/rocksdb/pull/7303 Test Plan: Run these tests. Reviewed By: riversand963 Differential Revision: D23280261 fbshipit-source-id: 0dca862e462f9e6d807f393320a1f82aa5b87e59
-
由 Akanksha Mahajan 提交于
Summary: RocksDb regression commands are exiting with error /usr/bin/ar: creating librocksdb.a /usr/bin/ld: ./cache/cache.o: relocation R_X86_64_32 against `.rodata.str1.1' can not be used when making a shared object; recompile with -fPIC Bug: It tries to link the static code into a shared lib. Fix: Added make clean before building shared_lib Pull Request resolved: https://github.com/facebook/rocksdb/pull/7300 Test Plan: make clean make -j$(nproc) static_lib make -j$(nproc) shared_lib Reviewed By: pdillinger Differential Revision: D23276842 Pulled By: akankshamahajan15 fbshipit-source-id: c2e69fa505893ad414786794fc486f3f22f059d5
-
- 22 8月, 2020 3 次提交
-
-
由 Jay Zhuang 提交于
Summary: Make sure deleting a running timer works fine. Pull Request resolved: https://github.com/facebook/rocksdb/pull/7292 Test Plan: unittest and an invalid benchmark command: `./db_bench --db=/tmp --use_existing_db=false --benchmarks=fred --compression_type=none` Reviewed By: riversand963 Differential Revision: D23248500 Pulled By: jay-zhuang fbshipit-source-id: 04111681b389a9aa23a439db4568d5ca351f1144
-
由 Akanksha Mahajan 提交于
Summary: When a memtable is trimmed in MemTableListVersion, the memtable is only added to delete list if it is the last reference. However it is not the last reference as it is held by the super version. But the super version would not be switched if the delete list is empty. So the memtable is never destroyed and memory usage increases beyond write_buffer_size + max_write_buffer_size_to_maintain. Pull Request resolved: https://github.com/facebook/rocksdb/pull/7296 Test Plan: 1. ./db_bench -benchmarks=randomtransaction -optimistic_transaction_db=1 -statistics -stats_interval_seconds=1 -duration=90 -num=500000 --max_write_buffer_size_to_maintain=16000000 --transaction_set_snapshot Reviewed By: ltamasi Differential Revision: D23267395 Pulled By: akankshamahajan15 fbshipit-source-id: 3a8d437fe9f4015f851ff84c0e29528aa946b650
-
由 Jay Zhuang 提交于
Summary: And change the internal time value from seconds to microseconds. Pull Request resolved: https://github.com/facebook/rocksdb/pull/7293 Reviewed By: pdillinger Differential Revision: D23253751 Pulled By: jay-zhuang fbshipit-source-id: 36aa9376b8801b85bd10163173590a17cf4f3a3a
-
- 21 8月, 2020 5 次提交
-
-
由 Peter Dillinger 提交于
Summary: On a read-write DB configured with DBOptions::file_checksum_gen_factory, BackupEngine::CreateNewBackup can fail intermittently, with non-OK status. This is due to a race between GetLiveFiles and GetLiveFilesChecksumInfo in creating backups. For patching 6.12 release (as this commit is intended for, except this is a forward-merged version), we can simply treat files for which we falsely failed to get checksum info as legacy files lacking checksum info. Pull Request resolved: https://github.com/facebook/rocksdb/pull/7294 Test Plan: unit test reproducer included Reviewed By: ajkr Differential Revision: D23253489 Pulled By: pdillinger fbshipit-source-id: 9e4945dad120b776ad3e753be10b962f61f28e14
-
由 mrambacher 提交于
Summary: This test uses database functionality and required more extensive work to get it to pass than the other tests. The DB functionality required for this test now passes the check. When it was unclear what the proper behavior was for unchecked status codes, a TODO was added. Pull Request resolved: https://github.com/facebook/rocksdb/pull/7283 Reviewed By: akankshamahajan15 Differential Revision: D23251497 Pulled By: ajkr fbshipit-source-id: 52b79629bdafa0a58de8ead1d1d66f141b331523
-
由 Stanislav Tkach 提交于
Summary: Pull Request resolved: https://github.com/facebook/rocksdb/pull/7289 Reviewed By: akankshamahajan15 Differential Revision: D23252520 Pulled By: ajkr fbshipit-source-id: 85cea485a6dcaa1c67c32a83eb49a1b623966609
-
由 Cheng Chang 提交于
Summary: The updates resolve comments left from https://github.com/facebook/rocksdb/pull/7164. Pull Request resolved: https://github.com/facebook/rocksdb/pull/7282 Test Plan: wal_edit_test Reviewed By: ltamasi Differential Revision: D23196824 Pulled By: cheng-chang fbshipit-source-id: 797f3fef27fc72114c2be777d9eadd3429da5301
-
由 Peter Dillinger 提交于
Summary: A recent build continued with confusing results after failing to "snap install cmake" so ensure failures in installs are fatal. Also upgrade snapd before "snap install" to hopefully avoid this error: error: cannot perform the following tasks: - Mount snap "cmake" (513) (snap "cmake" assumes unsupported features: command-chain (try to update snapd and refresh the core snap)) Pull Request resolved: https://github.com/facebook/rocksdb/pull/7239 Test Plan: watch Travis Reviewed By: akankshamahajan15 Differential Revision: D23244110 Pulled By: pdillinger fbshipit-source-id: 33dbf145f6999d0b90576cdfde484f15c5d1ac19
-
- 20 8月, 2020 3 次提交
-
-
由 Jay Zhuang 提交于
Summary: There's a potential deadlock caused by MockTimeEnv time value get to a large number, which causes TimedWait() wait forever. The test misuses the microseconds as seconds, making it more likely to happen. Pull Request resolved: https://github.com/facebook/rocksdb/pull/7277 Reviewed By: pdillinger Differential Revision: D23183873 Pulled By: jay-zhuang fbshipit-source-id: 6fc38ebd40b4125a99551204b271f91a27e70086
-
由 Will Angenent 提交于
Summary: Example of a faulty log message: Rate limiting is enabled with penalty 18203625after deleting file ... Pull Request resolved: https://github.com/facebook/rocksdb/pull/7286 Reviewed By: David132639 Differential Revision: D23215981 Pulled By: freewilll fbshipit-source-id: 8bdbbffea9f2942cc7a652f315a560d61c0f1068
-
由 Jay Zhuang 提交于
Summary: Seems it's only causing assert failure during compaction pick, but in production code, the problematic compactions are excluded at a later step. Pull Request resolved: https://github.com/facebook/rocksdb/pull/7281 Reviewed By: akankshamahajan15 Differential Revision: D23228000 Pulled By: jay-zhuang fbshipit-source-id: 2e4055aeebe0f5a2b07e299e0a2d51a1ad2e216d
-
- 19 8月, 2020 3 次提交
-
-
由 Levi Tamasi 提交于
Summary: Pull Request resolved: https://github.com/facebook/rocksdb/pull/7280 Test Plan: `make check` Reviewed By: riversand963 Differential Revision: D23195192 Pulled By: ltamasi fbshipit-source-id: 743b382de391963e62ba86119e9fbd0233ea3b3a
-
由 Akanksha Mahajan 提交于
Summary: This diff contains following changes: 1. Replace `FSSequentialFile` pointer with `FSSequentialFilePtr` object that wraps `FSSequentialFile` pointer in `SequenceFileReader`. Objective: If tracing is enabled, `FSSequentialFilePtr` returns `FSSequentialFileTracingWrapper` pointer that includes all necessary information in `IORecord` and calls underlying FileSystem and invokes `IOTracer` to dump that record in a binary file. If tracing is disabled then, underlying `FileSystem` pointer is returned directly. `FSSequentialFilePtr` wrapper class is added to bypass the `FSSequentialFileTracingWrapper` when tracing is disabled. Pull Request resolved: https://github.com/facebook/rocksdb/pull/7190 Test Plan: make check -j64 COMPILE_WITH_TSAN=1 make check -j64 Reviewed By: anand1976 Differential Revision: D23059616 Pulled By: akankshamahajan15 fbshipit-source-id: 1564b94dd1297cd0fbfe2ed5c9cc3e20f7395301
-
由 Andrew Kryczka 提交于
Summary: - Made it clear only one record in the tail is allowed to have a problem - Added detail about the valid use case instead of calling it legacy behavior Pull Request resolved: https://github.com/facebook/rocksdb/pull/7270 Reviewed By: riversand963 Differential Revision: D23169075 Pulled By: ajkr fbshipit-source-id: 2a4b45aa8641f17efa104523fbad765012a98fb0
-
- 18 8月, 2020 11 次提交
-
-
由 Peter Dillinger 提交于
Summary: Some tests like BackupableDBTest.FileCollision and ShareTableFilesWithChecksumsNewNaming are intermittently failing, probably due to unpredictable flushing with FillDB. This change should fix the failures seen and help to prevent similar flakiness in future tests in the file. Pull Request resolved: https://github.com/facebook/rocksdb/pull/7273 Test Plan: make check, and with valgrind Reviewed By: siying Differential Revision: D23176947 Pulled By: pdillinger fbshipit-source-id: 654b73a64db475f2b9b065ed53a889a8b9083c59
-
由 Jay Zhuang 提交于
Summary: `db_bench` is not linked with thirdparty libs in cmake, even `-DWITH_*` is specified. Pull Request resolved: https://github.com/facebook/rocksdb/pull/7264 Test Plan: `$ mkdir build; cd build; cmake .. -DWITH_SNAPPY=1; make db_bench; ./db_bench` `$ cmake .. -DWITH_SNAPPY=1 -DWITH_LZ4; make db_bench; ./db_bench -compression_type=lz4` Reviewed By: zhichao-cao Differential Revision: D23165077 Pulled By: jay-zhuang fbshipit-source-id: 9c6fead31c41664a5c75ecd6469f47402fcb7d62
-
由 sdong 提交于
Summary: After https://github.com/facebook/rocksdb/pull/7036, we still see extra DBTest that can timeout when running 10 or 20 in parallel. Expand skip-fsync mode in whole DBTest. Still preserve other tests from doing this mode to be conservative. This commit reinstates https://github.com/facebook/rocksdb/issues/7049, whose un-revert was lost in an automatic infrastructure mis-merge. Pull Request resolved: https://github.com/facebook/rocksdb/pull/7274 Test Plan: Run all existing files. Reviewed By: pdillinger Differential Revision: D23177444 fbshipit-source-id: 1f61690b2ac6333c3b2c87176fef6b2cba086b33
-
由 Andrew Kryczka 提交于
Summary: The two features are naturally incompatible. WAL recycling expects the recovery to succeed upon encountering a corrupt record at the point where new data ends and recycled data remains at the tail. However, `WALRecoveryMode::kTolerateCorruptedTailRecords` must fail upon encountering any such corrupt record, as it cannot differentiate between this and a real corruption, which would cause committed updates to be truncated. Pull Request resolved: https://github.com/facebook/rocksdb/pull/7271 Reviewed By: riversand963 Differential Revision: D23169923 Pulled By: ajkr fbshipit-source-id: 2cf8a3bcd2c9a0ecb0055a84725047a10fd4db50
-
由 Yanqin Jin 提交于
Summary: Add `kEntryDeleteWithTimestamp` to `EntryType` which is a public API. Pull Request resolved: https://github.com/facebook/rocksdb/pull/7195 Test Plan: make check Reviewed By: ajkr Differential Revision: D22914704 Pulled By: riversand963 fbshipit-source-id: 886f73c6b70c527cad1c8fc9fc8d3afe60e1ea39
-
由 Levi Tamasi 提交于
Summary: The patch makes sure that the functionality required for the new integrated BlobDB implementation (most importantly, the classes related to reading and writing blob files) is also built in LITE mode by removing the corresponding `#ifndef`s. Pull Request resolved: https://github.com/facebook/rocksdb/pull/7272 Test Plan: Ran `make check` in both regular and LITE mode. Reviewed By: zhichao-cao Differential Revision: D23173280 Pulled By: ltamasi fbshipit-source-id: 1596bd1a76409a8a6d83d8f1dbfe08bfdea7ffe6
-
由 sdong 提交于
Summary: There is potential data race related CompactRange() with level refitting. After the compaction step and refitting step, some automatic compaction could put data to the destination level and cause the DB to be corrupted. Fix the bug by checking the target level to be empty. Pull Request resolved: https://github.com/facebook/rocksdb/pull/7261 Test Plan: Add a unit test, which would fail with "Corruption: L1 have overlapping ranges '666F6F' seq:6, type:1 vs. '626172' seq:2, type:1", and now it succeeds. Reviewed By: ajkr Differential Revision: D23142269 fbshipit-source-id: 28bc14d5ac934c192260b23a4ce3f10a95e3ee91
-
由 Zitan Chen 提交于
Summary: The param tests did not take any effect previously. This PR re-enables it. Pull Request resolved: https://github.com/facebook/rocksdb/pull/7260 Test Plan: Some manual tests and `./backupable_db_test`. Reviewed By: siying Differential Revision: D23140902 Pulled By: pdillinger fbshipit-source-id: cd62b11b926affed25127d9074fa97a1c7f748c4
-
由 matthewvon 提交于
Summary: Looks like somebody simply missed initializing a member variable. The column family ID, cf_id, is not set during OnCompactionBegin. But it is set properly in the next function for OnCompactionCompleted. Need this cf_id for tracking progress of a Stardog optimize since there may be multiple compactions required for a given column family. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6938 Reviewed By: siying Differential Revision: D23153235 Pulled By: ajkr fbshipit-source-id: 932938de3a4ebbc7ac89702f655583862587d251
-
由 Hans Holmberg 提交于
Summary: This pull request adds the parameter --fs_uri to db_bench and db_stress, creating a composite env combining the default env with a specified registered rocksdb file system. This makes it easier to develop and test new RocksDB FileSystems. The pull request also registers the posix file system for testing purposes. Examples: ``` $./db_bench --fs_uri=posix:// --benchmarks=fillseq $./db_stress --fs_uri=zenfs://nullb1 ``` zenfs is a RocksDB FileSystem I'm developing to add support for zoned block devices, and in that case the zoned block device is specified in the uri (a zoned null block device in the above example). Pull Request resolved: https://github.com/facebook/rocksdb/pull/6878 Reviewed By: siying Differential Revision: D23023063 Pulled By: ajkr fbshipit-source-id: 8b3fe7193ce45e683043b021779b7a4d547af247
-
由 John Goerzen 提交于
Summary: pkg-config files are quite useful for communicating to users of a library how to compile against them. This commit generates and installs a pkg-config file that can be used for both static and dynamic builds against the RocksDB library. This should make life easier for developers of client programs, language bindings, etc. Example usage: ``` g++ `pkg-config --cflags rocksdb` -o simple_example simple_example.cc `pkg-config --libs rocksdb` g++ `pkg-config --cflags --static rocksdb` -static \ -o simple_example simple_example.cc `pkg-config --libs --static rocksdb` ``` The commit also adds the generated file to .gitignore, to the uninstall target, and to clean. No additional dependencies are added to RocksDB itself, and this does not make RocksDB use pkg-config as part of its build process. Resolves https://github.com/facebook/rocksdb/issues/4452 Pull Request resolved: https://github.com/facebook/rocksdb/pull/7244 Reviewed By: siying Differential Revision: D23146153 Pulled By: ajkr fbshipit-source-id: 3045aa650d68bd5ac42d40ed709570e9584ef004
-
- 15 8月, 2020 5 次提交
-
-
由 Jay Zhuang 提交于
Summary: Have a global StatsDumpScheduler for all DB instance stats dumping, including `DumpStats()` and `PersistStats()`. Before this, there're 2 dedicate threads for every DB instance, one for DumpStats() one for PersistStats(), which could create lots of threads if there're hundreds DB instances. Pull Request resolved: https://github.com/facebook/rocksdb/pull/7223 Reviewed By: riversand963 Differential Revision: D23056737 Pulled By: jay-zhuang fbshipit-source-id: 0faa2311142a73433ebb3317361db7cbf43faeba
-
由 Yanqin Jin 提交于
Summary: If user-defined timestamp is enabled, current implementation can expose newer data to queries even if an older sequence number is specified via read_options.snapshot. This PR makes Get() respect sequence-number-based snapshot. Solution is simple. Besides using <ukey, ts, seq> to search the index for the key, we also verify that the candidate result's seq is smaller than or equal to seq. This requires passing a seq via `GetContext`, which results in the majority of code change caused by this PR. Also added a few unit tests to demonstrate standard visibility during point lookup and range scan when timestamp and snapshot are both present. Test plan (devserver): ``` make check $./db_bench --benchmarks=fillseq,readrandom -cache_size=$[64*1024*1024] ``` Result this PR: readrandom : 4.827 micros/op 207180 ops/sec; 22.9 MB/s (1000000 of 1000000 found) master: readrandom : 4.936 micros/op 202610 ops/sec; 22.4 MB/s (1000000 of 1000000 found) Pull Request resolved: https://github.com/facebook/rocksdb/pull/7227 Reviewed By: ltamasi Differential Revision: D23015242 Pulled By: riversand963 fbshipit-source-id: ea7b85a728654553ba357d2e6a207b5e40f7376a
-
由 Daniel Smith 提交于
Summary: Pull Request resolved: https://github.com/facebook/rocksdb/pull/7248 Reviewed By: ajkr Differential Revision: D23108368 Pulled By: jay-zhuang fbshipit-source-id: 41765334b0bae26bd74a4d9250b9b98ea7ac6c3f
-
由 Zitan Chen 提交于
Summary: The flaky test in the title is caused by two problems. First, there is a bug in the BackupEngine that results in skipping computing the default crc32 checksum when `share_table_files` is enabled and the table is already backed up. Second, when `RestoreDBFromBackup` fails and the backup was being restored to the DB directory, it is likely that `RestoreDBFromBackup` has cleaned up the DB directory before it fails, and therefore, files in old backups may collide with files to be backed up if `share_files_with_checksum` is not enabled. New tests that cover the above problems are added. Pull Request resolved: https://github.com/facebook/rocksdb/pull/7254 Test Plan: `./backupable_db_test` Reviewed By: pdillinger Differential Revision: D23118715 Pulled By: gg814 fbshipit-source-id: 7be8de912808944be59e93d602c7431a54c079eb
-
由 Andrew Kryczka 提交于
Summary: Manual compaction with `CompactRangeOptions::change_levels` set could refit to a level targeted by another manual compaction. If force_consistency_checks were disabled, it could be possible for overlapping files to be written at that target level. This PR prevents the possibility by calling `DisableManualCompaction()` prior to `ReFitLevel()`. It also improves the manual compaction disabling mechanism to wait for pending manual compactions to complete before returning, and support disabling from multiple threads. Fixes https://github.com/facebook/rocksdb/issues/6432. Pull Request resolved: https://github.com/facebook/rocksdb/pull/7250 Test Plan: crash test command that repro'd the bug reliably: ``` $ TEST_TMPDIR=/dev/shm python tools/db_crashtest.py blackbox --simple -target_file_size_base=524288 -write_buffer_size=1048576 -clear_column_family_one_in=0 -reopen=0 -max_key=10000000 -column_families=1 -max_background_compactions=8 -compact_range_one_in=100000 -compression_type=none -compaction_style=1 -num_levels=5 -universal_min_merge_width=4 -universal_max_merge_width=8 -level0_file_num_compaction_trigger=12 -rate_limiter_bytes_per_sec=1048576000 -universal_max_size_amplification_percent=100 --duration=3600 --interval=60 --use_direct_io_for_flush_and_compaction=0 --use_direct_reads=0 --enable_compaction_filter=0 ``` Reviewed By: ltamasi Differential Revision: D23090800 Pulled By: ajkr fbshipit-source-id: afcbcd51b42ce76789fdb907d8b9ada790709c13
-