- 20 2月, 2020 1 次提交
-
-
由 Remington Brasga 提交于
Summary: TB = 1024 * GB Pull Request resolved: https://github.com/facebook/rocksdb/pull/6434 Differential Revision: D19978339 Pulled By: zhichao-cao fbshipit-source-id: 5a89890110b23f0ebda4a95223f66da6736321ac
-
- 19 2月, 2020 3 次提交
-
-
由 Yanqin Jin 提交于
Summary: Pull Request resolved: https://github.com/facebook/rocksdb/pull/6428 Differential Revision: D19959176 Pulled By: riversand963 fbshipit-source-id: 5b863944663462b246bb82883fa683f75ab33fc1
-
由 Andrew Kryczka 提交于
Summary: We were removing the file from `log_recycle_files_` before renaming it with `ReuseWritableFile()`. Since `ReuseWritableFile()` occurs outside the DB mutex, it was possible for a concurrent full purge to sneak in and delete the file before it could be renamed. Consequently, `SwitchMemtable()` would fail and the DB would enter read-only mode. The fix is to hold the old file number in `log_recycle_files_` until after the file has been renamed. Full purge uses that list to decide which files to keep, so it can no longer delete a file pending recycling. Pull Request resolved: https://github.com/facebook/rocksdb/pull/5900 Test Plan: new unit test Differential Revision: D19771719 Pulled By: ajkr fbshipit-source-id: 094346349ca3fb499712e62de03905acc30b5ce8
-
由 Andrew Kryczka 提交于
Summary: As discovered in https://github.com/facebook/rocksdb/issues/5260 and https://github.com/facebook/rocksdb/issues/5392, reads on the indexed batch do not account for range tombstones. So, return `Status::NotSupported` from `WriteBatchWithIndex::DeleteRange` until we properly support it. Pull Request resolved: https://github.com/facebook/rocksdb/pull/5393 Test Plan: added unit test Differential Revision: D19912360 Pulled By: ajkr fbshipit-source-id: 0bbfc978ea015d64516ca708fce2429abba524cb
-
- 15 2月, 2020 4 次提交
-
-
由 acelyc111 提交于
Summary: Pull Request resolved: https://github.com/facebook/rocksdb/pull/6412 Differential Revision: D19914063 Pulled By: ajkr fbshipit-source-id: 4e401e665d4b449d24c4cdec35a4585eeda95996
-
由 Manuel Ung 提交于
Summary: For write unprepared, some applications may bypass the transaction api, and write keys directly into the write batch. However, since they are not tracked, rollbacks (both for savepoint and transaction) are not aware that these keys have to be rolled back. The fix is to track them in `WriteUnpreparedTxn::untracked_keys_`. This is populated whenever we flush unprepared batches into the DB. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6404 Differential Revision: D19842023 Pulled By: lth fbshipit-source-id: a9edfc643d5c905fc89da9a9a9094d30c9b70108
-
由 Cheng Chang 提交于
Summary: `index` can be replaced by `iter`, saving the computation of `index++`. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6406 Test Plan: make check Differential Revision: D19905056 Pulled By: cheng-chang fbshipit-source-id: add4638959c0d2e4e77a11f3fa04ffabaf0de790
-
由 Cheng Chang 提交于
Summary: Previously, when recovering version set, LoadTableHandlers failures are ignored. If paranoid_checks is true, this failure should not be ignored, otherwise, the opened db might be in an inconsistent state. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6368 Test Plan: make check Differential Revision: D19713459 Pulled By: cheng-chang fbshipit-source-id: 68cb94f4f2cc43f8b024b14755193cd45cfcad55
-
- 14 2月, 2020 9 次提交
-
-
由 wolfkdy 提交于
Summary: I create a new branch from the branch new upsteram/master and "git merge --squash". Maybe it will fix everything. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6420 Differential Revision: D19897152 Pulled By: zhichao-cao fbshipit-source-id: 6575d9e3b23e360f42ee1480b43028b5fcc20136
-
由 Manuel Ung 提交于
Summary: During recovery, multiple (un)prepared batches could exist in the same WAL record due to group commit. This breaks an assertion in `MemTableInserter::MarkBeginPrepare`. To fix, reset unprepared_batch_ to false after `MarkEndPrepare`. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6419 Differential Revision: D19896148 Pulled By: lth fbshipit-source-id: b1a32ef88f775a0881264a18bd1a4a5b8c85eee3
-
由 Manuel Ung 提交于
Summary: Unfortunately, it seems like mysqld reuses xids across machine restarts. When that happens, we could have something like the following happening: ``` BEGIN_PREPARE(unprepared) Put(a) END_PREPARE(xid = 1) -- crash and recover with Put(a) rolled back as it was not prepared BEGIN_PREPARE(prepared) Put(b) END_PREPARE(xid = 1) COMMIT(xid = 1) -- crash and recover with both a, b ``` To solve this, we will have to log the rollback batch into the WAL during recovery. WritePrepared already logs the rollback batch into the WAL, if a rollback happens after prepare, so there is no problem there. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6418 Differential Revision: D19896151 Pulled By: lth fbshipit-source-id: 2ff65ddc5fe75efd57736fed4b7cd7a109d26609
-
由 sdong 提交于
Summary: A recent fix related to 2pc https://github.com/facebook/rocksdb/pull/6313/ writes something to WAL, but does not flush or sync. This causes assertion failure "impl->TEST_WALBufferIsEmpty()" if manual_wal_flush = true. We should fsync the entry to make sure a second power reset can recover. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6417 Test Plan: Add manual_wal_flush=true case in TransactionTest.DoubleCrashInRecovery and fix a bug in the test so that the bug can be reproduced. It passes with the fix. Differential Revision: D19894537 fbshipit-source-id: f1e84e49e2269f583c6019743118292cd8b6598e
-
由 Cheng Chang 提交于
Summary: The DecreaseNumBgThreads test keeps failing on Windows in AppVeyor. It fails because it depends on a timed wait for the tasks to be dequeued from the threadpool's internal queue, but within the specified time, the task might have not been scheduled onto the newly created threads. https://github.com/facebook/rocksdb/pull/6232 tries to fix this by waiting for longer time to let the threads scheduled. This PR tries to fix this by replacing the timed wait with a synchronization on the task's internal conditional variable. When the number of threads increases, instead of guessing the time needed for the task to be scheduled, it directly blocks on the conditional variable until the task starts running. But when thread number is reduced, it still does a timed wait, but this does not lead to the flakiness now, will try to remove these timed waits in a future PR. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6393 Test Plan: Wait to see whether AppVeyor tests pass. Differential Revision: D19890928 Pulled By: cheng-chang fbshipit-source-id: 4e56e4addf625c98c0876e62d9d57a6f0a156f76
-
由 sdong 提交于
Summary: Since IO Uring feature is not stable. Remove it from buck configuration. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6415 Test Plan: See internal build pass Differential Revision: D19892988 fbshipit-source-id: 7fc01efc2af5ed707fb8e4e4674223aeb83cd5ea
-
由 Huisheng Liu 提交于
Summary: It's observed on Windows DestroyDB failed to remove the log file because the logger is still alive in sst file manager and holding a handle to the log file. This fix makes sure the logger is released before attempt to clear the database directory. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6308 Differential Revision: D19818829 Pulled By: riversand963 fbshipit-source-id: 54c3e6859aadaaba4a49b3e851b73dc35ec7dc6a
-
由 sdong 提交于
Summary: A recent change https://github.com/facebook/rocksdb/pull/6386 broke LITE build in a trivial way. Fix it. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6411 Test Plan: Run "LITE=1 make all" Differential Revision: D19871765 fbshipit-source-id: 74f0ad3f8a9d666fbde0da7fd29ba1547a811f77
-
由 Cheng Chang 提交于
Summary: Seems like this caused the following test failure on AppVeyor: DBTest2.CrashInRecoveryMultipleCF c:\projects\rocksdb\db\db_test_util.cc(107): error: DestroyDB(dbname_, options) IO error: Failed to delete: C:\projects\rocksdb\db_tests\\testrocksdb-3112//db_test2_10791409581227174103/000013.sst: Access is denied. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6410 Test Plan: Wait to see whether the AppVeyor test passes. Differential Revision: D19879872 Pulled By: cheng-chang fbshipit-source-id: 59a9c55ca88566e9210c0b715ecc45a4fd9afe26
-
- 13 2月, 2020 2 次提交
-
-
由 sdong 提交于
Summary: We realized bugs related to IO Uring. Turn it off by default. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6405 Test Plan: Manually run build_tools/build_detect_platform and observe outputs. Differential Revision: D19862792 fbshipit-source-id: 5d5e8e2762997b72a145ae59389ef3d7e4ccd060
-
由 Burton Li 提交于
Summary: 1. `db_bench` now supports `value_size_distribution_type`, `value_size_min`, `value_size_max` options for generating random variable sized value. 2. Added `blob_db_compression_type` option for BlobDB to enable blob compression. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6386 Differential Revision: D19859406 Pulled By: zhichao-cao fbshipit-source-id: ace52674090023fde15d832392110bf288a8e215
-
- 12 2月, 2020 2 次提交
-
-
由 anand76 提交于
Summary: Unrevert the previous fix to propagate error status, and an additional fix to not treat a memtable lookup MergeInProgress status as an error. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6403 Test Plan: Unit tests Tried running stress tests but couldn't repro the stress failure Differential Revision: D19846721 Pulled By: anand1976 fbshipit-source-id: 7db10cccbdc863d9b559497f0a46b608d2488ca4
-
由 Tomas Kolda 提交于
Summary: It is very useful to support direct ByteBuffers in Java. It allows to have zero memory copy and some serializers are using that directly so one do not need to create byte[] array for it. This change also contains some fixes for Windows JNI build. Pull Request resolved: https://github.com/facebook/rocksdb/pull/2283 Differential Revision: D19834971 Pulled By: pdillinger fbshipit-source-id: 44173aa02afc9836c5498c592fd1ea95b6086e8e
-
- 11 2月, 2020 10 次提交
-
-
由 Yanqin Jin 提交于
Summary: Before this fix, the info_log_level passed from CreateLoggerFromOptions() will be ignored by AutoRollLogger::logger_. This PR fixes it by setting the info log level of logger_ during ResetLogger(). Test plan (dev server): ``` COMPILE_WITH_TSAN=1 make all && make check make all && make check ``` Pull Request resolved: https://github.com/facebook/rocksdb/pull/6388 Differential Revision: D19828045 Pulled By: riversand963 fbshipit-source-id: e1ac7de3a2090bee53b6c667f71a11f1774163e6
-
由 anand76 提交于
Summary: This reverts commit d70011bc. The commit is causing some stress test failure due to unexpected Status::MergeInProgress() return for some keys. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6401 Differential Revision: D19826623 Pulled By: anand1976 fbshipit-source-id: edd634cede9cb7bdd2cb8f46e662ea709b16d2f1
-
由 Cheng Chang 提交于
Summary: 1. remove AssertEmpty because calling methods on moved objects is discouraged. 2. add a test to assert that the internal buffer is moved instead of being copied. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6399 Test Plan: make slice_test && ./slice_test USE_CLANG=1 make analyze Differential Revision: D19825372 Pulled By: cheng-chang fbshipit-source-id: 2e26f8ce5ec3edbfce067db045e80bd433e704f4
-
由 Cheng Chang 提交于
Summary: Add a utility class `Defer` to defer the execution of a function until the Defer object goes out of scope. Used in VersionSet:: ProcessManifestWrites as an example. The inline comments for class `Defer` have more details. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6382 Test Plan: `make defer_test version_set_test && ./defer_test && ./version_set_test` Differential Revision: D19797538 Pulled By: cheng-chang fbshipit-source-id: b1a9b7306e4fd4f48ec2ab55783caa561a315f0f
-
由 Levi Tamasi 提交于
Summary: https://github.com/facebook/rocksdb/pull/6383 surfaced an issue with `VersionSet`/`ReactiveVersionSet` and `AtomicGroupReadBuffer::AddEdit` (which was added in https://github.com/facebook/rocksdb/pull/5411): `AddEdit` moves the `VersionEdit` passed to it into `replay_buffer_`, however, the client `VersionSet` classes keep using it afterwards. This *seemed to* work before the refactoring but it really did not: since `VersionEdit` used to have a user-declared destructor, no move constructor/move assignment operator was generated, and the `move` in `AddEdit` was really a copy. The patch makes the copy explicit. Note: it should be possible to rework this logic so that we can get away with the move but for now, this should fix the issue. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6400 Test Plan: `make check` `make analyze` Differential Revision: D19824466 Pulled By: ltamasi fbshipit-source-id: f38033967daf2a39c78dcd6e12978bafe37632b4
-
由 Zhichao Cao 提交于
Summary: In the current code base, RocksDB generate the checksum for each block and verify the checksum at usage. Current PR enable SST file checksum. After a SST file is generated by Flush or Compaction, RocksDB generate the SST file checksum and store the checksum value and checksum method name in the vs_info and MANIFEST as part for the FileMetadata. Added the enable_sst_file_checksum to Options to enable or disable file checksum. Added sst_file_checksum to Options such that user can plugin their own SST file checksum calculate method via overriding the SstFileChecksum class. The checksum information inlcuding uint32_t checksum value and a checksum name (string). A new tool is added to LDB such that user can dump out a list of file checksum information from MANIFEST. If user enables the file checksum but does not provide the sst_file_checksum instance, RocksDB will use the default crc32checksum implemented in table/sst_file_checksum_crc32c.h Pull Request resolved: https://github.com/facebook/rocksdb/pull/6216 Test Plan: Added the testing case in table_test and ldb_cmd_test to verify checksum is correct in different level. Pass make asan_check. Differential Revision: D19171461 Pulled By: zhichao-cao fbshipit-source-id: b2e53479eefc5bb0437189eaa1941670e5ba8b87
-
由 sdong 提交于
Summary: clang analysis shows following warning: options/options_test.cc:1554:24: warning: The left operand of '-' is a garbage value (file_size - 1) / readahead_size + 1); ~~~~~~~~~ ^ Explicitly initialize file_size and add an assertion to make clang analysis happy. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6398 Test Plan: Run "make analysis" and see the warning goes away. Differential Revision: D19819662 fbshipit-source-id: 1589ea91c0c8f78242538f01448e4ad0e5fbc219
-
由 sdong 提交于
Summary: Pull Request resolved: https://github.com/facebook/rocksdb/pull/6384 Test Plan: Wait and see the analysis result. Differential Revision: D19781072 fbshipit-source-id: 75e7cb6ee619ebd289841eaabea03dd075c09d3b
-
由 Yutian Li 提交于
Summary: When `no_slowdown` is enabled, it returns `Status::Incomplete("Write stall")` if a stall would occur. This patch adds descriptive text for when `no_slowdown` and `low_pri` are enabled. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6396 Differential Revision: D19808978 Pulled By: cheng-chang fbshipit-source-id: a53b0d25ed414c821a086531e0222027f925e627
-
由 Kefu Chai 提交于
Summary: to adapt the change in ceph upstream where the bufferlist::copy() method was removed in https://github.com/ceph/ceph/commit/c724369010a753bd44e11a534d1f42156c4fc12dSigned-off-by: NKefu Chai <tchaikov@gmail.com> Pull Request resolved: https://github.com/facebook/rocksdb/pull/6395 Differential Revision: D19816815 Pulled By: zhichao-cao fbshipit-source-id: 9210767b91af0ecdcf5dfaa3e70edcaeea55135f
-
- 08 2月, 2020 9 次提交
-
-
由 blackyblack 提交于
Summary: External CMAKE projects cannot add own gtest target due to CMP0002 policy. We should not add gtest target unless we build with tests. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6377 Differential Revision: D19801708 fbshipit-source-id: 662484683e8737e10397ddb0d17705da5243c4f9
-
由 anand76 提交于
Summary: Currently, any IO errors and checksum mismatches while reading data blocks, are being ignored by the batched MultiGet. Its only looking at the GetContext state. Fix that. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6387 Test Plan: Add unit tests Differential Revision: D19799819 Pulled By: anand1976 fbshipit-source-id: 46133dccbb04e64067b9fe6cda73e282203db969
-
由 Robert Yang 提交于
Summary: Fixed an error when compiled with -Og: db/write_thread.cc:183:14: error: 'state' may be used uninitialized in this function [-Werror=maybe-uninitialized] Signed-off-by: NRobert Yang <liezhi.yang@windriver.com> Pull Request resolved: https://github.com/facebook/rocksdb/pull/6275 Differential Revision: D19381755 fbshipit-source-id: a90bf3cd4a7248d9d71219e918fc6253deb97e3c
-
由 sdong 提交于
Summary: Right, when reading from option files, no readahead is used and 8KB buffer is used. It might introduce high latency if the file system provide high latency and doesn't do readahead. Instead, introduce a readahead to the file. When calling inside DB, infer the value from options.log_readahead. Otherwise, a default 512KB readahead size is used. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6372 Test Plan: Add --log_readahead_size in db_bench. Run it with several options and observe read size from option files using strace. Differential Revision: D19727739 fbshipit-source-id: e6d8053b0a64259abc087f1f388b9cd66fa8a583
-
由 Cheng Chang 提交于
Summary: It's logically correct for PinnableSlice to support move semantics to transfer ownership of the pinned memory region. This PR adds both move constructor and move assignment to PinnableSlice. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6374 Test Plan: A set of unit tests for the move semantics are added in slice_test. So `make slice_test && ./slice_test`. Differential Revision: D19739254 Pulled By: cheng-chang fbshipit-source-id: f898bd811bb05b2d87384ec58b645e9915e8e0b1
-
由 Chad Austin 提交于
Summary: liburing is a Linux-specific dependency, so make sure it's configured in the Linux-only Buck rules. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6378 Test Plan: ``` ~/fbcode $ cp internal_repo_rocksdb/repo/TARGETS rocksdb/src ~/fbcode $ buck build mode/mac eden ``` Reviewed By: chadaustin Differential Revision: D19760039 Pulled By: riversand963 fbshipit-source-id: 2abfce81c8b17965ef76012262cd117708e0294f
-
由 Levi Tamasi 提交于
Summary: This is a bunch of small improvements to `VersionEdit`. Namely, the patch * Makes the names and order of variables, methods, and code chunks related to the various information elements more consistent, and adds missing getters for the sake of completeness. * Initializes previously uninitialized stack variables. * Marks all getters const to improve const correctness. * Adds in-class initializers and removes the default ctor that would create an object with uninitialized built-in fields and call `Clear` afterwards. * Adds a new type alias for new files and changes the existing `typedef` for deleted files into a type alias as well. * Makes the helper method `DecodeNewFile4From` private. * Switches from long-winded iterator syntax to range based loops in a couple of places. * Fixes a couple of assignments where an integer 0 was assigned to boolean members. * Fixes a getter which used to return a `const std::string` instead of the intended `const std::string&`. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6383 Test Plan: make check Differential Revision: D19780537 Pulled By: ltamasi fbshipit-source-id: b0b4f09fee0ec0e7c7b7a6d76bfe5346e91824d0
-
由 Cheng Chang 提交于
Summary: Since the logic for handling IDENTITY file is now inside `NewDB`, the comment above `NewDB` is no longer relevant. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6379 Test Plan: not needed Differential Revision: D19795440 Pulled By: cheng-chang fbshipit-source-id: 0b1cca87ac6d92474701c46aa4c8d4d708bfa19b
-
由 Levi Tamasi 提交于
Summary: BlobDB keeps track of the mapping between SSTs and blob files using the `OnFlushCompleted` and `OnCompactionCompleted` callbacks of the `EventListener` interface: upon receiving a flush notification, a link is added between the newly flushed SST and the corresponding blob file; for compactions, links are removed for the inputs and added for the outputs. The earlier code performed this link deletion and addition even for trivially moved files; the new code walks through the two lists together (in a fashion that's similar to merge sort) and skips such files. This should mitigate https://github.com/facebook/rocksdb/issues/6338, wherein an assertion is triggered with the earlier code when a compaction notification for a trivial move precedes the flush notification for the moved SST. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6381 Test Plan: make check Differential Revision: D19773729 Pulled By: ltamasi fbshipit-source-id: ae0f273ded061110dd9334e8fb99b0d7786650b0
-