- 28 10月, 2017 5 次提交
-
-
由 Islam AbdelRahman 提交于
Summary: SstFileManager move files that need to be deleted into a trash directory. Deprecate this behaviour and instead add ".trash" extension to files that need to be deleted Closes https://github.com/facebook/rocksdb/pull/2970 Differential Revision: D5976805 Pulled By: IslamAbdelRahman fbshipit-source-id: 27374ece4315610b2792c30ffcd50232d4c9a343
-
由 Yi Wu 提交于
Summary: Changing blob file format and some code cleanup around the change. The change with blob log format are: * Remove timestamp field in blob file header, blob file footer and blob records. The field is not being use and often confuse with expiration field. * Blob file header now come with column family id, which always equal to default column family id. It leaves room for future support of column family. * Compression field in blob file header now is a standalone byte (instead of compact encode with flags field) * Blob file footer now come with its own crc. * Key length now being uint64_t instead of uint32_t * Blob CRC now checksum both key and value (instead of value only). * Some reordering of the fields. The list of cleanups: * Better inline comments in blob_log_format.h * rename ttlrange_t and snrange_t to ExpirationRange and SequenceRange respectively. * simplify blob_db::Reader * Move crc checking logic to inside blob_log_format.cc Closes https://github.com/facebook/rocksdb/pull/3081 Differential Revision: D6171304 Pulled By: yiwu-arbug fbshipit-source-id: e4373e0d39264441b7e2fbd0caba93ddd99ea2af
-
由 Dmitri Smirnov 提交于
Summary: Reuse WITH_JEMALLOC option in preparation for module search unification. Move jemalloc overrides into a separate .cc Remote obsolete JEMALLOC_NOINIT option. Closes https://github.com/facebook/rocksdb/pull/3078 Differential Revision: D6174826 Pulled By: yiwu-arbug fbshipit-source-id: 9970a0289b4490272d15853920d9d7531af91140
-
由 Prashant D 提交于
Summary: Coverity uninitialized member variable warnings in lru_cache Closes https://github.com/facebook/rocksdb/pull/3082 Differential Revision: D6173062 Pulled By: sagar0 fbshipit-source-id: 7bcfc653457bd362d46045d06527838c9a6adad6
-
由 Prashant D 提交于
Summary: db/column_family.h : 79 ColumnFamilyHandleInternal() CID 1322806 (#1 of 1): Uninitialized pointer field (UNINIT_CTOR) 2. uninit_member: Non-static class member internal_cfd_ is not initialized in this constructor nor in any functions that it calls. 80 : ColumnFamilyHandleImpl(nullptr, nullptr, nullptr) {} db/compacted_db_impl.cc: 18CompactedDBImpl::CompactedDBImpl( 19 const DBOptions& options, const std::string& dbname) 20 : DBImpl(options, dbname) { 2. uninit_member: Non-static class member cfd_ is not initialized in this constructor nor in any functions that it calls. 4. uninit_member: Non-static class member version_ is not initialized in this constructor nor in any functions that it calls. CID 1396120 (#1 of 1): Uninitialized pointer field (UNINIT_CTOR) 6. uninit_member: Non-static class member user_comparator_ is not initialized in this constructor nor in any functions that it calls. 21} db/compaction_iterator.cc: 9. uninit_member: Non-static class member current_user_key_sequence_ is not initialized in this constructor nor in any functions that it calls. 11. uninit_member: Non-static class member current_user_key_snapshot_ is not initialized in this constructor nor in any functions that it calls. CID 1419855 (#1 of 1): Uninitialized scalar field (UNINIT_CTOR) 13. uninit_member: Non-static class member current_key_committed_ is not initialized in this constructor nor in any functions that it calls. Closes https://github.com/facebook/rocksdb/pull/3084 Differential Revision: D6172999 Pulled By: sagar0 fbshipit-source-id: 084d73393faf8022c01359cfb445807b6a782460
-
- 27 10月, 2017 4 次提交
-
-
由 Prashant D 提交于
Pulled By: ajkr Differential Revision: D6170448 fbshipit-source-id: 5fd6d1608fc0df27c94d9f5059315ce7f79b8f5c
-
由 Prashant D 提交于
Summary: 228MutableDBOptions::MutableDBOptions() 229 : max_background_jobs(2), 230 base_background_compactions(-1), 231 max_background_compactions(-1), 232 avoid_flush_during_shutdown(false), 233 delayed_write_rate(2 * 1024U * 1024U), 234 max_total_wal_size(0), 235 delete_obsolete_files_period_micros(6ULL * 60 * 60 * 1000000), 236 stats_dump_period_sec(600), 2. uninit_member: Non-static class member bytes_per_sync is not initialized in this constructor nor in any functions that it calls. CID 1419857 (#1 of 1): Uninitialized scalar field (UNINIT_CTOR) 4. uninit_member: Non-static class member wal_bytes_per_sync is not initialized in this constructor nor in any functions that it calls. 237 max_open_files(-1) {} Closes https://github.com/facebook/rocksdb/pull/3069 Differential Revision: D6170424 Pulled By: ajkr fbshipit-source-id: 1f94e86b87611ad2330b8b1707911150978d68b8
-
由 Andrew Kryczka 提交于
Summary: - for `SeekToFirst()`, just convert it to a regular `Seek()` if lower bound is specified - for operations that iterate backwards over user keys (`SeekForPrev`, `SeekToLast`, `Prev`), change `PrevInternal` to check whether user key went below lower bound every time the user key changes -- same approach we use to ensure we stay within a prefix when `prefix_same_as_start=true`. Closes https://github.com/facebook/rocksdb/pull/3074 Differential Revision: D6158654 Pulled By: ajkr fbshipit-source-id: cb0e3a922e2650d2cd4d1c6e1c0f1e8b729ff518
-
由 Yi Wu 提交于
Summary: Adding the `min_blob_size` option to allow storing small values in base db (in LSM tree) together with the key. The goal is to improve performance for small values, while taking advantage of blob db's low write amplification for large values. Also adding expiration timestamp to blob index. It will be useful to evict stale blob indexes in base db by adding a compaction filter. I'll work on the compaction filter in future patches. See blob_index.h for the new blob index format. There are 4 cases when writing a new key: * small value w/o TTL: put in base db as normal value (i.e. ValueType::kTypeValue) * small value w/ TTL: put (type, expiration, value) to base db. * large value w/o TTL: write value to blob log and put (type, file, offset, size, compression) to base db. * large value w/TTL: write value to blob log and put (type, expiration, file, offset, size, compression) to base db. Closes https://github.com/facebook/rocksdb/pull/3066 Differential Revision: D6142115 Pulled By: yiwu-arbug fbshipit-source-id: 9526e76e19f0839310a3f5f2a43772a4ad182cd0
-
- 26 10月, 2017 3 次提交
-
-
由 Andrew Kryczka 提交于
Summary: When snapshots are held for a long time, files may reach the bottom level containing overwritten/deleted keys. We previously had no mechanism to trigger compaction on such files. This particularly impacted DBs that write to different parts of the keyspace over time, as such files would never be naturally compacted due to second-last level files moving down. This PR introduces a mechanism for bottommost files to be recompacted upon releasing all snapshots that prevent them from dropping their deleted/overwritten keys. - Changed `CompactionPicker` to compact files in `BottommostFilesMarkedForCompaction()`. These are the last choice when picking. Each file will be compacted alone and output to the same level in which it originated. The goal of this type of compaction is to rewrite the data excluding deleted/overwritten keys. - Changed `ReleaseSnapshot()` to recompute the bottom files marked for compaction when the oldest existing snapshot changes, and schedule a compaction if needed. We cache the value that oldest existing snapshot needs to exceed in order for another file to be marked in `bottommost_files_mark_threshold_`, which allows us to avoid recomputing marked files for most snapshot releases. - Changed `VersionStorageInfo` to track the list of bottommost files, which is recomputed every time the version changes by `UpdateBottommostFiles()`. The list of marked bottommost files is first computed in `ComputeBottommostFilesMarkedForCompaction()` when the version changes, but may also be recomputed when `ReleaseSnapshot()` is called. - Extracted core logic of `Compaction::IsBottommostLevel()` into `VersionStorageInfo::RangeMightExistAfterSortedRun()` since logic to check whether a file is bottommost is now necessary outside of compaction. Closes https://github.com/facebook/rocksdb/pull/3009 Differential Revision: D6062044 Pulled By: ajkr fbshipit-source-id: 123d201cf140715a7d5928e8b3cb4f9cd9f7ad21
-
由 Sagar Vemuri 提交于
Summary: I found that we continue accepting writes even when the blob db goes beyond the configured blob directory size limit. Now, we return an error for writes on reaching `blob_dir_size` limit and if `is_fifo` is set to false. (We cannot just drop any file when `is_fifo` is true.) Deleting the oldest file when `is_fifo` is true will be handled in a later PR. Closes https://github.com/facebook/rocksdb/pull/3060 Differential Revision: D6136156 Pulled By: sagar0 fbshipit-source-id: 2f11cb3f2eedfa94524fbfa2613dd64bfad7a23c
-
由 Islam AbdelRahman 提交于
Summary: When doing a Seek() or SeekForPrev() we should stop the moment we see a key with a different prefix as start if ReadOptions:: prefix_same_as_start was set to true Right now we don't stop if we encounter a tombstone outside the prefix while executing SeekForPrev() Closes https://github.com/facebook/rocksdb/pull/3067 Differential Revision: D6149638 Pulled By: IslamAbdelRahman fbshipit-source-id: 7f659862d2bf552d3c9104a360c79439ceba2f18
-
- 25 10月, 2017 1 次提交
-
-
由 zach shipko 提交于
Summary: A few simple changes to allow RocksDB to be built on OpenBSD. Let me know if any further changes are needed. Closes https://github.com/facebook/rocksdb/pull/3061 Differential Revision: D6138800 Pulled By: ajkr fbshipit-source-id: a13a17b5dc051e6518bd56a8c5efd1d24dd81b0c
-
- 24 10月, 2017 4 次提交
-
-
由 zawlazaw 提交于
Summary: Java's `Status.SubCode` was out of sync with `include/rocksdb/status.h:SubCode`. When running out of disc space this led to an `IllegalArgumentException` because of an invalid status code, rather than just returning the corresponding status code without an exception. I added the missing status codes. By this, we keep the behaviour of throwing an `IllegalArgumentException` in case of newly added status codes that are defined in C but not in Java. We could think of an alternative strategy: add in Java another code "UnknownCode" which acts as a catch-all for all those status codes that are not yet mirrored from C to Java. This approach would never throw an exception but simply return a non-OK status-code. I think the current approach of throwing an Exception in case of a C/Java inconsistency is fine, but if you have some opinion on the alternative strategy, then feel free to comment here. Closes https://github.com/facebook/rocksdb/pull/3050 Differential Revision: D6129682 Pulled By: sagar0 fbshipit-source-id: f2bf44caad650837cffdcb1f93eb793b43580c66
-
由 Yi Wu 提交于
Summary: With FIFO compaction we would like to get the oldest data time for monitoring. The problem is we don't have timestamp for each key in the DB. As an approximation, we expose the earliest of sst file "creation_time" property. My plan is to override the property with a more accurate value with blob db, where we actually have timestamp. Closes https://github.com/facebook/rocksdb/pull/2842 Differential Revision: D5770600 Pulled By: yiwu-arbug fbshipit-source-id: 03833c8f10bbfbee62f8ea5c0d03c0cafb5d853a
-
由 Dmitri Smirnov 提交于
Summary: MSVC does not support unused attribute at this time. A separate assignment line fixes the issue probably by being counted as usage for MSVC and it no longer complains about unused var. Closes https://github.com/facebook/rocksdb/pull/3048 Differential Revision: D6126272 Pulled By: maysamyabandeh fbshipit-source-id: 4907865db45fd75a39a15725c0695aaa17509c1f
-
由 Maysam Yabandeh 提交于
Summary: Enable concurrent_prepare flag for WritePrepared transactions and extend the existing transaction tests with this config. Closes https://github.com/facebook/rocksdb/pull/3046 Differential Revision: D6106534 Pulled By: maysamyabandeh fbshipit-source-id: 88c8d21d45bc492beb0a131caea84a2ac5e7d38c
-
- 21 10月, 2017 6 次提交
-
-
由 Sagar Vemuri 提交于
Summary: This test shouldn't be enabled under the lite version; and this fixes the failing contrun test due to #3006. Closes https://github.com/facebook/rocksdb/pull/3056 Differential Revision: D6114681 Pulled By: sagar0 fbshipit-source-id: dc5243549ae6b1353cec7edb820c771d95f66dda
-
由 Maysam Yabandeh 提交于
Summary: The test currently times out when it is run under tsan. This patch split it into 4 tests. Closes https://github.com/facebook/rocksdb/pull/3047 Differential Revision: D6106515 Pulled By: maysamyabandeh fbshipit-source-id: 03a28cdf8b1c097be2361b1b0cc3dc1acf2b5d63
-
由 Andrew Kryczka 提交于
Summary: Add options to `db_stress` (correctness testing tool) to randomly acquire snapshot and release it after some period of time. It's useful for correctness testing of #3009, as well as other parts of compaction that behave differently depending on which snapshots are held. Closes https://github.com/facebook/rocksdb/pull/3038 Differential Revision: D6086501 Pulled By: ajkr fbshipit-source-id: 3ec0d8666c78ac507f1f808887c4ff759ba9b865
-
由 Andrew Kryczka 提交于
Summary: fixup 6a541afc. This code didn't do anything because (1) `bytes_per_sync` is assigned in `EnvOptions`'s constructor; and (2) `OptimizeForCompactionTableWrite`'s return value was ignored, even though its only purpose is to return something. Closes https://github.com/facebook/rocksdb/pull/3055 Differential Revision: D6114132 Pulled By: ajkr fbshipit-source-id: ea4831770930e9cf83518e13eb2e1934d1f5487c
-
由 Andrew Kryczka 提交于
Summary: We should mention these: - `EventListener::OnStallConditionsChanged()` in 01542400 - `DeleteRange()` fix in 966b32b5 Closes https://github.com/facebook/rocksdb/pull/3054 Differential Revision: D6113989 Pulled By: ajkr fbshipit-source-id: d5e058e1ab07570df22936e8d5939fb30fb4d381
-
由 raistlin 提交于
Summary: Fix unstable floating point exception, tested on Windows, 64-bit build. The problem appeared in `SetCapacity()` method at line `high_pri_pool_capacity_ = capacity_ * high_pri_pool_ratio_;` `high_pri_pool_ratio_` was not initialized at that moment, because `SetHighPriorityPoolRatio()` is called after `SetCapacity()`. So, `high_pri_pool_ratio_` contained garbage, which caused "Floating point exception" sometimes. Closes https://github.com/facebook/rocksdb/pull/3052 Differential Revision: D6111161 Pulled By: yiwu-arbug fbshipit-source-id: d170329111ad12b4bf9bbcf37bcb6411523438ae
-
- 20 10月, 2017 2 次提交
-
-
由 Sagar Vemuri 提交于
Summary: ColumnFamilyOptions::compaction_options_fifo and all its sub-fields can be set dynamically now. Some of the ways in which the fifo compaction options can be set are: - `SetOptions({{"compaction_options_fifo", "{max_table_files_size=1024}"}})` - `SetOptions({{"compaction_options_fifo", "{ttl=600;}"}})` - `SetOptions({{"compaction_options_fifo", "{max_table_files_size=1024;ttl=600;}"}})` - `SetOptions({{"compaction_options_fifo", "{max_table_files_size=51;ttl=49;allow_compaction=true;}"}})` Most of the code has been made generic enough so that it could be reused later to make universal options (and other such nested defined-types) dynamic with very few lines of parsing/serializing code changes. Introduced a few new functions like `ParseStruct`, `SerializeStruct` and `GetStringFromStruct`. The duplicate code in `GetStringFromDBOptions` and `GetStringFromColumnFamilyOptions` has been moved into `GetStringFromStruct`. So they become just simple wrappers now. Closes https://github.com/facebook/rocksdb/pull/3006 Differential Revision: D6058619 Pulled By: sagar0 fbshipit-source-id: 1e8f78b3374ca5249bb4f3be8a6d3bb4cbc52f92
-
由 Dmitri Smirnov 提交于
Summary: Closes https://github.com/facebook/rocksdb/pull/3018 Differential Revision: D6079011 Pulled By: yiwu-arbug fbshipit-source-id: 988a721e7e7617967859dba71d660fc69f4dff57
-
- 19 10月, 2017 3 次提交
-
-
由 Sagar Vemuri 提交于
Summary: Update RocksDB Authors File. Reviewed By: yiwu-arbug Differential Revision: D6075453 fbshipit-source-id: dff52f483aab33c41de391f145a8273acfd6cbde
-
由 Gihwan Oh 提交于
Summary: instad of for specific level -> instead of a specific level Closes https://github.com/facebook/rocksdb/pull/3040 Differential Revision: D6090811 Pulled By: sagar0 fbshipit-source-id: 499edef0a6f596c448f61791e6aca8f5cce08e9c
-
由 Maysam Yabandeh 提交于
Summary: Disables GC during recovery of a WritePrepared txn db to avoid GCing uncommitted key values. Closes https://github.com/facebook/rocksdb/pull/2980 Differential Revision: D6000191 Pulled By: maysamyabandeh fbshipit-source-id: fc4d522c643d24ebf043f811fe4ecd0dd0294675
-
- 18 10月, 2017 7 次提交
-
-
由 Nikhil Benesch 提交于
Summary: As discussed on the mailing list (["Skipping entire SSTs while iterating"](https://groups.google.com/forum/#!topic/rocksdb/ujHCJVLrHlU)), this patch adds a `table_filter` to `ReadOptions` that allows specifying a callback to be executed during iteration before each table in the database is scanned. The callback is passed the table's properties; the table is scanned iff the callback returns true. This can be used in conjunction with a `TablePropertiesCollector` to dramatically speed up scans by skipping tables that are known to contain irrelevant data for the scan at hand. We're using this [downstream in CockroachDB](https://github.com/cockroachdb/cockroach/blob/master/pkg/storage/engine/db.cc#L2009-L2022) already. With this feature, under ideal conditions, we can reduce the time of an incremental backup in from hours to seconds. FYI, the first commit in this PR fixes a segfault that I unfortunately have not figured out how to reproduce outside of CockroachDB. I'm hoping you accept it on the grounds that it is not correct to return 8-byte aligned memory from a call to `malloc` on some 64-bit platforms; one correct approach is to infer the necessary alignment from `std::max_align_t`, as done here. As noted in the first commit message, the bug is tickled by having a`std::function` in `struct ReadOptions`. That is, the following patch alone is enough to cause RocksDB to segfault when run from CockroachDB on Darwin. ```diff --- a/include/rocksdb/options.h +++ b/include/rocksdb/options.h @@ -1546,6 +1546,13 @@ struct ReadOptions { // Default: false bool ignore_range_deletions; + // A callback to determine whether relevant keys for this scan exist in a + // given table based on the table's properties. The callback is passed the + // properties of each table during iteration. If the callback returns false, + // the table will not be scanned. + // Default: empty (every table will be scanned) + std::function<bool(const TableProperties&)> table_filter; + ReadOptions(); ReadOptions(bool cksum, bool cache); }; ``` /cc danhhz Closes https://github.com/facebook/rocksdb/pull/2265 Differential Revision: D5054262 Pulled By: yiwu-arbug fbshipit-source-id: dd6b28f2bba6cb8466250d8c5c542d3c92785476
-
由 Yi Wu 提交于
Summary: Blob db insert blob index to base db as kTypeBlobIndex type, to tell apart values written by plain rocksdb or blob db. This is to make it possible to migrate from existing rocksdb to blob db. Also with the patch blob db garbage collection get away from OptimisticTransaction. Instead it use a custom write callback to achieve similar behavior as OptimisticTransaction. This is because we need to pass the is_blob_index flag to DBImpl::Get but OptimisticTransaction don't support it. Closes https://github.com/facebook/rocksdb/pull/3000 Differential Revision: D6050044 Pulled By: yiwu-arbug fbshipit-source-id: 61dc72ab9977625e75f78cd968e7d8a3976e3632
-
由 Yi Wu 提交于
Summary: Previously each time we write a blob we write blog_record_header + key + value + blob_record_footer to blob log. The footer only contains a sequence and a crc for the sequence number. The sequence number was used in garbage collection to verify the value is recent. After #2703 we moved to use optimistic transaction and no longer use sequence number from the footer. Remove the footer altogether. There's another usage of sequence number and we are keeping it: Each blob log file keep track of sequence number range of keys in it, and use it to check if it is reference by a snapshot, before being deleted. Closes https://github.com/facebook/rocksdb/pull/3005 Differential Revision: D6057585 Pulled By: yiwu-arbug fbshipit-source-id: d6da53c457a316e9723f359a1b47facfc3ffe090
-
由 zhangjinpeng1987 提交于
Summary: Fix this [issue](https://github.com/facebook/rocksdb/issues/2989). ajkr PTAL Close #2989 Closes https://github.com/facebook/rocksdb/pull/3017 Differential Revision: D6078541 Pulled By: yiwu-arbug fbshipit-source-id: ef3db87b37b9156f83ca468aa39dea1f6dbde49d
-
由 Nikhil Benesch 提交于
Summary: As raised in #2265, the arena allocator will return memory that is improperly aligned to store a `std::function` on macOS. Oddly, I'm unable to tickle this bug without adding a `std::function` field to `struct ReadOptions`—but my proposal in #2265 does exactly that. In any case, here's a simple reproduction. Apply this bogus patch to get a `std::function` into `struct ReadOptions` ``` --- a/include/rocksdb/options.h +++ b/include/rocksdb/options.h @@ -1035,6 +1035,8 @@ struct ReadOptions { // Default: 0 uint64_t max_skippable_internal_keys; + std::function<void()> foo; + ReadOptions(); ReadOptions(bool cksum, bool cache); }; ``` then compile `db_properties_test` *with ubsan* and run `ReadLatencyHistogramByLevel`: ``` $ make COMPILE_WITH_UBSAN=1 db_properties_test $ ./db_properties_test --gtest_filter=DBPropertiesTest.ReadLatencyHistogramByLevel ``` ubsan will complain about several misaligned accesses: ``` Note: Google Test filter = DBPropertiesTest.ReadLatencyHistogramByLevel [==========] Running 1 test from 1 test case. [----------] Global test environment set-up. [----------] 1 test from DBPropertiesTest [ RUN ] DBPropertiesTest.ReadLatencyHistogramByLevel util/coding.h:372:12: runtime error: load of misaligned address 0x00010d85516c for type 'const unsigned long', which requires 8 byte alignment 0x00010d85516c: note: pointer points here 01 00 34 57 00 00 00 00 02 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 78 24 82 0a 01 00 00 00 ^ util/coding.h:362:3: runtime error: store to misaligned address 0x7fff5733fac4 for type 'unsigned long', which requires 8 byte alignment 0x7fff5733fac4: note: pointer points here 01 00 00 00 00 00 00 00 02 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 80 1d 96 0d 01 00 00 00 ^ util/coding.h:372:12: runtime error: load of misaligned address 0x00010d85516c for type 'const unsigned long', which requires 8 byte alignment 0x00010d85516c: note: pointer points here 01 00 34 57 00 00 00 00 02 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 78 24 82 0a 01 00 00 00 ^ version_set.cc:854: runtime error: constructor call on misaligned address 0x00010dbfa5c8 for type 'rocksdb::(anonymous namespace)::LevelFileIteratorState', which requires 16 byte alignment 0x00010dbfa5c8: note: pointer points here 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ^ version_set.cc:512: runtime error: constructor call on misaligned address 0x00010dbfa5c8 for type 'rocksdb::(anonymous namespace)::LevelFileIteratorState', which requires 16 byte alignment 0x00010dbfa5c8: note: pointer points here 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ^ version_set.cc:505: runtime error: constructor call on misaligned address 0x00010dbfa5e8 for type 'rocksdb::ReadOptions', which requires 16 byte alignment 0x00010dbfa5e8: note: pointer points here 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ^ options.h:931: runtime error: constructor call on misaligned address 0x00010dbfa5e8 for type 'rocksdb::ReadOptions', which requires 16 byte alignment 0x00010dbfa5e8: note: pointer points here 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ^ options.h:931: runtime error: constructor call on misaligned address 0x00010dbfa628 for type 'std::__1::function<void ()>', which requires 16 byte alignment 0x00010dbfa628: note: pointer points here 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ^ functional:1583: runtime error: constructor call on misaligned address 0x00010dbfa628 for type 'std::__1::function<void ()>', which requires 16 byte alignment 0x00010dbfa628: note: pointer points here 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ^ /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/../include/c++/v1/functional:1585:9: runtime error: member access within misaligned address 0x00010dbfa628 for type 'std::__1::function<void ()>', which requires 16 byte alignment 0x00010dbfa628: note: pointer points here 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ^ /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/../include/c++/v1/functional:1585:9: runtime error: store to misaligned address 0x00010dbfa648 for type '__base *' (aka '__base<void ()> *'), which requires 16 byte alignment 0x00010dbfa648: note: pointer points here 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ^ db/version_set.cc:864:29: runtime error: upcast of misaligned address 0x00010dbfa5c8 for type 'rocksdb::(anonymous namespace)::LevelFileIteratorState', which requires 16 byte alignment 0x00010dbfa5c8: note: pointer points here 00 00 00 00 a0 db 70 0a 01 00 00 00 00 00 00 00 00 00 00 00 90 14 98 0d 01 00 00 00 00 00 00 00 ^ db/version_set.cc:521:12: runtime error: member access within misaligned address 0x00010dbfa5c8 for type 'rocksdb::(anonymous namespace)::LevelFileIteratorState', which requires 16 byte alignment 0x00010dbfa5c8: note: pointer points here 00 00 00 00 a0 db 70 0a 01 00 00 00 00 00 00 00 00 00 00 00 90 14 98 0d 01 00 00 00 00 00 00 00 ^ db/version_set.cc:521:12: runtime error: load of misaligned address 0x00010dbfa5d8 for type 'rocksdb::TableCache *', which requires 16 byte alignment 0x00010dbfa5d8: note: pointer points here 00 00 00 00 90 14 98 0d 01 00 00 00 00 00 00 00 00 00 00 00 01 01 ff ff ff ff ff ff 00 00 00 00 ^ db/version_set.cc:522:9: runtime error: member access within misaligned address 0x00010dbfa5c8 for type 'rocksdb::(anonymous namespace)::LevelFileIteratorState', which requires 16 byte alignment 0x00010dbfa5c8: note: pointer points here 00 00 00 00 a0 db 70 0a 01 00 00 00 00 00 00 00 00 00 00 00 90 14 98 0d 01 00 00 00 00 00 00 00 ^ db/version_set.cc:522:9: runtime error: reference binding to misaligned address 0x00010dbfa5e8 for type 'const rocksdb::ReadOptions', which requires 16 byte alignment 0x00010dbfa5e8: note: pointer points here 00 00 00 00 01 01 ff ff ff ff ff ff 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ^ db/version_set.cc:522:24: runtime error: member access within misaligned address 0x00010dbfa5c8 for type 'rocksdb::(anonymous namespace)::LevelFileIteratorState', which requires 16 byte alignment 0x00010dbfa5c8: note: pointer points here 00 00 00 00 a0 db 70 0a 01 00 00 00 00 00 00 00 00 00 00 00 90 14 98 0d 01 00 00 00 00 00 00 00 ^ db/version_set.cc:522:38: runtime error: member access within misaligned address 0x00010dbfa5c8 for type 'rocksdb::(anonymous namespace)::LevelFileIteratorState', which requires 16 byte alignment 0x00010dbfa5c8: note: pointer points here 00 00 00 00 a0 db 70 0a 01 00 00 00 00 00 00 00 00 00 00 00 90 14 98 0d 01 00 00 00 00 00 00 00 ^ db/version_set.cc:522:57: runtime error: member access within misaligned address 0x00010dbfa5c8 for type 'rocksdb::(anonymous namespace)::LevelFileIteratorState', which requires 16 byte alignment 0x00010dbfa5c8: note: pointer points here 00 00 00 00 a0 db 70 0a 01 00 00 00 00 00 00 00 00 00 00 00 90 14 98 0d 01 00 00 00 00 00 00 00 ^ db/version_set.cc:522:57: runtime error: load of misaligned address 0x00010dbfa678 for type 'rocksdb::RangeDelAggregator *', which requires 16 byte alignment 0x00010dbfa678: note: pointer points here 01 00 00 00 d0 a1 bf 0d 01 00 00 00 00 00 00 00 00 00 00 00 f8 db 70 0a 01 00 00 00 00 00 00 00 ^ db/version_set.cc:523:54: runtime error: member access within misaligned address 0x00010dbfa5c8 for type 'rocksdb::(anonymous namespace)::LevelFileIteratorState', which requires 16 byte alignment 0x00010dbfa5c8: note: pointer points here 00 00 00 00 a0 db 70 0a 01 00 00 00 00 00 00 00 00 00 00 00 90 14 98 0d 01 00 00 00 00 00 00 00 ^ db/version_set.cc:523:54: runtime error: load of misaligned address 0x00010dbfa668 for type 'rocksdb::HistogramImpl *', which requires 16 byte alignment 0x00010dbfa668: note: pointer points here 01 00 00 00 c8 88 a5 0d 01 00 00 00 00 00 00 00 01 00 00 00 d0 a1 bf 0d 01 00 00 00 00 00 00 00 ^ db/version_set.cc:524:9: runtime error: member access within misaligned address 0x00010dbfa5c8 for type 'rocksdb::(anonymous namespace)::LevelFileIteratorState', which requires 16 byte alignment 0x00010dbfa5c8: note: pointer points here 00 00 00 00 a0 db 70 0a 01 00 00 00 00 00 00 00 00 00 00 00 90 14 98 0d 01 00 00 00 00 00 00 00 ^ db/version_set.cc:524:47: runtime error: member access within misaligned address 0x00010dbfa5c8 for type 'rocksdb::(anonymous namespace)::LevelFileIteratorState', which requires 16 byte alignment 0x00010dbfa5c8: note: pointer points here 00 00 00 00 a0 db 70 0a 01 00 00 00 00 00 00 00 00 00 00 00 90 14 98 0d 01 00 00 00 00 00 00 00 ^ db/version_set.cc:524:62: runtime error: member access within misaligned address 0x00010dbfa5c8 for type 'rocksdb::(anonymous namespace)::LevelFileIteratorState', which requires 16 byte alignment 0x00010dbfa5c8: note: pointer points here 00 00 00 00 a0 db 70 0a 01 00 00 00 00 00 00 00 00 00 00 00 90 14 98 0d 01 00 00 00 00 00 00 00 ^ db/table_cache.cc:228:33: runtime error: reference binding to misaligned address 0x00010dbfa5e8 for type 'const rocksdb::ReadOptions', which requires 16 byte alignment 0x00010dbfa5e8: note: pointer points here 00 00 00 00 01 01 ff ff ff ff ff ff 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ^ table/block_based_table_reader.cc:1554:41: runtime error: reference binding to misaligned address 0x00010dbfa5e8 for type 'const rocksdb::ReadOptions', which requires 16 byte alignment 0x00010dbfa5e8: note: pointer points here 00 00 00 00 01 01 ff ff ff ff ff ff 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ^ table/block_based_table_reader.cc:1396:21: runtime error: reference binding to misaligned address 0x00010dbfa5e8 for type 'const rocksdb::ReadOptions', which requires 16 byte alignment 0x00010dbfa5e8: note: pointer points here 00 00 00 00 01 01 ff ff ff ff ff ff 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ^ include/rocksdb/options.h:931:8: runtime error: reference binding to misaligned address 0x00010dbfa628 for type 'const std::function<void ()>', which requires 16 byte alignment 0x00010dbfa628: note: pointer points here 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ^ /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/../include/c++/v1/functional:1584:13: runtime error: load of misaligned address 0x00010dbfa648 for type '__base *const' (aka '__base<void ()> *const'), which requires 16 byte alignment 0x00010dbfa648: note: pointer points here 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 c8 a5 97 0d 01 00 00 00 38 36 9b 0d ^ table/block_based_table_reader.cc:1555:24: runtime error: reference binding to misaligned address 0x00010dbfa5e8 for type 'const rocksdb::ReadOptions', which requires 16 byte alignment 0x00010dbfa5e8: note: pointer points here 00 00 00 00 01 01 ff ff ff ff ff ff 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ^ db/table_cache.cc:244:54: runtime error: load of misaligned address 0x00010dbfa618 for type 'const bool', which requires 16 byte alignment 0x00010dbfa618: note: pointer points here 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ^ db/table_cache.cc:246:49: runtime error: reference binding to misaligned address 0x00010dbfa5e8 for type 'const rocksdb::ReadOptions', which requires 16 byte alignment 0x00010dbfa5e8: note: pointer points here 00 00 00 00 01 01 ff ff ff ff ff ff 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ^ db/version_set.cc:532:12: runtime error: member access within misaligned address 0x00010dbfa5c8 for type 'rocksdb::(anonymous namespace)::LevelFileIteratorState', which requires 16 byte alignment 0x00010dbfa5c8: note: pointer points here 00 00 00 00 a0 db 70 0a 01 00 00 00 00 00 00 00 00 00 00 00 90 14 98 0d 01 00 00 00 00 00 00 00 ^ db/version_set.cc:532:12: runtime error: member access within misaligned address 0x00010dbfa5e8 for type 'const rocksdb::ReadOptions', which requires 16 byte alignment 0x00010dbfa5e8: note: pointer points here 00 00 00 00 01 01 ff ff ff ff ff ff 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ^ db/version_set.cc:532:26: runtime error: load of misaligned address 0x00010dbfa5f8 for type 'const rocksdb::Slice *const', which requires 16 byte alignment 0x00010dbfa5f8: note: pointer points here 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ^ version_set.cc:493: runtime error: member call on misaligned address 0x00010dbfa5c8 for type 'rocksdb::(anonymous namespace)::LevelFileIteratorState', which requires 16 byte alignment 0x00010dbfa5c8: note: pointer points here 00 00 00 00 a0 db 70 0a 01 00 00 00 00 00 00 00 00 00 00 00 90 14 98 0d 01 00 00 00 00 00 00 00 ^ version_set.cc:493: runtime error: member call on misaligned address 0x00010dbfa5e8 for type 'rocksdb::ReadOptions', which requires 16 byte alignment 0x00010dbfa5e8: note: pointer points here 00 00 00 00 01 01 ff ff ff ff ff ff 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ^ options.h:931: runtime error: member call on misaligned address 0x00010dbfa5e8 for type 'rocksdb::ReadOptions', which requires 16 byte alignment 0x00010dbfa5e8: note: pointer points here 00 00 00 00 01 01 ff ff ff ff ff ff 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ^ options.h:931: runtime error: member call on misaligned address 0x00010dbfa628 for type 'std::__1::function<void ()>', which requires 16 byte alignment 0x00010dbfa628: note: pointer points here 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ^ functional:1765: runtime error: member call on misaligned address 0x00010dbfa628 for type 'std::__1::function<void ()>', which requires 16 byte alignment 0x00010dbfa628: note: pointer points here 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ^ /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/../include/c++/v1/functional:1766:9: runtime error: member access within misaligned address 0x00010dbfa628 for type 'std::__1::function<void ()>', which requires 16 byte alignment 0x00010dbfa628: note: pointer points here 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ^ /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/../include/c++/v1/functional:1766:9: runtime error: load of misaligned address 0x00010dbfa648 for type '__base *' (aka '__base<void ()> *'), which requires 16 byte alignment 0x00010dbfa648: note: pointer points here 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 c8 a5 97 0d 01 00 00 00 38 36 9b 0d ^ /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/../include/c++/v1/functional:1766:27: runtime error: member access within misaligned address 0x00010dbfa628 for type 'std::__1::function<void ()>', which requires 16 byte alignment 0x00010dbfa628: note: pointer points here 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ^ /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/../include/c++/v1/functional:1768:14: runtime error: member access within misaligned address 0x00010dbfa628 for type 'std::__1::function<void ()>', which requires 16 byte alignment 0x00010dbfa628: note: pointer points here 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ^ /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/../include/c++/v1/functional:1768:14: runtime error: load of misaligned address 0x00010dbfa648 for type '__base *' (aka '__base<void ()> *'), which requires 16 byte alignment 0x00010dbfa648: note: pointer points here 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 c8 a5 97 0d 01 00 00 00 38 36 9b 0d ^ [ OK ] DBPropertiesTest.ReadLatencyHistogramByLevel (1599 ms) [----------] 1 test from DBPropertiesTest (1599 ms total) [----------] Global test environment tear-down [==========] 1 test from 1 test case ran. (1599 ms total) [ PASSED ] 1 test. ``` So it seems the root cause is that the internal implementation of `std::function` on macOS (and perhaps with libc++ generally?) requires 16-byte aligned memory, but the arena allocator only guarantees that the returned memory will be `sizeof(void*)` aligned, which is only 8-byte alignment on my machine. This patch solves the problem by adjusting the allocator to derive the necessary alignment from `alignof(std::max_align_t)`, which is properly 16 bytes on my machine. As I mentioned in #2265, none of RocksDB's tests will cause this unaligned access to actually abort the process, but, on macOS, linking CockroachDB against a version of RocksDB with the above patch and letting it run for just a few seconds will cause a SIGABRT. ``` Process 19792 stopped * thread #2, stop reason = EXC_BAD_ACCESS (code=EXC_I386_GPFLT) frame #0: 0x0000000004f5e78f cockroach`DBNewIter + 95 cockroach`DBNewIter: -> 0x4f5e78f <+95>: callq *0x28(%rax) 0x4f5e792 <+98>: jmp 0x4f5e79e ; <+110> 0x4f5e794 <+100>: movq -0x50(%rbp), %rcx 0x4f5e798 <+104>: movq %rax, %rdi (lldb) bt * thread #2, stop reason = EXC_BAD_ACCESS (code=EXC_I386_GPFLT) * frame #0: 0x0000000004f5e78f cockroach`DBNewIter + 95 ``` I'd get you a backtrace, but [Go doesn't include cgo debug information on macOS](https://github.com/golang/go/issues/6942). I've also tried building against libc++ on Linux, where debug information would be available, but I can't seem to trigger the bug there. In any case, this PR both fixes the segfault in CockroachDB and fixes the warnings reported by ubsan. Closes https://github.com/facebook/rocksdb/pull/2347 Differential Revision: D5108596 Pulled By: yiwu-arbug fbshipit-source-id: bd5e4323b2ce915ed4fe78e123cb8996aec75a00
-
由 Changli Gao 提交于
Summary: Closes https://github.com/facebook/rocksdb/pull/3007 Differential Revision: D6077701 Pulled By: yiwu-arbug fbshipit-source-id: a6fd5b8a23f4feb1660b9ce027f651a7e90352b3
-
由 codeeply 提交于
Summary: When I impl my own comparator, and build in release mode. The following compile error occurs. undefined reference to `typeinfo for rocksdb::Comparator' This fix allows users build with RTTI off when has their own comparator. Closes https://github.com/facebook/rocksdb/pull/3008 Differential Revision: D6077354 Pulled By: yiwu-arbug fbshipit-source-id: 914c26dbab72f0ad1f0e15f8666a3fb2f10bfed8
-
- 17 10月, 2017 2 次提交
-
-
由 Yi Wu 提交于
Summary: * make `checksum_type_string_map` available for lite * comment out `FilesPerLevel` in lite mode. * travis and legocastle lite build also build `all` target and run tests Closes https://github.com/facebook/rocksdb/pull/3015 Differential Revision: D6069822 Pulled By: yiwu-arbug fbshipit-source-id: 9fe92ac220e711e9e6ed4e921bd25ef4314796a0
-
由 Andrew Kryczka 提交于
Summary: print throughput in MB/s upon finishing randomtransaction benchmark Closes https://github.com/facebook/rocksdb/pull/3016 Differential Revision: D6070426 Pulled By: ajkr fbshipit-source-id: 69df43beed4c374a36d826e761ca3a83e1fdcbf5
-
- 14 10月, 2017 2 次提交
-
-
由 Yi Wu 提交于
Summary: print more table_options to info log Closes https://github.com/facebook/rocksdb/pull/3003 Differential Revision: D6054490 Pulled By: yiwu-arbug fbshipit-source-id: 8e6f96e08bdc906077b6c62ade419d7cb739110f
-
由 Yi Wu 提交于
Summary: simply move BlobFile definition from blob_db_impl.h to blob_file.h. Closes https://github.com/facebook/rocksdb/pull/3002 Differential Revision: D6050143 Pulled By: yiwu-arbug fbshipit-source-id: a8fb6e094fe39bdeace6279569834bc65aa64a34
-
- 13 10月, 2017 1 次提交
-
-
由 Yi Wu 提交于
Summary: Allow `std::move(pinnable_slice)`. Closes https://github.com/facebook/rocksdb/pull/2997 Differential Revision: D6036782 Pulled By: yiwu-arbug fbshipit-source-id: 583fb0419a97e437ff530f4305822341cd3381fa
-