- 14 4月, 2022 1 次提交
-
-
由 Peter Dillinger 提交于
Summary: Tends to fill up /dev/shm Pull Request resolved: https://github.com/facebook/rocksdb/pull/9837 Test Plan: Some manual testing Reviewed By: hx235 Differential Revision: D35627568 Pulled By: pdillinger fbshipit-source-id: 22710f7b10bc287570475dae42318dd346f78db9
-
- 13 4月, 2022 1 次提交
-
-
由 Peter Dillinger 提交于
Summary: Especially after updating to C++17, I don't see a compelling case for *requiring* any folly components in RocksDB. I was able to purge the existing hard dependencies, and it can be quite difficult to strip out non-trivial components from folly for use in RocksDB. (The prospect of doing that on F14 has changed my mind on the best approach here.) But this change creates an optional integration where we can plug in components from folly at compile time, starting here with F14FastMap to replace std::unordered_map when possible (probably no public APIs for example). I have replaced the biggest CPU users of std::unordered_map with compile-time pluggable UnorderedMap which will use F14FastMap when USE_FOLLY is set. USE_FOLLY is always set in the Meta-internal buck build, and a simulation of that is in the Makefile for public CI testing. A full folly build is not needed, but checking out the full folly repo is much simpler for getting the dependency, and anything else we might want to optionally integrate in the future. Some picky details: * I don't think the distributed mutex stuff is actually used, so it was easy to remove. * I implemented an alternative to `folly::constexpr_log2` (which is much easier in C++17 than C++11) so that I could pull out the hard dependencies on `ConstexprMath.h` * I had to add noexcept move constructors/operators to some types to make F14's complainUnlessNothrowMoveAndDestroy check happy, and I added a macro to make that easier in some common cases. * Updated Meta-internal buck build to use folly F14Map (always) No updates to HISTORY.md nor INSTALL.md as this is not (yet?) considered a production integration for open source users. Pull Request resolved: https://github.com/facebook/rocksdb/pull/9546 Test Plan: CircleCI tests updated so that a couple of them use folly. Most internal unit & stress/crash tests updated to use Meta-internal latest folly. (Note: they should probably use buck but they currently use Makefile.) Example performance improvement: when filter partitions are pinned in cache, they are tracked by PartitionedFilterBlockReader::filter_map_ and we can build a test that exercises that heavily. Build DB with ``` TEST_TMPDIR=/dev/shm/rocksdb ./db_bench -benchmarks=fillrandom -num=10000000 -disable_wal=1 -write_buffer_size=30000000 -bloom_bits=16 -compaction_style=2 -fifo_compaction_max_table_files_size_mb=10000 -fifo_compaction_allow_compaction=0 -partition_index_and_filters ``` and test with (simultaneous runs with & without folly, ~20 times each to see convergence) ``` TEST_TMPDIR=/dev/shm/rocksdb ./db_bench_folly -readonly -use_existing_db -benchmarks=readrandom -num=10000000 -bloom_bits=16 -compaction_style=2 -fifo_compaction_max_table_files_size_mb=10000 -fifo_compaction_allow_compaction=0 -partition_index_and_filters -duration=40 -pin_l0_filter_and_index_blocks_in_cache ``` Average ops/s no folly: 26229.2 Average ops/s with folly: 26853.3 (+2.4%) Reviewed By: ajkr Differential Revision: D34181736 Pulled By: pdillinger fbshipit-source-id: ffa6ad5104c2880321d8a1aa7187e00ab0d02e94
-
- 12 4月, 2022 2 次提交
-
-
由 mrambacher 提交于
Summary: Added a Plugin class to the ObjectRegistry. Enabled compile-time and program-time addition of plugins to the Registry. Pull Request resolved: https://github.com/facebook/rocksdb/pull/7949 Reviewed By: mrambacher Differential Revision: D33517674 Pulled By: pdillinger fbshipit-source-id: c3e3270aab76a489bfa9e85d78cdfca951912557
-
由 gitbw95 提交于
Summary: ### **Summary:** When both LRU Cache and CompressedSecondaryCache are configured together, there possibly are some data blocks double cached. **Changes include:** 1. Update IS_PROMOTED to IS_IN_SECONDARY_CACHE to prevent confusions. 2. This PR updates SecondaryCacheResultHandle and use IsErasedFromSecondaryCache to determine whether the handle is erased in the secondary cache. Then, the caller can determine whether to SetIsInSecondaryCache(). 3. Rename LRUSecondaryCache to CompressedSecondaryCache. Pull Request resolved: https://github.com/facebook/rocksdb/pull/9747 Test Plan: **Test Scripts:** 1. Populate a DB. The on disk footprint is 482 MB. The data is set to be 50% compressible, so the total decompressed size is expected to be 964 MB. ./db_bench --benchmarks=fillrandom --num=10000000 -db=/db_bench_1 2. overwrite it to a stable state: ./db_bench --benchmarks=overwrite,stats --num=10000000 -use_existing_db -duration=10 --benchmark_write_rate_limit=2000000 -db=/db_bench_1 4. Run read tests with diffeernt cache setting: T1: ./db_bench --benchmarks=seekrandom,stats --threads=16 --num=10000000 -use_existing_db -duration=120 --benchmark_write_rate_limit=52000000 -use_direct_reads --cache_size=520000000 --statistics -db=/db_bench_1 T2: ./db_bench --benchmarks=seekrandom,stats --threads=16 --num=10000000 -use_existing_db -duration=120 --benchmark_write_rate_limit=52000000 -use_direct_reads --cache_size=320000000 -compressed_secondary_cache_size=400000000 --statistics -use_compressed_secondary_cache -db=/db_bench_1 T3: ./db_bench --benchmarks=seekrandom,stats --threads=16 --num=10000000 -use_existing_db -duration=120 --benchmark_write_rate_limit=52000000 -use_direct_reads --cache_size=520000000 -compressed_secondary_cache_size=400000000 --statistics -use_compressed_secondary_cache -db=/db_bench_1 T4: ./db_bench --benchmarks=seekrandom,stats --threads=16 --num=10000000 -use_existing_db -duration=120 --benchmark_write_rate_limit=52000000 -use_direct_reads --cache_size=20000000 -compressed_secondary_cache_size=500000000 --statistics -use_compressed_secondary_cache -db=/db_bench_1 **Before this PR** | Cache Size | Compressed Secondary Cache Size | Cache Hit Rate | |------------|-------------------------------------|----------------| |520 MB | 0 MB | 85.5% | |320 MB | 400 MB | 96.2% | |520 MB | 400 MB | 98.3% | |20 MB | 500 MB | 98.8% | **Before this PR** | Cache Size | Compressed Secondary Cache Size | Cache Hit Rate | |------------|-------------------------------------|----------------| |520 MB | 0 MB | 85.5% | |320 MB | 400 MB | 99.9% | |520 MB | 400 MB | 99.9% | |20 MB | 500 MB | 99.2% | Reviewed By: anand1976 Differential Revision: D35117499 Pulled By: gitbw95 fbshipit-source-id: ea2657749fc13efebe91a8a1b56bc61d6a224a12
-
- 07 4月, 2022 1 次提交
-
-
由 Peter Dillinger 提交于
Summary: Pull Request resolved: https://github.com/facebook/rocksdb/pull/9803 Only use Meta-internal version now. precommit_checker.py also now obsolete Bring back `make commit_prereq` in follow-up work Reviewed By: jay-zhuang Differential Revision: D35372283 fbshipit-source-id: 7428438ca51f878802c301d0d5591675e551a113
-
- 06 4月, 2022 1 次提交
-
-
由 Peter Dillinger 提交于
Summary: Various renaming and fixes to get rid of remaining uses of "backupable" which is terminology leftover from the original, flawed design of BackupableDB. Now any DB can be backed up, using BackupEngine. Pull Request resolved: https://github.com/facebook/rocksdb/pull/9792 Test Plan: CI Reviewed By: ajkr Differential Revision: D35334386 Pulled By: pdillinger fbshipit-source-id: 2108a42b4575c8cccdfd791c549aae93ec2f3329
-
- 05 4月, 2022 1 次提交
-
-
由 Jay Zhuang 提交于
Summary: Make `commit_prereq` work and a few other improvements: * Remove gcc 481 and gcc5xx which are no longer supported * Remove platform007 which is gone * `make clean` work for both mac and linux * `precommit_checker.py` to python3 Pull Request resolved: https://github.com/facebook/rocksdb/pull/9797 Test Plan: `make commit_prereq` Reviewed By: ajkr Differential Revision: D35338536 Pulled By: jay-zhuang fbshipit-source-id: 1e159962ab9d31c43c4b85de7d0f582d3e881ffe
-
- 30 3月, 2022 1 次提交
-
-
由 Jack Robison 提交于
Summary: Zlib (https://www.zlib.net/) has been updated to 1.2.12 due to CVE-2018-25032 - https://nvd.nist.gov/vuln/detail/CVE-2018-25032 - https://github.com/madler/zlib/issues/605 The source .tar.gz is no longer available, and the Makefile for rocksdb now fails as a result. This PR updates the dependency to the newer (and available) version, 1.2.12 Pull Request resolved: https://github.com/facebook/rocksdb/pull/9764 Reviewed By: ajkr Differential Revision: D35220367 Pulled By: jay-zhuang fbshipit-source-id: 1f68ff8f048a6dba42077f048ac143468f0e2478
-
- 25 3月, 2022 1 次提交
-
-
由 Jay Zhuang 提交于
Summary: Seems clean-rocksjava and clean-rocks conflict. Also remove unnecessary step in java CI build, otherwise it will rebuild the code again as java make sample do clean up first. Pull Request resolved: https://github.com/facebook/rocksdb/pull/9710 Test Plan: `make rocksdbjava && make clean` should return success Reviewed By: riversand963 Differential Revision: D35122872 Pulled By: jay-zhuang fbshipit-source-id: 2a15b83e7a763c0fc0e42e1f35aac9551f951ece
-
- 22 3月, 2022 1 次提交
-
-
由 Jay Zhuang 提交于
Summary: Pull Request resolved: https://github.com/facebook/rocksdb/pull/9714 Reviewed By: ajkr Differential Revision: D34983675 Pulled By: jay-zhuang fbshipit-source-id: 0252ec6ee38a0b960df4c92791c7c2bcbfba5ad8
-
- 19 3月, 2022 1 次提交
-
-
由 Peter Dillinger 提交于
Summary: The goal of this change is to allow changes to the "current" (in FileSystem) file temperatures to feed back into DB metadata, so that they can inform decisions and stats reporting. In part because of modular code factoring, it doesn't seem easy to do this automagically, where opening an SST file and observing current Temperature different from expected would trigger a change in metadata and DB manifest write (essentially giving the deep read path access to the write path). It is also difficult to do this while the DB is open because of the limitations of LogAndApply. This change allows updating file temperature metadata on a closed DB using an experimental utility function UpdateManifestForFilesState() or `ldb update_manifest --update_temperatures`. This should suffice for "migration" scenarios where outside tooling has placed or re-arranged DB files into a (different) tiered configuration without going through RocksDB itself (currently, only compaction can change temperature metadata). Some details: * Refactored and added unit test for `ldb unsafe_remove_sst_file` because of shared functionality * Pulled in autovector.h changes from https://github.com/facebook/rocksdb/issues/9546 to fix SuperVersionContext move constructor (related to an older draft of this change) Possible follow-up work: * Support updating manifest with file checksums, such as when a new checksum function is used and want existing DB metadata updated for it. * It's possible that for some repair scenarios, lighter weight than full repair, we might want to support UpdateManifestForFilesState() to modify critical file details like size or checksum using same algorithm. But let's make sure these are differentiated from modifying file details in ways that don't suspect corruption (or require extreme trust). Pull Request resolved: https://github.com/facebook/rocksdb/pull/9683 Test Plan: unit tests added Reviewed By: jay-zhuang Differential Revision: D34798828 Pulled By: pdillinger fbshipit-source-id: cfd83e8fb10761d8c9e7f9c020d68c9106a95554
-
- 17 3月, 2022 2 次提交
-
-
由 Yanqin Jin 提交于
Summary: Test only, no change to functionality. Extremely low risk of library regression. Update test key generation by maintaining existing and non-existing keys. Update db_crashtest.py to drive multiops_txn stress test for both write-committed and write-prepared. Add a make target 'blackbox_crash_test_with_multiops_txn'. Running the following commands caught the bug exposed in https://github.com/facebook/rocksdb/issues/9571. ``` $rm -rf /tmp/rocksdbtest/* $./db_stress -progress_reports=0 -test_multi_ops_txns -use_txn -clear_column_family_one_in=0 \ -column_families=1 -writepercent=0 -delpercent=0 -delrangepercent=0 -customopspercent=60 \ -readpercent=20 -prefixpercent=0 -iterpercent=20 -reopen=0 -ops_per_thread=1000 -ub_a=10000 \ -ub_c=100 -destroy_db_initially=0 -key_spaces_path=/dev/shm/key_spaces_desc -threads=32 -read_fault_one_in=0 $./db_stress -progress_reports=0 -test_multi_ops_txns -use_txn -clear_column_family_one_in=0 -column_families=1 -writepercent=0 -delpercent=0 -delrangepercent=0 -customopspercent=60 -readpercent=20 \ -prefixpercent=0 -iterpercent=20 -reopen=0 -ops_per_thread=1000 -ub_a=10000 -ub_c=100 -destroy_db_initially=0 \ -key_spaces_path=/dev/shm/key_spaces_desc -threads=32 -read_fault_one_in=0 ``` Running the following command caught a bug which will be fixed in https://github.com/facebook/rocksdb/issues/9648 . ``` $TEST_TMPDIR=/dev/shm make blackbox_crash_test_with_multiops_wc_txn ``` Pull Request resolved: https://github.com/facebook/rocksdb/pull/9568 Reviewed By: jay-zhuang Differential Revision: D34308154 Pulled By: riversand963 fbshipit-source-id: 99ff1b65c19b46c471d2f2d3b47adcd342a1b9e7
-
由 Peter Dillinger 提交于
Summary: some Makefile refactoring to support Meta-internal workflows, and add a basic crash_test flow to CircleCI Pull Request resolved: https://github.com/facebook/rocksdb/pull/9702 Test Plan: CI Reviewed By: riversand963 Differential Revision: D34934315 Pulled By: pdillinger fbshipit-source-id: 67f17280096d8968d8e44459293f72fb6fe339f3
-
- 09 3月, 2022 1 次提交
-
-
由 Yanqin Jin 提交于
Summary: Pull Request resolved: https://github.com/facebook/rocksdb/pull/9629 Pessimistic transactions use pessimistic concurrency control, i.e. locking. Keys are locked upon first operation that writes the key or has the intention of writing. For example, `PessimisticTransaction::Put()`, `PessimisticTransaction::Delete()`, `PessimisticTransaction::SingleDelete()` will write to or delete a key, while `PessimisticTransaction::GetForUpdate()` is used by application to indicate to RocksDB that the transaction has the intention of performing write operation later in the same transaction. Pessimistic transactions support two-phase commit (2PC). A transaction can be `Prepared()`'ed and then `Commit()`. The prepare phase is similar to a promise: once `Prepare()` succeeds, the transaction has acquired the necessary resources to commit. The resources include locks, persistence of WAL, etc. Write-committed transaction is the default pessimistic transaction implementation. In RocksDB write-committed transaction, `Prepare()` will write data to the WAL as a prepare section. `Commit()` will write a commit marker to the WAL and then write data to the memtables. While writing to the memtables, different keys in the transaction's write batch will be assigned different sequence numbers in ascending order. Until commit/rollback, the transaction holds locks on the keys so that no other transaction can write to the same keys. Furthermore, the keys' sequence numbers represent the order in which they are committed and should be made visible. This is convenient for us to implement support for user-defined timestamps. Since column families with and without timestamps can co-exist in the same database, a transaction may or may not involve timestamps. Based on this observation, we add two optional members to each `PessimisticTransaction`, `read_timestamp_` and `commit_timestamp_`. If no key in the transaction's write batch has timestamp, then setting these two variables do not have any effect. For the rest of this commit, we discuss only the cases when these two variables are meaningful. read_timestamp_ is used mainly for validation, and should be set before first call to `GetForUpdate()`. Otherwise, the latter will return non-ok status. `GetForUpdate()` calls `TryLock()` that can verify if another transaction has written the same key since `read_timestamp_` till this call to `GetForUpdate()`. If another transaction has indeed written the same key, then validation fails, and RocksDB allows this transaction to refine `read_timestamp_` by increasing it. Note that a transaction can still use `Get()` with a different timestamp to read, but the result of the read should not be used to determine data that will be written later. commit_timestamp_ must be set after finishing writing and before transaction commit. This applies to both 2PC and non-2PC cases. In the case of 2PC, it's usually set after prepare phase succeeds. We currently require that the commit timestamp be chosen after all keys are locked. This means we disallow the `TransactionDB`-level APIs if user-defined timestamp is used by the transaction. Specifically, calling `PessimisticTransactionDB::Put()`, `PessimisticTransactionDB::Delete()`, `PessimisticTransactionDB::SingleDelete()`, etc. will return non-ok status because they specify timestamps before locking the keys. Users are also prompted to use the `Transaction` APIs when they receive the non-ok status. Reviewed By: ltamasi Differential Revision: D31822445 fbshipit-source-id: b82abf8e230216dc89cc519564a588224a88fd43
-
- 08 3月, 2022 1 次提交
-
-
由 Adam Retter 提交于
Summary: Addresses the problems described in https://github.com/facebook/rocksdb/pull/9254#issuecomment-1054598516 and https://github.com/facebook/rocksdb/pull/9254#issuecomment-1059574837 that have blocked a RocksJava release **NOTE** Also needs to be ported to 6.29.fb branch. Pull Request resolved: https://github.com/facebook/rocksdb/pull/9662 Reviewed By: ajkr Differential Revision: D34689200 Pulled By: pdillinger fbshipit-source-id: c62fe34c54f05be5a00ee1daec8ec7454baa5eb8
-
- 04 3月, 2022 1 次提交
-
-
由 Jay Zhuang 提交于
Summary: Improve the CI build speed: - split the macos tests to 2 parallel jobs - split tsan tests to 2 parallel jobs - move non-shm tests to nightly build - slow jobs use lager machine - fast jobs use smaller machine - add microbench to no-test jobs - add run-microbench to nightly build Pull Request resolved: https://github.com/facebook/rocksdb/pull/9605 Reviewed By: riversand963 Differential Revision: D34358982 Pulled By: jay-zhuang fbshipit-source-id: d5091b3f4ef6d25c5c37920fb614f3342ee60e4a
-
- 24 2月, 2022 1 次提交
-
-
由 Bo Wang 提交于
Summary: **Summary:** RocksDB uses a block cache to reduce IO and make queries more efficient. The block cache is based on the LRU algorithm (LRUCache) and keeps objects containing uncompressed data, such as Block, ParsedFullFilterBlock etc. It allows the user to configure a second level cache (rocksdb::SecondaryCache) to extend the primary block cache by holding items evicted from it. Some of the major RocksDB users, like MyRocks, use direct IO and would like to use a primary block cache for uncompressed data and a secondary cache for compressed data. The latter allows us to mitigate the loss of the Linux page cache due to direct IO. This PR includes a concrete implementation of rocksdb::SecondaryCache that integrates with compression libraries such as LZ4 and implements an LRU cache to hold compressed blocks. Pull Request resolved: https://github.com/facebook/rocksdb/pull/9518 Test Plan: In this PR, the lru_secondary_cache_test.cc includes the following tests: 1. The unit tests for the secondary cache with either compression or no compression, such as basic tests, fails tests. 2. The integration tests with both primary cache and this secondary cache . **Follow Up:** 1. Statistics (e.g. compression ratio) will be added in another PR. 2. Once this implementation is ready, I will do some shadow testing and benchmarking with UDB to measure the impact. Reviewed By: anand1976 Differential Revision: D34430930 Pulled By: gitbw95 fbshipit-source-id: 218d78b672a2f914856d8a90ff32f2f5b5043ded
-
- 18 2月, 2022 2 次提交
-
-
由 Alan Paxton 提交于
Summary: Extend the plugin architecture to allow for the inclusion, building and testing of Java and JNI components of a plugin. This will cause the JAR built by `$ make rocksdbjava` to include the extra functionality provided by the plugin, and will cause `$ make jtest` to add the java tests provided by the plugin to the tests built and run by Java testing. The plugin's `<plugin>.mk` file can define: ``` <plugin>_JNI_NATIVE_SOURCES <plugin>_NATIVE_JAVA_CLASSES <plugin>_JAVA_TESTS ``` The plugin should provide java/src, java/test and java/rocksjni directories. When a plugin is required to be build it must be named in the ROCKSDB_PLUGINS environment variable (as per the plugin architecture). This now has the effect of adding the files specified by the above definitions to the appropriate parts of the build. An example of a plugin with a Java component can be found as part of the hdfs plugin in https://github.com/riversand963/rocksdb-hdfs-env - at the time of writing the Java part of this fails tests, and needs a little work to complete, but it builds correctly under the plugin model. Pull Request resolved: https://github.com/facebook/rocksdb/pull/9575 Reviewed By: hx235 Differential Revision: D34253948 Pulled By: riversand963 fbshipit-source-id: b3dde5da06f3d3c25c54246892097ae2a369b42d
-
由 Adam Retter 提交于
Summary: See https://github.com/facebook/rocksdb/issues/9388#issuecomment-1029583789 Pull Request resolved: https://github.com/facebook/rocksdb/pull/9500 Reviewed By: pdillinger Differential Revision: D34114687 Pulled By: jay-zhuang fbshipit-source-id: 22129d99ccd0dba7e8f1b263ddc5520d939641bf
-
- 17 2月, 2022 1 次提交
-
-
由 Andrew Kryczka 提交于
Summary: Users can set the priority for file reads associated with their operation by setting `ReadOptions::rate_limiter_priority` to something other than `Env::IO_TOTAL`. Rate limiting `VerifyChecksum()` and `VerifyFileChecksums()` is the motivation for this PR, so it also includes benchmarks and minor bug fixes to get that working. `RandomAccessFileReader::Read()` already had support for rate limiting compaction reads. I changed that rate limiting to be non-specific to compaction, but rather performed according to the passed in `Env::IOPriority`. Now the compaction read rate limiting is supported by setting `rate_limiter_priority = Env::IO_LOW` on its `ReadOptions`. There is no default value for the new `Env::IOPriority` parameter to `RandomAccessFileReader::Read()`. That means this PR goes through all callers (in some cases multiple layers up the call stack) to find a `ReadOptions` to provide the priority. There are TODOs for cases I believe it would be good to let user control the priority some day (e.g., file footer reads), and no TODO in cases I believe it doesn't matter (e.g., trace file reads). The API doc only lists the missing cases where a file read associated with a provided `ReadOptions` cannot be rate limited. For cases like file ingestion checksum calculation, there is no API to provide `ReadOptions` or `Env::IOPriority`, so I didn't count that as missing. Pull Request resolved: https://github.com/facebook/rocksdb/pull/9424 Test Plan: - new unit tests - new benchmarks on ~50MB database with 1MB/s read rate limit and 100ms refill interval; verified with strace reads are chunked (at 0.1MB per chunk) and spaced roughly 100ms apart. - setup command: `./db_bench -benchmarks=fillrandom,compact -db=/tmp/testdb -target_file_size_base=1048576 -disable_auto_compactions=true -file_checksum=true` - benchmarks command: `strace -ttfe pread64 ./db_bench -benchmarks=verifychecksum,verifyfilechecksums -use_existing_db=true -db=/tmp/testdb -rate_limiter_bytes_per_sec=1048576 -rate_limit_bg_reads=1 -rate_limit_user_ops=true -file_checksum=true` - crash test using IO_USER priority on non-validation reads with https://github.com/facebook/rocksdb/issues/9567 reverted: `python3 tools/db_crashtest.py blackbox --max_key=1000000 --write_buffer_size=524288 --target_file_size_base=524288 --level_compaction_dynamic_level_bytes=true --duration=3600 --rate_limit_bg_reads=true --rate_limit_user_ops=true --rate_limiter_bytes_per_sec=10485760 --interval=10` Reviewed By: hx235 Differential Revision: D33747386 Pulled By: ajkr fbshipit-source-id: a2d985e97912fba8c54763798e04f006ccc56e0c
-
- 16 2月, 2022 1 次提交
-
-
由 Peter Dillinger 提交于
Summary: After https://github.com/facebook/rocksdb/issues/9515 added a unique_ptr to Status, we see some warnings-as-error in some internal builds like this: ``` stderr: rocksdb/src/db/compaction/compaction_job.cc:2839:7: error: offset of on non-standard-layout type 'struct CompactionServiceResult' [-Werror,-Winvalid-offsetof] {offsetof(struct CompactionServiceResult, status), ^ ~~~~~~ ``` I see three potential solutions to resolving this: * Expand our use of an idiom that works around the warning (see offset_of functions removed in this change, inspired by https://gist.github.com/graphitemaster/494f21190bb2c63c5516) However, this construction is invoking undefined behavior that assumes consistent layout with no compiler-introduced indirection. A compiler incompatible with our assumptions will likely compile the code and exhibit undefined behavior. * Migrate to something in place of offset, like a function mapping CompactionServiceResult* to Status* (for the `status` field). This might be required in the long term. * **Selected:** Use our new C++17 dependency to use offsetof in a well-defined way when the compiler allows it. From a comment on https://gist.github.com/graphitemaster/494f21190bb2c63c5516: > A final note: in C++17, offsetof is conditionally supported, which > means that you can use it on any type (not just standard layout > types) and the compiler will error if it can't compile it correctly. > That appears to be the best option if you can live with C++17 and > don't need constexpr support. The C++17 semantics are confirmed on https://en.cppreference.com/w/cpp/types/offsetof, so we can suppress the warning as long as we accept that we might run into a compiler that rejects the code, and at that point we will find a solution, such as the more intrusive "migrate" solution above. Although this is currently only showing in our buck build, it will surely show up also with make and cmake, so I have updated those configurations as well. Also in the buck build, -Wno-expansion-to-defined does not appear to be needed anymore (both current compiler configurations) so I removed it. Pull Request resolved: https://github.com/facebook/rocksdb/pull/9563 Test Plan: Tried out buck builds with both current compiler configurations Reviewed By: riversand963 Differential Revision: D34220931 Pulled By: pdillinger fbshipit-source-id: d39436008259bd1eaaa87c77be69fb2a5b559e1f
-
- 15 2月, 2022 1 次提交
-
-
由 Peter Dillinger 提交于
Summary: * Fix LIB_MODE=shared for Meta-internal builds (use PIC libraries appropriately) * Fix gnu_parallel to recognize CircleCI and Travis builds as not connected to a terminal (was previously relying on the `| cat_ignore_eagain` stuff for Ubuntu 16). This problem could cause timeouts that should be 10m to balloon to 5h. Pull Request resolved: https://github.com/facebook/rocksdb/pull/9553 Test Plan: manual and CI Reviewed By: jay-zhuang Differential Revision: D34182886 Pulled By: pdillinger fbshipit-source-id: e95fd8002d94c8dc414bae1975e4fd348589f2b5
-
- 08 2月, 2022 1 次提交
-
-
由 Peter Dillinger 提交于
Summary: ... seen only in internal clang-analyze runs after https://github.com/facebook/rocksdb/issues/9481 * Mostly, this works around falsely reported leaks by using std::unique_ptr in some places where clang-analyze was getting confused. (I didn't see any changes in C++17 that could make our Status implementation leak memory.) * Also fixed SetBGError returning address of a stack variable. * Also fixed another false null deref report by adding an assert. Also, use SKIP_LINK=1 to speed up `make analyze` Pull Request resolved: https://github.com/facebook/rocksdb/pull/9515 Test Plan: Was able to reproduce the reported errors locally and verify they're fixed (except SetBGError). Otherwise, existing tests Reviewed By: hx235 Differential Revision: D34054630 Pulled By: pdillinger fbshipit-source-id: 38600ef3da75ddca307dff96b7a1a523c2885c2e
-
- 05 2月, 2022 1 次提交
-
-
由 Peter Dillinger 提交于
Summary: Drop support for some old compilers by requiring C++17 standard (or higher). See https://github.com/facebook/rocksdb/issues/9388 First modification based on this is to remove some conditional compilation in slice.h (also better for ODR) Also in this PR: * Fix some Makefile formatting that seems to affect ASSERT_STATUS_CHECKED config in some cases * Add c_test to NON_PARALLEL_TEST in Makefile * Fix a clang-analyze reported "potential leak" in lru_cache_test * Better "compatibility" definition of DEFINE_uint32 for old versions of gflags * Fix a linking problem with shared libraries in Makefile (`./random_test: error while loading shared libraries: librocksdb.so.6.29: cannot open shared object file: No such file or directory`) * Always set ROCKSDB_SUPPORT_THREAD_LOCAL and use thread_local (from C++11) * TODO in later PR: clean up that obsolete flag * Fix a cosmetic typo in c.h (https://github.com/facebook/rocksdb/issues/9488) Pull Request resolved: https://github.com/facebook/rocksdb/pull/9481 Test Plan: CircleCI config substantially updated. * Upgrade to latest Ubuntu images for each release * Generally prefer Ubuntu 20, but keep a couple Ubuntu 16 builds with oldest supported compilers, to ensure compatibility * Remove .circleci/cat_ignore_eagain except for Ubuntu 16 builds, because this is to work around a kernel bug that should not affect anything but Ubuntu 16. * Remove designated gcc-9 build, because the default linux build now uses GCC 9 from Ubuntu 20. * Add some `apt-key add` to fix some apt "couldn't be verified" errors * Generally drop SKIP_LINK=1; work-around no longer needed * Generally `add-apt-repository` before `apt-get update` as manual testing indicated the reverse might not work. Travis: * Use gcc-7 by default (remove specific gcc-7 and gcc-4.8 builds) * TODO in later PR: fix s390x "Assembler messages: Error: invalid switch -march=z14" failure AppVeyor: * Completely dropped because we are dropping VS2015 support and CircleCI covers VS >= 2017 Also local testing with old gflags (out of necessity when using ROCKSDB_NO_FBCODE=1). Reviewed By: mrambacher Differential Revision: D33946377 Pulled By: pdillinger fbshipit-source-id: ae077c823905b45370a26c0103ada119459da6c1
-
- 25 1月, 2022 3 次提交
-
-
由 Yanqin Jin 提交于
Summary: This PR moves RADOS support from RocksDB repo to a separate repo. The new (temporary?) repo in this PR serves as an example before we finalize the decision on where and who to host RADOS support. At this point, people can start from the example repo and fork. The goal is to include this commit in RocksDB 7.0 release. Reference: https://github.com/ajkr/dedupfs by ajkr Pull Request resolved: https://github.com/facebook/rocksdb/pull/9206 Test Plan: Follow instructions in https://github.com/riversand963/rocksdb-rados-env/blob/main/README.md and build test binary `env_librados_test` and run it. Also, make check Reviewed By: ajkr Differential Revision: D33751690 Pulled By: riversand963 fbshipit-source-id: 30466c62afa9e4619847a48567ed158e62835e35
-
由 Yanqin Jin 提交于
Summary: This PR is one proposal to resolve https://github.com/facebook/rocksdb/issues/9382. Looking at the code, I can't think of a reason why rdb is an internal component of RocksDB: it does not require any header files NOT in `include/rocksdb`. It's a better idea to host it somewhere else. Plus, rdb requires python2 which is not supported any more. No fixes or improvements will be made, even for potential security bugs (https://www.python.org/doc/sunset-python-2/). Pull Request resolved: https://github.com/facebook/rocksdb/pull/9399 Test Plan: make check Reviewed By: ajkr Differential Revision: D33641965 Pulled By: riversand963 fbshipit-source-id: 2a6a74693e5de36834f355e41d6865db206af48b
-
由 Yanqin Jin 提交于
Summary: This PR moves HDFS support from RocksDB repo to a separate repo. The new (temporary?) repo in this PR serves as an example before we finalize the decision on where and who to host hdfs support. At this point, people can start from the example repo and fork. Java/JNI is not included yet, and needs to be done later if necessary. The goal is to include this commit in RocksDB 7.0 release. Reference: https://github.com/ajkr/dedupfs by ajkr Pull Request resolved: https://github.com/facebook/rocksdb/pull/9170 Test Plan: Follow the instructions in https://github.com/riversand963/rocksdb-hdfs-env/blob/master/README.md. Build and run db_bench and db_stress. make check Reviewed By: ajkr Differential Revision: D33751662 Pulled By: riversand963 fbshipit-source-id: 22b4db7f31762ed417a20239f5a08dcd1696244f
-
- 13 1月, 2022 1 次提交
-
-
由 Eric Thérond 提交于
Summary: Fixes facebook/rocksdb#7720 Updated Makefile with flags to define target architecture when compiling/linking, and added goal `rocksdbjavastaticosxub` to build a OS X Universal Binary native library. Pull Request resolved: https://github.com/facebook/rocksdb/pull/9254 Reviewed By: mrambacher Differential Revision: D33551160 Pulled By: pdillinger fbshipit-source-id: 9ce9962e03aacf55014545a6cdf638b5b14b8fa9
-
- 23 12月, 2021 1 次提交
-
-
由 Adam Retter 提交于
Summary: * Added Docker build environment for RocksJava on s390x * Cache alignment size for s390x was incorrectly calculated on gcc 6.4.0 * Tighter control over which installed version of Java is used is required - build now correctly adheres to `JAVA_HOME` if it is set * Alpine build scripts should be used on Alpine (previously CentOS script worked by falling through to minimal gcc version) Pull Request resolved: https://github.com/facebook/rocksdb/pull/9321 Reviewed By: mrambacher Differential Revision: D33259624 Pulled By: jay-zhuang fbshipit-source-id: d791a5150581344925c3c3f9cbb9a3622d63b3b6
-
- 22 12月, 2021 1 次提交
-
-
由 Andreas Hindborg 提交于
Summary: Fix a bug introduced by https://github.com/facebook/rocksdb/issues/9198. The bug is triggered when a plugin does not provide any pkg-config requirements. Pull Request resolved: https://github.com/facebook/rocksdb/pull/9238 Reviewed By: riversand963 Differential Revision: D32771406 Pulled By: ajkr fbshipit-source-id: 79301871a8bf4e624d5e5eb9d219d7f13948c64d
-
- 17 12月, 2021 1 次提交
-
-
由 mrambacher 提交于
Summary: - Make MemoryAllocator and its implementations into a Customizable class. - Added a "DefaultMemoryAllocator" which uses new and delete - Added a "CountedMemoryAllocator" that counts the number of allocs and free - Updated the existing tests to use these new allocators - Changed the memkind allocator test into a generic test that can test the various allocators. - Added tests for creating all of the allocators - Added tests to verify/create the JemallocNodumpAllocator using its options. Pull Request resolved: https://github.com/facebook/rocksdb/pull/8980 Reviewed By: zhichao-cao Differential Revision: D32990403 Pulled By: mrambacher fbshipit-source-id: 6fdfe8218c10dd8dfef34344a08201be1fa95c76
-
- 10 12月, 2021 1 次提交
-
-
由 Si Ke 提交于
Summary: Closes https://github.com/facebook/rocksdb/pull/7737 Pull Request resolved: https://github.com/facebook/rocksdb/pull/9231 Reviewed By: hx235 Differential Revision: D32978332 Pulled By: pdillinger fbshipit-source-id: b28900b685d60c668529a90dbaa8e1b357b28f76
-
- 01 12月, 2021 2 次提交
-
-
由 Yanqin Jin 提交于
Summary: Not a CMake expert, and the current CMake build support added by this PR is unlikely the best way of doing it. Sending out the PR to demonstrate it can work. Pull Request resolved: https://github.com/facebook/rocksdb/pull/9214 Test Plan: Will need to update https://github.com/ajkr/dedupfs with CMake build. Also, PR https://github.com/facebook/rocksdb/issues/9170 and PR https://github.com/facebook/rocksdb/issues/9206 both include CMake support for their plugins, and can be used as a proof of concept. Reviewed By: ajkr Differential Revision: D32738273 Pulled By: riversand963 fbshipit-source-id: da87fb4377c716bbbd577a69763b48d22483f845
-
由 Andreas Hindborg 提交于
Summary: This patch fixes an issue that occur when dependencies of plugins are not installed to the same prefix as librocksdb. Because plugin dependencies are declared in the `Libs` field of rocksdb.pc, programs that link against librocksdb with `pkg-config --libs rocksdb` will link with `-L` flag for the path of librocksdb only. This patch allows plugin dependencies to be declared in the `Requires` field of rocksdb.pc, so that pkg-config will correctly provide `-L` flags for dependencies of plugins that are installed in other locations. Pull Request resolved: https://github.com/facebook/rocksdb/pull/9198 Reviewed By: akankshamahajan15 Differential Revision: D32596620 Pulled By: ajkr fbshipit-source-id: e17b2b6452b5f2e955b430140197c57e26a4a518
-
- 20 11月, 2021 2 次提交
-
-
由 Peter Dillinger 提交于
Summary: Generating megabytes of successful test output has caused issues / inconveniences for CI and internal sandcastle runs. This changes their configuration to only print output from failed tests. (Successful test output is still available in files under t/.) This likewise changes default behavior of parallel `make check` as a quick team poll showed interest in that. Pull Request resolved: https://github.com/facebook/rocksdb/pull/9188 Test Plan: Seed some test failures and observe * `make -j24 check` (new behavior) * `PRINT_PARALLEL_OUTPUTS=1 make -j24 check` (old CI behavior) * `QUIET_PARALLEL_TESTS=1 make -j24 check` (old manual run behavior) Reviewed By: siying Differential Revision: D32567392 Pulled By: pdillinger fbshipit-source-id: 8d8fb64aebd16bca103b11e3bd1f13c488a69611
-
由 Adam Retter 提交于
Summary: 1. Added a target for building a bundle jar for Sonatype Nexus - sometimes if the OSS Maven Central is misbehaving, it is quicker to upload a bundle to be processed for release. 2. Simplify the publish code by using a for-loop. Pull Request resolved: https://github.com/facebook/rocksdb/pull/9186 Reviewed By: jay-zhuang Differential Revision: D32564469 Pulled By: ajkr fbshipit-source-id: aaceac27e9143fb65b61dad2a46df346586672cd
-
- 19 11月, 2021 1 次提交
-
-
由 Peter Dillinger 提交于
Summary: * Checksums are now checked on meta blocks unless specifically suppressed or not applicable (e.g. plain table). (Was other way around.) This means a number of cases that were not checking checksums now are, including direct read TableProperties in Version::GetTableProperties (fixed in meta_blocks ReadTableProperties), reading any block from PersistentCache (fixed in BlockFetcher), read TableProperties in SstFileDumper (ldb/sst_dump/BackupEngine) before table reader open, maybe more. * For that to work, I moved the global_seqno+TableProperties checksum logic to the shared table/ code, because that is used by many utilies such as SstFileDumper. * Also for that to work, we have to know when we're dealing with a block that has a checksum (trailer), so added that capability to Footer based on magic number, and from there BlockFetcher. * Knowledge of trailer presence has also fixed a problem where other table formats were reading blocks including bytes for a non-existant trailer--and awkwardly kind-of not using them, e.g. no shared code checking checksums. (BlockFetcher compression type was populated incorrectly.) Now we only read what is needed. * Minimized code duplication and differing/incompatible/awkward abstractions in meta_blocks.{cc,h} (e.g. SeekTo in metaindex block without parsing block handle) * Moved some meta block handling code from table_properties*.* * Moved some code specific to block-based table from shared table/ code to BlockBasedTable class. The checksum stuff means we can't completely separate it, but things that don't need to be in shared table/ code should not be. * Use unique_ptr rather than raw ptr in more places. (Note: you can std::move from unique_ptr to shared_ptr.) Without enhancements to GetPropertiesOfAllTablesTest (see below), net reduction of roughly 100 lines of code. Pull Request resolved: https://github.com/facebook/rocksdb/pull/9163 Test Plan: existing tests and * Enhanced DBTablePropertiesTest.GetPropertiesOfAllTablesTest to verify that checksums are now checked on direct read of table properties by TableCache (new test would fail before this change) * Also enhanced DBTablePropertiesTest.GetPropertiesOfAllTablesTest to test putting table properties under old meta name * Also generally enhanced that same test to actually test what it was supposed to be testing already, by kicking things out of table cache when we don't want them there. Reviewed By: ajkr, mrambacher Differential Revision: D32514757 Pulled By: pdillinger fbshipit-source-id: 507964b9311d186ae8d1131182290cbd97a99fa9
-
- 18 11月, 2021 1 次提交
-
-
由 Peter Dillinger 提交于
Summary: Using deps for running blackbox and whitebox allows them to be parallelized, which doesn't seem to be working well. Pull Request resolved: https://github.com/facebook/rocksdb/pull/9180 Test Plan: make -j24 crash_test Reviewed By: siying Differential Revision: D32500851 Pulled By: pdillinger fbshipit-source-id: 364288c8d023b93e7ca2724ea40edae2f4eb0407
-
- 17 11月, 2021 2 次提交
-
-
由 Peter Dillinger 提交于
Summary: Some tests are timing out, so increase parallelism but also keep test output on console Large credit to jay-zhuang who is currently unavailable to revise & land https://github.com/facebook/rocksdb/issues/9135 Pull Request resolved: https://github.com/facebook/rocksdb/pull/9178 Test Plan: make check and valgrind_test, with and without PRINT_PARALLEL_OUTPUTS=1 https://www.internalfb.com/intern/sandcastle/group/nonce/6211105410048528/ Reviewed By: riversand963 Differential Revision: D32476680 Pulled By: pdillinger fbshipit-source-id: 8844947416e5baf4435e1ef6e33aeecc45621a89
-
由 Peter Dillinger 提交于
Summary: * Parallel `make check` would pass if a test binary failed to list gtest tests. This is now likely to report as a failure. * Crazy perl was generating some extra incorrect test names causing extra files and binary invocations. Fixed with cleaner awk. Pull Request resolved: https://github.com/facebook/rocksdb/pull/9160 Test Plan: For first part, add an 'assert(false);' to start of hash_test main and see 'make check' pass before, and fail after. For second part, inspect t/ directory before vs. after. Number of executed tests is same: $ cat log* | grep 'PASSED.*test' | awk '{ tot += $4; } END { print tot; }' 10469 Reviewed By: ajkr Differential Revision: D32372006 Pulled By: pdillinger fbshipit-source-id: 185b3db2b67e3f9198eb75322e4d0493e4fc1beb
-