- 01 10月, 2019 3 次提交
-
-
由 sdong 提交于
Summary: Recent changes trigger clang analyze warning. Fix it. Pull Request resolved: https://github.com/facebook/rocksdb/pull/5870 Test Plan: "USE_CLANG=1 TEST_TMPDIR=/dev/shm/rocksdb OPT=-g make -j60 analyze" and make sure it passes. Differential Revision: D17682533 fbshipit-source-id: 02716f2a24572550a22db4bbe9b54d4872dfae32
-
由 Jay Zhuang 提交于
Summary: ``` tools/block_cache_analyzer/block_cache_trace_analyzer.cc:653:48: error: implicit conversion loses integer precision: 'uint64_t' (aka 'unsigned long long') to 'std::__1::linear_congruential_engine<unsigned int, 48271, 0, 2147483647>::result_type' (aka 'unsigned int') [-Werror,-Wshorten-64-to-32] std::default_random_engine rand_engine(env_->NowMicros()); ``` Pull Request resolved: https://github.com/facebook/rocksdb/pull/5864 Differential Revision: D17668962 fbshipit-source-id: e08fa58b2a78a8dd8b334862b5714208f696b8ab
-
由 sdong 提交于
Summary: Two more bug fixes in db_stress: 1. this is to complete the fix of the regression bug causing overflowing when supporting FLAGS_prefix_size = -1. 2. Fix regression bug in compare iterator itself: (1) when creating control iterator, which used the same read option as the normal iterator by mistake; (2) the logic of comparing has some problems. Fix them. (3) disable validation for lower bound now, which generated some wildly different results. Disabling it to make normal tests pass while investigating it. 3. Cleaning up snapshots in verification failure cases. Memory is leaked otherwise. Pull Request resolved: https://github.com/facebook/rocksdb/pull/5867 Test Plan: Run "make crash_test" for a while and see at least 1 is fixed. Differential Revision: D17671712 fbshipit-source-id: 011f98ea1a72aef23e19ff28656830c78699b402
-
- 28 9月, 2019 4 次提交
-
-
由 Yanqin Jin 提交于
Summary: Atomic flush is incompatible with pipelined write. At least now. If pipelined write is enabled, a thread performing write can exit the write thread and start inserting into memtables. Consequently a thread performing flush will enter write thread and race with memtable insertion by the former. This will cause undefined result in terms of data persistence. Pull Request resolved: https://github.com/facebook/rocksdb/pull/5860 Test Plan: ``` $make all && make check ``` Differential Revision: D17638944 Pulled By: riversand963 fbshipit-source-id: abc578dc49a5dbe41bc5adcecf448f8e042a6d49
-
由 sdong 提交于
Summary: When prefix_size = -1, stress test crashes with run time error because of overflow. Fix it by not using -1 but 7 in prefix scan mode. Pull Request resolved: https://github.com/facebook/rocksdb/pull/5862 Test Plan: Run python -u tools/db_crashtest.py --simple whitebox --random_kill_odd \ 888887 --compression_type=zstd and see it doesn't crash. Differential Revision: D17642313 fbshipit-source-id: f029e7651498c905af1b1bee6d310ae50cdcda41
-
由 sdong 提交于
Summary: For now, crash_test is not able to report any failure for the logic related to iterator upper, lower bounds or iterators, or reseek. These are features prone to errors. Improve db_stress in several ways: (1) For each iterator run, reseek up to 3 times. (2) For every iterator, create control iterator with upper or lower bound, with total order seek. Compare the results with the iterator. (3) Make simple crash test to avoid prefix size to have more coverage. (4) make prefix_size = 0 a valid size and -1 to indicate disabling prefix extractor. Pull Request resolved: https://github.com/facebook/rocksdb/pull/5846 Test Plan: Manually hack the code to create wrong results and see they are caught by the tool. Differential Revision: D17631760 fbshipit-source-id: acd460a177bd2124a5ffd7fff490702dba63030b
-
由 Chen, You 提交于
Summary: Add unordered_write option api and related ut to rocksjava Pull Request resolved: https://github.com/facebook/rocksdb/pull/5839 Differential Revision: D17604446 Pulled By: maysamyabandeh fbshipit-source-id: c6b07e85ca9d5e3a92973ddb6ab2bc079e53c9c1
-
- 27 9月, 2019 2 次提交
-
-
由 Yanqin Jin 提交于
Summary: as title. Test Plan (on devserver): ``` $make all && make check ``` Pull Request resolved: https://github.com/facebook/rocksdb/pull/5855 Differential Revision: D17615125 Pulled By: riversand963 fbshipit-source-id: bd6ed8cf59eafff41f0d1fc044f39e8f3573172a
-
由 sdong 提交于
Summary: This is a bug occaionally shows up in crash test, and this unit test is to reproduce it. The bug is following: 1. Database has multiple CFs. 2. Between one DB restart, the last log file is corrupted in the middle (not the tail) 3. During restart, DB crashes between flushes between two CFs. The DB will fail to be opened again with error "SST file is ahead of WALs" Pull Request resolved: https://github.com/facebook/rocksdb/pull/5851 Test Plan: Run the test itself. Differential Revision: D17614721 fbshipit-source-id: 1b0abce49b203a76a039e38e76bc940429975f20
-
- 25 9月, 2019 2 次提交
-
-
由 Maysam Yabandeh 提交于
Summary: Partitioned filters make use of a top-level index to find the partition in which the filter resides. The top-level index has a key per partition. The key is guaranteed to be larger or equal than any key in that partition. When used with format_version 3, which excludes the sequence number form index keys, the separator key in the index could be equal to the prefix of the keys in the next partition. In this way, when searching for the key, the top-level index will lead us to the previous partition, which has no key with that prefix. The prefix bloom test thus returns false, although the prefix exists in the bloom of the next partition. The patch fixes that by a hack: It always adds the prefix of the first key of the next partition to the bloom of the current partition. In this way, in the corner cases that the index will lead us to the previous partition, we still can find the bloom filter there. Pull Request resolved: https://github.com/facebook/rocksdb/pull/5835 Differential Revision: D17513585 Pulled By: maysamyabandeh fbshipit-source-id: e2d1ff26c759e6e03875c4d57f4228316ecf50e9
-
由 Levi Tamasi 提交于
Summary: Pull Request resolved: https://github.com/facebook/rocksdb/pull/5832 Differential Revision: D17550773 Pulled By: ltamasi fbshipit-source-id: 66972bb008516e55b6fbba58ddd10234346d5d11
-
- 24 9月, 2019 3 次提交
-
-
由 WangQingping 提交于
Summary: Pull Request resolved: https://github.com/facebook/rocksdb/pull/5837 Differential Revision: D17529753 fbshipit-source-id: 98bbf22c690384b2f440286151dffdaaa744e97c
-
由 Yikun Jiang 提交于
Summary: The comparison of va_list and nullptr is always False under any arch, and will raise invalid operands of types error in aarch64 env (`error: invalid operands of types ‘va_list {aka __va_list}’ and ‘std::nullptr_t’ to binary ‘operator!=’`). This patch removes this invalid assert. Closes: https://github.com/facebook/rocksdb/issues/4277 Pull Request resolved: https://github.com/facebook/rocksdb/pull/5836 Differential Revision: D17532470 fbshipit-source-id: ca98078ecbc6a9416c69de3bd6ffcfa33a0f0185
-
由 Peter Dillinger 提交于
Summary: format-diff.sh, a.k.a. 'make format', would use 'master' to decide which commits are probably unpublished. Much better to use facebook remote master since local master may not be caught up and may have its own unpublished commits. Script now tries to compare against facebook remote master branch (branch pointer is updated with any fetch or pull), because those differences are what would be considered the differences for a pull request. Also, script would compare against *parent* of merge-base with that reference point, which is just wrong since that includes the last published commit. In case of problems, you can now customize the reference point, by setting the FORMAT_UPSTREAM variable. Pull Request resolved: https://github.com/facebook/rocksdb/pull/5831 Test Plan: manual Differential Revision: D17528462 Pulled By: pdillinger fbshipit-source-id: 50fdb8795d683bf3c14d449669c1a5299e0dfa8b
-
- 21 9月, 2019 2 次提交
-
-
由 sdong 提交于
Summary: Further apply formatter to more recent commits. Pull Request resolved: https://github.com/facebook/rocksdb/pull/5830 Test Plan: Run all existing tests. Differential Revision: D17488031 fbshipit-source-id: 137458fd94d56dd271b8b40c522b03036943a2ab
-
由 Vijay Nadimpalli 提交于
Summary: Added comment for Version Edit. Pull Request resolved: https://github.com/facebook/rocksdb/pull/5829 Test Plan: Run existing tests Differential Revision: D17486229 Pulled By: vjnadimpalli fbshipit-source-id: b4b31104fadd667356b64bd2dc409b3376ee46ca
-
- 20 9月, 2019 1 次提交
-
-
由 sdong 提交于
Summary: Some recent commits might not have passed through the formatter. I formatted recent 45 commits. The script hangs for more commits so I stopped there. Pull Request resolved: https://github.com/facebook/rocksdb/pull/5827 Test Plan: Run all existing tests. Differential Revision: D17483727 fbshipit-source-id: af23113ee63015d8a43d89a3bc2c1056189afe8f
-
- 19 9月, 2019 6 次提交
-
-
由 Maysam Yabandeh 提交于
Summary: The snap_refresh_nanos option didn't bring much benefit. Remove the feature to simplify the code. Pull Request resolved: https://github.com/facebook/rocksdb/pull/5826 Differential Revision: D17467147 Pulled By: maysamyabandeh fbshipit-source-id: 4f950b046990d0d1292d7fc04c2ccafaf751c7f0
-
由 Yanqin Jin 提交于
Summary: Make DeleteFileTest inherit DBTestBase to avoid code duplication. Test Plan (on devserver) ``` $make deletefile_test $./deletefile_test ``` Pull Request resolved: https://github.com/facebook/rocksdb/pull/5822 Differential Revision: D17456750 Pulled By: riversand963 fbshipit-source-id: 224e97967da7b98838a98981cd5095d3230a814f
-
由 Levi Tamasi 提交于
Summary: clang-analyzer has uncovered a bunch of places where the code is relying on pointers being valid and one case (in VectorIterator) where a moved-from object is being used: In file included from db/range_tombstone_fragmenter.cc:17: ./util/vector_iterator.h:23:18: warning: Method called on moved-from object 'keys' of type 'std::vector' current_(keys.size()) { ^~~~~~~~~~~ 1 warning generated. utilities/persistent_cache/block_cache_tier_file.cc:39:14: warning: Called C++ object pointer is null Status s = env->NewRandomAccessFile(filepath, file, opt); ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ utilities/persistent_cache/block_cache_tier_file.cc:47:19: warning: Called C++ object pointer is null Status status = env_->GetFileSize(Path(), size); ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ utilities/persistent_cache/block_cache_tier_file.cc:290:14: warning: Called C++ object pointer is null Status s = env_->FileExists(Path()); ^~~~~~~~~~~~~~~~~~~~~~~~ utilities/persistent_cache/block_cache_tier_file.cc:363:35: warning: Called C++ object pointer is null CacheWriteBuffer* const buf = alloc_->Allocate(); ^~~~~~~~~~~~~~~~~~ utilities/persistent_cache/block_cache_tier_file.cc:399:41: warning: Called C++ object pointer is null const uint64_t file_off = buf_doff_ * alloc_->BufferSize(); ^~~~~~~~~~~~~~~~~~~~ utilities/persistent_cache/block_cache_tier_file.cc:463:33: warning: Called C++ object pointer is null size_t start_idx = lba.off_ / alloc_->BufferSize(); ^~~~~~~~~~~~~~~~~~~~ utilities/persistent_cache/block_cache_tier_file.cc:515:5: warning: Called C++ object pointer is null alloc_->Deallocate(bufs_[i]); ^~~~~~~~~~~~~~~~~~~~~~~~~~~~ 7 warnings generated. ar: creating librocksdb_debug.a utilities/memory/memory_test.cc:68:25: warning: Called C++ object pointer is null cache_set->insert(db->GetDBOptions().row_cache.get()); ^~~~~~~~~~~~~~~~~~ 1 warning generated. The patch fixes these by adding assertions and explicitly passing in zero when initializing VectorIterator::current_ (which preserves the existing behavior). Pull Request resolved: https://github.com/facebook/rocksdb/pull/5821 Test Plan: Ran make check and make analyze to make sure the warnings have disappeared. Differential Revision: D17455949 Pulled By: ltamasi fbshipit-source-id: 363619618ea649a0674287f9f3b3393e390571ee
-
由 风 提交于
Summary: The dtor will automatically do unlock Pull Request resolved: https://github.com/facebook/rocksdb/pull/5809 Differential Revision: D17453694 Pulled By: ltamasi fbshipit-source-id: 5348bff8e6a620a05ff639a5454e8d82ae98a22d
-
由 Yanqin Jin 提交于
Summary: Make class ObsoleteFilesTest inherit from DBTestBase. Test plan (on devserver): ``` $COMPILE_WITH_ASAN=1 make obsolete_files_test $./obsolete_files_test ``` Pull Request resolved: https://github.com/facebook/rocksdb/pull/5820 Differential Revision: D17452348 Pulled By: riversand963 fbshipit-source-id: b09f4581a18022ca2bfd79f2836c0bf7083f5f25
-
由 Tomas Kolda 提交于
Summary: It is very useful method call to achieve https://github.com/facebook/rocksdb/wiki/Delete-A-Range-Of-Keys Pull Request resolved: https://github.com/facebook/rocksdb/pull/4031 Differential Revision: D13515418 Pulled By: vjnadimpalli fbshipit-source-id: 930b48e0992ef07fd1edd0b0cb5f780fabb1b4b5
-
- 18 9月, 2019 3 次提交
-
-
由 Yanqin Jin 提交于
Summary: Originally the loop of closing WAL in PurgeObsoleteFiles resides inside a loop iterating over the candidate files. It should be moved out. Test plan (devserver) ``` $COMPILE_WITH_ASAN=1 make -j32 all $make check ``` Pull Request resolved: https://github.com/facebook/rocksdb/pull/5804 Differential Revision: D17374350 Pulled By: riversand963 fbshipit-source-id: 2bee7343fc0481d9a385a87c7676491522285c96
-
由 sdong 提交于
Summary: We are seeing a bug of wrong results with merging iterator's reseek avoidence feature and prefix extractor. Disable this optimization for now. Pull Request resolved: https://github.com/facebook/rocksdb/pull/5815 Test Plan: Validated the same MyRocks case was fixed; run all existing tests. Differential Revision: D17430776 fbshipit-source-id: aef664277ba0ab8a2e68331ff0db6ae682535371
-
由 Yi Wu 提交于
Summary: purge_queue_ maybe contains thousands sst files, for example manual compact a range. If full scan is triggered at the same time and the total sst files number is large, RocksDB will be blocked at https://github.com/facebook/rocksdb/blob/master/db/db_impl_files.cc#L150 for several seconds. In our environment we have 140,000 sst files and the manual compaction delete about 1000 sst files, it blocked about 2 minutes. Commandeering https://github.com/facebook/rocksdb/issues/5290. Pull Request resolved: https://github.com/facebook/rocksdb/pull/5796 Differential Revision: D17357775 Pulled By: riversand963 fbshipit-source-id: 20eacca917355b8de975ccc7b1c9a3e7bd5b201a
-
- 17 9月, 2019 13 次提交
-
-
由 Maysam Yabandeh 提交于
Summary: https://github.com/facebook/rocksdb/issues/5797 charges the block cache with the total of user-provided charge plus the metadata charge. It had a bug where in MaintainPoolSize the user-provided charge was used instead of the total charge. The patch fixes that. Pull Request resolved: https://github.com/facebook/rocksdb/pull/5813 Differential Revision: D17412783 Pulled By: maysamyabandeh fbshipit-source-id: 45c0ac9f1e2233760db5ccd61399605cd74edc87
-
由 sdong 提交于
Summary: Doing some code reordering in DBIter::Seek() and DBIter::SeekForPrev(). The logic largely remains the same, except slight difference when handling some stats when valid_ = false, where they are not supposed to be used anyway. Also remove prefix_start_key_, which sometimes point a part of seek target, some times prefix_start_buf_, which is confusing. Pull Request resolved: https://github.com/facebook/rocksdb/pull/5794 Test Plan: Run all tests. Differential Revision: D17375257 fbshipit-source-id: 7339a23898cecd3a8475bf72340fcd6f82b933c5
-
由 andrew 提交于
Summary: Manual compaction may bring in very high load because sometime the amount of data involved in a compaction could be large, which may affect online service. So it would be good if the running compaction making the server busy can be stopped immediately. In this implementation, stopping manual compaction condition is only checked in slow process. We let deletion compaction and trivial move go through. Pull Request resolved: https://github.com/facebook/rocksdb/pull/3971 Test Plan: add tests at more spots. Differential Revision: D17369043 fbshipit-source-id: 575a624fb992ce0bb07d9443eb209e547740043c
-
由 sdong 提交于
Summary: Update version of dependencies. Pull Request resolved: https://github.com/facebook/rocksdb/pull/5777 Test Plan: make release Differential Revision: D17269421 fbshipit-source-id: e76dbe5389e1d7f811739d3bc1e404b482dfce34
-
由 sdong 提交于
Summary: Unity build fails because of name conflict of IsFileSectorAligned() after recent refactoring. Consolidate the function. Pull Request resolved: https://github.com/facebook/rocksdb/pull/5812 Test Plan: make unity. At least the failure goes away. Also "make all", "make release" and see no regression in normal cases. Differential Revision: D17411403 fbshipit-source-id: 09d5653471ae2c3a4d898e120a024f7dd08d9c9d
-
由 Peter Dillinger 提交于
Summary: Refactoring to consolidate implementation details of legacy Bloom filters. This helps to organize and document some related, obscure code. Also added make/cpp var TEST_CACHE_LINE_SIZE so that it's easy to compile and run unit tests for non-native cache line size. (Fixed a related test failure in db_properties_test.) Pull Request resolved: https://github.com/facebook/rocksdb/pull/5784 Test Plan: make check, including Recently added Bloom schema unit tests (in ./plain_table_db_test && ./bloom_test), and including with TEST_CACHE_LINE_SIZE=128U and TEST_CACHE_LINE_SIZE=256U. Tested the schema tests with temporary fault injection into new implementations. Some performance testing with modified unit tests suggest a small to moderate improvement in speed. Differential Revision: D17381384 Pulled By: pdillinger fbshipit-source-id: ee42586da996798910fc45ac0b6289147f16d8df
-
由 Maysam Yabandeh 提交于
Summary: For our default block cache, each additional entry has extra memory overhead. It include LRUHandle (72 bytes currently) and the cache key (two varint64, file id and offset). The usage is not negligible. For example for block_size=4k, the overhead accounts for an extra 2% memory usage for the cache. The patch charging the cache for the extra usage, reducing untracked memory usage outside block cache. The feature is enabled by default and can be disabled by passing kDontChargeCacheMetadata to the cache constructor. This PR builds up on https://github.com/facebook/rocksdb/issues/4258 Pull Request resolved: https://github.com/facebook/rocksdb/pull/5797 Test Plan: - Existing tests are updated to either disable the feature when the test has too much dependency on the old way of accounting the usage or increasing the cache capacity to account for the additional charge of metadata. - The Usage tests in cache_test.cc are augmented to test the cache usage under kFullChargeCacheMetadata. Differential Revision: D17396833 Pulled By: maysamyabandeh fbshipit-source-id: 7684ccb9f8a40ca595e4f5efcdb03623afea0c6f
-
由 Levi Tamasi 提交于
Summary: PR https://github.com/facebook/rocksdb/issues/4020 enabled partitioned indexes/filters in stress tests; however, this causes assertion failures in BatchedOpsStressTest. This patch disables them until we can root cause the failures. Pull Request resolved: https://github.com/facebook/rocksdb/pull/5811 Test Plan: Ran the script and made sure it only uses the binary search index. Differential Revision: D17399366 Pulled By: ltamasi fbshipit-source-id: adb116e6297f9c6ccd7ac15b6a16c9aa91f21ac5
-
由 Peter Dillinger 提交于
Summary: This will allow us to fix history by having the code changes for PR#5784 properly attributed to it. Pull Request resolved: https://github.com/facebook/rocksdb/pull/5810 Differential Revision: D17400231 Pulled By: pdillinger fbshipit-source-id: 2da8b1cdf2533cfedb35b5526eadefb38c291f09
-
由 sdong 提交于
Summary: Several functions of UniversalCompactionPicker share most of the parameters. Move these functions to a class with those shared arguments as class members. Hopefully this will make code slightly easier to maintain. Pull Request resolved: https://github.com/facebook/rocksdb/pull/5639 Test Plan: Run all existing test. Differential Revision: D16996403 fbshipit-source-id: fffafd1897ab132b420b1dec073542cffb5c44de
-
由 Peter (Stig) Edwards 提交于
Summary: As requested by siying in https://github.com/facebook/rocksdb/pull/5791#issuecomment-531417468 Pull Request resolved: https://github.com/facebook/rocksdb/pull/5807 Differential Revision: D17399349 fbshipit-source-id: 5986c3894f356becd393fee0f1aeadcd9affc798
-
由 sdong 提交于
Summary: file_reader_writer.h and .cc contain several files and helper function, and it's hard to navigate. Separate it to multiple files and put them under file/ Pull Request resolved: https://github.com/facebook/rocksdb/pull/5803 Test Plan: Build whole project using make and cmake. Differential Revision: D17374550 fbshipit-source-id: 10efca907721e7a78ed25bbf74dc5410dea05987
-
由 Peter Dillinger 提交于
Summary: DynamicBloom unit test now tests non-sequential as well as sequential keys in testing FP rates. Also now verifies larger structures. Pull Request resolved: https://github.com/facebook/rocksdb/pull/5805 Test Plan: thisisthetest Differential Revision: D17398109 Pulled By: pdillinger fbshipit-source-id: 374074206c76d242efa378afc27830448a0e892a
-
- 16 9月, 2019 1 次提交
-
-
由 Ariel Hurdle 提交于
Summary: Add Avrio's use case of RocksDB to USERS.md Pull Request resolved: https://github.com/facebook/rocksdb/pull/5748 Differential Revision: D17392483 Pulled By: riversand963 fbshipit-source-id: 673f4e1a5c82079ec8dfb2816db0cc9af9d38341
-