- 29 3月, 2019 4 次提交
-
-
由 Yanqin Jin 提交于
Summary: Fix some hdfs-related code so that it can compile and run 'db_stress' Pull Request resolved: https://github.com/facebook/rocksdb/pull/5122 Differential Revision: D14675495 Pulled By: riversand963 fbshipit-source-id: cac280479efcf5451982558947eac1732e8bc45a
-
由 anand76 提交于
Summary: WAL files are currently not subject to deletion rate limiting by DeleteScheduler. If the size of the WAL files is significant, this can cause a high delete rate on SSDs that may affect other operations. To fix it, force WAL file deletions to go through the SstFileManager. Original PR for this is #2768 Pull Request resolved: https://github.com/facebook/rocksdb/pull/5116 Differential Revision: D14669437 Pulled By: anand1976 fbshipit-source-id: c5f62d0640cebaa1574de841a1d01e4ce2faadf0
-
由 Siying Dong 提交于
Summary: Allow customized merge operator to be loaded from option file/map/string by allowing users to pre-regiester merge operators to object registry. Also update HISTORY.md and header files for the same feature for comparator. Pull Request resolved: https://github.com/facebook/rocksdb/pull/5123 Differential Revision: D14658488 Pulled By: siying fbshipit-source-id: 86ea2fbd2a0a04632d8ea9fceaffefd041f6ae61
-
由 Siying Dong 提交于
Summary: We see a failure of obsolete_files_test but aren't able to identify the issue. Improve the test in following way and hope we can debug better next time: 1. Place sync point before automatic compaction runs so race condition will always trigger. 2. Disable sync point before test finishes. 3. ASSERT_OK() instead of ASSERT_TRUE(status.ok()) Pull Request resolved: https://github.com/facebook/rocksdb/pull/5125 Differential Revision: D14669456 Pulled By: siying fbshipit-source-id: dccb7648e334501ad651eb212880096eef1f4ab2
-
- 28 3月, 2019 7 次提交
-
-
由 Burton Li 提交于
Summary: The existing code for env_win src and header file doesn't fully followed the recommended code style (https://google.github.io/styleguide/cppguide.html#Functions). Fix it for better readability. anand1976 siying Pull Request resolved: https://github.com/facebook/rocksdb/pull/5096 Differential Revision: D14585358 Pulled By: anand1976 fbshipit-source-id: 7ce35ffe9e922f5c1421b0bbaa5fce7abad57617
-
由 Siying Dong 提交于
Summary: Following files were run through automatic formatter: db/db_impl.cc db/db_impl.h db/db_impl_compaction_flush.cc db/db_impl_debug.cc db/db_impl_files.cc db/db_impl_readonly.h db/db_impl_write.cc db/dbformat.cc db/dbformat.h table/block.cc table/block.h table/block_based_filter_block.cc table/block_based_filter_block.h table/block_based_filter_block_test.cc table/block_based_table_builder.cc table/block_based_table_reader.cc table/block_based_table_reader.h table/block_builder.cc table/block_builder.h table/block_fetcher.cc table/block_prefix_index.cc table/block_prefix_index.h table/block_test.cc table/format.cc table/format.h I could easily run all the files, but I don't want people to feel that I'm doing it for lines of code changes :) Pull Request resolved: https://github.com/facebook/rocksdb/pull/5114 Differential Revision: D14633040 Pulled By: siying fbshipit-source-id: 3f346cb53bf21e8c10704400da548dfce1e89a52
-
由 Siying Dong 提交于
Summary: Automatically format public headers so it looks more consistent. Pull Request resolved: https://github.com/facebook/rocksdb/pull/5115 Differential Revision: D14632854 Pulled By: siying fbshipit-source-id: ce9929ea62f9dcd65c69660b23eed1931cb0ae84
-
由 Siying Dong 提交于
Summary: We follow Google C++ Style which indicates variable names should be all underscore: https://google.github.io/styleguide/cppguide.html#Variable_Names Fix some variable names under db/transaction_log_impl.* Pull Request resolved: https://github.com/facebook/rocksdb/pull/5112 Differential Revision: D14631157 Pulled By: siying fbshipit-source-id: 9525c9b0976b843bca377b03897700d87cc60af8
-
由 Fosco Marotto 提交于
Summary: Pull Request resolved: https://github.com/facebook/rocksdb/pull/5119 Differential Revision: D14645216 Pulled By: gfosco fbshipit-source-id: f7c83dca22c2486fc5d8697b61638c382889d073
-
由 Yi Wu 提交于
Summary: Currently `perf_context.user_key_comparison_count` is bump only in `InternalKeyComparator`. For places user comparator is used directly the counter is not bump. Fixing the majority of it. Index iterator and filter code also use user comparator directly and don't bump the counter. It is not fixed in this patch. Pull Request resolved: https://github.com/facebook/rocksdb/pull/5098 Differential Revision: D14603753 Pulled By: siying fbshipit-source-id: 1cd41035644ca9e49b97a51030a5d1e15f5f3cae
-
由 Siying Dong 提交于
Summary: The code convention we are following, Google C++ Style, discourage alias in header files, especially public headers: https://google.github.io/styleguide/cppguide.html#Aliases Remove some of them. Might removed some from .cc files as well to be consistent. Pull Request resolved: https://github.com/facebook/rocksdb/pull/5113 Differential Revision: D14633030 Pulled By: siying fbshipit-source-id: b990edc919d5de60295992284f980195e501d424
-
- 27 3月, 2019 7 次提交
-
-
由 Yanqin Jin 提交于
Summary: This PR allows RocksDB to run in single-primary, multi-secondary process mode. The writer is a regular RocksDB (e.g. an `DBImpl`) instance playing the role of a primary. Multiple `DBImplSecondary` processes (secondaries) share the same set of SST files, MANIFEST, WAL files with the primary. Secondaries tail the MANIFEST of the primary and apply updates to their own in-memory state of the file system, e.g. `VersionStorageInfo`. This PR has several components: 1. (Originally in #4745). Add a `PathNotFound` subcode to `IOError` to denote the failure when a secondary tries to open a file which has been deleted by the primary. 2. (Similar to #4602). Add `FragmentBufferedReader` to handle partially-read, trailing record at the end of a log from where future read can continue. 3. (Originally in #4710 and #4820). Add implementation of the secondary, i.e. `DBImplSecondary`. 3.1 Tail the primary's MANIFEST during recovery. 3.2 Tail the primary's MANIFEST during normal processing by calling `ReadAndApply`. 3.3 Tailing WAL will be in a future PR. 4. Add an example in 'examples/multi_processes_example.cc' to demonstrate the usage of secondary RocksDB instance in a multi-process setting. Instructions to run the example can be found at the beginning of the source code. Pull Request resolved: https://github.com/facebook/rocksdb/pull/4899 Differential Revision: D14510945 Pulled By: riversand963 fbshipit-source-id: 4ac1c5693e6012ad23f7b4b42d3c374fecbe8886
-
由 jsteemann 提交于
Summary: fbson library is still included in `third-party` directory, but is not needed by RocksDB anymore. Pull Request resolved: https://github.com/facebook/rocksdb/pull/5108 Differential Revision: D14622272 Pulled By: siying fbshipit-source-id: 52b24ed17d8d870a71364f85e5bac4eafb192df5
-
由 Shi Feng 提交于
Summary: Introduce CPU timers for iterator seek and next operations. Seek counter includes SeekToFirst, SeekToLast and SeekForPrev, w/ the caveat that SeekToLast timer doesn't include some post processing time if upper bound is defined. Pull Request resolved: https://github.com/facebook/rocksdb/pull/5076 Differential Revision: D14525218 Pulled By: fredfsh fbshipit-source-id: 03ba25df3b22b06c072621e4de0eacfa1445f0d9
-
由 Siying Dong 提交于
Summary: Even customized ldb may not be able to read data from some databases if comparator is not standard. We modify option helper to get comparator from object registry so that we can use customized ldb to read non-standard comparator. Pull Request resolved: https://github.com/facebook/rocksdb/pull/5106 Differential Revision: D14622107 Pulled By: siying fbshipit-source-id: 151dcb295a35a4c7d54f919cd4e322a89dc601c9
-
由 Siying Dong 提交于
Summary: Right now, BlobDB::Open() fails to put all trash files to delete scheduler, which causes some trash files permanently untracked. Pull Request resolved: https://github.com/facebook/rocksdb/pull/5103 Differential Revision: D14606095 Pulled By: siying fbshipit-source-id: 41a9437a2948abb235c0ed85f9a04612d0e50183
-
由 Yi Wu 提交于
Summary: Since `SstFileReader` don't know largest seqno of a file, it will fail this check when it open a file with global seqno: https://github.com/facebook/rocksdb/blob/ca89ac2ba997dfa0e135bd75d4ccf6f5774a7eff/table/block_based_table_reader.cc#L730 Changes: * Pass largest_seqno=kMaxSequenceNumber from `SstFileReader` and allow it to bypass the above check. * `BlockBasedTable::VerifyChecksum` also double check if checksum will match when excluding global seqno (this is to make the new test in sst_table_reader_test pass). Pull Request resolved: https://github.com/facebook/rocksdb/pull/5097 Differential Revision: D14607434 Pulled By: riversand963 fbshipit-source-id: 9008599227c5fccbf9b73fee46b3bf4a1523f023
-
由 Yi Wu 提交于
Summary: Pull Request resolved: https://github.com/facebook/rocksdb/pull/5104 Differential Revision: D14619000 Pulled By: maysamyabandeh fbshipit-source-id: c2895794a3f31b826c149dcb698c1952dacc2332
-
- 26 3月, 2019 3 次提交
-
-
由 Zhongyi Xie 提交于
Summary: User report has shown that sometimes `BlockBasedTable::SetupCacheKeyPrefix` would assert when trying to generate an id from the file. The actual cause seems to be hardware related but we might be better off without the incorrect assertion See T42178927 for more information Pull Request resolved: https://github.com/facebook/rocksdb/pull/5102 Differential Revision: D14604677 Pulled By: miasantreble fbshipit-source-id: fcb09207ebdc4fa66e941afbc0523d84797e7ad7
-
由 Siying Dong 提交于
Summary: With https://github.com/facebook/rocksdb/pull/3009 we go through every CF to check whether a bottommost compaction is needed to be triggered. This is done within DB mutex. What we do within DB mutex may heavily influece the write throughput we can achieve, so we always want to minimize work there. Here we try to avoid this for-loop by first check a global threshold. In most of the time, the CF loop can be avoided. Pull Request resolved: https://github.com/facebook/rocksdb/pull/5090 Differential Revision: D14582684 Pulled By: siying fbshipit-source-id: 968f6d9bb6affe1a5ebc4910b418300b076f166f
-
由 Zhongyi Xie 提交于
Summary: Right now ldb command doesn't allow cases where option values contain equals sign. For example, ``` ldb --db=/tmp/test scan --from='q=3' --max_keys=1 ``` after parsing, ldb will have one option 'db', 'max_keys' and one flag 'from'. This PR updates the parsing logic so that it now supports the above mentioned cases Pull Request resolved: https://github.com/facebook/rocksdb/pull/5088 Differential Revision: D14600869 Pulled By: miasantreble fbshipit-source-id: c6ef518c74a98d7b6675ea5954ae08b1bda5554e
-
- 22 3月, 2019 3 次提交
-
-
由 Rashmi Sharma 提交于
Summary: [RocksDB] Make it easier for users to load options from option file and set shared block cache. Right now, it requires several dynamic casting for users to set the shared block cache to their option struct cast from the option file. If people don't do that, every CF of every DB will generate its own 8MB block cache. It's not a usable setting. So we are dragging every user who loads options from the file into such a mess. Instead, we should allow them to pass their cache object to LoadLatestOptions() and LoadOptionsFromFile(), so that those loaded option structs will have the shared block cache. Pull Request resolved: https://github.com/facebook/rocksdb/pull/5063 Differential Revision: D14518584 Pulled By: rashmishrm fbshipit-source-id: c91430ff9425a0e67d76fc67931d755f491ca5aa
-
由 Burton Li 提交于
Summary: The original implementation of WinEnvIO::NowNanos() has a constant data overflow by: li.QuadPart *= std::nano::den; As a result, the api provides a incorrect result. e.g.: li.QuadPart=13477844301545 std::nano::den=1e9 The fix uses pre-computed nano_seconds_per_period_ to present the nano seconds per performance counter period, in the case if nano::den is divisible by perf_counter_frequency_. Otherwise it falls back to use high_resolution_clock. siying ajkr Pull Request resolved: https://github.com/facebook/rocksdb/pull/5062 Differential Revision: D14426842 Pulled By: anand1976 fbshipit-source-id: 127f1daf423dd4b30edd0dcf8ea0466f468bec12
-
由 Maysam Yabandeh 提交于
Summary: The patch reorders DBIter fields to put 1-byte fields together and let the compiler optimize the memory usage by using less 64-bit allocations for bools and enums. This might have a negative side effect of putting the variables that are accessed together into different cache lines and hence increasing the cache misses. Not sure what benchmark would verify that thought. I ran simple, single-threaded seekrandom benchmarks but the variance in the results is too much to be conclusive. ./db_bench --benchmarks=fillrandom --use_existing_db=0 --num=1000000 --db=/dev/shm/dbbench ./db_bench --benchmarks=seekrandom[X10] --use_existing_db=1 --db=/dev/shm/dbbench --num=1000000 --duration=60 --seek_nexts=100 Pull Request resolved: https://github.com/facebook/rocksdb/pull/5078 Differential Revision: D14562676 Pulled By: maysamyabandeh fbshipit-source-id: 2284655d46e079b6e9a860e94be5defb6f482167
-
- 21 3月, 2019 3 次提交
-
-
由 Levi Tamasi 提交于
Summary: The patch adds a new config option to LRUCacheOptions that enables users to choose whether to use an adaptive mutex for the LRU block cache (on platforms where adaptive mutexes are supported). The default is true if RocksDB is compiled with -DROCKSDB_DEFAULT_TO_ADAPTIVE_MUTEX, false otherwise. Pull Request resolved: https://github.com/facebook/rocksdb/pull/5054 Differential Revision: D14542749 Pulled By: ltamasi fbshipit-source-id: 0065715ab6cf91f10444b737fed8c8aee6a8a0d2
-
由 Alexandre Viau 提交于
Summary: Pull Request resolved: https://github.com/facebook/rocksdb/pull/5086 Differential Revision: D14542212 Pulled By: siying fbshipit-source-id: db2f38a3f7c9b64532655a5d4ac4b7715c392883
-
由 anand76 提交于
Summary: The stack buffer in rocksdb::autovector is currently defined as an array of elements of the template type. This results in unnecessary construction of those objects, which can be a significant overhead in some cases. This PR changes the type of the stack buf to char* and uses placement new to construct new objects when they are inserted into the autovector. Pull Request resolved: https://github.com/facebook/rocksdb/pull/5080 Differential Revision: D14533221 Pulled By: anand1976 fbshipit-source-id: 9378985c7d03f4e1a28951bdd2403c72f10f23d7
-
- 20 3月, 2019 4 次提交
-
-
由 Zhongyi Xie 提交于
Summary: In order to better understand compaction done by different priority thread pool, we now collect compaction stats by priority and also print them to info LOG through stats dump. ``` ** Compaction Stats [default] ** Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Low 0/0 0.00 KB 0.0 16.8 11.3 5.5 5.6 0.1 0.0 0.0 406.4 136.1 42.24 34.96 45 0.939 13M 8865K High 0/0 0.00 KB 0.0 0.0 0.0 0.0 11.4 11.4 0.0 0.0 0.0 76.2 153.00 35.74 12185 0.013 0 0 ``` Pull Request resolved: https://github.com/facebook/rocksdb/pull/5050 Differential Revision: D14408583 Pulled By: miasantreble fbshipit-source-id: e53746586ea27cb8abc9fec35805bd80ed30f608
-
由 Andrew Audibert 提交于
Summary: BackupEngine relies on write-ahead logs to back up the memtable. Disabling write-ahead logs can result in backups failing to preserve unflushed keys. This PR updates the documentation to specify this behavior, and suggest always flushing the memtable when write-ahead logs are disabled. Pull Request resolved: https://github.com/facebook/rocksdb/pull/5071 Differential Revision: D14524124 Pulled By: miasantreble fbshipit-source-id: 635f855f8a42ad60273b5efd226139b511e3e5d5
-
由 Wenjie Yang 提交于
Summary: Add an option to filter out READ or WRITE operations while tracing. Pull Request resolved: https://github.com/facebook/rocksdb/pull/5082 Differential Revision: D14515083 Pulled By: mrmiywj fbshipit-source-id: 2504c89a9abf1dd629cad44b4104092702d77610
-
由 Hiroaki Nakamura 提交于
Summary: Partly addresses https://github.com/facebook/rocksdb/issues/4999 I verified `make static_lib` runs fine. Pull Request resolved: https://github.com/facebook/rocksdb/pull/5077 Differential Revision: D14521101 Pulled By: maysamyabandeh fbshipit-source-id: ba88e74a51d2d793cac7260d505b1a54254b53af
-
- 19 3月, 2019 2 次提交
-
-
由 Shobhit Dayal 提交于
Summary: This is a feature to sample data-block compressibility and and report them as stats. 1 in N (tunable) blocks is sampled for compressibility using two algorithms: 1. lz4 or snappy for fast compression 2. zstd or zlib for slow but higher compression. The stats are reported to the caller as raw-bytes and compressed-bytes. The block continues to be compressed for storage using the specified CompressionType. The db_bench_tool how has a command line option for specifying the sampling rate. It's default value is 0 (no sampling). To test the overhead for a certain value, users can compare the performance of db_bench_tool, varying the sampling rate. It is unlikely to have a noticeable impact for high values like 20. Pull Request resolved: https://github.com/facebook/rocksdb/pull/4842 Differential Revision: D13629011 Pulled By: shobhitdayal fbshipit-source-id: 14ca668bcab6499b2a1734edf848eb62a4f4fafa
-
由 He Zhe 提交于
Summary: Initialize magic_number to zero to avoid such failure. utilities/blob_db/blob_log_format.cc:91:3: error: 'magic_number' may be used uninitialized in this function [-Werror=maybe-uninitialized] if (magic_number != kMagicNumber) { ^~ Signed-off-by: NHe Zhe <zhe.he@windriver.com> Pull Request resolved: https://github.com/facebook/rocksdb/pull/5074 Differential Revision: D14505514 Pulled By: miasantreble fbshipit-source-id: 4334462958c2b9c5a7c68c6ab24dadf94ad70902
-
- 16 3月, 2019 2 次提交
-
-
由 anand76 提交于
Summary: There is a potential failure case in DBImpl::SwitchMemtable() that is not handled properly. The call to cur_log_writer->WriteBuffer() can fail due to an IO error. In that case, we need to call SetBGError() in order set the background error since the WriteBuffer() failure may result in data loss. Also, the asserts for !new_mem and !new_log are incorrect, as those would have been allocated by the time this failure is detected. Pull Request resolved: https://github.com/facebook/rocksdb/pull/5072 Differential Revision: D14461384 Pulled By: anand1976 fbshipit-source-id: fb59bce9d61378f37d2dfcd28c0b704b0f43c3cf
-
由 Andrew Kryczka 提交于
Summary: Since this feature affects the WAL behavior, it seems important our crash-recovery tests cover it. Pull Request resolved: https://github.com/facebook/rocksdb/pull/5070 Differential Revision: D14470085 Pulled By: miasantreble fbshipit-source-id: 9b9682a718a926d57d055e0a5ec867efbd2eb9c1
-
- 15 3月, 2019 1 次提交
-
-
由 Zhichao Cao 提交于
Summary: In the current trace_analyzer implementation, once the trace file has corrupted content, which can be caused by unexpected tracing operations or other reasons, trace_analyzer will print the error and stop analyzing. By adding the -try_process_corrupted_trace option, user can try to process the corrupted trace file and get the analyzing results of the trace records from the beginning to the the first corrupted point in the trace file. Analyzing might fail even this option is enabled. Pull Request resolved: https://github.com/facebook/rocksdb/pull/5067 Differential Revision: D14433037 Pulled By: zhichao-cao fbshipit-source-id: d095233ba371726869af0def0cdee23b69896831
-
- 13 3月, 2019 2 次提交
-
-
由 Zhongyi Xie 提交于
Summary: Pull Request resolved: https://github.com/facebook/rocksdb/pull/5061 Differential Revision: D14418581 Pulled By: miasantreble fbshipit-source-id: be7f90e16586666ddd0cce36971e403782ab0892
-
由 Andrew Kryczka 提交于
Summary: Without `total_order_seek=true`, using this command with `prefix_extractor` set skips over lots of keys. Pull Request resolved: https://github.com/facebook/rocksdb/pull/5066 Differential Revision: D14425967 Pulled By: sagar0 fbshipit-source-id: f6f142733258d92604f920615be9266e1fe797f8
-
- 09 3月, 2019 2 次提交
-
-
由 Yi Wu 提交于
Summary: JEMALLOC_CXX_THROW is not defined for earlier versions of jemalloc (e.g. 3.6), causing builds to fail on some platforms. Fixing it. Closes #4869 Pull Request resolved: https://github.com/facebook/rocksdb/pull/5053 Differential Revision: D14390034 Pulled By: sagar0 fbshipit-source-id: b2b7a03cd377201ef385eb521f65bae85c558055
-
由 Maysam Yabandeh 提交于
Summary: Pull Request resolved: https://github.com/facebook/rocksdb/pull/5055 Differential Revision: D14395944 Pulled By: maysamyabandeh fbshipit-source-id: 385062b59428c132ada4e49b327685ba1f5d30e6
-