- 03 4月, 2019 1 次提交
-
-
由 Siying Dong 提交于
Summary: This reverts commit f29dc1b9. In BlockBasedTableIterator, index_iter_->key() is sometimes a user key, so it is wrong to call ExtractUserKey() against it. This is a bug introduced by #5101. Temporarily revert the diff to keep the branch clean. Pull Request resolved: https://github.com/facebook/rocksdb/pull/5132 Differential Revision: D14718584 Pulled By: siying fbshipit-source-id: 0ac55dc9b5dbc18c7809092146bdf7eb9364b9ad
-
- 02 4月, 2019 2 次提交
-
-
由 xinbenlv 提交于
Summary: Pull Request resolved: https://github.com/facebook/rocksdb/pull/5134 Differential Revision: D14719068 Pulled By: siying fbshipit-source-id: c09a544f06ff414dbe2f90792aaf2bb5b8550bee
-
由 Mike Kolupaev 提交于
Summary: Just like ReadOptions::background_purge_on_iterator_cleanup but for ColumnFamilyHandle instead of Iterator. In our use case we sometimes call ColumnFamilyHandle's destructor from low-latency threads, and sometimes it blocks the thread for a few seconds deleting the files. To avoid that, we can either offload ColumnFamilyHandle's destruction to a background thread on our side, or add this option on rocksdb side. This PR does the latter, to be consistent with how we solve exactly the same problem for iterators using background_purge_on_iterator_cleanup option. (EDIT: It's avoid_unnecessary_blocking_io now, and affects both CF drops and iterator destructors.) I'm not quite comfortable with having two separate options (background_purge_on_iterator_cleanup and background_purge_on_cf_cleanup) for such a rarely used thing. Maybe we should merge them? Rename background_purge_on_cf_cleanup to something like delete_files_on_background_threads_only or avoid_blocking_io_in_unexpected_places, and make iterators use it instead of the one in ReadOptions? I can do that here if you guys think it's better. Pull Request resolved: https://github.com/facebook/rocksdb/pull/5043 Differential Revision: D14339233 Pulled By: al13n321 fbshipit-source-id: ccf7efa11c85c9a5b91d969bb55627d0fb01e7b8
-
- 30 3月, 2019 4 次提交
-
-
由 Remington Brasga 提交于
Summary: NewEmptyInternalIterator with arena mistakenly used EmptyIterator to allocate the size from area but then initialized it to a totally different object: EmptyInternalIterator. The patch fixes that. Pull Request resolved: https://github.com/facebook/rocksdb/pull/4905 Differential Revision: D14689840 Pulled By: maysamyabandeh fbshipit-source-id: af64fd8ee93d5a4ad54691c792e5ecc5efabc887
-
由 Maysam Yabandeh 提交于
Summary: Compaction would depend on max_evicted_seq_ value. The ::Initialize method should do that after max_evicted_seq_ is properly initialized. The patch also back ports #4853 from WritePrepared txn to WriteUnPrepared. Pull Request resolved: https://github.com/facebook/rocksdb/pull/5128 Differential Revision: D14686562 Pulled By: maysamyabandeh fbshipit-source-id: b2355025712a72676ac3b20a95258adcf4774490
-
由 Yi Wu 提交于
Summary: `BlockBasedTableIterator` avoid reading next block on `Next()` if it detects the iterator will be out of bound, by checking against index key. The optimization was added in #2239, and by the time it only check the bound per block. It seems later change make it a per-key check, which introduce unnecessary key comparisons. Pull Request resolved: https://github.com/facebook/rocksdb/pull/5101 Differential Revision: D14678707 Pulled By: siying fbshipit-source-id: 2372446116753c7892ea4cec7b4b49ef87ba463e
-
由 Yanqin Jin 提交于
Summary: **This PR updates RepeatableThread::wait, breaking some tests on OS X. The rest of the PR fixes the tests on OS X.** `RepeatableThreadTest.MockEnvTest` uses `MockTimeEnv` and `RepeatableThread`. If `RepeatableThread::wait` calls `TimedWait` with a time smaller than or equal to the current (real) time, `TimedWait` returns immediately on certain platforms, e.g. OS X. #4560 addresses this issue by replacing `TimedWait` with `Wait` in test. This fixes the test but makes test/production code diverge, which is not optimal for test coverage. This PR proposes an alternative fix which unifies test and production code path for `RepeatableThread::wait`. We obtain the current (real) time in seconds and add 10 extra seconds to ensure that `RepeatableThread::wait` invokes `TimedWait` with a time greater than (real) current time. This is to prevent the `TimedWait` function from returning immediately without sleeping and releasing the mutex. If `TimedWait` returns immediately, the mutex will not be released, and `RepeatableThread::TEST_WaitForRun` never has a chance to execute the callback which, in this case, updates the result returned by `mock_env->NowMicros()`. Consequently, `RepeatableThread::wait` cannot break out of the loop, causing test to hang. The extra 10 seconds is a best-effort approach because there seems no reliable and deterministic way to provide the aforementioned guarantee. By the time `RepeatableThread::wait` is called, there is no guarantee that the `delay + mock_env->NowMicros()` will be greater than the current real time. However, 10 seconds should be sufficient in most cases. We will keep an eye for possible flakiness of this test. Pull Request resolved: https://github.com/facebook/rocksdb/pull/5107 Differential Revision: D14680885 Pulled By: riversand963 fbshipit-source-id: d1ecbe10e1dacd110bd464cd01e188bfee72b89e
-
- 29 3月, 2019 4 次提交
-
-
由 Yanqin Jin 提交于
Summary: Fix some hdfs-related code so that it can compile and run 'db_stress' Pull Request resolved: https://github.com/facebook/rocksdb/pull/5122 Differential Revision: D14675495 Pulled By: riversand963 fbshipit-source-id: cac280479efcf5451982558947eac1732e8bc45a
-
由 anand76 提交于
Summary: WAL files are currently not subject to deletion rate limiting by DeleteScheduler. If the size of the WAL files is significant, this can cause a high delete rate on SSDs that may affect other operations. To fix it, force WAL file deletions to go through the SstFileManager. Original PR for this is #2768 Pull Request resolved: https://github.com/facebook/rocksdb/pull/5116 Differential Revision: D14669437 Pulled By: anand1976 fbshipit-source-id: c5f62d0640cebaa1574de841a1d01e4ce2faadf0
-
由 Siying Dong 提交于
Summary: Allow customized merge operator to be loaded from option file/map/string by allowing users to pre-regiester merge operators to object registry. Also update HISTORY.md and header files for the same feature for comparator. Pull Request resolved: https://github.com/facebook/rocksdb/pull/5123 Differential Revision: D14658488 Pulled By: siying fbshipit-source-id: 86ea2fbd2a0a04632d8ea9fceaffefd041f6ae61
-
由 Siying Dong 提交于
Summary: We see a failure of obsolete_files_test but aren't able to identify the issue. Improve the test in following way and hope we can debug better next time: 1. Place sync point before automatic compaction runs so race condition will always trigger. 2. Disable sync point before test finishes. 3. ASSERT_OK() instead of ASSERT_TRUE(status.ok()) Pull Request resolved: https://github.com/facebook/rocksdb/pull/5125 Differential Revision: D14669456 Pulled By: siying fbshipit-source-id: dccb7648e334501ad651eb212880096eef1f4ab2
-
- 28 3月, 2019 7 次提交
-
-
由 Burton Li 提交于
Summary: The existing code for env_win src and header file doesn't fully followed the recommended code style (https://google.github.io/styleguide/cppguide.html#Functions). Fix it for better readability. anand1976 siying Pull Request resolved: https://github.com/facebook/rocksdb/pull/5096 Differential Revision: D14585358 Pulled By: anand1976 fbshipit-source-id: 7ce35ffe9e922f5c1421b0bbaa5fce7abad57617
-
由 Siying Dong 提交于
Summary: Following files were run through automatic formatter: db/db_impl.cc db/db_impl.h db/db_impl_compaction_flush.cc db/db_impl_debug.cc db/db_impl_files.cc db/db_impl_readonly.h db/db_impl_write.cc db/dbformat.cc db/dbformat.h table/block.cc table/block.h table/block_based_filter_block.cc table/block_based_filter_block.h table/block_based_filter_block_test.cc table/block_based_table_builder.cc table/block_based_table_reader.cc table/block_based_table_reader.h table/block_builder.cc table/block_builder.h table/block_fetcher.cc table/block_prefix_index.cc table/block_prefix_index.h table/block_test.cc table/format.cc table/format.h I could easily run all the files, but I don't want people to feel that I'm doing it for lines of code changes :) Pull Request resolved: https://github.com/facebook/rocksdb/pull/5114 Differential Revision: D14633040 Pulled By: siying fbshipit-source-id: 3f346cb53bf21e8c10704400da548dfce1e89a52
-
由 Siying Dong 提交于
Summary: Automatically format public headers so it looks more consistent. Pull Request resolved: https://github.com/facebook/rocksdb/pull/5115 Differential Revision: D14632854 Pulled By: siying fbshipit-source-id: ce9929ea62f9dcd65c69660b23eed1931cb0ae84
-
由 Siying Dong 提交于
Summary: We follow Google C++ Style which indicates variable names should be all underscore: https://google.github.io/styleguide/cppguide.html#Variable_Names Fix some variable names under db/transaction_log_impl.* Pull Request resolved: https://github.com/facebook/rocksdb/pull/5112 Differential Revision: D14631157 Pulled By: siying fbshipit-source-id: 9525c9b0976b843bca377b03897700d87cc60af8
-
由 Fosco Marotto 提交于
Summary: Pull Request resolved: https://github.com/facebook/rocksdb/pull/5119 Differential Revision: D14645216 Pulled By: gfosco fbshipit-source-id: f7c83dca22c2486fc5d8697b61638c382889d073
-
由 Yi Wu 提交于
Summary: Currently `perf_context.user_key_comparison_count` is bump only in `InternalKeyComparator`. For places user comparator is used directly the counter is not bump. Fixing the majority of it. Index iterator and filter code also use user comparator directly and don't bump the counter. It is not fixed in this patch. Pull Request resolved: https://github.com/facebook/rocksdb/pull/5098 Differential Revision: D14603753 Pulled By: siying fbshipit-source-id: 1cd41035644ca9e49b97a51030a5d1e15f5f3cae
-
由 Siying Dong 提交于
Summary: The code convention we are following, Google C++ Style, discourage alias in header files, especially public headers: https://google.github.io/styleguide/cppguide.html#Aliases Remove some of them. Might removed some from .cc files as well to be consistent. Pull Request resolved: https://github.com/facebook/rocksdb/pull/5113 Differential Revision: D14633030 Pulled By: siying fbshipit-source-id: b990edc919d5de60295992284f980195e501d424
-
- 27 3月, 2019 7 次提交
-
-
由 Yanqin Jin 提交于
Summary: This PR allows RocksDB to run in single-primary, multi-secondary process mode. The writer is a regular RocksDB (e.g. an `DBImpl`) instance playing the role of a primary. Multiple `DBImplSecondary` processes (secondaries) share the same set of SST files, MANIFEST, WAL files with the primary. Secondaries tail the MANIFEST of the primary and apply updates to their own in-memory state of the file system, e.g. `VersionStorageInfo`. This PR has several components: 1. (Originally in #4745). Add a `PathNotFound` subcode to `IOError` to denote the failure when a secondary tries to open a file which has been deleted by the primary. 2. (Similar to #4602). Add `FragmentBufferedReader` to handle partially-read, trailing record at the end of a log from where future read can continue. 3. (Originally in #4710 and #4820). Add implementation of the secondary, i.e. `DBImplSecondary`. 3.1 Tail the primary's MANIFEST during recovery. 3.2 Tail the primary's MANIFEST during normal processing by calling `ReadAndApply`. 3.3 Tailing WAL will be in a future PR. 4. Add an example in 'examples/multi_processes_example.cc' to demonstrate the usage of secondary RocksDB instance in a multi-process setting. Instructions to run the example can be found at the beginning of the source code. Pull Request resolved: https://github.com/facebook/rocksdb/pull/4899 Differential Revision: D14510945 Pulled By: riversand963 fbshipit-source-id: 4ac1c5693e6012ad23f7b4b42d3c374fecbe8886
-
由 jsteemann 提交于
Summary: fbson library is still included in `third-party` directory, but is not needed by RocksDB anymore. Pull Request resolved: https://github.com/facebook/rocksdb/pull/5108 Differential Revision: D14622272 Pulled By: siying fbshipit-source-id: 52b24ed17d8d870a71364f85e5bac4eafb192df5
-
由 Shi Feng 提交于
Summary: Introduce CPU timers for iterator seek and next operations. Seek counter includes SeekToFirst, SeekToLast and SeekForPrev, w/ the caveat that SeekToLast timer doesn't include some post processing time if upper bound is defined. Pull Request resolved: https://github.com/facebook/rocksdb/pull/5076 Differential Revision: D14525218 Pulled By: fredfsh fbshipit-source-id: 03ba25df3b22b06c072621e4de0eacfa1445f0d9
-
由 Siying Dong 提交于
Summary: Even customized ldb may not be able to read data from some databases if comparator is not standard. We modify option helper to get comparator from object registry so that we can use customized ldb to read non-standard comparator. Pull Request resolved: https://github.com/facebook/rocksdb/pull/5106 Differential Revision: D14622107 Pulled By: siying fbshipit-source-id: 151dcb295a35a4c7d54f919cd4e322a89dc601c9
-
由 Siying Dong 提交于
Summary: Right now, BlobDB::Open() fails to put all trash files to delete scheduler, which causes some trash files permanently untracked. Pull Request resolved: https://github.com/facebook/rocksdb/pull/5103 Differential Revision: D14606095 Pulled By: siying fbshipit-source-id: 41a9437a2948abb235c0ed85f9a04612d0e50183
-
由 Yi Wu 提交于
Summary: Since `SstFileReader` don't know largest seqno of a file, it will fail this check when it open a file with global seqno: https://github.com/facebook/rocksdb/blob/ca89ac2ba997dfa0e135bd75d4ccf6f5774a7eff/table/block_based_table_reader.cc#L730 Changes: * Pass largest_seqno=kMaxSequenceNumber from `SstFileReader` and allow it to bypass the above check. * `BlockBasedTable::VerifyChecksum` also double check if checksum will match when excluding global seqno (this is to make the new test in sst_table_reader_test pass). Pull Request resolved: https://github.com/facebook/rocksdb/pull/5097 Differential Revision: D14607434 Pulled By: riversand963 fbshipit-source-id: 9008599227c5fccbf9b73fee46b3bf4a1523f023
-
由 Yi Wu 提交于
Summary: Pull Request resolved: https://github.com/facebook/rocksdb/pull/5104 Differential Revision: D14619000 Pulled By: maysamyabandeh fbshipit-source-id: c2895794a3f31b826c149dcb698c1952dacc2332
-
- 26 3月, 2019 3 次提交
-
-
由 Zhongyi Xie 提交于
Summary: User report has shown that sometimes `BlockBasedTable::SetupCacheKeyPrefix` would assert when trying to generate an id from the file. The actual cause seems to be hardware related but we might be better off without the incorrect assertion See T42178927 for more information Pull Request resolved: https://github.com/facebook/rocksdb/pull/5102 Differential Revision: D14604677 Pulled By: miasantreble fbshipit-source-id: fcb09207ebdc4fa66e941afbc0523d84797e7ad7
-
由 Siying Dong 提交于
Summary: With https://github.com/facebook/rocksdb/pull/3009 we go through every CF to check whether a bottommost compaction is needed to be triggered. This is done within DB mutex. What we do within DB mutex may heavily influece the write throughput we can achieve, so we always want to minimize work there. Here we try to avoid this for-loop by first check a global threshold. In most of the time, the CF loop can be avoided. Pull Request resolved: https://github.com/facebook/rocksdb/pull/5090 Differential Revision: D14582684 Pulled By: siying fbshipit-source-id: 968f6d9bb6affe1a5ebc4910b418300b076f166f
-
由 Zhongyi Xie 提交于
Summary: Right now ldb command doesn't allow cases where option values contain equals sign. For example, ``` ldb --db=/tmp/test scan --from='q=3' --max_keys=1 ``` after parsing, ldb will have one option 'db', 'max_keys' and one flag 'from'. This PR updates the parsing logic so that it now supports the above mentioned cases Pull Request resolved: https://github.com/facebook/rocksdb/pull/5088 Differential Revision: D14600869 Pulled By: miasantreble fbshipit-source-id: c6ef518c74a98d7b6675ea5954ae08b1bda5554e
-
- 22 3月, 2019 3 次提交
-
-
由 Rashmi Sharma 提交于
Summary: [RocksDB] Make it easier for users to load options from option file and set shared block cache. Right now, it requires several dynamic casting for users to set the shared block cache to their option struct cast from the option file. If people don't do that, every CF of every DB will generate its own 8MB block cache. It's not a usable setting. So we are dragging every user who loads options from the file into such a mess. Instead, we should allow them to pass their cache object to LoadLatestOptions() and LoadOptionsFromFile(), so that those loaded option structs will have the shared block cache. Pull Request resolved: https://github.com/facebook/rocksdb/pull/5063 Differential Revision: D14518584 Pulled By: rashmishrm fbshipit-source-id: c91430ff9425a0e67d76fc67931d755f491ca5aa
-
由 Burton Li 提交于
Summary: The original implementation of WinEnvIO::NowNanos() has a constant data overflow by: li.QuadPart *= std::nano::den; As a result, the api provides a incorrect result. e.g.: li.QuadPart=13477844301545 std::nano::den=1e9 The fix uses pre-computed nano_seconds_per_period_ to present the nano seconds per performance counter period, in the case if nano::den is divisible by perf_counter_frequency_. Otherwise it falls back to use high_resolution_clock. siying ajkr Pull Request resolved: https://github.com/facebook/rocksdb/pull/5062 Differential Revision: D14426842 Pulled By: anand1976 fbshipit-source-id: 127f1daf423dd4b30edd0dcf8ea0466f468bec12
-
由 Maysam Yabandeh 提交于
Summary: The patch reorders DBIter fields to put 1-byte fields together and let the compiler optimize the memory usage by using less 64-bit allocations for bools and enums. This might have a negative side effect of putting the variables that are accessed together into different cache lines and hence increasing the cache misses. Not sure what benchmark would verify that thought. I ran simple, single-threaded seekrandom benchmarks but the variance in the results is too much to be conclusive. ./db_bench --benchmarks=fillrandom --use_existing_db=0 --num=1000000 --db=/dev/shm/dbbench ./db_bench --benchmarks=seekrandom[X10] --use_existing_db=1 --db=/dev/shm/dbbench --num=1000000 --duration=60 --seek_nexts=100 Pull Request resolved: https://github.com/facebook/rocksdb/pull/5078 Differential Revision: D14562676 Pulled By: maysamyabandeh fbshipit-source-id: 2284655d46e079b6e9a860e94be5defb6f482167
-
- 21 3月, 2019 3 次提交
-
-
由 Levi Tamasi 提交于
Summary: The patch adds a new config option to LRUCacheOptions that enables users to choose whether to use an adaptive mutex for the LRU block cache (on platforms where adaptive mutexes are supported). The default is true if RocksDB is compiled with -DROCKSDB_DEFAULT_TO_ADAPTIVE_MUTEX, false otherwise. Pull Request resolved: https://github.com/facebook/rocksdb/pull/5054 Differential Revision: D14542749 Pulled By: ltamasi fbshipit-source-id: 0065715ab6cf91f10444b737fed8c8aee6a8a0d2
-
由 Alexandre Viau 提交于
Summary: Pull Request resolved: https://github.com/facebook/rocksdb/pull/5086 Differential Revision: D14542212 Pulled By: siying fbshipit-source-id: db2f38a3f7c9b64532655a5d4ac4b7715c392883
-
由 anand76 提交于
Summary: The stack buffer in rocksdb::autovector is currently defined as an array of elements of the template type. This results in unnecessary construction of those objects, which can be a significant overhead in some cases. This PR changes the type of the stack buf to char* and uses placement new to construct new objects when they are inserted into the autovector. Pull Request resolved: https://github.com/facebook/rocksdb/pull/5080 Differential Revision: D14533221 Pulled By: anand1976 fbshipit-source-id: 9378985c7d03f4e1a28951bdd2403c72f10f23d7
-
- 20 3月, 2019 4 次提交
-
-
由 Zhongyi Xie 提交于
Summary: In order to better understand compaction done by different priority thread pool, we now collect compaction stats by priority and also print them to info LOG through stats dump. ``` ** Compaction Stats [default] ** Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Low 0/0 0.00 KB 0.0 16.8 11.3 5.5 5.6 0.1 0.0 0.0 406.4 136.1 42.24 34.96 45 0.939 13M 8865K High 0/0 0.00 KB 0.0 0.0 0.0 0.0 11.4 11.4 0.0 0.0 0.0 76.2 153.00 35.74 12185 0.013 0 0 ``` Pull Request resolved: https://github.com/facebook/rocksdb/pull/5050 Differential Revision: D14408583 Pulled By: miasantreble fbshipit-source-id: e53746586ea27cb8abc9fec35805bd80ed30f608
-
由 Andrew Audibert 提交于
Summary: BackupEngine relies on write-ahead logs to back up the memtable. Disabling write-ahead logs can result in backups failing to preserve unflushed keys. This PR updates the documentation to specify this behavior, and suggest always flushing the memtable when write-ahead logs are disabled. Pull Request resolved: https://github.com/facebook/rocksdb/pull/5071 Differential Revision: D14524124 Pulled By: miasantreble fbshipit-source-id: 635f855f8a42ad60273b5efd226139b511e3e5d5
-
由 Wenjie Yang 提交于
Summary: Add an option to filter out READ or WRITE operations while tracing. Pull Request resolved: https://github.com/facebook/rocksdb/pull/5082 Differential Revision: D14515083 Pulled By: mrmiywj fbshipit-source-id: 2504c89a9abf1dd629cad44b4104092702d77610
-
由 Hiroaki Nakamura 提交于
Summary: Partly addresses https://github.com/facebook/rocksdb/issues/4999 I verified `make static_lib` runs fine. Pull Request resolved: https://github.com/facebook/rocksdb/pull/5077 Differential Revision: D14521101 Pulled By: maysamyabandeh fbshipit-source-id: ba88e74a51d2d793cac7260d505b1a54254b53af
-
- 19 3月, 2019 2 次提交
-
-
由 Shobhit Dayal 提交于
Summary: This is a feature to sample data-block compressibility and and report them as stats. 1 in N (tunable) blocks is sampled for compressibility using two algorithms: 1. lz4 or snappy for fast compression 2. zstd or zlib for slow but higher compression. The stats are reported to the caller as raw-bytes and compressed-bytes. The block continues to be compressed for storage using the specified CompressionType. The db_bench_tool how has a command line option for specifying the sampling rate. It's default value is 0 (no sampling). To test the overhead for a certain value, users can compare the performance of db_bench_tool, varying the sampling rate. It is unlikely to have a noticeable impact for high values like 20. Pull Request resolved: https://github.com/facebook/rocksdb/pull/4842 Differential Revision: D13629011 Pulled By: shobhitdayal fbshipit-source-id: 14ca668bcab6499b2a1734edf848eb62a4f4fafa
-
由 He Zhe 提交于
Summary: Initialize magic_number to zero to avoid such failure. utilities/blob_db/blob_log_format.cc:91:3: error: 'magic_number' may be used uninitialized in this function [-Werror=maybe-uninitialized] if (magic_number != kMagicNumber) { ^~ Signed-off-by: NHe Zhe <zhe.he@windriver.com> Pull Request resolved: https://github.com/facebook/rocksdb/pull/5074 Differential Revision: D14505514 Pulled By: miasantreble fbshipit-source-id: 4334462958c2b9c5a7c68c6ab24dadf94ad70902
-