- 27 3月, 2019 2 次提交
-
-
由 Yi Wu 提交于
Summary: Since `SstFileReader` don't know largest seqno of a file, it will fail this check when it open a file with global seqno: https://github.com/facebook/rocksdb/blob/ca89ac2ba997dfa0e135bd75d4ccf6f5774a7eff/table/block_based_table_reader.cc#L730 Changes: * Pass largest_seqno=kMaxSequenceNumber from `SstFileReader` and allow it to bypass the above check. * `BlockBasedTable::VerifyChecksum` also double check if checksum will match when excluding global seqno (this is to make the new test in sst_table_reader_test pass). Pull Request resolved: https://github.com/facebook/rocksdb/pull/5097 Differential Revision: D14607434 Pulled By: riversand963 fbshipit-source-id: 9008599227c5fccbf9b73fee46b3bf4a1523f023
-
由 Yi Wu 提交于
Summary: Pull Request resolved: https://github.com/facebook/rocksdb/pull/5104 Differential Revision: D14619000 Pulled By: maysamyabandeh fbshipit-source-id: c2895794a3f31b826c149dcb698c1952dacc2332
-
- 26 3月, 2019 3 次提交
-
-
由 Zhongyi Xie 提交于
Summary: User report has shown that sometimes `BlockBasedTable::SetupCacheKeyPrefix` would assert when trying to generate an id from the file. The actual cause seems to be hardware related but we might be better off without the incorrect assertion See T42178927 for more information Pull Request resolved: https://github.com/facebook/rocksdb/pull/5102 Differential Revision: D14604677 Pulled By: miasantreble fbshipit-source-id: fcb09207ebdc4fa66e941afbc0523d84797e7ad7
-
由 Siying Dong 提交于
Summary: With https://github.com/facebook/rocksdb/pull/3009 we go through every CF to check whether a bottommost compaction is needed to be triggered. This is done within DB mutex. What we do within DB mutex may heavily influece the write throughput we can achieve, so we always want to minimize work there. Here we try to avoid this for-loop by first check a global threshold. In most of the time, the CF loop can be avoided. Pull Request resolved: https://github.com/facebook/rocksdb/pull/5090 Differential Revision: D14582684 Pulled By: siying fbshipit-source-id: 968f6d9bb6affe1a5ebc4910b418300b076f166f
-
由 Zhongyi Xie 提交于
Summary: Right now ldb command doesn't allow cases where option values contain equals sign. For example, ``` ldb --db=/tmp/test scan --from='q=3' --max_keys=1 ``` after parsing, ldb will have one option 'db', 'max_keys' and one flag 'from'. This PR updates the parsing logic so that it now supports the above mentioned cases Pull Request resolved: https://github.com/facebook/rocksdb/pull/5088 Differential Revision: D14600869 Pulled By: miasantreble fbshipit-source-id: c6ef518c74a98d7b6675ea5954ae08b1bda5554e
-
- 22 3月, 2019 3 次提交
-
-
由 Rashmi Sharma 提交于
Summary: [RocksDB] Make it easier for users to load options from option file and set shared block cache. Right now, it requires several dynamic casting for users to set the shared block cache to their option struct cast from the option file. If people don't do that, every CF of every DB will generate its own 8MB block cache. It's not a usable setting. So we are dragging every user who loads options from the file into such a mess. Instead, we should allow them to pass their cache object to LoadLatestOptions() and LoadOptionsFromFile(), so that those loaded option structs will have the shared block cache. Pull Request resolved: https://github.com/facebook/rocksdb/pull/5063 Differential Revision: D14518584 Pulled By: rashmishrm fbshipit-source-id: c91430ff9425a0e67d76fc67931d755f491ca5aa
-
由 Burton Li 提交于
Summary: The original implementation of WinEnvIO::NowNanos() has a constant data overflow by: li.QuadPart *= std::nano::den; As a result, the api provides a incorrect result. e.g.: li.QuadPart=13477844301545 std::nano::den=1e9 The fix uses pre-computed nano_seconds_per_period_ to present the nano seconds per performance counter period, in the case if nano::den is divisible by perf_counter_frequency_. Otherwise it falls back to use high_resolution_clock. siying ajkr Pull Request resolved: https://github.com/facebook/rocksdb/pull/5062 Differential Revision: D14426842 Pulled By: anand1976 fbshipit-source-id: 127f1daf423dd4b30edd0dcf8ea0466f468bec12
-
由 Maysam Yabandeh 提交于
Summary: The patch reorders DBIter fields to put 1-byte fields together and let the compiler optimize the memory usage by using less 64-bit allocations for bools and enums. This might have a negative side effect of putting the variables that are accessed together into different cache lines and hence increasing the cache misses. Not sure what benchmark would verify that thought. I ran simple, single-threaded seekrandom benchmarks but the variance in the results is too much to be conclusive. ./db_bench --benchmarks=fillrandom --use_existing_db=0 --num=1000000 --db=/dev/shm/dbbench ./db_bench --benchmarks=seekrandom[X10] --use_existing_db=1 --db=/dev/shm/dbbench --num=1000000 --duration=60 --seek_nexts=100 Pull Request resolved: https://github.com/facebook/rocksdb/pull/5078 Differential Revision: D14562676 Pulled By: maysamyabandeh fbshipit-source-id: 2284655d46e079b6e9a860e94be5defb6f482167
-
- 21 3月, 2019 3 次提交
-
-
由 Levi Tamasi 提交于
Summary: The patch adds a new config option to LRUCacheOptions that enables users to choose whether to use an adaptive mutex for the LRU block cache (on platforms where adaptive mutexes are supported). The default is true if RocksDB is compiled with -DROCKSDB_DEFAULT_TO_ADAPTIVE_MUTEX, false otherwise. Pull Request resolved: https://github.com/facebook/rocksdb/pull/5054 Differential Revision: D14542749 Pulled By: ltamasi fbshipit-source-id: 0065715ab6cf91f10444b737fed8c8aee6a8a0d2
-
由 Alexandre Viau 提交于
Summary: Pull Request resolved: https://github.com/facebook/rocksdb/pull/5086 Differential Revision: D14542212 Pulled By: siying fbshipit-source-id: db2f38a3f7c9b64532655a5d4ac4b7715c392883
-
由 anand76 提交于
Summary: The stack buffer in rocksdb::autovector is currently defined as an array of elements of the template type. This results in unnecessary construction of those objects, which can be a significant overhead in some cases. This PR changes the type of the stack buf to char* and uses placement new to construct new objects when they are inserted into the autovector. Pull Request resolved: https://github.com/facebook/rocksdb/pull/5080 Differential Revision: D14533221 Pulled By: anand1976 fbshipit-source-id: 9378985c7d03f4e1a28951bdd2403c72f10f23d7
-
- 20 3月, 2019 4 次提交
-
-
由 Zhongyi Xie 提交于
Summary: In order to better understand compaction done by different priority thread pool, we now collect compaction stats by priority and also print them to info LOG through stats dump. ``` ** Compaction Stats [default] ** Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Low 0/0 0.00 KB 0.0 16.8 11.3 5.5 5.6 0.1 0.0 0.0 406.4 136.1 42.24 34.96 45 0.939 13M 8865K High 0/0 0.00 KB 0.0 0.0 0.0 0.0 11.4 11.4 0.0 0.0 0.0 76.2 153.00 35.74 12185 0.013 0 0 ``` Pull Request resolved: https://github.com/facebook/rocksdb/pull/5050 Differential Revision: D14408583 Pulled By: miasantreble fbshipit-source-id: e53746586ea27cb8abc9fec35805bd80ed30f608
-
由 Andrew Audibert 提交于
Summary: BackupEngine relies on write-ahead logs to back up the memtable. Disabling write-ahead logs can result in backups failing to preserve unflushed keys. This PR updates the documentation to specify this behavior, and suggest always flushing the memtable when write-ahead logs are disabled. Pull Request resolved: https://github.com/facebook/rocksdb/pull/5071 Differential Revision: D14524124 Pulled By: miasantreble fbshipit-source-id: 635f855f8a42ad60273b5efd226139b511e3e5d5
-
由 Wenjie Yang 提交于
Summary: Add an option to filter out READ or WRITE operations while tracing. Pull Request resolved: https://github.com/facebook/rocksdb/pull/5082 Differential Revision: D14515083 Pulled By: mrmiywj fbshipit-source-id: 2504c89a9abf1dd629cad44b4104092702d77610
-
由 Hiroaki Nakamura 提交于
Summary: Partly addresses https://github.com/facebook/rocksdb/issues/4999 I verified `make static_lib` runs fine. Pull Request resolved: https://github.com/facebook/rocksdb/pull/5077 Differential Revision: D14521101 Pulled By: maysamyabandeh fbshipit-source-id: ba88e74a51d2d793cac7260d505b1a54254b53af
-
- 19 3月, 2019 2 次提交
-
-
由 Shobhit Dayal 提交于
Summary: This is a feature to sample data-block compressibility and and report them as stats. 1 in N (tunable) blocks is sampled for compressibility using two algorithms: 1. lz4 or snappy for fast compression 2. zstd or zlib for slow but higher compression. The stats are reported to the caller as raw-bytes and compressed-bytes. The block continues to be compressed for storage using the specified CompressionType. The db_bench_tool how has a command line option for specifying the sampling rate. It's default value is 0 (no sampling). To test the overhead for a certain value, users can compare the performance of db_bench_tool, varying the sampling rate. It is unlikely to have a noticeable impact for high values like 20. Pull Request resolved: https://github.com/facebook/rocksdb/pull/4842 Differential Revision: D13629011 Pulled By: shobhitdayal fbshipit-source-id: 14ca668bcab6499b2a1734edf848eb62a4f4fafa
-
由 He Zhe 提交于
Summary: Initialize magic_number to zero to avoid such failure. utilities/blob_db/blob_log_format.cc:91:3: error: 'magic_number' may be used uninitialized in this function [-Werror=maybe-uninitialized] if (magic_number != kMagicNumber) { ^~ Signed-off-by: NHe Zhe <zhe.he@windriver.com> Pull Request resolved: https://github.com/facebook/rocksdb/pull/5074 Differential Revision: D14505514 Pulled By: miasantreble fbshipit-source-id: 4334462958c2b9c5a7c68c6ab24dadf94ad70902
-
- 16 3月, 2019 2 次提交
-
-
由 anand76 提交于
Summary: There is a potential failure case in DBImpl::SwitchMemtable() that is not handled properly. The call to cur_log_writer->WriteBuffer() can fail due to an IO error. In that case, we need to call SetBGError() in order set the background error since the WriteBuffer() failure may result in data loss. Also, the asserts for !new_mem and !new_log are incorrect, as those would have been allocated by the time this failure is detected. Pull Request resolved: https://github.com/facebook/rocksdb/pull/5072 Differential Revision: D14461384 Pulled By: anand1976 fbshipit-source-id: fb59bce9d61378f37d2dfcd28c0b704b0f43c3cf
-
由 Andrew Kryczka 提交于
Summary: Since this feature affects the WAL behavior, it seems important our crash-recovery tests cover it. Pull Request resolved: https://github.com/facebook/rocksdb/pull/5070 Differential Revision: D14470085 Pulled By: miasantreble fbshipit-source-id: 9b9682a718a926d57d055e0a5ec867efbd2eb9c1
-
- 15 3月, 2019 1 次提交
-
-
由 Zhichao Cao 提交于
Summary: In the current trace_analyzer implementation, once the trace file has corrupted content, which can be caused by unexpected tracing operations or other reasons, trace_analyzer will print the error and stop analyzing. By adding the -try_process_corrupted_trace option, user can try to process the corrupted trace file and get the analyzing results of the trace records from the beginning to the the first corrupted point in the trace file. Analyzing might fail even this option is enabled. Pull Request resolved: https://github.com/facebook/rocksdb/pull/5067 Differential Revision: D14433037 Pulled By: zhichao-cao fbshipit-source-id: d095233ba371726869af0def0cdee23b69896831
-
- 13 3月, 2019 2 次提交
-
-
由 Zhongyi Xie 提交于
Summary: Pull Request resolved: https://github.com/facebook/rocksdb/pull/5061 Differential Revision: D14418581 Pulled By: miasantreble fbshipit-source-id: be7f90e16586666ddd0cce36971e403782ab0892
-
由 Andrew Kryczka 提交于
Summary: Without `total_order_seek=true`, using this command with `prefix_extractor` set skips over lots of keys. Pull Request resolved: https://github.com/facebook/rocksdb/pull/5066 Differential Revision: D14425967 Pulled By: sagar0 fbshipit-source-id: f6f142733258d92604f920615be9266e1fe797f8
-
- 09 3月, 2019 5 次提交
-
-
由 Yi Wu 提交于
Summary: JEMALLOC_CXX_THROW is not defined for earlier versions of jemalloc (e.g. 3.6), causing builds to fail on some platforms. Fixing it. Closes #4869 Pull Request resolved: https://github.com/facebook/rocksdb/pull/5053 Differential Revision: D14390034 Pulled By: sagar0 fbshipit-source-id: b2b7a03cd377201ef385eb521f65bae85c558055
-
由 Maysam Yabandeh 提交于
Summary: Pull Request resolved: https://github.com/facebook/rocksdb/pull/5055 Differential Revision: D14395944 Pulled By: maysamyabandeh fbshipit-source-id: 385062b59428c132ada4e49b327685ba1f5d30e6
-
由 Zhichao Cao 提交于
Summary: In the MixGraph benchmark of db_bench, The max buffer size used for value of KV-pair might be extremely large (64MB), which might cause function stack overflow in some platforms, reduced to 1MB. Added the finished ops printing in MixGraph benchmark. Pull Request resolved: https://github.com/facebook/rocksdb/pull/5051 Differential Revision: D14379571 Pulled By: zhichao-cao fbshipit-source-id: 24084fbe38f60f2902d9a40f6bc9a25e4e2c9bb9
-
由 Yi Wu 提交于
Summary: Print data block index type related options to info log Pull Request resolved: https://github.com/facebook/rocksdb/pull/5039 Differential Revision: D14387718 Pulled By: miasantreble fbshipit-source-id: 9df8f82eea83a8344c7d12a712486f656691bc4a
-
由 Maysam Yabandeh 提交于
Summary: Add a mutex to the test to synchronize before accessing the shared txn object. Pull Request resolved: https://github.com/facebook/rocksdb/pull/5052 Differential Revision: D14386861 Pulled By: maysamyabandeh fbshipit-source-id: 5b32e209840b210c35af53848dc77f489a76c95a
-
- 08 3月, 2019 1 次提交
-
-
由 Levi Tamasi 提交于
Summary: Remove BlobDBOptions.garbage_collection_interval_secs for now, since garbage collection is not yet implemented in BlobDB. Pull Request resolved: https://github.com/facebook/rocksdb/pull/5044 Differential Revision: D14354046 Pulled By: ltamasi fbshipit-source-id: 2b966b6d1e088ba9462f3ea73e115013562fbc04
-
- 07 3月, 2019 2 次提交
-
-
由 Maysam Yabandeh 提交于
Summary: The patch fixes an improbable race condition between AddPrepared from one write queue and AdvanceMaxEvictedSeq from another queue. In this scenario AddPrepared finds prepare_seq lower than max and adding to PrepareHeap as usual while AdvanceMaxEvictedSeq has finished checking PrepareHeap against the future max. Thus when AdvanceMaxEvictedSeq finishes off by updating the max_evicted_seq_, PrepareHeap ends up with a prepared_seq lower than it which breaks the PrepareHeap contract. The fix is that in AddPrepared we check against the future_max_evicted_seq_ instead, which is update before AdvanceMaxEvictedSeq acquire prepare_mutex_ and looks into PrepareHeap. A unit test added to test for the failure scenario. The code is also refactored a bit to remove the duplicate code between AdvanceMaxEvictedSeq and AddPrepared. Pull Request resolved: https://github.com/facebook/rocksdb/pull/5025 Differential Revision: D14249028 Pulled By: maysamyabandeh fbshipit-source-id: 072ea56663f40359662c05fafa6ac524417b0622
-
由 Maysam Yabandeh 提交于
Summary: The patch adds the sequence number of the rollback patch to the PrepareHeap when two_write_queues is enabled. Although the current behavior is still correct, the change simplifies reasoning about the code, by having all uncommitted batches registered with the PreparedHeap. Pull Request resolved: https://github.com/facebook/rocksdb/pull/5026 Differential Revision: D14249401 Pulled By: maysamyabandeh fbshipit-source-id: 1e3424edee5cd14e56ee35931ad3c93ed997cd5a
-
- 05 3月, 2019 2 次提交
-
-
由 Andrew Kryczka 提交于
Summary: The compiler flag `-DROCKSDB_FALLOCATE_PRESENT` was only set when `fallocate`, `FALLOC_FL_KEEP_SIZE`, and `FALLOC_FL_PUNCH_HOLE` were all present. However, the last of the three is not really necessary for the primary `fallocate` use case; furthermore, it was introduced only in later Linux kernel versions (2.6.38+). This PR changes the flag `-DROCKSDB_FALLOCATE_PRESENT` to only require `fallocate` and `FALLOC_FL_KEEP_SIZE` to be present. There is a separate check for `FALLOC_FL_PUNCH_HOLE` only in the place where it is used. Pull Request resolved: https://github.com/facebook/rocksdb/pull/5023 Differential Revision: D14248487 Pulled By: siying fbshipit-source-id: a10ed0b902fa755988e957bd2dcec9081ec0502e
-
由 SeterKwok 提交于
Summary: Fix #5008 Pull Request resolved: https://github.com/facebook/rocksdb/pull/5037 Differential Revision: D14302474 Pulled By: riversand963 fbshipit-source-id: dcd9dda5d4d6d459315692f355499a39e546d518
-
- 02 3月, 2019 6 次提交
-
-
由 Sagar Vemuri 提交于
Summary: Disabling `org.rocksdb.RocksDBTest.getApproximateSizes` test as it is frequently crashing on travis (#5020). It will be re-enabled once the root-cause is found and fixed. Pull Request resolved: https://github.com/facebook/rocksdb/pull/5035 Differential Revision: D14294736 Pulled By: sagar0 fbshipit-source-id: e28bff0d143a58ad6c82991fec3d4cf8c0209995
-
由 Siying Dong 提交于
Summary: This reverts commit ee181808. We are not ready to deprecate this feature. revert it for now. Pull Request resolved: https://github.com/facebook/rocksdb/pull/5034 Differential Revision: D14287246 Pulled By: siying fbshipit-source-id: e4beafdeaee1c94364fdaa6ba198218d158339f7
-
由 Andrew Kryczka 提交于
Summary: Added an option, `-use_existing_keys`, which can be set to run benchmarks against an arbitrary existing database. Now users can benchmark against their actual database rather than synthetic data. Before the run begins, it loads all the keys into memory, then uses that set of keys rather than synthesizing new ones in `GenerateKeyFromInt`. This is mainly intended for small-scale DBs where the memory consumption is not a concern. Pull Request resolved: https://github.com/facebook/rocksdb/pull/5017 Differential Revision: D14270303 Pulled By: riversand963 fbshipit-source-id: 6328df9dffb5e19170270dd00a69f4bbe424e5ed
-
由 Siying Dong 提交于
Summary: Right now, users can change statistics.stats_level while DB is running, but TSAN may report data race. We make stats_level_ to be atomic, and access them using accessors. Pull Request resolved: https://github.com/facebook/rocksdb/pull/5030 Differential Revision: D14267519 Pulled By: siying fbshipit-source-id: 37d7ebeff7a43a406230143422a16af899163f73
-
由 Fosco Marotto 提交于
[sync fix] Add defs.bzl
-
由 Maysam Yabandeh 提交于
Summary: This the hackish script we used to find the root cause of failures in transaction stress tests. It is not well-written and does not require rigorous reviewing but it is better than starting from scratch each time we observe an issue. The stress tests would just say that at which snapshots the sum of all the keys in a set is inconsistent with another set. To help debugging one need to know which key exactly returned inconsistent results. The script looks at the transactions between two conflicting snapshots, and performs thee changes manually to see for which key the read value was inconsistent. Pull Request resolved: https://github.com/facebook/rocksdb/pull/5033 Differential Revision: D14280362 Pulled By: maysamyabandeh fbshipit-source-id: d5826055c46711460ba81480d96cb5ea082814a5
-
- 01 3月, 2019 2 次提交
-
-
由 Maysam Yabandeh 提交于
Summary: PreReleaseCallback meant to be called before the writes are visible to the readers. Since the sequence number is known after the WAL write, there is no reason to delay calling PreReleaseCallback to after the memtable write, which would complicates the reader's logic in presence of our memtable writes that are made visible by the other write thread. Pull Request resolved: https://github.com/facebook/rocksdb/pull/5015 Differential Revision: D14221670 Pulled By: maysamyabandeh fbshipit-source-id: a504dd665cf923226d7af09cc8e9c7739a25edc6
-
由 Maysam Yabandeh 提交于
Summary: When two_write_queues is enabled we call ::AddPrepared only from the main queue, which writes to both WAL and memtable, and call ::AddCommitted from the 2nd queue, which writes only to WAL. This simplifies the logic by avoiding concurrency between AddPrepared and also between AddCommitted. The patch fixes one case that did not conform with the rule above. This would allow future refactoring. For example AdvaneMaxEvictedSeq, which is invoked by AddCommitted, can be simplified by assuming lack of concurrent calls to it. Pull Request resolved: https://github.com/facebook/rocksdb/pull/5014 Differential Revision: D14210493 Pulled By: maysamyabandeh fbshipit-source-id: 6db5ba372a294a568a14caa010576460917a4eab
-