- 24 8月, 2018 1 次提交
-
-
由 Yanqin Jin 提交于
Summary: We want to sample the file I/O issued by RocksDB and report the function calls. This requires us to include the file paths otherwise it's hard to tell what has been going on. Pull Request resolved: https://github.com/facebook/rocksdb/pull/4039 Differential Revision: D8670178 Pulled By: riversand963 fbshipit-source-id: 97ee806d1c583a2983e28e213ee764dc6ac28f7a
-
- 17 8月, 2018 2 次提交
-
-
由 Fenggang Wu 提交于
Summary: Add `--data_block_index_type` and `--data_block_hash_table_util_ratio` option to `db_bench`. `--data_block_index_type` can be either of `binary` (default) or `binary_and_hash`; `--data_block_hash_table_util_ratio` will be a double. The default value is `0.75`. Pull Request resolved: https://github.com/facebook/rocksdb/pull/4281 Differential Revision: D9361476 Pulled By: fgwu fbshipit-source-id: dc53e01acef9db81b9eec5e8a96f3bc8ed718c10
-
由 Siying Dong 提交于
Summary: Right now, `ldb idump` may have memory out of control if there is a big range of tombstones. Add an option to cut maxinum number of keys in GetAllKeyVersions(), and push down --max_num_ikeys from ldb. Pull Request resolved: https://github.com/facebook/rocksdb/pull/4271 Differential Revision: D9369149 Pulled By: siying fbshipit-source-id: 7cbb797b7d2fa16573495a7e84937456d3ff25bf
-
- 15 8月, 2018 3 次提交
-
-
由 Zhichao Cao 提交于
Summary: The wrong options are used in the trace_analyzer_test, removed. The potential loses integer precision are fixed. Pass the specified testing case, make asan_check Pull Request resolved: https://github.com/facebook/rocksdb/pull/4274 Reviewed By: yiwu-arbug Differential Revision: D9327811 Pulled By: zhichao-cao fbshipit-source-id: d62cb18d6586503a490cd323bfc1c672b68b346e
-
由 Anand Ananthabhotla 提交于
Summary: In the OnTableFileCreation() listener, assert on various TableProperties only when file size > 0 bytes. The listener can get called even for 0 byte SSTs which have been deleted. Pull Request resolved: https://github.com/facebook/rocksdb/pull/4273 Differential Revision: D9322738 Pulled By: anand1976 fbshipit-source-id: 17cdfb3d0da946b9a158d7328e5db1c87973956b
-
由 Maysam Yabandeh 提交于
Summary: Stress tests currently cover format_version 2 and 3. The patch adds 4 as well. Pull Request resolved: https://github.com/facebook/rocksdb/pull/4265 Differential Revision: D9323185 Pulled By: maysamyabandeh fbshipit-source-id: 54d11e41ecae09bae14cadd7313f07c9a3db5a57
-
- 14 8月, 2018 2 次提交
-
-
由 Zhichao Cao 提交于
Summary: A framework of trace analyzing for RocksDB After collecting the trace by using the tool of [PR #3837](https://github.com/facebook/rocksdb/pull/3837). User can use the Trace Analyzer to interpret, analyze, and characterize the collected workload. **Input:** 1. trace file 2. Whole keys space file **Statistics:** 1. Access count of each operation (Get, Put, Delete, SingleDelete, DeleteRange, Merge) in each column family. 2. Key hotness (access count) of each one 3. Key space separation based on given prefix 4. Key size distribution 5. Value size distribution if appliable 6. Top K accessed keys 7. QPS statistics including the average QPS and peak QPS 8. Top K accessed prefix 9. The query correlation analyzing, output the number of X after Y and the corresponding average time intervals **Output:** 1. key access heat map (either in the accessed key space or whole key space) 2. trace sequence file (interpret the raw trace file to line base text file for future use) 3. Time serial (The key space ID and its access time) 4. Key access count distritbution 5. Key size distribution 6. Value size distribution (in each intervals) 7. whole key space separation by the prefix 8. Accessed key space separation by the prefix 9. QPS of each operation and each column family 10. Top K QPS and their accessed prefix range **Test:** 1. Added the unit test of analyzing Get, Put, Delete, SingleDelete, DeleteRange, Merge 2. Generated the trace and analyze the trace **Implemented but not tested (due to the limitation of trace_replay):** 1. Analyzing Iterator, supporting Seek() and SeekForPrev() analyzing 2. Analyzing the number of Key found by Get **Future Work:** 1. Support execution time analyzing of each requests 2. Support cache hit situation and block read situation of Get Pull Request resolved: https://github.com/facebook/rocksdb/pull/4091 Differential Revision: D9256157 Pulled By: zhichao-cao fbshipit-source-id: f0ceacb7eedbc43a3eee6e85b76087d7832a8fe6
-
由 Yanqin Jin 提交于
Summary: Due to 4ea56b1b, we should also remove the assersion in stress test. This removal can be temporary, and we can add it back once we figure out the reason for the 0-byte SSTs. Pull Request resolved: https://github.com/facebook/rocksdb/pull/4268 Differential Revision: D9297186 Pulled By: riversand963 fbshipit-source-id: cebba9a68f42e815f8cf24471176d2cfdf962f63
-
- 10 8月, 2018 4 次提交
-
-
由 Yanqin Jin 提交于
Summary: TSAN fails due to comparison between signed int and unsigned long. Fix it by static_casting. Pull Request resolved: https://github.com/facebook/rocksdb/pull/4250 Differential Revision: D9256535 Pulled By: riversand963 fbshipit-source-id: c6bad23ff70c6d0ec58e2e85c401ce0ad45de609
-
由 Dmitri Smirnov 提交于
Summary: Although delete scheduler implementation allows for the interface not to be supported, the delete_scheduler_test does not allow for that. Address compiler warnings Make sst_dump_test use test directory structure as the current execution directory may not be writiable. Pull Request resolved: https://github.com/facebook/rocksdb/pull/4221 Differential Revision: D9210152 Pulled By: siying fbshipit-source-id: 381a74511e969ecb8089d5c4b4df87dc30c8df63
-
由 Yanqin Jin 提交于
Summary: We add two subcommands `write_extern_sst` and `ingest_extern_sst` to ldb. This PR avoids changing existing code because we hope to cherry-pick to earlier releases to support compatibility check for external SST file ingestion. Pull Request resolved: https://github.com/facebook/rocksdb/pull/4205 Differential Revision: D9112711 Pulled By: riversand963 fbshipit-source-id: 7cae88380d4de86da8440230e87eca66755648e4
-
由 Andrew Kryczka 提交于
Summary: db_bench's previous default compression level (-1) was not the default compression level in all libraries. In particular, in ZSTD negative values are valid compression levels, while ZSTD's default compression level is three. This PR changes db_bench's default to be RocksDB's library-independent default compression level (see #3895). I also changed a couple other flags to get their default values from an options object directly rather than hardcoding. Pull Request resolved: https://github.com/facebook/rocksdb/pull/4248 Differential Revision: D9235140 Pulled By: ajkr fbshipit-source-id: be4e0722d59fa1968832183db36d1d20fcf11e5b
-
- 07 8月, 2018 1 次提交
-
-
由 Andrew Kryczka 提交于
Summary: - Add `--compression_max_dict_bytes` and `--compression_zstd_max_train_bytes` flags to stress test - Randomly enable/disable the above flags in crash test - Set `--compression_type=zstd` in FB-specific crash test runs Pull Request resolved: https://github.com/facebook/rocksdb/pull/4234 Differential Revision: D9187207 Pulled By: ajkr fbshipit-source-id: 8d78cf8d8e1165f2cd1c32e069b73726b5bc1fd2
-
- 04 8月, 2018 1 次提交
-
-
由 Sagar Vemuri 提交于
Summary: Fix lite build failure in db_bench due to trace/replay feature. Pull Request resolved: https://github.com/facebook/rocksdb/pull/4225 Differential Revision: D9153303 Pulled By: sagar0 fbshipit-source-id: 9f7a8035429d0dcdbe99616d11389ed7bccf44be
-
- 03 8月, 2018 1 次提交
-
-
由 Pooja Malik 提交于
Summary: This PR includes fixes for some bugs that I encountered while testing the Optimizer with ODS stats support. Pull Request resolved: https://github.com/facebook/rocksdb/pull/4223 Differential Revision: D9140786 Pulled By: poojam23 fbshipit-source-id: 045cb3f27d075c2042040ac2d561938349419516
-
- 02 8月, 2018 1 次提交
-
-
由 Pooja Malik 提交于
Summary: This pull request adds a README file and a blog post for the Advisor tool. It also adds the missing tests for some Optimizer modules. Some comments are added to the classes being tested for improved readability. Pull Request resolved: https://github.com/facebook/rocksdb/pull/4201 Reviewed By: maysamyabandeh Differential Revision: D9125311 Pulled By: poojam23 fbshipit-source-id: aefcf2f06eaa05490cc2834ef5aa6e21f0d1dc55
-
- 01 8月, 2018 1 次提交
-
-
由 Sagar Vemuri 提交于
Summary: A framework for tracing and replaying RocksDB operations. A binary trace file is created by capturing the DB operations, and it can be replayed back at the same rate using db_bench. - Column-families are supported - Multi-threaded tracing is supported. - TraceReader and TraceWriter are exposed to the user, so that tracing to various destinations can be enabled (say, to other messaging/logging services). By default, a FileTraceReader and FileTraceWriter are implemented to capture to a file and replay from it. - This is not yet ideal to be enabled in production due to large performance overhead, but it can be safely tried out in a shadow setup, say, for analyzing RocksDB operations. Currently supported DB operations: - Writes: -- Put -- Merge -- Delete -- SingleDelete -- DeleteRange -- Write - Reads: -- Get (point lookups) Pull Request resolved: https://github.com/facebook/rocksdb/pull/3837 Differential Revision: D7974837 Pulled By: sagar0 fbshipit-source-id: 8ec65aaf336504bc1f6ed0feae67f6ed5ef97a72
-
- 31 7月, 2018 1 次提交
-
-
由 Yanqin Jin 提交于
Summary: Making generation of column families and keys virtual function so that subclasses of StressTest can override them to provide custom parameter generation for more flexibility. This will be useful for future tests. Pull Request resolved: https://github.com/facebook/rocksdb/pull/4046 Differential Revision: D9073382 Pulled By: riversand963 fbshipit-source-id: 2754f0fdfa5c24d95c1f92d4944bc479552fb665
-
- 28 7月, 2018 2 次提交
-
-
由 Yanqin Jin 提交于
Summary: RocksDB used to store global_seqno in external SST files written by SstFileWriter. During file ingestion, RocksDB uses `pwrite` to update the `global_seqno`. Since random write is not supported in some non-POSIX compliant file systems, external SST file ingestion is not supported on these file systems. To address this limitation, we no longer update `global_seqno` during file ingestion. Later RocksDB uses the MANIFEST and other information in table properties to deduce global seqno for externally-ingested SST files. Pull Request resolved: https://github.com/facebook/rocksdb/pull/4172 Differential Revision: D8961465 Pulled By: riversand963 fbshipit-source-id: 4382ec85270a96be5bc0cf33758ca2b167b05071
-
由 DorianZheng 提交于
Summary: If crash happen after a hard link established, Recover function may reuse the file number that has already assigned to the internal file, and this will overwrite the external file. To protect the external file, we have to make sure the file number will never being reused. Pull Request resolved: https://github.com/facebook/rocksdb/pull/4099 Differential Revision: D9034092 Pulled By: riversand963 fbshipit-source-id: 3f1a737440b86aa2ef01673e5013aacbb7c33e28
-
- 27 7月, 2018 1 次提交
-
-
由 Pooja Malik 提交于
Summary: In https://github.com/facebook/rocksdb/pull/3934 we introduced advisor scripts that make suggestions in the config options based on the log file and stats from a run of rocksdb. The optimizer runs the advisor on a benchmark application in a loop and automatically applies the suggested changes until the config options are optimized. This is a work in progress and the patch is the initial skeleton for the optimizer. The sample application that is run in the loop is currently dbbench. Pull Request resolved: https://github.com/facebook/rocksdb/pull/4169 Reviewed By: maysamyabandeh Differential Revision: D9023671 Pulled By: poojam23 fbshipit-source-id: a6192d475c462cf6eb2b316716f97cb400fcb64d
-
- 24 7月, 2018 1 次提交
-
-
由 Siying Dong 提交于
Summary: db_stress doesn't cover upper or lower bound in iterators. Try to cover it by randomly assigning a random one. Also in prefix scan tests, with 50% of the chance, set next prefix as the upper bound. Pull Request resolved: https://github.com/facebook/rocksdb/pull/4162 Differential Revision: D8953507 Pulled By: siying fbshipit-source-id: f0f04e9cb6c07cbebbb82b892ca23e0daeea708b
-
- 21 7月, 2018 2 次提交
-
-
由 Zhichao Cao 提交于
Summary: When running the tracing and analyzing, I found that MergeRandom benchmark in db_bench only access the default column family even the -num_column_families is specified > 1. changes: Using the db_with_cfh as DB to randomly select the column family to execute the Merge operation if -num_column_families is specified > 1. Tested with make asan_check and verified in tracing Pull Request resolved: https://github.com/facebook/rocksdb/pull/4155 Differential Revision: D8907888 Pulled By: zhichao-cao fbshipit-source-id: 2b4bc8fe0e99c8f262f5be6b986c7025d62cf850
-
由 Siying Dong 提交于
Summary: Lint is not happy with some new code recently committed. Format them. Pull Request resolved: https://github.com/facebook/rocksdb/pull/4161 Differential Revision: D8940582 Pulled By: siying fbshipit-source-id: c9b43b1ef8c88b5e923911058b44eb77234b36b7
-
- 19 7月, 2018 1 次提交
-
-
由 Pooja Malik 提交于
Summary: Adding the string "PERF_CONTEXT:" before the perf_context stats are printed. Setting the filter policy if it's a block based table even when options are being loaded from the provided FLAGS_options_file. Pull Request resolved: https://github.com/facebook/rocksdb/pull/4153 Differential Revision: D8905517 Pulled By: poojam23 fbshipit-source-id: 5956ed7882d39ec8ae654d5dadeb88727a36f0dd
-
- 14 7月, 2018 2 次提交
-
-
由 Maysam Yabandeh 提交于
Summary: The patch makes sure that two parallel test threads will operate on different db paths. This enables using open source tools such as gtest-parallel to run the tests of a file in parallel. Example: ``` ~/gtest-parallel/gtest-parallel ./table_test``` Pull Request resolved: https://github.com/facebook/rocksdb/pull/4135 Differential Revision: D8846653 Pulled By: maysamyabandeh fbshipit-source-id: 799bad1abb260e3d346bcb680d2ae207a852ba84
-
由 Zhongyi Xie 提交于
Summary: Pull Request resolved: https://github.com/facebook/rocksdb/pull/4118 Differential Revision: D8845554 Pulled By: miasantreble fbshipit-source-id: 13bd3c1259a7c30bad762a413fe3bb24eea650ba
-
- 13 7月, 2018 3 次提交
-
-
由 Zhongyi Xie 提交于
Summary: Right now there is no support for enabling compaction filter in db_bench, we should add support for that to facilitate testing of compaction filter. This PR adds a compaction filter called KeepFilter and make `Filter` always returns false, essentially a noop compaction filter. This will allow us to test compaction filter code path without having to support arbitrary compaction filters Pull Request resolved: https://github.com/facebook/rocksdb/pull/4106 Differential Revision: D8828517 Pulled By: miasantreble fbshipit-source-id: 9ad76d04103eaa9d00da98334b4a39e542d26c41
-
由 Andrew Kryczka 提交于
Summary: `DEFINE_uint32` was unavailable on some platforms, e.g., https://travis-ci.org/facebook/rocksdb/jobs/403352902. Use `DEFINE_uint64` instead which should work as it's used many times elsewhere in this file. Pull Request resolved: https://github.com/facebook/rocksdb/pull/4129 Differential Revision: D8830311 Pulled By: ajkr fbshipit-source-id: b4fc90ba3f50e649c070ce8069c68e530d731f05
-
由 Andrew Kryczka 提交于
Summary: give control of how often stats are printed, including jemalloc stats if enabled. Previously the default was 10 minutes so we'd only see updated stats for very long benchmark runs. Pull Request resolved: https://github.com/facebook/rocksdb/pull/4109 Differential Revision: D8796444 Pulled By: ajkr fbshipit-source-id: fd7902fe3f105fae89322c4ab63316bba4a2b15e
-
- 07 7月, 2018 1 次提交
-
-
由 Manuel Ung 提交于
Summary: This adds support for recovering WriteUnprepared transactions through the following changes: - The information in `RecoveredTransaction` is extended so that it can reference multiple batches. - `MarkBeginPrepare` is extended with a bool indicating whether it is an unprepared begin, and this is passed down to `InsertRecoveredTransaction` to indicate whether the current transaction is prepared or not. - `WriteUnpreparedTxnDB::Initialize` is overridden so that it will rollback unprepared transactions from the recovered transactions. This can be done without updating the prepare heap/commit map, because this is before the DB has finished initializing, and after writing the rollback batch, those data structures should not contain information about the rolled back transaction anyway. Commit/Rollback of live transactions is still unimplemented and will come later. Pull Request resolved: https://github.com/facebook/rocksdb/pull/4078 Differential Revision: D8703382 Pulled By: lth fbshipit-source-id: 7e0aada6c23bd39299f1f20d6c060492e0e6b60a
-
- 28 6月, 2018 1 次提交
-
-
由 Maysam Yabandeh 提交于
Summary: https://github.com/facebook/rocksdb/pull/3881 fixed a bug where PinnableSlice pin mmap files which could be deleted with background compaction. This is however a non-issue for ReadOnlyDB when there is no compaction running and max_open_files is -1. This patch reenables the pinning feature for that case. Closes https://github.com/facebook/rocksdb/pull/4053 Differential Revision: D8662546 Pulled By: maysamyabandeh fbshipit-source-id: 402962602eb0f644e17822748332999c3af029fd
-
- 27 6月, 2018 1 次提交
-
-
由 Peter (Stig) Edwards 提交于
Summary: Also remove redefined variable. As reported on https://lgtm.com/projects/g/facebook/rocksdb/ Closes https://github.com/facebook/rocksdb/pull/4057 Differential Revision: D8648342 Pulled By: ajkr fbshipit-source-id: afd2ba84d1364d316010179edd44777e64ca9183
-
- 24 6月, 2018 1 次提交
-
-
由 Yanqin Jin 提交于
Summary: Closes https://github.com/facebook/rocksdb/pull/4045 Differential Revision: D8595609 Pulled By: riversand963 fbshipit-source-id: 5ba5954d804b82b0e7264b2e18e1da4c94103b53
-
- 23 6月, 2018 2 次提交
-
-
由 Maysam Yabandeh 提交于
Summary: Top-level index in partitioned index/filter blocks are small and could be pinned in memory. So far we use that by cache_index_and_filter_blocks to false. This however make it difficult to keep account of the total memory usage. This patch introduces pin_top_level_index_and_filter which in combination with cache_index_and_filter_blocks=true keeps the top-level index in cache and yet pinned them to avoid cache misses and also cache lookup overhead. Closes https://github.com/facebook/rocksdb/pull/4037 Differential Revision: D8596218 Pulled By: maysamyabandeh fbshipit-source-id: 3a5f7f9ca6b4b525b03ff6bd82354881ae974ad2
-
由 Yi Wu 提交于
Summary: Fix db_stress failed to delete checkpoint pointer. It's caught by asan_crash test. Closes https://github.com/facebook/rocksdb/pull/4042 Differential Revision: D8592604 Pulled By: yiwu-arbug fbshipit-source-id: 7b2d67d5e3dfb05f71c33fcf320482303e97d3ef
-
- 22 6月, 2018 2 次提交
-
-
由 Andrew Kryczka 提交于
Summary: - Attempt to clean the checkpoint staging directory before starting a checkpoint. It was already cleaned up at the end of checkpoint. But it wasn't cleaned up in the edge case where the process crashed while staging checkpoint files. - Attempt to clean the checkpoint directory before calling `Checkpoint::Create` in `db_stress`. This handles the case where checkpoint directory was created by a previous `db_stress` run but the process crashed before cleaning it up. - Use `DestroyDB` for cleaning checkpoint directory since a checkpoint is a DB. Closes https://github.com/facebook/rocksdb/pull/4035 Reviewed By: yiwu-arbug Differential Revision: D8580223 Pulled By: ajkr fbshipit-source-id: 28c667400e249fad0fdedc664b349031b7b61599
-
由 Yanqin Jin 提交于
Summary: Closes https://github.com/facebook/rocksdb/pull/4032 Differential Revision: D8573061 Pulled By: riversand963 fbshipit-source-id: 112324dcb35956d6b3ec891073f4f21493933c8b
-
- 21 6月, 2018 2 次提交
-
-
由 Yanqin Jin 提交于
Summary: We potentially need this information for tracing, profiling and diagnosis. Closes https://github.com/facebook/rocksdb/pull/4026 Differential Revision: D8555214 Pulled By: riversand963 fbshipit-source-id: 4263e06c00b6d5410b46aa46eb4e358ff2161dd2
-
由 Andrew Kryczka 提交于
Summary: Once per `ingest_external_file_one_in` operations, uses SstFileWriter to create a file containing `ingest_external_file_width` consecutive keys. The file is named containing the thread ID to avoid clashes. The file is then added to the DB using `IngestExternalFile`. We can't enable it by default in crash test because `nooverwritepercent` and `test_batches_snapshot` both must be zero for the DB's whole lifetime. Perhaps we should setup a separate test with that config as range deletion also requires it. Closes https://github.com/facebook/rocksdb/pull/4018 Differential Revision: D8507698 Pulled By: ajkr fbshipit-source-id: 1437ea26fd989349a9ce8b94117241c65e40f10f
-