- 18 6月, 2019 1 次提交
-
-
由 Zhongyi Xie 提交于
Summary: This PR continues the work in https://github.com/facebook/rocksdb/pull/4748 and https://github.com/facebook/rocksdb/pull/4535 by adding a new DBOption `persist_stats_to_disk` which instructs RocksDB to persist stats history to RocksDB itself. When statistics is enabled, and both options `stats_persist_period_sec` and `persist_stats_to_disk` are set, RocksDB will periodically write stats to a built-in column family in the following form: key -> (timestamp in microseconds)#(stats name), value -> stats value. The existing API `GetStatsHistory` will detect the current value of `persist_stats_to_disk` and either read from in-memory data structure or from the hidden column family on disk. Pull Request resolved: https://github.com/facebook/rocksdb/pull/5046 Differential Revision: D15863138 Pulled By: miasantreble fbshipit-source-id: bb82abdb3f2ca581aa42531734ac799f113e931b
-
- 14 6月, 2019 1 次提交
-
-
由 Patrick Zhang 提交于
Summary: Verified with an Ampere Computing eMAG aarch64 system. Pull Request resolved: https://github.com/facebook/rocksdb/pull/5258 Differential Revision: D15807309 Pulled By: maysamyabandeh fbshipit-source-id: ab85d2fd3fe40e6094430ab0eba557b1e979510d
-
- 12 6月, 2019 1 次提交
-
-
由 haoyuhuang 提交于
Summary: This PR contains the first commit for block cache trace analyzer. It reads a block cache trace file and prints statistics of the traces. We will extend this class to provide more functionalities. Pull Request resolved: https://github.com/facebook/rocksdb/pull/5425 Differential Revision: D15709580 Pulled By: HaoyuHuang fbshipit-source-id: 2f43bd2311f460ab569880819d95eeae217c20bb
-
- 07 6月, 2019 1 次提交
-
-
由 haoyuhuang 提交于
Summary: This PR adds a help class block cache tracer to read/write block cache accesses. It uses the trace reader/writer to perform this task. Pull Request resolved: https://github.com/facebook/rocksdb/pull/5410 Differential Revision: D15612843 Pulled By: HaoyuHuang fbshipit-source-id: f30fd1e1524355ca87db5d533a5c086728b141ea
-
- 01 6月, 2019 2 次提交
-
-
由 Siying Dong 提交于
Summary: Many logging related source files are under util/. It will be more structured if they are together. Pull Request resolved: https://github.com/facebook/rocksdb/pull/5387 Differential Revision: D15579036 Pulled By: siying fbshipit-source-id: 3850134ed50b8c0bb40a0c8ae1f184fa4081303f
-
由 Vijay Nadimpalli 提交于
Summary: Pull Request resolved: https://github.com/facebook/rocksdb/pull/5390 Differential Revision: D15579388 Pulled By: vjnadimpalli fbshipit-source-id: 5bfc95e31554b8ff05b97b76d6534113f527f366
-
- 31 5月, 2019 3 次提交
-
-
由 Siying Dong 提交于
Summary: Move arena, allocator, and memory tools under util to a separate memory/ directory. Pull Request resolved: https://github.com/facebook/rocksdb/pull/5382 Differential Revision: D15564655 Pulled By: siying fbshipit-source-id: 9cd6b5d0d3d52b39606e19221fa154596e5852a5
-
由 Vijay Nadimpalli 提交于
Summary: Pull Request resolved: https://github.com/facebook/rocksdb/pull/5373 Differential Revision: D15559425 Pulled By: vjnadimpalli fbshipit-source-id: 5d6d6d615582bedd96a4b879bb25d429a6de8b55
-
由 Siying Dong 提交于
Summary: There are too many types of files under util/. Some test related files don't belong to there or just are just loosely related. Mo ve them to a new directory test_util/, so that util/ is cleaner. Pull Request resolved: https://github.com/facebook/rocksdb/pull/5377 Differential Revision: D15551366 Pulled By: siying fbshipit-source-id: 0f5c8653832354ef8caa31749c0143815d719e2c
-
- 30 5月, 2019 1 次提交
-
-
由 Siying Dong 提交于
Summary: util/ means for lower level libraries, so it's a good idea to move the files which requires knowledge to DB out. Create a file/ and move some files there. Pull Request resolved: https://github.com/facebook/rocksdb/pull/5375 Differential Revision: D15550935 Pulled By: siying fbshipit-source-id: 61a9715dcde5386eebfb43e93f847bba1ae0d3f2
-
- 03 5月, 2019 1 次提交
-
-
由 Adam Retter 提交于
Summary: This allows debug releases of RocksJava to be build with the Docker release targets. Pull Request resolved: https://github.com/facebook/rocksdb/pull/5274 Differential Revision: D15185067 Pulled By: sagar0 fbshipit-source-id: f3988e472f281f5844d9a07098344a827b1e7eb1
-
- 01 5月, 2019 1 次提交
-
-
由 Yuqi Gu 提交于
Summary: 1. Add Arm linear crc32c implemtation for RocksDB. 2. Arm runtime check for crc32 Pull Request resolved: https://github.com/facebook/rocksdb/pull/5221 Differential Revision: D15013685 Pulled By: siying fbshipit-source-id: 2c2983743d26656d93f212dc7c1a3cf66a1acf12
-
- 27 3月, 2019 1 次提交
-
-
由 Yanqin Jin 提交于
Summary: This PR allows RocksDB to run in single-primary, multi-secondary process mode. The writer is a regular RocksDB (e.g. an `DBImpl`) instance playing the role of a primary. Multiple `DBImplSecondary` processes (secondaries) share the same set of SST files, MANIFEST, WAL files with the primary. Secondaries tail the MANIFEST of the primary and apply updates to their own in-memory state of the file system, e.g. `VersionStorageInfo`. This PR has several components: 1. (Originally in #4745). Add a `PathNotFound` subcode to `IOError` to denote the failure when a secondary tries to open a file which has been deleted by the primary. 2. (Similar to #4602). Add `FragmentBufferedReader` to handle partially-read, trailing record at the end of a log from where future read can continue. 3. (Originally in #4710 and #4820). Add implementation of the secondary, i.e. `DBImplSecondary`. 3.1 Tail the primary's MANIFEST during recovery. 3.2 Tail the primary's MANIFEST during normal processing by calling `ReadAndApply`. 3.3 Tailing WAL will be in a future PR. 4. Add an example in 'examples/multi_processes_example.cc' to demonstrate the usage of secondary RocksDB instance in a multi-process setting. Instructions to run the example can be found at the beginning of the source code. Pull Request resolved: https://github.com/facebook/rocksdb/pull/4899 Differential Revision: D14510945 Pulled By: riversand963 fbshipit-source-id: 4ac1c5693e6012ad23f7b4b42d3c374fecbe8886
-
- 23 2月, 2019 1 次提交
-
-
由 Adam Retter 提交于
Summary: This is my latest round of changes to add missing items to RocksJava. More to come in future PRs. Pull Request resolved: https://github.com/facebook/rocksdb/pull/4833 Differential Revision: D14152266 Pulled By: sagar0 fbshipit-source-id: d6cff67e26da06c131491b5cf6911a8cd0db0775
-
- 20 2月, 2019 1 次提交
-
-
由 Yanqin Jin 提交于
Summary: Currently crash test covers cases with and without atomic flush, but takes too long to finish. Therefore it may be a better idea to put crash test with atomic flush in a separate set of tests. Pull Request resolved: https://github.com/facebook/rocksdb/pull/4945 Differential Revision: D13947548 Pulled By: riversand963 fbshipit-source-id: 177c6de865290fd650b0103408339eaa3f801d8c
-
- 14 2月, 2019 1 次提交
-
-
由 Yanqin Jin 提交于
Summary: as title. For people who continue to need Lua compaction filter, you can copy the include/rocksdb/utilities/rocks_lua/lua_compaction_filter.h and utilities/lua/rocks_lua_compaction_filter.cc to your own codebase. Pull Request resolved: https://github.com/facebook/rocksdb/pull/4971 Differential Revision: D14047468 Pulled By: riversand963 fbshipit-source-id: 9ad1a6484a7c94e478f1e108127a3184e4069f70
-
- 29 1月, 2019 1 次提交
-
-
由 Yanqin Jin 提交于
Summary: We used to call `printf $(t_run)` and later feed the result to GNU parallel in the recipe of target `check_0`. However, this approach is problematic when the length of $(t_run) exceeds the maximum length of a command and the `printf` command cannot be executed. Instead we use 'find -print' to avoid generating an overly long command. **This PR is actually the last commit of #4916. Prefer to merge this PR separately.** Pull Request resolved: https://github.com/facebook/rocksdb/pull/4922 Differential Revision: D13845883 Pulled By: riversand963 fbshipit-source-id: b56de7f7af43337c6ec89b931de843c9667cb679
-
- 25 1月, 2019 1 次提交
-
-
由 Yanqin Jin 提交于
Summary: Pull Request resolved: https://github.com/facebook/rocksdb/pull/4912 Differential Revision: D13789449 Pulled By: riversand963 fbshipit-source-id: 0639dae82049b7ac977c8f81851f1c9fdc346705
-
- 24 1月, 2019 1 次提交
-
-
由 Remington Brasga 提交于
Summary: False-negative about path not existing. The regex is ignoring the "." in front of a path. Example: "./path/to/file" Pull Request resolved: https://github.com/facebook/rocksdb/pull/4682 Differential Revision: D13777110 Pulled By: sagar0 fbshipit-source-id: 9f8173b7581407555fdc055580732aeab37d4ade
-
- 12 1月, 2019 1 次提交
-
-
由 Varadharajan 提交于
Summary: Build failing due to incorrect filename. Pull Request resolved: https://github.com/facebook/rocksdb/pull/4870 Differential Revision: D13637205 Pulled By: sagar0 fbshipit-source-id: 72da45d51b49bce32f696532ba0656ee0dc2b89f
-
- 11 1月, 2019 1 次提交
-
-
由 Siying Dong 提交于
Summary: Remove some components that we never heard people using them. Pull Request resolved: https://github.com/facebook/rocksdb/pull/4101 Differential Revision: D8825431 Pulled By: siying fbshipit-source-id: 97a12ad3cad4ab12c82741a5ba49669aaa854180
-
- 20 12月, 2018 1 次提交
-
-
由 Jakub Tomanik 提交于
Summary: This PR contains the following fixes: 1. Fixing Makefile to support non-default locations of developer tools 2. Fixing compile error using a patch from https://github.com/facebook/rocksdb/pull/4007 Pull Request resolved: https://github.com/facebook/rocksdb/pull/4687 Differential Revision: D13287263 Pulled By: riversand963 fbshipit-source-id: 4525eb42ba7b6f82af5f9bfb8e52fa4024e27ccc
-
- 19 12月, 2018 1 次提交
-
-
由 Adam Retter 提交于
Summary: Note that Snappy now requires CMake to build it, so I added a note about RocksJava to the README.md file. Pull Request resolved: https://github.com/facebook/rocksdb/pull/4761 Differential Revision: D13403811 Pulled By: ajkr fbshipit-source-id: 8fcd7e3dc7f7152080364a374d3065472f417eff
-
- 18 12月, 2018 2 次提交
-
-
由 Abhishek Madan 提交于
Summary: Now that v2 is fully functional, the v1 aggregator is removed. The v2 aggregator has been renamed. Pull Request resolved: https://github.com/facebook/rocksdb/pull/4778 Differential Revision: D13495930 Pulled By: abhimadan fbshipit-source-id: 9d69500a60a283e79b6c4fa938fc68a8aa4d40d6
-
由 Adam Retter 提交于
Summary: Avoids re-downloading the .tar.gz files for the static build of RocksJava if they are already present. Pull Request resolved: https://github.com/facebook/rocksdb/pull/4769 Differential Revision: D13491919 Pulled By: ajkr fbshipit-source-id: 9265f577e049838dc40335d54f1ff2b4f972c38c
-
- 28 11月, 2018 1 次提交
-
-
由 Huachao Huang 提交于
Summary: A user friendly sst file reader is useful when we want to access sst files outside of RocksDB. For example, we can generate an sst file with SstFileWriter and send it to other places, then use SstFileReader to read the file and process the entries in other ways. Also rename the original SstFileReader to SstFileDumper because of name conflict, and seems SstFileDumper is more appropriate for tools. TODO: there is only a very simple test now, because I want to get some feedback first. If the changes look good, I will add more tests soon. Pull Request resolved: https://github.com/facebook/rocksdb/pull/4717 Differential Revision: D13212686 Pulled By: ajkr fbshipit-source-id: 737593383264c954b79e63edaf44aaae0d947e56
-
- 22 11月, 2018 1 次提交
-
-
由 Abhishek Madan 提交于
Summary: The old RangeDelAggregator did expensive pre-processing work to create a collapsed, binary-searchable representation of range tombstones. With FragmentedRangeTombstoneIterator, much of this work is now unnecessary. RangeDelAggregatorV2 takes advantage of this by seeking in each iterator to find a covering tombstone in ShouldDelete, while doing minimal work in AddTombstones. The old RangeDelAggregator is still used during flush/compaction for now, though RangeDelAggregatorV2 will support those uses in a future PR. Pull Request resolved: https://github.com/facebook/rocksdb/pull/4649 Differential Revision: D13146964 Pulled By: abhimadan fbshipit-source-id: be29a4c020fc440500c137216fcc1cf529571eb3
-
- 13 11月, 2018 1 次提交
-
-
由 Sagar Vemuri 提交于
Summary: In #4652 we are setting -Os for lite builds only when LITE=1 is specified. But currently almost all the users invoke lite build via OPT="-DROCKSDB_LITE=1". So this diff tries to set LITE=1 when users already pass in -DROCKSDB_LITE=1 via the command line. Pull Request resolved: https://github.com/facebook/rocksdb/pull/4671 Differential Revision: D13033801 Pulled By: sagar0 fbshipit-source-id: e7b506cee574f9e3f42221ee6647915011c78d78
-
- 10 11月, 2018 1 次提交
-
-
由 Yi Wu 提交于
Summary: When running `make shared_lib` under fbcode, there's liblua link error: https://gist.github.com/yiwu-arbug/b796bff6b3d46d90c1ed878d983de50d This is because we link liblua.a when building shared lib. If we want to link with liblua, we need to link with liblua_pic.a instead. Fixing by simply not link with lua. Pull Request resolved: https://github.com/facebook/rocksdb/pull/4651 Differential Revision: D12964798 Pulled By: yiwu-arbug fbshipit-source-id: 18d6cee94afe20748068822b76e29ef255cdb04d
-
- 08 11月, 2018 1 次提交
-
-
由 Yi Wu 提交于
Summary: Set `-Os` for lite release build to minimize binary size. Pull Request resolved: https://github.com/facebook/rocksdb/pull/4652 Differential Revision: D12965427 Pulled By: yiwu-arbug fbshipit-source-id: c8b647642c24b3e5df6a2cd13112e452a08e8398
-
- 31 10月, 2018 1 次提交
-
-
由 Yanqin Jin 提交于
Summary: This PR adds test of atomic flush to our continuous stress tests. Pull Request resolved: https://github.com/facebook/rocksdb/pull/4605 Differential Revision: D12840607 Pulled By: riversand963 fbshipit-source-id: 0da187572791a59530065a7952697c05b1197ad9
-
- 25 10月, 2018 1 次提交
-
-
由 Abhishek Madan 提交于
Summary: Previously, range tombstones were accumulated from every level, which was necessary if a range tombstone in a higher level covered a key in a lower level. However, RangeDelAggregator::AddTombstones's complexity is based on the number of tombstones that are currently stored in it, which is wasteful in the Get case, where we only need to know the highest sequence number of range tombstones that cover the key from higher levels, and compute the highest covering sequence number at the current level. This change introduces this optimization, and removes the use of RangeDelAggregator from the Get path. In the benchmark results, the following command was used to initialize the database: ``` ./db_bench -db=/dev/shm/5k-rts -use_existing_db=false -benchmarks=filluniquerandom -write_buffer_size=1048576 -compression_type=lz4 -target_file_size_base=1048576 -max_bytes_for_level_base=4194304 -value_size=112 -key_size=16 -block_size=4096 -level_compaction_dynamic_level_bytes=true -num=5000000 -max_background_jobs=12 -benchmark_write_rate_limit=20971520 -range_tombstone_width=100 -writes_per_range_tombstone=100 -max_num_range_tombstones=50000 -bloom_bits=8 ``` ...and the following command was used to measure read throughput: ``` ./db_bench -db=/dev/shm/5k-rts/ -use_existing_db=true -benchmarks=readrandom -disable_auto_compactions=true -num=5000000 -reads=100000 -threads=32 ``` The filluniquerandom command was only run once, and the resulting database was used to measure read performance before and after the PR. Both binaries were compiled with `DEBUG_LEVEL=0`. Readrandom results before PR: ``` readrandom : 4.544 micros/op 220090 ops/sec; 16.9 MB/s (63103 of 100000 found) ``` Readrandom results after PR: ``` readrandom : 11.147 micros/op 89707 ops/sec; 6.9 MB/s (63103 of 100000 found) ``` So it's actually slower right now, but this PR paves the way for future optimizations (see #4493). ---- Pull Request resolved: https://github.com/facebook/rocksdb/pull/4449 Differential Revision: D10370575 Pulled By: abhimadan fbshipit-source-id: 9a2e152be1ef36969055c0e9eb4beb0d96c11f4d
-
- 24 10月, 2018 1 次提交
-
-
由 Yi Wu 提交于
Summary: In fbcode when we build with clang7++, although -faligned-new is available in compile phase, we link with an older version of libstdc++.a and it doesn't come with aligned-new support (e.g. `nm libstdc++.a | grep align_val_t` return empty). In this case the previous -faligned-new detection can pass but will end up with link error. Fixing it by only have the detection for non-fbcode build. Pull Request resolved: https://github.com/facebook/rocksdb/pull/4576 Differential Revision: D10500008 Pulled By: yiwu-arbug fbshipit-source-id: b375de4fbb61d2a08e54ab709441aa8e7b4b08cf
-
- 28 9月, 2018 1 次提交
-
-
由 Yi Wu 提交于
Summary: Introduce `RepeatableThread` utility to run task periodically in a separate thread. It is basically the same as the the same class in fbcode, and in addition provide a helper method to let tests mock time and trigger execution one at a time. We can use this class to replace `TimerQueue` in #4382 and `BlobDB`. Pull Request resolved: https://github.com/facebook/rocksdb/pull/4423 Differential Revision: D10020932 Pulled By: yiwu-arbug fbshipit-source-id: 3616bef108c39a33c92eedb1256de424b7c04087
-
- 18 9月, 2018 1 次提交
-
-
由 Abhishek Madan 提交于
Summary: To measure the results of upcoming DeleteRange v2 work, this commit adds simple benchmarks for RangeDelAggregator. It measures the average time for AddTombstones and ShouldDelete calls. Using this to compare the results before #4014 and on the latest master (using the default arguments) produces the following results: Before #4014: ``` ======================= Results: ======================= AddTombstones: 1356.28 us ShouldDelete: 0.401732 us ``` Latest master: ``` ======================= Results: ======================= AddTombstones: 740.82 us ShouldDelete: 0.383271 us ``` Pull Request resolved: https://github.com/facebook/rocksdb/pull/4363 Differential Revision: D9881676 Pulled By: abhimadan fbshipit-source-id: 793e7d61aa4b9d47eb917bbcc03f08695b5e5442
-
- 12 9月, 2018 1 次提交
-
-
由 Yanqin Jin 提交于
Summary: Before the fix: On a PowerPC machine, run the following ``` $ make jtest ``` The command will fail due to "undefined symbol: crc32c_ppc". It was caused by 'rocksdbjava' Makefile target not including crc32c_ppc object files when generating the shared lib. The fix is simple. Pull Request resolved: https://github.com/facebook/rocksdb/pull/4357 Differential Revision: D9779474 Pulled By: riversand963 fbshipit-source-id: 3c5ec9068c2b9c796e6500f71cd900267064fd51
-
- 31 8月, 2018 1 次提交
-
-
由 Zhongyi Xie 提交于
Summary: Currently unity-test is failing because both trace_replay.cc and trace_analyzer_tool.cc defined `DecodeCFAndKey` under anonymous namespace. It is supposed to be fine except unity test will dump all source files together and now we have a conflict. Another issue with trace_analyzer_tool.cc is that it is using some utility functions from ldb_cmd which is not included in Makefile for unity_test, I chose to update TESTHARNESS to include LIBOBJECTS. Feel free to comment if there is a less intrusive way to solve this. Pull Request resolved: https://github.com/facebook/rocksdb/pull/4323 Differential Revision: D9599170 Pulled By: miasantreble fbshipit-source-id: 38765b11f8e7de92b43c63bdcf43ea914abdc029
-
- 28 8月, 2018 1 次提交
-
-
由 Sagar Vemuri 提交于
Summary: Since bzip.org is no longer maintained, download the bzip2 packages from a snapshot taken by the internet archive until we figure out a more credible source. Fixes issue: #4305 Pull Request resolved: https://github.com/facebook/rocksdb/pull/4306 Differential Revision: D9514868 Pulled By: sagar0 fbshipit-source-id: 57c6a141a62e652f94377efc7ca9916b458e68d5
-
- 22 8月, 2018 1 次提交
-
-
由 Zhichao Cao 提交于
Summary: Previously, the trace_analyzer_tool will be complied with other libobjects, which let the GFLAGS of trace_analyzer appear in other tools (e.g., db_bench, rocksdb_dump, and etc.). When using '--help', the help information of trace_analyzer will appear in other tool help information, which will cause confusion issues. Currently, trace_analyzer_tool is built and used only by trace_analyzer and trace_analyzer_test to avoid the issues. Tested with make asan_check. Pull Request resolved: https://github.com/facebook/rocksdb/pull/4290 Differential Revision: D9413163 Pulled By: zhichao-cao fbshipit-source-id: ed5d20c4575a53ca15ff62a2ffe601d5cf278cc4
-
- 14 8月, 2018 1 次提交
-
-
由 Zhichao Cao 提交于
Summary: A framework of trace analyzing for RocksDB After collecting the trace by using the tool of [PR #3837](https://github.com/facebook/rocksdb/pull/3837). User can use the Trace Analyzer to interpret, analyze, and characterize the collected workload. **Input:** 1. trace file 2. Whole keys space file **Statistics:** 1. Access count of each operation (Get, Put, Delete, SingleDelete, DeleteRange, Merge) in each column family. 2. Key hotness (access count) of each one 3. Key space separation based on given prefix 4. Key size distribution 5. Value size distribution if appliable 6. Top K accessed keys 7. QPS statistics including the average QPS and peak QPS 8. Top K accessed prefix 9. The query correlation analyzing, output the number of X after Y and the corresponding average time intervals **Output:** 1. key access heat map (either in the accessed key space or whole key space) 2. trace sequence file (interpret the raw trace file to line base text file for future use) 3. Time serial (The key space ID and its access time) 4. Key access count distritbution 5. Key size distribution 6. Value size distribution (in each intervals) 7. whole key space separation by the prefix 8. Accessed key space separation by the prefix 9. QPS of each operation and each column family 10. Top K QPS and their accessed prefix range **Test:** 1. Added the unit test of analyzing Get, Put, Delete, SingleDelete, DeleteRange, Merge 2. Generated the trace and analyze the trace **Implemented but not tested (due to the limitation of trace_replay):** 1. Analyzing Iterator, supporting Seek() and SeekForPrev() analyzing 2. Analyzing the number of Key found by Get **Future Work:** 1. Support execution time analyzing of each requests 2. Support cache hit situation and block read situation of Get Pull Request resolved: https://github.com/facebook/rocksdb/pull/4091 Differential Revision: D9256157 Pulled By: zhichao-cao fbshipit-source-id: f0ceacb7eedbc43a3eee6e85b76087d7832a8fe6
-