- 06 7月, 2016 2 次提交
-
-
由 Aaron Gao 提交于
Summary: Currently, if several flush outputs are committed together, we issue each manifest write per batch (1 batch = 1 flush = 1 sst file = 1+ continuous memtables). Each manifest write requires one fsync and one fsync to parent directory. In some cases, it becomes the bottleneck of write. We should batch them and write in one manifest write when possible. Test Plan: ` ./db_bench -benchmarks="fillseq" -max_write_buffer_number=16 -max_background_flushes=16 -disable_auto_compactions=true -min_write_buffer_number_to_merge=1 -write_buffer_size=65536 -level0_stop_writes_trigger=10000 -level0_slowdown_writes_trigger=10000` **Before** ``` Initializing RocksDB Options from the specified file Initializing RocksDB Options from command-line flags RocksDB: version 4.9 Date: Fri Jul 1 15:38:17 2016 CPU: 32 * Intel(R) Xeon(R) CPU E5-2660 0 @ 2.20GHz CPUCache: 20480 KB Keys: 16 bytes each Values: 100 bytes each (50 bytes after compression) Entries: 1000000 Prefix: 0 bytes Keys per prefix: 0 RawSize: 110.6 MB (estimated) FileSize: 62.9 MB (estimated) Write rate: 0 bytes/second Compression: Snappy Memtablerep: skip_list Perf Level: 1 WARNING: Assertions are enabled; benchmarks unnecessarily slow ------------------------------------------------ Initializing RocksDB Options from the specified file Initializing RocksDB Options from command-line flags DB path: [/tmp/rocksdbtest-112628/dbbench] fillseq : 166.277 micros/op 6014 ops/sec; 0.7 MB/s ``` **After** ``` Initializing RocksDB Options from the specified file Initializing RocksDB Options from command-line flags RocksDB: version 4.9 Date: Fri Jul 1 15:35:05 2016 CPU: 32 * Intel(R) Xeon(R) CPU E5-2660 0 @ 2.20GHz CPUCache: 20480 KB Keys: 16 bytes each Values: 100 bytes each (50 bytes after compression) Entries: 1000000 Prefix: 0 bytes Keys per prefix: 0 RawSize: 110.6 MB (estimated) FileSize: 62.9 MB (estimated) Write rate: 0 bytes/second Compression: Snappy Memtablerep: skip_list Perf Level: 1 WARNING: Assertions are enabled; benchmarks unnecessarily slow ------------------------------------------------ Initializing RocksDB Options from the specified file Initializing RocksDB Options from command-line flags DB path: [/tmp/rocksdbtest-112628/dbbench] fillseq : 52.328 micros/op 19110 ops/sec; 2.1 MB/s ``` Reviewers: andrewkr, IslamAbdelRahman, yhchiang, sdong Reviewed By: sdong Subscribers: igor, andrewkr, dhruba, leveldb Differential Revision: https://reviews.facebook.net/D60075
-
由 omegaga 提交于
Summary: Reported in T11889874. When registering the cleanup function we should copy the option so that we can still access it if ReadOptions is deleted. Test Plan: Add a unit test to reproduce this bug. Reviewers: sdong Reviewed By: sdong Subscribers: andrewkr, dhruba Differential Revision: https://reviews.facebook.net/D60087
-
- 05 7月, 2016 2 次提交
-
-
由 Lucas Qiu 提交于
-
由 Andrew Kryczka 提交于
-
- 02 7月, 2016 3 次提交
-
-
由 Islam AbdelRahman 提交于
Summary: We saw instances where total_log_size is off the real value, but I'm not able to reproduce it. Add more logging to help debugging when it happens again. Test Plan: Run the unit test and see the logging. Reviewers: andrewkr, yhchiang, igor, IslamAbdelRahman Reviewed By: IslamAbdelRahman Subscribers: leveldb, andrewkr, dhruba Differential Revision: https://reviews.facebook.net/D60081
-
由 Wanning Jiang 提交于
Summary: LockFile is unnecessary in unit test Test Plan: env_basic_test.cc Reviewers: andrewkr Reviewed By: andrewkr Subscribers: andrewkr, dhruba, leveldb Differential Revision: https://reviews.facebook.net/D60285
-
由 Wanning Jiang 提交于
Summary: Read only backup engine return NotFound() on missing meta dir (for e2e test) Test Plan: backupable_db_test Reviewers: andrewkr Reviewed By: andrewkr Subscribers: andrewkr, dhruba, leveldb Differential Revision: https://reviews.facebook.net/D60273
-
- 01 7月, 2016 4 次提交
-
-
由 Aaron Gao 提交于
Summary: The current implementation of rate limiter has the possibility to introduce resource starvation when change its limit. This diff aims to fix this problem by consuming request bytes partially. Test Plan: ``` ./rate_limiter_test [==========] Running 4 tests from 1 test case. [----------] Global test environment set-up. [----------] 4 tests from RateLimiterTest [ RUN ] RateLimiterTest.OverflowRate [ OK ] RateLimiterTest.OverflowRate (0 ms) [ RUN ] RateLimiterTest.StartStop [ OK ] RateLimiterTest.StartStop (0 ms) [ RUN ] RateLimiterTest.Rate request size [1 - 1023], limit 10 KB/sec, actual rate: 10.355712 KB/sec, elapsed 2.00 seconds request size [1 - 1023], limit 20 KB/sec, actual rate: 19.136564 KB/sec, elapsed 2.00 seconds request size [1 - 2047], limit 20 KB/sec, actual rate: 20.783976 KB/sec, elapsed 2.10 seconds request size [1 - 2047], limit 40 KB/sec, actual rate: 39.308144 KB/sec, elapsed 2.10 seconds request size [1 - 4095], limit 40 KB/sec, actual rate: 40.318349 KB/sec, elapsed 2.20 seconds request size [1 - 4095], limit 80 KB/sec, actual rate: 79.667396 KB/sec, elapsed 2.20 seconds request size [1 - 8191], limit 80 KB/sec, actual rate: 81.807158 KB/sec, elapsed 2.30 seconds request size [1 - 8191], limit 160 KB/sec, actual rate: 160.659761 KB/sec, elapsed 2.20 seconds request size [1 - 16383], limit 160 KB/sec, actual rate: 160.700990 KB/sec, elapsed 3.00 seconds request size [1 - 16383], limit 320 KB/sec, actual rate: 317.639481 KB/sec, elapsed 2.50 seconds [ OK ] RateLimiterTest.Rate (22618 ms) [ RUN ] RateLimiterTest.LimitChangeTest [COMPLETE] request size 10 KB, new limit 20KB/sec, refill period 1000 ms [COMPLETE] request size 10 KB, new limit 5KB/sec, refill period 1000 ms [COMPLETE] request size 20 KB, new limit 40KB/sec, refill period 1000 ms [COMPLETE] request size 20 KB, new limit 10KB/sec, refill period 1000 ms [COMPLETE] request size 40 KB, new limit 80KB/sec, refill period 1000 ms [COMPLETE] request size 40 KB, new limit 20KB/sec, refill period 1000 ms [COMPLETE] request size 80 KB, new limit 160KB/sec, refill period 1000 ms [COMPLETE] request size 80 KB, new limit 40KB/sec, refill period 1000 ms [COMPLETE] request size 160 KB, new limit 320KB/sec, refill period 1000 ms [COMPLETE] request size 160 KB, new limit 80KB/sec, refill period 1000 ms [ OK ] RateLimiterTest.LimitChangeTest (5002 ms) [----------] 4 tests from RateLimiterTest (27620 ms total) [----------] Global test environment tear-down [==========] 4 tests from 1 test case ran. (27621 ms total) [ PASSED ] 4 tests. ``` Reviewers: sdong, IslamAbdelRahman, yiwu, andrewkr Reviewed By: andrewkr Subscribers: andrewkr, dhruba, leveldb Differential Revision: https://reviews.facebook.net/D60207
-
由 Andrew Kryczka 提交于
Summary: Previously we couldn't run env_basic_test on Env::Default (PosixEnv on our platforms) since GetChildren*() behavior was inconsistent with our other Envs. We can normalize the output of GetChildren*() such that these test cases work on PosixEnv too. Test Plan: ran env_basic_test Reviewers: wanning Reviewed By: wanning Subscribers: andrewkr, dhruba, leveldb Differential Revision: https://reviews.facebook.net/D59943
-
由 Andrew Kryczka 提交于
Summary: move cleanup to TearDown and handle directories, so cleanup will happen even if a test fails in the middle. Test Plan: ./env_basic_test Reviewers: wanning Reviewed By: wanning Subscribers: andrewkr, dhruba, leveldb Differential Revision: https://reviews.facebook.net/D60243
-
由 Islam AbdelRahman 提交于
Summary: Travis clang issue on linux is fixed now, re-enable linux on travis Test Plan: https://travis-ci.org/facebook/rocksdb/builds/141098948 Reviewers: yiwu, sdong, yhchiang, andrewkr Reviewed By: andrewkr Subscribers: andrewkr, dhruba Differential Revision: https://reviews.facebook.net/D60189
-
- 30 6月, 2016 4 次提交
-
-
由 John Alexander 提交于
* Fixed Windows build error in CMakeLists.txt and perf_level error in db_bench_tool.cc * Changed hard-coded perf levels in db_bench_tool.cc to enum values from perf_level.h * Replaced remaining FLAGS_perf_level > 0
-
由 Wanning Jiang 提交于
Summary: during backup, writable file should call close() before reset() Test Plan: backupable_db_test.cc Reviewers: andrewkr Reviewed By: andrewkr Subscribers: andrewkr, dhruba, leveldb Differential Revision: https://reviews.facebook.net/D60195
-
由 dhruba borthakur 提交于
-
由 Gunnar Kudrjavets 提交于
Summary: UBSan is unhappy because `cfd` is not initialized. This breaks UBSan build which in turn breaks MyRocks continuous integration with RocksDB which in turns makes me unhappy :-) Fix this. Test Plan: - `[p]arc diff --preview` + Sandcastle. - Verify that `COMPILE_WITH_UBSAN=1 OPT=-g make J=1 ubsan_check` gets past the break. Reviewers: andrewkr, hermanlee4, sdong Reviewed By: sdong Subscribers: andrewkr, dhruba Differential Revision: https://reviews.facebook.net/D60117
-
- 29 6月, 2016 2 次提交
-
-
由 Willem Jan Withagen 提交于
Submitted-by: NWillem Jan Withagen <wjw@digiware.nl>
-
由 Yueh-Hsuan Chiang 提交于
Summary: This diff makes the following improvement in regression_test.sh: 1. Add NUM_OPS and DELETE_TEST_PATH to regression_test.sh: * NUM_OPS: The number of operations that will be issued in EACH thread. Default: $NUM_KEYS / $NUM_THREADS * DELETE_TEST_PATH: If true, then the test directory will be deleted after the script ends. Default: 0 2. Add more information in SUMMARY.csv 3. Fix a bug in regression_test.sh where each thread in fillseq will all issue $NUM_KEYS writes. 4. Add --deletes in db_bench, which allows us to control the number of deletes instead of must using FLAGS_num. Test Plan: run regression test with and without DELETE_TEST_PATH and NUM_OPS Reviewers: yiwu, sdong, IslamAbdelRahman, gunnarku Reviewed By: gunnarku Subscribers: andrewkr, dhruba, leveldb Differential Revision: https://reviews.facebook.net/D60039
-
- 28 6月, 2016 2 次提交
-
-
由 sdong 提交于
Update DBTestUniversalCompaction.UniversalCompactionSingleSortedRun to use max_size_amplification_percent = 0 Summary: With max_size_amplification_percent = 0 to make sure that DBTestUniversalCompaction.UniversalCompactionSingleSortedRun tests the configuration to compact to one single sorted run. Test Plan: Run all existing tests Reviewers: yhchiang, andrewkr, IslamAbdelRahman Reviewed By: IslamAbdelRahman Subscribers: leveldb, andrewkr, dhruba Differential Revision: https://reviews.facebook.net/D60021
-
由 Reid Horuff 提交于
-
- 25 6月, 2016 5 次提交
-
-
由 charsyam 提交于
-
由 Andrew Kryczka 提交于
Summary: Overload RepairDB to take vector-of-ColumnFamilyDescriptor, which tells us CF name + options. Also takes a ColumnFamilyOptions for unspecified column families encountered during the repair. One potentially confusing thing is that we store options in the constructor and don't invoke AddColumnFamily() until discovering the CF in ScanTable. This is because we don't know the CF ID until we find a table belonging to that CF. Depends on D59781. Test Plan: $ ./repair_test Reviewers: yhchiang, IslamAbdelRahman, sdong Reviewed By: sdong Subscribers: andrewkr, dhruba, leveldb Differential Revision: https://reviews.facebook.net/D59853
-
由 Andrew Kryczka 提交于
Summary: This diff uses the CF ID and CF name properties in the SST file to associate recovered data with the proper column family. Depends on D59775. - In ScanTable(), create column families in VersionSet each time a new one is discovered (via reading SST file properties) - In ConvertLogToTable(), dump an SST file for every column family with data in the WAL - In AddTables(), make a VersionEdit per-column family that adds all of that CF's tables Test Plan: $ ./repair_test Reviewers: yhchiang, IslamAbdelRahman, sdong Reviewed By: sdong Subscribers: andrewkr, dhruba, leveldb Differential Revision: https://reviews.facebook.net/D59781
-
由 Wanning Jiang 提交于
Summary: ensure no 2nd level children under test_dir_ Test Plan: env_basic_test on 4 envs Reviewers: andrewkr Reviewed By: andrewkr Subscribers: andrewkr, dhruba Differential Revision: https://reviews.facebook.net/D59979
-
由 Andrew Kryczka 提交于
Summary: To support column families, it is easiest to use VersionSet to manage our column families (if we don't have Versions then ColumnFamilyData always behaves as a dummy column family). This diff only refactors the existing repair logic to use VersionSet; the next two parts will add support for multiple column families. Test Plan: $ ./repair_test Reviewers: yhchiang, IslamAbdelRahman, sdong Reviewed By: sdong Subscribers: andrewkr, dhruba, leveldb Differential Revision: https://reviews.facebook.net/D59775
-
- 24 6月, 2016 3 次提交
-
-
由 Gunnar Kudrjavets 提交于
-
由 Gunnar Kudrjavets 提交于
-
由 Gunnar Kudrjavets 提交于
-
- 23 6月, 2016 3 次提交
-
-
由 Islam AbdelRahman 提交于
Summary: Fix max_bytes_for_level_base comment Test Plan: none Reviewers: yhchiang, sdong Reviewed By: sdong Subscribers: andrewkr, dhruba Differential Revision: https://reviews.facebook.net/D59955
-
由 Wanning Jiang 提交于
Summary: test NativeHdfsEnv Test Plan: env_basic_test.cc Reviewers: andrewkr Reviewed By: andrewkr Subscribers: andrewkr, dhruba, leveldb Differential Revision: https://reviews.facebook.net/D59889
-
由 Gunnar Kudrjavets 提交于
Summary: Fix two minor typos and update the file name which is used to trigger the runs in case new changes have been committed. Test Plan: - Testing with a private Sandcastle instance. Reviewers: sdong, mung Reviewed By: mung Subscribers: andrewkr, dhruba Differential Revision: https://reviews.facebook.net/D59919
-
- 22 6月, 2016 5 次提交
-
-
由 omegaga 提交于
Summary: Add a read option `background_purge_on_iterator_cleanup` to avoid deleting files in foreground when destroying iterators. Instead, a job is scheduled in high priority queue and would be executed in a separate background thread. Test Plan: Add a variant of PurgeObsoleteFileTest. Turn on background purge option in the new test, and use sleeping task to ensure files are deleted in background. Reviewers: IslamAbdelRahman, sdong Reviewed By: IslamAbdelRahman Subscribers: andrewkr, dhruba Differential Revision: https://reviews.facebook.net/D59499
-
由 Islam AbdelRahman 提交于
Summary: DB::AddFile() right now always add the ingested file to L0 update the logic to add the file to the lowest possible level Test Plan: unit tests Reviewers: jkedgar, sdong Reviewed By: sdong Subscribers: andrewkr, dhruba, yoshinorim Differential Revision: https://reviews.facebook.net/D59637
-
由 Islam AbdelRahman 提交于
Summary: filter_deletes option was removed, remove it from crash_test to fix it Test Plan: make crash_test Reviewers: yhchiang, andrewkr, sdong Reviewed By: sdong Subscribers: andrewkr, dhruba Differential Revision: https://reviews.facebook.net/D59901
-
由 Gunnar Kudrjavets 提交于
Summary: The main PHP code churn is caused by extracting the common code from `FacebookArcanistConfiguration.php` and `FacebookOldArcanistConfiguration.php` into `RocksDBCommonDeterminator.php`. This is necessary both for reducing the duplication of code and making sure that we can execute the common core logic separately from continuous runs. The main logic in `RocksDBCommonDeterminator.php` remains quite the same with the exception of some things: - Adding separation between the cases when a diff is submitted //vs.// when the code is triggered from a continuous run. There are certain actions which we should do in a case of diff only. - Adding reporting - now the person who authored the diff will receive e-mail notifications if any of the jobs have failed. - Enabling assertions and making sure that we'll terminate on failure. This is an internal code used by competent engineers, so instead of `if (!condition) { echo "Something"; exit(1); }` for every invariant I think that `assert(condition)` provides better readability with the same behavior. Especially taking into account that we're talking about things which shouldn't ever happen. Enabling this entire process will be triggered internally and will be a subject of a separate code review. We should discuss the details of triggering continuous RocksDB build and tests on that diff. Test Plan: - Make sure that `[p]arc diff` scenario isn't broken by verifying that tests validating this diff will pass. - Private testing of triggering the continuous build script. - Once the changes will land then author an internal job which will use the script and verify its validity. Reviewers: sdong, yhchiang, kradhakrishnan Reviewed By: kradhakrishnan Subscribers: andrewkr, dhruba Differential Revision: https://reviews.facebook.net/D59811
-
由 Islam AbdelRahman 提交于
Summary: Update Makefile to show warnings when we have invalid paths in our make_config.mk file sample output ``` $ make static_lib -j64 Makefile:150: Warning: /mnt/gvfs/third-party2/libgcc/53e0eac8911888a105aa98b9a35fe61cf1d8b278/4.9.x/gcc-4.9-glibc-2.20/024dbc3/libs dont exist Makefile:150: Warning: /mnt/gvfs/third-party2/llvm-fb/b91de48a4974ec839946d824402b098d43454cef/stable/centos6-native/7aaccbe/../../src/clang/tools/scan-build/scan-build dont exist GEN util/build_version.cc ``` Test Plan: check that warning is printed visually Reviewers: sdong, yiwu, andrewkr Reviewed By: andrewkr Subscribers: andrewkr, dhruba Differential Revision: https://reviews.facebook.net/D59523
-
- 21 6月, 2016 2 次提交
-
-
由 Islam AbdelRahman 提交于
Summary: SstFileWriter ignore Options::bottommost_compression, update it to use bottommost_compression if available Test Plan: make check -j64 verified used compression using ./sst_dump Reviewers: sdong Reviewed By: sdong Subscribers: andrewkr, dhruba, yoshinorim Differential Revision: https://reviews.facebook.net/D59841
-
由 goroutine 提交于
-
- 18 6月, 2016 2 次提交
-
-
由 sdong 提交于
Summary: filter_deltes is not a frequently used feature. Remove it. Test Plan: Run all test suites. Reviewers: igor, yhchiang, IslamAbdelRahman Reviewed By: IslamAbdelRahman Subscribers: leveldb, andrewkr, dhruba Differential Revision: https://reviews.facebook.net/D59427
-
由 sdong 提交于
Summary: We introduced default slow down and stop condition, but didn't reset it in bulk load mode. Fix it. Test Plan: N/A Reviewers: igor, IslamAbdelRahman Reviewed By: IslamAbdelRahman Subscribers: leveldb, andrewkr, dhruba Differential Revision: https://reviews.facebook.net/D59757
-
- 17 6月, 2016 1 次提交
-
-
由 Yueh-Hsuan Chiang 提交于
Summary: Fixed a crash bug that incorrectly parse deprecated options in options_helper Test Plan: run db_bench with an old options file with memtable_prefix_bloom_probes ./db_bench --options_file=AN_OLD_OPTIONS_FILE --num=100 --benchmarks=fillseq Reviewers: sdong, IslamAbdelRahman Subscribers: andrewkr, dhruba, leveldb Differential Revision: https://reviews.facebook.net/D59787
-