- 13 1月, 2017 1 次提交
-
-
由 Aaron Gao 提交于
Summary: add build guard Closes https://github.com/facebook/rocksdb/pull/1771 Differential Revision: D4410779 Pulled By: siying fbshipit-source-id: 3796c30
-
- 12 1月, 2017 2 次提交
-
-
由 Aaron Gao 提交于
Summary: direct IO reads refactoring remove unnecessary classes and unified interfaces tested with db_bench need more change for options and ON/OFF for different files. Since disabled is default, it should be fine now Closes https://github.com/facebook/rocksdb/pull/1636 Differential Revision: D4307189 Pulled By: lightmark fbshipit-source-id: 6991e22
-
由 Anirban Rahut 提交于
Summary: …perly on travis There is some old code in PosixWritableFile::Close(), which truncates the file to the measured size and then does an extra fallocate with KEEP_SIZE. This is commented as a failsafe because in some cases ftruncate doesn't do the right job (I don't know of an instance of this btw). However doing an fallocate with KEEP_SIZE should not increase the file size. However on Travis Worker which is Docker (likely AUFS ) its not working. There are comments on web that show that the AUFS author had initially not implemented fallocate, and then did it later. So not sure what is the quality of the implementation. Closes https://github.com/facebook/rocksdb/pull/1765 Differential Revision: D4401340 Pulled By: anirbanr-fb fbshipit-source-id: e2d8100
-
- 11 1月, 2017 1 次提交
-
-
由 Andrew Kryczka 提交于
Summary: Previously the only way to increment a handle's refcount was to invoke Lookup(), which (1) did hash table lookup to get cache handle, (2) incremented that handle's refcount. For a future DeleteRange optimization, I added a function, Ref(), for when the caller already has a cache handle and only needs to do (2). Closes https://github.com/facebook/rocksdb/pull/1761 Differential Revision: D4397114 Pulled By: ajkr fbshipit-source-id: 9addbe5
-
- 10 1月, 2017 1 次提交
-
-
由 Dmitri Smirnov 提交于
Summary: Enable directIO on WritableFileImpl::Append with offset being current length of the file. Enable UniqueID tests on Windows, disable others but leeting them to compile. Unique tests are valuable to detect failures on different filesystems and upcoming ReFS. Clear output in WinEnv Getchildren.This is different from previous strategy, do not touch output on failure. Make sure DBTest.OpenWhenOpen works with windows error message Closes https://github.com/facebook/rocksdb/pull/1746 Differential Revision: D4385681 Pulled By: IslamAbdelRahman fbshipit-source-id: c07b702
-
- 09 1月, 2017 2 次提交
-
-
由 Maysam Yabandeh 提交于
Summary: This reverts commit 54d94e9c. The pull request was landed by mistake. Closes https://github.com/facebook/rocksdb/pull/1755 Differential Revision: D4391678 Pulled By: maysamyabandeh fbshipit-source-id: 36d5149
-
由 Maysam Yabandeh 提交于
Summary: Currently the point lookup values are copied to a string provided by the user. This incures an extra memcpy cost. This patch allows doing point lookup via a PinnableSlice which pins the source memory location (instead of copying their content) and releases them after the content is consumed by the user. The old API of Get(string) is translated to the new API underneath. Here is the summary for improvements: 1. value 100 byte: 1.8% regular, 1.2% merge values 2. value 1k byte: 11.5% regular, 7.5% merge values 3. value 10k byte: 26% regular, 29.9% merge values The improvement for merge could be more if we extend this approach to pin the merge output and delay the full merge operation until the user actually needs it. We have put that for future work. PS: Sometimes we observe a small decrease in performance when switching from t5452014 to this patch but with the old Get(string) API. The difference is a little and could be noise. More importantly it is safely cancelled Closes https://github.com/facebook/rocksdb/pull/1732 Differential Revision: D4374613 Pulled By: maysamyabandeh fbshipit-source-id: a077f1a
-
- 07 1月, 2017 1 次提交
-
-
由 Islam AbdelRahman 提交于
Summary: This function will return a list of supported compression types in RocksDB This is needed for MyRocks https://github.com/facebook/mysql-5.6/pull/446 Closes https://github.com/facebook/rocksdb/pull/1747 Differential Revision: D4385921 Pulled By: IslamAbdelRahman fbshipit-source-id: 2f5b59f
-
- 04 1月, 2017 1 次提交
-
-
由 Adam Retter 提交于
Summary: This fixes the Java API for Status#getState use in Native code and also simplifies the implementation of rocksdb::Status::getState. Closes https://github.com/facebook/rocksdb/issues/1688 Closes https://github.com/facebook/rocksdb/pull/1714 Differential Revision: D4364181 Pulled By: yiwu-arbug fbshipit-source-id: 8e073b4
-
- 29 12月, 2016 1 次提交
-
-
由 Siying Dong 提交于
Summary: File copying happens when creating checkpoints and bulkloading files from different FS partition. We should fsync the files when copying them to guarantee durability. A side effect will be that the dirty pages in file system buffers won't grow too large. Closes https://github.com/facebook/rocksdb/pull/1728 Differential Revision: D4371083 Pulled By: siying fbshipit-source-id: 579e14c
-
- 23 12月, 2016 2 次提交
-
-
由 Yi Wu 提交于
Summary: Improve cache options logging to info log. Also print the value of cache_index_and_filter_blocks_with_high_priority. Closes https://github.com/facebook/rocksdb/pull/1709 Differential Revision: D4358776 Pulled By: yiwu-arbug fbshipit-source-id: 8f030a0
-
由 Aaron Gao 提交于
Summary: rocksdb direct io support ``` [gzh@dev11575.prn2 ~/rocksdb] ./db_bench -benchmarks=fillseq --num=1000000 Initializing RocksDB Options from the specified file Initializing RocksDB Options from command-line flags RocksDB: version 5.0 Date: Wed Nov 23 13:17:43 2016 CPU: 40 * Intel(R) Xeon(R) CPU E5-2660 v2 @ 2.20GHz CPUCache: 25600 KB Keys: 16 bytes each Values: 100 bytes each (50 bytes after compression) Entries: 1000000 Prefix: 0 bytes Keys per prefix: 0 RawSize: 110.6 MB (estimated) FileSize: 62.9 MB (estimated) Write rate: 0 bytes/second Compression: Snappy Memtablerep: skip_list Perf Level: 1 WARNING: Assertions are enabled; benchmarks unnecessarily slow ------------------------------------------------ Initializing RocksDB Options from the specified file Initializing RocksDB Options from command-line flags DB path: [/tmp/rocksdbtest-112628/dbbench] fillseq : 4.393 micros/op 227639 ops/sec; 25.2 MB/s [gzh@dev11575.prn2 ~/roc Closes https://github.com/facebook/rocksdb/pull/1564 Differential Revision: D4241093 Pulled By: lightmark fbshipit-source-id: 98c29e3
-
- 22 12月, 2016 1 次提交
-
-
由 Islam AbdelRahman 提交于
Summary: Remove sst_file_manager option from LITE Closes https://github.com/facebook/rocksdb/pull/1690 Differential Revision: D4341331 Pulled By: IslamAbdelRahman fbshipit-source-id: 9f9328d
-
- 17 12月, 2016 2 次提交
-
-
由 Jianpeng Ma 提交于
Summary: ?e_file_max_buffer_size) If we overwrite WritableFile and has a buffer which has the same function of buf_. We hope remove the cache function of WritableFileWriter. So using options.writable_file_max_buffer_size = 0 to disable cache function. Signed-off-by: NJianpeng Ma <jianpeng.ma@intel.com> Closes https://github.com/facebook/rocksdb/pull/1628 Differential Revision: D4307219 Pulled By: yiwu-arbug fbshipit-source-id: 77a6e26
-
由 Daniel Black 提交于
Summary: Fixes compile error: In file included from ./util/statistics.h:17:0, from ./util/stop_watch.h:8, from ./util/perf_step_timer.h:9, from ./util/iostats_context_imp.h:8, from ./util/posix_logger.h:27, from ./port/util_logger.h:18, from ./db/auto_roll_logger.h:15, from db/auto_roll_logger.cc:6: ./util/thread_local.h:65:16: error: 'function' in namespace 'std' does not name a template type typedef std::function<void(void*, void*)> FoldFunc; Closes https://github.com/facebook/rocksdb/pull/1656 Differential Revision: D4318702 Pulled By: yiwu-arbug fbshipit-source-id: 8c5d17a
-
- 15 12月, 2016 3 次提交
-
-
由 Daniel Black 提交于
Summary: Bunch of commits related to insufficient buffer size. Errors in individual commits. Closes https://github.com/facebook/rocksdb/pull/1673 Differential Revision: D4332127 Pulled By: IslamAbdelRahman fbshipit-source-id: 878f73c
-
由 Daniel Black 提交于
Summary: hopefully the last of the gcc-7 compile errors Closes https://github.com/facebook/rocksdb/pull/1675 Differential Revision: D4332106 Pulled By: IslamAbdelRahman fbshipit-source-id: 139448c
-
由 Daniel Black 提交于
Summary: fixes error (that occurred on gcc-7): error: util/env_basic_test.cc: In member function 'virtual rocksdb::Status rocksdb::NormalizingEnvWrapper::GetChildren(const string&, std::vector<std::__cxx11::basic_string<char> >*)': util/env_basic_test.cc:27:21: error: 'remove_if' is not a member of 'std' result->erase(std::remove_if(result->begin(), result->end(), ^~~ Closes https://github.com/facebook/rocksdb/pull/1674 Differential Revision: D4331221 Pulled By: ajkr fbshipit-source-id: 9bbdc78
-
- 14 12月, 2016 3 次提交
-
-
由 Daniel Black 提交于
Summary: util/logging.cc:100:13: error: output may be truncated before the last format character [-Werror=format-length=] std::string NumberToHumanString(int64_t num) { ^~~~~~~~~~~~~~~~~~~ util/logging.cc:106:59: note: format output between 3 and 19 bytes into a destination of size 16 snprintf(buf, sizeof(buf), "%" PRIi64 "K", num / 1000); Closes https://github.com/facebook/rocksdb/pull/1653 Differential Revision: D4318687 Pulled By: yiwu-arbug fbshipit-source-id: 3a5c931
-
由 Daniel Black 提交于
Summary: sorry if these gcc-7/clang-4 cleanups are getting tedious. Closes https://github.com/facebook/rocksdb/pull/1658 Differential Revision: D4318792 Pulled By: yiwu-arbug fbshipit-source-id: 8e85891
-
由 Daniel Black 提交于
Summary: Increased buffer size to 1650. util/histogram.cc: In member function 'std::__cxx11::string rocksdb::HistogramStat::ToString() const': util/histogram.cc:189:13: error: '%.2f' directive output truncated writing between 4 and 313 bytes into a region of size 0 [-Werror=format-length=] std::string HistogramStat::ToString() const { ^~~~~~~~~~~~~ util/histogram.cc:205:30: note: format output between 69 and 1614 bytes into a destination of size 200 Percentile(99.99)); ^ cc1plus: all warnings being treated as errors Makefile:1521: recipe for target 'util/histogram.o' failed Closes https://github.com/facebook/rocksdb/pull/1660 Differential Revision: D4318820 Pulled By: yiwu-arbug fbshipit-source-id: 45ae6ea
-
- 13 12月, 2016 1 次提交
-
-
由 Andrew Kryczka 提交于
Summary: It'd be nice to use the error status type to distinguish between user error and system error. For example, GetChildren can fail listing a backup directory's contents either because a bad path was provided (user error) or because an operation failed, e.g., a remote storage service call failed (system error). In the former case, we want to continue and treat the backup directory as empty; in the latter case, we want to immediately propagate the error to the caller. This diff uses NotFound to indicate user error and IOError to indicate system error. Previously IOError indicated both. Closes https://github.com/facebook/rocksdb/pull/1644 Differential Revision: D4312157 Pulled By: ajkr fbshipit-source-id: 51b4f24
-
- 06 12月, 2016 2 次提交
-
-
由 Manuel Ung 提交于
Summary: This is an implementation of non-exclusive locks for pessimistic transactions. It is relatively simple and does not prevent starvation (ie. it's possible that request for exclusive access will never be granted if there are always threads holding shared access). It is done by changing `KeyLockInfo` to hold an set a transaction ids, instead of just one, and adding a flag specifying whether this lock is currently held with exclusive access or not. Some implementation notes: - Some lock diagnostic functions had to be updated to return a set of transaction ids for a given lock, eg. `GetWaitingTxn` and `GetLockStatusData`. - Deadlock detection is a bit more complicated since a transaction can now wait on multiple other transactions. A BFS is done in this case, and deadlock detection depth is now just a limit on the number of transactions we visit. - Expirable transactions do not work efficiently with shared locks at the moment, but that's okay for now. Closes https://github.com/facebook/rocksdb/pull/1573 Differential Revision: D4239097 Pulled By: lth fbshipit-source-id: da7c074
-
由 Anton Safonov 提交于
Summary: Made delete_obsolete_files_period_micros option dynamic. It can be updating using DB::SetDBOptions(). Closes https://github.com/facebook/rocksdb/pull/1595 Differential Revision: D4246569 Pulled By: tonek fbshipit-source-id: d23f560
-
- 02 12月, 2016 2 次提交
-
-
由 Islam AbdelRahman 提交于
Summary: Reduce number of comparisons in heap by caching which child node in the first level is smallest (left_child or right_child) So next time we can compare directly against the smallest child I see that the total number of calls to comparator drops significantly when using this optimization Before caching (~2mil key comparison for iterating the DB) ``` $ DEBUG_LEVEL=0 make db_bench -j64 && ./db_bench --benchmarks="readseq" --db="/dev/shm/heap_opt" --use_existing_db --disable_auto_compactions --cache_size=1000000000 --perf_level=2 readseq : 0.338 micros/op 2959201 ops/sec; 327.4 MB/s user_key_comparison_count = 2000008 ``` After caching (~1mil key comparison for iterating the DB) ``` $ DEBUG_LEVEL=0 make db_bench -j64 && ./db_bench --benchmarks="readseq" --db="/dev/shm/heap_opt" --use_existing_db --disable_auto_compactions --cache_size=1000000000 --perf_level=2 readseq : 0.309 micros/op 3236801 ops/sec; 358.1 MB/s user_key_comparison_count = 1000011 ``` It also improves Closes https://github.com/facebook/rocksdb/pull/1600 Differential Revision: D4256027 Pulled By: IslamAbdelRahman fbshipit-source-id: 76fcc66
-
由 Islam AbdelRahman 提交于
Summary: Travis fail because it uses clang 3.6 which don't recognize `__attribute__((__no_sanitize__("undefined")))` Closes https://github.com/facebook/rocksdb/pull/1601 Differential Revision: D4257175 Pulled By: IslamAbdelRahman fbshipit-source-id: fb4d1ab
-
- 29 11月, 2016 3 次提交
-
-
由 Islam AbdelRahman 提交于
Summary: disable UBSAN for functions with intentional left shift on -ve number / overflow These functions are rocksdb:: Hash FixedLengthColBufEncoder::Append FaultInjectionTest:: Key Closes https://github.com/facebook/rocksdb/pull/1577 Differential Revision: D4240801 Pulled By: IslamAbdelRahman fbshipit-source-id: 3e1caf6
-
由 Islam AbdelRahman 提交于
Summary: Having -ve value for max_write_buffer_number does not make sense and cause us to do a left shift on a -ve value number Closes https://github.com/facebook/rocksdb/pull/1579 Differential Revision: D4240798 Pulled By: IslamAbdelRahman fbshipit-source-id: bd6267e
-
由 Mike Kolupaev 提交于
Summary: In one deployment we saw high latencies (presumably from slow iterator operations) and a lot of CPU time reported by perf with this stack: ``` rocksdb::MergingIterator::Next rocksdb::DBIter::FindNextUserEntryInternal rocksdb::DBIter::Seek ``` I think what's happening is: 1. we create a snapshot iterator, 2. we do lots of Put()s for the same key x; this creates lots of entries in memtable, 3. we seek the iterator to a key slightly smaller than x, 4. the seek walks over lots of entries in memtable for key x, skipping them because of high sequence numbers. CC IslamAbdelRahman Closes https://github.com/facebook/rocksdb/pull/1413 Differential Revision: D4083879 Pulled By: IslamAbdelRahman fbshipit-source-id: a83ddae
-
- 24 11月, 2016 1 次提交
-
-
由 Siying Dong 提交于
Summary: Current write stalling system has the problem of lacking of positive feedback if the restricted rate is already too low. Users sometimes stack in very low slowdown value. With the diff, we add a positive feedback (increasing the slowdown value) if we recover from slowdown state back to normal. To avoid the positive feedback to keep the slowdown value to be to high, we add issue a negative feedback every time we are close to the stop condition. Experiments show it is easier to reach a relative balance than before. Also increase level0_stop_writes_trigger default from 24 to 32. Since level0_slowdown_writes_trigger default is 20, stop trigger 24 only gives four files as the buffer time to slowdown writes. In order to avoid stop in four files while 20 files have been accumulated, the slowdown value must be very low, which is amost the same as stop. It also doesn't give enough time for the slowdown value to converge. Increase it to 32 will smooth out the system. Closes https://github.com/facebook/rocksdb/pull/1562 Differential Revision: D4218519 Pulled By: siying fbshipit-source-id: 95e4088
-
- 22 11月, 2016 3 次提交
-
-
由 Nick Terrell 提交于
Summary: LZ4 1.7.3 emits warnings when calling the deprecated function `LZ4_compress_limitedOutput_continue()`. Starting in r129, LZ4 introduces `LZ4_compress_fast_continue()` as a replacement, and the two functions calls are [exactly equivalent](https://github.com/lz4/lz4/blob/dev/lib/lz4.c#L1408). Closes https://github.com/facebook/rocksdb/pull/1532 Differential Revision: D4199240 Pulled By: siying fbshipit-source-id: 138c2bc
-
由 Changli Gao 提交于
Summary: We should close the fd, before overriding it. This bug was introduced by f89caa12 Closes https://github.com/facebook/rocksdb/pull/1553 Differential Revision: D4214101 Pulled By: siying fbshipit-source-id: 0d65de0
-
由 Andrew Kryczka 提交于
Summary: - Made RangeDelAggregator's InternalKeyComparator member a reference-to-const so we don't need to copy-construct it. Also added InternalKeyComparator to ImmutableCFOptions so we don't need to construct one for each DBIter. - Made MemTable::NewRangeTombstoneIterator and the table readers' NewRangeTombstoneIterator() functions return nullptr instead of NewEmptyInternalIterator to avoid the allocation. Updated callers accordingly. Closes https://github.com/facebook/rocksdb/pull/1548 Differential Revision: D4208169 Pulled By: ajkr fbshipit-source-id: 2fd65cf
-
- 21 11月, 2016 1 次提交
-
-
由 Changli Gao 提交于
Summary: When calling StatisticsImpl::HistogramInfo::getMergedHistogram(), if there is a dying thread, which is calling ThreadLocalPtr::StaticMeta::OnThreadExit() to merge its thread values to HistogramInfo, deadlock will occur. Because the former try to hold merge_lock then ThreadMeta::mutex_, but the later try to hold ThreadMeta::mutex_ then merge_lock. In short, the locking order isn't the same. This patch addressed this issue by releasing merge_lock before folding thread values. Closes https://github.com/facebook/rocksdb/pull/1552 Differential Revision: D4211942 Pulled By: ajkr fbshipit-source-id: ef89bcb
-
- 20 11月, 2016 1 次提交
-
-
由 Manuel Ung 提交于
Summary: Currently, deadlock cycles are held in std::unordered_map. The problem with it is that it allocates/deallocates memory on every insertion/deletion. This limits throughput since we're doing this expensive operation while holding a global mutex. Fix this by using a vector which caches memory instead. Running the deadlock stress test, this change increased throughput from 39k txns/s -> 49k txns/s. The effect is more noticeable in MyRocks. Closes https://github.com/facebook/rocksdb/pull/1545 Differential Revision: D4205662 Pulled By: lth fbshipit-source-id: ff990e4
-
- 19 11月, 2016 2 次提交
-
-
由 Siying Dong 提交于
Summary: Currently, in the Direct I/O read mode, the last sector of the file, if not full, is not handled correctly. If the return value of pread is not multiplier of kSectorSize, we still go ahead and continue reading, even if the buffer is not aligned. With the commit, if the return value is not multiplier of kSectorSize, and all but the last sector has been read, we simply return. Closes https://github.com/facebook/rocksdb/pull/1550 Differential Revision: D4209609 Pulled By: lightmark fbshipit-source-id: cb0b439
-
由 Maysam Yabandeh 提交于
Summary: This patch clarifies the contract of PositionedAppend with some unit tests and also implements it for PosixWritableFile. (Tasks: 14524071) Closes https://github.com/facebook/rocksdb/pull/1514 Differential Revision: D4204907 Pulled By: maysamyabandeh fbshipit-source-id: 06eabd2
-
- 17 11月, 2016 1 次提交
-
-
由 Siying Dong 提交于
Summary: Closes https://github.com/facebook/rocksdb/pull/1496 Differential Revision: D4168080 Pulled By: siying fbshipit-source-id: 056ae62
-
- 16 11月, 2016 1 次提交
-
-
由 Yi Wu 提交于
Summary: It is hard to measure acutal memory usage by std containers. Even providing a custom allocator will miss count some of the usage. Here we only do a wild guess on its memory usage. Closes https://github.com/facebook/rocksdb/pull/1511 Differential Revision: D4179945 Pulled By: yiwu-arbug fbshipit-source-id: 32ab929
-
- 15 11月, 2016 1 次提交
-
-
由 Artemiy Kolesnikov 提交于
Summary: Closes https://github.com/facebook/rocksdb/pull/1509 Differential Revision: D4176426 Pulled By: yiwu-arbug fbshipit-source-id: b57689d
-