- 20 1月, 2018 1 次提交
-
-
由 Siying Dong 提交于
Summary: Closes https://github.com/facebook/rocksdb/pull/3383 Differential Revision: D6762375 Pulled By: siying fbshipit-source-id: dc1e0dc9718ffb59ffe42e2a2c844b67f935a5fb
-
- 19 1月, 2018 3 次提交
-
-
由 Yi Wu 提交于
Summary: Flush() call could be waiting indefinitely if min_write_buffer_number_to_merge is used. Consider the sequence: 1. User call Flush() with flush_options.wait = true 2. The manual flush started in the background 3. New memtable become immutable because of writes. The new memtable will not trigger flush if min_write_buffer_number_to_merge is not reached. 4. The manual flush finish. Because of the new memtable created at step 3 not being flush, previous logic of WaitForFlushMemTable() keep waiting, despite the memtables it intent to flush has been flushed. Here instead of checking if there are any more memtables to flush, WaitForFlushMemTable() also check the id of the earliest memtable. If the id is larger than that of latest memtable at the time flush was initiated, it means all the memtable at the time of flush start has all been flush. Closes https://github.com/facebook/rocksdb/pull/3378 Differential Revision: D6746789 Pulled By: yiwu-arbug fbshipit-source-id: 35e698f71c7f90b06337a93e6825f4ea3b619bfa
-
由 topilski 提交于
Summary: Fixes for msys2 and mingw, hide exceptions into cpp file. Closes https://github.com/facebook/rocksdb/pull/3377 Differential Revision: D6746707 Pulled By: yiwu-arbug fbshipit-source-id: 456b38df80bc48b8386a2cf87f669b5a4f9999a4
-
由 jonasf 提交于
Summary: We have seen cases where it could be good to change TTL on already open DB. Change ttl in TtlCompactionFilterFactory on open db. Next time a filter is created, it will filter accroding to the set TTL. Is this something that could be useful for others? Any downsides? Closes https://github.com/facebook/rocksdb/pull/3292 Differential Revision: D6731993 Pulled By: miasantreble fbshipit-source-id: 73b94d69237b11e8730734389052429d621a6b1e
-
- 18 1月, 2018 1 次提交
-
-
由 Andrew Kryczka 提交于
Summary: When calling `DisableFileDeletions` followed by `GetSortedWalFiles`, we guarantee the files returned by the latter call won't be deleted until after file deletions are re-enabled. However, `GetSortedWalFiles` didn't omit files already planned for deletion via `PurgeObsoleteFiles`, so the guarantee could be broken. We fix it by making `GetSortedWalFiles` wait for the number of pending purges to hit zero if file deletions are disabled. This condition is eventually met since `PurgeObsoleteFiles` is guaranteed to be called for the existing pending purges, and new purges cannot be scheduled while file deletions are disabled. Once the condition is met, `GetSortedWalFiles` simply returns the content of DB and archive directories, which nobody can delete (except for deletion scheduler, for which I plan to fix this bug later) until deletions are re-enabled. Closes https://github.com/facebook/rocksdb/pull/3341 Differential Revision: D6681131 Pulled By: ajkr fbshipit-source-id: 90b1e2f2362ea9ef715623841c0826611a817634
-
- 17 1月, 2018 5 次提交
-
-
由 Andrew Kryczka 提交于
Summary: After af92d4ad, only exclusive manual compaction can have conflict. dc360df8 updated the conflict-checking test case accordingly. But we missed the point that exclusive manual compaction can only conflict with automatic compactions scheduled after it, since it waits on pending automatic compactions before it begins running. This PR updates the test case to ensure the automatic compactions are scheduled after the manual compaction starts but before it finishes, thus ensuring a conflict. I also cleaned up the test case to use less space as I saw it cause out-of-space error on travis. Closes https://github.com/facebook/rocksdb/pull/3375 Differential Revision: D6735162 Pulled By: ajkr fbshipit-source-id: 020530a4e150a4786792dce7cec5d66b420cb884
-
由 Yi Wu 提交于
Summary: * Fix DBTest.CompactRangeWithEmptyBottomLevel lite build failure * Fix DBTest.AutomaticConflictsWithManualCompaction failure introduce by #3366 * Fix BlockBasedTableTest::IndexUncompressed should be disabled if snappy is disabled * Fix ASAN failure with DBBasicTest::DBClose test Closes https://github.com/facebook/rocksdb/pull/3373 Differential Revision: D6732313 Pulled By: yiwu-arbug fbshipit-source-id: 1eb9b9d9a8d795f56188fa9770db9353f6fdedc5
-
由 Bartek Wrona 提交于
Summary: Issue #3370 Simple fixes to make RocksDB project working also as a submodule of other bigger one. Closes https://github.com/facebook/rocksdb/pull/3372 Differential Revision: D6729595 Pulled By: ajkr fbshipit-source-id: eee2589e7a7c4322873dff8510eebd050301c54c
-
由 Sunguck Lee 提交于
Summary: If there's manual compaction in the queue, then "HaveManualCompaction(compaction_queue_.front())" will return true, and this cause too frequent MaybeScheduleFlushOrCompaction(). https://github.com/facebook/rocksdb/issues/3198 Closes https://github.com/facebook/rocksdb/pull/3366 Differential Revision: D6729575 Pulled By: ajkr fbshipit-source-id: 96da04f8fd33297b1ccaec3badd9090403da29b0
-
由 Anand Ananthabhotla 提交于
Summary: Currently, the only way to close an open DB is to destroy the DB object. There is no way for the caller to know the status. In one instance, the destructor encountered an error due to failure to close a log file on HDFS. In order to prevent silent failures, we add DB::Close() that calls CloseImpl() which must be implemented by its descendants. The main failure point in the destructor is closing the log file. This patch also adds a Close() entry point to Logger in order to get status. When DBOptions::info_log is allocated and owned by the DBImpl, it is explicitly closed by DBImpl::CloseImpl(). Closes https://github.com/facebook/rocksdb/pull/3348 Differential Revision: D6698158 Pulled By: anand1976 fbshipit-source-id: 9468e2892553eb09c4c41b8723f590c0dbd8ab7d
-
- 13 1月, 2018 4 次提交
-
-
由 Sagar Vemuri 提交于
Summary: Java static builds are again broken, this time due Snappy version upgrade introduced in 90c1d819 (#3331). This is due to two reasons: 1. The new Snappy packages should now be downloaded from https://github.com/google/snappy/archive/<pkg.tar.gz> instead of https://github.com/google/snappy/releases/download/<pkg.tar.gz> which we are using now. 1. In addition to the the above URL change, Snappy changed its build from using autotools to CMake based : https://github.com/google/snappy/blame/e69d9f880677f2aa3488c80b953ec4309f0dfa2e/README.md#L65-L72 So more changes are needed if we are going to upgrade to 1.1.7. Hence reverting the version upgrade until we figure them out. Closes https://github.com/facebook/rocksdb/pull/3363 Differential Revision: D6716983 Pulled By: sagar0 fbshipit-source-id: f451a1bc5eb0bb090f4da07bc3e5ba72cf89aefa
-
由 Andrew Kryczka 提交于
Summary: Split `JobContext::HaveSomethingToDelete` into two functions: itself and `JobContext::HaveSomethingToClean`. Now we won't call `DBImpl::PurgeObsoleteFiles` in cases where we really just need to call `JobContext::Clean`. The change is needed because I want to track pending calls to `PurgeObsoleteFiles` for a bug fix, which is much simpler if we only call it after `FindObsoleteFiles` finds files to delete. Closes https://github.com/facebook/rocksdb/pull/3350 Differential Revision: D6690609 Pulled By: ajkr fbshipit-source-id: 61502e7469288afe16a663a1b7df345baeaf246f
-
由 Andrew Kryczka 提交于
Summary: This test often causes out-of-space error when run on travis. We don't want such stress tests in our unit test suite. The bug in #596, which this test intends to expose, can be repro'd as long as the bottommost level(s) are empty when CompactRange is called. I rewrote the test to cover this simple case without writing a lot of data. Closes https://github.com/facebook/rocksdb/pull/3362 Differential Revision: D6710417 Pulled By: ajkr fbshipit-source-id: 9a1ec85e738c813ac2fee29f1d5302065ecb54c5
-
由 Sagar Vemuri 提交于
Summary: Java build on PPC64le has been broken since a few months, due to #2716. Fixing it with the least amount of changes. (We should cleanup a little around this code when time permits). This should fix the build failures seen in http://140.211.168.68:8080/job/Rocksdb/ . Closes https://github.com/facebook/rocksdb/pull/3359 Differential Revision: D6712938 Pulled By: sagar0 fbshipit-source-id: 3046e8f072180693de2af4762934ec1ace309ca4
-
- 12 1月, 2018 7 次提交
-
-
由 Andrew Kryczka 提交于
Summary: I installed the ruby dependencies and ran `bundle update nokogiri`. It depends on a newer version of "mini_portile2" which I missed in 9c2f64e1. Now `bundle install` works again. Closes https://github.com/facebook/rocksdb/pull/3361 Differential Revision: D6710164 Pulled By: ajkr fbshipit-source-id: 9a08d6cc6400ef495b715b3d68b04ce3f3367031
-
由 Peter (Stig) Edwards 提交于
Summary: Hello and thank you for RocksDB, While looking into the buffered io used when an `OPTIONS` file is read I noticed the `OPTIONS` files produced by RocksDB 5.8.8 (and head of master) were just over 4096 bytes in size, resulting in the version of glibc I am using (glibc-2.17-196.el7) (on the filesystem used) being passed a 4K buffer for the `fread_unlocked` call and 2 system call reads using a 4096 buffer being used to read the contents of the `OPTIONS` file. If the buffer size is increased to 8192 then 1 system call read is used to read the contents. As I think the buffer size is just used for reading `OPTIONS` files, and I thought it likely that `OPTIONS` files have increased in size (as more options are added), I thought I would suggest an increase. [ If the comments from the top of the `OPTIONS` file are removed, and white space from the start of lines is removed then the size can be reduced to be under 4K, but as more options are added the size seems likely to grow again. ] Create a new database: ``` > ./ldb --create_if_missing --db=/tmp/rdb_tmp put 1 1 OK ``` The OPTIONS file is 4252 bytes: ``` > stat /tmp/rdb_tmp/OPTIONS* | head -n 2 File: ‘/tmp/rdb_tmp/OPTIONS-000005’ Size: 4252 Blocks: 16 IO Block: 4096 regular file ``` Before, the 4096 byte buffer is used from 2 system read calls: ``` > strace -f ./ldb --try_load_options --db=/tmp/rdb_tmp get DOES_NOT_EXIST 2>&1 | grep -A 1 'RocksDB option file' read(3, "# This is a RocksDB option file."..., 4096) = 4096 read(3, "e\n metadata_block_size=4096\n c"..., 4096) = 156 ``` ltrace shows 4096 passed to fread_unlocked ``` > ltrace -S -f ./ldb --try_load_options --db=/tmp/rdb_tmp get DOES_NOT_EXIST 2>&1 | grep -C 3 'RocksDB option file' [pid 51013] fread_unlocked(0x7ffd5fbf2d50, 1, 4096, 0x7fd2e084e780 <unfinished ...> [pid 51013] fstat@SYS(3, 0x7ffd5fbf28f0) = 0 [pid 51013] mmap@SYS(nil, 4096, 3, 34, -1, 0) = 0x7fd2e318c000 [pid 51013] read@SYS(3, "# This is a RocksDB option file."..., 4096) = 4096 [pid 51013] <... fread_unlocked resumed> ) = 4096 ... ``` After, the 8192 byte buffer is used from 1 system read call: ``` > strace -f ./ldb --try_load_options --db=/tmp/rdb_tmp get DOES_NOT_EXIST 2>&1 | grep -A 1 'RocksDB option file' read(3, "# This is a RocksDB option file."..., 8192) = 4252 read(3, "", 4096) = 0 ``` ltrace shows 8192 passed to fread_unlocked ``` > ltrace -S -f ./ldb --try_load_options --db=/tmp/rdb_tmp get DOES_NOT_EXIST 2>&1 | grep -C 3 'RocksDB option file' [pid 146611] fread_unlocked(0x7ffcfba382f0, 1, 8192, 0x7fc4e844e780 <unfinished ...> [pid 146611] fstat@SYS(3, 0x7ffcfba380f0) = 0 [pid 146611] mmap@SYS(nil, 4096, 3, 34, -1, 0) = 0x7fc4eaee0000 [pid 146611] read@SYS(3, "# This is a RocksDB option file."..., 8192) = 4252 [pid 146611] read@SYS(3, "", 4096) = 0 [pid 146611] <... fread_unlocked resumed> ) = 4252 [pid 146611] feof(0x7fc4e844e780) = 1 ``` Closes https://github.com/facebook/rocksdb/pull/3294 Differential Revision: D6653684 Pulled By: ajkr fbshipit-source-id: 222f25f5442fefe1dcec18c700bd9e235bb63491
-
由 Changli Gao 提交于
Summary: Because the corresponding read_first_record_cache_ item wasn't erased, memory leaked. Closes https://github.com/facebook/rocksdb/pull/1712 Differential Revision: D4363654 Pulled By: ajkr fbshipit-source-id: 7da1adcfc8c380e4ffe05b8769fc2221ad17a225
-
由 Andrew Kryczka 提交于
Summary: bump nokogiri number Closes https://github.com/facebook/rocksdb/pull/3358 Differential Revision: D6708596 Pulled By: ajkr fbshipit-source-id: 6662c3ba4994374ecf8a13928e915b655a980b70
-
由 Bo Liu 提交于
Summary: to save a string copy for some use cases. The change is pretty straightforward, please feel free to let me know if you want to suggest any tests for it. Closes https://github.com/facebook/rocksdb/pull/3349 Differential Revision: D6706828 Pulled By: yiwu-arbug fbshipit-source-id: 873ce4442937bdc030b395c7f99228eda7f59eb7
-
由 Adam Retter 提交于
Summary: Closes https://github.com/facebook/rocksdb/pull/3356 Differential Revision: D6706909 Pulled By: sagar0 fbshipit-source-id: 6e4757d9eceab3e8a6c1b83c1be4108e86576cb2
-
由 Adam Retter 提交于
Summary: Tested on a clean FreeBSD 11.01 x64. Closes https://github.com/facebook/rocksdb/pull/1423 Closes https://github.com/facebook/rocksdb/pull/3357 Differential Revision: D6705868 Pulled By: sagar0 fbshipit-source-id: cbccbbdafd4f42922512ca03619a5d5583a425fd
-
- 11 1月, 2018 4 次提交
-
-
由 Dmitri Smirnov 提交于
Summary: Re-use metadata for reading Compression Dictionary on BlockBased table open, this saves two reads from disk. This helps to our 999 percentile in 5.6.1 where prefetch buffer is not present. Closes https://github.com/facebook/rocksdb/pull/3354 Differential Revision: D6695753 Pulled By: ajkr fbshipit-source-id: bb8acd9e9e66e65b89c548ab8940570ae360333c
-
由 Andrew Kryczka 提交于
Summary: It was using the same directory as `db_options_test` so transiently failed when unit tests were run in parallel. Closes https://github.com/facebook/rocksdb/pull/3352 Differential Revision: D6691649 Pulled By: ajkr fbshipit-source-id: bee433484fec4faedd5cadf2db3c92fdcc99a170
-
由 Anand Ananthabhotla 提交于
Summary: Add a new bool option index_uncompressed in BlockBasedTableOptions. Closes https://github.com/facebook/rocksdb/pull/3303 Differential Revision: D6686161 Pulled By: anand1976 fbshipit-source-id: 748b46993d48a01e5f89b6bd3e41f06a59ec6054
-
由 Andrew Kryczka 提交于
Summary: - Change directory name from "db_test" to "checkpoint_test". Previously it used the same directory as `db_test` - Systematically cleanup snapshot and snapshot staging directories before each test. Previously a failed test run caused subsequent runs to fail, particularly when the first failure caused "snapshot.tmp" to not be cleaned up. Closes https://github.com/facebook/rocksdb/pull/3351 Differential Revision: D6691015 Pulled By: ajkr fbshipit-source-id: 4fc2ac2e21ff2617ea0e96297c5132b5f2eefd79
-
- 10 1月, 2018 6 次提交
-
-
由 Siying Dong 提交于
Summary: Most popular versions of GCC can't identify platform on ARM if "-march=native" is specified. Remove it to unblock most people. Closes https://github.com/facebook/rocksdb/pull/3346 Differential Revision: D6690544 Pulled By: siying fbshipit-source-id: bbaba9fe2645b6b37144b36ea75beeff88992b49
-
由 Sagar Vemuri 提交于
Summary: Fixed RocksJava travis build failure due to a missing file in java/CMakeLists.txt. (from #3332). Closes https://github.com/facebook/rocksdb/pull/3344 Differential Revision: D6686472 Pulled By: sagar0 fbshipit-source-id: dd3281dff1342c3a7235c402890420aa56db0fe3
-
由 Siying Dong 提交于
Summary: I experienced weird segfault because of this mismatch of type in log formatting. Fix it. Closes https://github.com/facebook/rocksdb/pull/3345 Differential Revision: D6687224 Pulled By: siying fbshipit-source-id: c51fb1c008b7ebc3efdc353a4adad3e8f5b3e9de
-
由 Andrew Kryczka 提交于
Summary: got confused while reading `FindObsoleteFiles` due to thinking it's a local variable, so renamed it properly Closes https://github.com/facebook/rocksdb/pull/3342 Differential Revision: D6684797 Pulled By: ajkr fbshipit-source-id: a4df0aae1cccce99d4dd4d164aadc85b17707132
-
由 Yi Wu 提交于
Summary: The macro was added by mistake in #2372 Closes https://github.com/facebook/rocksdb/pull/3343 Differential Revision: D6681356 Pulled By: yiwu-arbug fbshipit-source-id: 4180172fb0eaef4189c07f219241e0c261c03461
-
由 Maysam Yabandeh 提交于
Summary: This patch addresses a couple of minor TODOs for WritePrepared Txn such as double checking some assert statements at runtime as well, skip extra AddPrepared in non-2pc transactions, and safety check for infinite loops. Closes https://github.com/facebook/rocksdb/pull/3302 Differential Revision: D6617002 Pulled By: maysamyabandeh fbshipit-source-id: ef6673c139cb49f64c0879508d2f573b78609aca
-
- 09 1月, 2018 2 次提交
-
-
由 Chris Lu 提交于
Summary: https://github.com/facebook/rocksdb/wiki/Creating-and-Ingesting-SST-files Need to expose these functions in the C API to be used by Go bindings. Closes https://github.com/facebook/rocksdb/pull/3011 Differential Revision: D6679563 Pulled By: sagar0 fbshipit-source-id: 536f844ddaeb0172c6d7e416d2a75e8f9e57c8ef
-
由 Adam Retter 提交于
Summary: Closes https://github.com/facebook/rocksdb/pull/3332 Differential Revision: D6667680 Pulled By: ajkr fbshipit-source-id: b2bb6889257850a4eb6f6cbd7106f62df7b82730
-
- 06 1月, 2018 4 次提交
-
-
由 Yi Wu 提交于
Summary: Previously on a blob db read, we are making a read of the blob value, and then make another read to get CRC checksum. I'm combining the two read into one. readrandom db_bench with 1G database with base db size of 13M, value size 1k: `./db_bench --db=/home/yiwu/tmp/db_bench --use_blob_db --value_size=1024 --num=1000000 --benchmarks=readrandom --use_existing_db --cache_size=32000000` master: throughput 234MB/s, get micros p50 5.984 p95 9.998 p99 20.817 p100 787 this PR: throughput 261MB/s, get micros p50 5.157 p95 9.928 p99 20.724 p100 190 Closes https://github.com/facebook/rocksdb/pull/3301 Differential Revision: D6615950 Pulled By: yiwu-arbug fbshipit-source-id: 052410c6d8539ec0cc305d53793bbc8f3616baa3
-
由 Sagar Vemuri 提交于
Summary: With the ZSTD dictionary generator support added in #3057 `PORTABLE=1 ROCKSDB_NO_FBCODE=1 make rocksdbjavastatic` fails as it can't find zdict.h. Specifically due to: https://github.com/facebook/rocksdb/blob/e3a06f12d27fd50af7b6c5941973f529601f9a3e/util/compression.h#L39 In java static builds zstd code gets directly downloaded from https://github.com/facebook/zstd , and in there zdict.h is under dictBuilder directory. So, I modified libzstd.a target to use `make install` to collect all the header files into a single location and used that as the zstd's include path. Closes https://github.com/facebook/rocksdb/pull/3260 Differential Revision: D6669850 Pulled By: sagar0 fbshipit-source-id: f8a7562a670e5aed4c4fb6034a921697590d7285
-
由 Dmitri Smirnov 提交于
Summary: Make dependacies switches compatible with other OS builds TODO: Make find_package work for Windows. Closes https://github.com/facebook/rocksdb/pull/3322 Differential Revision: D6667637 Pulled By: sagar0 fbshipit-source-id: 5afcd7bbfe69465310a4fbc8e589f01e506b95f5
-
由 Adam Retter 提交于
Summary: 1. Snappy 1.1.7 2. LZ4 1.8.0 3. ZSTD 1.3.3 Closes https://github.com/facebook/rocksdb/pull/3331 Differential Revision: D6667933 Pulled By: ajkr fbshipit-source-id: 21c526609df7580481195a389d31f733e2695e65
-
- 05 1月, 2018 1 次提交
-
-
由 Maysam Yabandeh 提交于
Summary: DestroyDB that is used in tests loops over the files returned by ::GetChildren and delete them one by one. Such files might be already deleted in the file system (during DeleteObsoleteFileImpl for example) but will get actually deleted with a delay sometimes before ::DeleteFile is called on the file name. We have some test failures where FaultInjectionTestEnv::DeleteFile fails on assert(s.ok()) during DestroyDB. This patch removes the assert statement to fix that. Closes https://github.com/facebook/rocksdb/pull/3324 Differential Revision: D6659545 Pulled By: maysamyabandeh fbshipit-source-id: 4c9552fbcd494dcf3e61d475c11fc965c4388b2c
-
- 04 1月, 2018 2 次提交
-
-
由 Andrew Kryczka 提交于
Summary: added support for C and asm files as required for e612e317. Closes https://github.com/facebook/rocksdb/pull/3299 Differential Revision: D6612479 Pulled By: ajkr fbshipit-source-id: 6263ed7c1602f249460421825c76b5721f396163
-
由 Adam Retter 提交于
Summary: As discovered during v5.9.2 release, and forward-ported. Closes https://github.com/facebook/rocksdb/pull/3323 Differential Revision: D6657209 Pulled By: siying fbshipit-source-id: b560d9f8ddb89e0ffaff7c895ec80f68ddf7dab4
-