- 12 7月, 2017 4 次提交
-
-
由 Adam Retter 提交于
Summary: Closes https://github.com/facebook/rocksdb/pull/2551 Differential Revision: D5399288 Pulled By: sagar0 fbshipit-source-id: dd3df2ed6cc5ae612db0998ea746cc29fccf568e
-
由 Islam AbdelRahman 提交于
Summary: Bump version to 5.7 Closes https://github.com/facebook/rocksdb/pull/2566 Differential Revision: D5400043 Pulled By: IslamAbdelRahman fbshipit-source-id: 74aae4ff143d370d7d89807e5be08a6ab827da40
-
由 Yi Wu 提交于
Summary: * Create info log before db open to make blob db able to log to LOG file. * Properly destroy blob db. Closes https://github.com/facebook/rocksdb/pull/2567 Differential Revision: D5400034 Pulled By: yiwu-arbug fbshipit-source-id: a49cfaf4b5c67d42d4cbb872bd5a9441828c17ce
-
由 Daniel Black 提交于
Summary: Found by gcc warning: x86_64-pc-linux-gnu-g++ --version x86_64-pc-linux-gnu-g++ (GCC) 7.1.1 20170710 tools/db_bench_tool.cc: In member function 'void rocksdb::Benchmark::RandomWithVerify(rocksdb::ThreadState*)': tools/db_bench_tool.cc:4430:8: error: '%lu' directive output may be truncated writing between 1 and 19 bytes into a region of size between 0 and 66 [-Werror=format-truncation=] void RandomWithVerify(ThreadState* thread) { ^~~~~~~~~~~~~~~~ tools/db_bench_tool.cc:4430:8: note: directive argument in the range [0, 9223372036854775807] tools/db_bench_tool.cc:4492:13: note: 'snprintf' output between 37 and 128 bytes into a destination of size 100 snprintf(msg, sizeof(msg), ~~~~~~~~^~~~~~~~~~~~~~~~~~ "( get:%" PRIu64 " put:%" PRIu64 " del:%" PRIu64 " total:%" \ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ PRIu64 " found:%" PRIu64 ")", ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ gets_done, puts_done, deletes_done, readwrites_, found); ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ cc1plus: all warnings being treated as errors Makefile:1707: recipe for target 'tools/db_bench_tool.o' failed Closes https://github.com/facebook/rocksdb/pull/2558 Differential Revision: D5398703 Pulled By: siying fbshipit-source-id: 6ffa552bbd8b59cfc2c36289f86ff9b9acca8ca6
-
- 11 7月, 2017 3 次提交
-
-
由 Aaron Gao 提交于
Summary: delete the checkout line Closes https://github.com/facebook/rocksdb/pull/2556 Differential Revision: D5394446 Pulled By: lightmark fbshipit-source-id: 4afaf35e7ad0378d12169d13a3131a92b428b5d2
-
由 Giuseppe Ottaviano 提交于
Summary: Instead of ignoring UBSan checks, fix the negative shifts in Hash(). Also add test to make sure the hash values are stable over time. The values were computed before this change, so the test also verifies the correctness of the change. Closes https://github.com/facebook/rocksdb/pull/2546 Differential Revision: D5386369 Pulled By: yiwu-arbug fbshipit-source-id: 6de4b44461a544d6222cc5d72d8cda2c0373d17e
-
由 Adam Retter 提交于
Summary: Closes https://github.com/facebook/rocksdb/pull/2554 Differential Revision: D5391019 Pulled By: siying fbshipit-source-id: a1d36037ef4c34b9ca77ab2127f3e17242c617cc
-
- 08 7月, 2017 3 次提交
-
-
由 Sagar Vemuri 提交于
Summary: FIFOCompactionWithTTLTests are flaky when run in parallel, as there is a time element involved to it. Temporarily disabling them while I investigate a more robust testing solution like, say, mocking time. Closes https://github.com/facebook/rocksdb/pull/2548 Differential Revision: D5386084 Pulled By: sagar0 fbshipit-source-id: 262886b25bdf091021d8553e780443a985e9bac4
-
由 Maysam Yabandeh 提交于
Summary: Closes https://github.com/facebook/rocksdb/pull/2549 Differential Revision: D5386307 Pulled By: maysamyabandeh fbshipit-source-id: 3032c95c54755053b6450765ec4dacbecb734f9d
-
由 Andrew Kryczka 提交于
Summary: As titled. Also fixed an off-by-one error causing us to add one less range deletion than the user specified. Closes https://github.com/facebook/rocksdb/pull/2544 Differential Revision: D5383451 Pulled By: ajkr fbshipit-source-id: cbd5890c33f09bbb5c0c1f4bb952a1add32336e0
-
- 07 7月, 2017 2 次提交
-
-
由 Aaron Gao 提交于
Summary: target `utilities/merge_operators/cassandra/test_utils.d' given more than once in the same rule remove the duplicate one Closes https://github.com/facebook/rocksdb/pull/2542 Differential Revision: D5379570 Pulled By: lightmark fbshipit-source-id: 353f38d085e9f627c1b3c53acef7a2c4bc71014d
-
由 Siying Dong 提交于
Summary: Closes https://github.com/facebook/rocksdb/pull/2529 Differential Revision: D5378787 Pulled By: siying fbshipit-source-id: 801ecacc2804f77cb3c9e5e665829e439db68800
-
- 06 7月, 2017 5 次提交
-
-
由 Maysam Yabandeh 提交于
Summary: Valgrind reports that it is not initialized. Closes https://github.com/facebook/rocksdb/pull/2541 Differential Revision: D5376084 Pulled By: maysamyabandeh fbshipit-source-id: 55c312f4f506863aa0d25ff92c8c34b57f48b860
-
由 Maysam Yabandeh 提交于
Summary: Closes https://github.com/facebook/rocksdb/pull/2537 Differential Revision: D5371433 Pulled By: maysamyabandeh fbshipit-source-id: 90d3e8bb1a8576f48b1ddf1bdbba5512b5986ba0
-
由 Aaron Gao 提交于
Summary: https://github.com/facebook/rocksdb/releases/tag/v5.5.1 Closes https://github.com/facebook/rocksdb/pull/2522 Differential Revision: D5365355 Pulled By: lightmark fbshipit-source-id: 63371a98ebc7429766fd86dc27c1ea35950c98f5
-
由 Andrew Kryczka 提交于
Summary: Valgrind had false positive complaints about the initialization pattern for `GetCurrentTime()`'s argument in #2480. We can instead have the client initialize the time variable before calling `GetCurrentTime()`, and have `GetCurrentTime()` promise to only overwrite it in success case. Closes https://github.com/facebook/rocksdb/pull/2526 Differential Revision: D5358689 Pulled By: ajkr fbshipit-source-id: 857b189f24c19196f6bb299216f3e23e7bc4be42
-
由 Maysam Yabandeh 提交于
Summary: Closes https://github.com/facebook/rocksdb/pull/2536 Differential Revision: D5371271 Pulled By: maysamyabandeh fbshipit-source-id: f1355ac658a79c9982a24986f0925c9e24fc39d5
-
- 03 7月, 2017 1 次提交
-
-
由 Maysam Yabandeh 提交于
Summary: Currently metadata_block_size controls only index partition size. With this patch a partition is cut after any of index or filter partitions reaches metadata_block_size. Closes https://github.com/facebook/rocksdb/pull/2452 Differential Revision: D5275651 Pulled By: maysamyabandeh fbshipit-source-id: 5057e4424b4c8902043782e6bf8c38f0c4f25160
-
- 01 7月, 2017 4 次提交
-
-
由 Andrew Kryczka 提交于
Summary: Mentioned changes: - #2477 - #2503 Closes https://github.com/facebook/rocksdb/pull/2528 Differential Revision: D5360185 Pulled By: ajkr fbshipit-source-id: 59d6ae465bcb0aa0a739317581fa3fc7871c6de6
-
由 Siying Dong 提交于
Summary: Expose pipeline write in db_bench and change the default to parallel memtable inserts Closes https://github.com/facebook/rocksdb/pull/2527 Differential Revision: D5359825 Pulled By: siying fbshipit-source-id: e30755feb07ff19a731c4058acf101e02de4e197
-
由 Maysam Yabandeh 提交于
Summary: Closes https://github.com/facebook/rocksdb/pull/2525 Differential Revision: D5356859 Pulled By: maysamyabandeh fbshipit-source-id: f593adad2a8aab12dcd6ab25db076eca51d30d34
-
由 Maysam Yabandeh 提交于
Summary: Adding/Correcting inline comments and clarify the sync rules. To make it simple to reason, the rules are a big general which ended up to some extra synchronizations. However such synchronizations are not on the fast path, and they are worth the simplicity. Closes https://github.com/facebook/rocksdb/pull/2517 Differential Revision: D5348239 Pulled By: maysamyabandeh fbshipit-source-id: ff2e59fb1e568c122d2cdbf598310f3613b7d212
-
- 30 6月, 2017 4 次提交
-
-
由 Andrew Kryczka 提交于
Summary: Issue: #2478 Fix: #2503 The bug happened when all of these conditions were satisfied: - A subcompaction generates no keys - `RangeDelAggregator::ShouldAddTombstones()` returns true because there's at least one non-obsoleted range deletion in its map - None of the non-obsolete tombstones overlap with the subcompaction key-range Under those conditions, we were creating a dedicated file for range deletions which was left empty, thus causing an error in VersionEdit. I verified this test case fails before the #2503 fix and passes after. Closes https://github.com/facebook/rocksdb/pull/2521 Differential Revision: D5352568 Pulled By: ajkr fbshipit-source-id: f619cae39984ce9bb9b7a4e7a9ac0f2bb2ce43e9
-
由 Maysam Yabandeh 提交于
Summary: AddDBStats is in two steps of load and store, which is more efficient than fetch_add. This is however not thread-safe. Currently we have to protect concurrent access to AddDBStats with a mutex which is less efficient that fetch_add. This patch adds the option to do fetch_add when AddDBStats. The results for my 2pc benchmark on sysbench is: - vanilla: 68618 tps - removing mutex on AddDBStats (unsafe): 69767 tps - fetch_add for all AddDBStats: 69200 tps - fetch_add only for concurrently access AddDBStats (this patch): 69579 tps Closes https://github.com/facebook/rocksdb/pull/2505 Differential Revision: D5330656 Pulled By: maysamyabandeh fbshipit-source-id: af64d7bee135b0e86b4fac323a4f9d9113eaa383
-
由 zhangjinpeng1987 提交于
Summary: When a compaction job output nothing, there is no necessary to generate a empty sst file which will cause `VersionEdit::EncodeTo` failed. ref https://github.com/facebook/rocksdb/issues/2478 Closes https://github.com/facebook/rocksdb/pull/2503 Differential Revision: D5350799 Pulled By: ajkr fbshipit-source-id: df0b4fcf3507fe1c3c435208b762e75478e00143
-
由 Yi Wu 提交于
Summary: The comma "," is not a valid separator for bash arrays. Closes https://github.com/facebook/rocksdb/pull/2516 Differential Revision: D5348101 Pulled By: yiwu-arbug fbshipit-source-id: 8f0afdac368e21076eb7366b7df7dbaaf158cf96
-
- 29 6月, 2017 4 次提交
-
-
由 Siying Dong 提交于
Summary: 11c5d474 introduces a bug that IsFastCrc32Supported() returns wrong result. Fix it. Also fix some FB internal scripts. Closes https://github.com/facebook/rocksdb/pull/2513 Differential Revision: D5343802 Pulled By: yiwu-arbug fbshipit-source-id: 057dc7ae3b262fe951413d1190ce60afc788cc05
-
由 Mike Kolupaev 提交于
Summary: We've got some DBs where iterators return Status with message "Corruption: block checksum mismatch" all the time. That's not very informative. It would be much easier to investigate if the error message contained the file name - then we would know e.g. how old the corrupted file is, which would be very useful for finding the root cause. This PR adds file name, offset and other stuff to some block corruption-related status messages. It doesn't improve all the error messages, just a few that were easy to improve. I'm mostly interested in "block checksum mismatch" and "Bad table magic number" since they're the only corruption errors that I've ever seen in the wild. Closes https://github.com/facebook/rocksdb/pull/2507 Differential Revision: D5345702 Pulled By: al13n321 fbshipit-source-id: fc8023d43f1935ad927cef1b9c55481ab3cb1339
-
由 Siying Dong 提交于
Summary: "make analyze" is reporting some errors. It's complicated to look but it seems to me that they are all false positive. Anyway, I think cleaning them up is a good idea. Some of the changes are hacky but I don't know a better way. Closes https://github.com/facebook/rocksdb/pull/2508 Differential Revision: D5341710 Pulled By: siying fbshipit-source-id: 6070e430e0e41a080ef441e05e8ec827d45efab6
-
由 Maysam Yabandeh 提交于
Summary: This is to resolve the asan complains. In the meanwhile I am working on clarifying/revisiting the sync rules. Closes https://github.com/facebook/rocksdb/pull/2510 Differential Revision: D5338660 Pulled By: yiwu-arbug fbshipit-source-id: ce6f6e0826d43a2c0bfa4328a00c78f73cd6498a
-
- 28 6月, 2017 5 次提交
-
-
由 Sagar Vemuri 提交于
Summary: Introducing FIFO compactions with TTL. FIFO compaction is based on size only which makes it tricky to enable in production as use cases can have organic growth. A user requested an option to drop files based on the time of their creation instead of the total size. To address that request: - Added a new TTL option to FIFO compaction options. - Updated FIFO compaction score to take TTL into consideration. - Added a new table property, creation_time, to keep track of when the SST file is created. - Creation_time is set as below: - On Flush: Set to the time of flush. - On Compaction: Set to the max creation_time of all the files involved in the compaction. - On Repair and Recovery: Set to the time of repair/recovery. - Old files created prior to this code change will have a creation_time of 0. - FIFO compaction with TTL is enabled when ttl > 0. All files older than ttl will be deleted during compaction. i.e. `if (file.creation_time < (current_time - ttl)) then delete(file)`. This will enable cases where you might want to delete all files older than, say, 1 day. - FIFO compaction will fall back to the prior way of deleting files based on size if: - the creation_time of all files involved in compaction is 0. - the total size (of all SST files combined) does not drop below `compaction_options_fifo.max_table_files_size` even if the files older than ttl are deleted. This feature is not supported if max_open_files != -1 or with table formats other than Block-based. **Test Plan:** Added tests. **Benchmark results:** Base: FIFO with max size: 100MB :: ``` svemuri@dev15905 ~/rocksdb (fifo-compaction) $ TEST_TMPDIR=/dev/shm ./db_bench --benchmarks=readwhilewriting --num=5000000 --threads=16 --compaction_style=2 --fifo_compaction_max_table_files_size_mb=100 readwhilewriting : 1.924 micros/op 519858 ops/sec; 13.6 MB/s (1176277 of 5000000 found) ``` With TTL (a low one for testing) :: ``` svemuri@dev15905 ~/rocksdb (fifo-compaction) $ TEST_TMPDIR=/dev/shm ./db_bench --benchmarks=readwhilewriting --num=5000000 --threads=16 --compaction_style=2 --fifo_compaction_max_table_files_size_mb=100 --fifo_compaction_ttl=20 readwhilewriting : 1.902 micros/op 525817 ops/sec; 13.7 MB/s (1185057 of 5000000 found) ``` Example Log lines: ``` 2017/06/26-15:17:24.609249 7fd5a45ff700 (Original Log Time 2017/06/26-15:17:24.609177) [db/compaction_picker.cc:1471] [default] FIFO compaction: picking file 40 with creation time 1498515423 for deletion 2017/06/26-15:17:24.609255 7fd5a45ff700 (Original Log Time 2017/06/26-15:17:24.609234) [db/db_impl_compaction_flush.cc:1541] [default] Deleted 1 files ... 2017/06/26-15:17:25.553185 7fd5a61a5800 [DEBUG] [db/db_impl_files.cc:309] [JOB 0] Delete /dev/shm/dbbench/000040.sst type=2 #40 -- OK 2017/06/26-15:17:25.553205 7fd5a61a5800 EVENT_LOG_v1 {"time_micros": 1498515445553199, "job": 0, "event": "table_file_deletion", "file_number": 40} ``` SST Files remaining in the dbbench dir, after db_bench execution completed: ``` svemuri@dev15905 ~/rocksdb (fifo-compaction) $ ls -l /dev/shm//dbbench/*.sst -rw-r--r--. 1 svemuri users 30749887 Jun 26 15:17 /dev/shm//dbbench/000042.sst -rw-r--r--. 1 svemuri users 30768779 Jun 26 15:17 /dev/shm//dbbench/000044.sst -rw-r--r--. 1 svemuri users 30757481 Jun 26 15:17 /dev/shm//dbbench/000046.sst ``` Closes https://github.com/facebook/rocksdb/pull/2480 Differential Revision: D5305116 Pulled By: sagar0 fbshipit-source-id: 3e5cfcf5dd07ed2211b5b37492eb235b45139174
-
由 Yi Wu 提交于
Summary: 1. The buckifier script assume each test "foo" comes with a .cc file of the same name (i.e. foo.cc). Update cassandra tests to follow this pattern so that the buckifier script can recognize them. 2. add blob_db_test Closes https://github.com/facebook/rocksdb/pull/2506 Differential Revision: D5331517 Pulled By: yiwu-arbug fbshipit-source-id: 86f3eba471fc621186ab44cbd073b6162cde8e57
-
由 Yi Wu 提交于
Summary: Allow numa >= 2.0.8 in buck TARGET file. Closes https://github.com/facebook/rocksdb/pull/2504 Differential Revision: D5330550 Pulled By: yiwu-arbug fbshipit-source-id: 8ffb6167b4ad913877eac16a20a91023b31f8d41
-
由 Siying Dong 提交于
Summary: Closes https://github.com/facebook/rocksdb/pull/2502 Differential Revision: D5326498 Pulled By: siying fbshipit-source-id: 2f0ac6dc6ca5ddb23cecf67a278c086e52646714
-
由 Yi Wu 提交于
Summary: update compatible test to include 5.5 and 5.6 branch. Closes https://github.com/facebook/rocksdb/pull/2501 Differential Revision: D5325220 Pulled By: yiwu-arbug fbshipit-source-id: 5f5271491e6dd2d7b2cf73a7142f38a571553bc4
-
- 27 6月, 2017 5 次提交
-
-
由 Siying Dong 提交于
Summary: A typo conversion fails Windows build. Fix it. Closes https://github.com/facebook/rocksdb/pull/2500 Differential Revision: D5325962 Pulled By: siying fbshipit-source-id: 2cefdafc9afbc85f856f403af7c876b622400630
-
由 Ewout Prangsma 提交于
Summary: This PR adds support for encrypting data stored by RocksDB when written to disk. It adds an `EncryptedEnv` override of the `Env` class with matching overrides for sequential&random access files. The encryption itself is done through a configurable `EncryptionProvider`. This class creates is asked to create `BlockAccessCipherStream` for a file. This is where the actual encryption/decryption is being done. Currently there is a Counter mode implementation of `BlockAccessCipherStream` with a `ROT13` block cipher (NOTE the `ROT13` is for demo purposes only!!). The Counter operation mode uses an initial counter & random initialization vector (IV). Both are created randomly for each file and stored in a 4K (default size) block that is prefixed to that file. The `EncryptedEnv` implementation is such that clients of the `Env` class do not see this prefix (nor data, nor in filesize). The largest part of the prefix block is also encrypted, and there is room left for implementation specific settings/values/keys in there. To test the encryption, the `DBTestBase` class has been extended to consider a new environment variable called `ENCRYPTED_ENV`. If set, the test will setup a encrypted instance of the `Env` class to use for all tests. Typically you would run it like this: ``` ENCRYPTED_ENV=1 make check_some ``` There is also an added test that checks that some data inserted into the database is or is not "visible" on disk. With `ENCRYPTED_ENV` active it must not find plain text strings, with `ENCRYPTED_ENV` unset, it must find the plain text strings. Closes https://github.com/facebook/rocksdb/pull/2424 Differential Revision: D5322178 Pulled By: sdwilsh fbshipit-source-id: 253b0a9c2c498cc98f580df7f2623cbf7678a27f
-
由 Siying Dong 提交于
Summary: Closes https://github.com/facebook/rocksdb/pull/2499 Differential Revision: D5324914 Pulled By: siying fbshipit-source-id: c69827d4dddafa81e651180633943a3380cdd5bb
-
由 Yi Wu 提交于
Summary: WriteBatchThreadedTest is failing, at least on Mac. The problem seems to be `wb` is getting GC before we finish write. Explicitly close it seems to fix it. Closes https://github.com/facebook/rocksdb/pull/2482 Differential Revision: D5307379 Pulled By: yiwu-arbug fbshipit-source-id: 8ff7f8170451078c941951f5aafae83afffb7933
-
由 Aaron Gao 提交于
Summary: https://github.com/facebook/rocksdb/pull/2380 introduces a regression by replacing __thread with ThreadLocalPtr. Revert the thread local implementation back. Closes https://github.com/facebook/rocksdb/pull/2485 Differential Revision: D5308050 Pulled By: lightmark fbshipit-source-id: 2676e9c22edf76e8133d3f4c50e2711e11a95480
-