- 08 7月, 2017 1 次提交
-
-
由 Andrew Kryczka 提交于
Summary: As titled. Also fixed an off-by-one error causing us to add one less range deletion than the user specified. Closes https://github.com/facebook/rocksdb/pull/2544 Differential Revision: D5383451 Pulled By: ajkr fbshipit-source-id: cbd5890c33f09bbb5c0c1f4bb952a1add32336e0
-
- 07 7月, 2017 2 次提交
-
-
由 Aaron Gao 提交于
Summary: target `utilities/merge_operators/cassandra/test_utils.d' given more than once in the same rule remove the duplicate one Closes https://github.com/facebook/rocksdb/pull/2542 Differential Revision: D5379570 Pulled By: lightmark fbshipit-source-id: 353f38d085e9f627c1b3c53acef7a2c4bc71014d
-
由 Siying Dong 提交于
Summary: Closes https://github.com/facebook/rocksdb/pull/2529 Differential Revision: D5378787 Pulled By: siying fbshipit-source-id: 801ecacc2804f77cb3c9e5e665829e439db68800
-
- 06 7月, 2017 5 次提交
-
-
由 Maysam Yabandeh 提交于
Summary: Valgrind reports that it is not initialized. Closes https://github.com/facebook/rocksdb/pull/2541 Differential Revision: D5376084 Pulled By: maysamyabandeh fbshipit-source-id: 55c312f4f506863aa0d25ff92c8c34b57f48b860
-
由 Maysam Yabandeh 提交于
Summary: Closes https://github.com/facebook/rocksdb/pull/2537 Differential Revision: D5371433 Pulled By: maysamyabandeh fbshipit-source-id: 90d3e8bb1a8576f48b1ddf1bdbba5512b5986ba0
-
由 Aaron Gao 提交于
Summary: https://github.com/facebook/rocksdb/releases/tag/v5.5.1 Closes https://github.com/facebook/rocksdb/pull/2522 Differential Revision: D5365355 Pulled By: lightmark fbshipit-source-id: 63371a98ebc7429766fd86dc27c1ea35950c98f5
-
由 Andrew Kryczka 提交于
Summary: Valgrind had false positive complaints about the initialization pattern for `GetCurrentTime()`'s argument in #2480. We can instead have the client initialize the time variable before calling `GetCurrentTime()`, and have `GetCurrentTime()` promise to only overwrite it in success case. Closes https://github.com/facebook/rocksdb/pull/2526 Differential Revision: D5358689 Pulled By: ajkr fbshipit-source-id: 857b189f24c19196f6bb299216f3e23e7bc4be42
-
由 Maysam Yabandeh 提交于
Summary: Closes https://github.com/facebook/rocksdb/pull/2536 Differential Revision: D5371271 Pulled By: maysamyabandeh fbshipit-source-id: f1355ac658a79c9982a24986f0925c9e24fc39d5
-
- 03 7月, 2017 1 次提交
-
-
由 Maysam Yabandeh 提交于
Summary: Currently metadata_block_size controls only index partition size. With this patch a partition is cut after any of index or filter partitions reaches metadata_block_size. Closes https://github.com/facebook/rocksdb/pull/2452 Differential Revision: D5275651 Pulled By: maysamyabandeh fbshipit-source-id: 5057e4424b4c8902043782e6bf8c38f0c4f25160
-
- 01 7月, 2017 4 次提交
-
-
由 Andrew Kryczka 提交于
Summary: Mentioned changes: - #2477 - #2503 Closes https://github.com/facebook/rocksdb/pull/2528 Differential Revision: D5360185 Pulled By: ajkr fbshipit-source-id: 59d6ae465bcb0aa0a739317581fa3fc7871c6de6
-
由 Siying Dong 提交于
Summary: Expose pipeline write in db_bench and change the default to parallel memtable inserts Closes https://github.com/facebook/rocksdb/pull/2527 Differential Revision: D5359825 Pulled By: siying fbshipit-source-id: e30755feb07ff19a731c4058acf101e02de4e197
-
由 Maysam Yabandeh 提交于
Summary: Closes https://github.com/facebook/rocksdb/pull/2525 Differential Revision: D5356859 Pulled By: maysamyabandeh fbshipit-source-id: f593adad2a8aab12dcd6ab25db076eca51d30d34
-
由 Maysam Yabandeh 提交于
Summary: Adding/Correcting inline comments and clarify the sync rules. To make it simple to reason, the rules are a big general which ended up to some extra synchronizations. However such synchronizations are not on the fast path, and they are worth the simplicity. Closes https://github.com/facebook/rocksdb/pull/2517 Differential Revision: D5348239 Pulled By: maysamyabandeh fbshipit-source-id: ff2e59fb1e568c122d2cdbf598310f3613b7d212
-
- 30 6月, 2017 4 次提交
-
-
由 Andrew Kryczka 提交于
Summary: Issue: #2478 Fix: #2503 The bug happened when all of these conditions were satisfied: - A subcompaction generates no keys - `RangeDelAggregator::ShouldAddTombstones()` returns true because there's at least one non-obsoleted range deletion in its map - None of the non-obsolete tombstones overlap with the subcompaction key-range Under those conditions, we were creating a dedicated file for range deletions which was left empty, thus causing an error in VersionEdit. I verified this test case fails before the #2503 fix and passes after. Closes https://github.com/facebook/rocksdb/pull/2521 Differential Revision: D5352568 Pulled By: ajkr fbshipit-source-id: f619cae39984ce9bb9b7a4e7a9ac0f2bb2ce43e9
-
由 Maysam Yabandeh 提交于
Summary: AddDBStats is in two steps of load and store, which is more efficient than fetch_add. This is however not thread-safe. Currently we have to protect concurrent access to AddDBStats with a mutex which is less efficient that fetch_add. This patch adds the option to do fetch_add when AddDBStats. The results for my 2pc benchmark on sysbench is: - vanilla: 68618 tps - removing mutex on AddDBStats (unsafe): 69767 tps - fetch_add for all AddDBStats: 69200 tps - fetch_add only for concurrently access AddDBStats (this patch): 69579 tps Closes https://github.com/facebook/rocksdb/pull/2505 Differential Revision: D5330656 Pulled By: maysamyabandeh fbshipit-source-id: af64d7bee135b0e86b4fac323a4f9d9113eaa383
-
由 zhangjinpeng1987 提交于
Summary: When a compaction job output nothing, there is no necessary to generate a empty sst file which will cause `VersionEdit::EncodeTo` failed. ref https://github.com/facebook/rocksdb/issues/2478 Closes https://github.com/facebook/rocksdb/pull/2503 Differential Revision: D5350799 Pulled By: ajkr fbshipit-source-id: df0b4fcf3507fe1c3c435208b762e75478e00143
-
由 Yi Wu 提交于
Summary: The comma "," is not a valid separator for bash arrays. Closes https://github.com/facebook/rocksdb/pull/2516 Differential Revision: D5348101 Pulled By: yiwu-arbug fbshipit-source-id: 8f0afdac368e21076eb7366b7df7dbaaf158cf96
-
- 29 6月, 2017 4 次提交
-
-
由 Siying Dong 提交于
Summary: 11c5d474 introduces a bug that IsFastCrc32Supported() returns wrong result. Fix it. Also fix some FB internal scripts. Closes https://github.com/facebook/rocksdb/pull/2513 Differential Revision: D5343802 Pulled By: yiwu-arbug fbshipit-source-id: 057dc7ae3b262fe951413d1190ce60afc788cc05
-
由 Mike Kolupaev 提交于
Summary: We've got some DBs where iterators return Status with message "Corruption: block checksum mismatch" all the time. That's not very informative. It would be much easier to investigate if the error message contained the file name - then we would know e.g. how old the corrupted file is, which would be very useful for finding the root cause. This PR adds file name, offset and other stuff to some block corruption-related status messages. It doesn't improve all the error messages, just a few that were easy to improve. I'm mostly interested in "block checksum mismatch" and "Bad table magic number" since they're the only corruption errors that I've ever seen in the wild. Closes https://github.com/facebook/rocksdb/pull/2507 Differential Revision: D5345702 Pulled By: al13n321 fbshipit-source-id: fc8023d43f1935ad927cef1b9c55481ab3cb1339
-
由 Siying Dong 提交于
Summary: "make analyze" is reporting some errors. It's complicated to look but it seems to me that they are all false positive. Anyway, I think cleaning them up is a good idea. Some of the changes are hacky but I don't know a better way. Closes https://github.com/facebook/rocksdb/pull/2508 Differential Revision: D5341710 Pulled By: siying fbshipit-source-id: 6070e430e0e41a080ef441e05e8ec827d45efab6
-
由 Maysam Yabandeh 提交于
Summary: This is to resolve the asan complains. In the meanwhile I am working on clarifying/revisiting the sync rules. Closes https://github.com/facebook/rocksdb/pull/2510 Differential Revision: D5338660 Pulled By: yiwu-arbug fbshipit-source-id: ce6f6e0826d43a2c0bfa4328a00c78f73cd6498a
-
- 28 6月, 2017 5 次提交
-
-
由 Sagar Vemuri 提交于
Summary: Introducing FIFO compactions with TTL. FIFO compaction is based on size only which makes it tricky to enable in production as use cases can have organic growth. A user requested an option to drop files based on the time of their creation instead of the total size. To address that request: - Added a new TTL option to FIFO compaction options. - Updated FIFO compaction score to take TTL into consideration. - Added a new table property, creation_time, to keep track of when the SST file is created. - Creation_time is set as below: - On Flush: Set to the time of flush. - On Compaction: Set to the max creation_time of all the files involved in the compaction. - On Repair and Recovery: Set to the time of repair/recovery. - Old files created prior to this code change will have a creation_time of 0. - FIFO compaction with TTL is enabled when ttl > 0. All files older than ttl will be deleted during compaction. i.e. `if (file.creation_time < (current_time - ttl)) then delete(file)`. This will enable cases where you might want to delete all files older than, say, 1 day. - FIFO compaction will fall back to the prior way of deleting files based on size if: - the creation_time of all files involved in compaction is 0. - the total size (of all SST files combined) does not drop below `compaction_options_fifo.max_table_files_size` even if the files older than ttl are deleted. This feature is not supported if max_open_files != -1 or with table formats other than Block-based. **Test Plan:** Added tests. **Benchmark results:** Base: FIFO with max size: 100MB :: ``` svemuri@dev15905 ~/rocksdb (fifo-compaction) $ TEST_TMPDIR=/dev/shm ./db_bench --benchmarks=readwhilewriting --num=5000000 --threads=16 --compaction_style=2 --fifo_compaction_max_table_files_size_mb=100 readwhilewriting : 1.924 micros/op 519858 ops/sec; 13.6 MB/s (1176277 of 5000000 found) ``` With TTL (a low one for testing) :: ``` svemuri@dev15905 ~/rocksdb (fifo-compaction) $ TEST_TMPDIR=/dev/shm ./db_bench --benchmarks=readwhilewriting --num=5000000 --threads=16 --compaction_style=2 --fifo_compaction_max_table_files_size_mb=100 --fifo_compaction_ttl=20 readwhilewriting : 1.902 micros/op 525817 ops/sec; 13.7 MB/s (1185057 of 5000000 found) ``` Example Log lines: ``` 2017/06/26-15:17:24.609249 7fd5a45ff700 (Original Log Time 2017/06/26-15:17:24.609177) [db/compaction_picker.cc:1471] [default] FIFO compaction: picking file 40 with creation time 1498515423 for deletion 2017/06/26-15:17:24.609255 7fd5a45ff700 (Original Log Time 2017/06/26-15:17:24.609234) [db/db_impl_compaction_flush.cc:1541] [default] Deleted 1 files ... 2017/06/26-15:17:25.553185 7fd5a61a5800 [DEBUG] [db/db_impl_files.cc:309] [JOB 0] Delete /dev/shm/dbbench/000040.sst type=2 #40 -- OK 2017/06/26-15:17:25.553205 7fd5a61a5800 EVENT_LOG_v1 {"time_micros": 1498515445553199, "job": 0, "event": "table_file_deletion", "file_number": 40} ``` SST Files remaining in the dbbench dir, after db_bench execution completed: ``` svemuri@dev15905 ~/rocksdb (fifo-compaction) $ ls -l /dev/shm//dbbench/*.sst -rw-r--r--. 1 svemuri users 30749887 Jun 26 15:17 /dev/shm//dbbench/000042.sst -rw-r--r--. 1 svemuri users 30768779 Jun 26 15:17 /dev/shm//dbbench/000044.sst -rw-r--r--. 1 svemuri users 30757481 Jun 26 15:17 /dev/shm//dbbench/000046.sst ``` Closes https://github.com/facebook/rocksdb/pull/2480 Differential Revision: D5305116 Pulled By: sagar0 fbshipit-source-id: 3e5cfcf5dd07ed2211b5b37492eb235b45139174
-
由 Yi Wu 提交于
Summary: 1. The buckifier script assume each test "foo" comes with a .cc file of the same name (i.e. foo.cc). Update cassandra tests to follow this pattern so that the buckifier script can recognize them. 2. add blob_db_test Closes https://github.com/facebook/rocksdb/pull/2506 Differential Revision: D5331517 Pulled By: yiwu-arbug fbshipit-source-id: 86f3eba471fc621186ab44cbd073b6162cde8e57
-
由 Yi Wu 提交于
Summary: Allow numa >= 2.0.8 in buck TARGET file. Closes https://github.com/facebook/rocksdb/pull/2504 Differential Revision: D5330550 Pulled By: yiwu-arbug fbshipit-source-id: 8ffb6167b4ad913877eac16a20a91023b31f8d41
-
由 Siying Dong 提交于
Summary: Closes https://github.com/facebook/rocksdb/pull/2502 Differential Revision: D5326498 Pulled By: siying fbshipit-source-id: 2f0ac6dc6ca5ddb23cecf67a278c086e52646714
-
由 Yi Wu 提交于
Summary: update compatible test to include 5.5 and 5.6 branch. Closes https://github.com/facebook/rocksdb/pull/2501 Differential Revision: D5325220 Pulled By: yiwu-arbug fbshipit-source-id: 5f5271491e6dd2d7b2cf73a7142f38a571553bc4
-
- 27 6月, 2017 9 次提交
-
-
由 Siying Dong 提交于
Summary: A typo conversion fails Windows build. Fix it. Closes https://github.com/facebook/rocksdb/pull/2500 Differential Revision: D5325962 Pulled By: siying fbshipit-source-id: 2cefdafc9afbc85f856f403af7c876b622400630
-
由 Ewout Prangsma 提交于
Summary: This PR adds support for encrypting data stored by RocksDB when written to disk. It adds an `EncryptedEnv` override of the `Env` class with matching overrides for sequential&random access files. The encryption itself is done through a configurable `EncryptionProvider`. This class creates is asked to create `BlockAccessCipherStream` for a file. This is where the actual encryption/decryption is being done. Currently there is a Counter mode implementation of `BlockAccessCipherStream` with a `ROT13` block cipher (NOTE the `ROT13` is for demo purposes only!!). The Counter operation mode uses an initial counter & random initialization vector (IV). Both are created randomly for each file and stored in a 4K (default size) block that is prefixed to that file. The `EncryptedEnv` implementation is such that clients of the `Env` class do not see this prefix (nor data, nor in filesize). The largest part of the prefix block is also encrypted, and there is room left for implementation specific settings/values/keys in there. To test the encryption, the `DBTestBase` class has been extended to consider a new environment variable called `ENCRYPTED_ENV`. If set, the test will setup a encrypted instance of the `Env` class to use for all tests. Typically you would run it like this: ``` ENCRYPTED_ENV=1 make check_some ``` There is also an added test that checks that some data inserted into the database is or is not "visible" on disk. With `ENCRYPTED_ENV` active it must not find plain text strings, with `ENCRYPTED_ENV` unset, it must find the plain text strings. Closes https://github.com/facebook/rocksdb/pull/2424 Differential Revision: D5322178 Pulled By: sdwilsh fbshipit-source-id: 253b0a9c2c498cc98f580df7f2623cbf7678a27f
-
由 Siying Dong 提交于
Summary: Closes https://github.com/facebook/rocksdb/pull/2499 Differential Revision: D5324914 Pulled By: siying fbshipit-source-id: c69827d4dddafa81e651180633943a3380cdd5bb
-
由 Yi Wu 提交于
Summary: WriteBatchThreadedTest is failing, at least on Mac. The problem seems to be `wb` is getting GC before we finish write. Explicitly close it seems to fix it. Closes https://github.com/facebook/rocksdb/pull/2482 Differential Revision: D5307379 Pulled By: yiwu-arbug fbshipit-source-id: 8ff7f8170451078c941951f5aafae83afffb7933
-
由 Aaron Gao 提交于
Summary: https://github.com/facebook/rocksdb/pull/2380 introduces a regression by replacing __thread with ThreadLocalPtr. Revert the thread local implementation back. Closes https://github.com/facebook/rocksdb/pull/2485 Differential Revision: D5308050 Pulled By: lightmark fbshipit-source-id: 2676e9c22edf76e8133d3f4c50e2711e11a95480
-
由 Siying Dong 提交于
Summary: Closes https://github.com/facebook/rocksdb/pull/2454 Differential Revision: D5255320 Pulled By: siying fbshipit-source-id: 0080830fa8eb5da6de25e17ba68aee91018c7913
-
由 Andrew Kryczka 提交于
Summary: as titled Closes https://github.com/facebook/rocksdb/pull/2497 Differential Revision: D5322732 Pulled By: ajkr fbshipit-source-id: 35a648a7af737032949ed99f430f4fd865ac9e9c
-
由 Siying Dong 提交于
Summary: Force people to write something other than file name while returning status for IOError. Closes https://github.com/facebook/rocksdb/pull/2493 Differential Revision: D5321309 Pulled By: siying fbshipit-source-id: 38bcf6c19e80831cd3e300a047e975cbb131d822
-
由 Siying Dong 提交于
Summary: With a regression bug was introduced two years ago, by https://github.com/facebook/rocksdb/commit/6e9fbeb27c38329f33ae541302c44c8db8374f8c , we fail to check return status of fsync call. This can cause we miss the information from the file system and can potentially cause corrupted data which we could have been detected. Closes https://github.com/facebook/rocksdb/pull/2495 Reviewed By: ajkr Differential Revision: D5321949 Pulled By: siying fbshipit-source-id: c68117914bb40700198fc37d0e4c63163a8a1031
-
- 25 6月, 2017 2 次提交
-
-
由 Maysam Yabandeh 提交于
Summary: Closes https://github.com/facebook/rocksdb/pull/2494 Differential Revision: D5317902 Pulled By: maysamyabandeh fbshipit-source-id: 097330292180816b3d0c9f4cbbdb6f68f0180200
-
由 Maysam Yabandeh 提交于
Summary: Throughput: 46k tps in our sysbench settings (filling the details later) The idea is to have the simplest change that gives us a reasonable boost in 2PC throughput. Major design changes: 1. The WAL file internal buffer is not flushed after each write. Instead it is flushed before critical operations (WAL copy via fs) or when FlushWAL is called by MySQL. Flushing the WAL buffer is also protected via mutex_. 2. Use two sequence numbers: last seq, and last seq for write. Last seq is the last visible sequence number for reads. Last seq for write is the next sequence number that should be used to write to WAL/memtable. This allows to have a memtable write be in parallel to WAL writes. 3. BatchGroup is not used for writes. This means that we can have parallel writers which changes a major assumption in the code base. To accommodate for that i) allow only 1 WriteImpl that intends to write to memtable via mem_mutex_--which is fine since in 2PC almost all of the memtable writes come via group commit phase which is serial anyway, ii) make all the parts in the code base that assumed to be the only writer (via EnterUnbatched) to also acquire mem_mutex_, iii) stat updates are protected via a stat_mutex_. Note: the first commit has the approach figured out but is not clean. Submitting the PR anyway to get the early feedback on the approach. If we are ok with the approach I will go ahead with this updates: 0) Rebase with Yi's pipelining changes 1) Currently batching is disabled by default to make sure that it will be consistent with all unit tests. Will make this optional via a config. 2) A couple of unit tests are disabled. They need to be updated with the serial commit of 2PC taken into account. 3) Replacing BatchGroup with mem_mutex_ got a bit ugly as it requires releasing mutex_ beforehand (the same way EnterUnbatched does). This needs to be cleaned up. Closes https://github.com/facebook/rocksdb/pull/2345 Differential Revision: D5210732 Pulled By: maysamyabandeh fbshipit-source-id: 78653bd95a35cd1e831e555e0e57bdfd695355a4
-
- 24 6月, 2017 2 次提交
-
-
由 Maysam Yabandeh 提交于
Summary: We currently do not support partitioning filters if indexes are not partitioned. The patch makes sure that these two are consistent. Closes https://github.com/facebook/rocksdb/pull/2455 Differential Revision: D5275644 Pulled By: maysamyabandeh fbshipit-source-id: b61701ac8914c2206d06f5e33ff6f67b24406d1d
-
由 jsteemann 提交于
Summary: should be boolean, not uint64_t MSVC complains about it during compilation with error `include\rocksdb\advanced_options.h(77): warning C4800: 'uint64_t': forcing value to bool 'true' or 'false' (performance warning)` Closes https://github.com/facebook/rocksdb/pull/2487 Differential Revision: D5310685 Pulled By: siying fbshipit-source-id: 719a33b3dba4f711aa72e3f229013c188015dc86
-
- 23 6月, 2017 1 次提交
-
-
由 Andrew Kryczka 提交于
Summary: Some users want to prevent rocksdb from entering read-only mode in certain error cases. This diff gives them a callback, `OnBackgroundError`, that they can use to achieve it. - call `OnBackgroundError` every time we consider setting `bg_error_`. Use its result to assign `bg_error_` but not to change the function's return status. - classified calls using `BackgroundErrorReason` to give the callback some info about where the error happened - renamed `ParanoidCheck` to something more specific so we can provide a clear `BackgroundErrorReason` - unit tests for the most common cases: flush or compaction errors Closes https://github.com/facebook/rocksdb/pull/2477 Differential Revision: D5300190 Pulled By: ajkr fbshipit-source-id: a0ea4564249719b83428e3f4c6ca2c49e366e9b3
-