- 27 6月, 2017 1 次提交
-
-
由 Siying Dong 提交于
Summary: With a regression bug was introduced two years ago, by https://github.com/facebook/rocksdb/commit/6e9fbeb27c38329f33ae541302c44c8db8374f8c , we fail to check return status of fsync call. This can cause we miss the information from the file system and can potentially cause corrupted data which we could have been detected. Closes https://github.com/facebook/rocksdb/pull/2495 Reviewed By: ajkr Differential Revision: D5321949 Pulled By: siying fbshipit-source-id: c68117914bb40700198fc37d0e4c63163a8a1031
-
- 25 6月, 2017 2 次提交
-
-
由 Maysam Yabandeh 提交于
Summary: Closes https://github.com/facebook/rocksdb/pull/2494 Differential Revision: D5317902 Pulled By: maysamyabandeh fbshipit-source-id: 097330292180816b3d0c9f4cbbdb6f68f0180200
-
由 Maysam Yabandeh 提交于
Summary: Throughput: 46k tps in our sysbench settings (filling the details later) The idea is to have the simplest change that gives us a reasonable boost in 2PC throughput. Major design changes: 1. The WAL file internal buffer is not flushed after each write. Instead it is flushed before critical operations (WAL copy via fs) or when FlushWAL is called by MySQL. Flushing the WAL buffer is also protected via mutex_. 2. Use two sequence numbers: last seq, and last seq for write. Last seq is the last visible sequence number for reads. Last seq for write is the next sequence number that should be used to write to WAL/memtable. This allows to have a memtable write be in parallel to WAL writes. 3. BatchGroup is not used for writes. This means that we can have parallel writers which changes a major assumption in the code base. To accommodate for that i) allow only 1 WriteImpl that intends to write to memtable via mem_mutex_--which is fine since in 2PC almost all of the memtable writes come via group commit phase which is serial anyway, ii) make all the parts in the code base that assumed to be the only writer (via EnterUnbatched) to also acquire mem_mutex_, iii) stat updates are protected via a stat_mutex_. Note: the first commit has the approach figured out but is not clean. Submitting the PR anyway to get the early feedback on the approach. If we are ok with the approach I will go ahead with this updates: 0) Rebase with Yi's pipelining changes 1) Currently batching is disabled by default to make sure that it will be consistent with all unit tests. Will make this optional via a config. 2) A couple of unit tests are disabled. They need to be updated with the serial commit of 2PC taken into account. 3) Replacing BatchGroup with mem_mutex_ got a bit ugly as it requires releasing mutex_ beforehand (the same way EnterUnbatched does). This needs to be cleaned up. Closes https://github.com/facebook/rocksdb/pull/2345 Differential Revision: D5210732 Pulled By: maysamyabandeh fbshipit-source-id: 78653bd95a35cd1e831e555e0e57bdfd695355a4
-
- 24 6月, 2017 2 次提交
-
-
由 Maysam Yabandeh 提交于
Summary: We currently do not support partitioning filters if indexes are not partitioned. The patch makes sure that these two are consistent. Closes https://github.com/facebook/rocksdb/pull/2455 Differential Revision: D5275644 Pulled By: maysamyabandeh fbshipit-source-id: b61701ac8914c2206d06f5e33ff6f67b24406d1d
-
由 jsteemann 提交于
Summary: should be boolean, not uint64_t MSVC complains about it during compilation with error `include\rocksdb\advanced_options.h(77): warning C4800: 'uint64_t': forcing value to bool 'true' or 'false' (performance warning)` Closes https://github.com/facebook/rocksdb/pull/2487 Differential Revision: D5310685 Pulled By: siying fbshipit-source-id: 719a33b3dba4f711aa72e3f229013c188015dc86
-
- 23 6月, 2017 3 次提交
-
-
由 Andrew Kryczka 提交于
Summary: Some users want to prevent rocksdb from entering read-only mode in certain error cases. This diff gives them a callback, `OnBackgroundError`, that they can use to achieve it. - call `OnBackgroundError` every time we consider setting `bg_error_`. Use its result to assign `bg_error_` but not to change the function's return status. - classified calls using `BackgroundErrorReason` to give the callback some info about where the error happened - renamed `ParanoidCheck` to something more specific so we can provide a clear `BackgroundErrorReason` - unit tests for the most common cases: flush or compaction errors Closes https://github.com/facebook/rocksdb/pull/2477 Differential Revision: D5300190 Pulled By: ajkr fbshipit-source-id: a0ea4564249719b83428e3f4c6ca2c49e366e9b3
-
由 Siying Dong 提交于
Summary: With c7004840, it's safe to open a DB with different prefix extractor. So it's safe to skip prefix extractor check. Closes https://github.com/facebook/rocksdb/pull/2474 Differential Revision: D5294700 Pulled By: siying fbshipit-source-id: eeb500da795eecb29b8c9c56a14cfd4afda12ecc
-
由 Siying Dong 提交于
Summary: CreateColumnFamily() releases DB mutex after adding column family to the set and install super version (to write option file), so if users call GetAggregatedIntProperty() in the middle, then super version will be null and the process will crash. Fix it by skipping those column families without super version installed. Maybe we should also fix the problem of releasing the lock when reading option file, but it is more risky. so I'm doing a quick and safer fix and we can investigate it later. Closes https://github.com/facebook/rocksdb/pull/2475 Differential Revision: D5298053 Pulled By: siying fbshipit-source-id: 4b3c8f91c60400b163fcc6cda8a0c77723be0ef6
-
- 22 6月, 2017 1 次提交
-
-
由 Siying Dong 提交于
Summary: Even if hard limit hits, flushing more memtable may not help cap the memory usage if already more than half data is scheduled for flush. Not triggering flush instead. Closes https://github.com/facebook/rocksdb/pull/2469 Differential Revision: D5284249 Pulled By: siying fbshipit-source-id: 8ab7ba1aba56a634dbe72b318fcab2093063972e
-
- 21 6月, 2017 3 次提交
-
-
由 Andrew Kryczka 提交于
Summary: DBImpl's instance variables should only be accessed with mutex held. I moved an assert later to uphold this rule. DBTest.LastWriteBufferDelay test was sporadically failing TSAN because it tried to flush around the same time the db was destroyed, so the variable was accessed simultaneously by two threads. Closes https://github.com/facebook/rocksdb/pull/2471 Differential Revision: D5286857 Pulled By: ajkr fbshipit-source-id: 435abd84efa601f667c254e320b0bb5a434b971f
-
由 Andrew Kryczka 提交于
Summary: When we create a column based on the `string::c_str()`, we need to make sure that char array doesn't get deleted when calls to `string::append()` cause the string to expand. Closes https://github.com/facebook/rocksdb/pull/2470 Differential Revision: D5285049 Pulled By: ajkr fbshipit-source-id: f918dd426ff3c024e7a293dcb10448f10b6c98e8
-
由 Dmitri Smirnov 提交于
Summary: Make default impl return NoSupported so the db_blob tests exist in a meaningful manner. Replace std::thread to port::Thread Closes https://github.com/facebook/rocksdb/pull/2465 Differential Revision: D5275563 Pulled By: yiwu-arbug fbshipit-source-id: cedf1a18a2c05e20d768c1308b3f3224dbd70ab6
-
- 20 6月, 2017 1 次提交
-
-
由 zhangjinpeng1987 提交于
Summary: Coredump will be triggered when ingest external sst file after delete range. ref https://github.com/facebook/rocksdb/issues/2398 Closes https://github.com/facebook/rocksdb/pull/2463 Differential Revision: D5275599 Pulled By: ajkr fbshipit-source-id: 0828dbc062ea8c74e913877cd63494fd3478a30d
-
- 19 6月, 2017 1 次提交
-
-
由 Andrew Kryczka 提交于
Summary: it's confusing to implementors of prefix extractor to implement an unused function Closes https://github.com/facebook/rocksdb/pull/2460 Differential Revision: D5267408 Pulled By: ajkr fbshipit-source-id: 2f1fe3131efc978f6098ae7a80e52bc7a0b13571
-
- 17 6月, 2017 1 次提交
-
-
由 Chen Shen 提交于
Summary: This PR implements the MergeOperator for Cassandra Row Values. Closes https://github.com/facebook/rocksdb/pull/2289 Differential Revision: D5055464 Pulled By: scv119 fbshipit-source-id: 45f276ef8cbc4704279202f6a20c64889bc1adef
-
- 16 6月, 2017 2 次提交
-
-
由 Maysam Yabandeh 提交于
Summary: This would simplify db_bench_tool.cc Closes https://github.com/facebook/rocksdb/pull/2457 Differential Revision: D5259035 Pulled By: maysamyabandeh fbshipit-source-id: 0a9c3abda624070fe2650200b885ad7e1c60182c
-
由 Maysam Yabandeh 提交于
Summary: Closes https://github.com/facebook/rocksdb/pull/2456 Differential Revision: D5259083 Pulled By: maysamyabandeh fbshipit-source-id: 1ed1746da7a8baadf4772d023d927c6c4e6b112a
-
- 15 6月, 2017 4 次提交
-
-
由 Ben Torfs 提交于
Summary: Closes https://github.com/facebook/rocksdb/pull/2209 Differential Revision: D5251951 Pulled By: sagar0 fbshipit-source-id: 03a73d025a7b4a322bb8d8d86f5d249fcd7dd00e
-
由 Sagar Vemuri 提交于
Summary: RateLimiterTest.Rate test has been failing continuously since many days on travis in Mac OSX PLATFORM_DEPENDENT test suite. Check https://travis-ci.org/facebook/rocksdb/pull_requests. Disabling this test for now, so that we can investigate more in depth. Closes https://github.com/facebook/rocksdb/pull/2451 Differential Revision: D5250147 Pulled By: sagar0 fbshipit-source-id: d58476a3c2792d20e875754d1516c4bc7174e86c
-
由 Yi Wu 提交于
Summary: `CompressBlock()` will return the uncompressed slice (i.e. `Slice(value_unc)`) if compression ratio is not good enough. This is undesired. We need to always assign the compressed slice to `value`. Closes https://github.com/facebook/rocksdb/pull/2447 Differential Revision: D5244682 Pulled By: yiwu-arbug fbshipit-source-id: 6989dd8852c9622822ba9acec9beea02007dff09
-
由 Yi Wu 提交于
Summary: I'm trying to improve unit test of blob db. I'm rewriting blob db test. In this patch: * Rewrite tests of basic put/write/delete operations. * Add disable_background_tasks to BlobDBOptionsImpl to allow me not running any background job for basic unit tests. * Move DestroyBlobDB out from BlobDBImpl to be a standalone function. * Remove all garbage collection related tests. Will rewrite them in following patch. * Disabled compression test since it is failing. Will fix in a followup patch. Closes https://github.com/facebook/rocksdb/pull/2446 Differential Revision: D5243306 Pulled By: yiwu-arbug fbshipit-source-id: 157c71ad3b699307cb88baa3830e9b6e74f8e939
-
- 14 6月, 2017 6 次提交
-
-
由 Sagar Vemuri 提交于
Summary: Added a flag, `ignore_unknown_options`, to skip unknown options when loading an options file (using `LoadLatestOptions`/`LoadOptionsFromFile`) or while verifying options (using `CheckOptionsCompatibility`). This will help in downgrading the db to an older version. Also added `--ignore_unknown_options` flag to ldb **Example Use case:** In MyRocks, if copying from newer version to older version, it is often impossible to start because of new RocksDB options that don't exist in older version, even though data format is compatible. MyRocks uses these load and verify functions in [ha_rocksdb.cc::check_rocksdb_options_compatibility](https://github.com/facebook/mysql-5.6/blob/e004fd9f416821d043ccc8ad4a345c33ac9953f0/storage/rocksdb/ha_rocksdb.cc#L3348-L3401). **Test Plan:** Updated the unit tests. `make check` ldb: $ ./ldb --db=/tmp/test_db --create_if_missing put a1 b1 OK Now edit /tmp/test_db/<OPTIONS-file> and add an unknown option. Try loading the options now, and it fails: $ ./ldb --db=/tmp/test_db --try_load_options get a1 Failed: Invalid argument: Unrecognized option DBOptions:: abcd Passes with the new --ignore_unknown_options flag $ ./ldb --db=/tmp/test_db --try_load_options --ignore_unknown_options get a1 b1 Closes https://github.com/facebook/rocksdb/pull/2423 Differential Revision: D5212091 Pulled By: sagar0 fbshipit-source-id: 2ec17636feb47dc0351b53a77e5f15ef7cbf2ca7
-
由 hyunwoo 提交于
Summary: fixed typo Closes https://github.com/facebook/rocksdb/pull/2430 Differential Revision: D5242471 Pulled By: IslamAbdelRahman fbshipit-source-id: 832eb3a4c70221444ccd2ae63217823fec56c748
-
由 haoxiang 提交于
Summary: fixed a typo in util/dynamic_bloom.h Closes https://github.com/facebook/rocksdb/pull/2442 Differential Revision: D5242397 Pulled By: IslamAbdelRahman fbshipit-source-id: c47fd18cc79afff6b022201a0410c0cd47626576
-
由 Andrew Kryczka 提交于
Summary: Allow users to rate limit background work based on read bytes, written bytes, or sum of read and written bytes. Support these by changing the RateLimiter API, so no additional options were needed. Closes https://github.com/facebook/rocksdb/pull/2433 Differential Revision: D5216946 Pulled By: ajkr fbshipit-source-id: aec57a8357dbb4bfde2003261094d786d94f724e
-
由 Yi Wu 提交于
Summary: At the beginning of write batch write, grab the latest sequence from base db and assume sequence number will increment by 1 for each put and delete, and write the exact sequence number with each put. This is assuming we are the only writer to increment sequence number (no external file ingestion, etc) and there should be no holes in the sequence number. Also having some minor naming changes. Closes https://github.com/facebook/rocksdb/pull/2402 Differential Revision: D5176134 Pulled By: yiwu-arbug fbshipit-source-id: cb4712ee44478d5a2e5951213a10b72f08fe8c88
-
由 Maysam Yabandeh 提交于
Summary: When Partitioning index/filter is enabled the user might need to check the index block size as well as the top-level index size via sst_dump. This patch records i) number of partitions, ii) top-level index size and make it accessible through sst_dump. The number of partitions for filters is the same as that of indexes. The top-level index for filters has a similar size to top-level index for indexes, so it is not repeated. Closes https://github.com/facebook/rocksdb/pull/2437 Differential Revision: D5224225 Pulled By: maysamyabandeh fbshipit-source-id: 5324598c75793523aef1bb7ee225a5475e95a9cb
-
- 13 6月, 2017 7 次提交
-
-
由 Siying Dong 提交于
Summary: 5582123d broken CLANG release build because of an unexpected change. Fix it. Closes https://github.com/facebook/rocksdb/pull/2443 Differential Revision: D5236297 Pulled By: siying fbshipit-source-id: 1b410adf13ded149c53e8235e9ea9f3130fb5403
-
由 Siying Dong 提交于
Summary: Currently for direct I/O, the large maximum buffer is always allocated. This will be wasteful if users flush the data in much smaller chunks. This diff fix this by changing the behavior of incremental buffer works. When we enlarge buffer, we try to copy the existing data in the buffer to the enlarged buffer, rather than flush the buffer first. This can make sure that no extra I/O is introduced because of buffer enlargement. Closes https://github.com/facebook/rocksdb/pull/2403 Differential Revision: D5178403 Pulled By: siying fbshipit-source-id: a8fe1e7304bdb8cab2973340022fe80ff83449fd
-
由 Anirban Rahut 提交于
Summary: initialize 2 additional fields tm_gmtoff and tm_zone, otherwise under strict warnings for initialization, we get errors in myrocks. Closes https://github.com/facebook/rocksdb/pull/2439 Differential Revision: D5229013 Pulled By: yiwu-arbug fbshipit-source-id: 9fc1615a1919656f36064791706ed41e10e9db84
-
由 Islam AbdelRahman 提交于
Summary: Update DeleteScheduler to delete files immediately if trash directory is >= 25% of DB size Closes https://github.com/facebook/rocksdb/pull/2436 Differential Revision: D5230384 Pulled By: IslamAbdelRahman fbshipit-source-id: 5cbda8ac536a3cc72c774641621edc02c8202482
-
由 Orgad Shaneh 提交于
Summary: cc tamird sagar0 Closes https://github.com/facebook/rocksdb/pull/2325 Differential Revision: D5098302 Pulled By: sagar0 fbshipit-source-id: 297c5506b5d9b2ed1d7719c8caf0b96cffe503b8
-
由 Sagar Vemuri 提交于
Summary: This diff syncs the Histogram Types in the Java API with the ones in C++ API (`statistics.h`), and brings it up-to-date. I also found that the enum ordering between Java and C++ has gotten out-of-sync, a few years back, with the addition of `SUBCOMPACTION_SETUP_TIME`. So updated the order as well. `READ_NUM_MERGE_OPERANDS` added in #2373 is needed for Cassandra-on-RocksDB work. Closes https://github.com/facebook/rocksdb/pull/2429 Differential Revision: D5215623 Pulled By: sagar0 fbshipit-source-id: bd136698c48197e53693275eb52acc9198ee5a4e
-
由 Andrew Kryczka 提交于
Summary: - mention range deletion + file ingestion - move post-5.6 stuff into new section Closes https://github.com/facebook/rocksdb/pull/2440 Differential Revision: D5229910 Pulled By: ajkr fbshipit-source-id: 1facfe41993fa1f3b1f6fa7dc77d2b11aa2b317a
-
- 12 6月, 2017 2 次提交
-
-
由 Siying Dong 提交于
Summary: We estimate number of reads per SST files, by updating the counter per file in sampled read requests. This information can later be used to trigger compactions to improve read performacne. Closes https://github.com/facebook/rocksdb/pull/2417 Differential Revision: D5193528 Pulled By: siying fbshipit-source-id: b4241c5ad0eaf444b61afb53f8e6290d9f5da2df
-
由 Siying Dong 提交于
Summary: Closes https://github.com/facebook/rocksdb/pull/2419 Differential Revision: D5193976 Pulled By: siying fbshipit-source-id: 62d115edee6043237e9d6ad3c2a05481e162c9eb
-
- 10 6月, 2017 1 次提交
-
-
由 Aaron Gao 提交于
Summary: update bug fixed. Closes https://github.com/facebook/rocksdb/pull/2434 Differential Revision: D5218601 Pulled By: lightmark fbshipit-source-id: 1f86b2c93345673612381081537d464e7d12e434
-
- 09 6月, 2017 3 次提交
-
-
由 Mike Kolupaev 提交于
Summary: We had a crash in this code: `fstat()` failed; `file_stats` contained garbage, in particular `file_stats.st_blksize == 6`; the expression `file_stats.st_blocks / (file_stats.st_blksize / 512)` divided by zero. Closes https://github.com/facebook/rocksdb/pull/2420 Differential Revision: D5216110 Pulled By: al13n321 fbshipit-source-id: 6d8fc5e7c4f98c1139e68c7829ebdbac68b0fce0
-
由 Yi Wu 提交于
Summary: Mingw is complaining about uninitialized variable in mock_env.cc. e.g. https://travis-ci.org/facebook/rocksdb/jobs/240132276 The fix is to initialize the variable. Closes https://github.com/facebook/rocksdb/pull/2428 Differential Revision: D5211306 Pulled By: yiwu-arbug fbshipit-source-id: ee02bf0327dcea8590a2aa087f0176fecaf8621c
-
由 Sagar Vemuri 提交于
Summary: Adding SSTFileWriter's newly introduced put, merge and delete apis to the Java api. The C++ APIs were first introduced in #2361. Add is deprecated in favor of Put. Merge is especially needed to support streaming for Cassandra-on-RocksDB work in https://issues.apache.org/jira/browse/CASSANDRA-13476. Closes https://github.com/facebook/rocksdb/pull/2392 Differential Revision: D5165091 Pulled By: sagar0 fbshipit-source-id: 6f0ad396a7cbd2e27ca63e702584784dd72acaab
-