- 09 8月, 2017 2 次提交
-
-
由 Andrew Kryczka 提交于
Summary: Closes https://github.com/facebook/rocksdb/pull/2702 Differential Revision: D5585389 Pulled By: ajkr fbshipit-source-id: cb54041eb481d0d759c440f82a8a2c5b34534173
-
由 Chang Liu 提交于
Summary: We should search wal_dir in Repairer::FindFiles function, and avoid use LogFileNmae(dbname, number) to get WAL file's name, which will get a wrong WAL filename. as following: ``` [WARN] [/home/liuchang/Workspace/rocksdb/db/repair.cc:310] Log #3: ignoring conversion error: IO error: While opening a file for sequentially reading: /tmp/rocksdbtest-1000/repair_test/000003.log: No such file or directory ``` I have added a new test case to repair_test.cc, which try to repair db with all WAL options. Signed-off-by: NChang Liu <liuchang0812@gmail.com> Closes https://github.com/facebook/rocksdb/pull/2692 Differential Revision: D5575888 Pulled By: ajkr fbshipit-source-id: 5b93e9f85cddc01663ccecd87631fa723ac466a3
-
- 08 8月, 2017 2 次提交
-
-
由 James Page 提交于
Summary: Markup i386 code paths resolving compilation failure under i386 with gcc-7. Signed-off-by: NJames Page <james.page@ubuntu.com> Closes https://github.com/facebook/rocksdb/pull/2700 Differential Revision: D5583047 Pulled By: maysamyabandeh fbshipit-source-id: fe31bcfeaf7cd2d3f51b55f5ae0b3b0cb3788fbc
-
由 Maysam Yabandeh 提交于
Summary: This patch splits Commit and Prepare into lock-related logic and db-write-related logic. It moves lock-related logic to PessimisticTransaction to be reused by all children classes and movies the existing impl of db-write-related to PrepareInternal, CommitSingleInternal, and CommitInternal in WriteCommittedTxnImpl. Closes https://github.com/facebook/rocksdb/pull/2691 Differential Revision: D5569464 Pulled By: maysamyabandeh fbshipit-source-id: d1b8698e69801a4126c7bc211745d05c636f5325
-
- 07 8月, 2017 2 次提交
-
-
由 Maysam Yabandeh 提交于
Summary: Closes https://github.com/facebook/rocksdb/pull/2697 Differential Revision: D5573894 Pulled By: maysamyabandeh fbshipit-source-id: 8fc03ea8ea6f3f3bc0f68b64cf90243a70562dc4
-
由 janlzlabs 提交于
Summary: I'd like to propose adding my company as a RocksDB user Closes https://github.com/facebook/rocksdb/pull/2694 Differential Revision: D5572113 Pulled By: ajkr fbshipit-source-id: 646143b955e3efddee56691cce912d7badaa6e8b
-
- 06 8月, 2017 1 次提交
-
-
由 Maysam Yabandeh 提交于
Summary: This opens space for the new implementations of TransactionDBImpl such as WritePreparedTxnDBImpl that has a different policy of how to write to DB. Closes https://github.com/facebook/rocksdb/pull/2689 Differential Revision: D5568918 Pulled By: maysamyabandeh fbshipit-source-id: f7eac866e175daf3793ae79da108f65cc7dc7b25
-
- 05 8月, 2017 3 次提交
-
-
由 Sagar Vemuri 提交于
Summary: In the condition: ``` if (range_del_agg != nullptr && range_del_agg->ShouldDelete( iter->key(), RangeDelAggregator::RangePositioningMode::kForwardTraversal) && filter != CompactionFilter::Decision::kRemoveAndSkipUntil) { ... } ``` it could be possible that all the work done in `range_del_agg->ShouldDelete` is wasted due to not having the right `filter` value later on. Instead, check `filter` value before even calling `range_del_agg->ShouldDelete`, which is a much more involved function. Closes https://github.com/facebook/rocksdb/pull/2690 Differential Revision: D5568931 Pulled By: sagar0 fbshipit-source-id: 17512d52360425c7ae9de7675383f5d7bc3dad58
-
由 Yi Wu 提交于
Summary: The FsyncFiles background job call Fsync() periodically for blob files. However it can access WritableFileWriter concurrently with a Put() or Write(). And WritableFileWriter does not support concurrent access. It will lead to WritableFileWriter buffer being flush with same content twice, and blob file end up corrupted. Fixing by simply let FsyncFiles hold write_mutex_. Closes https://github.com/facebook/rocksdb/pull/2685 Differential Revision: D5561908 Pulled By: yiwu-arbug fbshipit-source-id: f0bb5bcab0e05694e053b8c49eab43640721e872
-
由 Maysam Yabandeh 提交于
Summary: fixes #2555 Closes https://github.com/facebook/rocksdb/pull/2684 Differential Revision: D5560527 Pulled By: maysamyabandeh fbshipit-source-id: 6e1d874ae0b4e699a77203d9d52d0bb8f59013b0
-
- 04 8月, 2017 8 次提交
-
-
由 Andrew Kryczka 提交于
Summary: The background thread pools' sizes weren't easily configurable by `max_background_compactions` and `max_background_flushes` in multi-instance setups. Introduced separate arguments for their sizes. Closes https://github.com/facebook/rocksdb/pull/2680 Differential Revision: D5550675 Pulled By: ajkr fbshipit-source-id: bab5f0a7bc5db63bb084d0c10facbe437096367d
-
由 Cholerae Hu 提交于
Summary: maysamyabandeh IslamAbdelRahman PTAL Fix https://github.com/facebook/rocksdb/issues/2672Signed-off-by: NCholerae Hu <huyingqian@pingcap.com> Closes https://github.com/facebook/rocksdb/pull/2681 Differential Revision: D5561515 Pulled By: ajkr fbshipit-source-id: 676187802ebd8a87a6c051bb565818a1bf89d0a9
-
由 Yi Wu 提交于
Summary: The current blob db implementation use mix of int32_t, uint32_t and uint64_t for TTL and expiration. Update all timestamps to uint64_t for consistency. Closes https://github.com/facebook/rocksdb/pull/2683 Differential Revision: D5557103 Pulled By: yiwu-arbug fbshipit-source-id: e4eab2691629a755e614e8cf1eed9c3a681d0c42
-
由 Alan Somers 提交于
Summary: "/bin/bash" is a Linuxism. "/usr/bin/env bash" is portable. Closes https://github.com/facebook/rocksdb/pull/2646 Differential Revision: D5556259 Pulled By: ajkr fbshipit-source-id: cbffd38ecdbfffb2438969ec007ab345ed893ccb
-
由 Andrew Kryczka 提交于
Summary: When we had a single thread pool for compactions, a thread could be busy for a long time (minutes) executing a compaction involving the bottom level. In multi-instance setups, the entire thread pool could be consumed by such bottom-level compactions. Then, top-level compactions (e.g., a few L0 files) would be blocked for a long time ("head-of-line blocking"). Such top-level compactions are critical to prevent compaction stalls as they can quickly reduce number of L0 files / sorted runs. This diff introduces a bottom-priority queue for universal compactions including the bottom level. This alleviates the head-of-line blocking situation for fast, top-level compactions. - Added `Env::Priority::BOTTOM` thread pool. This feature is only enabled if user explicitly configures it to have a positive number of threads. - Changed `ThreadPoolImpl`'s default thread limit from one to zero. This change is invisible to users as we call `IncBackgroundThreadsIfNeeded` on the low-pri/high-pri pools during `DB::Open` with values of at least one. It is necessary, though, for bottom-pri to start with zero threads so the feature is disabled by default. - Separated `ManualCompaction` into two parts in `PrepickedCompaction`. `PrepickedCompaction` is used for any compaction that's picked outside of its execution thread, either manual or automatic. - Forward universal compactions involving last level to the bottom pool (worker thread's entry point is `BGWorkBottomCompaction`). - Track `bg_bottom_compaction_scheduled_` so we can wait for bottom-level compactions to finish. We don't count them against the background jobs limits. So users of this feature will get an extra compaction for free. Closes https://github.com/facebook/rocksdb/pull/2580 Differential Revision: D5422916 Pulled By: ajkr fbshipit-source-id: a74bd11f1ea4933df3739b16808bb21fcd512333
-
由 Yi Wu 提交于
Summary: I'm going with brute-force solution, just letting Put() and Write() holding a mutex before writing. May improve concurrent writing with finer granularity locking later. Closes https://github.com/facebook/rocksdb/pull/2682 Differential Revision: D5552690 Pulled By: yiwu-arbug fbshipit-source-id: 039abd675b5d274a7af6428198d1733cafecef4c
-
由 Yi Wu 提交于
Summary: Fix the bug where if blob db garbage collection revmoe keys with newer version. It shouldn't delete the key from base db when sequence number in base db is not equal to the one in blob log. Closes https://github.com/facebook/rocksdb/pull/2678 Differential Revision: D5549752 Pulled By: yiwu-arbug fbshipit-source-id: abb8649260963b5c389748023970fd746279d227
-
由 Maysam Yabandeh 提交于
Summary: https://github.com/facebook/rocksdb/issues/2559 reports an overflow in AwaitState. nbronson has debugged the issue and presented the fix, which is applied to this patch. Moreover this patch adds more comments to clarify the logic in AwaitState. I tried with both 16 and 64 threads on update benchmark. The fix lowers cpu usage by 1.6 but also lowers the throughput by 1.6 and 2% respectively. Apparently the bug had favored using the spinning more often. Benchmarks: TEST_TMPDIR=/dev/shm/tmpdb time ./db_bench --benchmarks="fillrandom" --threads=16 --num=2000000 TEST_TMPDIR=/dev/shm/tmpdb time ./db_bench --use_existing_db=1 --benchmarks="updaterandom[X3]" --threads=16 --num=2000000 TEST_TMPDIR=/dev/shm/tmpdb time ./db_bench --use_existing_db=1 --benchmarks="updaterandom[X3]" --threads=64 --num=200000 Results $ cat update-16t-bug.txt | tail -4 updaterandom [AVG 3 runs] : 234117 ops/sec; 51.8 MB/sec updaterandom [MEDIAN 3 runs] : 233581 ops/sec; 51.7 MB/sec 3896.42user 1539.12system 6:50.61elapsed 1323%CPU (0avgtext+0avgdata 331308maxresident)k 0inputs+0outputs (0major+1281001minor)pagefaults 0swaps $ cat update-16t-fixed.txt | tail -4 updaterandom [AVG 3 runs] : 230364 ops/sec; 51.0 MB/sec updaterandom [MEDIAN 3 runs] : 226169 ops/sec; 50.0 MB/sec 3865.46user 1568.32system 6:57.63elapsed 1301%CPU (0avgtext+0avgdata 315012maxresident)k 0inputs+0outputs (0major+1342568minor)pagefaults 0swaps $ cat update-64t-bug.txt | tail -4 updaterandom [AVG 3 runs] : 261878 ops/sec; 57.9 MB/sec updaterandom [MEDIAN 3 runs] : 262859 ops/sec; 58.2 MB/sec 926.27user 578.06system 2:27.46elapsed 1020%CPU (0avgtext+0avgdata 475480maxresident)k 0inputs+0outputs (0major+1058728minor)pagefaults 0swaps $ cat update-64t-fixed.txt | tail -4 updaterandom [AVG 3 runs] : 256699 ops/sec; 56.8 MB/sec updaterandom [MEDIAN 3 runs] : 256380 ops/sec; 56.7 MB/sec 933.47user 575.37system 2:30.41elapsed 1003%CPU (0avgtext+0avgdata 482340maxresident)k 0inputs+0outputs (0major+1078557minor)pagefaults 0swaps Closes https://github.com/facebook/rocksdb/pull/2679 Differential Revision: D5553732 Pulled By: maysamyabandeh fbshipit-source-id: 98b72dc3a8e0f22ea29d4f7c7790af10c369c5bb
-
- 03 8月, 2017 2 次提交
-
-
由 Maysam Yabandeh 提交于
Summary: This patch refactors TransactionImpl by separating the logic for pessimistic concurrency control from the implementation of how to write the data to rocksdb. The existing implementation is named WriteCommittedTxnImpl as it writes committed data to the db. A template named WritePreparedTxnImpl is also added which will be later completed to provide a an alternative implementation. Closes https://github.com/facebook/rocksdb/pull/2676 Differential Revision: D5549998 Pulled By: maysamyabandeh fbshipit-source-id: 16298e86b43ca4849324c1f35c731913c6d17bec
-
由 Andrew Kryczka 提交于
Summary: Move an option necessary for running db_bench on multiple CFs into the general initialization area, so it works with both flag-based init and OPTIONS-based init. Closes https://github.com/facebook/rocksdb/pull/2675 Differential Revision: D5541378 Pulled By: ajkr fbshipit-source-id: 169926cb4ae95c17974f744faf7cc794d41e5c0a
-
- 02 8月, 2017 2 次提交
-
-
由 Sagar Vemuri 提交于
Summary: I observed while doing a `make jtest` that the java sample was broken, due to the changes in #2551 . Closes https://github.com/facebook/rocksdb/pull/2674 Differential Revision: D5539807 Pulled By: sagar0 fbshipit-source-id: 2c7e9d84778099dfa1c611996b444efe3c9fd466
-
由 Yi Wu 提交于
Summary: * Dump blob db options to info log * Remove BlobDBOptionsImpl to disallow dynamic cast *BlobDBOptions into *BlobDBOptionsImpl. Move options there to be constants or into BlobDBOptions. The dynamic cast is broken after #2645 * Change some of the default options * Remove blob_db_options.min_blob_size, which is unimplemented. Will implement it soon. Closes https://github.com/facebook/rocksdb/pull/2671 Differential Revision: D5529912 Pulled By: yiwu-arbug fbshipit-source-id: dcd58ca981db5bcc7f123b65a0d6f6ae0dc703c7
-
- 01 8月, 2017 3 次提交
-
-
由 奏之章 提交于
Summary: this value ``` Compaction::is_trivial_move_ ``` uninitialized . under universal compaction , we enable ``` CompactionOptionsUniversal::allow_trivial_move ``` , https://github.com/facebook/rocksdb/blob/9b11d4345a0f01fc3de756e01460bf1b0446f326/db/compaction.cc#L245 here is a disastrous bug , some sst trivial move to target level without overlap check ... THEN , DATABASE DAMAGED , WE GOT A LEVEL WITH OVERLAP ! Closes https://github.com/facebook/rocksdb/pull/2634 Differential Revision: D5530722 Pulled By: siying fbshipit-source-id: 425ab55bca5967110377d634258360bcf88c200e
-
由 Andrew Kryczka 提交于
Summary: we were passing `record_read_stats` (a bool) as the `hist_type` argument, which meant we were updating either `rocksdb.db.get.micros` (`hist_type == 0`) or `rocksdb.db.write.micros` (`hist_type == 1`) with wrong data. Closes https://github.com/facebook/rocksdb/pull/2666 Differential Revision: D5520384 Pulled By: ajkr fbshipit-source-id: 2f7c956aec32f8b58c5c18845ac478e0230c9516
-
由 Siying Dong 提交于
Summary: Closes https://github.com/facebook/rocksdb/pull/2670 Differential Revision: D5529018 Pulled By: siying fbshipit-source-id: 10a378933d509035d2dbe502247dd85fcea09789
-
- 29 7月, 2017 6 次提交
-
-
由 Siying Dong 提交于
Summary: Replace dynamic_cast<> so that users can choose to build with RTTI off, so that they can save several bytes per object, and get tiny more memory available. Some nontrivial changes: 1. Add Comparator::GetRootComparator() to get around the internal comparator hack 2. Add the two experiemental functions to DB 3. Add TableFactory::GetOptionString() to avoid unnecessary casting to get the option string 4. Since 3 is done, move the parsing option functions for table factory to table factory files too, to be symmetric. Closes https://github.com/facebook/rocksdb/pull/2645 Differential Revision: D5502723 Pulled By: siying fbshipit-source-id: fd13cec5601cf68a554d87bfcf056f2ffa5fbf7c
-
由 Mike Kolupaev 提交于
Summary: This fixes OOMs that we (logdevice) are currently having in production. SkipListRep constructor does a couple small allocations from ConcurrentArena (see InlineSkipList constructor). ConcurrentArena would sometimes allocate an entire block for that, which is a few megabytes (we use Options::arena_block_size = 4 MB). So an empty memtable can take take 4 MB of memory. We have ~40k column families (spread across 15 DB instances), so 4 MB per empty memtable easily OOMs a machine for us. This PR makes ConcurrentArena always allocate from Arena's inline block when possible. So as long as InlineSkipList's initial allocations are below 2 KB there would be no blocks allocated for empty memtables. Closes https://github.com/facebook/rocksdb/pull/2569 Differential Revision: D5404029 Pulled By: al13n321 fbshipit-source-id: 568ec22a3fd1a485c06123f6b2dfc5e9ef67cd23
-
由 Sagar Vemuri 提交于
Summary: - FIFOCompactionWithTTLTest was flaky when run in parallel earlier, and hence it was disabled. Fixed it now. - Also, faking sleep now instead of really sleeping to make tests more realistic by using TTLs like 1 hour and 1 day. Closes https://github.com/facebook/rocksdb/pull/2650 Differential Revision: D5506038 Pulled By: sagar0 fbshipit-source-id: deb429a527f045e3e2c5138b547c3e8ac8586aa2
-
由 Yi Wu 提交于
Summary: Move blob_db/ttl_extractor.h into blob_db/blob_db.h Also exclude TTLExtractor from LITE build. Closes https://github.com/facebook/rocksdb/pull/2665 Differential Revision: D5520009 Pulled By: yiwu-arbug fbshipit-source-id: 4813dcc272c7cc4bf2cdac285256d9a17d78c7b7
-
由 Sagar Vemuri 提交于
Summary: I might have missed these while doing some recent cassandra code reviews. Closes https://github.com/facebook/rocksdb/pull/2663 Differential Revision: D5520138 Pulled By: sagar0 fbshipit-source-id: 340930afe9efe03c75f535a1da1f89bd3e53c1f9
-
由 Islam AbdelRahman 提交于
Summary: Simple component that will add a new entry in a log file every time we lookup/insert a key in SimCache. API: ``` SimCache::StartActivityLogging(<file_name>, <env>, <optional_max_size>) SimCache::StopActivityLogging() ``` Sending for review, Still need to add more comments. I was thinking about a better approach, but I ended up deciding I will use a mutex to sync the writes to the file, since this feature should not be heavily used and only used to collect info that will be analyzed offline. I think it's okay to hold the mutex every time we lookup/add to the SimCache. Closes https://github.com/facebook/rocksdb/pull/2295 Differential Revision: D5063826 Pulled By: IslamAbdelRahman fbshipit-source-id: f3b5daed8b201987c9a071146ddd5c5740a2dd8c
-
- 28 7月, 2017 8 次提交
-
-
由 Yi Wu 提交于
Summary: Introducing blob_db::TTLExtractor to replace extract_ttl_fn. The TTL extractor can be use to extract TTL from keys insert with Put or WriteBatch. Change over existing extract_ttl_fn are: * If value is changed, it will be return via std::string* (rather than Slice*). With Slice* the new value has to be part of the existing value. With std::string* the limitation is removed. * It can optionally return TTL or expiration. Other changes in this PR: * replace `std::chrono::system_clock` with `Env::NowMicros` so that I can mock time in tests. * add several TTL tests. * other minor naming change. Closes https://github.com/facebook/rocksdb/pull/2659 Differential Revision: D5512627 Pulled By: yiwu-arbug fbshipit-source-id: 0dfcb00d74d060b8534c6130c808e4d5d0a54440
-
由 Andrew Kryczka 提交于
Summary: Breaking commit: d12691b8 In the above commit, I moved the `TableCache` cleanup logic from `Version` destructor into `PurgeObsoleteFiles`. I missed cleaning up `TableCache` entries for the current `Version` during DB destruction. This PR adds that logic to `VersionSet` destructor. One unfortunate side effect is now we're potentially deleting `TableReader`s after `column_family_set_.reset()`, which means we can't call `BlockBasedTableReader::Close` a second time as the block cache might already be destroyed. Closes https://github.com/facebook/rocksdb/pull/2662 Differential Revision: D5515108 Pulled By: ajkr fbshipit-source-id: 2cb820e19aa813e0d258d17f76b2d7b6b7ee0b18
-
由 Yi Wu 提交于
Summary: We don't need to set them explicitly. Closes https://github.com/facebook/rocksdb/pull/2660 Differential Revision: D5514141 Pulled By: yiwu-arbug fbshipit-source-id: 10edebfc3cfe0afc00a34519f87fcea4d65069ae
-
由 Siying Dong 提交于
Summary: platform_dependent tests in Travis now builds all tests, which is not needed. Only build those tests we need to run. Closes https://github.com/facebook/rocksdb/pull/2647 Differential Revision: D5513954 Pulled By: siying fbshipit-source-id: 4d540b146124e70dd25586c47939d19f93655b0a
-
由 Aaron Gao 提交于
Summary: solved https://github.com/facebook/rocksdb/issues/2604 Closes https://github.com/facebook/rocksdb/pull/2648 Differential Revision: D5504875 Pulled By: lightmark fbshipit-source-id: c14bb62ccbdc9e7bda9cd914cae4ea0765d882ee
-
由 Siying Dong 提交于
Summary: This is to work around the problem of build error: util/threadpool_imp.o: file not recognized: File truncated Just to make the build go through. We should remove it later if we find the real long-term solution. Closes https://github.com/facebook/rocksdb/pull/2657 Differential Revision: D5511034 Pulled By: siying fbshipit-source-id: 229f024bd78ee96799017d4a89be74253058ec30
-
由 Andrew Kryczka 提交于
Summary: Post-compaction work holds onto db mutex for the longest time (found by tracing lock acquires/releases with LTTng and correlating timestamps with our info log). Further experimentation showed `TableCache::EraseHandle` is responsible for ~86% of time mutex is held. We can just release the handle outside the db mutex. Closes https://github.com/facebook/rocksdb/pull/2654 Differential Revision: D5507126 Pulled By: ajkr fbshipit-source-id: 703c01ddf2aea16bc0f9e33c08935d78aa6b781d
-
由 Andrew Kryczka 提交于
Summary: it should be a bool Closes https://github.com/facebook/rocksdb/pull/2653 Differential Revision: D5506148 Pulled By: ajkr fbshipit-source-id: f142f0f3aa8b678c68adef12e5ac6e1e163306f3
-
- 27 7月, 2017 1 次提交
-
-
由 Siying Dong 提交于
Summary: Closes https://github.com/facebook/rocksdb/pull/2649 Differential Revision: D5505778 Pulled By: siying fbshipit-source-id: 7e935603ede3d958ea087ed6b8cfc4121e8797bc
-