- 17 7月, 2018 3 次提交
-
-
由 Maysam Yabandeh 提交于
Summary: Refactor IndexBlockIter to reduce conditional branches on key_includes_seq_. IndexBlockIter::Prev is also separated from DataBlockIter::Prev, not to cache the prev entries as they are of less importance when iterating over the index block. Pull Request resolved: https://github.com/facebook/rocksdb/pull/4141 Differential Revision: D8866437 Pulled By: maysamyabandeh fbshipit-source-id: fdac76880426fc2be7d3c6354c09ab98f6657d4b
-
由 Sagar Vemuri 提交于
Summary: Allow ttl to be changed dynamically. Pull Request resolved: https://github.com/facebook/rocksdb/pull/4133 Differential Revision: D8845440 Pulled By: sagar0 fbshipit-source-id: c8c87ae643b3a8c4123e4c037c4645efc094a2d3
-
由 Siying Dong 提交于
Summary: Some logic only related to IndexBlockIter is separated from BlockIter to IndexBlockIter. This is done by writing an exclusive Seek() and SeekForPrev() for DataBlockIter, and all metadata block iter and tombstone block iter now use data block iter. Dealing with the BinarySeek() sharing problem by passing in the comparator to use. Pull Request resolved: https://github.com/facebook/rocksdb/pull/4136 Reviewed By: maysamyabandeh Differential Revision: D8859673 Pulled By: siying fbshipit-source-id: 703e5e6824b82b7cbf4721f3594b94127797ca9e
-
- 14 7月, 2018 10 次提交
-
-
由 Nathan VanBenschoten 提交于
Summary: Fixes #3391. This change adds a `DeleteRange` method to `SstFileWriter` and adds support for ingesting SSTs with range deletion tombstones. This is important for applications that need to atomically ingest SSTs while clearing out any existing keys in a given key range. Pull Request resolved: https://github.com/facebook/rocksdb/pull/3778 Differential Revision: D8821836 Pulled By: anand1976 fbshipit-source-id: ca7786c1947ff129afa703dab011d524c7883844
-
由 Zhongyi Xie 提交于
Summary: Our "rocksdb.sst.read.micros" stat includes time spent waiting for rate limiter. It probably only affects people rate limiting compaction reads, which is fairly rare. Pull Request resolved: https://github.com/facebook/rocksdb/pull/4102 Differential Revision: D8848506 Pulled By: miasantreble fbshipit-source-id: 01258ac5ae56e4eee372978cfc9143a6869f8bfc
-
由 Peter Mattis 提交于
Summary: Do not consider the range tombstone sentinel key as causing 2 adjacent sstables in a level to overlap. When a range tombstone's end key is the largest key in an sstable, the sstable's end key is so to a "sentinel" value that is the smallest key in the next sstable with a sequence number of kMaxSequenceNumber. This "sentinel" is guaranteed to not overlap in internal-key space with the next sstable. Unfortunately, GetOverlappingFiles uses user-keys to determine overlap and was thus considering 2 adjacent sstables in a level to overlap if they were separated by this sentinel key. This in turn would cause compactions to be larger than necessary. Note that this conflicts with https://github.com/facebook/rocksdb/pull/2769 and cases `DBRangeDelTest.CompactionTreatsSplitInputLevelDeletionAtomically` to fail. Pull Request resolved: https://github.com/facebook/rocksdb/pull/4050 Differential Revision: D8844423 Pulled By: ajkr fbshipit-source-id: df3f9f1db8f4cff2bff77376b98b83c2ae1d155b
-
由 Yanqin Jin 提交于
Summary: Make `ExternalSSTFileTest.IngestFileWithGlobalSeqnoRandomized` run faster. `make format` Pull Request resolved: https://github.com/facebook/rocksdb/pull/4131 Differential Revision: D8839952 Pulled By: riversand963 fbshipit-source-id: 4a7e842fde1cde4dc902e928a1cf511322578521
-
由 Maysam Yabandeh 提交于
Summary: The patch makes sure that two parallel test threads will operate on different db paths. This enables using open source tools such as gtest-parallel to run the tests of a file in parallel. Example: ``` ~/gtest-parallel/gtest-parallel ./table_test``` Pull Request resolved: https://github.com/facebook/rocksdb/pull/4135 Differential Revision: D8846653 Pulled By: maysamyabandeh fbshipit-source-id: 799bad1abb260e3d346bcb680d2ae207a852ba84
-
由 Zhongyi Xie 提交于
Summary: Pull Request resolved: https://github.com/facebook/rocksdb/pull/4118 Differential Revision: D8845554 Pulled By: miasantreble fbshipit-source-id: 13bd3c1259a7c30bad762a413fe3bb24eea650ba
-
由 Fosco Marotto 提交于
Summary: Picked up a task to convert this to use the gtest framework. It can't be this simple, can it? It works, but should all the std::cout be removed? ``` [$] ~/git/rocksdb [gft !]: ./merge_test [==========] Running 2 tests from 1 test case. [----------] Global test environment set-up. [----------] 2 tests from MergeTest [ RUN ] MergeTest.MergeDbTest Test read-modify-write counters... a: 3 1 2 a: 3 b: 1225 3 Compaction started ... Compaction ended a: 3 b: 1225 Test merge-based counters... a: 3 1 2 a: 3 b: 1225 3 Test merge in memtable... a: 3 1 2 a: 3 b: 1225 3 Test Partial-Merge Test merge-operator not set after reopen [ OK ] MergeTest.MergeDbTest (93 ms) [ RUN ] MergeTest.MergeDbTtlTest Opening database with TTL Test read-modify-write counters... a: 3 1 2 a: 3 b: 1225 3 Compaction started ... Compaction ended a: 3 b: 1225 Test merge-based counters... a: 3 1 2 a: 3 b: 1225 3 Test merge in memtable... Opening database with TTL a: 3 1 2 a: 3 b: 1225 3 Test Partial-Merge Opening database with TTL Opening database with TTL Opening database with TTL Opening database with TTL Test merge-operator not set after reopen [ OK ] MergeTest.MergeDbTtlTest (97 ms) [----------] 2 tests from MergeTest (190 ms total) [----------] Global test environment tear-down [==========] 2 tests from 1 test case ran. (190 ms total) [ PASSED ] 2 tests. ``` Pull Request resolved: https://github.com/facebook/rocksdb/pull/4114 Differential Revision: D8822886 Pulled By: gfosco fbshipit-source-id: c299d008e883c3bb911d2b357a2e9e4423f8e91a
-
由 Maysam Yabandeh 提交于
Summary: The transactions are currently tested with and without using StackableDB. This is mostly to check that the code path is consistent with stackable db as well. Slow, stress tests however do not benefit from being run again with StackableDB. The patch excludes StackableDB from such tests. On a single core it reduced the runtime of transaction_test from 199s to 135s. Pull Request resolved: https://github.com/facebook/rocksdb/pull/4132 Differential Revision: D8841655 Pulled By: maysamyabandeh fbshipit-source-id: 7b9aaba2673b542b195439dfb306cef26bd63b19
-
由 Anand Ananthabhotla 提交于
Summary: 1. Move kUniversalSubcompactions up before kEnd in db_test_util.h, so tests that cycle through all the option_configs include this 2. Skip kUniversalSubcompactions wherever kUniversalCompaction and kUniversalCompactionMultilevel are skipped Related to #3935 Pull Request resolved: https://github.com/facebook/rocksdb/pull/4125 Differential Revision: D8828637 Pulled By: anand1976 fbshipit-source-id: 650dee15fd27d85281cf9bb4ca8ab460e04cac6f
-
由 Tamir Duberstein 提交于
Summary: - Avoid `strdup` to use jemalloc on Windows - Use `size_t` for consistency - Add GCC 8 to Travis - Add CMAKE_BUILD_TYPE=Release to Travis Pull Request resolved: https://github.com/facebook/rocksdb/pull/3433 Differential Revision: D6837948 Pulled By: sagar0 fbshipit-source-id: b8543c3a4da9cd07ee9a33f9f4623188e233261f
-
- 13 7月, 2018 10 次提交
-
-
由 Zhongyi Xie 提交于
Summary: Right now there is no support for enabling compaction filter in db_bench, we should add support for that to facilitate testing of compaction filter. This PR adds a compaction filter called KeepFilter and make `Filter` always returns false, essentially a noop compaction filter. This will allow us to test compaction filter code path without having to support arbitrary compaction filters Pull Request resolved: https://github.com/facebook/rocksdb/pull/4106 Differential Revision: D8828517 Pulled By: miasantreble fbshipit-source-id: 9ad76d04103eaa9d00da98334b4a39e542d26c41
-
由 Andrew Kryczka 提交于
Summary: `DEFINE_uint32` was unavailable on some platforms, e.g., https://travis-ci.org/facebook/rocksdb/jobs/403352902. Use `DEFINE_uint64` instead which should work as it's used many times elsewhere in this file. Pull Request resolved: https://github.com/facebook/rocksdb/pull/4129 Differential Revision: D8830311 Pulled By: ajkr fbshipit-source-id: b4fc90ba3f50e649c070ce8069c68e530d731f05
-
由 Yanqin Jin 提交于
Summary: The original `EnvPosixTest.RunImmediately` assumes that after scheduling a background thread, the thread is guaranteed to complete after 0.1 second. I do not know about any non-real-time OS/runtime providing this guarantee. Nor does C++11 standard say anything about this in the documentation of `std::thread`. In fact, we have observed this test failure multiple times on appveyor, and we haven't been able to reproduce the failure deterministically. Therefore, I disable this test for now until we know for sure how it used to fail. Instead, I add another test `EnvPosixTest.RunEventually` that checks that a thread will be scheduled eventually. Pull Request resolved: https://github.com/facebook/rocksdb/pull/4126 Differential Revision: D8827086 Pulled By: riversand963 fbshipit-source-id: abc5cb655f90d50b791493da5eeb3716885dfe93
-
由 Yanqin Jin 提交于
Summary: Reduce the number of key ranges in `ExternalSSTFileTest.OverlappingRanges` so that the test completes in shorter time to avoid timeouts. Pull Request resolved: https://github.com/facebook/rocksdb/pull/4127 Differential Revision: D8827851 Pulled By: riversand963 fbshipit-source-id: a16387b0cc92a7c872b1c50f0cfbadc463afc9db
-
由 Maysam Yabandeh 提交于
Summary: BlockIter is getting crowded including details that specific only to either index or data blocks. The patch moves down such details to DataBlockIter and IndexBlockIter, both inheriting from BlockIter. Pull Request resolved: https://github.com/facebook/rocksdb/pull/4121 Differential Revision: D8816832 Pulled By: maysamyabandeh fbshipit-source-id: d492e74155c11d8a0c1c85cd7ee33d24c7456197
-
由 Andrew Kryczka 提交于
Summary: give control of how often stats are printed, including jemalloc stats if enabled. Previously the default was 10 minutes so we'd only see updated stats for very long benchmark runs. Pull Request resolved: https://github.com/facebook/rocksdb/pull/4109 Differential Revision: D8796444 Pulled By: ajkr fbshipit-source-id: fd7902fe3f105fae89322c4ab63316bba4a2b15e
-
由 Yanqin Jin 提交于
Summary: Reduce #iterations from 5000 to 1000 so that `ExternalSSTFileTest.CompactDuringAddFileRandom` can finish faster. On the one hand, 5000 iterations does not seem to improve the quality of unit test in comparison with 1000. On the other hand, long running tests should belong to stress tests. Pull Request resolved: https://github.com/facebook/rocksdb/pull/4123 Differential Revision: D8822514 Pulled By: riversand963 fbshipit-source-id: 0f439b8d5ccd9a4aed84638f8bac16382de17245
-
由 Nikhil Benesch 提交于
Summary: This fixes the same performance issue that #3992 fixes but with much more invasive cleanup. I'm more excited about this PR because it paves the way for fixing another problem we uncovered at Cockroach where range deletion tombstones can cause massive compactions. For example, suppose L4 contains deletions from [a, c) and [x, z) and no other keys, and L5 is entirely empty. L6, however, is full of data. When compacting L4 -> L5, we'll end up with one file that spans, massively, from [a, z). When we go to compact L5 -> L6, we'll have to rewrite all of L6! If, instead of range deletions in L4, we had keys a, b, x, y, and z, RocksDB would have been smart enough to create two files in L5: one for a and b and another for x, y, and z. With the changes in this PR, it will be possible to adjust the compaction logic to split tombstones/start new output files when they would span too many files in the grandparent level. ajkr please take a look when you have a minute! Pull Request resolved: https://github.com/facebook/rocksdb/pull/4014 Differential Revision: D8773253 Pulled By: ajkr fbshipit-source-id: ec62fa85f648fdebe1380b83ed997f9baec35677
-
由 Fosco Marotto 提交于
Summary: Per GitHub warning Pull Request resolved: https://github.com/facebook/rocksdb/pull/4116 Differential Revision: D8812291 Pulled By: gfosco fbshipit-source-id: 3c55adc4ac737e4be077ddf29322c8961018d67c
-
由 Siying Dong 提交于
Summary: Two CI tests never pass because of the environment problem. Delete them. Pull Request resolved: https://github.com/facebook/rocksdb/pull/4110 Differential Revision: D8805713 Pulled By: siying fbshipit-source-id: 6eb4813dc2094ee2045ec8ede7fe8967d546d6e8
-
- 12 7月, 2018 7 次提交
-
-
由 Anand Ananthabhotla 提交于
Summary: This test routinely exceeds the FB contbuild test timeout of 10 minutes, due to the large number of iterations. The large number (mainly due to 100 randomly selected window sizes) does not seem to add any value. Reduce it to allow the test to finish in < 10 mins. Pull Request resolved: https://github.com/facebook/rocksdb/pull/4117 Differential Revision: D8815646 Pulled By: anand1976 fbshipit-source-id: 260690d24f444767ad93b039dec3ae8b9cdd1843
-
由 Siying Dong 提交于
Summary: Pull Request resolved: https://github.com/facebook/rocksdb/pull/4113 Differential Revision: D8814172 Pulled By: siying fbshipit-source-id: cabc31db2c74803af9b2f99329155a1086eb1b22
-
由 Nikhil Benesch 提交于
Summary: Run the basic range deletion tests against the standard set of configurations. This testing exposed that files with hash indexes and partitioned indexes were not handling the case where the file contained only range deletions--i.e., where the index was empty. Additionally file a TODO about the fact that range deletions are broken when allow_mmap_reads = true is set. /cc ajkr nvanbenschoten Best viewed with ?w=1: https://github.com/facebook/rocksdb/pull/4021/files?w=1 Pull Request resolved: https://github.com/facebook/rocksdb/pull/4021 Differential Revision: D8811860 Pulled By: ajkr fbshipit-source-id: 3cc07e6d6210a2a00b932866481b3d5c59775343
-
由 Nicolas Pépin-Perreault 提交于
Summary: Exposes BackupEngine::CreateNewBackupWithMetadata and BackupInfo metadata to the Java API. Full disclaimer, I'm not familiar with JNI stuff, so I might have forgotten something (hopefully no memory leaks!). I also tried to find contributing guidelines but didn't see any, but I hope the PR style is consistent with the rest of the code base. Pull Request resolved: https://github.com/facebook/rocksdb/pull/4111 Differential Revision: D8811180 Pulled By: ajkr fbshipit-source-id: e38b3e396c7574328c2a1a0e55acc8d092b6a569
-
由 Sagar Vemuri 提交于
Summary: Remove external tracking of AlignedBuffer's size in `ReadaheadRandomAccessFile` and `FilePrefetchBuffer`. Pull Request resolved: https://github.com/facebook/rocksdb/pull/4105 Differential Revision: D8805724 Pulled By: sagar0 fbshipit-source-id: d61d8c203c7c500e3f36e912132d7852026ed023
-
由 Yanqin Jin 提交于
Summary: Prior to this PR, there was a race condition between `DBImpl::SetOptions` and `BackupEngine::CreateNewBackup`, as illustrated below. ``` Time thread 1 thread 2 | CreateNewBackup -> GetLiveFiles | SetOptions -> RenameTempFileToOptionsFile | SetOptions -> RenameTempFileToOptionsFile | SetOptions -> RenameTempFileToOptionsFile // unlink oldest OPTIONS file | copy the oldest OPTIONS // IO error! V ``` Proposed fix is to check the value of `DBImpl::disable_obsolete_files_deletion_` before calling `DeleteObsoleteOptionsFiles`. Pull Request resolved: https://github.com/facebook/rocksdb/pull/4108 Differential Revision: D8796360 Pulled By: riversand963 fbshipit-source-id: 02045317f793ea4c7d4400a5bf333b8502fa3e82
-
由 Sagar Vemuri 提交于
Summary: Copy data between buffers inside FilePrefetchBuffer only when chunk length is greater than 0. Otherwise AlignedBuffer was accessing memory out of its range causing crashes. Removing the tracking of buffer length outside of `AlignedBuffer`, i.e. in `FilePrefetchBuffer` and `ReadaheadRandomAccessFile`, will follow in a separate PR, as it is not the root cause of the crash reported in #4051. (`FilePrefetchBuffer` itself has been this way from its inception, and `ReadaheadRandomAccessFile` was updated to add the buffer length at some point). Comprehensive tests for `FilePrefetchBuffer` also to follow in a separate PR. Pull Request resolved: https://github.com/facebook/rocksdb/pull/4100 Differential Revision: D8792590 Pulled By: sagar0 fbshipit-source-id: 3578f45761cf6884243e767f749db4016ccc93e1
-
- 10 7月, 2018 1 次提交
-
-
由 Siying Dong 提交于
In delete scheduler, before ftruncate file for slow delete, check whether there is other hard links (#4093) Summary: Right now slow deletion with ftruncate doesn't work well with checkpoints because it ruin hard linked files in checkpoints. To fix it, check the file has no other hard link before ftruncate it. Pull Request resolved: https://github.com/facebook/rocksdb/pull/4093 Differential Revision: D8730360 Pulled By: siying fbshipit-source-id: 756eea5bce8a87b9a2ea3a5bfa190b2cab6f75df
-
- 07 7月, 2018 3 次提交
-
-
由 Manuel Ung 提交于
Summary: This adds support for recovering WriteUnprepared transactions through the following changes: - The information in `RecoveredTransaction` is extended so that it can reference multiple batches. - `MarkBeginPrepare` is extended with a bool indicating whether it is an unprepared begin, and this is passed down to `InsertRecoveredTransaction` to indicate whether the current transaction is prepared or not. - `WriteUnpreparedTxnDB::Initialize` is overridden so that it will rollback unprepared transactions from the recovered transactions. This can be done without updating the prepare heap/commit map, because this is before the DB has finished initializing, and after writing the rollback batch, those data structures should not contain information about the rolled back transaction anyway. Commit/Rollback of live transactions is still unimplemented and will come later. Pull Request resolved: https://github.com/facebook/rocksdb/pull/4078 Differential Revision: D8703382 Pulled By: lth fbshipit-source-id: 7e0aada6c23bd39299f1f20d6c060492e0e6b60a
-
由 Yanqin Jin 提交于
Summary: `std::map::at(key)` throws std::out_of_range if key does not exist. Current code does not handle this. Although this case is unlikely, I feel it's safe to use `std::map::find`. Pull Request resolved: https://github.com/facebook/rocksdb/pull/4098 Differential Revision: D8753865 Pulled By: riversand963 fbshipit-source-id: 9a9ba43badb0fb5e0d24cd87903931fd12f3f8ec
-
由 Yanqin Jin 提交于
Summary: This was caught by crash test, and the following is a simple way to reproduce it and verify the fix. One way to trigger this code path is to use the following configuration: - Compress SST file - Enable direct IO and prefetch buffer - Do NOT use compressed block cache Closes https://github.com/facebook/rocksdb/pull/4096 Differential Revision: D8742009 Pulled By: riversand963 fbshipit-source-id: f13381078bbb0dce92f60bd313a78ab602bcacd2
-
- 06 7月, 2018 2 次提交
-
-
由 Huachao Huang 提交于
Summary: The max_subcompactions option was introduced in https://github.com/facebook/rocksdb/pull/3775. Closes https://github.com/facebook/rocksdb/pull/4082 Differential Revision: D8743258 Pulled By: ajkr fbshipit-source-id: d60ee75769dfc19ab6f8754e4ff3a267848f1ed9
-
由 Yanqin Jin 提交于
Summary: Increase the size of each shard so that the number of cache hit/miss match expectation. Otherwise FilterBlockInBlockCache test will fail. Closes https://github.com/facebook/rocksdb/pull/4090 Differential Revision: D8736158 Pulled By: riversand963 fbshipit-source-id: 5cdbc06b02390389fd5b72a6d251d88949ad3d91
-
- 04 7月, 2018 1 次提交
-
-
由 Siying Dong 提交于
Summary: Now by default, with NewSstFileManager, checkpoints may be corrupted. Disable this feature to avoid this issue. Closes https://github.com/facebook/rocksdb/pull/4092 Differential Revision: D8729856 Pulled By: siying fbshipit-source-id: 914c321d6eaf52d8c5981171322d85dd29088307
-
- 03 7月, 2018 1 次提交
-
-
由 Adam Retter 提交于
Summary: It seems that compilation has been made stricter about unused args. Closes https://github.com/facebook/rocksdb/pull/4080 Differential Revision: D8712049 Pulled By: sagar0 fbshipit-source-id: 984af1982638af3568aac1a167f565f4741badee
-
- 30 6月, 2018 2 次提交
-
-
由 Andrey Zagrebin 提交于
Summary: to address issue #3849 Closes https://github.com/facebook/rocksdb/pull/3850 Differential Revision: D8695487 Pulled By: sagar0 fbshipit-source-id: 04baeb2127663934ed1321fe6d9a9ec23c86e16b
-
由 Daniel Black 提交于
Summary: clang version 7.0.0- (trunk) Target: x86_64-unknown-linux-gnu Thread model: posix InstalledDir: /usr/bin clang++-7 -DROCKSDB_USE_RTTI -g -W -Wextra -Wall -Wsign-compare -Wshadow -Wno-unused-parameter -Werror -I. -I./include -std=c++11 -DROCKSDB_PLATFORM_POSIX -DROCKSDB_LIB_IO_POSIX -DOS_LINUX -fno-builtin-memcmp -DROCKSDB_FALLOCATE_PRESENT -DSNAPPY -DGFLAGS=google -DZLIB -DBZIP2 -DROCKSDB_MALLOC_USABLE_SIZE -DROCKSDB_PTHREAD_ADAPTIVE_MUTEX -DROCKSDB_BACKTRACE -DROCKSDB_RANGESYNC_PRESENT -DROCKSDB_SCHED_GETCPU_PRESENT -Wshorten-64-to-32 -march=native -DHAVE_SSE42 -DROCKSDB_SUPPORT_THREAD_LOCAL -isystem ./third-party/gtest-1.7.0/fused-src -DTRAVIS -O2 -fno-omit-frame-pointer -momit-leaf-frame-pointer -Woverloaded-virtual -Wnon-virtual-dtor -Wno-missing-field-initializers -c utilities/transactions/transaction_test.cc -o utilities/transactions/transaction_test.o utilities/transactions/transaction_test.cc:2282:22: error: unused variable 'txn_options' [-Werror,-Wunused-variable] TransactionOptions txn_options; ^ utilities/transactions/transaction_test.cc:2822:22: error: unused variable 'txn_options' [-Werror,-Wunused-variable] TransactionOptions txn_options; ^ utilities/transactions/transaction_test.cc:2928:22: error: unused variable 'txn_options' [-Werror,-Wunused-variable] TransactionOptions txn_options; ^ utilities/transactions/transaction_test.cc:3109:22: error: unused variable 'txn_options' [-Werror,-Wunused-variable] TransactionOptions txn_options; ^ utilities/transactions/transaction_test.cc:4364:22: error: unused variable 'txn_options' [-Werror,-Wunused-variable] TransactionOptions txn_options; ^ Closes https://github.com/facebook/rocksdb/pull/4074 Differential Revision: D8698051 Pulled By: ajkr fbshipit-source-id: 6255618eefdd189962fbea1b02cf1eb5ae501274
-