1. 28 11月, 2018 2 次提交
    • A
      Test mapping of Histograms and HistogramsNameMap (#4720) · 1db4a096
      Adam Singer 提交于
      Summary:
      Adding sanity check test for mapping of `Histograms` and `HistogramsNameMap`
      
      ```
      [==========] Running 2 tests from 1 test case.
      [----------] Global test environment set-up.
      [----------] 2 tests from StatisticsTest
      [ RUN      ] StatisticsTest.SanityTickers
      [       OK ] StatisticsTest.SanityTickers (0 ms)
      [ RUN      ] StatisticsTest.SanityHistograms
      [       OK ] StatisticsTest.SanityHistograms (0 ms)
      [----------] 2 tests from StatisticsTest (0 ms total)
      
      [----------] Global test environment tear-down
      [==========] 2 tests from 1 test case ran. (0 ms total)
      [  PASSED  ] 2 tests.
      ```
      Pull Request resolved: https://github.com/facebook/rocksdb/pull/4720
      
      Differential Revision: D13217061
      
      Pulled By: ajkr
      
      fbshipit-source-id: 6427f4e684c36b2f3c3440808b74fee86a364683
      1db4a096
    • S
      Fix Java to C++ ticker conversions (#4719) · a2dec2ed
      Sagar Vemuri 提交于
      Summary:
      Added back `NO_ITERATORS` and moved `NO_ITERATOR_CREATED` to the end of `toCppTickers`.
      
      This is a leftover fix which is needed in addition to a138e351 to correctly convert java tickers to c++ tickers. a138e351 only updated `toJavaTickerType` but both `toJavaTickerType` and `toCppTickers` need to be changed.
      Pull Request resolved: https://github.com/facebook/rocksdb/pull/4719
      
      Differential Revision: D13208847
      
      Pulled By: sagar0
      
      fbshipit-source-id: 53a42f3d6ffe04034acfde972d73040b92b4c1af
      a2dec2ed
  2. 27 11月, 2018 4 次提交
    • P
      Fix build with ROCKSDB_LITE and -Wunused-private-field (#4715) · 60deb448
      Po-Chuan Hsieh 提交于
      Summary:
      The error message of databases/rocksdb-lite (FreeBSD port) is as follows:
      ```
        tools/db_bench_tool.cc:1976:16: error: private field 'trace_options_' is not used [-Werror,-Wunused-private-field]
          TraceOptions trace_options_;
                       ^
      ```
      Pull Request resolved: https://github.com/facebook/rocksdb/pull/4715
      
      Differential Revision: D13207902
      
      Pulled By: ajkr
      
      fbshipit-source-id: be3c612eba656aeddb77e35e2f201dd25dc92f7e
      60deb448
    • S
      FIX #3278: Move global const object definitions from .h to .cc (#4691) · f1837595
      Soli 提交于
      Summary:
      Summary
      
      We should declare constants in headers and define them in source files.
      But this commit is only aimed at compound types.
      
      I don't know if it is necessary to do the same thing to fundamental types.
      
      I used this command to find all of the constant definitions in header files.
      
      `find . -name "*.h" | xargs grep -e "^const .*=.*"`
      
      And here is what I found:
      
      ```
      ./db/version_edit.h:const uint64_t kFileNumberMask = 0x3FFFFFFFFFFFFFFF;
      ./include/rocksdb/env.h:const size_t kDefaultPageSize = 4 * 1024;
      ./include/rocksdb/statistics.h:const std::vector<std::pair<Tickers, std::string>> TickersNameMap = {
      ./include/rocksdb/statistics.h:const std::vector<std::pair<Histograms, std::string>> HistogramsNameMap = {
      ./include/rocksdb/table.h:const uint32_t kPlainTableVariableLength = 0;
      ./include/rocksdb/utilities/transaction_db.h:const uint32_t kInitialMaxDeadlocks = 5;
      ./port/port_posix.h:const uint32_t kMaxUint32 = std::numeric_limits<uint32_t>::max();
      ./port/port_posix.h:const int kMaxInt32 = std::numeric_limits<int32_t>::max();
      ./port/port_posix.h:const uint64_t kMaxUint64 = std::numeric_limits<uint64_t>::max();
      ./port/port_posix.h:const int64_t kMaxInt64 = std::numeric_limits<int64_t>::max();
      ./port/port_posix.h:const size_t kMaxSizet = std::numeric_limits<size_t>::max();
      ./port/win/port_win.h:const uint32_t kMaxUint32 = UINT32_MAX;
      ./port/win/port_win.h:const int kMaxInt32 = INT32_MAX;
      ./port/win/port_win.h:const int64_t kMaxInt64 = INT64_MAX;
      ./port/win/port_win.h:const uint64_t kMaxUint64 = UINT64_MAX;
      ./port/win/port_win.h:const size_t kMaxSizet = UINT64_MAX;
      ./port/win/port_win.h:const size_t kMaxSizet = UINT_MAX;
      ./port/win/port_win.h:const uint32_t kMaxUint32 = std::numeric_limits<uint32_t>::max();
      ./port/win/port_win.h:const int kMaxInt32 = std::numeric_limits<int>::max();
      ./port/win/port_win.h:const uint64_t kMaxUint64 = std::numeric_limits<uint64_t>::max();
      ./port/win/port_win.h:const int64_t kMaxInt64 = std::numeric_limits<int64_t>::max();
      ./port/win/port_win.h:const size_t kMaxSizet = std::numeric_limits<size_t>::max();
      ./port/win/port_win.h:const bool kLittleEndian = true;
      ./table/cuckoo_table_factory.h:const uint32_t kCuckooMurmurSeedMultiplier = 816922183;
      ./table/data_block_hash_index.h:const uint8_t kNoEntry = 255;
      ./table/data_block_hash_index.h:const uint8_t kCollision = 254;
      ./table/data_block_hash_index.h:const uint8_t kMaxRestartSupportedByHashIndex = 253;
      ./table/data_block_hash_index.h:const size_t kMaxBlockSizeSupportedByHashIndex = 1u << 16;
      ./table/data_block_hash_index.h:const double kDefaultUtilRatio = 0.75;
      ./table/filter_block.h:const uint64_t kNotValid = ULLONG_MAX;
      ./table/format.h:const int kMagicNumberLengthByte = 8;
      ./third-party/fbson/FbsonJsonParser.h:const char* const kJsonDelim = " ,]}\t\r\n";
      ./third-party/fbson/FbsonJsonParser.h:const char* const kWhiteSpace = " \t\n\r";
      ./third-party/gtest-1.7.0/fused-src/gtest/gtest.h:const BiggestInt kMaxBiggestInt =
      ./third-party/gtest-1.7.0/fused-src/gtest/gtest.h:const char kDeathTestStyleFlag[] = "death_test_style";
      ./third-party/gtest-1.7.0/fused-src/gtest/gtest.h:const char kDeathTestUseFork[] = "death_test_use_fork";
      ./third-party/gtest-1.7.0/fused-src/gtest/gtest.h:const char kInternalRunDeathTestFlag[] = "internal_run_death_test";
      ./third-party/gtest-1.7.0/fused-src/gtest/gtest.h:const char* pets[] = {"cat", "dog"};
      ./third-party/gtest-1.7.0/fused-src/gtest/gtest.h:const size_t kProtobufOneLinerMaxLength = 50;
      ./third-party/gtest-1.7.0/fused-src/gtest/gtest.h:const int kMaxStackTraceDepth = 100;
      ./third-party/gtest-1.7.0/fused-src/gtest/gtest.h:const T* WithParamInterface<T>::parameter_ = NULL;
      ./util/coding.h:const unsigned int kMaxVarint64Length = 10;
      ./util/filename.h:const size_t kFormatFileNumberBufSize = 38;
      ./util/testutil.h:const SliceTransform* RandomSliceTransform(Random* rnd, int pre_defined = -1);
      ./util/trace_replay.h:const std::string kTraceMagic = "feedcafedeadbeef";
      ./util/trace_replay.h:const unsigned int kTraceTimestampSize = 8;
      ./util/trace_replay.h:const unsigned int kTraceTypeSize = 1;
      ./util/trace_replay.h:const unsigned int kTracePayloadLengthSize = 4;
      ./util/trace_replay.h:const unsigned int kTraceMetadataSize =
      ./utilities/cassandra/serialize.h:const int64_t kCharMask = 0xFFLL;
      ./utilities/cassandra/serialize.h:const int32_t kBitsPerByte = 8;
      ```
      
      And these 3 lines are related to this commit:
      
      ```
      ./include/rocksdb/statistics.h:const std::vector<std::pair<Tickers, std::string>> TickersNameMap = {
      ./include/rocksdb/statistics.h:const std::vector<std::pair<Histograms, std::string>> HistogramsNameMap = {
      ./util/trace_replay.h:const std::string kTraceMagic = "feedcafedeadbeef";
      ```
      
      Any comments would be appreciated.
      Thanks.
      Pull Request resolved: https://github.com/facebook/rocksdb/pull/4691
      
      Differential Revision: D13208049
      
      Pulled By: ajkr
      
      fbshipit-source-id: e5ee55fdaec5447fc5798c6721e2821e7cdc0d5b
      f1837595
    • S
      RocksJava: Add the missing FIFO compaction options (#4609) · 0d65315c
      Sagar Vemuri 提交于
      Summary:
      Make CompactionOptionsFIFO's ttl and allow_compaction options to be available in RocksJava.
      Pull Request resolved: https://github.com/facebook/rocksdb/pull/4609
      
      Differential Revision: D12849503
      
      Pulled By: sagar0
      
      fbshipit-source-id: 47baa97918d252370f234c36c1af15ff2dad7658
      0d65315c
    • A
      Speed up range scans with range tombstones (#4677) · 85394a96
      Abhishek Madan 提交于
      Summary:
      Previously, every range tombstone iterator was seeked on every
      ShouldDelete call, which quickly degraded performance for long range
      scans. This PR improves performance by tracking iterator positions and
      only advancing iterators when necessary.
      Pull Request resolved: https://github.com/facebook/rocksdb/pull/4677
      
      Differential Revision: D13205373
      
      Pulled By: abhimadan
      
      fbshipit-source-id: 80c199dace1e19362a4c61c686bf01913eae87cb
      85394a96
  3. 25 11月, 2018 1 次提交
  4. 22 11月, 2018 4 次提交
    • A
      DeleteRange blog post (#4711) · e9372dcb
      Andrew Kryczka 提交于
      Summary:
      as titled
      Pull Request resolved: https://github.com/facebook/rocksdb/pull/4711
      
      Differential Revision: D13166391
      
      Pulled By: ajkr
      
      fbshipit-source-id: 3a3e537cebe2ba97a7ae6fcc3282db2ea755158e
      e9372dcb
    • A
      Fix ticker stat for number files closed (#4703) · 07cf0ee5
      Andrew Kryczka 提交于
      Summary:
      We haven't been populating `NO_FILE_CLOSES` since v1.5.8 even though it was never marked as deprecated. Start populating it again. Conveniently `DeleteTableReader` has an unused `void*` argument that we can use...
      
      Blame: 63f216ee
      
      Closes #4700.
      Pull Request resolved: https://github.com/facebook/rocksdb/pull/4703
      
      Differential Revision: D13146769
      
      Pulled By: ajkr
      
      fbshipit-source-id: ad8d6fb0493e701f60a165a3bca1787d255be008
      07cf0ee5
    • Y
      Revert "Move MemoryAllocator option from Cache to BlockBasedTableOpti… (#4697) · 05d9d821
      Yi Wu 提交于
      Summary:
      …ons (#4676)"
      
      This reverts commit b32d087d.
      
      `MemoryAllocator` needs to be with `Cache`, since cache entry can
      outlive DB and block based table. The cache needs to hold reference to
      memory allocator when deleting cache entry.
      Pull Request resolved: https://github.com/facebook/rocksdb/pull/4697
      
      Differential Revision: D13133490
      
      Pulled By: yiwu-arbug
      
      fbshipit-source-id: 8ef7e8a51263bfd929f892fd062665ff4ce9ce5a
      05d9d821
    • A
      Introduce RangeDelAggregatorV2 (#4649) · 457f77b9
      Abhishek Madan 提交于
      Summary:
      The old RangeDelAggregator did expensive pre-processing work
      to create a collapsed, binary-searchable representation of range
      tombstones. With FragmentedRangeTombstoneIterator, much of this work is
      now unnecessary. RangeDelAggregatorV2 takes advantage of this by seeking
      in each iterator to find a covering tombstone in ShouldDelete, while
      doing minimal work in AddTombstones. The old RangeDelAggregator is still
      used during flush/compaction for now, though RangeDelAggregatorV2 will
      support those uses in a future PR.
      Pull Request resolved: https://github.com/facebook/rocksdb/pull/4649
      
      Differential Revision: D13146964
      
      Pulled By: abhimadan
      
      fbshipit-source-id: be29a4c020fc440500c137216fcc1cf529571eb3
      457f77b9
  5. 21 11月, 2018 2 次提交
    • A
      Fix range tombstone covering short-circuit logic (#4698) · ed5aec5b
      Abhishek Madan 提交于
      Summary:
      Since a range tombstone seen at one level will cover all keys
      in the range at lower levels, there was a short-circuiting check in Get
      that reported a key was not found at most one file after the range
      tombstone was discovered. However, this was incorrect for merge
      operands, since a deletion might only cover some merge operands,
      which implies that the key should be found. This PR fixes this logic in
      the Version portion of Get, and removes the logic from the MemTable
      portion of Get, since the perforamnce benefit provided there is minimal.
      Pull Request resolved: https://github.com/facebook/rocksdb/pull/4698
      
      Differential Revision: D13142484
      
      Pulled By: abhimadan
      
      fbshipit-source-id: cbd74537c806032f2bfa564724d01a80df7c8f10
      ed5aec5b
    • A
      Fix compatibility of public ticker stats (#4701) · a138e351
      Andrew Kryczka 提交于
      Summary:
      - Added back the `NO_ITERATORS` that was removed in 5945e16d.
      - Marked it as deprecated since it is no longer populated, but kept for API compatibility.
      - Made sure the new tickers, `NO_ITERATOR_CREATED` and `NO_ITERATOR_DELETED`, are appended at the end of the enum, in case people are relying on the int values.
      
      The change where `NO_ITERATOR_CREATED` and `NO_ITERATOR_DELETED` were introduced is unreleased so I believe it is ok to change their ordering.
      Pull Request resolved: https://github.com/facebook/rocksdb/pull/4701
      
      Differential Revision: D13142887
      
      Pulled By: ajkr
      
      fbshipit-source-id: 29a336ce5b46632ce50ad42ccc4a29013f71d6d6
      a138e351
  6. 20 11月, 2018 2 次提交
    • Y
      JemallocAllocator: thread-local tcache (#4603) · 327097c9
      Yi Wu 提交于
      Summary:
      Add option to support  thread-local tcache to reduce mutex contention inside Jemalloc arena.
      Pull Request resolved: https://github.com/facebook/rocksdb/pull/4603
      
      Differential Revision: D12830738
      
      Pulled By: yiwu-arbug
      
      fbshipit-source-id: 59bd25b165b903f23a6a8531b18d72e140d69f65
      327097c9
    • M
      Run Define codemod in fbcode · 659d0e60
      Maggie Moss 提交于
      Summary:
      Found a callsite for `moo_translate` in the Scuba warnings and realized we have a few calls to `define()` left in fbcode.
      
      - I ran the `DefineCodemod` script against fbcode
      - Fixed broken tests, and ensured that tests that are explicitly testing the behaviour of `define()` were not changed.
      
      bypass-lint
      
      Reviewed By: kmeht
      
      Differential Revision: D12968447
      
      fbshipit-source-id: d8fd3649a2ce9868b8938d293e1bebf1a6d2fad8
      659d0e60
  7. 19 11月, 2018 1 次提交
  8. 17 11月, 2018 1 次提交
  9. 15 11月, 2018 5 次提交
    • Y
      Rollback memtable flush upon atomic flush fail (#4641) · 14769742
      Yanqin Jin 提交于
      Summary:
      This fixes an assertion.
      
      An atomic flush can have multiple flush jobs. Some of them may fail. If any of
      them fails, we need to rollback all of them.
      For the flush jobs that do fail, we already call `RollbackMemTableFlush` in
      `FlushJob::Run`. The tricky part is for flush jobs that have completed
      successfully. We need to call `RollbackMemTableFlush` for them as well.
      
      The newly added DBAtomicFlushTest.AtomicFlushRollbackSomeJobs will SigAbort
      without the corresponding change in AtomicFlushMemTablesToOutputFiles.
      Pull Request resolved: https://github.com/facebook/rocksdb/pull/4641
      
      Differential Revision: D12943649
      
      Pulled By: riversand963
      
      fbshipit-source-id: c66a4a664a1e0938e938fd41edc5a70c34cdd868
      14769742
    • A
      Modify FragmentedRangeTombstoneList member layout (#4632) · 6bee36a7
      Abhishek Madan 提交于
      Summary:
      Rather than storing a `vector<RangeTombstone>`, we now store a
      `vector<RangeTombstoneStack>` and a `vector<SequenceNumber>`. A
      `RangeTombstoneStack` contains the start and end keys of a range tombstone
      fragment, and indices into the seqnum vector to indicate which sequence
      numbers the fragment is located at. The diagram below illustrates an
      example:
      
      ```
      tombstones_:     [a, b) [c, e) [h, k)
                         | \   /  \   /  |
                         |  \ /    \ /   |
                         v   v      v    v
      tombstone_seqs_: [ 5 3 10 7 2 8 6  ]
      ```
      
      This format allows binary searching the tombstone list to use less key
      comparisons, which helps in cases where there are many overlapping
      tombstones. Also, this format makes it easier to add DBIter-like
      semantics to `FragmentedRangeTombstoneIterator` in the future.
      Pull Request resolved: https://github.com/facebook/rocksdb/pull/4632
      
      Differential Revision: D13053103
      
      Pulled By: abhimadan
      
      fbshipit-source-id: e8220cc712fcf5be4d602913bb23ace8ea5f8ef0
      6bee36a7
    • S
      Increase wait time in DBTest.SanitizeNumThreads (#4659) · f5c8cf5f
      Siying Dong 提交于
      Summary:
      DBTest.SanitizeNumThreads Sometimes fails. The test waited for 10ms timeout and expect all threads scheduled to be executed. This can be a source of flakiness. Make a check every 1ms and up to 10s.
      Pull Request resolved: https://github.com/facebook/rocksdb/pull/4659
      
      Differential Revision: D13074174
      
      Pulled By: siying
      
      fbshipit-source-id: b1d5ff87a326a4fc9eab8d1cc307bbb940dfe70c
      f5c8cf5f
    • M
      Fix ignoring params in default impl of GetForUpdate (#4679) · c2a20f17
      Maysam Yabandeh 提交于
      Summary:
      The default implementation of GetForUpdate that receives PinnableSlice was mistakenly dropping column_family and exclusive parameters.
      Pull Request resolved: https://github.com/facebook/rocksdb/pull/4679
      
      Differential Revision: D13062531
      
      Pulled By: maysamyabandeh
      
      fbshipit-source-id: 7625d0c1ba872a5d894b58ced42147d6c8556a6f
      c2a20f17
    • A
      Add max_scan_distance flag to db_bench (#4660) · 0ed738fd
      Abhishek Madan 提交于
      Summary:
      The new flag makes it possible to constrain iterator traversal
      by the upper/lower bound the iterator is expected to pass. This allows
      seekrandom results to be more easily comparable between DBs with and
      without deletions.
      Pull Request resolved: https://github.com/facebook/rocksdb/pull/4660
      
      Differential Revision: D13053111
      
      Pulled By: abhimadan
      
      fbshipit-source-id: 33e250f2e2d210b54c7726399da30a33f723c33c
      0ed738fd
  10. 14 11月, 2018 14 次提交
    • Y
      Improve result report of scan (#4648) · de651035
      Yanqin Jin 提交于
      Summary:
      When iterator becomes invalid, there are two possibilities.
      First, all data in the column family have been scanned and there is nothing
      more to scan.
      Second, an underlying error has occurred, causing `status()` to be !ok.
      Therefore, we need to check for both cases when `!iter->Valid()`.
      Pull Request resolved: https://github.com/facebook/rocksdb/pull/4648
      
      Differential Revision: D12959601
      
      Pulled By: riversand963
      
      fbshipit-source-id: 49c9382c9ea9e78f2e2b6f3708f0670b822ca8dd
      de651035
    • T
      Expose underlying Read/Write APIs for avoiding unnecessary memory copy (#2303) · 5cf5f472
      thisisbaozi 提交于
      Summary:
      adamretter
      
      As you already mentioned at #1247 .
      Pull Request resolved: https://github.com/facebook/rocksdb/pull/2303
      
      Differential Revision: D10209001
      
      Pulled By: sagar0
      
      fbshipit-source-id: bcbce004112c2edeaff116968d79c6f90aab4b6c
      5cf5f472
    • Z
      release db mutex when calling ApproximateSize (#4630) · d8df169b
      Zhongyi Xie 提交于
      Summary:
      `GenSubcompactionBoundaries` calls `VersionSet::ApproximateSize` which gets BlockBasedTableReader for every file and seeks in its index block to find `key`'s offset. If the table or index block aren't in memory already, this involves I/O. This can be improved by releasing DB mutex when calling ApproximateSize.
      Pull Request resolved: https://github.com/facebook/rocksdb/pull/4630
      
      Differential Revision: D13052653
      
      Pulled By: miasantreble
      
      fbshipit-source-id: cae31d46d10d0860fa8a26b8d5154b2d17d1685f
      d8df169b
    • S
      Remove two variables from BlockContents class and don't use class Block for... · b82e57d4
      Siying Dong 提交于
      Remove two variables from BlockContents class and don't use class Block for compressed block (#4650)
      
      Summary:
      We carry compression type and "cachable" variables for every block in the block cache, while they take well-known values. 8-byte is wasted for each block (2-byte for useful information but it takes 8 bytes because of padding). With this change, these two variables are removed.
      
      The cachable information is only useful in the process of reading the block. We use other information to infer from it. For compressed blocks, the compression type is a part of the block content itself so we can get it from there.
      
      Some code is slightly refactored so that the cachable information can flow better.
      
      Another change is to only use class BlockContents for compressed block, and narrow the class Block to only be used for uncompressed blocks, including blocks in compressed block cache. This can make the Block class less confusing. It also saves tens of bytes for each block in compressed block cache.
      Pull Request resolved: https://github.com/facebook/rocksdb/pull/4650
      
      Differential Revision: D12969070
      
      Pulled By: siying
      
      fbshipit-source-id: 548b62724e9eb66993026429fd9c7c3acd1f95ed
      b82e57d4
    • Z
      apply ReadOptions.iterate_upper_bound to transaction iterator (#4656) · b76398a8
      Zhongyi Xie 提交于
      Summary:
      Currently transaction iterator does not apply `ReadOptions.iterate_upper_bound` when iterating. This PR attempts to fix the problem by having `BaseDeltaIterator` enforcing the upper bound check when iterator state is changed.
      Pull Request resolved: https://github.com/facebook/rocksdb/pull/4656
      
      Differential Revision: D13039257
      
      Pulled By: miasantreble
      
      fbshipit-source-id: 909eb9f6b4597a4d80418fb139f32ec82c6ec1d1
      b76398a8
    • S
      optimized the performance of autovector::emplace_back. (#4606) · a2de8e52
      Simon Liu 提交于
      Summary:
      It called the autovector::push_back simply in autovector::emplace_back.
      This was not efficient, and then optimazed this function through the
      perfect forwarding.
      
      This was the src and result of the benchmark(using the google'benchmark library, the type of elem in
      autovector was std::string, and call emplace_back with the "char *" type):
      
      https://gist.github.com/monadbobo/93448b89a42737b08cbada81de75c5cd
      
      PS: The benchmark's result of  previous PR was not accurate, and so I update the test case and result.
      Pull Request resolved: https://github.com/facebook/rocksdb/pull/4606
      
      Differential Revision: D13046813
      
      Pulled By: sagar0
      
      fbshipit-source-id: 19cde1bcadafe899aa454b703acb35737a1cc02d
      a2de8e52
    • Y
      Move MemoryAllocator option from Cache to BlockBasedTableOptions (#4676) · b32d087d
      Yi Wu 提交于
      Summary:
      Per offline discussion with siying, `MemoryAllocator` and `Cache` should be decouple. The idea is that memory allocator handles memory allocation, while cache handle cache policy.
      
      It is normal that external cache libraries pack couple the two components for better optimization. If we want to integrate with such library in the future, we can make a wrapper of the library implementing both `Cache` and `MemoryAllocator` interface.
      Pull Request resolved: https://github.com/facebook/rocksdb/pull/4676
      
      Differential Revision: D13047662
      
      Pulled By: yiwu-arbug
      
      fbshipit-source-id: cd42e246d80ab600b4de47d073f7d2db308ce6dd
      b32d087d
    • S
      Add a unit test to assert number of preads (#4657) · abb1a8fc
      Siying Dong 提交于
      Summary:
      We used to have a bug, which caused every block to be read twice, and none of our tests caught it. Add a very simply unit test to make sure that when reading a data block, we only issue one pread against the SST file.
      Pull Request resolved: https://github.com/facebook/rocksdb/pull/4657
      
      Differential Revision: D13005260
      
      Pulled By: siying
      
      fbshipit-source-id: 03167b554ad2451192b1707415536d7d05e9026c
      abb1a8fc
    • Y
      BlobDB: use char array instead of string as buffer (#4662) · 05dab3aa
      Yi Wu 提交于
      Summary:
      As pointed out in #4059, we miss use string as buffer for file read. Changing to use char array instead.
      
      Closing #4059
      Pull Request resolved: https://github.com/facebook/rocksdb/pull/4662
      
      Differential Revision: D13012998
      
      Pulled By: yiwu-arbug
      
      fbshipit-source-id: 41234ba17c0bccea65bd647e362a0e979152bd1e
      05dab3aa
    • Q
      Expose num entries and deletions of sst files (#4623) · 4f0fcb78
      QingpingWang 提交于
      Summary:
      he ratio of num_deletions to num_entries of a level can be useful to determine if a manual compaction needs to be triggered on a level.
      Also refer #3980
      Pull Request resolved: https://github.com/facebook/rocksdb/pull/4623
      
      Differential Revision: D13045744
      
      Pulled By: sagar0
      
      fbshipit-source-id: 71f3c8e363a8ffd194ec3bb0ed0b69612231f0b3
      4f0fcb78
    • S
      Divide `NO_ITERATORS` into two counters `NO_ITERATOR_CREATED` and `NO_ITERATOR_DELETE` (#4498) · 5945e16d
      Soli Como 提交于
      Summary:
      Currently, `Statistics` can record tick by `recordTick()` whose second parameter is an `uint64_t`.
      That means tick can only increase.
      If we want to reduce tick, we have to work around like `RecordTick(statistics_, NO_ITERATORS, uint64_t(-1));`.
      That's kind of a hack.
      
      So, this PR divide `NO_ITERATORS` into two counters `NO_ITERATOR_CREATED` and `NO_ITERATOR_DELETE`, making the counters increase only.
      
      Fixes #3013 .
      Pull Request resolved: https://github.com/facebook/rocksdb/pull/4498
      
      Differential Revision: D10395010
      
      Pulled By: sagar0
      
      fbshipit-source-id: cfb523b22a37411c794b4e9da090f1ae30293db2
      5945e16d
    • S
      Fix #3840: only `SyncClosedLogs` for multiple CFs (#4460) · a4786822
      Soli 提交于
      Summary:
      Call `SyncClosedLogs()` only if there are more than one column families.
      
      Update several unit tests (in `fault_injection_test` and `db_flush_test`) correspondingly.
      
      See #3840 for more info.
      Pull Request resolved: https://github.com/facebook/rocksdb/pull/4460
      
      Differential Revision: D12896377
      
      Pulled By: riversand963
      
      fbshipit-source-id: f49afdaec32568f12f001219a3aec1dfde3b32bf
      a4786822
    • A
      Backup engine support for direct I/O reads (#4640) · ea945470
      Andrew Kryczka 提交于
      Summary:
      Use the `DBOptions` that the backup engine already holds to figure out the right `EnvOptions` to use when reading the DB files. This means that, if a user opened a DB instance with `use_direct_reads=true`, then using `BackupEngine` to back up that DB instance will use direct I/O to read files when calculating checksums and copying. Currently the WALs and manifests would still be read using buffered I/O to prevent mixing direct I/O reads with concurrent buffered I/O writes.
      Pull Request resolved: https://github.com/facebook/rocksdb/pull/4640
      
      Differential Revision: D13015268
      
      Pulled By: ajkr
      
      fbshipit-source-id: 77006ad6f3e00ce58374ca4793b785eea0db6269
      ea945470
    • Z
      use per-level perfcontext for DB::Get calls (#4617) · b3130193
      Zhongyi Xie 提交于
      Summary:
      this PR adds two more per-level perf context counters to track
      * number of keys returned in Get call, break down by levels
      * total processing time at each level during Get call
      Pull Request resolved: https://github.com/facebook/rocksdb/pull/4617
      
      Differential Revision: D12898024
      
      Pulled By: miasantreble
      
      fbshipit-source-id: 6b84ef1c8097c0d9e97bee1a774958f56ab4a6c4
      b3130193
  11. 13 11月, 2018 4 次提交
    • S
      Fix RocksDB Lite build (#4675) · 2993cd20
      Sagar Vemuri 提交于
      Summary:
      Our internal CI test caught RocksDB Lite build failures. The failures are due to a new test introduced in #4665 using `SSTFileWriter` and `IngestExternalFile`, but these is not exposed under lite mode. Fixed by #ifdef'ing out the test.
      
      ```
      db/db_test2.cc: In member function ‘virtual void rocksdb::DBTest2_TestCompactFiles_Test::TestBody()’:
      db/db_test2.cc:2907:3: error: ‘SstFileWriter’ is not a member of ‘rocksdb’
         rocksdb::SstFileWriter sst_file_writer{rocksdb::EnvOptions(), options};
         ^
      In file included from ./util/testharness.h:15:0,
                       from ./table/mock_table.h:23,
                       from ./db/db_test_util.h:44,
                       from db/db_test2.cc:13:
      db/db_test2.cc:2912:13: error: ‘sst_file_writer’ was not declared in this scope
         ASSERT_OK(sst_file_writer.Open(external_file1));
      ```
      Pull Request resolved: https://github.com/facebook/rocksdb/pull/4675
      
      Differential Revision: D13035984
      
      Pulled By: sagar0
      
      fbshipit-source-id: c1ceac550dfac1a85eeea436693dc7dd467519a6
      2993cd20
    • S
      Automatically set LITE=1 on passing OPT="-DROCKSDB_LITE" (#4671) · dd742e24
      Sagar Vemuri 提交于
      Summary:
      In #4652 we are setting -Os for lite builds only when LITE=1 is specified. But currently almost all the users invoke lite build via OPT="-DROCKSDB_LITE=1". So this diff tries to set LITE=1 when users already pass in -DROCKSDB_LITE=1 via the command line.
      Pull Request resolved: https://github.com/facebook/rocksdb/pull/4671
      
      Differential Revision: D13033801
      
      Pulled By: sagar0
      
      fbshipit-source-id: e7b506cee574f9e3f42221ee6647915011c78d78
      dd742e24
    • A
      Fix flaky DBDynamicLevelTest.DynamicLevelMaxBytesBase2 (#4668) · 7d04ef46
      Abhishek Madan 提交于
      Summary:
      Part of the test required that a compaction start before a
      manual flush, but this was not enforced by the test. In some cases,
      particularly when writing to tmpfs, this could lead to the compaction
      starting after the flush, which caused the base level to be higher than
      it was expected to be. Add a sync point in the test to ensure that the
      flush and compaction happen simultaneously.
      
      The test also had some stale comments, so those have been removed or
      modified, and the test has been simplified so that it no longer uses sleeps
      and writes uncompressed SSTs.
      Pull Request resolved: https://github.com/facebook/rocksdb/pull/4668
      
      Differential Revision: D13032440
      
      Pulled By: abhimadan
      
      fbshipit-source-id: 3f23b583a096454dafb8d8ea75678605dec80209
      7d04ef46
    • F
      Update history and version for future 5.18 release. · 050d7355
      Fosco Marotto 提交于
      Summary: Pull Request resolved: https://github.com/facebook/rocksdb/pull/4669
      
      Differential Revision: D13031522
      
      Pulled By: gfosco
      
      fbshipit-source-id: d09e655a9d5556594f195c5d1b786900932145ce
      050d7355