- 20 3月, 2019 2 次提交
-
-
由 Wenjie Yang 提交于
Summary: Add an option to filter out READ or WRITE operations while tracing. Pull Request resolved: https://github.com/facebook/rocksdb/pull/5082 Differential Revision: D14515083 Pulled By: mrmiywj fbshipit-source-id: 2504c89a9abf1dd629cad44b4104092702d77610
-
由 Hiroaki Nakamura 提交于
Summary: Partly addresses https://github.com/facebook/rocksdb/issues/4999 I verified `make static_lib` runs fine. Pull Request resolved: https://github.com/facebook/rocksdb/pull/5077 Differential Revision: D14521101 Pulled By: maysamyabandeh fbshipit-source-id: ba88e74a51d2d793cac7260d505b1a54254b53af
-
- 19 3月, 2019 1 次提交
-
-
由 Shobhit Dayal 提交于
Summary: This is a feature to sample data-block compressibility and and report them as stats. 1 in N (tunable) blocks is sampled for compressibility using two algorithms: 1. lz4 or snappy for fast compression 2. zstd or zlib for slow but higher compression. The stats are reported to the caller as raw-bytes and compressed-bytes. The block continues to be compressed for storage using the specified CompressionType. The db_bench_tool how has a command line option for specifying the sampling rate. It's default value is 0 (no sampling). To test the overhead for a certain value, users can compare the performance of db_bench_tool, varying the sampling rate. It is unlikely to have a noticeable impact for high values like 20. Pull Request resolved: https://github.com/facebook/rocksdb/pull/4842 Differential Revision: D13629011 Pulled By: shobhitdayal fbshipit-source-id: 14ca668bcab6499b2a1734edf848eb62a4f4fafa
-
- 16 3月, 2019 1 次提交
-
-
由 anand76 提交于
Summary: There is a potential failure case in DBImpl::SwitchMemtable() that is not handled properly. The call to cur_log_writer->WriteBuffer() can fail due to an IO error. In that case, we need to call SetBGError() in order set the background error since the WriteBuffer() failure may result in data loss. Also, the asserts for !new_mem and !new_log are incorrect, as those would have been allocated by the time this failure is detected. Pull Request resolved: https://github.com/facebook/rocksdb/pull/5072 Differential Revision: D14461384 Pulled By: anand1976 fbshipit-source-id: fb59bce9d61378f37d2dfcd28c0b704b0f43c3cf
-
- 02 3月, 2019 2 次提交
-
-
由 Siying Dong 提交于
Summary: This reverts commit ee181808. We are not ready to deprecate this feature. revert it for now. Pull Request resolved: https://github.com/facebook/rocksdb/pull/5034 Differential Revision: D14287246 Pulled By: siying fbshipit-source-id: e4beafdeaee1c94364fdaa6ba198218d158339f7
-
由 Siying Dong 提交于
Summary: Right now, users can change statistics.stats_level while DB is running, but TSAN may report data race. We make stats_level_ to be atomic, and access them using accessors. Pull Request resolved: https://github.com/facebook/rocksdb/pull/5030 Differential Revision: D14267519 Pulled By: siying fbshipit-source-id: 37d7ebeff7a43a406230143422a16af899163f73
-
- 01 3月, 2019 2 次提交
-
-
由 Maysam Yabandeh 提交于
Summary: PreReleaseCallback meant to be called before the writes are visible to the readers. Since the sequence number is known after the WAL write, there is no reason to delay calling PreReleaseCallback to after the memtable write, which would complicates the reader's logic in presence of our memtable writes that are made visible by the other write thread. Pull Request resolved: https://github.com/facebook/rocksdb/pull/5015 Differential Revision: D14221670 Pulled By: maysamyabandeh fbshipit-source-id: a504dd665cf923226d7af09cc8e9c7739a25edc6
-
由 Siying Dong 提交于
Summary: Statistics cost too much CPU for some use cases. Add two stats levels so that people can choose to skip two types of expensive stats, timers and histograms. Pull Request resolved: https://github.com/facebook/rocksdb/pull/5027 Differential Revision: D14252765 Pulled By: siying fbshipit-source-id: 75ecec9eaa44c06118229df4f80c366115346592
-
- 27 2月, 2019 1 次提交
-
-
由 Maysam Yabandeh 提交于
Summary: The read path includes a callback function, ReadCallback, which would eventually calls IsInSnapshot to figure if a particular seq is in the reading snapshot or not. This callback is virtual, which adds the cost of multiple virtual function call to each read. The first few checks in IsInSnapshot, however, are quite trivial and take care of majority of the cases. The patch moves those to a non-virtual function in the the parent class, ReadCallback, to lower the virtual callback cost. Pull Request resolved: https://github.com/facebook/rocksdb/pull/5018 Differential Revision: D14226562 Pulled By: maysamyabandeh fbshipit-source-id: 6feed5b34f3b082e52092c5ef143e29b49c46b44
-
- 21 2月, 2019 1 次提交
-
-
由 Zhongyi Xie 提交于
Summary: This PR adds public `GetStatsHistory` API to retrieve stats history in the form of an std map. The key of the map is the timestamp in microseconds when the stats snapshot is taken, the value is another std map from stats name to stats value (stored in std string). Two DBOptions are introduced: `stats_persist_period_sec` (default 10 minutes) controls the intervals between two snapshots are taken; `max_stats_history_count` (default 10) controls the max number of history snapshots to keep in memory. RocksDB will stop collecting stats snapshots if `stats_persist_period_sec` is set to 0. (This PR is the in-memory part of https://github.com/facebook/rocksdb/pull/4535) Pull Request resolved: https://github.com/facebook/rocksdb/pull/4748 Differential Revision: D13961471 Pulled By: miasantreble fbshipit-source-id: ac836d401ecb84ea92216bf9966f969dedf4ad04
-
- 20 2月, 2019 3 次提交
-
-
由 Siying Dong 提交于
Summary: DBTest2.PresetCompressionDict is flaky. Temparily disable it for now. Pull Request resolved: https://github.com/facebook/rocksdb/pull/5003 Differential Revision: D14139505 Pulled By: siying fbshipit-source-id: ebf1872d364b76b2cb021b489ea2f17ee997116a
-
由 Michael Liu 提交于
Summary: Use C++11’s override and remove virtual where applicable. Change are automatically generated. bypass-lint drop-conflicts Reviewed By: igorsugak Differential Revision: D14131816 fbshipit-source-id: f20e7f7cecf2e699d70f5fa036f72c0e3f59b50e
-
由 Zhongyi Xie 提交于
Summary: MyRocks calls `GetForUpdate` on `INSERT`, for unique key check, and in almost all cases GetForUpdate returns empty result. For such cases, whole key bloom filter is helpful. Pull Request resolved: https://github.com/facebook/rocksdb/pull/4985 Differential Revision: D14118257 Pulled By: miasantreble fbshipit-source-id: d35cb7109c62fd5ad541a26968e3a3e16d3e85ea
-
- 16 2月, 2019 3 次提交
-
-
由 Siying Dong 提交于
Summary: The info log header feature never worked well, because log level Header was not translated to Logger::LogHeader() call. Fix it. Pull Request resolved: https://github.com/facebook/rocksdb/pull/4980 Differential Revision: D14087283 Pulled By: siying fbshipit-source-id: 7e7d03ce35fa8d13d4ee549f46f7326f7bc0006d
-
由 Siying Dong 提交于
Summary: Right now when a flush is triggered, the memory consumption is logged but data size is not. It's useful to log both when we debug unexpected small flushed file size. Pull Request resolved: https://github.com/facebook/rocksdb/pull/4979 Differential Revision: D14071979 Pulled By: siying fbshipit-source-id: 0cd60449c5205eb00e0fbc299084418f609904ed
-
由 Aubin Sanyal 提交于
Summary: We introduced ttl option in CompactionOptionsFIFO when ttl-based file deletion (compaction) was supported only as part of FIFO Compaction. But with the extension of ttl semantics even to Level compaction, CompactionOptionsFIFO.ttl can now be deprecated. Instead we will start using ColumnFamilyOptions.ttl for FIFO compaction as well. Pull Request resolved: https://github.com/facebook/rocksdb/pull/4965 Differential Revision: D14072960 Pulled By: sagar0 fbshipit-source-id: c98cc2ae695a28136295787cd88d36a220fc219e
-
- 15 2月, 2019 2 次提交
-
-
由 Michael Liu 提交于
Summary: Use C++11’s override and remove virtual where applicable. Change are automatically generated. Reviewed By: Orvid Differential Revision: D14090024 fbshipit-source-id: 1e9432e87d2657e1ff0028e15370a85d1739ba2a
-
由 Andrew Kryczka 提交于
Summary: If `CompressionOptions::max_dict_bytes` and/or `CompressionOptions::zstd_max_train_bytes` are set, `SstFileWriter` will now generate files respecting those options. I refactored the logic a bit for deciding when to use dictionary compression. Previously we plumbed `is_bottommost_level` down to the table builder and used that. However it was kind of confusing in `SstFileWriter`'s context since we don't know what level the file will be ingested to. Instead, now the higher-level callers (e.g., flush, compaction, file writer) are responsible for building the right `CompressionOptions` to give the table builder. Pull Request resolved: https://github.com/facebook/rocksdb/pull/4978 Differential Revision: D14060763 Pulled By: ajkr fbshipit-source-id: dc802c327896df2b319dc162d6acc82b9cdb452a
-
- 14 2月, 2019 1 次提交
-
-
由 Yanqin Jin 提交于
Summary: if an operation just involves a single column family, then we do not have to set the kInAtomicGroup tag when writing to MANIFEST. This change can fix a compatibility test failure, i.e. 5.15 and earlier cannot recognize kInAtomicGroup tag. Pull Request resolved: https://github.com/facebook/rocksdb/pull/4981 Differential Revision: D14072687 Pulled By: riversand963 fbshipit-source-id: 46b0c61e399f16c6b7169de0b33430d0ed90d6d4
-
- 13 2月, 2019 3 次提交
-
-
由 Yanqin Jin 提交于
Summary: Make file ingestion atomic. as title. Ingesting external SST files into multiple column families should be atomic. If a crash occurs and db reopens, either all column families have successfully ingested the files before the crash, or non of the ingestions have any effect on the state of the db. Also add unit tests for atomic ingestion. Note that the unit test here does not cover the case of incomplete atomic group in the MANIFEST, which is covered in VersionSetTest already. Pull Request resolved: https://github.com/facebook/rocksdb/pull/4895 Differential Revision: D13718245 Pulled By: riversand963 fbshipit-source-id: 7df97cc483af73ad44dd6993008f99b083852198
-
由 Siying Dong 提交于
Summary: Previously, stats were logged in warning level. This was done in that way because people reported that it wasn't logged in MyRocks. However, later we learned that it turns out to be due to a bug in MyRocks, which is fixed in https://github.com/facebook/mysql-5.6/commit/79bb705e74b239d7030b724ea6bbd635eceec531 Now we revert the stats logging to INFO level, so that it doesn't pollute the warning level logging. Pull Request resolved: https://github.com/facebook/rocksdb/pull/4977 Differential Revision: D14058485 Pulled By: siying fbshipit-source-id: 19fab323c19d9bc88184287f209551f9a77ca0e6
-
由 Yanqin Jin 提交于
Summary: In `DBImpl::AtomicFlushMemTablesToOutputFiles`, we need to call fsync only once on the same data directory. If two column families share a common directory for their data, we call fsync only once. Pull Request resolved: https://github.com/facebook/rocksdb/pull/4817 Differential Revision: D13543689 Pulled By: riversand963 fbshipit-source-id: 4701d77c96a47802fbf6cb9f3337ee65d46b95f5
-
- 12 2月, 2019 4 次提交
-
-
由 Andrew Kryczka 提交于
Summary: Our previous approach was to train one compression dictionary per compaction, using the first output SST to train a dictionary, and then applying it on subsequent SSTs in the same compaction. While this was great for minimizing CPU/memory/I/O overhead, it did not achieve good compression ratios in practice. In our most promising potential use case, moderate reductions in a dictionary's scope make a major difference on compression ratio. So, this PR changes compression dictionary to be scoped per-SST. It accepts the tradeoff during table building to use more memory and CPU. Important changes include: - The `BlockBasedTableBuilder` has a new state when dictionary compression is in-use: `kBuffered`. In that state it accumulates uncompressed data in-memory whenever `Add` is called. - After accumulating target file size bytes or calling `BlockBasedTableBuilder::Finish`, a `BlockBasedTableBuilder` moves to the `kUnbuffered` state. The transition (`EnterUnbuffered()`) involves sampling the buffered data, training a dictionary, and compressing/writing out all buffered data. In the `kUnbuffered` state, a `BlockBasedTableBuilder` behaves the same as before -- blocks are compressed/written out as soon as they fill up. - Samples are now whole uncompressed data blocks, except the final sample may be a partial data block so we don't breach the user's configured `max_dict_bytes` or `zstd_max_train_bytes`. The dictionary trainer is supposed to work better when we pass it real units of compression. Previously we were passing 64-byte KV samples which was not realistic. Pull Request resolved: https://github.com/facebook/rocksdb/pull/4952 Differential Revision: D13967980 Pulled By: ajkr fbshipit-source-id: 82bea6f7537e1529c7a1a4cdee84585f5949300f
-
由 Maysam Yabandeh 提交于
Summary: If IsInSnapshot(seq2, snapshot) determines that the snapshot is released, the future queries IsInSnapshot(seq1, snapshot) could still return a definitive answer of true if for example seq1 is too old that is determined visible in all snapshots. This violates a recently added assert statement to compaction iterator. The patch relaxes the assert. Pull Request resolved: https://github.com/facebook/rocksdb/pull/4969 Differential Revision: D14030998 Pulled By: maysamyabandeh fbshipit-source-id: 6db53db0e37d0a20e8997ef2c1004b8627614ab9
-
由 Yanqin Jin 提交于
Summary: Always enable properties block checksum verification for block-based table. For external SST file ingested with 'write_global_seqno==true', we use 'DecodeEntrySlow' to parse its blocks' contents so that the process will not die upon failing the assertion possibly caused by corruption. Pull Request resolved: https://github.com/facebook/rocksdb/pull/4956 Differential Revision: D14012741 Pulled By: riversand963 fbshipit-source-id: 8b766e6f54b36f8f9e074c0e19e0926ec3cce186
-
由 Siying Dong 提交于
Summary: https://github.com/facebook/rocksdb/pull/4960 introduced ignorable manfiest record. Adding a test to it. Pull Request resolved: https://github.com/facebook/rocksdb/pull/4964 Differential Revision: D14012667 Pulled By: siying fbshipit-source-id: e5f10ecc68dec2716e178d44f0fe2b76c3d857ef
-
- 09 2月, 2019 2 次提交
-
-
由 tang-jianfeng 提交于
Summary: Implement trace sampling to allow user to specify the sampling frequency, i.e. save one per how many requests, so that a user does not need to log all if he/she is interested in only a sampled set. Pull Request resolved: https://github.com/facebook/rocksdb/pull/4963 Differential Revision: D14011190 Pulled By: tang-jianfeng fbshipit-source-id: 078b631d9319b67cb089dd2c30e21d0df8dc406a
-
由 Siying Dong 提交于
Summary: We want to reserve some right that some extra information added manifest in the future can be forward compatible by previous versions. Now we create a place holder for that. A bit in tag is added to indicate that a field can be safely ignored. Pull Request resolved: https://github.com/facebook/rocksdb/pull/4960 Differential Revision: D14000484 Pulled By: siying fbshipit-source-id: cbf5bad3f9d5ec798f789806f244d1c20d3b66d6
-
- 08 2月, 2019 2 次提交
-
-
由 Siying Dong 提交于
Summary: We found that the behavior of CompactionFilter::IgnoreSnapshots() = false isn't what we have expected. We thought that snapshot will always be preserved. However, we just realized that, if no snapshot is created while compaction starts, and a snapshot is created after that, the data seen from the snapshot can successfully be dropped by the compaction. This creates a strange behavior to the feature, which is hard to explain. Like what is documented in code comment, this feature is not very useful with snapshot anyway. The decision is to deprecate the feature. We keep the function to avoid to break users code. However, we will fail compactions if false is returned. Pull Request resolved: https://github.com/facebook/rocksdb/pull/4954 Differential Revision: D13981900 Pulled By: siying fbshipit-source-id: 2db8c2c3865acd86a28dca625945d1481b1d1e36
-
由 Siying Dong 提交于
Summary: Cuckoo Hash is less useful than we initially expected. Remove it. Pull Request resolved: https://github.com/facebook/rocksdb/pull/4953 Differential Revision: D13979264 Pulled By: siying fbshipit-source-id: 2a60afdaa989f045357398b43a1cc5d46f4492ed
-
- 06 2月, 2019 5 次提交
-
-
由 Zhongyi Xie 提交于
Summary: This will fix the following build error: > db/db_test.cc: In member function ‘virtual void rocksdb::DBTest_CompactFilesShouldTriggerAutoCompaction_Test::TestBody()’: > db/db_test.cc:5462:8: error: ‘class rocksdb::DB’ has no member named ‘GetColumnFamilyMetaData’ > db_->GetColumnFamilyMetaData(db_->DefaultColumnFamily(), &cf_meta_data); > db/db_test.cc:5490:8: error: ‘class rocksdb::DB’ has no member named ‘GetColumnFamilyMetaData’ > db_->GetColumnFamilyMetaData(db_->DefaultColumnFamily(), &cf_meta_data); > db/db_test.cc:5499:8: error: ‘class rocksdb::DB’ has no member named ‘GetColumnFamilyMetaData’ > db_->GetColumnFamilyMetaData(db_->DefaultColumnFamily(), &cf_meta_data); Pull Request resolved: https://github.com/facebook/rocksdb/pull/4950 Differential Revision: D13965378 Pulled By: miasantreble fbshipit-source-id: a975435476fe555b1cd9d5da263ee3da3acdea56
-
由 Zhongyi Xie 提交于
Summary: Existing implementation of PerfContext does not define copy constructor or assignment operator, which could potentially cause problems when user create copies and resets the builtin one. This PR address the issue by providing these two constructors with deep copy semantics. Pull Request resolved: https://github.com/facebook/rocksdb/pull/4919 Differential Revision: D13960406 Pulled By: miasantreble fbshipit-source-id: 36aab5aaee65d4480f537e4e22148faa45e8e334
-
由 Jay Zhuang 提交于
Summary: CompactFiles() may block auto compaction which could cuase DB hang when it reachs level0_stop_writes_trigger. Pull Request resolved: https://github.com/facebook/rocksdb/pull/4940 Differential Revision: D13929648 Pulled By: cooldoger fbshipit-source-id: 10842df38df3bebf862cd1a120a88ce961fdd381
-
由 Siying Dong 提交于
Summary: In NotFound cases, stats BYTES_READ and perf_context.get_read_bytes is still be increased. The amount increased will be whatever size of the string or PinnableSlice that users passed in as the output data structure. This is wrong. Fix this by not increasing these two counters. Pull Request resolved: https://github.com/facebook/rocksdb/pull/4938 Differential Revision: D13908963 Pulled By: siying fbshipit-source-id: 60bce42e4fbb9862bba3da36dbc27b2963ea6162
-
由 yangzhijia 提交于
Summary: Fix the ouput overlap bug when using subcompactions, the upper bound of output file was extended incorrectly. Pull Request resolved: https://github.com/facebook/rocksdb/pull/4898 Differential Revision: D13736107 Pulled By: ajkr fbshipit-source-id: 21dca09f81d5f07bf2766bf566f9b50dcab7d8e3
-
- 02 2月, 2019 2 次提交
-
-
由 Maysam Yabandeh 提交于
Summary: The patch fixes the following analyze error by checking the return status of ParseInternalKey. ``` db/merge_helper.cc:306:23: warning: The right operand of '==' is a garbage value assert(kTypeMerge == orig_ikey.type); ``` Pull Request resolved: https://github.com/facebook/rocksdb/pull/4937 Differential Revision: D13908506 Pulled By: maysamyabandeh fbshipit-source-id: 68d7771e75519da3d4bd807fd231675ec12093f6
-
由 Ming Zhao 提交于
Summary: Pull Request resolved: https://github.com/facebook/rocksdb/pull/4927 Differential Revision: D13889458 Pulled By: mzhaom fbshipit-source-id: d6b66db85901a9eb90748fba6a9dc4e7457b9c5e
-
- 01 2月, 2019 2 次提交
-
-
由 Yanqin Jin 提交于
Summary: 1. this commit fixes our handling of a combination of two separate edge cases. If a flush job does not pick any memtable to flush (because another flush job has already picked the same memtables), and the column family assigned to the flush job is dropped right before RocksDB calls rocksdb::InstallMemtableAtomicFlushResults, our original code passes a FileMetaData object whose file number is 0, failing the assertion in rocksdb::InstallMemtableAtomicFlushResults (assert(m->GetFileNumber() > 0)). 2. Also piggyback a small change: since we already create a local copy of column family's mutable CF options to eliminate potential race condition with `SetOptions` call, we might as well use the local copy in other function calls in the same scope. Pull Request resolved: https://github.com/facebook/rocksdb/pull/4932 Differential Revision: D13901322 Pulled By: riversand963 fbshipit-source-id: b936580af7c127ea0c6c19ea10cd5fcede9fb0f9
-
由 Maysam Yabandeh 提交于
Summary: FlushMemTablesToOutputFiles calls FlushMemTableToOutputFile for each column family. The patch moves the take-snapshot logic to outside FlushMemTableToOutputFile so that it does it once for all the flushes. This also addresses a deadlock issue for resetting the managed snapshot of job_snapshot in the 2nd call to FlushMemTableToOutputFile. Pull Request resolved: https://github.com/facebook/rocksdb/pull/4934 Differential Revision: D13900747 Pulled By: maysamyabandeh fbshipit-source-id: f3cd650c5fff24cf95c1aaf8a10c149d42bf042c
-
- 30 1月, 2019 1 次提交
-
-
由 Alexander Zinoviev 提交于
Summary: Measure CPU time consumed for a compaction and report it in the stats report Enable NowCPUNanos() to work for MacOS Pull Request resolved: https://github.com/facebook/rocksdb/pull/4889 Differential Revision: D13701276 Pulled By: zinoale fbshipit-source-id: 5024e5bbccd4dd10fd90d947870237f436445055
-