- 03 6月, 2017 5 次提交
-
-
由 Aaron Gao 提交于
Summary: … headers https://github.com/facebook/rocksdb/pull/2199 should not reference RocksDB-specific macros (like ROCKSDB_SUPPORT_THREAD_LOCAL in this case) to public headers, `iostats_context.h` and `perf_context.h`. We shouldn't do that because users have to provide these compiler flags when building their binary with RocksDB. We should hide the thread local global variable inside our implementation and just expose a function api to retrieve these variables. It may break some users for now but good for long term. make check -j64 Closes https://github.com/facebook/rocksdb/pull/2380 Differential Revision: D5177896 Pulled By: lightmark fbshipit-source-id: 6fcdfac57f2e2dcfe60992b7385c5403f6dcb390
-
由 Mike Kolupaev 提交于
Summary: Fixes the following scenario: 1. Set prefix extractor. Enable bloom filters, with `whole_key_filtering = false`. Use compaction filter that sometimes returns `kRemoveAndSkipUntil`. 2. Do a compaction. 3. Compaction creates an iterator with `total_order_seek = false`, calls `SeekToFirst()` on it, then repeatedly calls `Next()`. 4. At some point compaction filter returns `kRemoveAndSkipUntil`. 5. Compaction calls `Seek(skip_until)` on the iterator. The key that it seeks to happens to have prefix that doesn't match the bloom filter. Since `total_order_seek = false`, iterator becomes invalid, and compaction thinks that it has reached the end. The rest of the compaction input is silently discarded. The fix is to make compaction iterator use `total_order_seek = true`. The implementation for PlainTable is quite awkward. I've made `kRemoveAndSkipUntil` officially incompatible with PlainTable. If you try to use them together, compaction will fail, and DB will enter read-only mode (`bg_error_`). That's not a very graceful way to communicate a misconfiguration, but the alternatives don't seem worth the implementation time and complexity. To be able to check in advance that `kRemoveAndSkipUntil` is not going to be used with PlainTable, we'd need to extend the interface of either `CompactionFilter` or `InternalIterator`. It seems unlikely that anyone will ever want to use `kRemoveAndSkipUntil` with PlainTable: PlainTable probably has very few users, and `kRemoveAndSkipUntil` has only one user so far: us (logdevice). Closes https://github.com/facebook/rocksdb/pull/2349 Differential Revision: D5110388 Pulled By: lightmark fbshipit-source-id: ec29101a99d9dcd97db33923b87f72bce56cc17a
-
由 Siying Dong 提交于
Summary: Improve write buffer manager in several ways: 1. Size is tracked when arena block is allocated, rather than every allocation, so that it can better track actual memory usage and the tracking overhead is slightly lower. 2. We start to trigger memtable flush when 7/8 of the memory cap hits, instead of 100%, and make 100% much harder to hit. 3. Allow a cache object to be passed into buffer manager and the size allocated by memtable can be costed there. This can help users have one single memory cap across block cache and memtable. Closes https://github.com/facebook/rocksdb/pull/2350 Differential Revision: D5110648 Pulled By: siying fbshipit-source-id: b4238113094bf22574001e446b5d88523ba00017
-
由 Andrew Kryczka 提交于
Summary: Some users want to monitor column family activity in their custom memtable implementations. Previously there was no way to figure out with which column family a memtable is associated. This diff: - adds an overload to MemTableRepFactory::CreateMemTableRep() that provides the CF ID. For compatibility, its default implementation calls the old overload. - updates MemTable to create MemTableRep's using the new overload. Closes https://github.com/facebook/rocksdb/pull/2346 Differential Revision: D5108061 Pulled By: ajkr fbshipit-source-id: 3a1921214a348dd8ea0f54e1cab3b71c3d46d616
-
由 Yi Wu 提交于
Summary: rocksdb::Random is not thread-safe. Have one Random for each thread instead. Closes https://github.com/facebook/rocksdb/pull/2400 Differential Revision: D5173919 Pulled By: yiwu-arbug fbshipit-source-id: 1a99c7b877f3893eb22355af49e321bcad4e53e6
-
- 02 6月, 2017 8 次提交
-
-
由 Andrew Kryczka 提交于
Summary: The range deletion meta-block iterators weren't getting cleaned up properly since they don't support arena allocation. I didn't implement arena support since, in the general case, each iterator is used only once and separately from all other iterators, so there should be no benefit to data locality. Anyways, this diff fixes up #2370 by treating range deletion iterators as non-arena-allocated. Closes https://github.com/facebook/rocksdb/pull/2399 Differential Revision: D5171119 Pulled By: ajkr fbshipit-source-id: bef6f5c4c5905a124f4993945aed4bd86e2807d8
-
由 Andrew Kryczka 提交于
Summary: also changed the `>` in the comparison against `level0_file_num_compaction_trigger` into a `>=` since exactly `level0_file_num_compaction_trigger` can trigger a compaction from L0. Closes https://github.com/facebook/rocksdb/pull/2179 Differential Revision: D4915772 Pulled By: ajkr fbshipit-source-id: e38fec6253de6f9a40e61734615c6670d84038aa
-
由 Andrew Gallagher 提交于
Reviewed By: igorsugak fbshipit-source-id: 906b744c179eb932f5a388b39f93209cecd50a80
-
由 Aaron Gao 提交于
Summary: For level_compaction_style regression test. Closes https://github.com/facebook/rocksdb/pull/2397 Differential Revision: D5168545 Pulled By: lightmark fbshipit-source-id: 195e4d84917e7c261d9f4fbe9aee5d104c9cb9a2
-
https://github.com/facebook/rocksdb/pull/2082由 Maysam Yabandeh 提交于
Summary: This is a manual commit of this PR: Retire InMemoryEnv in favor of MockEnv #2082 With MockEnv doing the same yet being more mature, InMemoryEnv is redundant. Reviewed By: IslamAbdelRahman Differential Revision: D5162323 fbshipit-source-id: 59fd0082a891dc99cc531e4da9d68bf891eae3f5
-
由 Islam AbdelRahman 提交于
-
由 Volker Mische 提交于
Summary: When the `TwoLevelIndexSearch` was introduced, it wasn't added to the C-API. Closes https://github.com/facebook/rocksdb/pull/2395 Differential Revision: D5165127 Pulled By: maysamyabandeh fbshipit-source-id: d077f16ab5646c18158d8202a33b0fd076c6c8ad
-
由 Daniel Black 提交于
Summary: Use later xcode version from https://docs.travis-ci.com/user/osx-ci-environment Closes https://github.com/facebook/rocksdb/pull/2128 Differential Revision: D4907471 Pulled By: yiwu-arbug fbshipit-source-id: debf8e27baef71a5833c845401b1865bc75ac977
-
- 01 6月, 2017 4 次提交
-
-
由 Tamir Duberstein 提交于
Summary: This fixes a compilation failure on Linux when the system libc is not glibc. jemalloc's configure script incorrectly assumes that glibc is always used on Linux systems, producing glibc-style signatures; when the system libc is e.g. musl, the following error is observed: ``` [ 0%] Building CXX object CMakeFiles/rocksdb.dir/db/db_impl.cc.o In file included from /go/src/github.com/cockroachdb/cockroach/c-deps/rocksdb.src/table/block.h:19:0, from /go/src/github.com/cockroachdb/cockroach/c-deps/rocksdb.src/db/db_impl.cc:77: /x-tools/x86_64-unknown-linux-musl/x86_64-unknown-linux-musl/sysroot/usr/include/malloc.h:19:8: error: declaration of 'size_t malloc_usable_size(void*)' has a different exception specifier size_t malloc_usable_size(void *); ^~~~~~~~~~~~~~~~~~ In file included from /go/src/github.com/cockroachdb/cockroach/c-deps/rocksdb.src/db/db_impl.cc:20:0: /go/native/x86_64-unknown-linux-musl/jemalloc/include/jemalloc/jemalloc.h:78:33: note: from previous declaration 'size_t malloc_usable_size(void*) throw ()' # define je_malloc_usable_size malloc_usable_size ^ /go/native/x86_64-unknown-linux-musl/jemalloc/include/jemalloc/jemalloc.h:239:41: note: in expansion of macro 'je_malloc_usable_size' JEMALLOC_EXPORT size_t JEMALLOC_NOTHROW je_malloc_usable_size( ^~~~~~~~~~~~~~~~~~~~~ CMakeFiles/rocksdb.dir/build.make:350: recipe for target 'CMakeFiles/rocksdb.dir/db/db_impl.cc.o' failed ``` This works around the issue by rearranging the sources such that jemalloc's headers are never in the same scope as the system's malloc header. The jemalloc issue has been reported as well, see: https://github.com/jemalloc/jemalloc/issues/778. cc tschottdorf Closes https://github.com/facebook/rocksdb/pull/2188 Differential Revision: D5163048 Pulled By: siying fbshipit-source-id: c553125458892def175c1be5682b0330d80b2a0d
-
由 Aaron Gao 提交于
Summary: fix regression test by not reporting stats when building db Closes https://github.com/facebook/rocksdb/pull/2390 Differential Revision: D5159909 Pulled By: lightmark fbshipit-source-id: c3f4b9deb9c6799ff84207fd341c529144f8158d
-
由 Andrew Kryczka 提交于
Summary: Previously we returned NotSupported when ingesting files into a database containing any range deletions. This diff adds the support. - Flush if any memtable contains range deletions overlapping the to-be-ingested file - Place to-be-ingested file before any level that contains range deletions overlapping it. - Added support for `Version` to return iterators over range deletions in a given level. Previously, we piggybacked getting range deletions onto `Version`'s `Get()` / `AddIterator()` functions by passing them a `RangeDelAggregator*`. But file ingestion needs to get iterators over range deletions, not populate an aggregator (since the aggregator does collapsing and doesn't expose the actual ranges). Closes https://github.com/facebook/rocksdb/pull/2370 Differential Revision: D5127648 Pulled By: ajkr fbshipit-source-id: 816faeb9708adfa5287962bafdde717db56e3f1a
-
由 Yi Wu 提交于
Summary: Blob db rely on base db returning sequence number through write batch after DB::Write(). However after recent changes to the write path, DB::Writ()e no longer return sequence number in some cases. Fixing it by have WriteBatchInternal::InsertInto() always encode sequence number into write batch. Stacking on #2375. Closes https://github.com/facebook/rocksdb/pull/2385 Differential Revision: D5148358 Pulled By: yiwu-arbug fbshipit-source-id: 8bda0aa07b9334ed03ed381548b39d167dc20c33
-
- 31 5月, 2017 5 次提交
-
-
由 Siying Dong 提交于
Summary: Add a histogram in statistics to help users understand how many merge operands they merge. Closes https://github.com/facebook/rocksdb/pull/2373 Differential Revision: D5139983 Pulled By: siying fbshipit-source-id: 61b9ba8ca83f358530a4833d68f0103b56a0e182
-
由 Yi Wu 提交于
Summary: Re-enable blob_db_test with some update: * Commented out delay at the end of GC tests. Will update the logic later with sync point to properly trigger GC. * Added some helper functions. Also update make files to include blob_dump tool. Closes https://github.com/facebook/rocksdb/pull/2375 Differential Revision: D5133793 Pulled By: yiwu-arbug fbshipit-source-id: 95470b26d0c1f9592ba4b7637e027fdd263f425c
-
由 Aaron Gao 提交于
Summary: abandon fillseqdeterministic test locally Closes https://github.com/facebook/rocksdb/pull/2290 Differential Revision: D5151867 Pulled By: lightmark fbshipit-source-id: 4c8a24cc937212ffb5ceb9bfaf7288eb8726d0c1
-
由 Tamir Duberstein 提交于
Summary: yiwu-arbug see individual commits. Closes https://github.com/facebook/rocksdb/pull/2318 Differential Revision: D5141520 Pulled By: yiwu-arbug fbshipit-source-id: 7987c92ab4461eef36afce5a133d3a0ee0c96300
-
由 Tamir Duberstein 提交于
Summary: This collapses all the "platform dependent" tests into a single travis builder in an effort to reduce overall CI times. These builds currently take a combined 21-23 minutes, but each one has to compile the library, so combining them should yield some time savings (5-10 minutes). Unfortunately the other builders don't duplicate work, so combining them is unlikely to provide benefit. Closes https://github.com/facebook/rocksdb/pull/2306 Differential Revision: D5147850 Pulled By: yiwu-arbug fbshipit-source-id: d947dc8b9f49639fe22f3c8ab9a82a8d730ddddf
-
- 27 5月, 2017 7 次提交
-
-
由 Sagar Vemuri 提交于
Summary: 5.4.5 release blog post. Tag created at: https://github.com/facebook/rocksdb/releases/tag/rocksdb-5.4.5 Closes https://github.com/facebook/rocksdb/pull/2378 Differential Revision: D5141648 Pulled By: sagar0 fbshipit-source-id: 43faadfbd7feb8bb01c3d56127ef3453e23dc28a
-
由 Sagar Vemuri 提交于
Summary: Adding my name to the authors list so that I can publish a post to rocksdb blog (rocksdb.org). Closes https://github.com/facebook/rocksdb/pull/2379 Differential Revision: D5143582 Pulled By: sagar0 fbshipit-source-id: d85163f8b59aaeb07ac2a1cdd776ae335c7062b9
-
由 赵星宇 提交于
Summary: Closes https://github.com/facebook/rocksdb/pull/2377 Differential Revision: D5141274 Pulled By: lightmark fbshipit-source-id: c237a285b73ad93488c080ea80c71a29a17f1be0
-
由 Aaron Gao 提交于
Summary: Previously sst_file_writer only supports kTypeValue, we need kTypeMerge and kTypeDeletion also as user requested. Closes https://github.com/facebook/rocksdb/pull/2361 Differential Revision: D5139402 Pulled By: lightmark fbshipit-source-id: 092a60756d01692539d817a3765ebfd58a8d7f88
-
由 Siying Dong 提交于
Summary: Reorder variables of ReadOptions so that its size is reduced from 64 to 48 bytes. Closes https://github.com/facebook/rocksdb/pull/2366 Differential Revision: D5124043 Pulled By: siying fbshipit-source-id: 70e9c204c34f97fad011f2fe2297ba292d85df7a
-
由 Sagar Vemuri 提交于
Summary: This log message shouldn't be a warning; some services are seeing high warning count due to this. The count for the below line is a few hundreds of millions, as per Logview: ``` [rocksdb/src/db/column_family.cc:729] [checkpoints] Increasing compaction threads because we have 2 level-0 files ``` Closes https://github.com/facebook/rocksdb/pull/2364 Differential Revision: D5123565 Pulled By: sagar0 fbshipit-source-id: a07ce499a4f82f0ebde9cda9f4948fb9df6a734c
-
由 Sagar Vemuri 提交于
Summary: zstd files are downloaded and used as part of JNI build, but are left behind even after doing a `make clean`. This PR updates the `clean` target to remove these zstd files as well. Closes https://github.com/facebook/rocksdb/pull/2365 Differential Revision: D5123537 Pulled By: sagar0 fbshipit-source-id: a8f355da5ba961aa89d5852e35751ffc35de03ea
-
- 26 5月, 2017 3 次提交
-
-
由 Andrew Gallagher 提交于
Reviewed By: meyering Differential Revision: D5094332 fbshipit-source-id: 3df2f693def8ca418bc9febe3e20ccf051f2e19d
-
由 Yi Wu 提交于
Summary: Lite build does not recognize FLAGS_use_blob_db. Fixing it. Closes https://github.com/facebook/rocksdb/pull/2372 Reviewed By: anirbanr-fb Differential Revision: D5130773 Pulled By: yiwu-arbug fbshipit-source-id: 43131d9d0be5811f2129af562be72cca26369cb3
-
由 Aaron Gao 提交于
Summary: rand_ has data race risk TEST_TMPDIR=\/dev\/shm\/rocksdb OPT=-g COMPILE_WITH_TSAN=1 CRASH_TEST_KILL_ODD=1887 make J=1 crash_test Closes https://github.com/facebook/rocksdb/pull/2368 Differential Revision: D5127424 Pulled By: lightmark fbshipit-source-id: b7f4d1430a5769b57da9f99037106749264b2ced
-
- 25 5月, 2017 7 次提交
-
-
由 Andrew Kryczka 提交于
Summary: stop calling Close() at the end of tests holding a compaction pressure token since it causes the write controller to be deleted while it's still needed. these calls were pointless anyways since Close() is already called in the test's destructor. Closes https://github.com/facebook/rocksdb/pull/2367 Differential Revision: D5125906 Pulled By: ajkr fbshipit-source-id: 6cad8673e5546a82ff602ac0ba59cc3f68dbde46
-
由 Daniel Black 提交于
Summary: just realised when I updated the .travis.yml to trusty the llvm repo was still precise. Update this and clang-4.0. Closes https://github.com/facebook/rocksdb/pull/2127 Differential Revision: D4869427 Pulled By: sagar0 fbshipit-source-id: b7f906b6fac28e60cacc6a1f1959d6acf8269906
-
由 Aaron Gao 提交于
Summary: update targets file for release Closes https://github.com/facebook/rocksdb/pull/2358 Differential Revision: D5115705 Pulled By: lightmark fbshipit-source-id: 96a3c7e15b5807b5d0f5a9bb73850b92754b5794
-
由 Andrew Kryczka 提交于
Summary: - `max_background_flushes` and `max_background_compactions` are still supported for backwards compatibility - `base_background_compactions` is completely deprecated. Now we just throttle to one background compaction when there's no pressure. - `max_background_jobs` is added to automatically partition the concurrent background jobs into flushes vs compactions. Currently it's very simple as we just allocate one-fourth of the jobs to flushes, and the remaining can be used for compactions. - The test cases that set `base_background_compactions > 1` needed to be updated. I just grab the pressure token such that the desired number of compactions can be scheduled. Closes https://github.com/facebook/rocksdb/pull/2205 Differential Revision: D4937461 Pulled By: ajkr fbshipit-source-id: df52cbbd497e13bbc9a60560a5ac2a2526b3f1f9
-
由 Adam Retter 提交于
Summary: Closes https://github.com/facebook/rocksdb/pull/2356 Differential Revision: D5122240 Pulled By: sagar0 fbshipit-source-id: 3764797dd261d5b2a04d58b3967969e66c541bbb
-
由 Siying Dong 提交于
Summary: It's hard for RocksDB to come up with a good default of delayed write rate. Use rate given by rate limiter if it is availalbe. This provides the I/O order of magnitude. Closes https://github.com/facebook/rocksdb/pull/2357 Differential Revision: D5115324 Pulled By: siying fbshipit-source-id: 341065ad2211c981fc804011c0f0e59a50c7e754
-
由 Siying Dong 提交于
Summary: We forgot to add the new flag in internal build script. Add it. Closes https://github.com/facebook/rocksdb/pull/2360 Differential Revision: D5121428 Pulled By: siying fbshipit-source-id: af72d48cd855b37df1ce3c1fbb00c80377ba6e4f
-
- 24 5月, 2017 1 次提交
-
-
由 Sagar Vemuri 提交于
Summary: Fixing the build errors seen with GCC 4.8.1. ``` Makefile:105: Warning: Compiling in debug mode. Don't use the resulting binary in production utilities/blob_db/blob_dump_tool.cc: In member function ‘rocksdb::Status rocksdb::blob_db::BlobDumpTool::DumpBlobLogFooter(uint64_t, uint64_t*)’: utilities/blob_db/blob_dump_tool.cc:149:42: error: expected ‘)’ before ‘PRIu64’ fprintf(stdout, " Blob count : %" PRIu64 "\n", footer.GetBlobCount()); ^ utilities/blob_db/blob_dump_tool.cc:149:76: error: spurious trailing ‘%’ in format [-Werror=format=] fprintf(stdout, " Blob count : %" PRIu64 "\n", footer.GetBlobCount()); ^ utilities/blob_db/blob_dump_tool.cc:149:76: error: too many arguments for format [-Werror=format-extra-args] utilities/blob_db/blob_dump_tool.cc: In member function ‘rocksdb::Status rocksdb::blob_db::BlobDumpTool::DumpRecord(rocksdb::blob_db::BlobDumpTool::DisplayType, rocksdb::blob_db::BlobDumpTool::DisplayType, uint64_t*)’: utilities/blob_db/blob_dump_tool.cc:161:49: error: expected ‘)’ before ‘PRIx64’ fprintf(stdout, "Read record with offset 0x%" PRIx64 " (%" PRIu64 "):\n", ^ utilities/blob_db/blob_dump_tool.cc:162:27: error: spurious trailing ‘%’ in format [-Werror=format=] *offset, *offset); ^ utilities/blob_db/blob_dump_tool.cc:162:27: error: too many arguments for format [-Werror=format-extra-args] utilities/blob_db/blob_dump_tool.cc:176:38: error: expected ‘)’ before ‘PRIu64’ fprintf(stdout, " blob size : %" PRIu64 "\n", record.GetBlobSize()); ^ utilities/blob_db/blob_dump_tool.cc:176:71: error: spurious trailing ‘%’ in format [-Werror=format=] fprintf(stdout, " blob size : %" PRIu64 "\n", record.GetBlobSize()); ^ utilities/blob_db/blob_dump_tool.cc:176:71: error: too many arguments for format [-Werror=format-extra-args] utilities/blob_db/blob_dump_tool.cc:178:38: error: expected ‘)’ before ‘PRIu64’ fprintf(stdout, " time : %" PRIu64 "\n", record.GetTimeVal()); ^ utilities/blob_db/blob_dump_tool.cc:178:70: error: spurious trailing ‘%’ in format [-Werror=format=] fprintf(stdout, " time : %" PRIu64 "\n", record.GetTimeVal()); ^ utilities/blob_db/blob_dump_tool.cc:178:70: error: too many arguments for format [-Werror=format-extra-args] utilities/blob_db/blob_dump_tool.cc:214:38: error: expected ‘)’ before ‘PRIu64’ fprintf(stdout, " sequence : %" PRIu64 "\n", record.GetSN()); ^ utilities/blob_db/blob_dump_tool.cc:214:65: error: spurious trailing ‘%’ in format [-Werror=format=] fprintf(stdout, " sequence : %" PRIu64 "\n", record.GetSN()); ``` Closes https://github.com/facebook/rocksdb/pull/2359 Differential Revision: D5117684 Pulled By: sagar0 fbshipit-source-id: 7480346bcd96205fcae890927c5e68cf004e87be
-