- 21 10月, 2020 3 次提交
-
-
由 mrambacher 提交于
Summary: Further refinement of the earlier PR. Now the Status is NotFound with a subcode of PathNotFound. Also the existing functions for options parsing/loading are reverted to return InvalidArgument no matter in which way the user-provided arguments are deemed invalid. Pull Request resolved: https://github.com/facebook/rocksdb/pull/7563 Reviewed By: zhichao-cao Differential Revision: D24422491 Pulled By: ajkr fbshipit-source-id: ba6b237cd0584d3f925c5ba0d349aeb8c250af67
-
由 Cheng Chang 提交于
Summary: as title Pull Request resolved: https://github.com/facebook/rocksdb/pull/7571 Test Plan: make check Reviewed By: zhichao-cao Differential Revision: D24392577 Pulled By: cheng-chang fbshipit-source-id: c94f92db48270d0e215aa0f2782b0ff2e31bf708
-
由 Akanksha Mahajan 提交于
Summary: Ignore read error in 'FilePrefetchBuffer::TryReadFromCache' as status is ignored and bool value is returned. Return error if prefetch fails in 'PrefetchTail' as we have planned to return Prefetch failures to users. Pull Request resolved: https://github.com/facebook/rocksdb/pull/7574 Test Plan: make check -j64, python -u tools/db_crashtest.py blackbox Reviewed By: zhichao-cao Differential Revision: D24408825 Pulled By: akankshamahajan15 fbshipit-source-id: feebda771415998253fbe54632f13e6e75b7a243
-
- 20 10月, 2020 2 次提交
-
-
由 anand76 提交于
Summary: This PR adds support for writing a location identifier of the DB host to SST files as a table property. By default, the hostname is used, but can be overridden by the user. There have been some recent corruptions in files written by ```SstFileWriter``` before checksumming, so this property can be used to trace it back to the writing host and checking the host for hardware isues. Pull Request resolved: https://github.com/facebook/rocksdb/pull/7479 Test Plan: Add new unit tests Reviewed By: pdillinger Differential Revision: D24340671 Pulled By: anand1976 fbshipit-source-id: 2038949fd8d160c0633ccb4f9da77740f19fa2a2
-
由 Cheng Chang 提交于
Summary: In order to be able to introduce more locking protocols, we need to abstract out the locking subsystem in TransactionDB into a set of interfaces. PR https://github.com/facebook/rocksdb/pull/7013 introduces interface `LockTracker`. This PR is a follow up to take the first step to abstract out a `LockManager` interface. Further modifications to the interface may be needed when introducing the first implementation of range lock. But the idea here is to put the range lock implementation based on range tree under the `utilities/transactions/lock/range/range_tree`. Pull Request resolved: https://github.com/facebook/rocksdb/pull/7532 Test Plan: point_lock_manager_test Reviewed By: ajkr Differential Revision: D24238731 Pulled By: cheng-chang fbshipit-source-id: 2a9458cd8b3fb008d9529dbc4d3b28c24631f463
-
- 17 10月, 2020 2 次提交
-
-
由 Stanislav Tkach 提交于
Summary: Pull Request resolved: https://github.com/facebook/rocksdb/pull/7501 Reviewed By: ltamasi Differential Revision: D24344109 Pulled By: pdillinger fbshipit-source-id: d9a2b1b1cc8c8d8a96f13b8ae6814380caa10c96
-
由 Adam Retter 提交于
Summary: Fixes the URL for the badge and link. Pull Request resolved: https://github.com/facebook/rocksdb/pull/7561 Reviewed By: ltamasi Differential Revision: D24336928 Pulled By: pdillinger fbshipit-source-id: 41e04a5b036458e303aac3ae3b6129f572f0e9ce
-
- 16 10月, 2020 1 次提交
-
-
由 Levi Tamasi 提交于
Summary: The patch adds blob file support to the `Get` API by extending `Version` so that whenever a blob reference is read from a file, the blob is retrieved from the corresponding blob file and passed back to the caller. (This is assuming the blob reference is valid and the blob file is actually part of the given `Version`.) It also introduces a cache of `BlobFileReader`s called `BlobFileCache` that enables sharing `BlobFileReader`s between callers. `BlobFileCache` uses the same backing cache as `TableCache`, so `max_open_files` (if specified) limits the total number of open (table + blob) files. TODO: proactively open/cache blob files and pin the cache handles of the readers in the metadata objects similarly to what `VersionBuilder::LoadTableHandlers` does for table files. Pull Request resolved: https://github.com/facebook/rocksdb/pull/7540 Test Plan: `make check` Reviewed By: riversand963 Differential Revision: D24260219 Pulled By: ltamasi fbshipit-source-id: a8a2a4f11d3d04d6082201b52184bc4d7b0857ba
-
- 15 10月, 2020 10 次提交
-
-
由 Adam Retter 提交于
Summary: Pull Request resolved: https://github.com/facebook/rocksdb/pull/7560 Reviewed By: akankshamahajan15 Differential Revision: D24323048 Pulled By: zhichao-cao fbshipit-source-id: 2094ef089c37f570d4ea30d28d2f46f9fa1ce0f1
-
由 mrambacher 提交于
Summary: Make LoadLatestOptions return PathNotFound if the options file does not exist. Added tests for the LoadOptions related methods. Pull Request resolved: https://github.com/facebook/rocksdb/pull/7554 Reviewed By: akankshamahajan15 Differential Revision: D24298985 Pulled By: zhichao-cao fbshipit-source-id: c9ae3cb12fc4a5bbef07743e1c1300f98a2441b3
-
由 Adam Retter 提交于
Summary: * Avoid some unnecessary array copy operations on read/write * Remove some duplicated code * Don't leak arrays on some exceptions * Fixed some doc comments Pull Request resolved: https://github.com/facebook/rocksdb/pull/7371 Reviewed By: jay-zhuang Differential Revision: D24312932 Pulled By: pdillinger fbshipit-source-id: 422fe6b98bbdb922a148922ac0d2d965c715176e
-
由 Stanislav Tkach 提交于
Summary: Pull Request resolved: https://github.com/facebook/rocksdb/pull/7550 Reviewed By: jay-zhuang Differential Revision: D24315343 Pulled By: ajkr fbshipit-source-id: fc7855b630a50c00dcb940241942295932732f39
-
由 Tomasz Posluszny 提交于
Summary: - Takes the burden off developer to close ColumnFamilyHandle instances before closing RocksDB instance - The change is backward-compatible ---- Previously the pattern for working with Column Families was: ```java try (final ColumnFamilyOptions cfOpts = new ColumnFamilyOptions().optimizeUniversalStyleCompaction()) { // list of column family descriptors, first entry must always be default column family final List<ColumnFamilyDescriptor> cfDescriptors = Arrays.asList( new ColumnFamilyDescriptor(RocksDB.DEFAULT_COLUMN_FAMILY, cfOpts), new ColumnFamilyDescriptor("my-first-columnfamily".getBytes(), cfOpts) ); // a list which will hold the handles for the column families once the db is opened final List<ColumnFamilyHandle> columnFamilyHandleList = new ArrayList<>(); try (final DBOptions options = new DBOptions() .setCreateIfMissing(true) .setCreateMissingColumnFamilies(true); final RocksDB db = RocksDB.open(options, "path/to/do", cfDescriptors, columnFamilyHandleList)) { try { // do something } finally { // NOTE user must explicitly frees the column family handles before freeing the db for (final ColumnFamilyHandle columnFamilyHandle : columnFamilyHandleList) { columnFamilyHandle.close(); } } // frees the column family options } } // frees the db and the db options ``` With the changes in this PR, the Java user no longer has to worry about manually closing the Column Families, which allows them to write simpler symmetrical create/free oriented code like this: ```java try (final ColumnFamilyOptions cfOpts = new ColumnFamilyOptions().optimizeUniversalStyleCompaction()) { // list of column family descriptors, first entry must always be default column family final List<ColumnFamilyDescriptor> cfDescriptors = Arrays.asList( new ColumnFamilyDescriptor(RocksDB.DEFAULT_COLUMN_FAMILY, cfOpts), new ColumnFamilyDescriptor("my-first-columnfamily".getBytes(), cfOpts) ); // a list which will hold the handles for the column families once the db is opened final List<ColumnFamilyHandle> columnFamilyHandleList = new ArrayList<>(); try (final DBOptions options = new DBOptions() .setCreateIfMissing(true) .setCreateMissingColumnFamilies(true); final RocksDB db = RocksDB.open(options, "path/to/do", cfDescriptors, columnFamilyHandleList)) { // do something } // frees the column family options, then frees the db and the db options } } ``` **NOTE**: The changes in this PR are backwards API compatible, which means existing code using the original approach will also continue to function correctly. Pull Request resolved: https://github.com/facebook/rocksdb/pull/7428 Reviewed By: cheng-chang Differential Revision: D24063348 Pulled By: jay-zhuang fbshipit-source-id: 648d7526669923128c863ead94516bf4d50ac658
-
由 Akanksha Mahajan 提交于
Summary: fix for clang_analyzer build failure in table_test because of potential memory leak of memtable in case of ASSERT failure. Pull Request resolved: https://github.com/facebook/rocksdb/pull/7553 Test Plan: USE_CLANG=1 make analyze; make check -j64 Reviewed By: jay-zhuang Differential Revision: D24295042 Pulled By: akankshamahajan15 fbshipit-source-id: e9ea184367970fff3b520e33f3ceebf28d66ac8d
-
由 Tomasz Posluszny 提交于
Summary: Allows adding event listeners in RocksJava. * Adds listeners getter and setter in `Options` and `DBOptions` classes. * Adds `EventListener` Java interface and base class for implementing custom event listener callbacks - `AbstractEventListener`, which has an underlying native callback class implementing C++ `EventListener` class. * `AbstractEventListener` class has mechanism for selectively enabling its callback methods in order to prevent invoking Java method if it is not implemented. This decreases performance cost in case only subset of event listener callback methods is needed - the JNI code for remaining "no-op" callbacks is not executed. * The code is covered by unit tests in `EventListenerTest.java`, there are also tests added for setting/getting listeners field in `OptionsTest.java` and `DBOptionsTest.java`. Pull Request resolved: https://github.com/facebook/rocksdb/pull/7425 Reviewed By: pdillinger Differential Revision: D24063390 Pulled By: jay-zhuang fbshipit-source-id: 508c359538983d6b765e70d9989c351794a944ee
-
由 Zhichao Cao 提交于
Summary: Pull Request resolved: https://github.com/facebook/rocksdb/pull/7555 Test Plan: ASSERT_STATUS_CHECKED=1 make -j48 error_handler_fs_test Reviewed By: riversand963 Differential Revision: D24299387 Pulled By: zhichao-cao fbshipit-source-id: 6c8aa91c4b6e2bc82580b8d2264c177068f5a32c
-
由 Akanksha Mahajan 提交于
Summary: Right now all I/O failures under PartitionFilterBlock::CacheDependencies() is swallowed. Return error in case prefetch fails. On returning error in PartitionedFilterBlockReader::CacheDependencies was causing stress test failure because PrefetchBuffer is initialized with enable_ = true, as result when PosixMmapReadableFile::Read is called from Prefetch, scratch is ignored causing buffer to fill with garbage values. Initializing prefetch buffer by CreatePrefetchBuffer that sets enable_ with !ioptions.allow_mmap_reads fixed the problem as it returns without prefetching data if allow_mmap_reads is set. Pull Request resolved: https://github.com/facebook/rocksdb/pull/7543 Test Plan: make check -j64; python -u tools/db_crashtest.py --simple blackbox Reviewed By: anand1976 Differential Revision: D24284596 Pulled By: akankshamahajan15 fbshipit-source-id: f3f0fd44b59dcf60645730436f28564a07884868
-
由 Akanksha Mahajan 提交于
Summary: Update IOTrace operations in stackabledb.h and also trace few other IO operations. Pull Request resolved: https://github.com/facebook/rocksdb/pull/7514 Test Plan: make check -j64 Reviewed By: anand1976 Differential Revision: D24151202 Pulled By: akankshamahajan15 fbshipit-source-id: 112cd3d2041f8c6398b7b0ba1a783b8c93224d4a
-
- 14 10月, 2020 5 次提交
-
-
由 Jay Zhuang 提交于
Summary: Pull Request resolved: https://github.com/facebook/rocksdb/pull/7549 Reviewed By: ajkr, zhichao-cao Differential Revision: D24292032 Pulled By: jay-zhuang fbshipit-source-id: 0442283386ae20d10410a8d013a431d7cd282b22
-
由 Jay Zhuang 提交于
Summary: db_stress prints key in both id and hex. https://github.com/facebook/rocksdb/issues/7531 Pull Request resolved: https://github.com/facebook/rocksdb/pull/7533 Test Plan: local test Reviewed By: akankshamahajan15 Differential Revision: D24252725 Pulled By: jay-zhuang fbshipit-source-id: f0c1409a0568874df36949d5da139316d978fa98
-
由 Andrew Kryczka 提交于
Summary: These notes existed on the release branches where they were backported, but were never added on master branch. Added them now and mentioned what minor release the fix originally appeared. Pull Request resolved: https://github.com/facebook/rocksdb/pull/7545 Reviewed By: riversand963 Differential Revision: D24281759 Pulled By: ajkr fbshipit-source-id: 7422e984b667793d6260dd32a7492afcb2ff1c4b
-
由 Zhichao Cao 提交于
Summary: Add plain_table_db_test to ASSERT_STATUS_CHECKED list Pull Request resolved: https://github.com/facebook/rocksdb/pull/7482 Test Plan: ASSERT_STATUS_CHECKED=1 make -j48 plain_table_db_test Reviewed By: riversand963 Differential Revision: D24034987 Pulled By: zhichao-cao fbshipit-source-id: e61c937d55ded0947cc8936937362dafed572a60
-
由 mrambacher 提交于
Summary: Added unit tests that have paranoid_check = true that perform range deletions. At the moment, the deleted ranges do not appear to be checked as part of the paranoid checks. Pull Request resolved: https://github.com/facebook/rocksdb/pull/7521 Reviewed By: zhichao-cao Differential Revision: D24262175 Pulled By: ajkr fbshipit-source-id: 1035e968f7ab8ccaa7af086b835a4e72c7e56743
-
- 13 10月, 2020 4 次提交
-
-
由 Zhichao Cao 提交于
Summary: Added status check enforcement for db_flush_test Pull Request resolved: https://github.com/facebook/rocksdb/pull/7476 Test Plan: ASSERT_STATUS_CHECKED=1 make -j48 db_flush_test Reviewed By: akankshamahajan15 Differential Revision: D24033752 Pulled By: zhichao-cao fbshipit-source-id: d957934e1666d0043bebdd8a4149e94cdcbbb89b
-
由 Jay Zhuang 提交于
Summary: If crash test fails, don't delete the `expected_values_file` for later debug. More details: https://github.com/facebook/rocksdb/issues/7530 Pull Request resolved: https://github.com/facebook/rocksdb/pull/7534 Test Plan: local host Reviewed By: ajkr Differential Revision: D24239655 Pulled By: jay-zhuang fbshipit-source-id: 3566f91a30aae1e27d2f51d910cddd08edb7d4cf
-
由 Jay Zhuang 提交于
Summary: ThreadSanitizer: data race for `DummyOldStats.num_rt`. Failed build: https://app.circleci.com/pipelines/github/facebook/rocksdb/3991/workflows/b47c3ae1-5531-4489-ac51-11854abdfd0f/jobs/42305 Pull Request resolved: https://github.com/facebook/rocksdb/pull/7526 Reviewed By: akankshamahajan15 Differential Revision: D24226736 Pulled By: jay-zhuang fbshipit-source-id: e05ce354d0c0db0eba242d59d4b0e89ce7c25acf
-
由 Cheng Chang 提交于
Summary: as title Pull Request resolved: https://github.com/facebook/rocksdb/pull/7536 Test Plan: see the new `build-macos` tests pass in circleci Reviewed By: jay-zhuang Differential Revision: D24243218 Pulled By: cheng-chang fbshipit-source-id: 9b5f8a859e54c99a9ebe7efff6f336458a5d42de
-
- 12 10月, 2020 1 次提交
-
-
由 Andrew Kryczka 提交于
Summary: The old flag-based APIs (`BlockBasedTableOptions::pin_l0_filter_and_index_blocks_in_cache` and `BlockBasedTableOptions::pin_top_level_index_and_filter`) were insufficient for our needs. For example, it was impossible to pin only unpartitioned meta-blocks, which could prevent block cache contention when turning on dictionary compression or during a migration to partitioned indexes/filters. It was also impossible to pin all meta-blocks in memory while having predictable memory usage via block cache. If we had continued adding flags to address these scenarios, they would have had significant overlap causing confusion. Instead, this PR deprecates the flags and starts a new API with non-overlapping options. Pull Request resolved: https://github.com/facebook/rocksdb/pull/7520 Test Plan: - new unit test - added new options to stress/crash test and ran for a while: `$ python tools/db_crashtest.py blackbox --simple --max_key=1000000 -write_buffer_size=1048576 -target_file_size_base=1048576 -max_bytes_for_level_base=4194304 --interval=10 -value_size_mult=33 -column_families=1 -reopen=0` Reviewed By: pdillinger Differential Revision: D24200034 Pulled By: ajkr fbshipit-source-id: 3fa7cfc71e7960f7a867511dd6ae5834dd73b13e
-
- 10 10月, 2020 2 次提交
-
-
由 Cheng Chang 提交于
Summary: This option determines whether WALs will be tracked in MANIFEST and verified on recovery. Pull Request resolved: https://github.com/facebook/rocksdb/pull/7275 Test Plan: db_options_test options_test Reviewed By: pdillinger Differential Revision: D23181418 Pulled By: cheng-chang fbshipit-source-id: 5dd1cdc166f3dfc1c93c094df4a2f7734e3b4547
-
由 Akanksha Mahajan 提交于
Summary: Update release history for 6.14 Pull Request resolved: https://github.com/facebook/rocksdb/pull/7525 Test Plan: No code change Reviewed By: jay-zhuang Differential Revision: D24224690 Pulled By: akankshamahajan15 fbshipit-source-id: 95441aefde96672fea5a6af5d7e67cdafb1ebdd2
-
- 09 10月, 2020 4 次提交
-
-
由 sdong 提交于
Summary: Cover paranoid_file_checks in crash test. Pull Request resolved: https://github.com/facebook/rocksdb/pull/7489 Test Plan: Run crash tests for hours and didn't see any failure. Reviewed By: ajkr Differential Revision: D24063868 fbshipit-source-id: 7b48b110e66ce78ae5d0c99a9f32af86edd34c1e
-
由 Akanksha Mahajan 提交于
Summary: Add status enforcement for following tests: 1. import_column_family_test 2. memory_test 3. table_test Pull Request resolved: https://github.com/facebook/rocksdb/pull/7500 Reviewed By: zhichao-cao Differential Revision: D24095887 Pulled By: akankshamahajan15 fbshipit-source-id: db8e1ec595852df143fad78a0c07bfdd27dc3c84
-
由 Levi Tamasi 提交于
Summary: The `std::pair(const T1& x, const T2& y);` constructor is `constexpr` only starting from C++14; relying on this breaks compilation on certain compilers/platforms we need to support. Pull Request resolved: https://github.com/facebook/rocksdb/pull/7519 Test Plan: `make check` Reviewed By: ajkr Differential Revision: D24195747 Pulled By: ltamasi fbshipit-source-id: 665e8fbc9747675bb49c5d895aad3dcf2714750f
-
由 Jay Zhuang 提交于
Summary: It's a known issue, which is tracked in https://github.com/facebook/rocksdb/issues/7405, https://github.com/facebook/rocksdb/issues/7470. Disable it for now. Pull Request resolved: https://github.com/facebook/rocksdb/pull/7511 Reviewed By: zhichao-cao Differential Revision: D24145075 Pulled By: jay-zhuang fbshipit-source-id: 1858497972f2baba617867aaeac30d93b8305c80
-
- 08 10月, 2020 6 次提交
-
-
由 Zhichao Cao 提交于
Summary: Add ldb_cmd_test to ASSERT_STATUS_CHECKED list Pull Request resolved: https://github.com/facebook/rocksdb/pull/7499 Test Plan: pass ASSERT_STATUS_CHECKED=1 make -j48 ldb_cmd_test Reviewed By: cheng-chang Differential Revision: D24086203 Pulled By: zhichao-cao fbshipit-source-id: 29592202b1d4335e566de15e7937269d98d57841
-
由 Yanqin Jin 提交于
Summary: Pull Request resolved: https://github.com/facebook/rocksdb/pull/7518 Test Plan: ``` $USE_CLANG=1 make analyze ``` Reviewed By: zhichao-cao Differential Revision: D24175390 Pulled By: riversand963 fbshipit-source-id: c70121652908cf5d450120c38ab65cc595332ca7
-
由 Levi Tamasi 提交于
Summary: The patch does some cleanup in and around the legacy `BlobLogReader` class: * It renames the class to `BlobLogSequentialReader` to emphasize that it is for sequentially iterating through blobs in a blob file, as opposed to doing random point reads using `BlobIndex`es (which is `BlobFileReader`'s jurisdiction). * It removes some dead code from the old BlobDB implementation that references `BlobLogReader` (namely the method `BlobFile::OpenRandomAccessReader`). * It cleans up some `#include`s and forward declarations. * It fixes some incorrect/outdated comments related to the reader class. * It adds a few assertions to the `Read` methods of the class. Pull Request resolved: https://github.com/facebook/rocksdb/pull/7517 Test Plan: `make check` Reviewed By: riversand963 Differential Revision: D24172611 Pulled By: ltamasi fbshipit-source-id: 43e2ae1eba5c3dd30c1070cb00f217edc45bd64f
-
由 Levi Tamasi 提交于
Summary: The patch adds a class called `BlobFileReader` that can be used to retrieve blobs using the information available in blob references (e.g. blob file number, offset, and size). This will come in handy when implementing blob support for `Get`, `MultiGet`, and iterators, and also for compaction/garbage collection. When a `BlobFileReader` object is created (using the factory method `Create`), it first checks whether the specified file is potentially valid by comparing the file size against the combined size of the blob file header and footer (files smaller than the threshold are considered malformed). Then, it opens the file, and reads and verifies the header and footer. The verification involves magic number/CRC checks as well as checking for unexpected header/footer fields, e.g. incorrect column family ID or TTL blob files. Blobs can be retrieved using `GetBlob`. `GetBlob` validates the offset and compression type passed by the caller (because of the presence of the header and footer, the specified offset cannot be too close to the start/end of the file; also, the compression type has to match the one in the blob file header), and retrieves and potentially verifies and uncompresses the blob. In particular, when `ReadOptions::verify_checksums` is set, `BlobFileReader` reads the blob record header as well (as opposed to just the blob itself) and verifies the key/value size, the key itself, as well as the CRC of the blob record header and the key/value pair. In addition, the patch exposes the compression type from `BlobIndex` (both using an accessor and via `DebugString`), and adds a blob file read latency histogram to `InternalStats` that can be used with `BlobFileReader`. Pull Request resolved: https://github.com/facebook/rocksdb/pull/7461 Test Plan: `make check` Reviewed By: riversand963 Differential Revision: D23999219 Pulled By: ltamasi fbshipit-source-id: deb6b1160d251258b308d5156e2ec063c3e12e5e
-
由 Akanksha Mahajan 提交于
Summary: Add following stats for MultiGet in Histogram to get more insight on MultiGet. 1. Number of index and filter blocks read from file as part of MultiGet request per level. 2. Number of data blocks read from file per level. 3. Number of SST files loaded from file system per level. Pull Request resolved: https://github.com/facebook/rocksdb/pull/7366 Reviewed By: anand1976 Differential Revision: D24127040 Pulled By: akankshamahajan15 fbshipit-source-id: e63a003056b833729b277edc0639c08fb432756b
-
由 Jay Zhuang 提交于
Summary: If `BottommostLevelCompaction.kForce*` is set, compaction should avoid trivial move and always compact the sst to the target size. Pull Request resolved: https://github.com/facebook/rocksdb/pull/7368 Reviewed By: ajkr Differential Revision: D23629525 Pulled By: jay-zhuang fbshipit-source-id: 79f23c79ecb31587e0593b28cce43131107bbcd0
-