- 25 7月, 2020 1 次提交
-
-
由 Yanqin Jin 提交于
Summary: As title. Pull Request resolved: https://github.com/facebook/rocksdb/pull/7170 Test Plan: Manually invoke the commands. Reviewed By: siying Differential Revision: D22732256 Pulled By: riversand963 fbshipit-source-id: d331e5ee84658ac079814292ff1a1eacfd14bfdf
-
- 21 7月, 2020 2 次提交
-
-
由 sdong 提交于
Summary: Two TSAN tests occaionaly fail. Exclude them for now: [ RUN ] DeleteFileTest.BackgroundPurgeCFDropTest db/deletefile_test.cc:122: Failure Expected equality of these values: required_manifest Which is: 1 manifest_cnt Which is: 2 [ RUN ] FormatLatest/ColumnFamilyTest.FlushCloseWALFiles/0 db/column_family_test.cc:3004: Failure Expected equality of these values: 2 env.num_open_wal_file_.load() Which is: 1 Pull Request resolved: https://github.com/facebook/rocksdb/pull/7152 Test Plan: Watch CircleCI restuls Reviewed By: ajkr Differential Revision: D22632285 fbshipit-source-id: 29fa348e8be917be0237c74812a8b0b04978e84e
-
由 Jay Zhuang 提交于
Summary: To have test report. Pull Request resolved: https://github.com/facebook/rocksdb/pull/7137 Reviewed By: siying Differential Revision: D22630798 Pulled By: jay-zhuang fbshipit-source-id: bc07ba673c0bceed5a4829b4af2d9a74435379c7
-
- 18 7月, 2020 1 次提交
-
-
由 sdong 提交于
Summary: In CircleCI tests, we failed to fail tests properly if parallel doesn't return an error code. It's probably would happen when unit tests fail with signals, rather than return values. Fix them. Pull Request resolved: https://github.com/facebook/rocksdb/pull/7147 Test Plan: Manually ingest a failure and see it to fail. Reviewed By: jay-zhuang Differential Revision: D22611594 fbshipit-source-id: 88a42425a41d1213d29bd2e7c80731d2bdd5644b
-
- 15 7月, 2020 2 次提交
-
-
由 Adam Retter 提交于
Summary: This fixes an issue introduced in 0c56fc4d whereby the location of Python is evaluated many times and leads to excessive logging of unknown python locations of CentOS 6. The location is now only checked once. Pull Request resolved: https://github.com/facebook/rocksdb/pull/7123 Reviewed By: zhichao-cao Differential Revision: D22532274 Pulled By: ajkr fbshipit-source-id: cade71b4b46e9a23d63ecb4dd36a4ac8ae217970
-
由 sdong 提交于
Summary: It is helpful to add some TSAN coverage before a pull request is committed. This diff adds some of them. Some slow tests are excluded for the running speed. Some are blacklisted because they show warnings. Will investigate these warnings and see whether we can fix or suppress them. Pull Request resolved: https://github.com/facebook/rocksdb/pull/7122 Test Plan: Watch CIrcleCI runs Reviewed By: riversand963 Differential Revision: D22532133 fbshipit-source-id: 81ddd02d9df19c513a12811979e8ddabae911354
-
- 10 7月, 2020 1 次提交
-
-
由 mrambacher 提交于
Summary: Cleans up some of the dependencies on test code in the Makefile while building tools: - Moves the test::RandomString, DBBaseTest::RandomString into Random - Moves the test::RandomHumanReadableString into Random - Moves the DestroyDir method into file_utils - Moves the SetupSyncPointsToMockDirectIO into sync_point. - Moves the FaultInjection Env and FS classes under env These changes allow all of the tools to build without dependencies on test_util, thereby simplifying the build dependencies. By moving the FaultInjection code, the dependency in db_stress on different libraries for debug vs release was eliminated. Tested both release and debug builds via Make and CMake for both static and shared libraries. More work remains to clean up how the tools are built and remove some unnecessary dependencies. There is also more work that should be done to get the Makefile and CMake to align in their builds -- what is in the libraries and the sizes of the executables are different. Pull Request resolved: https://github.com/facebook/rocksdb/pull/7097 Reviewed By: riversand963 Differential Revision: D22463160 Pulled By: pdillinger fbshipit-source-id: e19462b53324ab3f0b7c72459dbc73165cc382b2
-
- 08 7月, 2020 3 次提交
-
-
由 Peter Dillinger 提交于
Summary: On some platforms like MacOS, a second 'make check' can lead to /bin/rm: Argument list too long This is fixed by replacing with a 'find'. Also, using '-f' for more rm calls to avoid prompt. Pull Request resolved: https://github.com/facebook/rocksdb/pull/7095 Test Plan: 'make check' on Linux and MacOS Reviewed By: riversand963 Differential Revision: D22415808 Pulled By: pdillinger fbshipit-source-id: 0fd1ebae13739c9d81f9e813e99b062715604d6b
-
由 Peter Dillinger 提交于
Summary: by tracking and linking against runtime dependent libraries in Makefile Pull Request resolved: https://github.com/facebook/rocksdb/pull/7098 Test Plan: look for fix in CircleCI Reviewed By: riversand963 Differential Revision: D22420860 Pulled By: pdillinger fbshipit-source-id: d211d709214bf5306db68e43b7a2f18169281022
-
由 Peter Dillinger 提交于
Summary: Primarily, this change adds a way to work around a bug limiting the effective output (and therefore debugability) of the Linux builds using parallel make. We would get make[1]: write error: stdout probably due to a kernel bug, apparently affecting both available ubuntu 16 machine images (maybe not affecting docker images, less horsepower). https://bugs.launchpad.net/ubuntu/+source/linux-signed/+bug/1814393 Now in the CircleCI config, make output on Ubuntu is piped through a custom 'cat' that ignores EAGAIN errors, which seems to fix the problem. Significant other changes: * Add another linux build that combines * LIB_MODE=shared, to ensure this works with compile and unit test execution * Alternative rocksdb namespace, to ensure this works (not rely on Travis) * ASSERT_STATUS_CHECKED=1, but with building all unit tests and running those expected to pass with it * Run release build with and without gflags. (Was running only without, ignore large swaths of code in a normal release build! Two regressions in this build, only with gflags, in the last week not caught by CI!) * Use gflags with unity and LITE build, as typical case. Debugability improvements: * Use V=1 to show commands being executed (thanks to EAGAIN work-around) * Print kernel version and compiler versions as part of V=1 output from Makefile Cosmetic other changes: * Put more commands on one line, for less clutter in CircleCI output pages * Remove redundant "all" in "make all check" and put make command options before targets * Change some recursive "make clean" into dependency on "clean," toward minimizing unnecessary overhead (detect platform, build version, etc.) of extra recursive makes Pull Request resolved: https://github.com/facebook/rocksdb/pull/7078 Reviewed By: siying Differential Revision: D22391647 Pulled By: pdillinger fbshipit-source-id: d446fccf5a8c568b37dc8748621c8a5c546fe135
-
- 07 7月, 2020 1 次提交
-
-
由 Peter Dillinger 提交于
Summary: (a) use STRESS_LIBRARY for db_stress and make sure STRESS_LIBRARY has other stress test dependencies (as in buck build) (b) fix rpath option to be accepted on MacOS. It still doesn't fully work for me e.g. to run a LIB_MODE=shared unit test binary from another directory, as it does on Linux, but the option is now accepted, and running unit tests from current directory works for me. Also adding LIB_MODE=shared to Travis. (Later TBD where best to fit in in CircleCI.) Pull Request resolved: https://github.com/facebook/rocksdb/pull/7066 Test Plan: manual Reviewed By: cheng-chang Differential Revision: D22364068 Pulled By: pdillinger fbshipit-source-id: 6fa98a222f89f808ee786474de1100d92c1adec3
-
- 02 7月, 2020 1 次提交
-
-
由 Peter Dillinger 提交于
Summary: Follow-up to https://github.com/facebook/rocksdb/issues/6660. Release build had linker error. fbcode+clang+shared build was erroring on unused parameter '-nostdinc'. Fixes https://github.com/facebook/rocksdb/issues/7061 Pull Request resolved: https://github.com/facebook/rocksdb/pull/7062 Test Plan: make release, USE_CLANG=1 LIB_MODE=shared make check, etc. Reviewed By: siying Differential Revision: D22335663 Pulled By: pdillinger fbshipit-source-id: 261cd959ca1f6c273dc763a70020a535ba8e81de
-
- 01 7月, 2020 1 次提交
-
-
由 mrambacher 提交于
Summary: Change the linking of tests/tools to be against a library rather than a list of objects. This change substantially reduces the size of the objects produced. peterd clean repo size: 264M Before this change, with make all: 40G After this change, with make all: 28G With make LIB_MODE=shared all: 7.0G The list of TESTS was changed from being hard-coded to generated from the test sources variable. Note that there are some test sources that are not built as tests (though the set of tests is identical to the previous version). Added OBJ_DIR option to Makefile to allow objects to be placed in an alternative location. By default, OBJ_DIR is the same as before ("./"). This change is a precursor to being able to build/run the tests/tools linked against static libraries. Additionally, it should be possible to clean up and merge some of the rules for building tests and the like if so desired. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6660 Reviewed By: riversand963 Differential Revision: D22244463 Pulled By: pdillinger fbshipit-source-id: db9c6341d81ed62c2270374f4ede02fb9604c754
-
- 27 6月, 2020 1 次提交
-
-
由 sdong 提交于
Summary: We are still keeping unity build working. So it's a good idea to add to a pre-commit CI. A latest GCC docker image just to get a little bit more coverage. Fix three small issues to make it pass. Also make unity_test to run db_basic_test rather than db_test to cut the test time. There is no point to run expensive tests here. It was set to run db_test before db_basic_test was separated out. Pull Request resolved: https://github.com/facebook/rocksdb/pull/7026 Test Plan: watch tests to pass. Reviewed By: zhichao-cao Differential Revision: D22223197 fbshipit-source-id: baa3b6cbb623bf359829b63ce35715c75bcb0ed4
-
- 20 6月, 2020 1 次提交
-
-
由 Peter Dillinger 提交于
Summary: We don't need them. Pull Request resolved: https://github.com/facebook/rocksdb/pull/7008 Test Plan: "make check" and ensure "make crash_test" starts Reviewed By: ajkr Differential Revision: D22143838 Pulled By: pdillinger fbshipit-source-id: 72c8e16603abc59f4954e304466bc4dc1f58f94e
-
- 19 6月, 2020 1 次提交
-
-
由 Akanksha Mahajan 提交于
Summary: 1. As part of IOTracing project, Add a class IOTracer, IOTraceReader and IOTracerWriter that writes the file operations information in a binary file. IOTrace Record contains record information and right now it contains access_timestamp, file_operation, file_name, io_status, len, offset and later other options will be added when file system APIs will be call IOTracer. 2. Add few unit test cases that verify that reading and writing to a IO Trace file is working properly and before start trace and after ending trace nothing is added to the binary file. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6958 Test Plan: 1. make check -j64 2. New testcases for IOTracer. Reviewed By: anand1976 Differential Revision: D21943375 Pulled By: akankshamahajan15 fbshipit-source-id: 3532204e2a3eab0104bf411ab142e3fdd4fbce54
-
- 13 6月, 2020 1 次提交
-
-
由 Yanqin Jin 提交于
Summary: Add crash test for the case of best-efforts recovery. After a certain amount of time, we kill the db_stress process, randomly delete some certain table files and restart db_stress. Given the randomness of file deletion, it is difficult to verify against a reference for data correctness. Therefore, we just check that the db can restart successfully. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6819 Test Plan: ``` ./db_stress -best_efforts_recovery=true -disable_wal=1 -reopen=0 ./db_stress -best_efforts_recovery=true -disable_wal=0 -skip_verifydb=1 -verify_db_one_in=0 -continuous_verification_interval=0 make crash_test_with_best_efforts_recovery ``` Reviewed By: anand1976 Differential Revision: D21436753 Pulled By: riversand963 fbshipit-source-id: 0b3605c922a16c37ed17d5ab6682ca4240e47926
-
- 09 6月, 2020 1 次提交
-
-
由 sdong 提交于
Summary: Moving towards the long term goal of moving most CI build to CircleCI when possible, add some Linux tests in CircleCI. This is not all what we can include to CircleCI. For example, Java builds are not includ ed. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6937 Test Plan: Watch CI build results. Reviewed By: pdillinger Differential Revision: D21941605 fbshipit-source-id: db6aead3c45f523386d4fb30d224cfde573cccad
-
- 06 6月, 2020 1 次提交
-
-
由 Cheng Chang 提交于
Summary: Currently, `DeleteDir` only deletes the directory if there are no other directories under the target dir. This PR makes it delete directories recursively. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6934 Test Plan: Added a new unit test in testutil_test.cc. `make testutil_test` Reviewed By: zhichao-cao Differential Revision: D21884211 Pulled By: cheng-chang fbshipit-source-id: 0b9a48a200f494ee007aef5d1763b4aa331f8b5a
-
- 04 6月, 2020 2 次提交
-
-
由 sdong 提交于
Summary: This reverts commit 8d87e9ce. Based on offline discussions, it's too early to upgrade to gtest 1.10, as it prevents some developers from using an older version of gtest to integrate to some other systems. Revert it for now. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6923 Reviewed By: pdillinger Differential Revision: D21864799 fbshipit-source-id: d0726b1ff649fc911b9378f1763316200bd363fc
-
由 Peter Dillinger 提交于
Summary: People keep breaking the gcc 4.8 compilation due to different warnings for shadowing member functions with locals. Adding to Travis to keep compatibility. (gcc 4.8 is default on CentOS 7.) Pull Request resolved: https://github.com/facebook/rocksdb/pull/6915 Test Plan: local and Travis Reviewed By: siying Differential Revision: D21842894 Pulled By: pdillinger fbshipit-source-id: bdcd4385127ee5d1cc222d87e53fb3695c32a9d4
-
- 02 6月, 2020 1 次提交
-
-
由 Adam Retter 提交于
Summary: Pull Request resolved: https://github.com/facebook/rocksdb/pull/6808 Reviewed By: anand1976 Differential Revision: D21483984 Pulled By: pdillinger fbshipit-source-id: 70c5eff2bd54ddba469761d95e4cd4611fb8e598
-
- 30 5月, 2020 1 次提交
-
-
由 Peter Dillinger 提交于
Summary: RocksDB Makefile was assuming existence of 'python' command, which is not present in CentOS 8. We avoid using 'python' if 'python3' is available. Also added fancy logic to format-diff.sh to make clang-format-diff.py for Python2 work even with Python3 only (as some CentOS 8 FB machines come equipped) Also, now use just 'python3' for PYTHON if not found so that an informative "command not found" error will result rather than something weird. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6883 Test Plan: manually tried some variants, 'make check' on a fresh CentOS 8 machine without 'python' executable or Python2 but with clang-format-diff.py for Python2. Reviewed By: gg814 Differential Revision: D21767029 Pulled By: pdillinger fbshipit-source-id: 54761b376b140a3922407bdc462f3572f461d0e9
-
- 23 5月, 2020 2 次提交
-
-
由 Peter Dillinger 提交于
Summary: * Print stack trace on status checked failure * Make folly_synchronization_distributed_mutex_test a parallel test * Disable ldb_test.py and rocksdb_dump_test.sh with ASSERT_STATUS_CHECKED (broken) * Fix shadow warning in random_access_file_reader.h reported by gcc 4.8.5 (ROCKSDB_NO_FBCODE), also https://github.com/facebook/rocksdb/issues/6866 * Work around compiler bug on max_align_t for gcc < 4.9 * Remove an apparently wrong comment in status.h * Use check_some in Travis config (for proper diagnostic output) * Fix ignored Status in loop in options_helper.cc Pull Request resolved: https://github.com/facebook/rocksdb/pull/6871 Test Plan: manual, CI Reviewed By: ajkr Differential Revision: D21706619 Pulled By: pdillinger fbshipit-source-id: daf6364173d6689904eb394461a69a11f5bee2cb
-
由 Peter Dillinger 提交于
Summary: Fixed some option handling code that recently broke the ASSERT_STATUS_CHECKED build for options_test. Added all other existing tests that pass under ASSERT_STATUS_CHECKED to the whitelist. Added a Travis configuration to run all whitelisted tests with ASSERT_STATUS_CHECKED. (Someday we might enable this check by default in debug builds.) Pull Request resolved: https://github.com/facebook/rocksdb/pull/6870 Test Plan: ASSERT_STATUS_CHECKED=1 make check, Travis Reviewed By: ajkr Differential Revision: D21704374 Pulled By: pdillinger fbshipit-source-id: 15daef98136a19d7a6843fa0c9ec08738c2ac693
-
- 21 5月, 2020 1 次提交
-
-
由 Peter Dillinger 提交于
Summary: ... so that we have freedom to upgrade it (see https://github.com/facebook/rocksdb/issues/6808). As a side benefit, gtest will no longer be linked into main library in buck build. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6858 Test Plan: fb internal build & link Reviewed By: riversand963 Differential Revision: D21652061 Pulled By: pdillinger fbshipit-source-id: 6018104af944debde576b5beda6c134e737acedb
-
- 20 5月, 2020 1 次提交
-
-
由 Akanksha Mahajan 提交于
Summary: Added status check enforcement for io_posix_test and options_settable_test Pull Request resolved: https://github.com/facebook/rocksdb/pull/6857 Test Plan: ASSERT_STATUS_CHECKED=1 make -j48 check Reviewed By: ajkr Differential Revision: D21647904 Pulled By: akankshamahajan15 fbshipit-source-id: b7f2321eb6c141a88cd5e1270ecb7d58f00341af
-
- 15 5月, 2020 1 次提交
-
-
由 Cheng Chang 提交于
Summary: Currently, in direct IO mode, `MultiGet` retrieves the data blocks one by one instead of in parallel, see `BlockBasedTable::RetrieveMultipleBlocks`. Since direct IO is supported in `RandomAccessFileReader::MultiRead` in https://github.com/facebook/rocksdb/pull/6446, this PR applies `MultiRead` to `MultiGet` so that the data blocks can be retrieved in parallel. Also, in direct IO mode and when data blocks are compressed and need to uncompressed, this PR only allocates one continuous aligned buffer to hold the data blocks, and then directly uncompress the blocks to insert into block cache, there is no longer intermediate copies to scratch buffers. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6815 Test Plan: 1. added a new unit test `BlockBasedTableReaderTest::MultiGet`. 2. existing unit tests and stress tests contain tests against `MultiGet` in direct IO mode. Reviewed By: anand1976 Differential Revision: D21426347 Pulled By: cheng-chang fbshipit-source-id: b8446ae0e74152444ef9111e97f8e402ac31b24f
-
- 09 5月, 2020 1 次提交
-
-
由 Andrew Kryczka 提交于
Summary: Tried making Status object enforce that it is checked in some way. In cases it is not checked, `PermitUncheckedError()` must be called explicitly. Added a way to run tests (`ASSERT_STATUS_CHECKED=1 make -j48 check`) on a whitelist. The effort appears significant to get each test to pass with this assertion, so I only fixed up enough to get one test (`options_test`) working and added it to the whitelist. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6798 Reviewed By: pdillinger Differential Revision: D21377404 Pulled By: ajkr fbshipit-source-id: 73236f9c8df38f01cf24ecac4a6d1661b72d077e
-
- 01 5月, 2020 1 次提交
-
-
由 Yanqin Jin 提交于
Summary: Add Github Action to perform some basic sanity check for PR, inclding the following. 1) Buck TARGETS file. On the one hand, The TARGETS file is used for internal buck, and we do not manually update it. On the other hand, we need to run the buckifier scripts to update TARGETS whenever new files are added, etc. With this Github Action, we make sure that every PR does not forget this step. The GH Action uses a Makefile target called check-buck-targets. Users can manually run `make check-buck-targets` on local machine. 2) Code format We use clang-format-diff.py to format our code. The GH Action in this PR makes sure this step is not skipped. The checking script build_tools/format-diff.sh assumes that `clang-format-diff.py` is executable. On host running GH Action, it is difficult to download `clang-format-diff.py` and make it executable. Therefore, we modified build_tools/format-diff.sh to handle the case in which there is a non-executable clang-format-diff.py file in the top-level rocksdb repo directory. Test Plan (Github and devserver): Watch for Github Action result in the `Checks` tab. On dev server ``` make check-format make check-buck-targets make check ``` Pull Request resolved: https://github.com/facebook/rocksdb/pull/6761 Test Plan: Watch for Github Action result in the `Checks` tab. Reviewed By: pdillinger Differential Revision: D21260209 Pulled By: riversand963 fbshipit-source-id: c646e2f37c6faf9f0614b68aa0efc818cff96787
-
- 28 4月, 2020 1 次提交
-
-
由 Peter Dillinger 提交于
Summary: Some common build variables like USE_CLANG and COMPILE_WITH_UBSAN did not work if specified as make variables, as in `make USE_CLANG=1 check` etc. rather than (in theory less hygienic) `USE_CLANG=1 make check`. This patches Makefile to export some commonly used ones to build_detect_platform so that they work. (I'm skeptical of a broad `export` in Makefile because it's hard to predict how random make variables might affect various invoked tools.) Pull Request resolved: https://github.com/facebook/rocksdb/pull/6740 Test Plan: manual / CI Reviewed By: siying Differential Revision: D21229011 Pulled By: pdillinger fbshipit-source-id: b00c69b23eb2a13105bc8d860ce2d1e61ac5a355
-
- 25 4月, 2020 1 次提交
-
-
由 Cheng Chang 提交于
Summary: In https://github.com/facebook/rocksdb/pull/6455, we modified the interface of `RandomAccessFileReader::Read` to be able to get rid of memcpy in direct IO mode. This PR applies the new interface to `BlockFetcher` when reading blocks from SST files in direct IO mode. Without this PR, in direct IO mode, when fetching and uncompressing compressed blocks, `BlockFetcher` will first copy the raw compressed block into `BlockFetcher::compressed_buf_` or `BlockFetcher::stack_buf_` inside `RandomAccessFileReader::Read` depending on the block size. then during uncompressing, it will copy the uncompressed block into `BlockFetcher::heap_buf_`. In this PR, we get rid of the first memcpy and directly uncompress the block from `direct_io_buf_` to `heap_buf_`. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6689 Test Plan: A new unit test `block_fetcher_test` is added. Reviewed By: anand1976 Differential Revision: D21006729 Pulled By: cheng-chang fbshipit-source-id: 2370b92c24075692423b81277415feb2aed5d980
-
- 17 4月, 2020 1 次提交
-
-
由 Adam Retter 提交于
Summary: Updates the version of bzip2 used for RocksJava static builds. Please, can we also get this cherry-picked to: 1. 6.7.fb 2. 6.8.fb 3. 6.9.fb Pull Request resolved: https://github.com/facebook/rocksdb/pull/6714 Reviewed By: cheng-chang Differential Revision: D21067233 Pulled By: pdillinger fbshipit-source-id: 8164b7eb99c5ca7b2021ab8c371ba9ded4cb4f7e
-
- 11 4月, 2020 2 次提交
-
-
由 anand76 提交于
Summary: This PR implements a fault injection mechanism for injecting errors in reads in db_stress. The FaultInjectionTestFS is used for this purpose. A thread local structure is used to track the errors, so that each db_stress thread can independently enable/disable error injection and verify observed errors against expected errors. This is initially enabled only for Get and MultiGet, but can be extended to iterator as well once its proven stable. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6538 Test Plan: crash_test make check Reviewed By: riversand963 Differential Revision: D20714347 Pulled By: anand1976 fbshipit-source-id: d7598321d4a2d72bda0ced57411a337a91d87dc7
-
由 Yanqin Jin 提交于
Summary: Towards making compaction logic compatible with user timestamp. When computing boundaries and overlapping ranges for inputs of compaction, We need to compare SSTs by user key without timestamp. Test plan (devserver): ``` make check ``` Several individual tests: ``` ./version_set_test --gtest_filter=VersionStorageInfoTimestampTest.GetOverlappingInputs ./db_with_timestamp_compaction_test ./db_with_timestamp_basic_test ``` Pull Request resolved: https://github.com/facebook/rocksdb/pull/6645 Reviewed By: ltamasi Differential Revision: D20960012 Pulled By: riversand963 fbshipit-source-id: ad377fa9eb481bf7a8a3e1824aaade48cdc653a4
-
- 10 4月, 2020 1 次提交
-
-
由 Luca Giacchino 提交于
Summary: New memory technologies are being developed by various hardware vendors (Intel DCPMM is one such technology currently available). These new memory types require different libraries for allocation and management (such as PMDK and memkind). The high capacities available make it possible to provision large caches (up to several TBs in size), beyond what is achievable with DRAM. The new allocator provided in this PR uses the memkind library to allocate memory on different media. **Performance** We tested the new allocator using db_bench. - For each test, we vary the size of the block cache (relative to the size of the uncompressed data in the database). - The database is filled sequentially. Throughput is then measured with a readrandom benchmark. - We use a uniform distribution as a worst-case scenario. The plot shows throughput (ops/s) relative to a configuration with no block cache and default allocator. For all tests, p99 latency is below 500 us. ![image](https://user-images.githubusercontent.com/26400080/71108594-42479100-2178-11ea-8231-8a775bbc92db.png) **Changes** - Add MemkindKmemAllocator - Add --use_cache_memkind_kmem_allocator db_bench option (to create an LRU block cache with the new allocator) - Add detection of memkind library with KMEM DAX support - Add test for MemkindKmemAllocator **Minimum Requirements** - kernel 5.3.12 - ndctl v67 - https://github.com/pmem/ndctl - memkind v1.10.0 - https://github.com/memkind/memkind **Memory Configuration** The allocator uses the MEMKIND_DAX_KMEM memory kind. Follow the instructions on[ memkind’s GitHub page](https://github.com/memkind/memkind) to set up NVDIMM memory accordingly. Note on memory allocation with NVDIMM memory exposed as system memory. - The MemkindKmemAllocator will only allocate from NVDIMM memory (using memkind_malloc with MEMKIND_DAX_KMEM kind). - The default allocator is not restricted to RAM by default. Based on NUMA node latency, the kernel should allocate from local RAM preferentially, but it’s a kernel decision. numactl --preferred/--membind can be used to allocate preferentially/exclusively from the local RAM node. **Usage** When creating an LRU cache, pass a MemkindKmemAllocator object as argument. For example (replace capacity with the desired value in bytes): ``` #include "rocksdb/cache.h" #include "memory/memkind_kmem_allocator.h" NewLRUCache( capacity /*size_t*/, 6 /*cache_numshardbits*/, false /*strict_capacity_limit*/, false /*cache_high_pri_pool_ratio*/, std::make_shared<MemkindKmemAllocator>()); ``` Refer to [RocksDB’s block cache documentation](https://github.com/facebook/rocksdb/wiki/Block-Cache) to assign the LRU cache as block cache for a database. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6214 Reviewed By: cheng-chang Differential Revision: D19292435 fbshipit-source-id: 7202f47b769e7722b539c86c2ffd669f64d7b4e1
-
- 09 4月, 2020 1 次提交
-
-
由 Cheng Chang 提交于
Summary: Although there are tests related to locking in transaction_test, this new test directly tests against TransactionLockMgr. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6599 Test Plan: make transaction_lock_mgr_test && ./transaction_lock_mgr_test Reviewed By: lth Differential Revision: D20673749 Pulled By: cheng-chang fbshipit-source-id: 1fa4a13218e68d785f5a99924556751a8c5c0f31
-
- 08 4月, 2020 1 次提交
-
-
由 Sagar Vemuri 提交于
Summary: Adding a simple timer support to schedule work at a fixed time. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6543 Test Plan: TODO: clean up the unit tests, and make them better. Reviewed By: siying Differential Revision: D20465390 Pulled By: sagar0 fbshipit-source-id: cba143f70b6339863e1d0f8b8bf92e51c2b3d678
-
- 05 4月, 2020 1 次提交
-
-
由 Peter Dillinger 提交于
Summary: When Travis times out, it's hard to determine whether the last executing thing took an excessively long time or the sum of all the work just exceeded the time limit. This change inserts some timestamps in the output that should make this easier to determine. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6643 Test Plan: CI (Travis mostly) Reviewed By: anand1976 Differential Revision: D20843901 Pulled By: pdillinger fbshipit-source-id: e7aae5434b0c609931feddf238ce4355964488b7
-
- 02 4月, 2020 1 次提交
-
-
由 Ziyue Yang 提交于
Summary: This PR adds support for pipelined & parallel compression optimization for `BlockBasedTableBuilder`. This optimization makes block building, block compression and block appending a pipeline, and uses multiple threads to accelerate block compression. Users can set `CompressionOptions::parallel_threads` greater than 1 to enable compression parallelism. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6262 Reviewed By: ajkr Differential Revision: D20651306 fbshipit-source-id: 62125590a9c15b6d9071def9dc72589c1696a4cb
-