Skip to content

R
rocksdb

kvdb / rocksdb
10 个月 前同步成功

通知 0
Star 0
Fork 0
R
rocksdb

GitCode（gitcode.net）2024年7月9日维护升级公告

前往新版Gitcode，体验更适合开发者的 AI 搜索 >>

切换分支/标签
  1. 13 2月, 2023 1 次提交
    • W
      remove dependency on options.h for port_posix.h andport_win.h (#11214) · 42d6652b
      由 Wentian Guo 提交于 
      Summary:
The files in `port/`, such as `port_posix.h`, are layering over the system libraries, so shouldn't include the DB-specific files like `options.h`. This PR remove this dependency.

# How
The reason that `port_posix.h` (or `port_win.h`) include `options.h` is to use `CpuPriority`, as there is a method `SetCpuPriority()` in `port_posix.h` that uses `CpuPriority.`
- I think `SetCpuPriority()` make sense to exist in `port_posix.h` as it provides has platform-dependent implementation
- `CpuPriority` enum is defined in `env.h`, but used in `rocksdb/include` and `port/`.

Hence, let us define `CpuPriority` enum in a common file, say `port_defs.h`, such that both directories `rocksdb/include` and `port/` can include.

When we remove this dependency, some other files have compile errors because they can't find definitions, so add header files to resolve

# Test
make all check -j

Pull Request resolved: https://github.com/facebook/rocksdb/pull/11214

Reviewed By: pdillinger

Differential Revision: D43196910

Pulled By: guowentian

fbshipit-source-id: 70deccb72844cfb08fcc994f76c6ef6df5d55ab9
      42d6652b
  3. 11 2月, 2023 1 次提交
  5. 10 2月, 2023 3 次提交
    • A
      Extend existing benchmarks seekrandom and multiread to run with async_io (#11170) · ab2157fa
      由 akankshamahajan 提交于 
      Summary:
=======================================================================
Benchmark seekrandom_asyncio

=======================================================================

db_bench_cmd=$(which time) -p ./db_bench       --benchmarks=seekrandom
--db=/tmp/rocksdb/regression_test/db --wal_dir=
--use_existing_db=0       --perf_level=1
--disable_auto_compactions       --threads=1       --num=1073741824
--reads=1073741824       --writes=1073741824       --deletes=1073741824
--key_size=100       --value_size=900       --cache_size=1073741824
--statistics=0              --compression_ratio=0.5       --histogram=1
--seek_nexts=10       --stats_per_interval=1
--stats_interval_seconds=600       --max_background_flushes=4
--num_multi_db=1       --max_background_compactions=16
--num_high_pri_threads=4       --num_low_pri_threads=16
--seed=1675181789       --multiread_batched=true       --batch_size=128
--multiread_stride=12       --async_io=true
--optimize_multiget_for_io=false 2>&1
RocksDB:    version 8.0.0

=======================================================================
 Benchmark multireadrandom_asyncio

====================================================================

db_bench_cmd=$(which time) -p ./db_bench
--benchmarks=multireadrandom --db=/tmp/rocksdb/regression_test/db
--wal_dir=       --use_existing_db=0       --perf_level=1
--disable_auto_compactions       --threads=1       --num=1073741824
--reads=1073741824       --writes=1073741824       --deletes=1073741824
--key_size=100       --value_size=900       --cache_size=1073741824
--statistics=0              --compression_ratio=0.5       --histogram=1
--seek_nexts=10       --stats_per_interval=1
--stats_interval_seconds=600       --max_background_flushes=4
--num_multi_db=1       --max_background_compactions=16
--num_high_pri_threads=4       --num_low_pri_threads=16
--seed=1675181841       --multiread_batched=true       --batch_size=128
--multiread_stride=12       --async_io=true
--optimize_multiget_for_io=true 2>&1
RocksDB:    version 8.0.0
Date:       Tue Jan 31 08:17:22 2023
CPU:        32 * Intel Xeon Processor (Skylake)
CPUCache:   16384 KB

Pull Request resolved: https://github.com/facebook/rocksdb/pull/11170

Reviewed By: ajkr, anand1976

Differential Revision: D42889107

Pulled By: akankshamahajan15

fbshipit-source-id: b819be2bd5f00d1db654b9e829b84f11e6bcab92
      ab2157fa
    • P
      Put Cache and CacheWrapper in new public header (#11192) · 3cacd4b4
      由 Peter Dillinger 提交于 
      Summary:
The definition of the Cache class should not be needed by the vast majority of RocksDB users, so I think it is just distracting to include it in cache.h, which is primarily needed for configuring and creating caches. This change moves the class to a new header advanced_cache.h. It is just cut-and-paste except for modifying the class API comment.

In general, operations on shared_ptr<Cache> should continue to work when only a forward declaration of Cache is available, as long as all the Cache instances provided are already shared_ptr. See https://stackoverflow.com/a/17650101/454544

Also, the most common way to customize a Cache is by wrapping an existing implementation, so it makes sense to provide CacheWrapper in the public API. This was a cut-and-paste job except removing the implementation of Name() so that derived classes must provide it.

Intended follow-up: consolidate Release() into one function to reduce customization bugs / confusion

Pull Request resolved: https://github.com/facebook/rocksdb/pull/11192

Test Plan: `make check`

Reviewed By: anand1976

Differential Revision: D43055487

Pulled By: pdillinger

fbshipit-source-id: 7b05492df35e0f30b581b4c24c579bc275b6d110
      3cacd4b4
    • P
      Attempt fix flaky DBWriteTest.LockWALInEffect (#11209) · b7747bbc
      由 Peter Dillinger 提交于 
      Summary:
Example failure:
```
[ RUN      ] DBWriteTestInstance/DBWriteTest.LockWALInEffect/1
db/db_write_test.cc:646: Failure
Put("key3", "value")
Corruption: Not active
```
Presumably from a background compaction prior to Put.

Pull Request resolved: https://github.com/facebook/rocksdb/pull/11209

Test Plan: watch CI

Reviewed By: akankshamahajan15

Differential Revision: D43147727

Pulled By: pdillinger

fbshipit-source-id: a1c34ac5ab124bfe2f23205a30777990056e9082
      b7747bbc
  7. 09 2月, 2023 3 次提交
    • P
      Improve SmallEnumSet (#11178) · 34bb3ddc
      由 Peter Dillinger 提交于 
      Summary:
In anticipation of using this to represent sets of CacheEntryRole for including or excluding kinds of blocks in block cache tiers, add significant new features to SmallEnumSet, including at least:

* List initialization
* Applicative constexpr operations
* copy/move/equality ops
* begin/end/const_iterator for iteration
* Better comments

Pull Request resolved: https://github.com/facebook/rocksdb/pull/11178

Test Plan: unit tests added/expanded

Reviewed By: ltamasi

Differential Revision: D42973723

Pulled By: pdillinger

fbshipit-source-id: 40783486feda931c3f7c6fcc9a300acd6a4b0a0a
      34bb3ddc
    • P
      Mitigate presumed OOM in CircleCI (#11206) · ee5305fa
      由 Peter Dillinger 提交于 
      Summary:
We've seen many instances of
build-linux-static_lib-alt_namespace-status_checked failing like this:
```
g++: fatal error: Killed signal terminated program cc1plus
compilation terminated.
make: *** [Makefile:2507: utilities/transactions/transaction_test.o]
Error 1
```

It's understandable that so many static linking jobs could exhaust memory.

The executor only has 16 vcores, so going from 32 down to 24 shouldn't hurt build time.

Pull Request resolved: https://github.com/facebook/rocksdb/pull/11206

Test Plan: will watch CI

Reviewed By: ajkr

Differential Revision: D43137246

Pulled By: pdillinger

fbshipit-source-id: 050b0f700c285dd913bcae8b4a76a44d04bb0356
      ee5305fa
    • A
      Fix bug in WAL streaming uncompression (#11198) · 77b61abc
      由 anand76 提交于 
      Summary:
Fix a bug in the calculation of the input buffer address/offset in log_reader.cc. The bug is when consecutive fragments of a compressed record are located at the same offset in the log reader buffer, the second fragment input buffer is treated as a leftover from the previous input buffer. As a result, the offset in the `ZSTD_inBuffer` is not reset.

Pull Request resolved: https://github.com/facebook/rocksdb/pull/11198

Test Plan: Add a unit test in log_test.cc that fails without the fix and passes with it.

Reviewed By: ajkr, cbi42

Differential Revision: D43102692

Pulled By: anand1976

fbshipit-source-id: aa2648f4802c33991b76a3233c5a58d4cc9e77fd
      77b61abc
  9. 08 2月, 2023 4 次提交
    • L
      Add compaction filter support for wide-column entities (#11196) · 876d2815
      由 Levi Tamasi 提交于 
      Summary:
The patch adds compaction filter support for wide-column entities by introducing
a new `CompactionFilter` API called `FilterV3`. This API is called for regular
key-values, merge operands, and wide-column entities as well. It is passed the
existing value/operand or wide-column structure and it can update the value or
columns or keep/delete/etc. the key-value as usual. For compatibility, the default
implementation of `FilterV3` keeps all wide-column entities and falls back to calling
`FilterV2` for plain old key-values and merge operands.

Pull Request resolved: https://github.com/facebook/rocksdb/pull/11196

Test Plan: `make check`

Reviewed By: akankshamahajan15

Differential Revision: D43094147

Pulled By: ltamasi

fbshipit-source-id: 75acabe9a35254f7f404ba6173ee9c2774382ebd
      876d2815
    • H
      Remove a couple deprecated convenience.h APIs (#11120) · 6650ca24
      由 Hui Xiao 提交于 
      Summary:
**Context/Summary:**
As instructed by convenience.h comments, a few deprecated APIs are removed.

Pull Request resolved: https://github.com/facebook/rocksdb/pull/11120

Test Plan:
- make check & CI
- eyeball check on test semantics.

Reviewed By: pdillinger

Differential Revision: D42937507

Pulled By: hx235

fbshipit-source-id: a9e4709387da01b1d0e9148c2e210f02e9746ee1
      6650ca24
    • P
      Revert to LIB_MODE=static for optimized builds (#11195) · b5827c80
      由 Peter Dillinger 提交于 
      Summary:
Continuous performance testing indicates there's a small performance hit with shared library (-fPIC) builds, so while retaining the motivation for https://github.com/facebook/rocksdb/issues/11168, we set the default for DEBUG_LEVEL=0 Makefile builds back to LIB_MODE=static.

Pull Request resolved: https://github.com/facebook/rocksdb/pull/11195

Test Plan: CI, with some updated checks and removal of some now obsolete LIB_MODE overrides

Reviewed By: cbi42

Differential Revision: D43090576

Pulled By: pdillinger

fbshipit-source-id: 755fe5d07005f85caf24e16f90228ffd46a6e250
      b5827c80
    • S
      Add kForceOptimized option to jni (#11181) · 68fa90ca
      由 Symious 提交于 
      Summary:
Currently the option of "KForceOptimized" is not included in CompactRangeOptions.BottommostLevelCompaction.

This PR is to add this option.

Pull Request resolved: https://github.com/facebook/rocksdb/pull/11181

Reviewed By: ajkr

Differential Revision: D43056453

Pulled By: cbi42

fbshipit-source-id: 22fd53f980ab1a86c61dd42e948902542065128f
      68fa90ca
  11. 07 2月, 2023 4 次提交
  13. 04 2月, 2023 5 次提交
    • P
      Fix compile gettid on older Linux (#11184) · 27cf0917
      由 Peter Dillinger 提交于 
      Summary:
Seen only in post-PR CI job benchmark-linux. Some context: https://stackoverflow.com/questions/30680550/c-gettid-was-not-declared-in-this-scope

Pull Request resolved: https://github.com/facebook/rocksdb/pull/11184

Test Plan: watch CI

Reviewed By: cbi42

Differential Revision: D43013891

Pulled By: pdillinger

fbshipit-source-id: 48b3b7231080a0f803fdc36d13946d5524770302
      27cf0917
    • P
      Use LIB_MODE=shared build by default with make (#11168) · cf756ed9
      由 Peter Dillinger 提交于 
      Summary:
With https://github.com/facebook/rocksdb/issues/11150 this becomes a practical change that I think is overall good for developer efficiency.

Pull Request resolved: https://github.com/facebook/rocksdb/pull/11168

Test Plan:
More efficient build of all unit tests and tools:

```
$ git clean -fdx
$ du -sh .
522M    .
$ /usr/bin/time make -j32 LIB_MODE=static
...
14270.63user 1043.33system 11:19.85elapsed 2252%CPU (0avgtext+0avgdata 1929944maxresident)k
...
$ du -sh .
62G     .
$
```
Vs.
```
$ git clean -fdx
$ du -sh .
522M    .
$ /usr/bin/time make -j32 LIB_MODE=shared
...
9479.87user 478.26system 7:20.82elapsed 2258%CPU (0avgtext+0avgdata 1929272maxresident)k
...
$ du -sh .
5.4G    .
$
```

So 1/3 less build time and >90% less space usage.

Individual unit test edit-compile-run is not too different. Modifying an average unit test source file:
```
$ touch db/version_builder_test.cc
$ /usr/bin/time make -j32 LIB_MODE=static version_builder_test
...
34.74user 3.37system 0:38.29elapsed 99%CPU (0avgtext+0avgdata 945520maxresident)k
```
Vs.
```
$ touch db/version_builder_test.cc
$ /usr/bin/time make -j32 LIB_MODE=shared version_builder_test
...
116.26user 43.91system 0:28.65elapsed 559%CPU (0avgtext+0avgdata 675160maxresident)k
```
A little faster with shared.

However, modifying an average DB implementation file has an extra linking step with shared lib:
```
$ touch db/db_impl/db_impl_files.cc
$ /usr/bin/time make -j32 LIB_MODE=static version_builder_test
...
33.17user 5.13system 0:39.70elapsed 96%CPU (0avgtext+0avgdata 945544maxresident)k
```
Vs.
```
$ touch db/db_impl/db_impl_files.cc
$ /usr/bin/time make -j32 LIB_MODE=shared version_builder_test
...
40.80user 4.66system 0:45.54elapsed 99%CPU (0avgtext+0avgdata 1056340maxresident)k
```
A little slower with shared.

On the whole, should be faster and lighter weight because of the many unit test files case

Reviewed By: cbi42

Differential Revision: D42894004

Pulled By: pdillinger

fbshipit-source-id: 9e827e52ace79b86f849b6a24466e318b4b605a7
      cf756ed9
    • P
      Support stack traces with gdb (and debugger invocation) (#11150) · e17f3105
      由 Peter Dillinger 提交于 
      Summary:
LIB_MODE=shared is much more efficient for building all the unit tests but comes with the downside of ugly stack traces, generally missing name demangling and source line info. Searching the internet suggests the reliable way to get stack traces with dynamic loading is with gdb.

This change automatically tries to use gdb to get a stack trace if built with LIB_MODE=shared, and only on Linux because that's where we have the capability to attach to the proper thread. (We could revise the exact conditions in the future.) If there's a failure invoking gdb, it falls back on the old method. Obscure details of making the output reasonable / pretty are in the source code comments.

Based on this, it was easy to make it so that running a test command with ROCKSDB_DEBUG=1 would invoke gdb whenever the stack trace handler was invoked, so I included that.

Intended follow-up: make LIB_MODE=shared the new default `make` build config

Pull Request resolved: https://github.com/facebook/rocksdb/pull/11150

Test Plan:
manual, mostly by injecting an "assert(false)" into a unit test and trying different build modes etc.

Although gdb is slower to start showing stack trace output, it seems overall faster in many if not most cases, presumably because it doesn't reload the symbol table for each stack entry. At least with parallel test runs, having many tests dumping stacks with the old method can take so long it appears to hang the test run.

Reviewed By: cbi42

Differential Revision: D42894064

Pulled By: pdillinger

fbshipit-source-id: 608143309d8c69c40049c9a4abcde4f22e87b4d8
      e17f3105
    • P
      Deprecate write_global_seqno and default to false (#11179) · 0cf1008f
      由 Peter Dillinger 提交于 
      Summary:
This option has long been intended to be set to false by default and deprecated. It might never be practical to completely remove the feature, so that we can continue to test for backward compatibility by keeping the ability to generate DBs in the old way.

Also improved API comments.

Pull Request resolved: https://github.com/facebook/rocksdb/pull/11179

Test Plan: existing tests (with one tiny update)

Reviewed By: hx235

Differential Revision: D42973927

Pulled By: pdillinger

fbshipit-source-id: e9bc161cb933266e094aea2dff8cc03753c39dab
      0cf1008f
    • P
      Ensure LockWAL() stall cleared for UnlockWAL() return (#11172) · 390cc0b1
      由 Peter Dillinger 提交于 
      Summary:
Fixes https://github.com/facebook/rocksdb/issues/11160

By counting the number of stalls placed on a write queue, we can check in UnlockWAL() whether the stall present at the start of UnlockWAL() has been cleared by the end, or wait until it's cleared.

More details in code comments and new unit test.

Pull Request resolved: https://github.com/facebook/rocksdb/pull/11172

Test Plan: unit test added. Yes, it uses sleep to amplify failure on buggy behavior if present, but using a sync point to only allow new behavior would fail with the old code only because it doesn't contain the new sync point. Basically, using a sync point in UnlockWAL() could easily mask a regression by artificially limiting key behaviors. The test would only check that UnlockWAL() invokes code that *should* do the right thing, without checking that it *does* the right thing.

Reviewed By: ajkr

Differential Revision: D42894341

Pulled By: pdillinger

fbshipit-source-id: 15c9da0ca383e6aec845b29f5447d76cecbf46c3
      390cc0b1
  15. 03 2月, 2023 4 次提交
    • A
      Return any errors returned by ReadAsync to the MultiGet caller (#11171) · 63da9cfa
      由 anand76 提交于 
      Summary:
Currently, we incorrectly return a Status::Corruption to the MultiGet caller if the file system ReadAsync cannot issue a read and returns an error for some reason, such as IOStatus::NotSupported(). In this PR, we copy the ReadAsync error to the request status so it can be returned to the user.

Tests:
Update existing unit tests and add a new one for this scenario

Pull Request resolved: https://github.com/facebook/rocksdb/pull/11171

Reviewed By: akankshamahajan15

Differential Revision: D42950057

Pulled By: anand1976

fbshipit-source-id: 85ffcb015fa6c064c311f8a28488fec78c487869
      63da9cfa
    • Y
      Enable crash test for user-defined timestamp and BlobDB combination (#11163) · 701a19cc
      由 Yu Zhang 提交于 
      Summary:
Enable the set of crash test for when user defined timestamp is enabled in combination with BlobDB.

Pull Request resolved: https://github.com/facebook/rocksdb/pull/11163

Test Plan: `make check` and `db_stress`/`db_crashtest.py` with various combinations.

Reviewed By: ltamasi

Differential Revision: D42906457

Pulled By: jowlyzhang

fbshipit-source-id: 6bec6449a4213b536c787420ff30a7d17b676deb
      701a19cc
    • C
      Remove NUMA setting for benchmark-linux (#11180) · fec5c8de
      由 changyubi 提交于 
      Summary:
benchmark-linux is failing on main branch after https://github.com/facebook/rocksdb/issues/11074 with the following error msg:
```
/usr/bin/time -f '%e %U %S' -o /tmp/benchmark-results/8.0.0/benchmark_overwriteandwait.t1.s0.log.time numactl --interleave=all timeout 1200 ./db_bench --benchmarks=overwrite,waitforcompaction,stats --use_existing_db=1 --sync=0 --level0_file_num_compaction_trigger=4 --level0_slowdown_writes_trigger=20 --level0_stop_writes_trigger=30 --max_background_jobs=4 --max_write_buffer_number=8 --undefok=use_blob_cache,use_shared_block_and_blob_cache,blob_cache_size,blob_cache_numshardbits,prepopulate_blob_cache,multiread_batched,cache_low_pri_pool_ratio,prepopulate_block_cache --db=/tmp/rocksdb-benchmark-datadir --wal_dir=/tmp/rocksdb-benchmark-datadir --num=20000000 --key_size=20 --value_size=400 --block_size=8192 --cache_size=10737418240 --cache_numshardbits=6 --compression_max_dict_bytes=0 --compression_ratio=0.5 --compression_type=none --bytes_per_sync=1048576 --cache_index_and_filter_blocks=1 --cache_high_pri_pool_ratio=0.5 --cache_low_pri_pool_ratio=0 --benchmark_write_rate_limit=0 --write_buffer_size=16777216 --target_file_size_base=16777216 --max_bytes_for_level_base=67108864 --verify_checksum=1 --delete_obsolete_files_period_micros=62914560 --max_bytes_for_level_multiplier=8 --statistics=0 --stats_per_interval=1 --stats_interval_seconds=20 --report_interval_seconds=1 --histogram=1 --memtablerep=skip_list --bloom_bits=10 --open_files=-1 --subcompactions=1 --compaction_style=0 --num_levels=8 --min_level_to_compress=3 --level_compaction_dynamic_level_bytes=true --pin_l0_filter_and_index_blocks_in_cache=1 --duration=600 --threads=1 --merge_operator="put" --seed=1675372532 --report_file=/tmp/benchmark-results/8.0.0/benchmark_overwriteandwait.t1.s0.log.r.csv 2>&1 | tee -a /tmp/benchmark-results/8.0.0/benchmark_overwriteandwait.t1.s0.log
/usr/bin/time: cannot run numactl: No such file or directory
```
This PR removes the newly added NUMA setting.

Pull Request resolved: https://github.com/facebook/rocksdb/pull/11180

Test Plan: check next main branch run for benchmark-linux

Reviewed By: ajkr

Differential Revision: D42975930

Pulled By: cbi42

fbshipit-source-id: f084d39aeba9877c0752502e879c5e612b507653
      fec5c8de
    • A
      CI Benchmarking. Small configuration changes based on performance analysis. (#11074) · 6781009e
      由 Alan Paxton 提交于 
      Summary:
First, we made a small reduction in DURATION_RW as runs were exceeding 1 hour and colliding with subsequent runs.

See Mark Callaghan’s blog post at http://smalldatum.blogspot.com/2023/01/variance-in-rocksdb-benchmarks-on-cloud.html

Configuration parameters which are not consistent with the following email from Mark (see the blog post for more context) have been updated. Where Mark has defined the parameter and we haven't, we define it explicitly. We will need to further monitor for an expected reduction in variance of test times:

To match what I did:
 ---

nsecs=1800
dbdir=/data/m/rx
resultdir=bm.lc.nt1.cm1.d0

env WRITE_BUFFER_SIZE_MB=16 TARGET_FILE_SIZE_BASE_MB=16 MAX_BYTES_FOR_LEVEL_BASE_MB=64 MAX_BACKGROUND_JOBS=4 NUM_KEYS=20000000 CACHE_SIZE_MB=10240 DURATION_RW=$nsecs DURATION_RO=$nsecs MB_WRITE_PER_SEC=2 NUM_THREADS=1 COMPRESSION_TYPE=none CACHE_INDEX_AND_FILTER_BLOCKS=1 VALUE_SIZE=400 NUMA=1 MIN_LEVEL_TO_COMPRESS=3 COMPACTION_STYLE=leveled bash benchmark_compare.sh $dbdir $resultdir 7.8.fb

env WRITE_BUFFER_SIZE_MB=16 TARGET_FILE_SIZE_BASE_MB=16 MAX_BYTES_FOR_LEVEL_BASE_MB=64 MAX_BACKGROUND_JOBS=4 NUM_KEYS=200000000 CACHE_SIZE_MB=10240 DURATION_RW=$nsecs DURATION_RO=$nsecs MB_WRITE_PER_SEC=2 NUM_THREADS=1 COMPRESSION_TYPE=lz4 CACHE_INDEX_AND_FILTER_BLOCKS=1 VALUE_SIZE=400 NUMA=1 MIN_LEVEL_TO_COMPRESS=3 COMPACTION_STYLE=leveled bash benchmark_compare.sh $dbdir $resultdir 7.8.fb

Pull Request resolved: https://github.com/facebook/rocksdb/pull/11074

Reviewed By: ajkr

Differential Revision: D42969668

Pulled By: cbi42

fbshipit-source-id: 1ea4e6a3901be4016108f93817eb58f74baac21a
      6781009e
  17. 02 2月, 2023 2 次提交
    • A
      Update HISTORY.md for #11136 (#11177) · 6af16ac7
      由 Andrew Kryczka 提交于 
      Summary: Pull Request resolved: https://github.com/facebook/rocksdb/pull/11177

Reviewed By: cbi42

Differential Revision: D42948946

Pulled By: ajkr

fbshipit-source-id: 783d3d9007faaa036923a0364cdd0bfbd8e78062
      6af16ac7
    • L
      Clean up InvokeFilterIfNeeded a bit (#11174) · df680b24
      由 Levi Tamasi 提交于 
      Summary:
The patch makes some code quality enhancements in `CompactionIterator::InvokeFilterIfNeeded`
including the renaming of `filter` (which is most likely a remnant of the days before the `FilterV2`
API when the compaction filter used to return a boolean) to `decision`, the removal of some
outdated comments, the elimination of an `error` flag which was only used in one failure case
out of many, as well as some small stylistic improvements. (Some the above will also come in
handy when adding compaction filter support for wide-column entities.)

Pull Request resolved: https://github.com/facebook/rocksdb/pull/11174

Test Plan: `make check`

Reviewed By: akankshamahajan15

Differential Revision: D42901408

Pulled By: ltamasi

fbshipit-source-id: ab382d59a4990c5dfe1cee219d49e1d80902b666
      df680b24
  19. 01 2月, 2023 3 次提交
  21. 31 1月, 2023 3 次提交
    • P
      Cleanup, improve, stress test LockWAL() (#11143) · 94e3beec
      由 Peter Dillinger 提交于 
      Summary:
The previous API comments for LockWAL didn't provide much about why you might want to use it, and didn't really meet what one would infer its contract was. Also, LockWAL was not in db_stress / crash test. In this change:

* Implement a counting semantics for LockWAL()+UnlockWAL(), so that they can safely be used concurrently across threads or recursively within a thread. This should make the API much less bug-prone and easier to use.
* Make sure no UnlockWAL() is needed after non-OK LockWAL() (to match RocksDB conventions)
* Make UnlockWAL() reliably return non-OK when there's no matching LockWAL() (for debug-ability)
* Clarify API comments on LockWAL(), UnlockWAL(), FlushWAL(), and SyncWAL(). Their exact meanings are not obvious, and I don't think it's appropriate to talk about implementation mutexes in the API comments, but about what operations might block each other.
* Add LockWAL()/UnlockWAL() to db_stress and crash test, mostly to check for assertion failures, but also checks that latest seqno doesn't change while WAL is locked. This is simpler to add when LockWAL() is allowed in multiple threads.
* Remove unnecessary use of sync points in test DBWALTest::LockWal. There was a bug during development of above changes that caused this test to fail sporadically, with and without this sync point change.

Pull Request resolved: https://github.com/facebook/rocksdb/pull/11143

Test Plan: unit tests added / updated, added to stress/crash test

Reviewed By: ajkr

Differential Revision: D42848627

Pulled By: pdillinger

fbshipit-source-id: 6d976c51791941a31fd8fbf28b0f82e888d9f4b4
      94e3beec
    • S
      DB Stress to fix a false assertion (#11164) · 36174d89
      由 sdong 提交于 
      Summary:
Seeting this error in stress test:

db_stress: internal_repo_rocksdb/repo/db_stress_tool/db_stress_test_base.cc:2459: void rocksdb::StressTest::Open(rocksdb::SharedState *): Assertion `txn_db_ == nullptr' failed. Received signal 6 (Aborted)
......

It doesn't appear that txn_db_ is set to nullptr at all. We set ithere.

Pull Request resolved: https://github.com/facebook/rocksdb/pull/11164

Test Plan: Run db_stress transaction and non-transation with low kill rate and see restarting without assertion

Reviewed By: ajkr

Differential Revision: D42855662

fbshipit-source-id: 06816d37cce9c94a81cb54ab238fb73aa102ed46
      36174d89
    • Y
      Use user key on sst file for blob verification for Get and MultiGet (#11105) · 24ac53d8
      由 Yu Zhang 提交于 
      Summary:
Use the user key on sst file for blob verification for `Get` and `MultiGet` instead of the user key passed from caller.

Add tests for `Get` and `MultiGet` operations when user defined timestamp feature is enabled in a BlobDB.

Pull Request resolved: https://github.com/facebook/rocksdb/pull/11105

Test Plan:
make V=1 db_blob_basic_test
./db_blob_basic_test --gtest_filter="DBBlobTestWithTimestamp.*"

Reviewed By: ltamasi

Differential Revision: D42716487

Pulled By: jowlyzhang

fbshipit-source-id: 5987ecbb7e56ddf46d2467a3649369390789506a
      24ac53d8
  23. 28 1月, 2023 3 次提交
    • A
      Move ExternalSSTTestEnv to FileSystemWrapper (#11139) · 79e57a39
      由 akankshamahajan 提交于 
      Summary:
Migrate ExternalSSTTestEnv to FileSystemWrapper

Pull Request resolved: https://github.com/facebook/rocksdb/pull/11139

Reviewed By: anand1976

Differential Revision: D42780180

Pulled By: akankshamahajan15

fbshipit-source-id: 9a4448c9fe5186b518235fe11e1a34dcad897cdd
      79e57a39
    • S
      Remove RocksDB LITE (#11147) · 4720ba43
      由 sdong 提交于 
      Summary:
We haven't been actively mantaining RocksDB LITE recently and the size must have been gone up significantly. We are removing the support.

Most of changes were done through following comments:

unifdef -m -UROCKSDB_LITE `git grep -l ROCKSDB_LITE | egrep '[.](cc|h)'`

by Peter Dillinger. Others changes were manually applied to build scripts, CircleCI manifests, ROCKSDB_LITE is used in an expression and file db_stress_test_base.cc.

Pull Request resolved: https://github.com/facebook/rocksdb/pull/11147

Test Plan: See CI

Reviewed By: pdillinger

Differential Revision: D42796341

fbshipit-source-id: 4920e15fc2060c2cd2221330a6d0e5e65d4b7fe2
      4720ba43
    • Y
      Remove deprecated util functions in options_util.h (#11126) · 6943ff6e
      由 Yu Zhang 提交于 
      Summary:
Remove the util functions in options_util.h that have previously been marked deprecated.

Pull Request resolved: https://github.com/facebook/rocksdb/pull/11126

Test Plan: `make check`

Reviewed By: ltamasi

Differential Revision: D42757496

Pulled By: jowlyzhang

fbshipit-source-id: 2a138a3c207d0e0e0bbb4d99548cf2cadb44bcfb
      6943ff6e
  25. 27 1月, 2023 4 次提交