Skip to content

R
rocksdb

kvdb / rocksdb
12 个月 前同步成功

通知 0
Star 0
Fork 0
R
rocksdb

体验新版 GitCode，发现更多精彩内容 >>

切换分支/标签
  1. 04 12月, 2020 1 次提交
  3. 03 12月, 2020 4 次提交
    • Z
      Add kManifestWriteNoWAL to BackgroundErrorReason to handle Flush IO Error when... · 29e8f6a6
      由 Zhichao Cao 提交于 
      Add kManifestWriteNoWAL to BackgroundErrorReason to handle Flush IO Error when WAL is disabled (#7693)

Summary:
In the current code base, all the manifest writes with IO error will be set with reason: BackgroundErrorReason::kManifestWrite, which will be mapped to the kHardError if the IO Error is retryable. However, if the system does not use the WAL, all the retryable IO error should be mapped to kSoftError. Create this PR to handle is special case by adding kManifestWriteNoWAL to BackgroundErrorReason.

Pull Request resolved: https://github.com/facebook/rocksdb/pull/7693

Test Plan: make check, add new testing cases to error_handler_fs_test

Reviewed By: anand1976

Differential Revision: D25066204

Pulled By: zhichao-cao

fbshipit-source-id: d59553896c2eac3fb37c05238544d2b265379462
      29e8f6a6
    • P
      Skip minimum rate check in Sandcastle (#7728) · 3b9bfe8f
      由 Peter Dillinger 提交于 
      Summary:
The minimum rate check in RateLimiterTest.Rate can fail in
Facebook's CI system Sandcastle, presumably due to heavily loaded
machines. This change disables the minimum rate check for Sandcastle
runs, and cleans up the code disabling it on other CI environments. (The
amount of conditionally compiled code shall be minimized.)

Pull Request resolved: https://github.com/facebook/rocksdb/pull/7728

Test Plan: try new test with and without setting envvar SANDCASTLE=1

Reviewed By: ltamasi

Differential Revision: D25247642

Pulled By: pdillinger

fbshipit-source-id: d786233af37af9a874adbb3a9e2707ec52c27a5a
      3b9bfe8f
    • J
      Make CompactRange and GetApproximateSizes work with timestamp (#7684) · 7fec715d
      由 Jay Zhuang 提交于 
      Summary:
Add timestamp to the `CompactRange()` and `GetApproximateSizes` range keys if needed.

Pull Request resolved: https://github.com/facebook/rocksdb/pull/7684

Test Plan: make check

Reviewed By: riversand963

Differential Revision: D25015421

Pulled By: jay-zhuang

fbshipit-source-id: 51ca0756087eb053a3b11801e5c7ce1c6e2d38a9
      7fec715d
    • Y
      Fix assertion failure in bg flush (#7362) · e062a719
      由 Yanqin Jin 提交于 
      Summary:
https://github.com/facebook/rocksdb/issues/7340 reports and reproduces an assertion failure caused by a combination of the following:
- atomic flush is disabled.
- a column family can appear multiple times in the flush queue at the same time. This behavior was introduced in release 5.17.

Consequently, it is possible that two flushes race with each other. One bg flush thread flushes all memtables. The other thread calls `FlushMemTableToOutputFile()` afterwards, and hits the assertion error below.

```
  assert(cfd->imm()->NumNotFlushed() != 0);
  assert(cfd->imm()->IsFlushPending());
```

Fix this by reverting the behavior. In non-atomic-flush case, a column family can appear in the flush queue at most once at the same time.

Pull Request resolved: https://github.com/facebook/rocksdb/pull/7362

Test Plan:
make check
Also run stress test successfully for 10 times.
```
make crash_test
```

Reviewed By: ajkr

Differential Revision: D25172996

Pulled By: riversand963

fbshipit-source-id: f1559b6366cc609e961e3fc83fae548f1fad08ce
      e062a719
  5. 02 12月, 2020 3 次提交
  7. 01 12月, 2020 2 次提交
    • A
      Fix kPointInTimeRecovery handling of truncated WAL (#7701) · eb65d673
      由 Andrew Kryczka 提交于 
      Summary:
WAL may be truncated to an incomplete record due to crash while writing
the last record or corruption. In the former case, no hole will be
produced since no ACK'd data was lost. In the latter case, a hole could
be produced without this PR since we proceeded to recover the next WAL
as if nothing happened. This PR changes the record reading code to
always report a corruption for incomplete records in
`kPointInTimeRecovery` mode, and the upper layer will only ignore them
if the next WAL has consecutive seqnum (i.e., we are guaranteed no
hole).

While this solves the hole problem for the case of incomplete
records, the possibility is still there if the WAL is corrupted by
truncation to an exact record boundary. This PR also regresses how much data
can be recovered when writes are mixed with/without
`WriteOptions::disableWAL`, as then we can not distinguish between a
seqnum gap caused by corruption and a seqnum gap caused by a `disableWAL` write.

Pull Request resolved: https://github.com/facebook/rocksdb/pull/7701

Test Plan:
Interestingly there already was a test for this case
(`DBWALTestWithParams.kPointInTimeRecovery`); it just had a typo bug in
the verification that prevented it from noticing holes in recovery.

Reviewed By: anand1976

Differential Revision: D25111765

Pulled By: ajkr

fbshipit-source-id: 5e330b13b1ee2b5be096cea9d0ff6075843e57b6
      eb65d673
    • S
      Fix merge operator docs typo (#7716) · cc431ece
      由 Steve Yen 提交于 
      Summary: Pull Request resolved: https://github.com/facebook/rocksdb/pull/7716

Reviewed By: pdillinger

Differential Revision: D25214340

Pulled By: zhichao-cao

fbshipit-source-id: 143a8e7d076917e60bbe6993d60ec55f33e2ab56
      cc431ece
  9. 24 11月, 2020 2 次提交
    • L
      Integrated blob garbage collection: relocate blobs (#7694) · 51a8dc6d
      由 Levi Tamasi 提交于 
      Summary:
The patch adds basic garbage collection support to the integrated BlobDB
implementation. Valid blobs residing in the oldest blob files are relocated
as they are encountered during compaction. The threshold that determines
which blob files qualify is computed based on the configuration option
`blob_garbage_collection_age_cutoff`, which was introduced in https://github.com/facebook/rocksdb/issues/7661 .
Once a blob is retrieved for the purposes of relocation, it passes through the
same logic that extracts large values to blob files in general. This means that
if, for instance, the size threshold for key-value separation (`min_blob_size`)
got changed or writing blob files got disabled altogether, it is possible for the
value to be moved back into the LSM tree. In particular, one way to re-inline
all blob values if needed would be to perform a full manual compaction with
`enable_blob_files` set to `false`, `enable_blob_garbage_collection` set to
`true`, and `blob_file_garbage_collection_age_cutoff` set to `1.0`.

Some TODOs that I plan to address in separate PRs:

1) We'll have to measure the amount of new garbage in each blob file and log
`BlobFileGarbage` entries as part of the compaction job's `VersionEdit`.
(For the time being, blob files are cleaned up solely based on the
`oldest_blob_file_number` relationships.)
2) When compression is used for blobs, the compression type hasn't changed,
and the blob still qualifies for being written to a blob file, we can simply copy
the compressed blob to the new file instead of going through decompression
and compression.
3) We need to update the formula for computing write amplification to account
for the amount of data read from blob files as part of GC.

Pull Request resolved: https://github.com/facebook/rocksdb/pull/7694

Test Plan: `make check`

Reviewed By: riversand963

Differential Revision: D25069663

Pulled By: ltamasi

fbshipit-source-id: bdfa8feb09afcf5bca3b4eba2ba72ce2f15cd06a
      51a8dc6d
    • A
      Return `Status` from `MemTable` mutation functions (#7656) · dd6b7fc5
      由 Andrew Kryczka 提交于 
      Summary:
This PR updates `MemTable::Add()`, `MemTable::Update()`, and
`MemTable::UpdateCallback()` to return `Status` objects, and adapts the
client code in `MemTableInserter`. The goal is to prepare these
functions for key-value checksum, where we want to verify key-value
integrity while adding to memtable. After this PR, the memtable mutation
functions can report a failed integrity check by returning `Status::Corruption`.

Pull Request resolved: https://github.com/facebook/rocksdb/pull/7656

Reviewed By: riversand963

Differential Revision: D24900497

Pulled By: ajkr

fbshipit-source-id: 1a7e80581e3774676f2bbba2f0a0b04890f40009
      dd6b7fc5
  11. 23 11月, 2020 1 次提交
    • P
      Add Ribbon schema test to bloom_test (#7696) · 0baa5055
      由 Peter Dillinger 提交于 
      Summary:
These new unit tests should ensure that we don't accidentally
change the interpretation of bits for what I call Standard128Ribbon
filter internally, available publicly as NewExperimentalRibbonFilterPolicy.
There is very little intuitive reason for the values we check against in
these tests; I just plug in the right expected values upon watching the
test fail initially.

Most (but not all) of the tests are essentially "whitebox" "round-trip." We
create a filter from fixed keys, and first compare the checksum of those
filter bytes against a saved value. We also run queries against other fixed
keys, comparing which return false positives against a saved set.

Pull Request resolved: https://github.com/facebook/rocksdb/pull/7696

Test Plan: test addition and refactoring only

Reviewed By: jay-zhuang

Differential Revision: D25082289

Pulled By: pdillinger

fbshipit-source-id: b5ca646fdcb5a1c2ad2085eda4a1fd44c4287f67
      0baa5055
  13. 21 11月, 2020 1 次提交
  15. 20 11月, 2020 3 次提交
  17. 19 11月, 2020 2 次提交
    • C
      Do not track empty WALs (#7697) · 7169ca9c
      由 Cheng Chang 提交于 
      Summary:
An empty WAL won't be backed up by the BackupEngine. So if we track the empty WALs in MANIFEST, then when restoring from a backup, it may report corruption that the empty WAL is missing, which is correct because the WAL is actually in the main DB but not in the backup DB, but missing an empty WAL does not logically break DB consistency.

Pull Request resolved: https://github.com/facebook/rocksdb/pull/7697

Test Plan: watch existing tests to pass

Reviewed By: pdillinger

Differential Revision: D25077194

Pulled By: cheng-chang

fbshipit-source-id: 01917b57234b92b6063925f2ee9452c5732bdc03
      7169ca9c
    • C
      Call out a bug in HISTORY (#7690) · 8a97f356
      由 Cheng Chang 提交于 
      Summary:
It's worth mentioning the corner case bug fixed in PR 7621.

Pull Request resolved: https://github.com/facebook/rocksdb/pull/7690

Test Plan: N/A

Reviewed By: zhichao-cao

Differential Revision: D25056678

Pulled By: cheng-chang

fbshipit-source-id: 1ab42ec080f3ffe21f5d97acf65ee0af993112ba
      8a97f356
  19. 18 11月, 2020 5 次提交
  21. 17 11月, 2020 6 次提交
    • Y
      Fix the logic of setting read_amp_bytes_per_bit from OPTIONS file (#7680) · 84a70081
      由 Yanqin Jin 提交于 
      Summary:
Instead of using `EncodeFixed32` which always serialize a integer to
little endian, we should use the local machine's endianness when
populating a native data structure during options parsing.
Without this fix, `read_amp_bytes_per_bit` may be populated incorrectly
on big-endian machines.

Pull Request resolved: https://github.com/facebook/rocksdb/pull/7680

Test Plan: make check

Reviewed By: pdillinger

Differential Revision: D24999166

Pulled By: riversand963

fbshipit-source-id: dc603cff6e17f8fa32479ce6df93b93082e6b0c4
      84a70081
    • Y
      Clean up after two test failures in db_basic_test (#7682) · 869f0538
      由 Yanqin Jin 提交于 
      Summary:
In db_basic_test.cc, there are two tests that rely on the underlying
system's `LockFile` support to function correctly:
DBBasicTest.OpenWhenOpen and DBBasicTest.CheckLock. In both tests,
re-opening a db using `DB::Open` is expected to fail because the second
open cannot lock the LOCK file. Some distributed file systems, e.g. HDFS
do not support the POSIX-style file lock. Therefore, these unit tests will cause
assertion failure and the second `Open` will create a db instance.
Currently, these db instances are not closed after the assertion
failure. Since these db instances are registered with some process-wide, static
data structures, e.g. `PeriodicWorkScheduler::Default()`, they can still be
accessed after the unit tests. However, the `Env` object created for this db
instance is destroyed when the test finishes in `~DBTestBase()`. Consequently,
it causes illegal memory access.

Pull Request resolved: https://github.com/facebook/rocksdb/pull/7682

Test Plan:
Run the following on a distrubited file system:
```
make check
```

Reviewed By: anand1976

Differential Revision: D25004215

Pulled By: riversand963

fbshipit-source-id: f4327d7716c0e72b13bb43737ec9a5d156da4d52
      869f0538
    • A
      Use default FileSystem in GenerateUniqueId (#7672) · 9627e342
      由 anand76 提交于 
      Summary:
Use ```FileSystem::Default``` to read ```/proc/sys/kernel/uuid```, so it works for ```Envs``` with remote ```FileSystem``` as well.

Pull Request resolved: https://github.com/facebook/rocksdb/pull/7672

Reviewed By: riversand963

Differential Revision: D24998702

Pulled By: anand1976

fbshipit-source-id: fa95c1d70f0e4ed17561201f047aa055046d06c3
      9627e342
    • A
      Fail early when `merge_operator` not configured (#7667) · 1c5f13f2
      由 Andrew Kryczka 提交于 
      Summary:
An application may accidentally write merge operands without properly configuring `merge_operator`. We should alert them as early as possible that there's an API misuse. Previously RocksDB only notified them when a query or background operation needed to merge but couldn't. With this PR, RocksDB notifies them of the problem before applying the merge operand to the memtable (although it may already be in WAL, which seems it'd cause a crash loop until they enable `merge_operator`).

Pull Request resolved: https://github.com/facebook/rocksdb/pull/7667

Reviewed By: riversand963

Differential Revision: D24933360

Pulled By: ajkr

fbshipit-source-id: 3a4a2ceb0b7aed184113dd03b8efd735a8332f7f
      1c5f13f2
    • J
      add ArangoDB to USERS.md, and fix typos in that file (#7675) · 7582c568
      由 jsteemann 提交于 
      Summary:
Add ArangoDB to USERS.md.
We are using RocksDB since 2016.

Pull Request resolved: https://github.com/facebook/rocksdb/pull/7675

Reviewed By: riversand963

Differential Revision: D24998955

Pulled By: ajkr

fbshipit-source-id: 82c656bf56589e52aff8c491bab6fbc19b52cc91
      7582c568
    • M
      Add arena_block_size flag to db_bench (#7654) · 1861de45
      由 Mammo, Mulugeta 提交于 
      Summary:
db_bench currently does not allow overriding the default `arena_block_size `calculation ([memtable size/8](https://github.com/facebook/rocksdb/blob/master/db/column_family.cc#L216)). For memtables whose size is in gigabytes, the `arena_block_size` defaults to hundreds of megabytes (affecting performance).

Exposing this option in db_bench would allow us to test the workloads with various `arena_block_size` values.

Pull Request resolved: https://github.com/facebook/rocksdb/pull/7654

Reviewed By: jay-zhuang

Differential Revision: D24996812

Pulled By: ajkr

fbshipit-source-id: a5e3d2c83d9f89e1bb8382f2e8dd476c79e33bef
      1861de45
  23. 16 11月, 2020 1 次提交
  25. 14 11月, 2020 2 次提交
    • C
      Do not track WAL in MANIFEST when fsync is disabled in a test (#7669) · 1aae4178
      由 Cheng Chang 提交于 
      Summary:
If fsync is disabled in a unit test, then do not track WAL in MANIFEST, because on DB recovery, the WAL might be missing because the directory is not fsynced.

Pull Request resolved: https://github.com/facebook/rocksdb/pull/7669

Test Plan: Tests with fsync enabled should pass.

Reviewed By: riversand963

Differential Revision: D24941431

Pulled By: cheng-chang

fbshipit-source-id: ab3ff0f90769795cfb4e4d6dcf084ea5545d1975
      1aae4178
    • Y
      Hack to load OPTIONS file for read_amp_bytes_per_bit (#7659) · 9aa1b1dc
      由 Yanqin Jin 提交于 
      Summary:
A temporary hack to work around a bug in 6.10, 6.11, 6.12, 6.13 and
6.14. The bug will write out 8 bytes to OPTIONS file from the starting
address of BlockBasedTableOptions.read_amp_bytes_per_bit which is
actually a uint32. Consequently, the value of read_amp_bytes_per_bit
written in the OPTIONS file is wrong. From 6.15, RocksDB will
try to parse the read_amp_bytes_per_bit from OPTIONS file as a uint32.
To be able to load OPTIONS file generated by affected releases before
the fix, we need to manually parse read_amp_bytes_per_bit with this hack.

Pull Request resolved: https://github.com/facebook/rocksdb/pull/7659

Test Plan:
Generate a db with current 6.14.fb (head at https://github.com/facebook/rocksdb/commit/b6db05dbb5364c658c5401a8078d73697bb5f31d). Maybe use db_stress.

Checkout this PR, run
```
 ~/rocksdb/ldb --db=. --try_load_options --ignore_unknown_options idump --count_only
```
Expect success, and should not see
```
Failed: Invalid argument: Error parsing read_amp_bytes_per_bit:17179869184
```

Also
make check

Reviewed By: anand1976

Differential Revision: D24954752

Pulled By: riversand963

fbshipit-source-id: c7b802fc3e52acd050a4fc1cd475016122234394
      9aa1b1dc
  27. 13 11月, 2020 7 次提交
    • A
      Update option "allow_data_in_errors" in BuildOptions (#7665) · e300ce21
      由 Akanksha Mahajan 提交于 
      Summary:
"allow_data_in_errors" is not updated in BuildOptions. So it
would assume default value when BuildOptions is called.

Pull Request resolved: https://github.com/facebook/rocksdb/pull/7665

Test Plan: make check -j64

Reviewed By: zhichao-cao

Differential Revision: D24929100

Pulled By: akankshamahajan15

fbshipit-source-id: dd6225a6c9f13b20027ff1b6de8e79801b57b3f7
      e300ce21
    • P
      Experimental (production candidate) SST schema for Ribbon filter (#7658) · 60af9643
      由 Peter Dillinger 提交于 
      Summary:
Added experimental public API for Ribbon filter:
NewExperimentalRibbonFilterPolicy(). This experimental API will
take a "Bloom equivalent" bits per key, and configure the Ribbon
filter for the same FP rate as Bloom would have but ~30% space
savings. (Note: optimize_filters_for_memory is not yet implemented
for Ribbon filter. That can be added with no effect on schema.)

Internally, the Ribbon filter is configured using a "one_in_fp_rate"
value, which is 1 over desired FP rate. For example, use 100 for 1%
FP rate. I'm expecting this will be used in the future for configuring
Bloom-like filters, as I expect people to more commonly hold constant
the filter accuracy and change the space vs. time trade-off, rather than
hold constant the space (per key) and change the accuracy vs. time
trade-off, though we might make that available.

### Benchmarking

```
$ ./filter_bench -impl=2 -quick -m_keys_total_max=200 -average_keys_per_filter=100000 -net_includes_hashing
Building...
Build avg ns/key: 34.1341
Number of filters: 1993
Total size (MB): 238.488
Reported total allocated memory (MB): 262.875
Reported internal fragmentation: 10.2255%
Bits/key stored: 10.0029
----------------------------
Mixed inside/outside queries...
  Single filter net ns/op: 18.7508
  Random filter net ns/op: 258.246
    Average FP rate %: 0.968672
----------------------------
Done. (For more info, run with -legend or -help.)
$ ./filter_bench -impl=3 -quick -m_keys_total_max=200 -average_keys_per_filter=100000 -net_includes_hashing
Building...
Build avg ns/key: 130.851
Number of filters: 1993
Total size (MB): 168.166
Reported total allocated memory (MB): 183.211
Reported internal fragmentation: 8.94626%
Bits/key stored: 7.05341
----------------------------
Mixed inside/outside queries...
  Single filter net ns/op: 58.4523
  Random filter net ns/op: 363.717
    Average FP rate %: 0.952978
----------------------------
Done. (For more info, run with -legend or -help.)
```

168.166 / 238.488 = 0.705  -> 29.5% space reduction

130.851 / 34.1341 = 3.83x construction time for this Ribbon filter vs. lastest Bloom filter (could make that as little as about 2.5x for less space reduction)

### Working around a hashing "flaw"

bloom_test discovered a flaw in the simple hashing applied in
StandardHasher when num_starts == 1 (num_slots == 128), showing an
excessively high FP rate.  The problem is that when many entries, on the
order of number of hash bits or kCoeffBits, are associated with the same
start location, the correlation between the CoeffRow and ResultRow (for
efficiency) can lead to a solution that is "universal," or nearly so, for
entries mapping to that start location. (Normally, variance in start
location breaks the effective association between CoeffRow and
ResultRow; the same value for CoeffRow is effectively different if start
locations are different.) Without kUseSmash and with num_starts > 1 (thus
num_starts ~= num_slots), this flaw should be completely irrelevant.  Even
with 10M slots, the chances of a single slot having just 16 (or more)
entries map to it--not enough to cause an FP problem, which would be local
to that slot if it happened--is 1 in millions. This spreadsheet formula
shows that: =1/(10000000*(1 - POISSON(15, 1, TRUE)))

As kUseSmash==false (the setting for Standard128RibbonBitsBuilder) is
intended for CPU efficiency of filters with many more entries/slots than
kCoeffBits, a very reasonable work-around is to disallow num_starts==1
when !kUseSmash, by making the minimum non-zero number of slots
2*kCoeffBits. This is the work-around I've applied. This also means that
the new Ribbon filter schema (Standard128RibbonBitsBuilder) is not
space-efficient for less than a few hundred entries. Because of this, I
have made it fall back on constructing a Bloom filter, under existing
schema, when that is more space efficient for small filters. (We can
change this in the future if we want.)

TODO: better unit tests for this case in ribbon_test, and probably
update StandardHasher for kUseSmash case so that it can scale nicely to
small filters.

### Other related changes

* Add Ribbon filter to stress/crash test
* Add Ribbon filter to filter_bench as -impl=3
* Add option string support, as in "filter_policy=experimental_ribbon:5.678;"
where 5.678 is the Bloom equivalent bits per key.
* Rename internal mode BloomFilterPolicy::kAuto to kAutoBloom
* Add a general BuiltinFilterBitsBuilder::CalculateNumEntry based on
binary searching CalculateSpace (inefficient), so that subclasses
(especially experimental ones) don't have to provide an efficient
implementation inverting CalculateSpace.
* Minor refactor FastLocalBloomBitsBuilder for new base class
XXH3pFilterBitsBuilder shared with new Standard128RibbonBitsBuilder,
which allows the latter to fall back on Bloom construction in some
extreme cases.
* Mostly updated bloom_test for Ribbon filter, though a test like
FullBloomTest::Schema is a next TODO to ensure schema stability
(in case this becomes production-ready schema as it is).
* Add some APIs to ribbon_impl.h for configuring Ribbon filters.
Although these are reasonably covered by bloom_test, TODO more unit
tests in ribbon_test
* Added a "tool" FindOccupancyForSuccessRate to ribbon_test to get data
for constructing the linear approximations in GetNumSlotsFor95PctSuccess.

Pull Request resolved: https://github.com/facebook/rocksdb/pull/7658

Test Plan:
Some unit tests updated but other testing is left TODO. This
is considered experimental but laying down schema compatibility as early
as possible in case it proves production-quality. Also tested in
stress/crash test.

Reviewed By: jay-zhuang

Differential Revision: D24899349

Pulled By: pdillinger

fbshipit-source-id: 9715f3e6371c959d923aea8077c9423c7a9f82b8
      60af9643
    • L
      Add options for integrated blob GC (#7661) · bbbb5a28
      由 Levi Tamasi 提交于 
      Summary:
This patch simply adds a couple of options that will enable users to
configure garbage collection when using the integrated BlobDB
implementation. The actual GC logic will be added in a separate step.

Pull Request resolved: https://github.com/facebook/rocksdb/pull/7661

Test Plan: `make check`

Reviewed By: riversand963

Differential Revision: D24906544

Pulled By: ltamasi

fbshipit-source-id: ee0e056a712a4b4475cd90de8b27d969bd61b7e1
      bbbb5a28
    • Y
      Add full_history_ts_low_ to FlushJob (#7655) · 76ef894f
      由 Yanqin Jin 提交于 
      Summary:
https://github.com/facebook/rocksdb/issues/7556 enables `CompactionIterator` to perform garbage collection during compaction according
to a lower bound (user-defined) timestamp `full_history_ts_low_`.
This PR adds a data member `full_history_ts_low_` of type `std::string` to `FlushJob`, and
`full_history_ts_low_` does not change during flush. `FlushJob` will pass a pointer to this data member
to the `CompactionIterator` used during flush.

Also refactored flush_job_test.cc to re-use some existing code, which is actually the majority of this PR.

Pull Request resolved: https://github.com/facebook/rocksdb/pull/7655

Test Plan: make check

Reviewed By: ltamasi

Differential Revision: D24933340

Pulled By: riversand963

fbshipit-source-id: 2e584bfd0cf6e5c295ab1af264e68e9d6a12fca3
      76ef894f
    • L
      Fix InternalStats::DumpCFStats (#7666) · bb69b4ce
      由 Levi Tamasi 提交于 
      Summary:
https://github.com/facebook/rocksdb/pull/7461 accidentally broke
`InternalStats::DumpCFStats` by making `DumpCFFileHistogram` overwrite
the output of `DumpCFStatsNoFileHistogram` instead of appending to it,
resulting in only the file histogram related information getting logged.
The patch fixes this by reverting to appending in `DumpCFFileHistogram`.

Fixes https://github.com/facebook/rocksdb/issues/7664 .

Pull Request resolved: https://github.com/facebook/rocksdb/pull/7666

Test Plan: Ran `make check` and checked the info log of `db_bench`.

Reviewed By: riversand963

Differential Revision: D24929051

Pulled By: ltamasi

fbshipit-source-id: 636a3d5ebb5ce23de4f3fe4f03ad3f16cb2858f8
      bb69b4ce
    • Y
      Add full_history_ts_low_ to CompactionJob (#7657) · cf9d8e45
      由 Yanqin Jin 提交于 
      Summary:
https://github.com/facebook/rocksdb/issues/7556 enables `CompactionIterator` to perform garbage collection during compaction according
to a lower bound (user-defined) timestamp `full_history_ts_low_`.

This PR adds a data member `full_history_ts_low_` of type `std::string` to `CompactionJob`, and
`full_history_ts_low_` does not change during compaction. `CompactionJob` will pass a pointer to this
data member to the `CompactionIterator` used during compaction.

Also refactored compaction_job_test.cc to re-use some existing code, which is actually the majority of this PR.

Pull Request resolved: https://github.com/facebook/rocksdb/pull/7657

Test Plan: make check

Reviewed By: ltamasi

Differential Revision: D24913803

Pulled By: riversand963

fbshipit-source-id: 11ad5329ddac365667152e7b3b02f84182c0ca8e
      cf9d8e45
    • L
      Clean up CompactionProxy (#7662) · 0dc437d6
      由 Levi Tamasi 提交于 
      Summary:
`CompactionProxy` is currently both a concrete class used for actual `Compaction`s
and a base class that `FakeCompaction` (which is used in `compaction_iterator_test`)
is derived from. This is bad from an OO design standpoint, and also results in
`FakeCompaction` containing an (uninitialized and unused) `Compaction*` member.
The patch fixes this by making `CompactionProxy` a pure interface and introducing
a separate concrete class `RealCompaction` for non-test/non-fake compactions. It
also removes an unused parameter from the virtual method `level`.

Pull Request resolved: https://github.com/facebook/rocksdb/pull/7662

Test Plan: `make check`

Reviewed By: riversand963

Differential Revision: D24907680

Pulled By: ltamasi

fbshipit-source-id: c100ecb1beef4b0ada35e799116c5bda71719ee7
      0dc437d6