cache.txt 11.9 KB
Newer Older
J
Joe Thornber 已提交
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52
Introduction
============

dm-cache is a device mapper target written by Joe Thornber, Heinz
Mauelshagen, and Mike Snitzer.

It aims to improve performance of a block device (eg, a spindle) by
dynamically migrating some of its data to a faster, smaller device
(eg, an SSD).

This device-mapper solution allows us to insert this caching at
different levels of the dm stack, for instance above the data device for
a thin-provisioning pool.  Caching solutions that are integrated more
closely with the virtual memory system should give better performance.

The target reuses the metadata library used in the thin-provisioning
library.

The decision as to what data to migrate and when is left to a plug-in
policy module.  Several of these have been written as we experiment,
and we hope other people will contribute others for specific io
scenarios (eg. a vm image server).

Glossary
========

  Migration -  Movement of the primary copy of a logical block from one
	       device to the other.
  Promotion -  Migration from slow device to fast device.
  Demotion  -  Migration from fast device to slow device.

The origin device always contains a copy of the logical block, which
may be out of date or kept in sync with the copy on the cache device
(depending on policy).

Design
======

Sub-devices
-----------

The target is constructed by passing three devices to it (along with
other parameters detailed later):

1. An origin device - the big, slow one.

2. A cache device - the small, fast one.

3. A small metadata device - records which blocks are in the cache,
   which are dirty, and extra hints for use by the policy object.
   This information could be put on the cache device, but having it
   separate allows the volume manager to configure it differently,
53 54
   e.g. as a mirror for extra robustness.  This metadata device may only
   be used by a single cache device.
J
Joe Thornber 已提交
55 56 57 58 59 60

Fixed block size
----------------

The origin is divided up into blocks of a fixed size.  This block size
is configurable when you first create the cache.  Typically we've been
61 62
using block sizes of 256KB - 1024KB.  The block size must be between 64
(32KB) and 2097152 (1GB) and a multiple of 64 (32KB).
J
Joe Thornber 已提交
63 64 65 66 67 68 69 70

Having a fixed block size simplifies the target a lot.  But it is
something of a compromise.  For instance, a small part of a block may be
getting hit a lot, yet the whole block will be promoted to the cache.
So large block sizes are bad because they waste cache space.  And small
block sizes are bad because they increase the amount of metadata (both
in core and on disk).

J
Joe Thornber 已提交
71 72
Cache operating modes
---------------------
J
Joe Thornber 已提交
73

J
Joe Thornber 已提交
74 75
The cache has three operating modes: writeback, writethrough and
passthrough.
J
Joe Thornber 已提交
76 77 78 79 80 81 82 83 84

If writeback, the default, is selected then a write to a block that is
cached will go only to the cache and the block will be marked dirty in
the metadata.

If writethrough is selected then a write to a cached block will not
complete until it has hit both the origin and cache devices.  Clean
blocks should remain clean.

J
Joe Thornber 已提交
85 86 87 88
If passthrough is selected, useful when the cache contents are not known
to be coherent with the origin device, then all reads are served from
the origin device (all reads miss the cache) and all writes are
forwarded to the origin device; additionally, write hits cause cache
89 90 91 92 93 94 95 96 97
block invalidates.  To enable passthrough mode the cache must be clean.
Passthrough mode allows a cache device to be activated without having to
worry about coherency.  Coherency that exists is maintained, although
the cache will gradually cool as writes take place.  If the coherency of
the cache can later be verified, or established through use of the
"invalidate_cblocks" message, the cache device can be transitioned to
writethrough or writeback mode while still warm.  Otherwise, the cache
contents can be discarded prior to transitioning to the desired
operating mode.
J
Joe Thornber 已提交
98

J
Joe Thornber 已提交
99
A simple cleaner policy is provided, which will clean (write back) all
100 101 102 103 104 105 106 107 108 109
dirty blocks in a cache.  Useful for decommissioning a cache or when
shrinking a cache.  Shrinking the cache's fast device requires all cache
blocks, in the area of the cache being removed, to be clean.  If the
area being removed from the cache still contains dirty blocks the resize
will fail.  Care must be taken to never reduce the volume used for the
cache's fast device until the cache is clean.  This is of particular
importance if writeback mode is used.  Writethrough and passthrough
modes already maintain a clean cache.  Future support to partially clean
the cache, above a specified threshold, will allow for keeping the cache
warm and in writeback mode during resize.
J
Joe Thornber 已提交
110 111 112 113 114 115

Migration throttling
--------------------

Migrating data between the origin and cache device uses bandwidth.
The user can set a throttle to prevent more than a certain amount of
116
migration occurring at any one time.  Currently we're not taking any
J
Joe Thornber 已提交
117 118 119 120 121 122 123 124 125 126
account of normal io traffic going to the devices.  More work needs
doing here to avoid migrating during those peak io moments.

For the time being, a message "migration_threshold <#sectors>"
can be used to set the maximum number of sectors being migrated,
the default being 204800 sectors (or 100MB).

Updating on-disk metadata
-------------------------

127 128 129 130 131
On-disk metadata is committed every time a FLUSH or FUA bio is written.
If no such requests are made then commits will occur every second.  This
means the cache behaves like a physical disk that has a volatile write
cache.  If power is lost you may lose some recent writes.  The metadata
should always be consistent in spite of any crash.
J
Joe Thornber 已提交
132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186

The 'dirty' state for a cache block changes far too frequently for us
to keep updating it on the fly.  So we treat it as a hint.  In normal
operation it will be written when the dm device is suspended.  If the
system crashes all cache blocks will be assumed dirty when restarted.

Per-block policy hints
----------------------

Policy plug-ins can store a chunk of data per cache block.  It's up to
the policy how big this chunk is, but it should be kept small.  Like the
dirty flags this data is lost if there's a crash so a safe fallback
value should always be possible.

For instance, the 'mq' policy, which is currently the default policy,
uses this facility to store the hit count of the cache blocks.  If
there's a crash this information will be lost, which means the cache
may be less efficient until those hit counts are regenerated.

Policy hints affect performance, not correctness.

Policy messaging
----------------

Policies will have different tunables, specific to each one, so we
need a generic way of getting and setting these.  Device-mapper
messages are used.  Refer to cache-policies.txt.

Discard bitset resolution
-------------------------

We can avoid copying data during migration if we know the block has
been discarded.  A prime example of this is when mkfs discards the
whole block device.  We store a bitset tracking the discard state of
blocks.  However, we allow this bitset to have a different block size
from the cache blocks.  This is because we need to track the discard
state for all of the origin device (compare with the dirty bitset
which is just for the smaller cache device).

Target interface
================

Constructor
-----------

 cache <metadata dev> <cache dev> <origin dev> <block size>
       <#feature args> [<feature arg>]*
       <policy> <#policy args> [policy args]*

 metadata dev    : fast device holding the persistent metadata
 cache dev	 : fast device holding cached data blocks
 origin dev	 : slow device holding original data blocks
 block size      : cache unit size in sectors

 #feature args   : number of feature arguments passed
187
 feature args    : writethrough or passthrough (The default is writeback.)
J
Joe Thornber 已提交
188 189 190 191 192 193 194 195 196 197 198 199 200 201 202

 policy          : the replacement policy to use
 #policy args    : an even number of arguments corresponding to
                   key/value pairs passed to the policy
 policy args     : key/value pairs passed to the policy
		   E.g. 'sequential_threshold 1024'
		   See cache-policies.txt for details.

Optional feature arguments are:
   writethrough  : write through caching that prohibits cache block
		   content from being different from origin block content.
		   Without this argument, the default behaviour is to write
		   back cache block contents later for performance reasons,
		   so they may differ from the corresponding origin blocks.

203 204 205 206 207 208 209
   passthrough	 : a degraded mode useful for various cache coherency
		   situations (e.g., rolling back snapshots of
		   underlying storage).	 Reads and writes always go to
		   the origin.	If a write goes to a cached origin
		   block, then the cache block is invalidated.
		   To enable passthrough mode the cache must be clean.

J
Joe Thornber 已提交
210 211 212 213 214 215 216 217 218
A policy called 'default' is always registered.  This is an alias for
the policy we currently think is giving best all round performance.

As the default policy could vary between kernels, if you are relying on
the characteristics of a specific policy, always request it by name.

Status
------

219 220 221 222
<metadata block size> <#used metadata blocks>/<#total metadata blocks>
<cache block size> <#used cache blocks>/<#total cache blocks>
<#read hits> <#read misses> <#write hits> <#write misses>
<#demotions> <#promotions> <#dirty> <#features> <features>*
223
<#core args> <core args>* <policy name> <#policy args> <policy args>*
224 225 226 227 228 229 230 231 232 233

metadata block size	 : Fixed block size for each metadata block in
			     sectors
#used metadata blocks	 : Number of metadata blocks used
#total metadata blocks	 : Total number of metadata blocks
cache block size	 : Configurable block size for the cache device
			     in sectors
#used cache blocks	 : Number of blocks resident in the cache
#total cache blocks	 : Total number of cache blocks
#read hits		 : Number of times a READ bio has been mapped
J
Joe Thornber 已提交
234
			     to the cache
235
#read misses		 : Number of times a READ bio has been mapped
J
Joe Thornber 已提交
236
			     to the origin
237
#write hits		 : Number of times a WRITE bio has been mapped
J
Joe Thornber 已提交
238
			     to the cache
239
#write misses		 : Number of times a WRITE bio has been
J
Joe Thornber 已提交
240
			     mapped to the origin
241
#demotions		 : Number of times a block has been removed
J
Joe Thornber 已提交
242
			     from the cache
243
#promotions		 : Number of times a block has been moved to
J
Joe Thornber 已提交
244
			     the cache
245
#dirty			 : Number of blocks in the cache that differ
J
Joe Thornber 已提交
246
			     from the origin
247 248 249 250
#feature args		 : Number of feature args to follow
feature args		 : 'writethrough' (optional)
#core args		 : Number of core arguments (must be even)
core args		 : Key/value pairs for tuning the core
J
Joe Thornber 已提交
251
			     e.g. migration_threshold
252
policy name		 : Name of the policy
253 254 255
#policy args		 : Number of policy arguments to follow (must be even)
policy args		 : Key/value pairs
			     e.g. sequential_threshold
J
Joe Thornber 已提交
256 257 258 259 260 261 262 263 264 265 266 267 268 269 270

Messages
--------

Policies will have different tunables, specific to each one, so we
need a generic way of getting and setting these.  Device-mapper
messages are used.  (A sysfs interface would also be possible.)

The message format is:

   <key> <value>

E.g.
   dmsetup message my_cache 0 sequential_threshold 1024

271 272 273

Invalidation is removing an entry from the cache without writing it
back.  Cache blocks can be invalidated via the invalidate_cblocks
274
message, which takes an arbitrary number of cblock ranges.  Each cblock
275 276 277 278 279 280
range's end value is "one past the end", meaning 5-10 expresses a range
of values from 5 to 9.  Each cblock must be expressed as a decimal
value, in the future a variant message that takes cblock ranges
expressed in hexidecimal may be needed to better support efficient
invalidation of larger caches.  The cache must be in passthrough mode
when invalidate_cblocks is used.
281 282 283 284 285 286

   invalidate_cblocks [<cblock>|<cblock begin>-<cblock end>]*

E.g.
   dmsetup message my_cache 0 invalidate_cblocks 2345 3456-4567 5678-6789

J
Joe Thornber 已提交
287 288 289 290 291
Examples
========

The test suite can be found here:

292
https://github.com/jthornber/device-mapper-test-suite
J
Joe Thornber 已提交
293 294 295 296 297 298

dmsetup create my_cache --table '0 41943040 cache /dev/mapper/metadata \
	/dev/mapper/ssd /dev/mapper/origin 512 1 writeback default 0'
dmsetup create my_cache --table '0 41943040 cache /dev/mapper/metadata \
	/dev/mapper/ssd /dev/mapper/origin 1024 1 writeback \
	mq 4 sequential_threshold 1024 random_threshold 8'