@@ -34,7 +34,7 @@ So Redis offers more features:<br/><br/><ul><li> Keys can store different data t
<ul><li> We wrote a <ahref="http://retwis.antirez.com"target="_blank">simple Twitter Clone</a> using just Redis as database. Download the source code from the download section and imagine to write it with a plain key-value DB without support for lists and sets... it's <b>much</b> harder.</li></ul>
<ul><li> Multiple DBs. Using the SELECT command the client can select different datasets. This is useful because Redis provides a MOVE atomic primitive that moves a key form a DB to another one, if the target DB already contains such a key it returns an error: this basically means a way to perform locking in distributed processing.</li></ul>
<ul><li><b>So what is Redis really about?</b> The User interface with the programmer. Redis aims to export to the programmer the right tools to model a wide range of problems. <b>Sets, Lists with O(1) push operation, lrange and ltrim, server-side fast intersection between sets, are primitives that allow to model complex problems with a key value database</b>.</li></ul>
<h1><aname="Isn't this key-value thing just hype?">Isn't this key-value thing just hype?</a></h1>I imagine key-value DBs, in the short term future, to be used like you use memory in a program, with lists, hashes, and so on. With Redis it's like this, but this special kind of memory containing your data structures is shared, atomic, persistent.<br/><br/>When we write code it is obvious, when we take data in memory, to use the most sensible data structure for the work, right? Incredibly when data is put inside a relational DB this is no longer true, and we create an absurd data model even if our need is to put data and get this data back in the same order we put it inside (an ORDER BY is required when the data should be already sorted. Strange, dont' you think?).<br/><br/>Key-value DBs bring this back at home, to create sensible data models and use the right data structures for the problem we are trying to solve.<h1><aname="Can I backup a Redis DB while the server is working?">Can I backup a Redis DB while the server is working?</a></h1>Yes you can. When Redis saves the DB it actually creates a temp file, then rename(2) that temp file name to the destination file name. So even while the server is working it is safe to save the database file just with the <i>cp</i> unix command. Note that you can use master-slave replication in order to have redundancy of data, but if all you need is backups, cp or scp will do the work pretty well.<h1><aname="What's the Redis memory footprint?">What's the Redis memory footprint?</a></h1>Worst case scenario: 1 Million keys with the key being the natural numbers from 0 to 999999 and the string "Hello World" as value use 100MB on my Intel macbook (32bit). Note that the same data stored linearly in an unique string takes something like 16MB, this is the norm because with small keys and values there is a lot of overhead. Memcached will perform similarly.<br/><br/>With large keys/values the ratio is much better of course.<br/><br/>64 bit systems will use much more memory than 32 bit systems to store the same keys, especially if the keys and values are small, this is because pointers takes 8 bytes in 64 bit systems. But of course the advantage is that you can have a lot of memory in 64 bit systems, so to run large Redis servers a 64 bit system is more or less required.<h1><aname="I like Redis high level operations and features, but I don't like it takes everything in memory and I can't have a dataset larger the memory. Plans to change this?">I like Redis high level operations and features, but I don't like it takes everything in memory and I can't have a dataset larger the memory. Plans to change this?</a></h1>The whole key-value hype started for a reason: performances. Redis takes the whole dataset in memory and writes asynchronously on disk in order to be very fast, you have the best of both worlds: hyper-speed and persistence of data, but the price to pay is exactly this, that the dataset must fit on your computers RAM.<br/><br/>If the data is larger then memory, and this data is stored on disk, what happens is that the bottleneck of the disk I/O speed will start to ruin the performances. Maybe not in benchmarks, but once you have real load with distributed key accesses the data must come from disk, and the disk is damn slow. Not only, but Redis supports higher level data structures than the plain values. To implement this things on disk is even slower.<br/><br/>Redis will always continue to hold the whole dataset in memory because this days scalability requires to use RAM as storage media, and RAM is getting cheaper and cheaper. Today it is common for an entry level server to have 16 GB of RAM! And in the 64-bit era there are no longer limits to the amount of RAM you can have in theory.<h1><aname="Ok but I absolutely need to have a DB larger than memory, still I need the Redis features">Ok but I absolutely need to have a DB larger than memory, still I need the Redis features</a></h1>One possible solution is to use both MySQL and Redis at the same time, basically take the state on Redis, and all the things that get accessed very frequently: user auth tokens, Redis Lists with chronologically ordered IDs of the last N-comments, N-posts, and so on. Then use MySQL as a simple storage engine for larger data, that is just create a table with an auto-incrementing ID as primary key and a large BLOB field as data field. Access MySQL data only by primary key (the ID). The application will run the high traffic queries against Redis but when there is to take the big data will ask MySQL for specific resources IDs.<h1><aname="I have an empty Redis server but INFO and logs are reporting megabytes of memory in use!">I have an empty Redis server but INFO and logs are reporting megabytes of memory in use!</a></h1>This may happen and it's prefectly ok. Redis objects are small C structures allocated and freed a lot of times. This costs a lot of CPU so instead of being freed, released objects are taken into a free list and reused when needed. This memory is taken exactly by this free objects ready to be reused.<h1><aname="What happens if Redis runs out of memory?">What happens if Redis runs out of memory?</a></h1>With modern operating systems malloc() returning NULL is not common, usually the server will start swapping and Redis performances will be disastrous so you'll know it's time to use more Redis servers or get more RAM.<br/><br/>However it is planned to add a configuration directive to tell Redis to stop accepting queries but instead to SAVE the latest data and quit if it is using more than a given amount of memory. Also the new INFO command (work in progress in this days) will report the amount of memory Redis is using so you can write scripts that monitor your Redis servers checking for critical conditions.<br/><br/>Update: redis SVN is able to know how much memory it is using and report it via the <ahref="InfoCommand.html">INFO</a> command.<h1><aname="What Redis means actually?">What Redis means actually?</a></h1>Redis means two things:
<h1><aname="Isn't this key-value thing just hype?">Isn't this key-value thing just hype?</a></h1>I imagine key-value DBs, in the short term future, to be used like you use memory in a program, with lists, hashes, and so on. With Redis it's like this, but this special kind of memory containing your data structures is shared, atomic, persistent.<br/><br/>When we write code it is obvious, when we take data in memory, to use the most sensible data structure for the work, right? Incredibly when data is put inside a relational DB this is no longer true, and we create an absurd data model even if our need is to put data and get this data back in the same order we put it inside (an ORDER BY is required when the data should be already sorted. Strange, dont' you think?).<br/><br/>Key-value DBs bring this back at home, to create sensible data models and use the right data structures for the problem we are trying to solve.<h1><aname="Can I backup a Redis DB while the server is working?">Can I backup a Redis DB while the server is working?</a></h1>Yes you can. When Redis saves the DB it actually creates a temp file, then rename(2) that temp file name to the destination file name. So even while the server is working it is safe to save the database file just with the <i>cp</i> unix command. Note that you can use master-slave replication in order to have redundancy of data, but if all you need is backups, cp or scp will do the work pretty well.<h1><aname="What's the Redis memory footprint?">What's the Redis memory footprint?</a></h1>Worst case scenario: 1 Million keys with the key being the natural numbers from 0 to 999999 and the string "Hello World" as value use 100MB on my Intel macbook (32bit). Note that the same data stored linearly in an unique string takes something like 16MB, this is the norm because with small keys and values there is a lot of overhead. Memcached will perform similarly.<br/><br/>With large keys/values the ratio is much better of course.<br/><br/>64 bit systems will use much more memory than 32 bit systems to store the same keys, especially if the keys and values are small, this is because pointers takes 8 bytes in 64 bit systems. But of course the advantage is that you can have a lot of memory in 64 bit systems, so to run large Redis servers a 64 bit system is more or less required.<h1><aname="I like Redis high level operations and features, but I don't like it takes everything in memory and I can't have a dataset larger the memory. Plans to change this?">I like Redis high level operations and features, but I don't like it takes everything in memory and I can't have a dataset larger the memory. Plans to change this?</a></h1>The whole key-value hype started for a reason: performances. Redis takes the whole dataset in memory and writes asynchronously on disk in order to be very fast, you have the best of both worlds: hyper-speed and persistence of data, but the price to pay is exactly this, that the dataset must fit on your computers RAM.<br/><br/>If the data is larger then memory, and this data is stored on disk, what happens is that the bottleneck of the disk I/O speed will start to ruin the performances. Maybe not in benchmarks, but once you have real load from multiple clients with distributed key accesses the data must come from disk, and the disk is damn slow. Not only, but Redis supports higher level data structures than the plain values. To implement this things on disk is even slower.<br/><br/>Redis will always continue to hold the whole dataset in memory because this days scalability requires to use RAM as storage media, and RAM is getting cheaper and cheaper. Today it is common for an entry level server to have 16 GB of RAM! And in the 64-bit era there are no longer limits to the amount of RAM you can have in theory.<h1><aname="Ok but I absolutely need to have a DB larger than memory, still I need the Redis features">Ok but I absolutely need to have a DB larger than memory, still I need the Redis features</a></h1>You may try to load a dataset larger than your memory in Redis and see what happens, basically if you are using a modern Operating System, and you have a lot of data in the DB that is rarely accessed, the OS's virtual memory implementation will try to swap rarely used pages of memory on the disk, to only recall this pages when they are needed. If you have many large values rarely used this will work. If your DB is big because you have tons of little values accessed at random without a specific pattern this will not work (at low level a page is usually 4096 bytes, and you can have different keys/values stored at a single page. The OS can't swap this page on disk if there are even few keys used frequently).<br/><br/>Another possible solution is to use both MySQL and Redis at the same time, basically take the state on Redis, and all the things that get accessed very frequently: user auth tokens, Redis Lists with chronologically ordered IDs of the last N-comments, N-posts, and so on. Then use MySQL as a simple storage engine for larger data, that is just create a table with an auto-incrementing ID as primary key and a large BLOB field as data field. Access MySQL data only by primary key (the ID). The application will run the high traffic queries against Redis but when there is to take the big data will ask MySQL for specific resources IDs.<h1><aname="I have an empty Redis server but INFO and logs are reporting megabytes of memory in use!">I have an empty Redis server but INFO and logs are reporting megabytes of memory in use!</a></h1>This may happen and it's prefectly ok. Redis objects are small C structures allocated and freed a lot of times. This costs a lot of CPU so instead of being freed, released objects are taken into a free list and reused when needed. This memory is taken exactly by this free objects ready to be reused.<h1><aname="What happens if Redis runs out of memory?">What happens if Redis runs out of memory?</a></h1>With modern operating systems malloc() returning NULL is not common, usually the server will start swapping and Redis performances will be disastrous so you'll know it's time to use more Redis servers or get more RAM.<br/><br/>However it is planned to add a configuration directive to tell Redis to stop accepting queries but instead to SAVE the latest data and quit if it is using more than a given amount of memory. Also the new INFO command (work in progress in this days) will report the amount of memory Redis is using so you can write scripts that monitor your Redis servers checking for critical conditions.<br/><br/>Update: redis SVN is able to know how much memory it is using and report it via the <ahref="InfoCommand.html">INFO</a> command.<h1><aname="What Redis means actually?">What Redis means actually?</a></h1>Redis means two things:
<ul><li> it's a joke on the word Redistribute (instead to use just a Relational DB redistribute your workload among Redis servers)</li><li> it means REmote DIctionary Server</li></ul>
<h1><aname="Why did you started the Redis project?">Why did you started the Redis project?</a></h1>In order to scale <ahref="http://lloogg.com"target="_blank">LLOOGG</a>. But after I got the basic server working I liked the idea to share the work with other guys, and Redis was turned into an open source project.
<!-- This is a (PRE) block. Make sure it's left aligned or your toc title will be off. -->
<b>ProtocolSpecification: Contents</b><br> <ahref="#Protocol Specification">Protocol Specification</a><br> <ahref="#Networking layer">Networking layer</a><br> <ahref="#Simple INLINE commands">Simple INLINE commands</a><br> <ahref="#Bulk commands">Bulk commands</a><br> <ahref="#Bulk replies">Bulk replies</a><br> <ahref="#Bulk reply error reporting">Bulk reply error reporting</a><br> <ahref="#Multi-Bulk replies">Multi-Bulk replies</a><br> <ahref="#Nil elements in Multi-Bulk replies">Nil elements in Multi-Bulk replies</a><br> <ahref="#Multi-Bulk replies errors">Multi-Bulk replies errors</a><br> <ahref="#Status code reply">Status code reply</a><br> <ahref="#Integer reply">Integer reply</a><br> <ahref="#Single line reply">Single line reply</a><br> <ahref="#Multiple commands and pipelining">Multiple commands and pipelining</a>
<b>ProtocolSpecification: Contents</b><br> <ahref="#Protocol Specification">Protocol Specification</a><br> <ahref="#Networking layer">Networking layer</a><br> <ahref="#Simple INLINE commands">Simple INLINE commands</a><br> <ahref="#Bulk commands">Bulk commands</a><br> <ahref="#Bulk replies">Bulk replies</a><br> <ahref="#Multi-Bulk replies">Multi-Bulk replies</a><br> <ahref="#Nil elements in Multi-Bulk replies">Nil elements in Multi-Bulk replies</a><br> <ahref="#Single line reply">Single line reply</a><br> <ahref="#Integer reply">Integer reply</a><br> <ahref="#Multiple commands and pipelining">Multiple commands and pipelining</a>
server/client chat (the server chat starts with S:, the client chat with C:)<br/><br/><preclass="codeblock python"name="code">
C: PING
S: +PONG
</pre>An inline command is a CRLF-terminated string sent to the client. The server
usually replies to inline commands with a single line that can be a number
or a return code.<br/><br/>When the server replies with a status code (that is a one line reply just indicating if the operation succeeded or not), if the first character of the
reply is a "+" then the command succeeded, if it is a "-" then the following
part of the string is an error.<br/><br/>The following is another example of an INLINE command returning an integer:<br/><br/><preclass="codeblock python python"name="code">
</pre>An inline command is a CRLF-terminated string sent to the client. The server can reply to commands in different ways:
<ul><li> With an error message (the first byte of the reply will be "-")</li><li> With a single line reply (the first byte of the reply will be "+)</li><li> With bulk data (the first byte of the reply will be "$")</li><li> With multi-bulk data, a list of values (the first byte of the reply will be "<codename="code"class="python">*</code>")</li><li> With an integer number (the first byte of the reply will be ":")</li></ul>
The following is another example of an INLINE command returning an integer:<br/><br/><preclass="codeblock python python"name="code">
C: EXISTS somekey
S: 0
</pre>Since 'somekey' does not exist the server returned '0'.<br/><br/>Note that the EXISTS command takes one argument. Arguments are separated
S: :0
</pre>Since 'somekey' does not exist the server returned ':0'.<br/><br/>Note that the EXISTS command takes one argument. Arguments are separated
simply by spaces.<h2><aname="Bulk commands">Bulk commands</a></h2>A bulk command is exactly like an inline command, but the last argument
of the command must be a stream of bytes in order to send data to the server.
the "SET" command is a bulk command, see the following example:<br/><br/><preclass="codeblock python python python"name="code">
...
...
@@ -58,82 +56,60 @@ sent by the client in the above sample:<br/><br/><blockquote>"SET mykey 6\r
<h2><aname="Bulk replies">Bulk replies</a></h2>The server may reply to an inline or bulk command with a bulk reply. See
the following example:<br/><br/><preclass="codeblock python python python python"name="code">
C: GET mykey
S: 6
S: $6
S: foobar
</pre>A bulk reply is very similar to the last argument of a bulk command. The
server sends as the first line the number of bytes of the actual reply
followed by CRLF, then the bytes are sent followed by additional two bytes
for the final CRLF. The exact sequence sent by the server is:<br/><br/><blockquote>"6\r\nfoobar\r\n"</blockquote>
server sends as the first line a "$" byte followed by the number of bytes
of the actual reply followed by CRLF, then the bytes are sent followed by
additional two bytes for the final CRLF. The exact sequence sent by the
server is:<br/><br/><blockquote>"$6\r\nfoobar\r\n"</blockquote>
If the requested value does not exist the bulk reply will use the special
value 'nil' instead to send the line containing the number of bytes to read.
This is an example:<br/><br/><preclass="codeblock python python python python python"name="code">
value -1 as data length, example:<br/><br/><preclass="codeblock python python python python python"name="code">
C: GET nonexistingkey
S: nil
</pre>The client library API should not return an empty string, but a nil object.
S: $-1
</pre>The client library API should not return an empty string, but a nil object, when the requested object does not exist.
For example a Ruby library should return 'nil' while a C library should return
NULL.<h2><aname="Bulk reply error reporting">Bulk reply error reporting</a></h2>Bulk replies can signal errors, for example trying to use GET against a list
value is not permitted. Bulk replies use a negative bytes count in order to
signal an error. An error string of ABS(bytes_count) bytes will follow. See
the following example:<br/><br/><preclass="codeblock python python python python python python"name="code">
S: GET alistkey
S: -38
S: -ERR Requested element is not a string
</pre>-38 means: sorry your operation resulted in an error, but a 38 bytes string
that explains this error will follow. Client APIs should abort on this kind
of errors, for example a PHP client should call the die() function.<br/><br/>The following commands reply with a bulk reply: GET, KEYS, LINDEX, LPOP, RPOP<h2><aname="Multi-Bulk replies">Multi-Bulk replies</a></h2>Commands similar to LRANGE needs to return multiple values (every element
NULL, and so forth.<h2><aname="Multi-Bulk replies">Multi-Bulk replies</a></h2>Commands similar to LRANGE needs to return multiple values (every element
of the list is a value, and LRANGE needs to return more than a single element). This is accomplished using multiple bulk writes,
prefixed by an initial line indicating how many bulk writes will follow.
The first byte of a multi bulk reply is always <codename="code"class="python">*</code>. Example:<br/><br/><preclass="codeblock python python python python python python"name="code">
C: LRANGE mylist 0 3
S: 4
S: 3
S: *4
S: $3
S: foo
S: 3
S: $3
S: bar
S: 5
S: $5
S: Hello
S: 5
S: $5
S: World
</pre>The first line the server sent is "4\r\n" in order to specify that four bulk
</pre>The first line the server sent is "<b>4\r\n" in order to specify that four bulk
write will follow. Then every bulk write is transmitted.<br/><br/>If the specified key does not exist instead of the number of elements in the
list, the special value 'nil' is sent. Example:<br/><br/><preclass="codeblock python python python python python python python python"name="code">
list, the special value -1 is sent as count. Example:<br/><br/><preclass="codeblock python python python python python python python"name="code">
C: LRANGE nokey 0 1
S: nil
S: *-1
</pre>A client library API SHOULD return a nil object and not an empty list when this
happens. This makes possible to distinguish between empty list and non existing ones.<h2><aname="Nil elements in Multi-Bulk replies">Nil elements in Multi-Bulk replies</a></h2>Single elements of a multi bulk reply may have -1 length, in order to signal that this elements are missing and not empty strings. This can happen with the SORT command when used with the GET <i>pattern</i> option when the specified key is missing. Example of a multi bulk reply containing an empty element:<br/><br/><preclass="codeblock python python python python python python python python python"name="code">
S: 3
S: 3
happens. This makes possible to distinguish between empty list and non existing ones.<h2><aname="Nil elements in Multi-Bulk replies">Nil elements in Multi-Bulk replies</a></h2>Single elements of a multi bulk reply may have -1 length, in order to signal that this elements are missing and not empty strings. This can happen with the SORT command when used with the GET <i>pattern</i> option when the specified key is missing. Example of a multi bulk reply containing an empty element:<br/><br/><preclass="codeblock python python python python python python python python"name="code">
S: *3
S: $3
S: foo
S: -1
S: 3
S: $-1
S: $3
S: bar
</pre>The second element is nul. The client library should return something like this:<br/><br/><preclass="codeblock python python python python python python python python python python"name="code">
</pre>The second element is nul. The client library should return something like this:<br/><br/><preclass="codeblock python python python python python python python python python"name="code">
["foo",nil,"bar"]
</pre><h2><aname="Multi-Bulk replies errors">Multi-Bulk replies errors</a></h2>Like bulk reply errors Multi-bulk reply errors are reported using a negative
</pre>The following commands reply with a multi-bulk reply: LRANGE, LINTER<br/><br/>Check the Bulk replies errors section for more information.<h2><aname="Status code reply">Status code reply</a></h2>As already seen a status code reply is in the form of a single line string
terminated by "\r\n". For example:<br/><br/><preclass="codeblock python python python python python python python python python python python python"name="code">
</pre><h2><aname="Single line reply">Single line reply</a></h2>As already seen a single line reply is in the form of a single line string
starting with "+" terminated by "\r\n". For example:<br/><br/><preclass="codeblock python python python python python python python python python python"name="code">
</pre>are two examples of status code replies. The first character of a status code reply is always "+" or "-".<br/><br/>The following commands reply with a status code reply:
PING, SET, SELECT, SAVE, BGSAVE, SHUTDOWN, RENAME, LPUSH, RPUSH, LSET, LTRIM<h2><aname="Integer reply">Integer reply</a></h2>This type of reply is just a CRLF terminated string representing an integer. For example "0\r\n", or "1000\r\n" are integer replies.<br/><br/>With commands like INCR or LASTSAVE using the integer reply to actually return a value there is no special meaning for the returned integer. It is just an incremental number for INCR, a UNIX time for LASTSAVE and so on.<br/><br/>Some commands like EXISTS will return 1 for true and 0 for false.<br/><br/>Other commands like SADD, SREM and SETNX will return 1 if the operation was actually done, 0 otherwise, and <b>a negative value</b> if the operation is invalid (for example SADD against a non-set value), accordingly to this table:
-2 operation against the a key holding a value of the wrong type
-3 source and destiantion objects/dbs are the same
-4 argument out of range
</pre>
In all this cases it is mandatory that the client raises an error instead to pass the negative value to the caller. Please check the commands documentation for the exact behaviour.<br/><br/>The following commands will reply with an integer reply: SETNX, DEL, EXISTS, INCR, INCRBY, DECR, DECRBY, DBSIZE, LASTSAVE, RENAMENX, MOVE, LLEN, SADD, SREM, SISMEMBER, SCARD<br/><br/>The commands that will never return a negative integer (commands that can't fail) are: INCR, DECR, INCRBY, DECRBY, LASTSAVE, EXISTS, SETNX, DEL, DBSIZE.<h2><aname="Single line reply">Single line reply</a></h2>This replies are just single line strings terminated by CRLF. Only two commands reply in this way currently, RANDOMKEY and TYPE.<h2><aname="Multiple commands and pipelining">Multiple commands and pipelining</a></h2>A client can use the same connection in order to issue multiple commands.
</pre>The client library should return everything after the "+", that is, the string "OK" in the example.<br/><br/>The following commands reply with a status code reply:
PING, SET, SELECT, SAVE, BGSAVE, SHUTDOWN, RENAME, LPUSH, RPUSH, LSET, LTRIM<h2><aname="Integer reply">Integer reply</a></h2>This type of reply is just a CRLF terminated string representing an integer, prefixed by a ":" byte. For example ":0\r\n", or ":1000\r\n" are integer replies.<br/><br/>With commands like INCR or LASTSAVE using the integer reply to actually return a value there is no special meaning for the returned integer. It is just an incremental number for INCR, a UNIX time for LASTSAVE and so on.<br/><br/>Some commands like EXISTS will return 1 for true and 0 for false.<br/><br/>Other commands like SADD, SREM and SETNX will return 1 if the operation was actually done, 0 otherwise.<br/><br/>The following commands will reply with an integer reply: SETNX, DEL, EXISTS, INCR, INCRBY, DECR, DECRBY, DBSIZE, LASTSAVE, RENAMENX, MOVE, LLEN, SADD, SREM, SISMEMBER, SCARD<h2><aname="Multiple commands and pipelining">Multiple commands and pipelining</a></h2>A client can use the same connection in order to issue multiple commands.
Pipelining is supported so multiple commands can be sent with a single
write operation by the client, it is not needed to read the server reply
in order to issue the next command. All the replies can be read at the end.<br/><br/>Usually Redis server and client will have a very fast link so this is not
very important to support this feature in a client implementation, still
if an application needs to issue a very large number of commands in short
time to use pipelining can be much faster.<br/><br/>
