REmote DIctionary Server.
http://redis.io/
open source, advanced key-value store, data
structure server
binzhang@ebay.com
http://www.yumianfeilong.com
Hello Redis
Install
$ wget http://redis.googlecode.com/files/redis-2.4.8.tar.gz
$ tar xzf redis-2.4.8.tar.gz
$ cd redis-2.4.8
$ make
Start Redis server & make connection
[binzhang@phxrueidb04 redis-2.4.8]$ src/redis-server redis.conf
[16849] 04 Mar 02:03:59 * Server started, Redis version 2.4.8
[16849] 04 Mar 02:03:59 * The server is now ready to accept connections on
port 6379
Connect server
$ src/redis-cli
redis > set player:666:name binzhang
OK
redis > get player:666:name
"binzhang"
2
Learn More : data structure server
3
Agenda
Redis Manifesto 宣言
Data structures : strings, hashes, lists, sets and
sorted sets.
Leveraging Redis
Redis Admin and maintenance
The architecture of REDIS
Cases
4
Redis Manifesto
http://antirez.com/post/redis-manifesto.html
1. Redis is a DSL (Domain Specific Language) that manipulates
abstract data types and implemented as a TCP daemon. keys are
binary-safe strings and values are different kinds of abstract data
types.
2. Redis has persistence option but Memory storage is #1.
3. The Redis API is a direct consequence of fundamental data
structures.
4. Code is like a poem.
5. We believe designing systems is a fight against complexity. Most of
the time the best way to fight complexity is by not creating it at all.
6. Redis API has two levels: 1) a subset of the API fits naturally into a
distributed version of Redis and 2) a more complex API that
supports multi-key operations.
7. We optimize for joy. When there is no longer joy in writing code,
the best thing to do is stop.
5
Redis Manifesto
: where is Redis?
1. Redis is extremely fast, making it perfectly suited for applications that are writeheavy, data that changes often, and data that naturally fits one of Redis’s data
structures (for instance, analytics data).
2. A scenario where you probably shouldn’t use Redis is if you have a very large
dataset of which only a small part is “hot” (accessed often) or a case where
your dataset doesn’t fit in memory.
6
Who is using Redis
Taobao
Sina weibo
7
Agenda
Redis Manifesto 宣言
Simple/fast/
 Data structure : strings, hashes, lists, sets and sorted sets.
Leveraging Redis
The architecture of REDIS
Redis Admin and maintenance
Cases
8
Data structure: Key-Value Data
Store
Keys are strings which identify pieces of data
(values)
Values are arbitrary byte arrays that Redis
doesn't care about
Redis is implemented as five specialized data
structures
Strings, hash, list, set, sort set
Pub/Sub
Querying with Redis ;
the above make Redis fast and easy to use, but not
9
Data structure : KEY value
Before we dive into the specific data types, it is important to look at a few things you should keep in
mind when designing the key structure that holds your data.
1.a key can contain any characters, you can use separators to define a namespace with a semantic
value for your business. An example might be using cache:project:319:tasks, where the colon acts as a
namespace separator.
2.When defining your keys, try to limit them to a reasonable size. Retrieving a key from storage
requires comparison operations, so keeping keys as small as possible is a good idea. Additionally,
smaller keys are more effective in terms of memory usage.
3.Even though keys shouldn’t be exceptionally large, there are no big performance improvements for
extremely small keys. This means you should design your keys in such a way that combines
readability (to help you) and regular key sizes (to help Redis).
10
Data structure : KEY value
Redis support different kind of values
1.Binary-safe strings.
2.Lists of binary-safe strings.
3.Hash map of strings
4.Sets of binary-safe strings, that are collection
of unique unsorted elements.
5.Sorted sets, similar to Sets but where every
element is associated to a floating number
score. The elements are taken sorted by score.
11
Data structure : strings,
hashes, lists, sets , sorted sets.
1. Strings: the simplest and most basic data type (max 512M in length)
2. Redis Strings are binary safe, this means that a Redis string can contain any kind of data, for
instance a JPEG image or a serialized Ruby object.
– redis > users:leto "{name: leto, planet: dune, likes: [spice]}"
– OK
The C structure sdshdr
– redis > x users:leto "{name: leto, planet: dune,
likes: in
[spice]}"
declared
sds.h represents a
– (integer) 0
redis > strlen users:leto
(integer) 42
redis > getrange users:leto 27 40
"likes: [spice]"
redis > append users:leto " OVER 9000!!"
(integer) 54
redis > get users:leto
"{name: leto, planet: dune, likes: [spice]} OVER 9000!!"
Redis string:
struct sdshdr { long len;
long free;
char buf[];
};
12
•
Data structure : strings, hashes,
lists, sets , sorted sets.
if we store a counter key, we can use commands such as INCR (or INCRBY) and DECR (or
DECRBY) to increment or decrement its contained value.
– To store page visit data, we could have a key
“visits:pageid:totals”
redis > SET visits:2:totals 1367894
OK
redis > get visits:2:totals
"1367894"
redis > INCR visits:635:totals
(integer) 1
redis > INCR visits:2:totals
(integer) 1367895
redis > get visits:2:totals
13
Data structure :
•
•
strings
, hashes
, lists, sets , sorted sets.
Much like traditional hashtables, hashes in Redis store several fields and their values inside a
specific key. so they are the perfect data type to represent objects (eg: A User with a number
of fields like name, surname, age, and so forth):
Example: Designing a key namespace to store our users.
redis> hset users:jdoe name "John Doe"
(integer) 1
redis> hmset users:jdoe email jdoe@test.com phone "+1555313940"
OK
redis> hincrby users:jdoe visits 1
(integer) 1
redis > hget users:jdoe phone
"+1555313940"
•
•
A hash with a few fields (where few means up to one hundred or so) is stored in a way that takes
very little space, so you can store millions of objects in a small Redis instance.
Every hash can store up to 232 - 1 field-value pairs (more than 4 billion).
14
•
•
•
•
•
•
•
•
•
Data structure : strings, hashes,
lists, sets , sorted sets.
HDEL key field [field ...] Delete one or more hash fields
HGETALL key Get all the fields and values in a hash
HINCRBY key field increment Increment the integer value of a hash field by the given number
HKEYS key Get all the fields in a hash
HLEN key Get the number of fields in a hash
HMGET key field [field ...] Get the values of all the given hash fields
HMSET key field value [field value ...] Set multiple hash fields to multiple values
HSETNX key field value Set the value of a hash field, only if the field does not exist
HVALS key Get all the values in a hash
redis > hgetall users:jdoe
1) "name"
2) "John Doe"
3) "email"
4) "jdoe@test.com"
5) "phone"
6) "+1555313940"
7) "visits"
8) "1"
15
Data structure :
•
strings
,
hashes,
lists
, sets , sorted sets.
Redis Lists are simply lists of strings, sorted by insertion order. It is possible to add elements
to a Redis List pushing new elements on the head (on the left) or on the tail (on the right) of
the list.
LPUSH mylist a # now the list is "a"
LPUSH mylist b # now the list is "b","a"
RPUSH mylist c # now the list is "b","a","c" (RPUSH was
used this time)
•
•
•
support for constant time insertion and deletion of elements near the head and tail, even
with many millions of inserted items. Accessing elements is very fast near the extremes of
the list but is slow if you try accessing the middle of a very big list, as it is an O(N) operation.
You might want to use lists in order to implement structures such as queues
The max length of a list is 232 - 1 elements (4294967295, more than 4 billion of elements per
list).
16
Data structure : strings, hashes,
lists, sets , sorted sets.
•
Model a timeline in a social network, using LPUSH in order to add new elements in the user
time line, and using LRANGE in order to retrieve a few of recently inserted items.
–
•
You can use LPUSH together with LTRIM (O(N))to create a list that never exceeds a given
number of elements, but just remembers the latest N elements.
–
•
Lrange to do paging
Capped Collections in MongoDB.
Lists can be used as a message passing primitive,
–
–
–
–
–
–
–
–
BLPOP key [key ...] timeout Remove and get the first element in a list, or block until one is available
BRPOP key [key ...] timeout Remove and get the last element in a list, or block until one is available
LINDEX key index Get an element from a list by its index
LLEN key Get the length of a list
LPOP key Remove and get the first element in a list
RPOP key Remove and get the last element in a list
LRANGE key start stop Get a range of elements from a list
LTRIM key start stop Trim a list to the specified range
17
Data structure :
•
•
•
•
strings
,
hashes, lists,
sets ,
sorted sets.
Sets are an unordered collection of Strings. It is possible to add, remove, and test for
existence of members in O(1) (constant time regardless of the number of elements contained
inside the Set).
Elements in a given set can have no duplicates. this means that adding a member does not
require a check if exists then add operation.
Sets are a natural fit for circles, because sets represent collections of data, and have native
functionality to do interesting things like intersections and unions.
The max number of members in a set is 232 - 1 (4294967295, more than 4 billion of members
per set).
–
–
–
You can track unique things using Redis Sets. Want to know all the unique IP addresses visiting a
given blog post? Simply use SADD every time you process a page view.
Redis Sets are good to represent relations.
You can use Sets to extract elements at random using the SPOP or SRANDMEMBER commands.
18
Data structure :
strings
,
hashes, lists,
sets ,
sorted sets.
 We want to store several circles for each of our users, so it makes sense for our key to
include a bit about the user and a bit about the actual circle. (circle:jdoe:family etc)
redis>
sadd
circle:jdoe:family
users:anna
redis>
sinter
circle:jdoe:family
circle:jdoe:soccer
redis> sadd circle:jdoe:family users:richard
1) "users:mike"
redis> sadd circle:jdoe:family users:mike
(integer)
redis> 1sunion circle:jdoe:family circle:jdoe:soccer
redis> sadd circle:jdoe:soccer users:mike
1) "users:anna"
redis> sadd circle:jdoe:soccer users:adam
2) "users:mike"
redis>
sadd circle:jdoe:soccer users:toby
redis> sadd circle:jdoe:soccer users:apollo
3) "users:apollo"
(integer) 1
4) "users:adam"
redis>
smembers circle:jdoe:family
5) "users:richard"
1) "users:richard"
"users:toby"
2)6)"users:mike"
3) "users:anna"
redis> hgetall users:mike
(...)
19
Data structure : strings, hashes,
lists, sets , sorted sets.
•
•
•
•
•
•
•
•
•
•
•
•
•
SADD key member [member ...] Add one or more members to a set
SCARD key Get the number of members in a set
SDIFF key [key ...] Subtract multiple sets
SDIFFSTORE destination key [key ...] Subtract multiple sets and store the resulting set in a
key
SINTER key [key ...]
Intersect multiple sets
SISMEMBER key member
Determine if a given value is a member of a set
SMEMBERS key
Get all the members in a set
SMOVE source destination member Move a member from one set to another
SPOP key Remove and return a random member from a set
SRANDMEMBER key Get a random member from a set
SREM key member [member ...] Remove one or more members from a set
SUNION key [key ...] Add multiple sets
SUNIONSTORE destination key [key ...] Add multiple sets and store the resulting set in a key
20
•
Data structure : strings, hashes,
sets.
Use setlists,
to sets , sorted
To get all the tags for a given object :
implement tags
redis>smembers news:1000:tags
1. 5
2. 1
3. 77
4. 2
– sadd
news:1000:tags
1
– (integer) 1
we may want the list of all the objects having as
– sadd
tags 1, 2, 10, and 27 at the same time
news:1000:tags
Sinter tag:1:objects tag:2:objects tag:10:objects
2
tag:27:objects
– (integer) 1
– sadd
21
news:1000:tags
Set: Wildcard autocomplete
• Split every username to three letter chunks
– Simonw => sim, imo, mon, onw
• Create a set for each chunk
– Sim => { simonw, asimov, fasim}
• If the user types “simo”, return the
intersection of the “sim” and “imo”.
22
1.
2.
3.
Data structure : strings, hashes,
lists, sets , sorted sets.
Every member of a Sorted Set is associated with score, that is to sort set, from the smallest to the greatest
score. members are unique, scores may be repeated.
With sorted sets you can add, remove, or update elements in a very fast way (in a time proportional to the
logarithm of the number of elements, O(log(N))).
Get ranges by score or by rank (position) in a very fast way.
– ZADD can be used both to add items to the set and to update the score of an existing member.
 The ZRANGE family of commands return items by their index position within the ordered set. The optional
WITHSCORES argument returns the score for each item in the same response.
 ZRANGEBYSCORE query the ordered set by score, instead of by index.
zadd friends:leto 100 ghanima 95 paul 95 chani 75 jessica 1 vladimir
redis > zrange friends:leto 0 -1 withscores
1) "vladimir"
2) "1"
3) "jessica"
4) "75"
5) "chani"
6) "95"
7) "paul"
8) "95"
9) "ghanima"
10) "100"
23
Data structure : strings, hashes,
lists, sets , sorted sets.
Zset as index : any time you need to look up data based on range queries, you should be storing it in a sorted set.
They're indexes that you have to maintain yourself.
redis > zadd hackers 1940 "Alan Kay" 1953 "Richard Stallman" 1969 "Linus Torvalds" 1912 "Alan Turing"
redis > zrange hackers 0 -1
1) "Alan Turing"
2) "Alan Kay"
3) "Richard Stallman"
4) "Linus Torvalds“
redis > zrangebyscore hackers 1950 1990
1) "Richard Stallman"
2) "Linus Torvalds“
redis > zrangebyscore hackers -inf 1950
1) "Alan Turing"
2) "Alan Kay"
redis > zremrangebyscore hackers 1940 1960
(integer) 2
redis > zrange hackers 0 10
1) "Alan Turing"
2) "Linus Torvalds"
24
Sort set : Prefix autocomplete
• Type “binz”, return “binzhang”
1. Turn the first 4 or 5 characters of the strings into an
integer (you can
imagine every char as a digit of a radix 256 number
for instance, but
there are better representation) and add all your
usernames into a sorted set with score=integer.
2. Then using ZRANGEBYSCORE you can get all the
elements between a given range.
--ZRANGEBYSCORE key min max [WITHSCORES] [LIMIT offset count]
25
sorted sets : Inverted-Index Text Search with Redis
1.
2.
3.
4.
5.
an inverted index - a bunch of sets mapping terms to document IDs.
Assign each document an ID
Apply stemming and stopwords first
Create an inverted index, with one set per word
Create a set of docIDs for each term
–
ZINTERSTORE destination-zset number-of-zsets-to-intersect zset1 [zset2 ...] [WEIGHTS weight1 [weight2 ...]]
[AGGREGATE SUM | MIN | MAX]
26
Learn More Data structure : Pub/Sub
• Redis has native support for the
publish/subscribe (or pub/sub) pattern
1. receivers subscribe to messages that match a
specific pattern (for instance, messages that are
sent to a specific “channel”),
2. an procedurer/emitter to send messages to me
3. emitter and receivers to be loosely coupled-- hey
don’t need to know each other.
• The pub/sub command
1.
PSUBSCRIBE pattern [pattern ...] Listen for messages published to channels matching the given
patterns
27
Learn More Data structure :
redis > PUBLISH irc:football "Rock you"
(integer) 0
redis > PUBLISH irc:football "have a good day"
(integer) 1
redis > PUBLISH irc:football "water"
Pub/Sub
redis > subscribe irc:football
Reading messages... (press Ctrl-C to quit)
1) "subscribe"
2) "irc:football"
3) (integer) 1
1) "message"
2) "irc:football"
3) "have a good day"
1) "message"
2) "irc:football"
3) "water"
28
Agenda
Redis Manifesto 宣言
Data structure : strings, hashes, lists, sets and
sorted sets.
Leveraging Redis
The architecture of REDIS
Redis Admin and maintenance
Cases
29
Leveraging Redis
1.
2.
3.
4.
5.
6.
7.
Operations on KEYS
Big O Notation
Sort
EXPIRE
Transaction
Optimistic locking using check-and-set ( select for update)
Pipelining ( commands in batch)
30
Keys operation
 KEYS pattern
Lists all the keys in the current database that match the given pattern. [slow]
 TYPE key-name Tells the type of the key. Possible types are: string, list, hash, set, zset, and
none.
 MONITOR
Outputs the commands received by the Redis server in real time. [debug
purpose only]
KEYS h*llo
KEYS h?llo
KEYS h[ae]llo.
redis > type circle:jdoe:soccer
Set
DEL/EXISTS/EXPIRE/TTL/PERSIST/RANDOMKEY/RE
NAME
Redis can handle up to 2^32 keys
31
Big O Notation
How fast a command is based on the number of
items we are dealing with.
O(1): fastest, Whether we are dealing with 5
items or 5 million, you'll get the same
performance.
 Sismember : if a value belongs to a set
O(N) : linear commands, fts etc
 Keys
 ltrim, N is the number of elements being removed.
O(log(N)):
32
 zadd is a O(log(N)) command, where N is the number of
sort
•
Sort the values within a list, set or sorted set.
redis > rpush users:leto:guesses 5 9 10 2 4 10 19 2
(integer) 8
redis > sort users:leto:guesses
1) "2"
2) "2"
3) "4"
4) "5"
5) "9"
6) "10"
7) "10"
8) "19"
redis 9> sadd friends:ghanima leto paul chani jessica alia duncan
(integer) 6
redis > sort friends:ghanima limit 0 3 desc alpha
1) "paul"
2) "leto"
3) "jessica"
•
•
Redis is single-thread, so sort on salve if large dataset
Sort can store result to a key, one pattern for paginating through expensive sort results
(millions of items, for example) is to save the result to a temporary key, set an expiry on it
and use that for pagination via the LRANGE command.
33
Transaction
Every Redis command is atomic, including the ones that do multiple things.
–
–
–
–
incr is essentially a get followed by a set
getset sets a new value and returns the original
setnx first checks if the key exists, and only sets the value if it does not
Msetnx fails if any key already exist (less important, now we’ve hash)
MULTI command can run multiple commands as an atomic group.
1.
Marks the start of a transaction block. Subsequent commands will be queued for atomic execution using EXEC.

DISCARD can be used in order to abort a transaction. In this case, no commands are executed and the state of the connection is restored to
normal.
2.
3.
The commands will be executed in order
The commands will be executed as a single atomic operation (without another client's command being executed halfway
through)
4. That either all or none of the commands in the transaction will be executed
redis > multi
OK
redis > hincrby groups:1percent balance -9000000000
QUEUED
redis > hincrby groups:1percent balance -9000000000
QUEUED
redis > exec
1) (integer) -9000000000
2) (integer) -18000000000
34
Optimistic locking using check-andset
•
WATCH / UNWATCH
–
keys are monitored in order to detect changes against them. If at least one watched key is modified before
the EXEC command, the whole transaction aborts, and EXEC returns a Null multi-bulk reply to notify that the
transaction failed
•
.
let's suppose Redis doesn't have INCR
WATCH mykey
val = GET mykey
val = val + 1
MULTI
SET mykey $val
EXEC
•
Using WATCH to implement ZPOP
WATCH zset
element = ZRANGE zset 0 0
MULTI
ZREM zset element
EXEC
35
Expiration
1. Redis allows you to mark a key for expiration.
2. You can give it an absolute time in the form of a Unix timestamp (seconds since January 1,
1970) or a time to live in seconds.
a. expire pages:about 30
b. expireat pages:about 1356933600
-- delete key after 30 seconds
-- delete key at 12:00 a.m. December 31st, 2012.
c. ttl pages:about
d. persist pages:about
-- check ttl
-- remove expire limit
e. setex pages:about 30 '<h1>about us</h1>’
--set a string and specify a expire time
3. Lazy Expiration algorithm
Keys are expired simple when some clients tries to
access a key and the key is found to be time out
4. Once every second,
I. Tests 100 random keys from expired keys set.
II. Deletes all the keys found expired.
III. If more than 25 keys were expired, it starts again from
36
What if no available memory
 Redis will return an error on write operations, but read-only query still works
 Can specify “maxmemory” to define a hard limit for memory usage
 maxmemory-policy: specify the algorithm to use when we need to reclaim memory
a.
b.
c.
d.
e.
volatile-lru (default) remove a key among the ones with an expire set, trying to remove keys not recently used.
volatile-ttl remove a key among the ones with an expire set, trying to remove keys with short remaining time to live.
volatile-random remove a random key among the ones with an expire set.
allkeys-lru like volatile-lru, but will remove every kind of key, both normal keys or keys with an expire set.
allkeys-random like volatile-random, but will remove every kind of keys, both normal keys and keys with an expire set.
 LRU and minimal TTL algorithms
a)
b)
are not precise algorithms
for default Redis will check three keys(“maxmemory-samples”) and pick the one that was used less recently
redis > set newkey "maxsize"
(error) ERR command not allowed when used memory > 'maxmemory'
37
Redis Pipelining(How)
Send multiple commands to the server without waiting for the replies at all, and finally read the
replies in a single step.
$ (echo -en "PING\r\nPING\r\nPING\r\n"; sleep 1) | nc localhost 6379
+PONG
+PONG
+PONG
Not paying the cost of RTT for every call;
Client: INCR X
Client: INCR X
Client: INCR X
Client: INCR X
Server: 1
Server: 2
Server: 3
Server: 4
38
Redis Pipelining(why we need)
• Redis is a TCP server using the client-server
model and what is called a Request/Response
protocol.
– The client sends a query to the server, and reads
from the socket, usually in a blocking way, for the
server response.
– The server processes the command and sends
the response back to the client.
So for instance a four commands sequence is something like this:
Client: INCR X
Server: 1
Client: INCR X
Server: 2
Client: INCR X
Network Round Trip: Latency?
Server: 3
Client: INCR X
Server: 4
39
Agenda
Redis Manifesto 宣言
Data structure : strings, hashes, lists, sets and
sorted sets.
Leveraging Redis

Keys/O(n)/sort/expire/transaction/watch
Redis Admin and maintenance









Select database
Monitor Redis
Configure Persistence
Starting a Redis Slave
Handling a Dataset larger than memory
Upgrade Redis
BackUp Redis
Sharding Redis
Benchmarks
40
Databases in Redis
1.A database contains a set of data.
2.A database is to group all of an application's
data together and to keep it separate from
another application's.
3.databases are simply identified by a number
with the default database being number 0.
4.Number of databases is set via “databases”
param in config
5.change to a different database via select
command
41
Monitor Redis
1. MONITOR , is actually part of the Redis replication system. If you telnet directly to Redis and
type monitor, you'll see a live dump of all commands executing against the database. This is
really useful for debugging.
2. config set slowlog-log-slower-than 0
3. Redis-stat : similar like prstat
4. Info command
–
–
–
–
–
–
–
–
–
–
–
–
–
–
–
–
–
–
–
–
–
redis > info
redis_version:2.4.8
redis_git_sha1:00000000
redis_git_dirty:0
arch_bits:64
multiplexing_api:epoll
gcc_version:4.1.2
process_id:5898
uptime_in_seconds:163519
uptime_in_days:1
lru_clock:1033766
used_cpu_sys:1.19
used_cpu_user:2.36
used_cpu_sys_children:0.00
used_cpu_user_children:0.00
connected_clients:2
connected_slaves:1
client_longest_output_list:0
client_biggest_input_buf:0
blocked_clients:0
used_memory:1384280
42
Configure Persistence(1)

Persistence Mode: snapshotting and AOF. It should be configured in a way that suits your
dataset and usage patterns.
1. snapshotting, which consists of saving the entire database to disk in the RDB format (a
compressed database dump). This can be done periodically at set times, or every time a
configurable number of keys changes.
# save <seconds> <changes>
save 900 1 --after 900 sec (15 min) if at least 1 key changed
save 300 10 --after 300 sec (5 min) if at least 10 keys changed
save 60 10000 --after 60 sec if at least 10000 keys changed
2. The alternative is using an Append Only File (AOF). This might be a better option if you have
a large dataset or your data doesn’t change very frequently.
./redis-server --appendonly yes
3.
4.
5.
6.
It is possible to combine both AOF and RDB in the same instance.
Master ->> Slave can be a option.
Both are sequential IO
When Redis starts, it will read RDB or AOF to load all data into memory.
43
Configure Persistence(2)
1. Snapshotting
–
–
–
performs point-in-time snapshots of dataset at specified intervals. (
a full dump of your database to disk, overwriting the previous dump only if successful.
Can manually trigger snapshotting with the SAVE and BGSAVE commands.
•
•
–
BGSAVE forks the main Redis process and saves the DB to disk in the background.
SAVE performs the same operation as BGSAVE but does so in the foreground, thereby blocking your Redis server.
are also used when performing a master -> slave synchronization.
2. Append Only File(AOF)
–
–
–
keeps a log of the commands that change your dataset in a separate file.
an append only log. no seeks, nor corruption problems (redis-check-aof )
Appendfsync: how often the AOF gets synched to disk (fsync syscall) :
•
–
Always (be able to group commit), every sec, and no.
BGREWRITEAOF rewrites the AOF to match the current database; can reduce size of AOF greatly. (For
example, if you are incrementing a counter 100 times, you'll end up with a single key in your dataset
containing the final value, but 100 entries in your AOF. 99 of those entries are not needed to rebuild the
current state.)
44
Redis Admin and maintenance:
Master --Slave
 Use slave : Load balance read queires, standby, Backup ,DW queries
 master-slave replication natively:



A master can have multiple slaves.
Slaves are able to accept other slaves connections.
Redis replication is non-blocking on the master side, this means that the master will continue to serve
queries when one or more slaves perform the first synchronization.
 configure replication on the configuration file before starting a server


slaveof master-ip-or-hostname masterport
masterauth master-password
 by connecting to a running server and using the SLAVEOF command.


SLAVEOF master-ip-or-hostname [masterport]
CONFIG SET masterauth password
45
Handling a Dataset Larger Than
Memory
•
Memory (VM) since version 2.0 (deprecated after Redis 2.4) .
vm-enabled yes
vm-swap-file
•
Allow a dataset bigger than your available RAM by swapping rarely used values to disk and
keeping all the keys and the frequently used values in memory.
1.
2.
3.
4.
5.
The keys are always kept in memory. Values can be swapped.
Redis server might end up blocking clients in order to fetch the values from disk.
Slow snapshot, Redis needs to read all the values swapped to disk in order to write them to the RDB file. AOF
is better at this case.
VM also affects the speed of replication, because Redis masters need to perform a BGSAVE when a new slave
connects.
SSDs such as Flash is encouraged
46
Upgrading Redis
Redis can’t do online binary upgrades
solution
1. starting a new Redis server in slave mode,
2. switching over the clients to the slave
3. promoting the new server to the master role.
make sure to test before doing it on your
production servers.
47
Backing up Redis
• Depending on which Redis persistence model
you’re using.
1.With the default persistence model
(snapshotting), you’re best off using a
snapshot as a backup. /* cold backup */
redis-cli BGSAVE
Copy
2.If you’re using only AOF, you’ll have to back up
your log in order to be able to replay it on
startup.
48
Sharding Redis
•
Where is Redis Cluster
–
•
Under development. Probably reasonable beta for
summer 2012 and ship the first stable one before end
of 2012.
Have to implemented in the client library or
application
–
–
you should probably use consistent hashing.
you will not be able to perform some operations that
affect multiple keys, because those keys might be in
different shards (servers).
49
Benchmarks--- How fast is Redis?
•
redis-benchmark utility that simulates SETs/GETs done by N clients at the same time sending M total
queries
[hadoop@phxrueidb03 src]$ ./redis-benchmark -q -n 100000
PING (inline): 97370.98 requests per second
PING: 101214.58 requests per second
MSET (10 keys): 66357.00 requests per second
SET: 105263.16 requests per second
GET: 103199.18 requests per second
INCR: 104493.20 requests per second
LPUSH: 104931.80 requests per second
LPOP: 104384.13 requests per second
SADD: 104931.80 requests per second
SPOP: 103950.10 requests per second
LPUSH (again, in order to bench LRANGE): 104931.80 requests per second
LRANGE (first 100 elements): 44964.03 requests per second
LRANGE (first 300 elements): 22825.84 requests per second
LRANGE (first 450 elements): 16564.52 requests per second
LRANGE (first 600 elements): 12701.64 requests per second
50
Benchmarks- Redis VS memcached
51
Benchmarks--- How fast is Redis?
•
Redis is a server: all commands involve network or IPC roundtrips. Cost of most operations is
precisely dominated by network/protocol management.
–
•
•
Redis commands return an acknowledgment for all usual commands.
Redis is an in-memory data store with some optional persistency options. Some persistency
option would bring latency.
–
•
low latency network
huge page & SSD
Redis is a single-threaded server. It is not designed to benefit from multiple CPU cores.
People are supposed to launch several Redis instances to scale out on several cores if needed.
–
Redis favors fast CPUs with large caches and not many cores.
52
Agenda
Redis Manifesto 宣言
Data structure : strings, hashes, lists, sets and
sorted sets.
Leveraging Redis
Redis Admin and maintenance
Configure Persistence/Redis Slave/Handling a
Dataset larger than memory
Upgrade Redis/BackUp Redis/Sharding Redis/
benchmarks
The architecture of REDIS
 How Redis works
53
How Redis works
How a command received by a client is processed internally by Redis:
1.
2.
Redis uses a single thread that manages synchronously all network connection. A thin event library has
been implemented to abstract several unix system calls (epoll, select, kqueue).
Requests are managed with commands. Using a command table and according what event is read from
sockets a command handler is invoked to perform desired action.
54
Latency in Redis
•
Latency induced by network and communication
–
•
Single threaded nature of Redis
–
–
•
–
a request is slow to serve all the other clients will wait for this request to be served
commands operating on many elements, like SORT, LREM, SUNION and others. For instance taking
the intersection of two big sets can take a considerable amount of time.
run all your slow queries on replciations
Latency generated by fork
–
•
•
a mostly single threaded design ( I/O threads in background since 2.4)
all the requests are served sequentially
Latency generated by slow commands
–
–
•
use aggregated commands (MSET/MGET) and Pipelining
The fork operation (running in the main thread) can induce latency by itself.
Latency induced by swapping (operating system paging)
Latency due to AOF and disk I/O
55
Memory efficient for list

adlist.h: A generic doubly linked list implementation
typedef struct list {
listNode *head;
listNode *tail;
void *(*dup)(void *ptr);
void (*free)(void *ptr);
int (*match)(void *ptr, void *key);
unsigned int len;
} list;
typedef struct listNode {
struct listNode *prev;
struct listNode *next;
void *value;
} listNode;


O(1) is cool but *prev/*next would take amounts of bytes if *value is few bytes.
Ziplist (list-max-ziplist-entries 512 & list-max-ziplist-value 64)
I.Save memory by using a little more CPU
II.Pack list in a single block of memory
III.Value header holds encoding / value length
IV.O(memory size) LPUSH / LPOP
V.Good fit for small payload, limited size
56
Memory efficient for hash
•
Zmap (hash-max-zipmap-entries 512 & hash-max-zipmap-value 64)
–
•
keys and values are prefixed length "objects", the lookup will take O(N) where N is the number of elements in the
zipmap and *not* the number of bytes needed to represent the zipmap.
Other data structure also have similar improve ( sort sets, intset)
57
Skip-list for sort set
1.
2.
3.
4.
5.
6.
Consists of several levels, Each level is a sorted list
All keys appear in level 1
If key x appears in level n, then it also appears in all levels below n
An element in level n points (via down pointer) to the element with same key in the level below
Each level has int_min and int_max
Top points to the smallest element in the highest level
58
Memory structure

lazy rehashing：The more operation you run into an hash table that is rhashing, the more rehashing "steps" are
performed, so if the server is idle the rehashing is never complete and some more memory is used by the hash table.

active rehashing： uses 1 millisecond every 100 milliseconds of CPU time in order to help rehashing the main Redis hash
table (the one mapping top-level keys to values).
59
memory fragmentation

Info
used_memory:21279952
memory allocated to redis
used_memory_human:20.29M
used_memory_rss:23654400
memory from OS, result of ps or top
used_memory_peak:21704152
used_memory_peak_human:20.70M
mem_fragmentation_ratio:1.11
= used_memory_rss/used_memory
mem_allocator:jemalloc-2.2.5
default in linux 2.4 and 2.6
 String:
 dictEntry(12bytes)+sds(store key)+redisObject(12bytes)+sds(store value)
Set hello word = 16(dictEtnry) + 16 (redisObject) + 16(“hello”) + 16(“world”),
60
Redis Security
1. Redis is designed to be accessed by trusted clients inside trusted environments.

Firewall on redis port
2. Redis is not optimized for maximum security but for maximum performance and simplicity.
3. Authentication feature


The password is in clear text inside redis.conf file and client configuration
AUTH command, like every other Redis command, is sent unencrypted
4. Data encryption support - None
5. Disabling of specific commands -- rename-command FLUSHALL ""
61
Agenda
Redis Manifesto 宣言
Data structure : strings, hashes, lists, sets and
sorted sets.
Leveraging Redis
Redis Admin and maintenance
The architecture of REDIS
Event library /Memory efficient
Latency /Security
Cases
 http://lloogg.com
 a simple Twitter clone
62
CASE 1: http://lloogg.com/
1.
2.
3.
List, lpush,ltrim show access history
Strings:incr show pageveiws
zset: show opt references, ref as a score
63
a simple Twitter clone
Register:
INCR global:nextUserId => 1000
SET uid:1000:username antirez
SET uid:1000:password p1pp0
Circles and posts:
SET username:antirez:uid 1000
uid:1000:followers => Set of uids of all the followers users
uid:1000:following => Set of uids of all the following user
uid:1000:posts => a List of post ids, every new post is LPUSHed here.
Cookie:
SET uid:1000:auth fea5e81ac8ca77622bed1c2132a021f9
SET auth:fea5e81ac8ca77622bed1c2132a021f9 1000
New post
INCR global:nextPostId => 10343
SET post:10343 "$owner_id|$time|I'm having fun with Retwis"
Pageing
$posts = $r->lrange($key,$start,$start+$count);
Making it horizontally scalable
1. Split by hash key
LPUSH to user’s followers
foreach($followers as $fid) {
$r->push("uid:$fid:posts",$postid,false);
}
Push to latest news:
$r->push("global:timeline",$postid,false);
$r->ltrim("global:timeline",0,1000);
64
Sina weibo
65
Summary
Redis Manifesto
 Memory #1;data structure server
Data structure :
 strings, hashes, lists, sets and sorted sets, pub sub
Leveraging Redis
 KEYS/Big O /Sort/EXPIRE/Transaction/Optimistic locking/Pipelining
Redis Admin and maintenance
 Select /Redis/Persistence/Replication/
 VM/Upgrade/BackUp/Sharding /Benchmarks
The architecture of REDIS
 Event library /Memory efficient
 Latency /Security
Cases
66
Redis
• Q&A
67