Let’s Learn Together Sessions: Redis

This article briefly describes Redis, one of the most popular in-memory data storage, database, cache, and messaging agent NoSQL solutions.

Photo by Aaron Burden on Unsplash

What is Redis?

Redis is one of the fastest in-memory data stores which its database model is based on key-value stores. The name of Redis comes from the abbreviation of the words “Remote”, “Dictionary” and “Server”. It was developed by Salvatore Sanfilippo to improve the scalability performance of his startup company which provides a real-time analytics service called lloogg.com.

Redis is an open-source solution written in C language, which was published on Github. On the other hand, Redis also has a BSD3-Clause commercial license, which protects the redistribution and use of source code and binary forms, together with printed terms.

Redis has a big open source community which not only supports it to maintain its simplicity and performance but also adding enterprise capabilities and functionalities and integrations for different modules and designs.

This community was called Redis Labs is a private company was founded in 2011 by Ofer Bengal and Yiftach Shoolman. The founder of Redis, Salvatore Sanfilippo is also part of the Redis Labs which leads the open-source development of Redis.

Yiftach Shoolman (on left) and Ofer Bengal (on right), the founders of Redis Labs.

There are many clients that you can use Redis with different languages ​​like Java, Node.js, C, C #, Swift, Python, etc. You can reach the Redis client alternatives and recommended clients by the Redis community from the link below:

Redis

Redis supports many key-value-based data structures such as Hashes, Sets, Sorted Sets, and Lists. In addition, Redis supports pub/sub message brokers where published messages are assigned to channels without knowing of connected subscribers. Subscribers who are connected to related channels receive published messages.

Redis provides improved read performance and stable failover mechanism and faster data recovery thanks to its master-slave replication and sharding mechanism support.

Redis also supports persisting to disk. You can easily enable or disable this property and manage it parametric. ( automatically in a specified period such as each day at 9:00 or each hour, or manually calling by “bgsave” command or trigger it automatically at shutdown). So Redis is a really good alternative for designing consistent and partition tolerant data stores.

Image Credit: https://datasciencepedia.com/ :: CAP Theorem

Redis provides a highly available structure through a master-slave replication mechanism. You can easily scale your Redis by adding new nodes or removing nodes. This is a very secure and stable way to scale your application under the lights of your application needs.

Where to use Redis?

Redis is used for different purposes in many different sectors and domains. The well-known companies that use Redis are Twitter, Github, Pinterest, Snapchat, Stackoverflow, and so on…

One of the main use cases of Redis is caching. It decreases data access latency time and increases the data operations (access or write) performance. It provides a fast response time, even lower than 1 millisecond for mostly or frequently used data. You can increase response time and write time performance with some configurations such as read from slave and write to master nodes functionality etc…

Redis is also a good alternative for the pub/sub and queue mechanisms thanks to its pattern matching and different data structure support. You can create a room or channel-based chat applications, real-time content streams, and also social media feed infrastructures.

Redis is also used in-memory session store by frontend and backend applications. It provides to store session data, user sessions, and credentials in a highly scalable and persistent way.

Redis sorted set data structure is a preferable solution for game developers to develop a real-time leaderboard functionality. Many popular game development companies such as Roblox and Jelly Button Games etc and leading game live streaming companies such as Twitch uses Redis as their in-memory data stores needs. You can reach the detailed documentation and solution of Redis labs in the gaming sector with the link below:

Redis Enterprise Databases for Game Development | Redis Labs

In addition to these use cases, you can prefer Redis for many domains to increase your application scalability and performance.
How does Redis provide this, let’s find out?

How does it work?

If you have decided to use Redis, you must first choose which Redis mode to use. Redis provides four different modes as below for their users:

• Standalone mode
• Replicated mode
• Cluster mode
• Sentinel mode

Redis standalone mode is similar to relational databases in terms of scalability. It supports vertical scalability and does not support high availability because of its single point of failure. Standalone mode is easy to set up but it is not suggested for production environments.

The second mode, replicated mode, solves the availability issue of standalone mode thanks to its master-slave structure. There are master nodes and some replicas. Master nodes update replicas about new data and replicas have no network and data transfer between them. The main problem of replicated mode is resilience. You have to apply operations and manages nodes manually. If the master is down, you have to assign a replica as a new master manually and restart all the clients.

Sentinel mode has been developed as an alternative to the first two nodes to solve the resilience problem. There are sentinel nodes besides master or slave nodes. Sentinel mode aims to the resilience of both master-slave nodes and sentinel modes. Every node knows who is the master node and automatically promotes a new master in a fallback scenario.

Sentinel mode uses a quorum system to provide a master promotion process. Quorum means the minimum number of nodes required to assign the new master. Quorum number can be changed according to your Redis sentinel design(number of masters or slaves — number of sentinels so on ) and your resources. Sentinel monitor, analyze, and report the cluster. If the master node is down, it creates a connection between other nodes and provides assigning of a new master. Or it extracts unhealthy applications from the cluster and reconnects it to cluster back when it is healthy.

The last mode, cluster mode, provides resilience and scalability functionality with the multi-master configuration. It is thought of as one master with N slaves. The master node is responsible for all write operations, and slaves are just read-only. This decreases the traffic on the master node and prevents latency on reading operations.

In cluster mode, data is sharded into 16k buckets. Each bucket assigns to a master and replicated twice at least. In cluster mode, each node can be either a master or replica node. For a highly available and stable cluster structure, there must be at least 6 nodes( 3 of them must be masters, 3 of them must be slaves) So if you would like to use Redis with cluster mode, you have to need more resources compared to other deployment modes.

And also you have to check the client’s support of cluster mode on selected development language because all Redis clients do not support cluster mode.

For this article, I developed a demo project which supports three modes: standalone, sentinel, and cluster. So you can use your selected deployment mode by enabling related configuration and Redis utility class.

Where to start?

Firstly, you have to download the stable Redis version from the Redis.io download page. This download page only works in Unix or Linux distributed computers. If you would like to install Redis on a Windows machine, you can use below Microsoft archive page link.

microsoftarchive/redis

After Redis installation is completed, you can install redis-cli package to manage your Redis cluster and make queries on your Redis servers. To install redis-cli you can use the following npm command:

npm install -g redis-cli

To start a simple Redis server on your machine, you must go to the /usr/local/etc folder and execute redis-server command without any arguments. If you run the Redis server without arguments, it automatically creates a Redis server on port 6379. It uses the default redis configuration, called redis.conf, which was located under /usr/local/etc folder. You will see the below screen if everything goes fine :)

Then you can check the health status of Redis with redis-cli also with the following command:

my-pc:etc rootuser$ redis-cli
127.0.0.1:6379> ping
PONG
127.0.0.1:6379>

As I already said, Redis provides many data structures such as Hash, Set, List, Counts, and Sorted Sets, etc. Redis documentation tells you how to use these data structures and how to manage data with basic CRUD operations(set, get, update, delete). For example, if you use a hash on your Redis cluster, you can add data to your hash with the following hmset command:

hmset student_id name "George" major "Mathematics"

Then you can reach any field on this data with use hmget command with student_id parameter like:

hmget student_id name

Redis provides a bunch of really good commands to manage your data. You can reach the full list of commands with the following Redis documentation link:

Command reference - Redis

How to use Redis with Java?

If you would like to use Redis with Java, you have to check suggested Redis clients from the Redis clients page. There are a bunch of open-source Redis Java clients, but three of them are recommended from the Redis community: Jedis, lettuce, Redisson.

For my demo project, I need to use multiple Redis modes, standalone, sentinel, and cluster. Because Jedis supports all of these modes, I choose Jedis as my Java client. Also, the other two recommended clients support these three modes, so you can also use them in your applications. I have not tried other clients, so I can not compare.

To use Jedis on a Spring boot project, You have to add Maven or Gradle dependency as follow:

Maven->

<dependency>
   <groupId>redis.clients</groupId>
   <artifactId>jedis</artifactId>
</dependency>

Gradle ->

compile group: 'redis.clients', name: 'jedis'

To use Jedis, you have to initialize a Jedis pool with a JedisPool object. You can add Redis-specific connection configurations to Jedis Pool as a config. I will use the same connection configurations for three Redis modes, so I created an abstract class, called as RedisConfiguration, which create a simple Jedis pool configuration that is extended from the generic Java GenericObjectPoolConfig class as follow:

public abstract class RedisConfiguration {

    public JedisPoolConfig getJedisPoolConfig(){

        JedisPoolConfig jedisPoolConfig = new JedisPoolConfig();

        jedisPoolConfig.setBlockWhenExhausted(false); 
        jedisPoolConfig.setMaxIdle(15);  
        jedisPoolConfig.setMinIdle(1);  
        jedisPoolConfig.setMaxTotal(20); 

        return jedisPoolConfig;
    }
}

The first parameter, “blockWhenExchausted” is set to false for the triggering client to return an error when a client tries to request a connection and none of the Redis servers are available. “MaxIdle” property means maximum number of idle(used by an application and waits in the pool to use) connections for the pool. The default value for this property is 8, but we set it to 15 to give permission connecting of more clients to the pool. “MinIdle” property defines the minimum number of idle connections for making the pool work. Default is 0, but you can change this value like the above example. You can limit the number of total connections(idle + used) by using “MaxTotal” property.

I used this configuration pool for all of the Redis modes, but you can create different configurations for different Redis modes. Then, I created Jedis pools and Jedis cluster object configurations with distinct configuration classes:

• RedisStandaloneConfiguration
• RedisSentinelConfiguration
• RedisClusterConfiguration

For standalone configuration, we created a JedisPool object with the pool configuration which we defined above as below:

JedisPoolConfig jedisPoolConfig = getJedisPoolConfig();


return new JedisPool(jedisPoolConfig,
        appConf.getRedis().getStandalone().getHost(),
        appConf.getRedis().getStandalone().getPort(),
        appConf.getRedis().getStandalone().getTimeout());

We set host, port, and timeout values from the application configuration file. We start the Redis server at localhost with port 6379, and set the timeout value to 3 seconds as follow:

redis.standalone.host= localhost
redis.standalone.port= 6379
redis.standalone.timeout = 3000

For the sentinel configuration, we created a JedisSentinelPool object similar to the JedisPool object in a standalone configuration. For sentinel mode, we created three sentinel nodes in ports 5000,5001,5002. JedisSentinelPool object creation process implemented as below:

@Bean
public JedisSentinelPool getJedisSentinel(){

    String[] sentinelNodes = appConf.getRedis().getSentinel().getHost().split(";");

    Set sentinels = new HashSet<>();


    for(String sentinelNode : sentinelNodes){

        if(sentinelNode!= null &&
                !sentinelNode.isEmpty()){
            sentinels.add(sentinelNode);

        }

    }

    JedisPoolConfig jedisPoolConfig = getJedisPoolConfig();

    return new JedisSentinelPool("mymaster",sentinels,jedisPoolConfig,appConf.getRedis().getSentinel().getTimeout());

}

We added all hosts from the application.configuration one by one and also timeout value also comes from the configuration file:

redis.sentinel.host= 127.0.0.1:5000;127.0.0.1:5001;127.0.0.1:5002
redis.sentinel.timeout = 3000

Cluster mode configuration is a little bit different from standalone and sentinel configurations. Instead of creating a pool object, cluster mode has its pool structure so we can create a simple JedisCluster object for connecting a Redis cluster:

@Bean
public JedisCluster getJedisCluster(){

    String[] clusterNodes = appConf.getRedis().getCluster().getHost().split(";");


    Set<HostAndPort> clusters = new HashSet<>();


    for(String clusterNode : clusterNodes){

        if(clusterNode!= null &&
                !clusterNode.isEmpty()){

            String[] clusterNodeEndpoint = clusterNode.split(":");

            if(clusterNodeEndpoint.length == 2)
                clusters.add(new HostAndPort(clusterNodeEndpoint[0],Integer.parseInt(clusterNodeEndpoint[1])));

        }

    }

    JedisPoolConfig jedisPoolConfig = getJedisPoolConfig();

    return new JedisCluster(clusters,
            appConf.getRedis().getCluster().getTimeout(),
            appConf.getRedis().getCluster().getTimeout(),jedisPoolConfig);


}

We have assigned cluster nodes individually with HostAndPort objects. We created 6 nodes( 3 of them is master,3 of them is a slave) with the ports from 7000 to 7005. Also, unlike other configurations, we set the socket timeout value to 3 seconds. We also added pool configuration like other configurations to overwrite the default pool mechanism of the Redis cluster. We get configuration parameters from the application configuration file as follow:

redis.cluster.host= 127.0.0.1:7000;127.0.0.1:7001;127.0.0.1:7002;127.0.0.1:7003;127.0.0.1:7004;127.0.0.1:7005
redis.cluster.timeout = 3000

We can use all three modes in the demo project. To do this you need to set the ActiveRedisMode property to the desired mode, as follows:

# Redis Modes -> standalone,sentinel,cluster

activeRedisMode= standalone

To create a standalone Redis server, go to the /usr/local/etc folder and execute redis-server command. For sentinel Redis mode, you can follow directions at below Redis documentation page:

Redis Sentinel Documentation - Redis

To create a Redis cluster with 6 cluster nodes, you can follow the Cluster documentation on the Redis page:

Redis cluster tutorial - Redis

For my demo project, I created a simple POJO class named Student. I would like to store all students in Redis with Redis “hash” object, and also I need to manage students with basic CRUD operations. For that aim, we created a single interface, called “RedisCacheManager”. We added the below functionalities to the Manager class:

public interface RedisCacheManager {

    StudentCacheObject getObjectFromHashWithKey(String key);

    String getSingleFieldFromHashWithKey(String key, String valueType);

    void updateObjectInHashWithKey(String key, StudentCacheObject studentCacheObject);

    void setNewObjectToHashWithKey(StudentCacheObject studentCacheObject);

    void removeSingleValueFromHashWithKey(String key, String removedType);

    void removeObjectFromHashWithKey(String key);

    List<StudentCacheObject> getAllObjects();

}

The first two methods are classic get operations. While the first method returns all object fields in a Student hash object, the second one just returns desired field from Student such as email and, GPA, etc. Any get operation on the Redis hash object is done by using “hget” or “hgetall” commands. If you would like to reach a single field in a hash object, you must use “hget” command. On the other hand, you must use “hgetall” command to get all fields from the hash object. We defined these two objects for all three Redis mode manager classes as in the following code snippet:( RedisStandaloneCacheManager, RedisSentinelCacheManager, RedisClusterCacheManager)

public StudentCacheObject getObjectFromHashWithKey(String key) {
    Map<String, String> redisMap = redisClient.hgetAll(key);
    ObjectMapper mapper = new ObjectMapper();

    return mapper.convertValue(redisMap, StudentCacheObject.class);
}

public String getSingleFieldFromHashWithKey(String key, String valueType) {
    return redisClient.hget(key, valueType);
}

Sometimes we need to update some fields of an object in the database. For these types of operations, we can use update operations. In the Redis hash data structure, there is no distinct update method. We use “hmset” command to update already defined objects with desired parameters. “updateObjectInHashWithKey” method provides to update any field in the Student hash object by using key:

public void updateObjectInHashWithKey(String key, StudentCacheObject studentCacheObject) {

    if (studentCacheObject == null)
        throw new IllegalArgumentException("[(updateObjectInHashWithKey)] Student object cannot be null");

    if (key == null)
        throw new IllegalArgumentException("[(updateObjectInHashWithKey)] Key cannot be null");

    Map<String, String> studentObjectMap = convertStudentObjectToMapObject(studentCacheObject);

    redisClient.hmset(key, studentObjectMap);
}

Another and most used operation in classic APIs is the insert or add operation.
The “setNewObjectToHashWithKey” method is used to add a new Student object with parameters defined in Redis. “hmset” command is used to add a new object to hash in Redis.
We have implemented this operation as follows:

public void setNewObjectToHashWithKey(StudentCacheObject studentCacheObject) {

    if (studentCacheObject == null)
        throw new IllegalArgumentException("[(setNewObjectToHashWithKey)] Student object cannot be null");

    if (studentCacheObject.getStudentId() == null)
        throw new IllegalArgumentException("[(setNewObjectToHashWithKey)] Student id cannot be null");

    Map<String, String> studentObjectMap = convertStudentObjectToMapObject(studentCacheObject);

    redisClient.hmset(studentCacheObject.getStudentId(), studentObjectMap);

}

If you can add a new object to Redis, you should be able to remove it at any time. Or you may want to remove a single area for an object. It is possible to remove a field or field list from the object with the “hdel” command in Redis. The following two processes fully meet this need:

public void removeSingleValueFromHashWithKey(String key, String removedType) {

    redisClient.hdel(key, removedType);
}

public void removeObjectFromHashWithKey(String key) {

    redisClient.hdel(key, "studentId", "name", "email", "major", "gpa");
}

The last method, “getAllObjects” is another get operation that returns all Student objects in Redis one by one. For this process, we first use the “keys” command to get all the keys on the Redis server. Then we use the “hgetall” command to get all the fields of Redis hash objects.

public List<StudentCacheObject> getAllObjects() {

    List<StudentCacheObject> studentCacheObjectList = new ArrayList<>();

    Set<String> keys = redisClient.keys("*");
    if (!CollectionUtils.isEmpty(keys)) {
        for (String key : keys) {

            Map<String, String> redisObjectMap = redisClient.hgetAll(key);

            ObjectMapper mapper = new ObjectMapper();

            studentCacheObjectList.add(mapper.convertValue(redisObjectMap, StudentCacheObject.class));
        }

        return studentCacheObjectList;
    }

    return null;
}

For the demo application, we created also a controller class to test our Redis hash functionalities. StudentController class can be used to test all of these methods with different Redis modes.

@PostMapping("/addNewStudent")
public String addNewStudent(StudentCacheObject studentCacheObject) {

    redisCacheManager.setNewObjectToHashWithKey(studentCacheObject);

    return "OK";
}


@GetMapping("/getStudentInfo")
public StudentCacheObject getStudentInfo(@RequestParam String studentId) {

    StudentCacheObject studentCacheObject = redisCacheManager.getObjectFromHashWithKey(studentId);

    if(studentCacheObject.getStudentId() == null)
        throw new NullPointerException("Student does not exist on system.");

    return studentCacheObject;
}

@GetMapping("/getStudentSingleField")
public String getStudentSingleField(@RequestParam String studentId,@RequestParam String fieldType) {

    return redisCacheManager.getSingleFieldFromHashWithKey(studentId,fieldType);

}

@GetMapping("/getAllStudents")
public List<StudentCacheObject> getAllStudents(){

    return redisCacheManager.getAllObjects();

}

@PutMapping("/updateStudentInfo")
public String updateStudentInfo(@RequestParam String studentId,
                                StudentCacheObject studentCacheObject) {

    redisCacheManager.updateObjectInHashWithKey(studentId, studentCacheObject);

    return "OK";

}

@DeleteMapping("/removeStudent")
public String removeStudent(@RequestParam String studentId) {

    redisCacheManager.removeObjectFromHashWithKey(studentId);

    return "OK";
}

Conclusion

Redis is a really fast in-memory data store in the database world. With Redis and similar data stores, it is possible to scale your databases horizontally and increase data performance.

Redis offers you different distribution modes according to your needs and resources. You can use Redis as a basic data store that caches your frequently accessed data to have more powerful and faster data operations and processing.

On the other hand, you can think of Redis as an alternative data storage solution to your session stores, pub/sub mechanisms, real-time social media feed, and comment data structures, and even gaming leaderboard functionalities.

Redis supports many different data structures from simple hash objects to counter objects. You can use Redis on many OS systems Unix, Linux, or even Windows(with a Microsoft extension).

Many Redis clients make it possible to manage Redis for many programming languages ​​such as Java, C #, Python and Node.js, etc…

Redis is easy to learn, and documentation and community are really powerful and grow day by day. Redis Labs also provides both open source and enterprise solutions for Redis.

I used Redis on Node.js application as a highly available data store solution, and highly recommend Redis you to try for your data storage needs in your backend and frontend systems.

The code is available on Github.