Improve the Quality of Breaks With Kafka
Who is the best? To answer this question once and for all we created a tool with which we can track scores, rankings, and statistics. Why you might ask, it is simple, playing kicker, foosball or töggele is in our DNA. So is creating awesome tools with awesome components.
Who does not like breaks? I do like them. One could say that I love them. There does not one day go by in our office when we don’t go to the töggelikasten to play a game. Even more so, I like them so much that I convinced the guys at Confluent that it makes perfect sense to talk about breaks together with Kafka at the KafkaSummit.
What is that game, you ask? It is famously played all around Europe and also (maybe even especially) in Switzerland. In essence, it is a fast-paced, semi-competitive, 2v2 game that crudely imitates football (ehrm, soccer). It is aptly named foosball or table soccer, depending on who you ask, and in swiss german, it is called töggele (from the noun töggel, which means figurine).
However, playing such a game very often leads to some inconveniences.
Firstly, people tend to have a favorite position, either defense or offense. The issue starts when more than two players have a strong preference for one spot because they need to agree who plays in what place. The second issue is when the skill level is vastly different. Thus fair teambuilding is hard for people. It turns out this is a child’s play for a computer. Simply randomizing the teams leads to good results and no discussions apart from the occasional glare from people who were assigned the same role repeatedly.
Secondly, I like to know how good I am. And for this, I want statistics, rankings, or generally, data. Thus creating a ranking seemed obvious. Additionally, to a simple ranking, I think it is vital that people who are ranked higher should receive fewer points than people who are ranked lower. Such that the spread does not get too big. Implementing some diminishing return functionality seemed crucial.
To alleviate us from those pains, we decided to band together and create an application called the Töggelomat during a hackathon.
So what the hell does this game have to do with Kafka? As you probably already know, Kafka is a streaming platform for event sourcing and processing, among other use-cases. Generally, most people would not consider using Kafka for such an application and would instead go for a relational database. We are not most people. For us, it makes perfect sense to think of matches as events and process them accordingly. Also, there were no time or budget restrictions because we did it in our spare time and were free to use the technology we love and learn more about it at the same time. : heart eyes:
There is one minor downside to using Kafka Streams for such a small tool. Operating a Kafka cluster is not trivial, and it makes no sense to do this overhead for a small application like this. There are two choices available to fix this. Either use an already existing cluster and write the data there or go for Confluent Cloud, which has very decent pricing for small use cases. For me, it makes perfect sense to use Confluent Cloud for that because it feels natural.
Other things that we wanted to try out were Quarkus and ksqlDB. Quarkus is an alternative to spring boot. It has been made to build applications for the cloud and includes several libraries that work well together. ksqlDB seems like an obvious choice when you want to experiment with Kafka and use the newest and fanciest tools. I had no previous experience with it and was eager to try it out.
Given those conditions, let me tell you the story of our data. It is a story of little Emily. She applied to SPOUD and got the job. She was delighted. When she then came to the office, everyone welcomed her, and she was happy. Her start at SPOUD caused an event to be triggered, which writes into the spoud employee topic. This topic has many uses, but for this, it is used to create the players. With ksqlDB, we listen to the employee stream and devise players from them.
Emily had a fun first day, with coffee, water, and a lot of snacks. At about nine, the team invited her to play a game of kicker. She was enthusiastic. She is not so good, but she likes to play regardless. The team filled out the application, and she was assigned a position. She tried her best and scored some points.
Nevertheless, she lost the game. The team entered the score and went back to their places. The result of the match is another event. It gets published to match the results topic with the current state of the players.
Emily feels a bit sad because she lost and wants to check how many points she lost. In the records, she sees the entry of her last match with the calculated points. She lost 20 points. For the evaluation of the scores, winners, and losers, the service listens to the match result topic’s events. After the calculation, it publishes these specifications into another topic called enriched results.
Emily wanted to check where she is on the ranking and noticed that she only lost 19 points. After exploring, she finds that her points changed differently than what she first expected because each player is always rewarded one point for playing, she was delighted. After publishing the enriched results, a ksqlDB definition calculates the score changes for each player. These are published in a different topic called point change. This topic is needed to enable multiple sources of points like betting or achievements. Those changes are then applied to the players, and the player stream gets updated with the new state.
After a while, Emily wants to regain her points. She plays again with the chaps. To everyone’s surprise, Emily wins and is very happy. After checking the history, she notices that she gained more points than she lost before. The current state of the player influences the scores that are given or taken. Meaning if two lower-ranked players play against two higher-ranked ones, the latter can only win a small number of points while the former can earn a lot. This is what happened when Emily played again because she is ranked lower; she could win more points.
The story of Emily ends here. And it shows very well how the data flows through the application. I used her as an example of how we used Apache Kafka to navigate through the töggelomat. This use case is rather simple and easily understandable, but the same techniques can be applied to more complex workflows. The producer of data (in our case, the entry of match results) should not have to care about what is done with it afterward. The whole process does not need to be coupled. With this architecture and data flow, we can easily extend it to other implementations. Some guys want to add a SlackBot for writing the match results; I want to add achievements and betting, some other guy probably wants to add statistics to know which side is generally better. All of that can easily be integrated by consuming and producing events to the correct topics. Finding those topics can be hard at times, but even for that, there are solutions.
There are some pitfalls and learnings that I think are important:
1. writing complex objects using ksqlDB is challenging. I would personally advise you, for now, to use different sources if you need to write nested objects.
2. it is crucial to keep in mind that event sourcing is asynchronous. Data is not immediately available to the front end after sending it back
3. cyclic dependencies of data can bite you. In our use case, the current state of the player influences the points this player can gain, which then, in turn, influences the scores the player wins. To avoid wrong reprocessing, we added the current state of the players directly within the match event. This solution works perfectly fine for us because realistically, the frequency of match events will never exceed one per five minutes. Your use case might need a different answer.
With these things in mind, I think everyone can benefit from writing a simple use case with Kafka first. You will gain some experience and know-how to solve business problems fast, without sacrificing extendability. Kafka is a tool that is well suited for a lot of situations, not only for ones that need an incredible amount of load. Writing distributed systems with event sourcing leads to a decoupling of use cases. This is one of the major benefits of using Apache Kafka to improve your data culture.
To learn more about the Töggelomat check out my talk on the KafkaSummit or contact us on www.agoora.com. We can also talk with you about other Apache Kafka related solutions.
