Context

My Role : Product Owner / Product Manager

Team Size : 1 UI Designer | 1 UI Engineer | 1 Full Stack Engineer | 1 Junior Game / Product Designer

Guild of Guardians is a hardcore RPG game where users create teams of 5 guardians to duke it out against monsters, enemies, and other players. In the first 7 to 10 days, the users are expected to play the Adventure mode where they explore the story, learn about the characters, and fight some easy monsters to learn about the game. They need to earn gold to strengthen their equipment and collect souls to level up their heroes.

We had a problem, our D7 retention, the percent of players still continuing to play by the 7th day since they installed, was low. Which means we lost players who we spend lots of money to market to and acquire which reduced the efficiency of our marketing budget.

Action

Being a live product, looking at data is always my first step. D7 retention is usually an indicator if players had a wow experience on their first day and if the excitement of the wow stuck. It is an objective measure of subjective fun.

A good sub-indicator of retention is to look at engagement, high engagement (many hours played) and low retention may indicate players had lots of fun at the start and then either forgot your game or got burned out. Low engagement means they simply did not enjoy it and chose not to come back.

A good stand in for habitual behaviour is to look at sessions per day, how many unique times did a user login to the game. Logging in daily means your game was both - "wow" and forming habits. Sometimes a good, fun game fails to build habits and gets forgotten because humans have access to so much entertainment.

The average sessions per day in a user's first 7 days from install was 1.3. That's good! But is it? An average does not often tell the full story. Instead, this was the full story.

There were two segments of users, some who played a lot, more than 1.3 times per day, and some who played less than 1.3. So if you're playing an average of less than 1, that means you're not playing everyday. That's not forming any habits. In nerd terms, this is a bimodal distribution and very few players are actually having the "average" experience, and those below the average where definitely not sticking around.

I ran a logistic regression test on D7 retention as the dependent binary variable and sessions per day as the independent continuous variable. Basically, I want to see how much session/day explain D7 retention as these are session per day till the 6th day. Turns out it only explain 24% of the variance in the model, but the result is significant and the odds ratio explains that every 1 unit increase in average sessions/day would increase probability of player retaining by 16%. This definitely proves that something about logging in everyday made players retain to D7. Pretty obvious when you say it out aloud but you always have to be sure about that relationship.

In a world where I had more time, I would have loved to create a multivariable regression model exploring the relationship of D7 retention to session per day, activation days, activation days density, and game inventory linked to account level i.e. did player progress enough or feel accomplished enough to want to return the next day. Alas, for another challenge.

The averages are different and it explains correlation. But how do I know if the problem is with session per day and not with overall ability of the game to delight? Well obviously I have a hunch because I play and develop video games for a living. In Guild of Guardians, you get gold for playing. And you can use that gold to get stronger swords and sturdier shields to defeat bigger monsters. What's the catch? If you don't login in enough times, you can get as much gold as you're supposed to. So you're weaker, the boss is more difficult because your weapons are rustier. Let's actually ascertain that by talking to the players

I asked the community manager for permission and pasted the following question in Discord

And came in hundreds of responses

There was a question about how the game fits in people's lifestyle. Due to consent reasons I cannot post answers, but interestingly the some recurring themes were

1. 1. "When I am picking up my child and waiting in the car"
   2. "When I'm on the toilet"
   3. "When I'm in my commute"

This meant that the game was finding place in a pretty crowded lifestyle, and had to take the shape that the users needed for it to work for them. Changing their behaviour to do what we wanted was going to be futile. Additionally, survivor bias had to be eliminated as people who churned did not answer these surveys, so a buffer was added to not overestimate how much users claimed to engage.

So comes the solution

At the current state of the game, you needed to spend 1 energy to defeat monsters automatically and automatically get gold. Given how we designed the product, a user had to login 3 times a day to get the most rewards. We just saw with data and opinions that such a cadence does not fit in the lifestyle of most of our players. And if they did not login 3 times a day, they will never get strong enough to win successfully.

So we had to do two things, build habits, and build them within a busy lifestyle.

In the original flow, users had to come back to take an action and be rewarded. This did not generate any anticipation excitement at the end of your previous session for you to not forget this game and want to come back.

So we simply moved the fantasy in such a way that at the end of your session, we just let users know that rewards are accumulating and they should come back before these rewards overflow and spill out

While to the user the change was as simple not needing energy by rather having things collected for you, under the hood we changed a lot of numbers to make sure that when players came back, they had just the exact amount of gold for that new shiny sword. This created a satisfying moment upon return to immediately then get a sword and go kill a bigger bad guy. This was a pretty invisible but highly impactful, under the hood change to most of the noob players that we were building this for to make their experience smoother and kinder.

Result

We saw our D7 retention increase by nearly 20% within the first week. The bimodal distribution became a normal distribution, and we lost fewer users to this issue.

Learning

While there were many visual iterations and the golden path was updated many times in testing, I was most proud of being able to highlight that many user needs are solved by out of the box (screen) thinking. The screen is just an abstraction, have we really checked their lives and habits to see if our products are failing in ways that we can't see? Solving real user needs requires looking past the screen and understanding the user.

Josiah Wallace, Senior Game Design Manager

"Data driven design is now a keystone in modern game development and there are few others like Bramha who combine the Game Design know-how with the Data Analysis process & procedure as well as he does. The decisions, features, and projects he stands behind always produce measurable results, drive revenue growth, and increase payer retention."

 

Daniel Paez, Executive Producer

"Bramha was always able to breakdown each design decision into its core target audiences and their motivations, helping cut through the franticness and rash decision-making and driving towards elegant, effective solutions."