Making guitar learning accessible to 1 million users

I joined the team in 2020 when Rocksmith+ was in pre-launch production and Ubisoft wanted a global release on mobile, PC, and next-gen game consoles (PlayStation 5 and Xbox Series X). Very few people on the UX team had prior experience building mobile user experiences and responsive UIs and interactions for multiple platforms.

The MVP for Rocksmith+ was also built only for PC. My goal was to deliver it on Mobile and Tablets in addition to Consoles to invite a whole new audience of guitar learners from around the globe.

The first piece of work started with extensive user testing. I requested internal and external volunteers to play a crude, ported version of the PC game on iPad without any scaling and no interactions except simple taps. My first few initial tests were to discover how people set up a portable device compared to playing on a PC or a TV to understand the context and environment in which Rocksmith+ would be experienced.

The tests gleaned insights using three primary methods:

1. Observational studies and contextual interviews with internal and external volunteers (4–6 waves with panels sized 12–15)
2. Diary Studies to see if and how user needs evolved as users went from beginners to power users (10 users)
3. Surveys after each playtest to track key success signals via ability to learn on mobile as compared to PC, satisfaction of experience, and NPS surveys

Throughout virtual and in-person testing sessions I made notes of various environments users learn guitar in:

1. Placing their phone on various physical contexts ranging from kitchen countertops to dedicated desks
2. Plugging in their guitars into a myriad of equipment — from pedals to headphones so a parent can practice without disturbing the children
3. Placing devices at different distances to have room to play their instruments

Another important highlight was the amount of personalisation needed:

We also included questions related to eyesight. I wanted to know how many users suffered from eyesight issues including near/far sightedness and colour vision deficiency. I personally suffer from near sightedness and colour vision deficiency which impassioned my desire for an accessible UI.

I wanted to know how many contexts the general user stories would have and how many devices a user imagined themselves playing on.

What I quickly realised was that the UI cannot be one size fits all for these many user segments. Users need tons of customisability to change the experience to their needs:

1. The ability to zoom in and zoom out on the guitar lessons based on device distance
2. The ability to scale the UI so that buttons are bigger if the device is farther away
3. Other less frequently used but important controls such as changing field of view

I first began with working on a core responsive UI that would offer the base experience on all devices. It involved not only designing and validating layouts in Figma, but also working closely with UI engineers to build appropriate tools:

1. UI component anchoring and scaling so that UI elements know exactly where to stay on the hundreds of devices (TVs, monitors, tablets, and phones of up to 8 standard aspect ratio configurations)
2. Aspect ratio based scaling rules — percentage based scaling to maintain consistency without too many breakpoints
3. Aspect ratio break point table for bespoke rules for brackets of devices

The tricky part was making 3D objects in the UI responsive. I worked very closely with the UI engineers to develop a custom system where an anchor point and multiple boundary points were projected from the 2D screen into 3D space. The camera adjusted so that the 3D point was placed perfectly where the 2D point was supposed to be on screen.

It would automatically know how much to zoom in or out and how to move left and right to make sure all the 3D objects appeared at the correct places on the screen no matter the aspect ratio and screen resolution. The MVP achieved usable display results on all 5 target aspect ratios encompassing 90% of expected devices out of the box.

Live test demonstrating the 3D-to-2D anchoring system, ensuring all elements remain legible and correctly positioned

User Testing Gallery

Accessibility was a core tenet of the experience. One of the first accessibility features we developed was allowing users to modify the scale of their UI to fine tune their mobile and tablet experience based on how far they kept their device. The distance between user and device changed heavily based on skill level, instrument type, and physical setup.

This customisation system was easy to build as my team had already worked on a dynamic system to make our 3D UI responsive. The original system, being built on solid technical foundations, meant that users could change their configuration at run time — whenever they wanted and however many times they wanted.

Since one subscription of Rocksmith+ gave access on every platform, we expected users to use the app on multiple platforms — think Netflix. We needed consistent outcomes from the same input on different platforms.

From all our extensive user studies, one thing was absolutely clear — users will only have one hand free as they had a guitar in the other. Whether a mouse, a game controller, or a touch screen, the entire interface had to be usable with only 1 hand.

Since users were occupied with guitar chords and memorising complex techniques, it made no sense to add the cognitive load of also learning how to interact with this app on multiple platforms.

I created a 3-dimensional interaction matrix that mapped outcome → component → action on each platform and conducted many surveys to build an exhaustive list of commonly mapped translations between interactions. For example, hovering with a mouse cursor on desktop was majorly translated as touch and hold for mobile users.

The original Rocksmith+ experience was built with only PC in mind, so many components had to undergo changes to be intuitive for mobile and console UI interactions. For example, on PC you scroll through a carousel by clicking arrows. On touch devices, you simply swipe.

A large part of my role was making all these small changes to the global design system and its component libraries to improve intuitiveness on mobile, PC, and consoles.

Another major issue I tackled was menu navigation for consoles. The original prototypes included expanders with nested scrolls to hide secondary information. While common on desktop and mobile, on consoles this created a major issue.

On consoles, you cannot independently focus on a separate item — all directional input is mapped to the item in focus. You can only scroll the expanded container all the way up till you reach the header to close it.

My first gut reaction was to try mapping one stick for the nested container and the other for the root container.

This solution had 2 core problems:

1. Not a natural pattern in video game menu UIs. No mainstream games use two sticks for two UI elements — not intuitive to users
2. One of our main goals was one-hand operation as the other hand holds a guitar. This pattern required 2 hands

Going back to the drawing board, a technical designer proposed using a two-axis design pattern — vertical axis for moving between components, horizontal axis for accessing information. I created multiple quick iterations in Figma to explore commonalities across platforms.

I worked very closely with a Senior Engineer on building consistent focusing in a two-axis interaction — whenever users would vertically scroll, the item closest to the current item in focus would gain focus.

When we specifically asked playtesting moderator "comparing to other main menus such as your console's game library, did you have any issues with the Rocksmith+ main menu?" — 100% of results came in as "no". This was a big success.