Microsoft HoloLens Developer Contest. Ep.2
This episode will cover Phase 2 of the contest, and mainly focus on the Settings design of the HoloLens with a digital piano.
After our initial application, Microsoft announced we were among its top 10 finalist submissions. This meant we were advancing to Phase 2, for another 4 months of development before final submission to the HoloLens Store. We were excited, but soon filled with worry. We knew a significant challenge for us in Phase 2 would be:
How can the selection jury properly test our application without a piano?
Well, the short answer was, they couldn’t. We needed to send them a digital piano. Our app uses a Bluetooth connection to send out a MIDI signal from the digital piano to the HoloLens. In short, for the HoloLens to talk to the piano, it needed a shared language. We decided to send a piano to the jury for the final submission. Now for another problem: What kind to send?
88 keys? 76 keys? 61 keys? 49 keys?… 25 keys? We knew we needed to design for different sizes. To learn more about piano specs, especially the key sizes, and the piano market overall, we went to Guitar Center to gather data. This allowed us to account for a variety of Bluetooth keyboard options. (We ended up sending them a 61 key model.)
Now that we got the piano question out of the way, how do we make sure the user can set it up well? Check alignment? Troubleshoot problems?
In looking back on the bulk of our playtesting, our demo sessions at CMU, GDC , NIME, and Siggraph, virtual alignment and settings were always completed by a member of our team. It relied on instinct and guessing, a truly mysterious endeavor. The keyboard’s position would often shift in the air, requiring the user to hunt for its position. More than once, we’d find it floating around in another room.
How can we make sure the HoloLens successfully connects with the piano? What happens if this connection fails, or the virtual keyboard is not aligned well? What’s the user experience if the keyboard starts to drift? With our prototype, the answer was to let a team member reset it manually for the user. We needed a better way.
In order to design a settings process that accounted for a variety of potential setup issues, in order to make it user friendly, we needed to go back to the start. We needed to think about it from all different angles.
Necessary Evil: Long setting time.
In order to ensure a smooth setup, we needed account for:
- How to lead the user’s vision within the limited field of view(FOV) to see the whole virtual graphic without being cut off.
- Connecting the HoloLens with the digital piano via Bluetooth.
- The user’s digital piano size.
- Generate, align, and lock the virtual keyboard on the real piano.
- Solve a Bluetooth disconnection.
- Re-align the piano when it drifts away.
As you can feel reading the list, the settings process is pretty long. Some concerns for this might be that the longer the process, the more complicated it will be. The user experience can’t be good if it’s too long. But we felt:
A long settings process doesn’t mean it’s complicated. If each step has a clear purpose and intuitive instructions, the user experience (UX) won’t be bad.
Unfortunately, we had a false start. Enter the Roadies.
Throughout our user testing, the virtual characters, the musicians and singers, consistently reinforced user engagement. They jumped, they played with you, they danced with their little dance moves. “They are so cute,” was common feedback. The team thought, what if we used virtual characters to guide the user through setup? This made sense on the surface. We could bring consistency throughout the experience. The characters would be appealing, and in theme with the Music Everywhere world. Our solution seemed simple: The Roadies would be our setup characters.
Complete with propeller hats that help them zip around the air, the Roadies helped and encouraged users through the tedious settings process. The interactions forms included VO, talking bubbles and blackboard instructions to communicate information on different levels.
- Voice Over: One-time information, like normal conversations which are ok to be missed.
- Talking Bubble: The important information hints that appear on the top of the Roadies. It stays until advancing to the next step.
- Blackboard: Permanent information panel, which is used to show the big amount and sequential of important information.
So… after spending almost 2 months on designing the characters, their interaction styles, animations and everything else that comes with character design, we scrapped it. It simply required too much development time to get right, and time was something we did not have. Remember, the Phase 2 clock was ticking.
The main problems we encountered through the Roadies were:
- People have strong opinions about characters, including how the voice sounds, how the characters act, what the characters say. By giving them personality, we unintentionally invited the users to be irritated by that personality. Clippy from Microsoft Office comes to mind…
- Implementing virtual characters is not an efficient way to use the limited FOV since Roadies occupy a lot of space. For example: In the blackboard mode, it’s challenging to show Roadies, talking bubble, and the message on the board within the FOV.
- Essentially, the Roadies idea is an delicate interactive animation project. We need more time to craft and test in order to combine the audio, animation and setting process properly which is truly challenging to be done by a team of 3.
I think the main issue at the core of this problem was:
Time spent developing the settings process, was time away from developing the main experience.
Settings set up should be: Neutral, Simple, Accurate.
Are the settings part of the main experience, or separate from it?
If the settings procedure is a person, what is this person like? Probably someone who is very helpful, not opinionated, and says yes to every user’s demand. If the settings procedure is a person, this is an incredibly helpful person who has no personality. Settings should be neutral.
In the end, we designed a linear setting process with simple graphics and instructions which guild users how to set up step by step.
One last design case study I’d like to share:
At the start of our design process for the settings setup, we needed to lead user gaze to lock on the settings panel in the designated area: 6–8 inches above the piano’s center. This allows users to see the virtual information and real piano keys in the same FOV. If the instructions are too high, it’s out of FOV. If they are too low, it blocks the real keyboard. We needed it just right.
Initially, we asked users to Look Straight or Look Forward. But we encountered issues as people come in different heights, hold their heads differently, consider the terms straight or forward differently. It still wasn’t right.
I tried to add measurements to this instruction, but learned that adding numbers gave people added anxiety and pressure. This was proving to be quite a headache to solve. I started to ask around our office:
How do you measure 6–8 inches? How would someone eyeball that distance?
I had hard time trying to solve this problem until I noticed that in the States, people often used the gesture six (the distance between your thumb and your pinky) to measure things because the distance is around 6 inches. This observation ended up being both intuitive and accurate.
In addition to this, we created an intro animation to give the user a clear picture of Music Everywhere, and cultivate some setting background information.
To design for the HoloLens UX, being pessimistic is a good attitude.
“What if our design totally fails, or for people who don’t have the Bluetooth compatible digital piano?” Byunghwan asked. Thanks to him, we design the “No-Piano Demo”option for people who can’t get the settings to work, or without a proper piano. Through this demo, people can get a preliminary understanding for how Music Everywhere works. It also gave us a backup plan in case the contest jury couldn’t connect the Bluetooth piano.
From this pessimistic perspective, we developed a full FAQ section. Here, users could troubleshoot by themselves with guided instructions, and get to know more details about Music Everywhere.
Developing for the HoloLens, we’ve encountered many surprises. Between navigating the foundation UX of AR, integrating with other hardware, working around a limited FOV, experience stability is not guaranteed. Thinking pessimistically drove us to discover some fundamental problems we ignored earlier in our process, led us to optimize more playing conditions.
