Microsoft HoloLens Developer Contest. Ep.3
This last episode will cover the main experience of Music Everywhere:
Augmented Reality Piano Lessons
People might ask:
- “Is Music Everywhere better than a real piano teacher?”
- “Can I make significant improvement with Music Everywhere?”
Or might criticize (Youtube Comments.):
- “It’s not the right way to learn piano!”
- “A piano teacher could never be replaced by this!”
Music Everywhere creates a new path for people who fail at learning piano the traditional way.
Our CEO, Seth, went to Julliard school and has decades of experiences in teaching piano in NYC. All the lesson content and details were designed by him. To our critics, he says,
“Music Everywhere creates a playful and fun environment to help people who have failed with the traditional piano learning way to learn again. We are not replacing real piano teachers. Above all, learning with a good piano teacher is the only way to become a good pianist.”
Our goal is to guide and inspire people on how they can express themselves with playing piano, through improvisation. In our lessons, the virtual musicians teach improv knowledge. Users can follow along with instructions in Following Mode, or experiment with different notes in Improv Mode.
It’s not about learning to play perfectly, it’s more about learning the value of jumping right into playing music.
The Beauty of AR: Projecting Instructions Directly above the Piano’s Keys
What sets Music Everywhere apart from other applications like Guitar Hero, or even Youtube, is the fact that we integrate a real piano with the tech platform. Unlike learning with a computer or mobile screen, Music Everywhere leverages AR technology to project instructions above the keys. There’s no need to keep looking back and forth from a digital screen to the physical keyboard. Everything you need to see is in one place.
Music Everywhere provides a intuitive interactive learning experience unencumbered by extra screens or fake instruments. We called it On Instrument Learning. We plan to expand to other instruments. On Instrument Learning with a piano is just the start.
Lesson Editor: the Heart of Interactive Lesson Design.
Designing an interactive lesson is complicated and time-consuming because you need to combine 3D animation, audio design, UI instructions and user feedback. To help mitigate this potential pipeline disaster, our CTO Byunghwan created a Lesson Editor tool that allowed Seth to craft lessons quickly, through a drag and drop interface. Seth imports material and applies function modules to produce a variety of interactive events on the editor’s linear timeline.
The Level Editor was a game changer for us because it allowed Seth to create lessons more quickly and effectively, saving valuable time in the development pipeline.
Virtual Musician: Blurring the Lines
The virtual musicians were a hit. The way they moved in sync with the audio design and their animations made them come alive. To design these lively characters, I combined my 3D art with Byunghwan’s code and Seth’s audio design. Combining these elements required a great deal of iteration, but we knew it was critical for user immersion. We wanted to make them seem real, as if you could reach out and touch them. We wanted to blur the lines between what was real and what was virtual.
Here were some challenges I had during development:
- In order to build the synced animation with audio (e.g. to make the guitarist plays the chords with the rhythm), I needed to cut the whole animation into smaller pieces and build the complex animation structure in Unity. After this, I needed to test out many times with the scripts.
- I optimized 12 characters from the old prototype. From retopology, style to rig and animation, I spent so much time on fine-tuning them. While I could have easily spent over a month to fine tune each character, time was of the essence. “Good enough” became a guiding principle, keeping me from obsessing too much in the little 3D world.
- Because of multi-jobs(Design plus 3D Art), I needed to transfer my brain from the fully-focus 3D art mode to the flexible designer mode to solve some development problems. It was great to have Byunghwan sitting net to me, as he is a clever game programmer, and usually gave me good ideas and new perspectives to problem solve.
Music Everywhere is Music Learning for the Future
Last, I would like to say, “ Music Everywhere is preparing for the future lives.” Maybe it’s hard to see the value in the current market, but we believe in Augmented Reality. As the AR market matures, Music Everywhere will be there to give new musicians a great place to learn and jam.
Let’s play some music together. Learn to play Music Everywhere.
