Yair Lavi is Founder and CEO of Tonara, the company that designed an app which helps musicians navigate digital sheet music and allows them to intelligently learn, rehearse and perform with score-following technology and a suite of other interactive features. As a pianist and experienced algorithm designer and researcher in the machine learning and image processing fields, Yair decided to combine his love of technology and passion for music while working on his Ph.D. Excited by the prospect of a technology that would listen to him while he played sheet music, transcribe the notes and turn the pages for him, he began to research possibilities alongside his friend, computer scientist and future Tonara co-Founder Evgeni Begelfor to see if they could create a tool that would process polyphonic input. After several months of research and testing, they came up with the ideas behind the Tonara iPad app, which is bolstered by a music-recognition algorithm that allows the device to listen to musicians as they play, show them where they are in a score and turn the pages automatically in real time, even when they change tempo or make mistakes. Yair also holds a BSc. with honors in Mathematics and Physics from the Hebrew University and an MSc. in Physics from Tel-Aviv University. Tonara celebrated its flagship product’s first year on September 12, 2012.
Yair talked to me about how he came up with the technology behind Tonara, how it helps musicians learn and grow and some new products and tools his company plans to launch in the coming months. He also shared his predictions about where digital sheet music is headed in the next decade.
Thanks so much for taking some time to talk, Yair. How did you get involved with music and technology?
I started from a technology background. I studied physics and mathematics, and then my Ph.D studies were a combination of physics and computer science. While I’m not a professional pianist, I’ve always played piano for fun. At one point, I thought about how cool it would be if there was a technology that listened to me while I played, transcribed the notes I was playing, and turned the pages for me. I began to study some basic things about music recognition technology, and discovered much to my surprise that there was nothing really advanced on the market. There were a lot of theories out there about the subject, but I didn’t find a lot of products in the industry that worked well. Some applications could listen to monophonic instruments and provide feedback, but no one could translate polyphonic input.
This was when I decided to talk to my friend, who is now Tonara’s co-founder, Evgeni Begelfor. He’s a computer genius, and we started to think it would be possible to create a product that worked for polyphonic input. We were working on our Ph.D.s at the time, and on the weekends, began to develop Tonara. After several months of work, we started to see some nice results and began to demonstrate it to several companies in the music industry. We came to the conclusion that people were very enthusiastic about the possibility, and it had enough potential for us to focus on Tonara full time.
We’ve been working to perfect the app technology for the past several years. Once we had a strong working product, we were able to secure a seed investment, and today the product has been in the market for one year and gaining strong momentum everyday. We feel our one-year anniversary is an important milestone.
Yes. A year in the app world is like 15 years in the regular business world, too, because everything changes so quickly around you.
And how does Tonara work?
The product we launched a year ago really functions as an interactive electronic sheet music notebook. We look at what we’re doing for the sheet music industry like what the Kindle did for the book industry, and iTunes did for recorded music.
So, when you play into a device, Tonara recognizes exactly where you are in the music?
Exactly. But let me first take a step back. Tonara brings digitization and interactivity to the sheet music industry. You can organize all of your scores in one portable device instead of carrying stacks of books and single sheets of paper around. If you need or want a new song, you can quickly purchase it with one click.
On top of that, we have advanced interactive technology that is unique to Tonara. There are a few other applications out there that provide sheet music in different genres, but none of them intelligently listen to the person playing, indicate their position and tempo, and turn the pages at exactly the right times. Our technology is based on a polyphonic score-following technology, and is designed to locate a position in a score in real time, whether it’s from one person playing an acoustic instrument or several people playing select instruments.
Very soon, we plan to launch new features for music accompaniment. It will be like a smarter version of Music Minus One, because it will be able to adjust to your tempo even in the middle of a performance. It’s designed to be like true human accompaniment.
I’m a bass player, which is how I first got into the music industry. Let’s say I wanted to try to play piano, which I’m miserable at playing. How accurate do you have to be as a musician for the system to recognize you?
It’s hard to quantify in exact percentages. But in the tests and demonstrations we’ve done, we’ve discovered you can be a true beginner, and the software would still be able to follow you very precisely. It’s very tolerant to mistakes.
That’s great. So, you’ve basically built a pretty non-judgmental app.
That’s exactly what we tried to do. If a person plays something that’s not great but otherwise recognizable, the software will be able to follow. We encourage people to test out the app themselves. If you give the app to someone who is a total beginner and then give it to someone who is an accomplished musician, you’ll see that it will really follow both people well.
Tonara is robust enough to handle both mistakes and tempo changes. Musicians actually use tempo changes quite often to express emotions in the performance, so that flexibility is really important. The app is also very tolerant to external noise. So, if a person is playing the melody, someone else could make noises or play a different melody in the same room, and it would still follow the performance.
That’s really interesting. It can sort out music from background noise or other background music.
Tonara is an excellent flagship product and first company. It’s surprising to me that it has this many capabilities and is still in its development phase. How do you plan to grow the company and use this technology in the future?
The Tonara product is not only a fun tool, but also has a lot of educational properties, which you and I haven’t discussed yet. We are constantly improving it and adding new features based on our own work and on feedback from users.
We also have some other products coming out soon. Right now we support all instrumentalists, except guitar players who play using chords and percussionists. In 2013, we plan to launch a product for guitarists that will show them chords in letter form and use tabs with the lyrics. It will follow them in the same way that the current app will follow pianists and violinists, etc.
We also see an opportunity for singers and plan to launch another application we call “Smart Karaoke,” which adjusts the playback of the song to the singer’s tempo. A lot of times in karaoke clubs, the singer will fall behind the song, and it sounds really terrible. Because we can locate the performer’s exact location in the melody, we can adjust the playback position and speed so it always stays with the singer. The product will also be able to adjust the key. So, if the singer starts singing in the wrong key, we’ll be able to change the key to match the singer. That way, everything sounds perfect. This will make for some better Karaoke experiences!
I’m pretty evangelical about my belief that surviving as a musician involves being musically literate, although there’s a lot of musical illiteracy out there, especially among rock musicians and hobbyists. But it sounds like this technology could really be helpful in helping people learn to read music and get better at their instruments. Is this true?
Tonara does offer a great platform for learning to read music, and we’ll continue to enhance this in 2013. Because we know a person’s position in the score, we can hide the current measure they’re playing, which would force them to read ahead. This is how teachers teach students to sight read, by hiding notes with their fingers, so the student is forced to read a couple measures ahead at all times.
We’ll also provide educational feedback about the performance. After the piece is performed, we’ll color the different measures according to how well they were played. If some measures were played very poorly, we can color them red, and the other measures green. We’ll then indicate the measures the performer needs to practice. Once the measures are performed perfectly, we’ll let the person play the next level piece. Our objective is to provide an automatic monitoring system for students that will motivate them to improve, so they can play more difficult pieces.
Ultimately, we want to make sure music students can easily share everything with teachers, as well as give their teachers the actual recordings of the practice sessions. One important point to make is we’re in no way trying to eliminate teachers, because we could never provide feedback for non-technical components of the performance. For instance, we can’t gauge emotion and artistic expression; we only provide feedback on the tempo, duration and pitch.
If my assumption is correct, part of what you were asking is also if we’ll be able to use the technology to do automatic transcription. For example, can we transcribe the notes being playing for polyphonic performances? This actually relates to what I thought about when I first started studying music recognition technology. I was composing, and I wanted something that would write what I played. This is actually extremely difficult for polyphonic instruments, and many people who have tried unsuccessfully to do it. Because music has harmony and melody, you play many different notes at the same time, with a lot of overlapping frequencies – this is what makes music sound beautiful to the human ear. But from the computer’s point of view, it’s a nightmare, because the computer has to differentiate between different sources that have a lot of overlapping frequencies. Our brains are actually very good at pattern recognition; it’s one of the only things our brains can do better than computers at this point.
We don’t intend to work on automatic transcription within the next five years or so, because technology is not there yet.
You have a really interesting world view, because you are somebody who has studied so much about audio recognition technologies. And you’ve given me a great sense of your own product. Where do you see audio recognition technology going, above and beyond your product in the future?
What we’ve seen from our own users is that they like the technology a lot, and couldn’t go back to using regular sheet music, without automatic page turning and the marker that follows along with them.
We actually also provide a lot of other things that are not totally unique to our technology, but stem from the fact that we digitize the scores. For instance, every time a user plays a piece, we automatically record it, and offer the user an automatic playback button. Right now, if you play from paper sheet music, you could technically open your computer at the same time, plug in the microphone and record using recording software. Then, after you finish playing, you could press the button and listen to how you did. But obviously, not a lot of people do it. With a product like Tonara, you get recorded by the same program that is allowing you to see the sheet music.
Music recognition gives musicians the option to listen to their practice and performance sessions right away, which they’ve not been able to do previously. And, a product like Tonara lets them see the marker over the score, so they can see how they did in exact positions in the score and even navigate and jump ahead or turn back in the score and have the marker follow them as they go.
Another thing that goes along with digitizing sheet music is allowing users to annotate in layers. Obviously, on regular sheet music, you can write down notations. But when you digitize sheet music, you can write notes in different layers, so you can decide which layers you want to see depending on the situation. For example, maybe you’re playing in an orchestra, and you have annotations on your music from your private teacher that aren’t relevant to the performance. So, you can hide these annotations and then make them visible again in a different situation.
This feature also allows users to share annotations. So, if you’re a teacher and you have 20 students that are learning the same piece, you only have to write your annotations once on your own iPad, then email them to all your students, who will get it automatically through the application.
There are a lot of features like these that compliment our advanced technology. I’m pretty sure that within three, to five years, the majority of scores that are sold and played will be in digital formats. We’re working very hard to get the Tonara experience as close to perfect as possible so musicians use the application for this. That’s the same thing that happened with Kindle. These days, Kindle books sell much more copies than paper books. There’s no reason to believe the same thing won’t happen within the music industry.
I didn’t realize that sales of digital books had surpassed sales of paper books. But I’m sure music publishers who are making money off sheet music are excited about that prospect.
It began to happen about a year ago. And now Amazon is selling more digital books than regular books, and digital books are selling more than regular books across the whole industry. The evolution happened really quickly. The young generation especially has really connected with them.
In the broader scope of technology, I think some really amazing things will emerge in the next five, to ten years. The polyphonic transcription, which has, as I mentioned, been very difficult to figure out, will get much better. Machine learning – which is the new thing in artificial intelligence – really accelerates everyday. As I mentioned, the human brain is better at pattern recognition than computers, but it won’t necessarily stay that way forever. Machine learning has really become quite highly advanced and already enables computers to do face and speech recognition. And it will be able to do automatic transcription at some point.
This is all really amazing to me, and I believe its value to any musician or composer is obvious. Imagine just being able to compose something on your piano, and never needing to go through the process of writing it down. But also, the composer or musician will be able to input an audio recording and have the software automatically transcribe it. Then they would be able to give all the other musicians they’re playing with all the parts. People will be able to quickly learn how to play every song. These are exciting times!
To learn more about Yair Lavi, his company and the free Tonara app, which is available through the App store, visit the official Tonara website.