skip to content
Tushar's Avatar Tushar Tripathi

Computer keyboard as MIDI Piano Input over Wifi

/ 4 min read

programming , music

So a few weeks back, I got interested in music again and started the Duolingo’s music course which focuses mainly on reading sheet music. I also tried a few other apps on the PlayStore for practicing sheet music. There was one problem, I didn’t have an actual piano to practice with. So I was using the inbuilt Piano within those apps.

And… it’s not the best experience. Misclicks were frequent with the touch interface, especially with poorly designed keyboard like below, with random gaps between keys which throwing off your muscle memory. You have to pay attention to where are you pressing in addition to figuring out what the current note is.

Duolingo's Piano

This was a bad experience, but I wasn’t yet ready to buy a piano . I didn’t want to spend money on something which I wasn’t sure I would be committed to. So I looked for if I could somehow use my laptop’s keyboard as an interface to these apps. It’s more than enough for the beginner stage with the top keyboard row(QWERTY...]\) covering almost 2 octaves. I didn’t find a direct way to do what I wanted, so I ended up writing a python script which can listen to keyboard events and pass them to my phone.

The complete source code is available here. A few concepts and libraries I needed -

  • MIDI - a standard to transmit and store music. It’s a binary format, with a quite elegant and logical structure. It contains the instruments to be played, time signatures, tempo, timestamps for when notes are pressed/released, and the velocity with which the notes are hit. For real time use cases, there are two main events which need to be sent -
# note on when a key is pressed, velocity decides the
# volume, how heavily the piano keys were pressed
note_on(note_name, velocity)


# when the key is released
# there can be multiple pressed keys in parallel
note_off(note_name)
  • RTP MIDI - It’s a protocol introduced by Apple to transport MIDI messages within RTP packets(usually over UDP for low latency). In simple words, the piano can send what is being played to your mobile over wifi. The communication happens over UDP and there is logic for error recovery, retransmission of lost packets etc. There are two counter parts -
    • RTP server - in our case, this is the mobile device, it will be listening to incoming RTP packets from RTP client(usually musical instruments). The server has two UDP ports open, a control port(for connection establishment etc.) and a data port(for sending the actual MIDI messages).
    • RTP Client - in our case this is the computer where the script would be running. The script pretends to be a piano sending RTP packets corresponding to keys pressed.
  • PyMidi - Python library which wraps over the RTP MIDI protocol. The current released version only had support for acting as a RTP server, so I had to use the main branch directly.
  • Sound Font - Similar to font files for text, but for sound. They’re used to play MIDI files or just plain notes. They’re based on high quality recording of music instruments. There is a big list of sound fonts here.
  • TinySoundFont - A synthesizer library which can understand SoundFont files and play the requested note with passed in velocity.
  • PyGame - It’s a big library aimed at game developed in Python. I’ve used it for two things -

The overall flow is straightforward, listen to keyboard events and do these three things on each key press/release -

  • update the GUI to highlight the current pressed keys(there could be many keys pressed at the same time)
  • play/stop that sound playing via the SoundFont file
  • send on/off signal to the RTP MIDI server(mobile app)

Here is a small demo(the tune being played is from Ode to Joy) -

Github Source.

Current Status

Eventually it turned out I did like playing enough to buy an actual digital piano with built in MIDI support. Using laptop’s keyboard was becoming quite clumsy as I advanced further with accidentals coming into picture.