Making music requires a closed feedback loop to evaluate the created with the intended sound. This becomes a challenging task for those with hearing disabilities who are nevertheless interested in learning to play an instrument. Due to limited access to the auditory channel, deaf people have less information to evaluate their performance. The result is a feedback loop-gap.
We developed Music Sensory Substitution (MuSS) Bits, small wearable plug-and-play sensor-display pairs that communicate wirelessly peer-to-peer. The Sensory-Bit captures real-world sounds from various audio sources, extracts rhythm information and sends it to the Display-Bit which provides a visual and vibrotactile representation of the rhythm. The vibrotactile intensity and visual brightness can be adjusted to the user’s need. The decoupling of the display from the sensor part of the sensory-substitution system allows the performer to spatially deploy the Display-Bit, for example on his or her body, provides free limb movement and permits a simple way of selecting the desired audio source. Real-time feedback has been established through onboard processing, nRF communication, and responsive vibration motors.
We believe that MuSS-Bits are a first step towards customizable exploratory music-sensory-substitution systems for deaf people.
