Algorithm

To the right, we can see a block diagram of our system. The main idea, derived from our original motivation to pursue this project, is to take in any piano song and transform it into something that may have been played from any instrument. To do so, we decided to “chunk up” or “sample” our original audio file, using an algorithm that we developed which lines up each sample into a fixed length, or keypress. From this, we can tell what the most prominent frequency is being played for that particular sample, which is deciphered from the spectrogram, showing a “plot”, essentially, of the sample versus the frequency power, highlighting the frequency containing the most power. Now that we have the most prominent frequency, we can take that array and turn it into something that is matched to theoretically, any instrument. From there, we are done, aside from some filtering and normalization features that we add on to make the audio cleaner to the human ear.

General Procedure

When taking in an audio file, we are simply given the amplitude of each sample from some original audio file. This varies with each audio sample that we take in so we must normalize the audio signal, which is done at the end of our program. At the beginning of our algorithm, we are most interested in the actual frequencies that are being played at some point in time, or at some sample in time. Spectrograms heavily aid us because we only want the frequency at a point in time, as that is what we recover at the end of our conversion process. However, instead of just taking the spectrogram itself and reading it into something that lines up with real frequencies, we noticed that several overtones existed at higher frequencies. To get rid of these overtones, we applied a bandpass filter to the audio signal in order to limit the spectrogram to a fixed set of frequencies that our algorithm could easily handle. We chose an equiripple, bandpass filter, with relatively low taps in order to reduce the computational energy required to filter out these frequencies.

Once we convert our original audio file into a spectrogram containing information about the frequencies at different points in time, we can take the most prominent frequency that is being played for each sample in time. We do this with the max() function in MATLAB, which allows us to save each maximum value from the spectrogram into an array. We can then compare the values in this array to actual notes that we would have on a piano. This process “matches'' each sample to its particular note that we expect to hear out of an instrument However, there is not a corresponding piano note for every frequency we recover, so we embed a lookup table of all frequency values that can be played on a grand piano. This allows us to line up all our guesses (maximum values from the spectrogram) and use the lookup table to match each guess to the closest value that could be played on a piano.

Audio File to Frequency Conversion

Frequency Conversion to Instrument Matching

We now have an array of frequency values that we hope to play. Now, we need to convert this into something that is actually audible. We accomplished this task by first studying the different waveforms that were expected of different instruments, and matched up these expected waveforms with different sinusoids. For example, the pulse shaping of an organ is made up of the product of sinusoids of varying amplitudes that all have different frequencies. This procedure is described as frequency modulation synthesis. We also use a process called additive synthesis, why we add different sinusoids to produce different audio effects. The steel drum sound is made up of a decaying exponential multiplied by a sinusoid of the desired frequency, which produces a spiked sinusoid. This spike is what we expect to hear out of a steel drum.

These shapes are produced by going through the recovered notes, matched to the piano note lookup table, and inputting these frequencies into a cosine. At the simplest level, when developing an 8-bit sound, all we need to do is to line up these frequencies to cosines, and change the frequency of that cosine for each sample, and then we are done. When we want different instruments, we use the procedure described above in order to obtain a more dynamic sound. From there, we have a playable array of cosine. Because the amplitudes of each audio sample varies greatly, we normalize this array of cosines. Finally, we subtract out the DC term bias that is introduced through the use of the Discrete Fourier Transform to produce a more crisp sound.

A filtered spectrogram of "Happy Birtday". Notice how our bandpass filter attenuates frequencies higher than ~600 Hz.

An unfiltered spectrogram of "Harry Potter Theme". Notice how all frequencies are allowed without a filter.