Group
Future Work
What we Learned
Improved Instrumental Synthesis
Harmonic Detection and Handling
Chord Detection and Handling
Our project is only capable of synthesizing four different instrumental effects: 8-bit, steel drum, organ, and oboe. Moving forward, we would like to be able to synthesize more instrumental effects, making our project more versatile as a musical synthesizer. Some other instrumental effects could include brass instruments, string instruments, percussion instruments (other than steel drum), and vocal tones.
We can also improve the accuracy of our current instrumental effects. While these effects are reminiscent of their intended instruments, they definitely possess a “digital” quality and sound somewhat unrealistic for certain frequencies. Improving the performance of these effects would require experimentation with more advanced synthesis techniques and research into the behavior of the sound waves produced by these instruments.
A limitation of our current audio file to frequency conversion algorithm is its ability to detect and handle harmonics of the fundamental frequency for some notes. Our algorithm essentially uses the spectrogram of an audio file to find the frequency with the maximum power for each specified time interval. This approach generally works well; however, there are some notes, particularly those with low fundamental frequencies, for which the fundamental frequency has a power less than that of its harmonics. We attempted to handle this issue by applying a bandpass filter to the audio file to hopefully remove harmonics of the fundamental frequencies. The bandpass filter improved the performance of the algorithm, but it does not completely resolve the issue. To completely fix this issue in the future, we would like to be able to detect all non-negligible harmonics of the fundamental frequency (harmonics with power above a certain threshold) and their respective magnitudes for each time interval. This information would be stored in a matrix. With this information, we could play a weighted sum of cosines at the harmonic frequencies to more accurately replicate the note from the original audio. Using this approach, we could also separate any given number of harmonics, including the fundamental frequency, from the rest of the detected frequencies. The first harmonic (fundamental frequency) would be the lowest frequency of non-negligible power, the second harmonic would be the next lowest frequency, and so on.
Another limitation of our current audio file to frequency conversion algorithm is that it assumes only a single note is played during a time interval. This is clearly an unrealistic assumption, as most songs use chords where multiple notes are played simultaneously. Detecting and handling chords would require a more complex algorithm than what we used. Similar to our proposed method for handling harmonics, we could adjust our algorithm to detect every frequency with a non-negligible power and its magnitude at each time interval. We would store the frequencies and their respective magnitudes in a matrix. An example of how we could organize said matrix is to assign the first two columns to the first time interval, the second two to the second interval, and so on. Within each time interval, the recovered frequencies would be in different rows of the first column and their respective powers would be in the same row of the second column. Using this approach, we could play chords as a weighted sum of cosines at the recovered frequencies. With these adjustments, we could also identify which chords are being played based on the combination of notes that are identified.
Through this project, we learned how to utilize a spectrogram to analyze and process a digital audio signal. This enabled us to extract the fundamental notes and the times of each note for audio files. We learned about audio synthesis techniques such as additive synthesis, digital attenuation, and frequency modulation (FM) synthesis which are all frequently used in audio effects and digital audio processing. We also learned about harmonics and overtones and how to handle these harmonics/overtones using an equiripple bandpass filter.
Chord, as shown on musical staff.
First through seventh harmonic of standing sine wave