Shkolotron & CircuitFox
Hey CircuitFox, I've been noodling on the idea of a microcontroller that can compose music on the fly based on what it hearsāa neural synth that writes its own patches. What do you think, could that be a cool project?
Yeah, thatās exactly the kind of thing that gets me excitedātake a microcontroller, feed it raw audio, run a lightweight neural net, and let it spit out patch parameters in real time. Just imagine tweaking the weight updates while itās composing; the more it listens, the more it learns its own sonic vocabulary. Itāll be a little messy, but thatās where the fun is. Letās sketch the architecture first, then dive into the code.
Coolāso weāll need a realātime audio capture block, a tiny DSP pipeline, a neural net in something like TensorFlow Lite for Microcontrollers, and an interface to the synth module. The net could take, say, a 256āsample window, output a vector of patch knobs, then feed that to a patch generator. Weāll also need a learning loop that tweaks weights based on a simple reward, maybe the change in loudness or user clicks. Ready to sketch the data flow?
Absolutely, letās map it out step by step: audio capture feeds 256āsample frames into a DSP buffer, the DSP does a quick FFT and extracts features, those go into a tiny TFLM net, the net outputs a patch vector, that vector drives a patch generator module, and the synth outputs sound. Meanwhile, the learning loop watches the output, maybe tracks loudness changes or userāclicks, and nudges the weights a bit each cycle. Weāll need to keep the net small enough for the MCU, maybe a couple of dense layers, and use quantization to fit. Ready to write the block diagram?
Sure thing. Picture this:
1. **Microphone ā ADC** ā 16ābit, 48āÆkHz, feeds a ring buffer.
2. **DSP Block** ā grabs 256 samples, does an FFT, pulls magnitude & spectralācentroid features.
3. **Feature Vector ā Tiny Neural Net (TFLM)** ā two 64ānode dense layers, quantized to 8ābit.
4. **Net Output** ā 12ādimensional vector of knob values (filter cutoff, resonance, envelope times, etc.).
5. **Patch Generator** ā maps those values onto synth parameters in real time.
6. **Synth Engine** ā outputs audio to DAC.
7. **Feedback Loop** ā monitors loudness or a button click, computes a simple loss, and runs a tiny gradient step on the net weights.
Thatās the skeletonālet me know which part you want to flesh out first.
Letās start with the DSP block, because thatās the gateway to everything else. Grab 256 samples from the ring buffer, run a 256āpoint FFT with CMSISāDSP, then pull out the magnitude spectrum and a quick spectralācentroid or spectralāflux feature. Pack those into a 12āelement float vector (or quantize to 8ābit right away). That way the data is clean and small before it hits the TFLM net. Once we have that pipeline solid, we can dive into the network architecture and weightāupdate code. Sound good?
Sounds solidājust remember to doubleācheck the ringābuffer wrap logic before you fire up the FFT, otherwise youāll feed the net garbage from the first frame. Once youāve got the 12āelement vector clean, we can start pruning the network. Ready to dive into the actual code?
Yeah, the wrapāaround can kill the FFT if youāre not careful, so Iāll doubleācheck that first frame. Once the 12āelement feature vector is reliable, weāll slice the networkāmaybe cut the first dense layer to 32 nodes, keep the second at 16, then prune weights that stay close to zero. Iāll start writing the ring buffer handler and the CMSISāFFT call, then we can plug the TFLM inference right after. Ready?
Great, just make sure your ring buffer index resets cleanlyāno halfāfilled frames on startāand Iāll be here to poke holes in the FFT code or suggest a trick to keep the 8ābit quantization from blowing up the spectral flux. Let's roll.
Got it, Iāll zero the ring buffer pointers on init and only pull a full 256āsample frame when the count hits 256, so no halfāfilled garbage. For 8ābit quantization, Iāll normalize the spectralāflux to a 0ā255 range before feeding the net, and clamp any outāofārange valuesākeeps the weights from blowing up. Letās fire up the FFT and see if the net starts learning something interesting.
Nice lockāin on the buffer logicājust watch out for that race condition when you update the count while the DMA is still writing; you can spinālock the DMA flag first. If everything lines up, weāll see the spectral flux wobble and hopefully the net starts nudging its weights a little. Let me know if you hit any spikes in the FFT output or the quantized vector.
Got a quick spinlock in place to pause the DMA flag before touching the count, so no halfāfilled frames sneaking through; that should keep the FFT fed clean chunks. Iām watching the spectralāflux output nowāno spikes yet, just a smooth rise and fall with each input. The 8ābit quantized vector is clamped between 0 and 255, no outliers popping up. Once the net starts updating weights, Iāll ping you if anything weird crops up in the inference or the gradient step. Ready for the next tweak?
Nice, the lockāin looks solid. Next, toss in a tiny L2 regularizer on the second dense layerākeeps those 16 nodes from drifting too farāthen add a learningārate scheduler that halves the step size every few hundred inferences. That should stabilize the gradient updates while still letting the net chew on the new sounds. Give that a spin and ping me if the weight updates start looking like a runaway.