FlyTTS: training a fly connectome to generate raw waveform in the browser

After the tagger and FlyDoom, the next test is deliberately easy to state: can the MaleCNS connectome participate in a text → waveform transformation without a vocoder, mel spectrogram, or acoustic codec?

The first experiment does not aim for generalization. It starts with a cheaper, falsifiable target: keep the system learning continuously in the browser until it can reproduce one short recording that already has an associated transcript.

The loop#

The brain is the same compact artifact used by FlyDoom: malecns_l3_compact.mcns, kept frozen and executed inside a Web Worker.

At every step:

text at the current instant ──────┐
                                  ├─> frozen MaleCNS
previous generated waveform ──────┘       │
                                          ▼
                                512 sampled states
                                          │
                                          ▼
                           trainable readout → 128 PCM samples
                                          │
                         target WAV ─ error┘
                                          │
                                      NLMS update
                                          │
                                       next step

When it reaches the end of the WAV, it immediately starts another pass without clearing the learned readout. Loss, correlation, and the generated waveform update lap after lap while the system keeps running.

The target voice#

The initial target is LJ001-0002 from LJSpeech, transcribed as “in being comparatively modern.” The dataset pairs mono 22.05 kHz PCM WAV files with their transcripts. The demo tries to load that clip directly and also accepts a local WAV if cross-origin loading is unavailable.

The training target is the waveform itself. There is no vocoder hidden downstream.

What actually learns#

In this MVP MaleCNS stays fixed. The rate dynamics and the artificial text/audio coupling are engineering choices, not a claim that this is a literal biophysical simulation of Drosophila speech.

The online learner is a small readout over 512 sampled connectome states. Text and the waveform just produced are injected as artificial sensory drives. The first question is therefore intentionally narrow: does the recurrent connectome provide a useful state from which the raw wave can converge?

If it does, the next steps are to make the input adapter trainable as well and then move from single-utterance overfitting to multiple text/audio pairs.

The key feedback loop is already present: the generated sound is not only something we can listen to. It becomes part of the next neural input. In that limited operational sense, the fly continuously “hears” what it just said.