The one-sentence version start here
AudioForge takes a 10-second audio clip and tells you which of 200 possible sounds are happening in it — Dog, Applause, Thunder, Acoustic guitar — and it can do that with more than one sound tagged at once, because real recordings usually aren't just one clean sound.
That's the whole job. Everything else in this document — the data, the two models, the AWS servers, the graphs — exists to make that one sentence true as accurately as possible.
The data: FSD50K what the model learns from
A model can't learn "what a dog bark sounds like" out of thin air — it needs thousands of real examples, each one labeled by a human. That dataset is called FSD50K (Freesound Dataset 50K), built from real recordings uploaded to Freesound.org and labeled against a 200-category vocabulary borrowed from Google's AudioSet ontology.
"Held back" matters more than it sounds. If you graded a student using the exact questions they memorized, you'd learn nothing about whether they actually understood the material. The 4,170 validation clips are never shown to the model during training — every score in this document comes from clips the model has never seen.
It's also multi-label, not multi-class: one clip can be tagged Dog and Bark and Animal simultaneously, because those things are all true at once about the same three seconds of audio. That single fact shapes almost every technical decision downstream — the loss function, the output layer, the evaluation metric all have to handle "several correct answers at once" instead of "exactly one."
Sound → numbers the feature pipeline
A neural network can't listen to a .wav file — it only multiplies grids of numbers together. So before anything resembling "learning" happens, every clip goes through a fixed, mechanical conversion:
flowchart LR
A["raw .wav file
(any length, any sample rate)"] --> B["resample to 16kHz,
mix to mono, crop/pad to 10s"]
B --> C["FFT in overlapping windows
→ spectrogram"]
C --> D["compress to mel scale
(128 frequency bins)"]
D --> E["log-mel spectrogram
a 128 × ~313 grid of numbers"]
E --> F(("model"))
The end result — the log-mel spectrogram — is genuinely just a picture of sound: time runs left to right, pitch runs bottom to top, and brightness is loudness at that pitch, at that instant. "Mel scale" means the frequency axis is warped to match human hearing, which is much more sensitive to differences at low pitches than high ones. It's the same reason a piano's lowest octave has visibly wider keys of *perceptual* difference than its highest — mel spacing does that mathematically for frequency.
Once sound has become a 128×313 grid of numbers, it's no longer really "audio" as far as the model is concerned — it's an image. That reframing is exactly what lets us borrow two very different tools from image processing to actually learn from it.
The two models two philosophies, one job
AudioForge trains two completely different models on the same spectrograms, on purpose — one built from nothing, one built on a giant's shoulders. Comparing them tells you something real about the tradeoff between the two philosophies.
scratch_cnn — learns from zero
A convolutional neural network (CNN) with every single one of its 2,412,008 numbers starting at random and being shaped entirely by our 36,796 training clips. Nothing is borrowed.
It works in 5 stages. Each stage looks at small 3×3 patches of the spectrogram, doubles how many different "patterns" it tracks (32 → 64 → 128 → 256 channels), and shrinks the image in half at the same time. By the last stage it isn't looking at raw pixels anymore — it's looking at abstract shapes built from shapes built from edges.
Cheap to train. Knows nothing you didn't teach it directly.
ast — stands on a pretrained giant
The Audio Spectrogram Transformer, a 86.8-million-parameter model already trained by MIT on 2 million AudioSet clips before we ever touched it. It already "knows" what thunder, engines, and speech look like as spectrograms.
Rather than retrain all 86.8M numbers (slow, and risks erasing what it already knows), we freeze the entire pretrained model and attach small trainable "adapters" — a technique called LoRA. Only 450,248 numbers (0.52%) ever get updated.
Expensive to pretrain (someone else already paid that cost). Cheap for us to adapt.
LoRA, concretely
Every frozen weight matrix W inside the pretrained model stays
completely untouched. Alongside it, LoRA adds two tiny matrices, A
and B, and the model actually computes
output = W·x + (B·A)·x. W never moves. Only the
small A/B pair — and a brand-new classifier head,
since the pretrained model only knew AudioSet's 527 categories, not our 200 —
receive any gradient at all.
The intuition: adapting a model that already understands sound in general to our specific 200 categories is a small correction, not a full re-education. A low-rank patch captures most of that correction at roughly 1/200th the trainable parameter count of retraining everything.
What "training" actually means in plain language
Strip away the jargon and training is one loop, repeated tens of thousands of times:
- Show the model a small batch of spectrograms it hasn't adjusted for yet.
- It guesses, for each of the 200 categories, "how likely is this sound present" (a number between 0 and 1 per category).
- Compare those guesses to the real human labels — the gap between guess and truth is the loss.
- Nudge every trainable number very slightly in the direction that would have made that guess less wrong (this is gradient descent — literally following the slope downhill).
- Repeat, with a new batch, thousands of times.
One full pass through every training clip is an epoch. We train scratch_cnn for 5 epochs and AST for 5 epochs. Because the loss surface is enormous and jagged, we don't take huge steps blindly — a learning rate schedule starts small, ramps up for the first ~5% of training (warmup, so the model doesn't lurch wildly before it's learned anything), then decays smoothly toward zero as training finishes, for finer and finer adjustments.
How we grade it mAP, precision, recall
Loss is what the model optimizes during training; it's not what we trust to judge whether the model is actually good. For that we use metrics computed only on the 4,170 held-back validation clips:
- Precision
- Of everything the model flagged as "Dog," what fraction really was a dog? Low precision = too many false alarms.
- Recall
- Of everything that really was a dog, what fraction did the model actually catch? Low recall = too many misses.
- mAP (mean Average Precision)
- The headline number. For each of the 200 categories, rank every validation clip by how confident the model was, and check whether the truly-positive clips cluster near the top of that ranking. Average that across all 200 categories. 1.0 is a perfect ranking; a model that guesses randomly scores near 0.
mAP is the fairest single number here specifically because it's threshold-free — it grades the model's ranking of confidence, not one arbitrary cutoff, which matters a lot when 200 categories have wildly different numbers of positive examples.
The real results both models, fully trained
Every number below came off two real training runs on real AWS hardware — nothing here is projected or simulated.
| Metric | scratch_cnn (final) | ast + LoRA (best) |
|---|---|---|
| mAP | 0.3020 | 0.5567 |
| Micro Average Precision | 0.5343 | 0.7166 |
| Micro F1 | 0.4260 | 0.6443 |
| Macro F1 | 0.1183 | 0.3843 |
| Macro Precision / Recall | 0.2651 / 0.0888 | 0.6113 / 0.3235 |
| Micro Precision / Recall | 0.8082 / 0.2892 | 0.8091 / 0.5353 |
| Trainable parameters | 2,412,008 (100%) | 450,248 (0.52%) |
| Total parameters | 2,412,008 | 86,792,848 |
| Epochs | 5 | 5 |
| Wall-clock to train | ~22.4 minutes | ~71 minutes |
| GPU | 1× A10G (g5.xlarge) | 1× A10G (g5.xlarge) |
scratch_cnn's mAP climbed on every single evaluation — 0.083 → 0.090 → 0.147 → 0.162 → 0.200 → 0.212 → 0.245 → 0.278 → 0.280 → 0.302 — never once regressing, exactly what healthy training looks like.
ast + LoRA's mAP climbed even faster — 0.113 → 0.363 → 0.403 → 0.457 → 0.498 → 0.513 → 0.518 → 0.537 → 0.544 → 0.546 → 0.551 → 0.555 → 0.557 (best, step 10,000 of 11,500) → 0.556 (final step) — a small dip in the very last steps that's normal noise near convergence, not a real regression, which is why the trainer keeps the best checkpoint, not just the last one.
The comparison is the real payoff here: AST, updating only 0.52% of its parameters, beat the fully-trained from-scratch CNN's mAP by 84% (0.302 → 0.557) — while training in roughly 3× the wall-clock time, not 200×. That's the entire argument for transfer learning made concrete, in one table.
Bugs found & fixed the honest engineering log
None of this ran perfectly the first time. Here's what was actually broken along the way, caught before it could waste real GPU hours — which is the entire point of an audit pass and a smoke test.
| What was broken | Root cause | Fix |
|---|---|---|
| Smoke test did nothing | Config file was empty (silently fell back to full-dataset defaults); real smoke params were misplaced inside a .sh file with no shebang |
Moved YAML to the right file, rewrote the script as real bash |
| A metric function was silently wrong | compute_per_class_average_precision scored every prediction against a hardcoded all-zero array, ignoring the model's real output |
Added a real threshold on actual scores |
| An entire loss module was dead code | training/losses.py was fully implemented but the trainer hardcoded BCEWithLogitsLoss directly, never calling it |
Wired it in via a loss_fn config key and factory function |
| Two competing config systems | A pydantic AudioForgeConfig was exported as if canonical, but nothing in training/inference/serving used it |
Removed entirely — one real config, the trainer dataclass |
| Anomaly-detection configs lied about their method | beats_knn.yaml implied a pretrained BEATs encoder; the actual code was classical log-mel-statistics + k-NN, and all four configs produced identical output regardless of name |
Removed the whole DCASE pipeline per project scope decision |
| torchaudio crashed on the real GPU instance | torchaudio.load() needs torchcodec, which itself needs system-level FFmpeg shared libraries the base AMI didn't have — invisible until actually run on real hardware |
uv add torchcodec + apt-get install ffmpeg, verified end-to-end before trusting it |
| LoRA silently adapted nothing | target_modules=["query","value"] matched zero modules — this transformers version names AST's attention q_proj/k_proj/v_proj/o_proj, not the older convention |
Caught by a dedicated AST smoke test before the real ~1hr run; verified the real names directly against the loaded model and fixed both configs |
The AWS journey not just "we rented a GPU"
Training a model — even the "small" scratch_cnn — means millions of matrix multiplications per second, sustained for tens of minutes. A laptop CPU could technically do it, just a hundred times slower. Getting an actual GPU provisioned and working, though, was its own real story:
What actually had to happen, in order
| Step | What happened |
|---|---|
| VRAM sizing | Calculated weights + gradients + optimizer state + activations by hand for both models before picking a GPU, to avoid over- or under-provisioning |
| Service quota | New AWS accounts default to 0 vCPU quota for GPU instance families — a separate gate from billing/credits entirely. Had to file two increase requests (on-demand L-DB2E81BA and spot L-3819A6DF, both to 8 vCPUs), each going to manual review, approved ~30 minutes later |
| Spot capacity | Tried Spot first (cheaper, ~$0.46–0.93/hr) across multiple Availability Zones and even a different instance size — every attempt returned "insufficient capacity." Fell back to On-Demand and said so plainly rather than silently eating the cost difference |
| Security group churn | SSH access was scoped to a single home IP address (least-privilege, not 0.0.0.0/0) — that IP changed three separate times over the course of this project as the ISP reassigned it, each time requiring a live security-group update before SSH would reconnect |
| Dataset download | FSD50K (~29GB across 9 split-zip files) comes from Zenodo — a CERN-run academic repository, not a CDN, hosted in Europe with per-client rate limiting. Took a few hours at ~3–4 MB/s sustained, downloaded straight onto the instance rather than the laptop |
| Cost discipline | The instance was stopped (not terminated) between every distinct phase — download → prep → smoke test → scratch_cnn → AST — so compute billing only ran while something was actually happening on the GPU. EBS storage keeps billing regardless of instance state, but at ~$0.40/day it's trivial next to ~$1.01/hr compute |
GPU internals, for real measured, not estimated
The two models put dramatically different load on the same A10G GPU — and unlike everything estimated earlier in this document, these numbers were measured live, from real step timings during the real training runs.
| Metric | scratch_cnn | ast + LoRA |
|---|---|---|
| Sequence/spatial size per sample | 128×313 spectrogram | 1,214 tokens |
| Forward FLOPs per sample | ~4.43 GFLOPs | ~260 GFLOPs |
| Measured throughput | ~4.35 steps/sec | ~3.1 steps/sec |
| Achieved compute | ~1.85 TFLOPS | ~38.7 TFLOPS |
| A10G utilization (of 125 TFLOPS FP16 peak) | ~1.5% | ~31% |
scratch_cnn is so small that the GPU spends most of its time on fixed per-kernel launch overhead rather than actual math — its 5-block CNN just doesn't give the A10G's 9,216 CUDA cores and 288 Tensor Cores enough work per instruction to stay busy. AST is the opposite story: a 1,214-token attention mechanism (the query/value projections and the 1,214×1,214 attention score matrix, recomputed for every one of 12 layers) is exactly the kind of large, regular matrix multiplication Tensor Cores are built for — ~20× higher GPU utilization, measured, not assumed.
Git & Hugging Face the two places this project lives
GitHub — the code's memory
Every fix, config change, and script in this project is tracked with git and pushed to GitHub (Auro-rium/audioforge). Practically, that means two things: nothing is ever silently lost (every prior version is recoverable), and the exact code that produced a given result is pinned to a specific, checkable commit — when we say "0.302 mAP" or "0.5567 mAP," you can trace either back to the exact config and code that produced it. The key commits in this project's history: the LoRA target-module fix (ffd5507), the scratch_cnn results (787894c), and the AST results (7cbf891).
Hugging Face Hub — the models' home
Trained model weights are deliberately kept out of git (they're large binary files that don't diff or compress like code) and instead pushed to the Hugging Face Hub — the standard place ML models get shared, versioned, and downloaded from, the same way GitHub is for code. Both models are live:
Current status a live snapshot, not a permanent fact
| Step | Status | Detail |
|---|---|---|
| FSD50K download & prep | Done | 36,796 train / 4,170 val / 10,231 test clips, 200 labels, zero missing audio |
| scratch_cnn training | Done | 0.302 mAP, pushed to GitHub + Hugging Face |
| ast + LoRA smoke test | Done | caught & fixed a real LoRA target-module bug before the full run |
| ast + LoRA full training | Done | 0.5567 mAP, ~71 min on the A10G |
| ast → Hugging Face export | Done | LoRA adapter live at auro-rirum/audioforge-ast-fsd50k |
| Raw dataset cleanup | Done | ~55GB removed from the instance once both checkpoints were safely pulled down |
| GPU instance | Stopped | no compute billing while idle; both trained checkpoints are safely off-instance |
Glossary quick reference
- Spectrogram
- A picture of sound: time left-to-right, pitch bottom-to-top, brightness = loudness.
- Multi-label classification
- Several labels can be true for the same input at once, unlike a single "cat vs. dog" choice.
- Parameter
- One trainable number inside the model. scratch_cnn has ~2.4M; AST has ~86.8M.
- Epoch
- One full pass through every training clip.
- Loss
- A single number measuring how wrong the model's current guesses are; training tries to shrink it.
- Transfer learning
- Reusing a model already trained on a related, larger task instead of starting from random weights.
- LoRA
- Freeze the pretrained weights; train only a small pair of "correction" matrices alongside them.
- mAP
- Mean Average Precision — the headline accuracy metric here, averaged across all 200 categories.
- Checkpoint
- A saved snapshot of every trained number, at a point in training — what actually gets deployed.