Voice Standard Bench ranks LLMs for transcript extraction
News
Voice Standard Bench compared more than 50 Gateway models on JSON extraction from transcripts, with rankings varying by transcript length.
Voice Standard Bench has published a transcript-specific comparison of large language models for structured data extraction, arguing that spoken calls pose different problems from PDFs or web pages. In the October 28, 2026 results cited by the post, the top model changes by transcript length, while schema validity and groundedness remain central to the scoring.
Evidence and context
The post, published by AssemblyAI on October 27, 2026 and labeled current as of October 28, 2026, says the benchmark evaluates more than 50 models on information extraction from speech-to-text output. It focuses on a fixed JSON shape for participants, action items, decisions and named entities, and it requires models to output only the object, with every key present. If a field has nothing to report, the benchmark uses an empty array or the string “unspecified” rather than a guess or a missing key.
According to the methodology described in the post, scoring begins with a mechanical schema check. Outputs that fail to return valid JSON with all required keys receive the worst rubric score and are also counted separately as schema failures. Valid outputs are then judged on groundedness and completeness by Claude Opus 5.5 and GPT-5.6 Sol, with the two judges tracked separately and then combined. The benchmark runs at four transcript lengths: 30 seconds, 2 minutes, 15 minutes and 60 minutes, using 10 real transcripts per bucket.
The published cards show that the best model varies with length. For 30-second transcripts, gpt-5.6-sol scores 94. For 2-minute transcripts, nemotron-lightning-3p5-30b-a3b is listed at 100, but the post flags that result as an outlier because the model recorded 15 errored requests in that bucket and errored requests are excluded from the score. Its 2-minute median response time is 18.1 seconds, and the post notes that its 30-second extraction score is 26. The next-highest 2-minute scores are gpt-5.6-luna and gemini-4-argon, both at 91.
For 15-minute support calls, gpt-6-luna is listed as the highest-performance model at 85, while the cheapest entry is nemotron-3-nano-30b-a3b at 49 and the fastest is qwen3.5-4b-32k-fast at 33 with a 1.49-second median response time. For 60-minute meetings, gpt-6.1-sol leads at 85, with nemotron-3-nano-30b-a3b the lowest-cost option at 52 and gemini-3.5-flash-lite the fastest at 59 with a 2.86-second median response time.
The post uses these results to make a narrower point: transcript length changes the extraction problem. It says short inputs favor small, fast, cheap models, while longer transcripts put more weight on context handling and completeness. At 30 seconds, 43 of the 52 models on the tab score 80 or higher; at 60 minutes, only 9 models do. The same post says qwen3.5-4b-32k-fast scores 80 at 30 seconds but falls to 33 at 15 minutes, illustrating the shift.
What the benchmark says to watch
The article identifies four recurring failure modes in transcript extraction: misheard or spelled-out numbers, spelled names and hedging, speaker attribution, and cross-talk or disfluency. It argues that a model trained to produce fluent text can invent plausible but unsupported details, which is especially problematic when the output is meant to populate downstream JSON fields.
AssemblyAI also notes that its Gateway product supports structured outputs through response_format and JSON Schema for several model families, and that tool calling is an alternative. The post includes a sample extraction workflow that transcribes audio first, then passes the transcript to an LLM with a strict schema. It recommends validating outputs, retrying once on failure, and using empty fields or “unspecified” rather than forcing the model to guess.
The benchmark results are presented as guidance for model selection, not a guarantee for production use. The post explicitly says the live page is the source of truth and will change as models change. It also notes that the benchmark’s live cards do not apply a quality floor to the cheapest and fastest options, so users are told to check the score before shipping one of those picks.
As of the publication date cited in the source material, the next verified catalyst was not supplied.
