My data, my benchmark: Swapping the speech recognition (ASR) engine for my lecture note system

Introduction: Why official scores aren’t enough

My workflow for turning recorded lectures into notes goes in a straight line: transcript → matching slides → final summary.

The foundation of this line is automatic speech recognition (ASR). However much it mishears, every subsequent layer of notes will be just as wrong. Originally, I was using OpenAI’s Whisper large-v3.

The starting point for all this was actually me scrolling on my phone and seeing MediaTek Research announce their Taiwanese model, Breeze-ASR-26. A Taiwanese model is intriguing enough on its own, but clicking through, I learned they had already released Breeze-ASR-25 last year. Fine-tuned from Whisper-large-v2, it was optimized specifically for Taiwanese Mandarin and Mandarin-English code-switching, and open-sourced under Apache 2.0. Medical lectures happen to be ground zero for Mandarin-English code-switching. I had somehow missed this for a whole year.

So I was facing two models at once, and their official data was completely asymmetrical.

Breeze-ASR-25 provided a full report card, with word error rate (WER, lower is better):

(The official columns say WLV3-Auto / WLV2-Auto, meaning Whisper with automatic language detection turned on.)

It even beat the large-v3 I was using at the time, and the two code-switching rows were just overwhelming. I have no need to prove this part again.

And Breeze-ASR-26? It only had one score for Taiwanese—but that one was solid. The official benchmark compared its Taiwanese character error rate (CER, lower is better) against four systems:

It beat Gemini 3 Flash on Taiwanese, and it beat Yating. This is real capability; it shouldn’t be ignored.

But its report card ended right there:

• No Mandarin benchmark
• No English benchmark
• No code-switching benchmark
• No timestamp/alignment data at all

In other words, aside from Taiwanese, the official docs didn’t say a word about its performance in any other scenario. Yet it was fine-tuned from the very same Whisper family; it’s a sibling to 25. So is it “25 plus Taiwanese,” or did it “sacrifice other things for Taiwanese”? Nobody knew, because nobody had tested it.

(Let’s be clear upfront: it never claimed to have capabilities beyond Taiwanese, so what follows isn’t a teardown. It’s just filling in the pieces the official docs left out.)

This is why I had to do it myself. Three questions, and the official benchmark couldn’t answer a single one:

1. Can it catch my domain terminology? The official test used everyday conversational corpora like ASCEND and CommonVoice, not physical medicine and rehabilitation (PM&R) lectures stuffed with nucleus pulposus, piriformis, and spondylolisthesis.
2. Do the timestamps align with the slides? My notes rely on timestamps to paste every sentence onto its corresponding slide. The transcript can be as accurate as you want, but if it can’t segment paragraphs, it’s useless. And neither model card mentioned this.
3. What happens when you take that Taiwanese model into non-Taiwanese scenarios? They didn’t test it, so I did.

An official benchmark is their exam. It tests what they care about, not what you care about—and the things they didn’t test are usually exactly what will break your system.

Methods

If there’s no gold standard, build your own yardstick

The ideal reference standard is a human-refined transcript. I don’t have one, and I didn’t want to handcraft one just for testing. So I switched to two yardsticks that can be computed programmatically.

Yardstick 1: Domain vocabulary glossary (measuring vocabulary quality)

• Automatically extracted over 180,000 English terms from hundreds of physical medicine and rehabilitation (PM&R) textbooks (keeping any word that appeared more than three times in these books).
• Compared each model’s output against the glossary to calculate two numbers: capture (how many real, existing terms it caught) and garbage (how many fake words it generated that aren’t in the glossary).
• The two numbers must be looked at together: a model that completely fails to hear any English would have a perfect precision of 100%, but it wouldn’t have captured anything.

This yardstick doesn’t measure “medical term accuracy,” but something more fundamental: does the word this model spit out actually exist in the physical medicine and rehabilitation (PM&R) literature, or did it just make it up? Invented words like motibra, grutius, or idioma are instantly exposed when checked against the glossary.

Yardstick 2: Paragraph granularity (measuring timestamp usability)

• Slides change roughly every 1 to 2 minutes, so the transcript must be chopped finely enough to match them.
• Threshold: At least 10 segments per minute, with a median segment gap of no more than 8 seconds.
• This yardstick later weeded out one model, and it was a problem that only this yardstick could catch.

A model must pass both checks. If you only measure vocabulary, you’ll end up putting a model into your pipeline that “hears every word right, but smears the whole paragraph together.”

Testing discipline

Three rules, all learned the hard way:

1. n ≥ 3, test variables separately. Single-sample measurements suffer from three biases simultaneously: instrumental error, confirmation bias, and the argument from silence where “not documented” is treated as “model can’t do it.” In my first round, I only used one audio clip to test the waters, and all three of my conclusions were later overturned (including “Breeze is twice as slow,” which was purely an instrumental error from loading two models into the same process and having them fight for memory on an 8GB GPU). This is exactly the same pitfall you have to watch out for when reading a diagnostic study in clinical practice.
2. One model, one process. Following from the above, this is the source of instrumental error, and a hard limit of an 8GB GPU.
3. Leave the scoring to code, not the model. If you let a large language model (LLM) pick its own examples to compare, what it notices will naturally skew toward supporting the new model, bringing confirmation bias right along with it. Full-population statistics must be hardcoded into the scripts.

Corpora

15 clips × 8 minutes each (6 classroom lectures + 9 case discussions), plus 6 English lectures; the retest for paragraph granularity was 4 clips. All were recordings of actual physical medicine and rehabilitation (PM&R) lectures, not read-aloud speech corpora.

Results

1. The model: Breeze-ASR-25 wins outright, and it’s faster

It captured 49% more real terms, and it was faster. The official benchmark said it was better on general corpora; my data confirmed this advantage extends to medical domain terminology, which was the part they didn’t—and couldn’t possibly—test for me.

It’s worth noting that part of the gain came from the settings. Swapping the model wasn’t the only variable.

The last row is the Taiwanese version, Breeze-ASR-26. I included it to answer the inevitable “what about 26?” question: on Mandarin medical lectures, it only caught 182 real terms—less than a third of Breeze-ASR-25.

But this number can’t be taken literally; I misread it myself at first.

Where did the 182 come from: did it not understand, or was my yardstick blind?

Someone asked me (an excellent challenge): could it be that Breeze-ASR-26 was transcribing the English as Chinese, so my English glossary couldn’t catch it?

Looking at the raw output, the answer is yes, a huge portion of it was. Same audio clip:

It transliterated the English terminology into Chinese characters. This is actually exactly the design written on its model card: outputs Chinese characters, not Taiwanese orthography.

It becomes clearer when quantified (same batch of 15 clips):

Its output volume didn’t collapse; it actually had the most Chinese characters of the three. It was only the English that tanked. It wasn’t failing to speak; it just switched to speaking in Chinese characters. (By the way, having the most Chinese characters doesn’t mean its Mandarin is better—“印特拉蜜拉” alone takes up 5 characters.)

So that 182 mixes two things: genuinely not hearing it, and hearing it but writing it in characters. My English glossary could only see the failure of the latter; it couldn’t see whether the model actually understood it or not.

Did the conclusion change? No, but the reasoning is completely different.

For my purposes, this is still a failure: my notes need interlamina, not “印特拉蜜拉”. For searching the literature, matching slides, or showing to colleagues, transliterated characters are completely useless. So the conclusion that you shouldn’t use Breeze-ASR-26 for Mandarin medical lectures remains unchanged.

But the correct way to say it isn’t “it doesn’t understand English terminology,” but rather “it will transliterate English terms into Chinese characters, and that’s not the output format I want.” These two statements grade the model very differently, and I initially wrote the former.

As a side note, Whisper large-v3 also transliterates (“英特拉米拉”, “發射”), just much less frequently. This is a common ailment of the Whisper family, and 26 amplified it.

2. Settings: Both switches are “Off”

• Voice activity detection (VAD) → Off. Its original intent is to cut out silence and only feed in the segments with sound. But pauses in lectures, breaths, and the gaps while flipping slides were often swallowed whole, taking half a word from either side down with them. With it off, the model listens from start to finish, and the boundaries don’t get butchered.
• condition_on_previous_text → Off. This parameter tells the model to use the output of the previous segment as the context for the next one. Theoretically, it makes the text coherent; practically, once a segment is misheard, the error is fed in as “previous context,” and everything that follows gets it wrong too. In severe cases, the whole paragraph gets stuck in an infinite loop of the same sentence. With it off, every segment is decoded independently, and the errors aren’t contagious.

In an actual test of 2,236 lines of output, silence hallucinations (inventing a “Thanks for watching” when nobody is speaking) and repetitive loops occurred 0 times each.

The third row’s beam size 10 is very interesting: it had the highest precision across the board (96.4%), but it was almost twice as slow, and it captured 45 fewer terms. The extra compute traded off for a more conservative output, not a better one, so I didn’t adopt it.

Also, turning off VAD had a side effect: faster-whisper’s batch mode doesn’t support VAD being off, so it automatically falls back to sequential decoding. It actually ended up being faster (3.0 vs 3.4 minutes) because it saved the overhead of chunking and stitching.

3. Negative results: Two smart-sounding ideas both lost

Negative results are still results, and writing them down saves someone else from stepping on the same landmine. The following were all measured using the same batch of 15 clips and the same yardstick.

Feeding terminology prompts (hotwords/initial_prompt): No upside, only cost

I prepared 25 physical medicine and rehabilitation (PM&R) terms (like prolotherapy, nucleus pulposus, piriformis, sacroiliac) and fed them in using faster-whisper’s hotwords:

First, look at what it didn’t do: the real term capture barely budged (587→580, 435→446). The main selling point of feeding a glossary—making it easier for the model to catch the proper nouns you care about—didn’t happen at all.

Then look at what it did do: both models saw an increase in fake words (29→36, 51→56), and precision dropped.

The failure mechanism is worth noting more than the score: a prompt isn’t a dictionary; it’s a “pull.” It yanks the sound the model hears toward the word you gave it, and if it can’t quite reach, it spawns a mangled new typo—and that’s exactly where the new fake words came from: Breeze started spitting out sacroilar and buteroli, and large-v3 spit out paravertibone. These words simply didn’t exist in the versions without the prompt.

The ultimate irony was that nucleus pulposus and prolotherapy, the very words I wanted to save the most, were written out letter-for-letter in the prompt and still didn’t get caught.

It’s a weak guide, not a dictionary lookup. If you really want domain vocabulary, you should fine-tune the model, not pass it cheat sheets during inference.

Taking the union of two models: Lost on every front

The real terms didn’t increase, but the fake words became 5 times as many, and it took twice the compute. The reason is simple: a union sucks in all the words the other model invented out of thin air. The merge logic has been deleted.

But I did pick up a byproduct here: the agreement rate between the two models on a word-by-word basis is an excellent confidence indicator. Normal paragraphs had a character agreement rate between 88% and 91%; one paragraph dropped to 57%, and when I went back and listened, it was off-mic chatter where both models were hallucinating. Now, anything below 75% gets flagged for manual review and isn’t written directly into the notes.

English lectures: A tie, so unify

For the English clips (6 clips), Breeze and large-v3 were practically tied (1535 vs 1537 words), precision was slightly better (98.6% vs 97.8%), and there were no Chinese hallucinations. Since it didn’t lose, I just unified everything to Breeze, leaving me with one less pipeline to maintain.

4. The time alignment regression in Breeze-ASR-26 (The main new finding of this post)

This section is something you won’t find on the official model card.

First, let’s look at what the two official docs say. The Breeze-ASR-25 model card treats time alignment as a selling point:

Enhanced time alignment, suitable for automatic captioning.

But in the model card for the Taiwanese Breeze-ASR-26, it doesn’t say a single word about timestamps, alignment, or captioning. It only mentions one thing: it used about 10,000 hours of synthetic Taiwanese speech for training, dropping the character error rate (CER) on the Taiwanese benchmark from Breeze-ASR-25’s 49.99% down to 30.13%.

Measuring it with Yardstick 2, there’s a reason it was “not mentioned”:

Notice that median segment length: a flat 30.0 seconds, and the max value is also 30.0 seconds. That isn’t its sentence-breaking habit; that happens to be exactly the length of Whisper’s decoding window.

In other words, it wasn’t predicting paragraph boundaries at all. Whisper relies on a special “timestamp token” to decide where a sentence ends, but after Breeze-ASR-26 was fine-tuned on synthetic Taiwanese corpora, this ability regressed, so it just spits out the entire 30-second window in one chunk. Take a 10-minute audio clip for example: Breeze-ASR-25 chopped it into 253 segments; Breeze-ASR-26 only cut out 20.

At this point, my original conclusion was: it’s unusable. But that conclusion was also wrong.

Because timestamp tokens are only one of the paths Whisper uses to generate timestamps. The other path is called word timestamps, which runs on the model’s internal cross-attention, bypassing timestamp tokens completely. And I hadn’t flipped this switch on once.

Once turned on, same audio clip, same model (n=4 clips):

The word-level timestamps themselves were healthy: time monotonically increasing, coverage near 100%, almost zero zero-length words. The tradeoff was that decoding was 1.29x slower.

So the correct way to put it isn’t “Breeze-ASR-26 can’t align timestamps,” but “it can’t with its default settings.” This is a settings issue, not a ceiling on the model’s capabilities.

Discussion

What this means for anyone wanting to use Breeze-ASR-26

If you plan on using Breeze-ASR-26 for anything that requires timestamps (Taiwanese subtitles, transcript alignment, matching slides, editing positioning), take note: its default paragraph segmentation is broken, and you must turn on word_timestamps to use it. That is the most practical sentence in this entire post, and it isn’t written in any official documentation.

As for whether you should switch to 26 for Mandarin content: no. It transliterates English terminology into characters (interlamina → 印特拉蜜拉), and my notes need the original English words so they can be searched and matched to slides. Besides, even if you salvage the granularity by turning on word-level timestamps, it only barely scrapes past the 10-segment-per-minute threshold; Breeze-ASR-25 natively has 18 or more. “Technically viable” and “worth the switch” are two different things.

But please note the scope of that sentence: I’m saying “it’s not suited for my kind of code-switched medical lectures that need English terminology.” It beat Gemini 3 Flash on its home turf (Taiwanese), and that’s its own achievement, which I’m not here to negate.

Limitations (Please don’t over-interpret my results)

• I didn’t test Taiwanese. My corpus doesn’t contain a single Taiwanese audio clip. Breeze-ASR-26’s CER of 30.13% is MediaTek’s number, not something I measured, so I won’t endorse it, nor will I trash-talk it.
• I tested the paragraph granularity of a Taiwanese model using Mandarin audio. The regression of timestamp tokens comes from the model weights, so theoretically it should happen the exact same way on Taiwanese audio, but that’s an inference, not a result I measured. If you really want to use it for Taiwanese, you’ll still have to record a clip and test it yourself.
• My yardstick is blind to transliteration. This is the most important lesson I learned this time. An English glossary can only judge “did the model write this term in English”; it cannot judge “did the model actually understand it.” When Breeze-ASR-26 wrote interlamina as “印特拉蜜拉”, my yardstick gave it a 0—even though it actually heard the sound right. Giving it a 0 for my use case (notes need to be searchable and slide-matched) is correct, but if what you want is “did it understand it,” this yardstick will systematically underestimate any model that leans toward outputting Chinese characters. Before you use it, check: are you measuring “did it understand it,” or “did it write it in the format you want”? I was measuring the latter.
• The glossary isn’t a perfect reference standard. It collects “words that have appeared in the literature,” so it’s mixed with general English words; it’s not a pure medical terminology list. But it maps directly to the outcome I care about: was the key information heard, and is the model making things up.
• My yardstick isn’t necessarily your yardstick. If you care about latency, punctuation, speaker diarization, or emotion, then you need to swap in a yardstick for that.

How should you actually use official benchmarks?

They aren’t useless; they answer “is this model good?" What they can’t answer is “is this model good for my task?" This time, it saved me the verification work of “is 25 actually better than Whisper,” but it also missed two things that were make-or-break for me: domain terminology, and time alignment.

And for one of those, even the unwritten parts of the model card were hiding information. Breeze-ASR-25 specifically wrote “Enhanced time alignment, suitable for automatic captioning,” and Breeze-ASR-26 didn’t say a word. I didn’t read that signal at the time; the yardstick read it for me.

The process is actually very simple: clearly define the result you want → find the yardstick to measure it (maybe more than one) → multiple samples, test variables separately → record the negative results too.

And one more sentence to add: when the yardstick measures a “fail,” first ask, is the model incapable, or did I not set it up right? On the Breeze-ASR-26 hurdle, the answer was the latter. The score will tell you there’s a problem, but it won’t tell you whose fault it is. That step you have to take apart and look at yourself.

I open-sourced this yardstick

I’ve packaged the entire testing method into a repo, which you can take and use to test your own recordings:

👉 github.com/drpwchen/asr-benchmark

Inside are the scripts to build the glossary (just point it at your own corpus), the scoring scripts for both yardsticks, and the scripts to run the “model × setting” matrix—including the variants I tried that didn’t work, so you can reproduce the negative results yourself, no need to take my word for it.

I also included my own 187,000-word glossary as an example, so you can take it for a spin without having to prep your own corpus first; my fellow physical medicine and rehabilitation (PM&R) colleagues can just use it directly. But if you’re in another domain, please use the script to build your own glossary. Taking a physical medicine and rehabilitation (PM&R) yardstick to measure a cardiology lecture is just using the wrong yardstick to begin with, which is exactly the point this whole post is trying to make.

—

Model links: Breeze-ASR-25, Breeze-ASR-26. I previously used this same approach of “testing with my own data” on graph RAG and a skill for reading papers. If you’re interested, you can check them out together.

—