Sentence overlap, training data and Danish speech recognition benchmarks.
November 3, 2026 · 7 min read
Mattias Fält
Head of AI
We recently released Pianissimo, a fast state-of-the-art speech-to-text model for Danish. But if Pianissimo is the best Danish speech-to-text model, why isn't it at the top of every leaderboard?
It's a completely valid question, but it has a complicated answer. We asked ourselves the same question when we finished training Pianissimo for Danish. It looked really good in our tests, but it only placed 11th on the leaderboard.
Most public leaderboards have a set of open benchmarks that they use to score different models. This means that model creators try to score as well as possible on these datasets. However, scoring well on a specific dataset does not necessarily translate to being good in general.
Why did we only place eleventh?
On the public Danish ASR leaderboard, Pianissimo only placed eleventh. Because of the amount and quality of our data, we expected it to perform a lot better. This made us delve into the details of leaderboards and datasets to understand why we performed worse than the other models on this specific leaderboard.
The first thing we did was to create a new benchmark with the goal of scoring how models perform on data they had not seen before. It consisted of three new datasets: European Parliament speeches, television programmes, and DanPASS, a collection of conversations collected by researchers at Copenhagen University.
When we ran the evaluations on this set, Pianissimo scored the best out of all the models we could test. Its mean word error rate (WER) was 11.8%, lower than top models from companies such as ElevenLabs and NVIDIA.
Results on our benchmark sets
Mean accuracy across three Danish test sets
88.2%Klang Pianissimo, mean accuracy
Klang Pianissimo
Open models
Closed models
Mean accuracy across our three test sets. Accuracy = 100 − WER.Full results: all 25 models
Word error rate (%) · lower is better
Model
Mean
European Parliament
DR TV
DanPASS
Klang Pianissimo
11.83
7.86
18.24
9.38
ElevenLabs Scribe v2
12.33
8.59
19.07
9.31
edda-v0.2-duo
13.08
9.23
20.37
9.64
brage-v1
13.24
9.05
19.95
10.73
saga-2-m
13.28
8.79
20.13
10.93
hviske-v6
13.48
8.37
21.36
10.72
edda-v0.1
13.75
9.36
21.13
10.75
Mistral Voxtral Small Transcribe 3
13.83
8.78
19.08
13.62
ekko-v1-tiny
13.94
8.95
22.46
10.42
Syv Transcribe
14.04
8.66
23.79
9.68
whisper-large-v3-turbo-danish
14.05
7.80
20.78
13.56
svale-600M
14.07
8.67
22.72
10.82
edda-v0.2-large
14.14
9.83
20.87
11.73
OpenAI GPT Transcribe
14.16
9.97
19.88
12.62
whisper-small-danish
14.17
8.77
20.97
12.76
svale-110M
14.30
9.58
21.96
11.36
edda-v0.2
14.32
10.39
21.49
11.07
Mistral Voxtral Mini Transcribe 3
14.58
9.15
19.73
14.86
Google Gemini 3.5 Transcribe
14.86
9.15
21.10
14.34
hviske-v5.1
16.15
9.54
22.89
16.02
Whisper large-v3
17.00
10.50
22.83
17.68
Whisper large-v3-turbo
17.62
11.11
23.05
18.69
hviske-v5.3
19.51
13.91
33.05
11.56
Canary-1B-v2
20.66
10.92
25.36
25.71
Parakeet v3
25.53
16.28
32.31
28.01
Training sentences in the test set
So why is it that Pianissimo scores highest on one dataset, but only eleventh on another?
Part of the answer is differences in output conventions such as whether or not disfluencies are dropped, how numbers are normalised, etc. But it was not the main cause. So we started looking into the leaderboard datasets and what data the other models trained on.
We suspected that several of the other models on the leaderboard might be heavily overfitting to these specific datasets. To investigate if this was the case, we began by looking at overlaps between the training and the test sets of the leaderboard. It turns out that for some datasets, e.g. the CoRal read-aloud dataset, over 70% of the sentences in the test set also exist in the train set, but spoken by different speakers. This means that if you were to train heavily on the train set, a model could learn those specific sentences verbatim and perform really well on the test sets.
To measure if this was actually the case, we compared the word error rate of 49 models on:
sentences that existed in both the train and test data
sentences that were unique to the test set.
We computed the "overlap advantage" as a percentage of how much better a model performed on the first vs the second case. In the graph below, you can see the results of this test.
Full comparison: all 49 modelsExpand to see the full results for all models on both the leaderboard and our own benchmarkOur WER averages three test sets. Saga-2-l was not included in our tests because we could not find any public API.
We see that several of the top models heavily overperform on the examples where there is overlap. For example, syv-transcribe had 5.1% WER on repeated sentences compared to 9.1% for new ones, a 43% decrease.
Pianissimo showed very little difference, 14.7% on repeated sentences and 14.9% on new ones.
What is the point of a benchmark?
A benchmark measures the results on a very specific set of examples. The goal is not to know how well a model performs on those specific tests. The hope is that the test will say something in general about the model.
It's the same with any type of test. Assume that you are trying to learn general "History". If a tester picks a random set of questions from different eras, topics and geographic areas, then your result is likely to reflect your general knowledge of "History".
But this only works if the questions can be considered random. For public benchmarks they are often not random. They often correspond to a few specific areas like "Danish Kings", "French Wars" and "Greek Philosophers". Testing on these sets could reflect your general "History" knowledge if you didn't know the specific topics in advance. But if you studied specifically on these topics, it's likely that your score would not reflect your general knowledge.
For the case of the CoRal dataset it's even worse than described above, it's closer to a test where 70% of the questions were known beforehand.
This is a complicated way of saying that leaderboards are often biased, and it's a hard problem to solve. There is a real advantage to releasing the tests publicly so that anyone can run it themselves. Even if they are kept secret, it's well known that it's hard to stop submissions from specializing even on hidden datasets.[1] Moreover, there is a limited amount of high quality data freely available, so it's no surprise that models are trained on these train splits, even if they bias the results.
Danish ASR Leaderboard
The Danish ASR Leaderboard has obviously inspired many organizations to train and release models. This is a great achievement, and it's one of the secondary goals of a benchmark: to help push the frontier.
However, the problems above also mean that these scores can't be used alone to judge how well a model performs on Danish in general.
When we take a closer look at the data used to train the models on the leaderboard, we see this pattern clearly. Several of the models like Edda v0.2, Edda v0.2-large and Brage v1 have over 80% of their training data from the same datasets as the leaderboard datasets.[2][3] This is not cheating or unethical, it is completely reasonable to train on the available datasets. However, it does create a significant bias in the results.
Pianissimo was also trained on these datasets, but they accounted for only 1.5% of our training data. Like several of the closed models, such as ElevenLabs Scribe v2, Voxtral Transcribe 3 and GPT Transcribe, there is little or no advantage on repeated sentences because of the relative size of the datasets.
On the other hand, a great example of the potential bias is the Hviske 5.3 model. It was finetuned for 5 epochs on the CoRal training splits from the Hviske 5.1 model.[4] It improved from 17.8% WER to 9.3% on the corresponding test set, but increased its relative errors by 20% on our test sets. A similar bias was measured and declared for Brage v1 by the authors themselves.[3]
As an extra check to investigate the overfitting, we compared the advantage on repeated sentences with the advantage on the public leaderboard. Across 24 models, the two closely follow each other: models that benefit most from repeated sentences also tend to lose more of their leaderboard advantage on our test sets.
Sentence overlap and leaderboard overperformance
PianissimoOther modelsLinear fit
Relative WER reductions: repeated vs. new sentences (X), and public leaderboard vs. our test sets (Y). Pearson r = 0.77; Spearman ρ = 0.83.
This is what we would expect from models that have become too specialised for the benchmark. It also makes the overlap advantage a useful warning sign. A large overlap advantage is likely to mean that the model performs better on the leaderboard than on general Danish speech.
Conclusions
So does this mean that leaderboards are useless? No. But it means we have to be careful in how we judge models, and we have to remember that there might be strong biases in these leaderboards.
You can try Pianissimo yourself here or download it from Hugging Face.
Thanks to CoRal and RyeAI for the datasets and published outputs that made the overlap analysis possible, and to Nina Grønnum for providing access to the DanPASS data.
