Browsing by Author "Hansen, Lars Kai"

Now showing 1 - 1 of 1

listelement.badge.access-status Open Access ,
Danoliteracy of Generative Large Language Models
(University of Tartu Library, 2025-03) Vejlgaard Holm, Søren; Hansen, Lars Kai; Nielsen, Martin Carsten; Johansson, Richard; Stymne, Sara
The language technology moonshot moment of Generative Large Language Models (GLLMs) was not limited to English: These models brought a surge of technological applications, investments, and hype to low-resource languages as well. However, the capabilities of these models in languages such as Danish were, until recently, difficult to verify beyond qualitative demonstrations due to a lack of applicable evaluation corpora. We present a GLLM benchmark to evaluate Danoliteracy, a measure of Danish language and cultural competency across eight diverse scenarios such as Danish citizenship tests and abstractive social media question answering. This limited-size benchmark was found to produce a robust ranking that correlates to human feedback at $\rho \sim 0.8$ with GPT-4 and Claude Opus models achieving the highest rankings. Analyzing these model results across scenarios, we find one strong underlying factor explaining $95\%$ of scenario performance variance for GLLMs in Danish, suggesting a $g$ factor of model consistency in language adaptation.