Ármannsson, BjarkiIngimundarson, Finnur ÁgústSigurðsson, Einar FreyrJohansson, RichardStymne, Sara2025-02-172025-02-172025-03https://hdl.handle.net/10062/107196This paper introduces a linguistic benchmark for Icelandic-language LLMs, the first of its kind manually constructed by native speakers. We report on the scores obtained by current state-of-the-art models, which indicate room for improvement, and discuss the theoretical problems involved in creating such a benchmark and scoring a model's performance.enAttribution-NonCommercial-NoDerivatives 4.0 Internationalhttps://creativecommons.org/licenses/by-nc-nd/4.0/Article