Sirvi Autor "Torstensson, Olle" järgi

Tulemuste filtreerimiseks trükkige paar esimest tähte
Nüüd näidatakse 1 - 1 1
  • Pisipilt
    listelement.badge.dso-type Kirje ,
    A Grammar-Based Method for Instilling Empirical Dependency Structure in LLMs
    (University of Tartu Library, 2025-03) Torstensson, Olle; Holmström, Oskar; Trosterud, Trond; Wiechetek, Linda; Pirinen, Flammie
    We investigate whether synthetic pretraining data generated from a formal grammar modeling syntactic dependencies can improve English language models. Building upon the structured pretraining data approach of Papadimitriou and Jurafsky (2023), we develop a grammar that more closely mirrors empirical dependency structures. Our results are negative – this type of pretraining significantly degrades model performance, with both our and their pretraining approach performing worse than no pretraining at all. We analyze potential explanations for these findings and discuss implications for future work on structured-data pretraining.