Model selection and AIC

Abstract

This master thesis investigates the role of Akaike Information Criterion (AIC) in choosing statistical models, combining theoretical foundations with practical simulations to evaluate its effectiveness. The theoretical part of the thesis shows that AIC is based on Kullback-Leibler divergence and cross-entropy, emphasizing its role in minimizing information loss while maintaining a balance between model fit and complexity. In practical analysis, samples from parametric distributions supported on unit interval are simulated to evaluate the effectiveness of AIC in correctly identifying the true model among various competing alternatives. The results demonstrate that AIC successfully prevents overfitting by penalizing excessive parameters. Moreover, the asymptotic behavior of AIC is also analyzed to see if and how the probability of choosing the correct model converges as the sample size increases. In addition, AIC bias-corrected estimate of the cross-entropy is analyzed and compared with other bias-corrected estimates in order to evaluate its effectiveness in reducing bias. Overall, this thesis demonstrates the importance of AIC in model selection by optimizing the trade-off between model fit and complexity.