Õpitulemuste parendamine ja mõõtmine küberkaitseõppustel
Files
Date
2017
Authors
Journal Title
Journal ISSN
Volume Title
Publisher
Abstract
Küberõppusi peetakse üheks efektiivseimaks meetodiks erinevate sihtgruppide koolitamisel, see sobib nii (sõjaväelistele) professionaalsetele meeskondadele kui individuaalsetele õpilastele. Samas põhinevad teadmised õppustel saavutatud õpitulemustest peamiselt suulisel infol ja metoodika efektiivsust pole tõestatud. Käesolev töö käsitleb õppimist küberkaitseõppustel ning keskendub õpitulemuste hindamisele. Erinevate õppuste formaatide seast on antud töö aluseks valitud tehnilised küberkaitseõppused, milles on esindatud punaste ja siniste meeskonnad. Töös analüüsitakse kübekaitseõppusi lähtuvalt täiskasvanu õpiteooriatest ja õpitulemuste mõõtmise hetkeolukorda küberkaitseõppuste raamistikus. Õpitulemusi mõõdeti kahel küberkaitseõppusel, Locked Shields ja Crossed Swords. Neist esimene on suurim avalik küberkaitseõppus maailmas peaaegu 900 osalejaga ning peamiseks koolitusgrupiks on siniste meeskonnad. Teine õppus on väiksemahuline punaste meeskonna õppus. Locked Shields ja Crossed Swords on korraldatud NATO küberkaitsekeskuse poolt. Sellised õppused on tehniliselt väga komplekssed ning nii korraldajatele kui osalejatele keerukad. Seetõttu vajavad nii õppuse disain kui õpitulemuste mõõtmine suuremat tähelepanu. Käesolev töö pakub välja uudse ja skaleeritava õpitulemuste mõõtmise metoodika, nn. “5-ajatempli metoodika”. Metoodika hõlmab nii efektiivset tagasisidet (s.h. võrdlusvõimalus) kui õpitulemuste mõõtmist. See võimaldab hinnata meeskondade tegevustulemust, ja väidab, et tulemuste muutus ajas näitab ka õpitulemusi. Ajatempleid saab koguda nii traditsiooniliste meetoditega (nt. intervjuud, vaatlused ja küsimustikud), aga ka potentsiaalselt mitte-intrusiivselt võrgulogidest (nt. pcap’id). Metoodika aitab parandada tagasisidet, tuvastada õppuse disaininõrkusi ja näidata kübekaitseõppuste õpiväärtust. Crossed Swords õppuse hindamisel keskenduti eelkõige osalejatele (punaste meeskond) kohese tagasiside andmisele nende tegevuste kohta. Käesolev töö annab olulise panuse küberkaitseõppuste õpitulemuste hindamise teoreetiliste ja praktiliste aluste kohta ning pakub välja praktilised soovitused õpikogemuse parendamiseks.
Cyber security exercises are believed to be the most effective training for all training audiences from top (military) professional teams to individual students. However, evidence of learning outcomes for those exercises are often anecdotal and not validated. This thesis takes a fresh look at learning in Cyber Defence Exercises (CDXs) and focuses on measuring learning outcomes. As such exercises come in a variety of formats, this thesis focuses on technical CDXs with Red and Blue teaming elements. The review of adult learning theories and current state of learning measurement in CDXs context are presented. The learning measurements are performed at two CDXs: Locked Shields and Crossed Swords. First one is the largest unclassified live-fire CDX in the world with nearly 900 participants (with Blue teams as main training audience). Second one is a small scale exercise designed to train Red teams. Both exercises are organised by the NATO Cooperative Cyber Defence Centre of Excellence (CCD COE). Such top-end CDXs are highly complex, which makes it hard for organisers and participants to handle. Therefore, both learning design and measurement need careful consideration. This work proposes a novel and scalable learning measurement methodology, called the “5-timestamp methodology”. This method aims at accommodating for both—effective feedback (including benchmarking opportunity) and learning measurement. The method is capable of assessing team performance, and argues that changes in performance over time equal learning. The timestamps can either be collected using traditional methods, such as interviews, observations and surveys, but also potentially be obtained non-obtrusively from raw network traces (such as pcaps). The method enhances the feedback loop, allows identifying learning design flaws, and provides solid evidence of learning value for CDXs. Crossed Swords measurement focused on providing the training audience (Red team) with instant feedback about their actions to ensure effective learning. This work contributes to theoretical foundations and in practical terms by providing practical recommendations readily applicable for improvement of learning experience in CDXs.
Cyber security exercises are believed to be the most effective training for all training audiences from top (military) professional teams to individual students. However, evidence of learning outcomes for those exercises are often anecdotal and not validated. This thesis takes a fresh look at learning in Cyber Defence Exercises (CDXs) and focuses on measuring learning outcomes. As such exercises come in a variety of formats, this thesis focuses on technical CDXs with Red and Blue teaming elements. The review of adult learning theories and current state of learning measurement in CDXs context are presented. The learning measurements are performed at two CDXs: Locked Shields and Crossed Swords. First one is the largest unclassified live-fire CDX in the world with nearly 900 participants (with Blue teams as main training audience). Second one is a small scale exercise designed to train Red teams. Both exercises are organised by the NATO Cooperative Cyber Defence Centre of Excellence (CCD COE). Such top-end CDXs are highly complex, which makes it hard for organisers and participants to handle. Therefore, both learning design and measurement need careful consideration. This work proposes a novel and scalable learning measurement methodology, called the “5-timestamp methodology”. This method aims at accommodating for both—effective feedback (including benchmarking opportunity) and learning measurement. The method is capable of assessing team performance, and argues that changes in performance over time equal learning. The timestamps can either be collected using traditional methods, such as interviews, observations and surveys, but also potentially be obtained non-obtrusively from raw network traces (such as pcaps). The method enhances the feedback loop, allows identifying learning design flaws, and provides solid evidence of learning value for CDXs. Crossed Swords measurement focused on providing the training audience (Red team) with instant feedback about their actions to ensure effective learning. This work contributes to theoretical foundations and in practical terms by providing practical recommendations readily applicable for improvement of learning experience in CDXs.