Teksti vektorkujul esitamise kasutamine teksti segmenteerimiseks
Date
2019
Authors
Journal Title
Journal ISSN
Volume Title
Publisher
Abstract
Tekstipõhised andmestikud on tihti struktuuritud lausete kogumid ning seega raskesti kasutatavad paljude eesmärkide täitmiseks. Tekstis struktuuri loomine teemade või mõtete kaupa aitab teksti kokkuvõtmisel, tehisnärvivõrkudega masintõlkel ning teistel rakendustel, kus üksik lause võib pakkuda liiga vähe konteksti. Teksti segmenteerimiseks loodud meetodid on olnud kas juhendamata ning põhinevad sõnade koosesinemise vaatlemisel või juhendatud ning põhinevad sõnade ja lausete vektoresitustel. Selle magistritöö eesmärgiks on üldise teksti segmenteerimise meetodi arendamine, mis kasutab sõna-vektoreid ning koosinuskaugust. Loodud meetodi implementatsioone võrreldakse naiivse tõenäosusliku lahendusega, et hinnata loodud lahenduse efektiivsust. Ühte mudelit kasutati ka osana teksti kokkuvõtmise algoritmi osana, et hinnata lähenemise praktilist kasu. Tulemuste põhjal võib öelda, et kuigi loodud lahendus töötab paremini kui võrdlusalus, edasise uurimistööga on võimalik lähenemise võimekust märkimisväärselt tõsta.
Textual data is often an unstructured collection of sentences and thus difficult to use for many purposes. Creating structure in the text according to topics or concepts can aid in text summarization, neural machine translation and other fields where a single sentence can provide too little context. There have been methods of text segmentation that are either unsupervised and based on word occurrences or supervised and based on vector representations of words and sentences. The purpose of this Master’s Thesis is to develop a general unsupervised method of text segmentation using word vector. The created ap-proach is implemented and compared to a naïve baseline to assess the viability of this method. An implemented model is used as part of extractive text summarization to as-sess the benefit of the proposed approach. The results show that while the approach out-performs the baseline, further research can greatly improve its efficacy.
Textual data is often an unstructured collection of sentences and thus difficult to use for many purposes. Creating structure in the text according to topics or concepts can aid in text summarization, neural machine translation and other fields where a single sentence can provide too little context. There have been methods of text segmentation that are either unsupervised and based on word occurrences or supervised and based on vector representations of words and sentences. The purpose of this Master’s Thesis is to develop a general unsupervised method of text segmentation using word vector. The created ap-proach is implemented and compared to a naïve baseline to assess the viability of this method. An implemented model is used as part of extractive text summarization to as-sess the benefit of the proposed approach. The results show that while the approach out-performs the baseline, further research can greatly improve its efficacy.