Finding Non-Trivially Similar Documents from a Large Document Corpus
Files
Date
2011
Authors
Journal Title
Journal ISSN
Volume Title
Publisher
Tartu Ülikool
Abstract
Käesoleva magistritöö eesmärgiks on uurida, kuidas leida mittetriviaalselt sarnaseid dokumente suurest dokumentide hulgast. Antud töös kirjeldatakse nii traditsioonilisi meetodeid dokumentide sarnasuse uurimiseks kui ka tutvustatakse uusi. Lisaks viiakse läbi eksperimendid, et uurida väljapakutud mõõtude käitumist andmetel.
Traditsioonilised dokumentide sarnasusmeetodid mõõdavad sarnaste sõnade esinemist kahes dokumendis. Antud töös käsitleme, mis probleemid kaasnevad kui me kasutame dokumentide sarnasusmõõdu arvutamisel vaid viimastes leiduvaid sõnu, tutvustame olemasolevaid kui ka pakume välja uusi mõõte nende probleemide
ületamiseks. Dokumendid on mittetriviaalselt sarnased, kui nad sisaldavad vähe
ühiseid sõnu, kuid on kontekstuaalselt sarnased.
Selleks, et tuvastada dokumentide konteksti pakume töös välja taustgraafi kontseptsiooni. Taustgraafi eesmärk on modelleerida sõnade ehk kontseptsioonidevahelist seost, andes rohkem kaalu nendele sõnadele, mis esinevad tihti koos. Saadud taustgraafi kasutame erinevate dokumentidevaheliste sarnasusmõõtude arvutamiseks.
Käesolevas töös käsitletakse ka kasutaja käitumise ja sarnasusmõõtude vahelist seost. Töös antakse lühiülevaade järjestuste kaevandamise põhimõistetest ning kasutakse neid, et uurida, kuidas erinevad sarnasusmõõdud modelleerivad kasutaja käitumist.
Töös viiakse läbi erinevaid eksperimente uudisportaali Postimees.ee andmetel. Taustgraafi uurimisel näeme, et loodud graaf kirjeldab kontekstisiseseid kontseptsioonide vahelisi seoseid väga hästi. Uurides sarnasusmõõte näeme, et
üleüldisel uudiste soovitamisel töötab meie väljapakutud meetoditest paremini traditsiooniline meetod. Mõõdud, mis kasutavad taustgraafi informatsiooni, annavad paremaid tulemusi traditsioonilistest meetoditest, juhul kui me kasutame väheseid, kuid kvaliteetseid andmeid dokumendi kohta.
Käesolev magistritöö pakub välja uue metoodi dokumentide sarnasuse leidmiseks ning näeme, et antud meetodid töötavad kindlatel juhtudel paremini kui varem kasutusel olnud mõõdud.
This thesis introduces the methods which are used for measuring the similarity between documents. The document similarity measures are an important topic in information retrieval and in document classification systems. Finding similar documents from a document corpus is applicable in many different fields - web search engines, news aggregation services, advertising systems et cetera. An important aspect for a document similarity measure is, that the human opinion of the similarity should concur with the score of similarity. The problem of semantic similarity arises. The standard way to find similarity between documents is to compare the co-occurrence of words in them. Thus it is possible, that two documents which are contextually very similar, but to dot contain the same words, are marked dissimilar by the standard document similarity measures. The goal of the semantic similarity measures is to take into account the context of the documents and use this information for measuring the similarity. The goal of this thesis is to first give an overview of different methods which are used for standard and for semantic document similarity. The second goal is to experiment with the document similarity measures on a news portal dataset and to explore whether we can find some interesting properties of those measures. The motivation for the topic originates from an idea to create a new advertising network which is able to target advertisements better than the networks currently in the market. The goal was to analyse whether we could find a simple, intuitive, yet effective method for finding the non-trivial similarity between documents.
This thesis introduces the methods which are used for measuring the similarity between documents. The document similarity measures are an important topic in information retrieval and in document classification systems. Finding similar documents from a document corpus is applicable in many different fields - web search engines, news aggregation services, advertising systems et cetera. An important aspect for a document similarity measure is, that the human opinion of the similarity should concur with the score of similarity. The problem of semantic similarity arises. The standard way to find similarity between documents is to compare the co-occurrence of words in them. Thus it is possible, that two documents which are contextually very similar, but to dot contain the same words, are marked dissimilar by the standard document similarity measures. The goal of the semantic similarity measures is to take into account the context of the documents and use this information for measuring the similarity. The goal of this thesis is to first give an overview of different methods which are used for standard and for semantic document similarity. The second goal is to experiment with the document similarity measures on a news portal dataset and to explore whether we can find some interesting properties of those measures. The motivation for the topic originates from an idea to create a new advertising network which is able to target advertisements better than the networks currently in the market. The goal was to analyse whether we could find a simple, intuitive, yet effective method for finding the non-trivial similarity between documents.