SEMANTIC RELATEDNESS CALCULATION METHOD // МЕТОД ВИЗНАЧЕННЯ СЕМАНТИЧНОЇ ЗВ'ЯЗНОСТІ (original) (raw)
The article describes a new method for determination of semantic relatedness. The method is based on statistical data collected from text corpora and principles of distributive semantics. A set of basic hypotheses lies at the basis of the method. Each hypothesis is a feature of semantic relatedness itself and can be used separately from the method. Total offered more than 170 hypotheses (including sub-hypothesis). The main hypotheses can be split on the following classes: 1. Basic hypotheses - reflects the frequency characteristics of the common occurrences of the words. 2. Hypotheses with normalization by length of the context. 3. Hypotheses with normalization by the number of words. 4. Distances based hypotheses. 5. Hypotheses with normalization by the number of documents. 6. Hypotheses based on the calculation of weighted distances on a graph 7. A set of methods with variational calculation of the combined information. 8. A set of methods with logarithm of values and variational calculation of the combined information. 9. Hypotheses with different PMI modefications. 10. Hypotheses that calculate relatedness only for words with common occurrences above the certain thresholds and mixed hypotheses based on PMI, calculated over statistics from other hypotheses. The article shows graphs for Spearman and Pearson correlations for each hypotheses on the benchmark sets. The plots contain marked boundaries for hypothesis’s classes and a correlation for each class. Also, noted various behavior of the same hypothesis on different benchmark sets, which confirms our observation of the different nature of the benchmark sets. Further, on the basis of the proposed hypotheses, created aggregating model for evaluating semantic relatedness. Model measured on all benchmark sets. Received ratings exceeded the evaluation of other existing methods, this confirms the effectiveness of the proposed model.
Sign up for access to the world's latest research.
checkGet notified about relevant papers
checkSave papers to use in your research
checkJoin the discussion with peers
checkTrack your impact
Loading Preview
Sorry, preview is currently unavailable. You can download the paper by clicking the button above.