Klasifikavimo metodų galia: uždaviniai, kuriuos galima išspręsti lietuvių kalbai

ENDespite the existence of many effective computational linguistics methods for such widely used languages as English, there is no answer whether these methods are suitable for the substantially different languages. In this research we attempted to solve the following computational linguistics problems for Lithuanian: topic classification, named entity recognition, sentiment analysis, and dependency parsing. However, these tasks were more complex due to the several reasons. First, we were dealing with the Lithuanian language which is highly inflective, has rich morphology, vocabulary, word derivation system, and relatively free word-order in a sentence. Besides, non-normative Lithuanian language texts - in particular, internet comments or forum posts- usually lack of diacritics, but are full of foreign language insertions and neologisms that even do not exist in the Lithuanian dictionary. Moreover, Lithuanian language is resource-scarce, because grammatical tools (such as diacritics restorations, lemmatizers adjusted for the contemporary language, etc.) and wide range of annotated corpora (labeled with topics, sentiment labels or annotated with named entities) are not yet available. We experimentally investigated different text pre-processing techniques (diacritics replacement, emoticons replacement, etc.), text features (lemmas, grammatical information, word/character n-grams, etc.), and methods (supervised machine learning, dictionary-based, parsing) able to solve our tasks and obtained encouraging results.