TY - CHAP
T1 - Detecting inflection patterns in natural language by minimization of morphological model
AU - Gelbukh, Alexander
AU - Alexandrov, Mikhail
AU - Han, Sang Yong
PY - 2004
Y1 - 2004
N2 - One of the most important steps in text processing and information retrieval is stemming - reducing of words to stems expressing their base meaning, e.g., bake, baked, bakes, baking → bak-. We suggest an unsupervised method of recognition such inflection patterns automatically, with no a priori information on the given language, basing exclusively on a list of words extracted from a large text. For a given word list V we construct two sets of strings: stems S and endings E, such that each word from V is a concatenation of a stem from S and ending from E. To select an optimal model, we minimize the total number of elements in S and E. Though such a simplistic model does not reflect many phenomena of real natural language morphology, it shows surprisingly promising results on different European languages. In addition to practical value, we believe that this can also shed light on the nature of human language.
AB - One of the most important steps in text processing and information retrieval is stemming - reducing of words to stems expressing their base meaning, e.g., bake, baked, bakes, baking → bak-. We suggest an unsupervised method of recognition such inflection patterns automatically, with no a priori information on the given language, basing exclusively on a list of words extracted from a large text. For a given word list V we construct two sets of strings: stems S and endings E, such that each word from V is a concatenation of a stem from S and ending from E. To select an optimal model, we minimize the total number of elements in S and E. Though such a simplistic model does not reflect many phenomena of real natural language morphology, it shows surprisingly promising results on different European languages. In addition to practical value, we believe that this can also shed light on the nature of human language.
UR - http://www.scopus.com/inward/record.url?scp=24344505975&partnerID=8YFLogxK
U2 - 10.1007/978-3-540-30463-0_54
DO - 10.1007/978-3-540-30463-0_54
M3 - Capítulo
AN - SCOPUS:24344505975
SN - 3540235272
T3 - Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)
SP - 432
EP - 438
BT - Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)
A2 - Sanfeliu, Alberto
A2 - Martinez-Trinidad, Jose Francisco
A2 - Carrasco-Ochoa, Jesus Ariel
PB - Springer Verlag
ER -