LEX – Lexical Anaysis

LEX – Lexical Word Analysis Module and MAP1DIC, LST1DIC dictionaries

De-stems inflected items and decomposes compound and/or composite vocabulary items in the text stream (e.g., “schoolhouse,” “houselike”, “coldish”, etc.). Removes high frequency inflectional/conjugation endings and attaches them to their corresponding dictionary entries so that subsequent analysis knows that they were present on the original form.

LEX also normalizes all irregular forms, e.g. “goed” for “went” so that their linguistic composition is consistent with run-of-the-mill forms. Such forms are maintained in an editable list.

Special note: The LEX module is actually the interpreter for a specialized programming language of the same name whose lines of code are comprised of specific commands for assembling/disassembling words.

The lexical MAP1DIC dictionaries are comprised of over 78,000 entries based on Webster’s New Ninth Collegiate Dictionary with associated JG categorizations. The Lexical Analysis module also relies on the LST1DIC dictionary which is also included.