DUTCH MODELS
The Dutch grammar as well as the extraction procedure are described in
the following book chapter;
Moot, R. (2010), Automated extraction of type-logical supertags from the Spoken Dutch Corpus, in Srinivas Bangalore and Aravind Joshi, eds, `Complexity of Lexical Descriptions and its Relevance to Natural Language Processing: A Supertagging
Approach', MIT Press.
- A tarbal of the part-of-speech model
directory for the Dutch language, trained using the Spoken Dutch Corpus
using the detailed tagset of 320 POS tags (96.8% of which are
correctly assigned on unseen data).
- A tarbal of the part-of-speech model
directory for the Dutch language, trained using the Spoken Dutch Corpus
using the reduced tagset of 70 POS tags (using the reduced set of 64
POS tags, with the SPEC tag separated into its distinct forms
as used for the detailed POS tag set, it assigns 97.6% of POS tags
correctly on unseen data).
- A tarbal of the supertag model
directory for the Dutch language, trained using the Spoken Dutch Corpus
and a set of 4,457 supertags (82.0% of which are
correctly assigned on unseen data). This is a version of the
split extraction procedure described in the article, but using both a more
detailed supertag set (hence the supertag set which is around double
the size) and having (considerably) superior performance in spite of
this (due to a slightly cleaner tagset).
UTILITIES
The different corpus utilities are classified below according to their function.
CORPUS DISPLAY
- portray.tcl is a small TclTk program for displaying the CGN annotation graphs. Note that if you plan on using TIGERSearch, a graph display option is included in that program as well.
CORPUS TRANSFORMATION
- renumber.tcl compiles several .syn files into one big one, renumbering the sentences; useful for producing one big input file for TIGERSearch.
- filtercgn converts a .syn file into a .out file, where `orphan' nodes, that is isolated nodes without either a parent or children, have been removed.
- splitsent converts a .syn file containing sentences with multiple roots into a file where each connected component is a proper sentence.
LEXICON EXTRACTION
- cgntoform is a parametric extraction algorithm which transforms an annotated treebank into a lexicon in Prolog-like syntax.
MAXIMUM ENTROPY
These scripts perform several steps for transform the extracted lexicon into a format suitable for the maximum entropy supertagger and for evaluation the supertagger on the test data.
- a Makefile performs all the operations needed for extracting a grammar and training and testing a maximum entropy supertagger.
- treecount counts the number of occurrences of different formulas in the lexicon.
- cutoff2forms transforms the output of treecounts into a number of auxiliary files necessary for further processing.
- formula2counts a Prolog program for creating a `counts' file, indicating for each formula occuring in the lexicon and for each atomic formula the sum of its positive and negative occurrences.
- lex2me uses the auxiliary files produced by cutoff2forms to produce the input files for training and testing the maximum entropy supertagger.
- partition separates the phrases in a maximum entropy sentence file into training and test phrases.
- evalkmaxent compares the test data tagged by the supertagger to the correct test data and outputs the results.
- eval2tex transforms the output of the evaluation script into a bar graph to be typeset with LaTeX.
SUPERTAGGER
A series of Java files based on and requiring an installation of the 