CORPUS TOOLS FOR TYPE-LOGICAL GRAMMARS
INTRODUCTION
This is the first public release of my suite of corpus tools for
type-logical grammars. At the moment, it is a considerably less
polished (and more idiosyncratic) collection of utilities than Grail 3 which
the collection is designed to work with. I hope people will find it
useful nonetheless.
This research has received the generous
support from the Conseil Regional d'Aquitaine, and benefited from the grants
Grammaire du Français and Itipy.
All source code and model files are provided under the GNU lesser general public license and downloading any of these files constitutes an agreement to all the terms of this license.
MODELS
The models files below are meant to be used in conjunction with the Clark &
Curran tools, which I highly recommend.
FRENCH MODELS
The French grammars as well as the procedure followed for
extracting it are described in my TALN 2010 articles.
Moot, R. (2010), Semi-automated Extraction of a Wide-Coverage Type-Logical
Grammar for French, Proceedings of Traitement Automatique des Langues
Naturelles 2010,
(TALN 2010), (pdf).
DUTCH MODELS
The Dutch grammar as well as the extraction procedure are described in
the following book chapter;
Moot, R. (2010), Automated extraction of type-logical supertags from the Spoken Dutch Corpus, in Srinivas Bangalore and Aravind Joshi, eds, `Complexity of Lexical Descriptions and its Relevance to Natural Language Processing: A Supertagging
Approach', MIT Press.
- A tarbal of the part-of-speech model
directory for the Dutch language, trained using the Spoken Dutch Corpus
using the detailed tagset of 320 POS tags (96.8% of which are
correctly assigned on unseen data).
- A tarbal of the part-of-speech model
directory for the Dutch language, trained using the Spoken Dutch Corpus
using the reduced tagset of 70 POS tags (using the reduced set of 64
POS tags, with the SPEC tag separated into its sever distinct forms
as used for the detailed POS tag set, it assigns 97.6% of POS tags
correctly on unseen data).
- A tarbal of the supertag model
directory for the Dutch language, trained using the Spoken Dutch Corpus
and a set of 4,457 supertags (82.0% of which are
correctly assigned on unseen data).
UTILITIES
The different corpus utilities are classified below according to their function.
CORPUS DISPLAY
- portray.tcl is a small TclTk program for displaying the CGN annotation graphs. Note that if you plan on using TIGERSearch, a graph display option is included in that program as well.
CORPUS TRANSFORMATION
- renumber.tcl compiles several .syn files into one big one, renumbering the sentences; useful for producing one big input file for TIGERSearch.
- filtercgn converts a .syn file into a .out file, where `orphan' nodes, that is isolated nodes without either a parent or children, have been removed.
- splitsent converts a .syn file containing sentences with multiple roots into a file where each connected component is a proper sentence.
LEXICON EXTRACTION
- cgntoform is a parametric extraction algorithm which transforms an annotated treebank into a lexicon in Prolog-like syntax.
MAXIMUM ENTROPY
These scripts perform several steps for transform the extracted lexicon into a format suitable for the maximum entropy supertagger and for evaluation the supertagger on the test data.
- a Makefile performs all the operations needed for extracting a grammar and training and testing a maximum entropy supertagger.
- treecount counts the number of occurrences of different formulas in the lexicon.
- cutoff2forms transforms the output of treecounts into a number of auxiliary files necessary for further processing.
- formula2counts a Prolog program for creating a `counts' file, indicating for each formula occuring in the lexicon and for each atomic formula the sum of its positive and negative occurrences.
- lex2me uses the auxiliary files produced by cutoff2forms to produce the input files for training and testing the maximum entropy supertagger.
- partition separates the phrases in a maximum entropy sentence file into training and test phrases.
- evalkmaxent compares the test data tagged by the supertagger to the correct test data and outputs the results.
- eval2tex transforms the output of the evaluation script into a bar graph to be typeset with LaTeX.
SUPERTAGGER
A series of Java files based on and requiring an installation of the Edinburgh maximum entropy tools. Used in coordination with the scripts above for training and evaluating the supertagger.
CORPUS SEARCH
These scripts provide some search functions I needed in order to get an idea of some of the constructions found in the corpus and the number of occurrences.
- findlex find occurrences of a pattern of lexical entries in a series of lexicon files, for example "de-*-die" will find all instances of the word "de" followed by any other word followed by the word "die".
- subvertices lists all occurences of a vertex label in the annotation files. For example "subvertices du filenames" will list all discourse units in the corpus files.
- subedges lists all occurences of an edge label in the annotation files. For example "subedges lp filenames" will enumerate all list parts in the corpus files.
- childedges generates a list of all edges which occur as child of a parent vertex.
- subparentedges generates all occurrences of an edge-parent pair. For example "subparentedges tag du" will list all constituents marked as tag which are daughters of a discourse unit.
- sisternodes generates all instances of two nodes occuring as sisters, specified by vertex or edge label, or both. For example "sisternodes tw1 det '*' hd" will generate all numerals which function as determiner which are sisters of any category which functions as a head.
ERROR TRACKING
These scripts compare a tagged file to a correct file in order to find instances of a specific error made by the supertagger. This may help improving the maximum entropy models either by adding new features or by improving the extraction algorithm.
- finderror finds instances of one supertag being mistaken for another.
- findworderror finds instances of one word being assigned an incorrect supertag.
MODELS
DOWNLOAD
All files in one big tarball.
OTHER CORPUS RESOURCES
- The TIGERSearch corpus search tools have aided me greatly in finding constructions in the corpora I've worked with.
- The Edinburgh maximum entropy tools is an excellent Java package for training and using maximum entropy models. My supertagger requires these tools and is to a great extent inspired by their POS tagger.
Richard.Moot@labri.fr