Workshop on Cross-Framework and Cross-Domain Parser Evaluation Shared Task Manchester, August 23, 2008

(In connection with the 22nd International Conference on Computational Linguistics)

This page is devoted to the shared task at the Coling 2008 workshop on Cross-Framework and Cross-Domain Parser Evaluation.

Jump to:

• Description
• Data Sets
• Obtaining the Data NEW!
• Submissions
• Contact

Lightweight Shared Task

One of the workshop goals is to establish an improved shared knowledge among participants of the strengths and weaknesses of extant annotation and evaluation schemes. In order to create a joint focus and in-depth discussion, there will be a 'lightweight' shared task.

For a selection of 34 sentences (of which ten are considered obligatory, the rest optional) for which gold-standard annotations are provided in several formats potentially useful for parser evaluation, we invite contributors to scrutinize existing gold-standard representations contrastively, identify perceived deficiencies, and sketch what can be done to address these.

As an optional component, participants in the shared task are welcome to include 'native', framework-specific output representations and actual results for a parsing system of their choice (be it their own or not) in the contrastive study. In either case, submissions to the shared task should aim to reflect on the nature of different representations, highlight which additional distinctions are made in either scheme, and discuss why these are useful (for some task) or unmotivated (in general).

The parser evaluation shared task is not a competition or a bake-off, but a collaborative effort intended to provide a focus for discussion on current issues on parser evaluation.

Authors submitting to the workshop can decide whether they want to participate in the shared task or submit a more general paper on parser evaluation (or both). Depending on the volume and distribution of accepted papers, we anticipate that the presentation and discussion of shared task results may account for about half of the available time at the workshop.

Finally, where format conversions have been established successfully, we will establish a repository of existing tools (post-processing scripts, typically) and results that we hope may provide utility beyond the workshop itself.

Back to top

Data

There are three data sets. All shared task participants are required to use the first data set, which will serve as a common basis for discussion and comparison of different representations.

1. WSJ 10 sentences (required)

   This set contains 10 sentences from the Wall Street Journal portion of the Penn Treebank. The following representation formats are provided (external links to format descriptions):

   • Penn Treebank (PTB): phrase structure trees.
   • CoNLL-2008 shared task (CoNLL08): labeled syntactic dependencies extracted from the PTB annotations, and predicate-argument dependencies extracted from PropBank and NomBank.
   • RASP Grammatical Relations (GR): the Grammatical Relation scheme proposed by Briscoe, Carroll and colleagues for parser evaluation.
   • UTokyo HPSG Treebank Predicate-Argument structures (HPSG-PA): predicate-argument dependencies extracted from the University of Tokyo HPSG Treebank.
   • CCGBank Predicate-Argument structures (CCG-PA): predicate-argument dependencies extracted from the CCGBank.
   • PARC Dependency structures (PARC): dependency structures in the scheme used by King et al. in the PARC 700 dependency bank.
   • Stanford Dependencies (Stanford): Dependencies in the scheme designed by de Marneffe et al. for representation of typed dependencies from PTB structures.
2. WSJ 15 sentences (optional)

   This set contains an additional 15 sentences from the Wall Street Journal portion of the Penn Treebank.

   Annotation is provided in the same formats as above, except for PARC.
   

3. GENIA 9 sentences (optional)

   This set contains 9 sentences from the GENIA Treebank, which contains syntactic structures for abstracts of biomedical papers. The sentences in this data set correspond to one abstract from the GENIA Treebank.

   Annotation is provided in the following formats: PTB, GR, HPSG-PA.

4. BNC 10 sentence (optional) NEW!

   This set contains 10 sentences from the British National Corpus.

   Annotation is provided in the follwing formats: PTB and PARC.

Back to top

Obtaining the Data

The data sets are now available for download: PEsharedtask-data.tgz.

Back to top

Submissions

Shared task submission deadline: May 5, 2008.

Submissions to the shared task should discuss any aspect of parser evaluation using one or more of the representation formats provided. In their submissions, participants are required to discuss specific examples from the 10 WSJ sentences in the obligatory data set. Discussions that include a contrastive comparison between two or more formats are encouraged, although discussions that focus on an in-depth analysis of the strengths and weaknesses of one of the formats are also appropriate.

Shared task submissions should follow the same guidelines as submissions for the workshop:

All submissions must be formatted according to the standard templates for the main conference. Contrary to the main conference, however, submissions must not exceed a length of six pages, excluding bibliographic references (i.e. the bibliography is not counted against the page limit). For papers accepted for presentation at the workshop, we anticipate making available another few pages, so as to enable authors to accommodate reviewer suggestions, as appropriate. Submissions must be entered into the on-line START system on or before the submission deadline (May 5).

Back to top

Contact

If you have any questions or comments, please send email to the shared task organizers at pest AT delph-in DOT net.

Back to top

This page is maintained by Yusuke Miyao and Kenji Sagae.