Experience of using GATE for NLP R&D.

Hamish Cunningham, Diana Maynard,
Kalina Bontcheva, Valentin Tablan, Yorick Wilks
Department of Computer Science and
Institute for LAnguage, Speech and Hearing,
University of Sheffield, UK
{hamish,diana,kalina,valyt,yorick}@dcs.shef.ac.uk

Abstract

1  Introduction

This paper relates experiences in projects that have used GATE (General Architecture for Text Engineering) over the four years since its initial release in 1996.

We begin in section 2 with some of the motivation behind this type of system, and then give a definition of architecture in this context (section 3). Section 4 briefly describes GATE; section 5 covers a range of projects that have used the system; and section 6 examines some additional ways in which GATE has been used. These experiences form the input to section 7, which discusses the system's strengths and weaknesses.

2  Motivation

If you're researching human language processing you should probably not be writing code to:

• store data on disk;
• display data;
• load processor modules and data stores into processes;
• initiate and administer processes;
• divide computation between client and server;
• pass data between processes and machines.

A Software Architecture for language processing should do all this for you. You will have to parameterise it, and sometimes deployment of your work into applications software will require some low-level fiddling for optimisation purposes, but in the main these activities should be carried out by infrastructure for the language sciences, not by each researcher in the field.

We can go further and say that you shouldn't have to reinvent components and resources outside of your specialism if there is already something that could do the job. A statistician doesn't need to know the details of the IEEE Floating Point computation standard; a discourse processing specialist doesn't need to understand all the ins and outs of part-of-speech tagging (or worse still how to install a particular POS tagger on a particular machine).

If you're a professional mathematician, you probably regard a tool like SPSS or Mathematica as necessary infrastructure for your work. If you're a computational linguist or a language engineer, the chances are that large parts of your work have no such infrastructural support. Where there is infrastructure, it tends to be specific to restricted areas. GATE, a General Architecture for Text Engineering [ Cunningham et al. 1997], represents an attempt to fill this gap, and is a software architecture for language processing R&D.

We now have four years of experience with GATE, work on which began in 1995, with a first widespread release late in 1996. The system is currently at a pivotal point in its development, with a new version due for release. The experiences reported in this paper contribute to the requirements analysis and design of the new system.

3  Infrastructure for Language Processing R&D

What does infrastructure mean for Natural Language Processing (NLP)? What sorts of tasks should be delegated to a general tool, and which should be left to individual projects? The position we took in designing GATE is to focus on the common elements of NLP systems.

There are many useful tools around for performing specific tasks such as developing feature structure grammars for evaluation under unification, or collecting statistical measures across corpora. To varying extents, they entail the adoption of particular theories. The only common factor of NLP systems, alas, seems to be that they very often create information about text. Developers of such systems create modules and data resources that handle text, and they store this data, exchange it between various modules, compare results of test runs, and generally spend inordinate amounts of time pouring over samples of it when they really should be enjoying a slurp of something relaxing instead.

The types of data structure typically involved are large and complex, and without good tools to manage and allow succinct viewing of the data we work below our potential. At this stage in the progress of our field, no one should really have to write a tree viewing program for the output of a syntax analyser, for example, or even have to do significant work to get an existing viewing tool to process their data.

In addition, many common language processing tasks have been solved to an acceptable degree by previous work and should be reused. Instead of writing a new part of speech tagger, or sentence splitter, or list of common nominal compounds, we should have available a store of reusable tools and data that can be plugged into our new systems with minimal effort. Such reuse is much less common than it should be, often because of installation and integration problems that have to be solved afresh in each case [ Cunningham et al. 1994].

In sum, we define our infrastructure as an architecture, framework and development environment, where an architecture is a macro-level organisational pattern for the components and data resources that make up a language processing system; a framework is a class library implementing the architecture; a development environment adds graphical tools to access the services provided by the architecture.

4  GATE

GATE version 1.n does three things:

• manages textual data storage and exchange;
• supports visual assembly and execution of modular NLP systems plus visualisation of data structures associated with text;
• provides plug-in modularity of text processing components.

The architecture does this using three subsystems:

• GDM, the GATE Document Manager;
• GGI, the GATE Graphical Interface;
• CREOLE, a Collection of REusable Objects for Language Engineering.

GDM manages the information about texts produced and consumed by NLP processes; GGI provides visual access to this data and manages control flow; CREOLE is the set of resources so far integrated. Developers working with GATE begin with a subset of CREOLE that does some basic tasks, perhaps tokenisation, sentence and paragraph identification and part-of-speech tagging. They then add or modify modules for their specific tasks. They use a single API for accessing the data and for storing their data back into the central database. With a few lines of configuration information they allow the system to display their data in friendly graphical form, including tree diagrams where appropriate. The system takes care of data storage and module loading, and can be used to deliver embeddable subsystems by stripping the graphical interface. It supports modules in any language including Prolog, Lisp, Perl, Java, C++ and Tcl.

5  Projects involving GATE

Over the past 4 years, GATE has been used in a number of projects, not only within the University of Sheffield, but also by a variety of external institutions. In this section, we outline some of the main projects that have used GATE, and examine its performance in each case.

5.1  ECRAN

5.2  Cass-SWE

5.3  GIE

5.4  AVENTINUS

5.5  SVENSK

5.6  LOTTIE

5.7  EUDICO

5.8  LaSIE

5.9  EMPathIE

5.10  TRESTLE

5.11  PASTA

5.12  German Named Entity Recognition

6  Usage of GATE

In addition to the projects described above, GATE has been used as a research tool, for applications development and for teaching. Some examples of these uses are given below.

Our colleagues in the Universities of Edinburgh, UMIST in Manchester, and Sussex (amongst others) have reported using the system for teaching, and the University of Stuttgart produced a tutorial in German for the same purposes (see http://www.dcs.shef.ac.uk/gate/contrib/michael.dorna).

Numerous postgraduates in locations as diverse as Israel, Copenhagen and Surrey are using the system in order to avoid having to write simple things like sentence splitters from scratch, and to enable visualisation and management of data.

Turning to applications development, ESTEAM Inc., of Gothenburg and Athens are using the system for adding name recognition to their MT systems (for 26 language pairs) to improve performance on unknown words.

Syntalex Ltd., of London, are developing a product that automatically applies amendments to legal documents within the GATE framework.

Both British Gas Research and Master Foods NV (owner of the Mars confectionery brand) used the LaSIE system for competitor intelligence systems. LaSIE was configured as an embedded library using GATE's deployment facilities.

[ Gotoh et al. 1998] report experiments using Named Entity tagged language models for large vocabulary connected speech recognition. The modelling data was created by running LaSIE as a batch process using GATE's command line interface.

Competitor intelligence researchers in Finland are using GATE in the BRIEFS project [ Keijola1999] (Brief driven Information retrieval and Extraction for Strategy), which concerns business intelligence for companies. Although they had some initial difficulties getting the system to run on their platform (Linux Red Hat 6.0), they were able to get good results in a short period of time, which was one of their main reasons for choosing GATE in the first place¹.

The Polytechnic of Catalunya in Barcelona have used GATE as part of the ITEM project for multilingual IE and retrieval for Spanish, Catalan and Basque. They used GATE as a framework for integrating their tools for different languages, and as a visualisation tool for the results. They found it particularly useful for providing a user-friendly environment for non-experts, although its slow speed in processing large corpora was a problem.²

GATE has also been used for Information Extraction in English, German, French, Spanish, Greek and Swedish.

In these and other projects, researchers have contributed a diverse and growing set of components to CREOLE, the collection of language resources.

CREOLE now includes:

• the VIE³ English IE components (tokeniser and text structure analysers; sentence splitter; several POS taggers; morphological analyser; chart parser; name matcher; discourse interpreter);

• the Alvey Natural Language Tools morphological analyser and parser;

• the Plink parser;

• the Parseval tree comparison software;

• the MUC scoring tools;

• French parsing and morphological analysis tools from Fribourg and INRIA; Italian corpus analysis tools from Rome Tor Vergata;

• a wide range of Swedish language processing tools.

The main deficiency in this set is a bias towards language analysis, and towards Processing Resources above Language Resources.

A partial list of GATE licensees (of which there were over 300 at the end of 1999) is available at http://www.dcs.shef.ac.uk/nlp/gate/users.html. Along with the experiences cited above, this list indicates that take-up of the system is healthy and that LE R&D workers have found the system useful in many contexts.

7  Strengths and Weaknesses

The strengths of the final version 1 release of GATE are that it:

• facilitates reuse of NLP components by reducing the overheads of integration, documentation and data visualisation;

• facilitates multi-site collaboration on IE research by providing a modular base-line system (VIE) with which others can experiment;

• facilitates comparative evaluation of different methods by making it easy to interchange components;

• facilitates task-based evaluation, both of `internal' components such as taggers and parsers, and of whole systems, e.g. by using materials from the ARPA MUC programme [ Grishman and Sundheim1996] (whose scoring software is available in GATE, as is the Parseval tree scoring tool [ Harrison1991], and a generic annotation scoring tool [ Rodgers et al. 1997]);

• provides a reasonably easy to use, rich graphical interface;

• contributes to increased software-level robustness, quality and efficiency in NLP systems, and to their cross-platform portability (all UNIX systems, Windows NT and Windows 95; native support for Java, C++ and Tcl);

• contributes to the portability of NLP systems across problem domains by providing a markup tool for generating training data for example-based learning (it can also take input from the Alembic tool [ Day et al. 1997] for this purpose);

• unifies the two best approaches to managing information about text by combining a TIPSTER-style database with SGML input/output filters (developed using tools from Edinburgh's Language Technology Group [ McKelvie et al. 1997]).

The principal problems with version 1 are that:

• It is biased towards algorithmic components for language processing, and neglects data resource components (PRs vs. LRs).

• It is biased towards text analysis components, and neglects text generation components.

• The database implementation is space and time inefficient.

• The visual interface is complex and somewhat non-standard.

• The task graph generation and management process does not scale beyond small component sets: ``GGI suffers from the scaling problem [ Burnett et al. 1987], as the size of the custom graph quickly becomes unmanageable'' [ Rodgers et al. 1997].

• Only the annotator component model is extensible; adding new viewers or tools is not possible.

• Installing and supporting the system is a skilled job, and it runs better on some platforms than on others (UNIX vs. Windows).

• Sharing of components depends on sharing of annotation definitions (but isomorphic transformations are relatively easy to implement).

• It only caters for textual documents, not for multi-media documents.

• It only supports 8-bit character sets.

• Module reset cascades through all previously run PRs that made non-monotonic database updates.

Work is currently in progress on Version 2 of GATE, which aims to combat some of these problems and extend the range of the system [ Cunningham2000]. Further details of requirements for this type of system, and how to evaluate them, can be found in [ Cunningham et al. 2000].

8  Conclusion

Based on the collective experiences of a sizeable user base across the EU and elsewhere, the system can claim to be a viable infrastructure for certain sections of the field. Given further development, we hope that it can take on this role for a wider variety of tasks.

9  Acknowledgements

This work was supported by EPSRC grants GR/K25267 and GR/M31699.

References

Footnotes:

¹ (Matti Keijola, personal communication, October 1999)

² (Jordi Batalla, personal communication, March 1999.)

³ A `Vanilla IE' system related to LaSIE