References to graphical objects in interactive multimodal queries

Daqing He, Graeme Ritchie, John Lee

Research output: Contribution to journalArticle

1 Citation (Scopus)


Traditionally, interactive natural language systems assume a semantic model in which the entities referred to are in some abstract representation of a real or imagined world. In a system where graphical objects such as diagrams may be on the screen during the language interaction, there is possibility that the user may want to allude to these visual entities. Since graphical objects have their own properties (colour, shape, position on the screen, etc.) but may also represent items in a knowledge base which have other associated properties (price, geographical location, technical specifications, etc.), some systematic way is needed to enable such objects to be referred to in terms of either their screen properties or their associated attributes from the domain under discussion. In this paper, we present a formalisation for these arrangements, and show how our logical definitions can be used to generate constraints suitable for reference resolution within a natural language interpreter. (c) 2008 Elsevier B.V. All rights reserved.

Original languageEnglish
Pages (from-to)617-628
Number of pages12
JournalKnowledge-Based Systems
Issue number7
Early online date31 Mar 2008
Publication statusPublished - Oct 2008


  • Source ambiguities
  • Intelligence multimodal interfaces
  • Constraint satisfaction problems
  • Reference resolution


Dive into the research topics of 'References to graphical objects in interactive multimodal queries'. Together they form a unique fingerprint.

Cite this