An evaluation of concept suggestion strategies for professional multimedia archives

Choosing the optimal terms to represent a search engine query is not trivial, and may involve an iterative process such as relevance feedback, repeated unaided attempts by the user or the automatic suggestion of additional terms, which the user may select or reject. This is particularly true of a multimedia search engine which searches on concepts as well as user-input terms, since the user is unlikely to be familiar with all the system-known concepts. We propose three concept suggestion strategies: suggestion by normalised textual matching, by semantic similarity, and by the use of a similarity matrix. We have evaluated these three strategies by comparing machine suggestions with the suggestions produced by professional annotators, using the measures of micro- and macro- precision and recall. The semantic similarity strategy outperformed the use of a similarity matrix at a range of thresholds. Normalised textual matching, which is the simplest strategy, performed almost as well as the semantic similarity one on recall-based measures, and even better on precision-based and F-based measures.