Human-Machine Collaboration in Intelligence Analysis: An Expert Evaluation

Alice Toniolo* (Corresponding Author), Federico Cerutti, Timothy J. Norman, Nir Oren, John A. Allen

*Corresponding author for this work

Research output: Contribution to journalArticlepeer-review

2 Citations (Scopus)


In this paper we illustrate how novel AI methods can improve the performance
of intelligence analysts. These analysts aim to make sense of — often conflicting or incomplete — information, weighing up competing hypotheses which
serve to explain an observed situation. Analysts have access to numerous visual analytic tools which support the temporal and/or conceptual structuring
of information and collection, and on the other hand other tools can be used to
evaluate alternative hypotheses. We believe, however, that there are currently
no tools or methods which allow them to combine the recording and interpretation of information, and that there is little understanding by analysts about how software tools can facilitate the hypothesis formation process. Following the identification of these requirements, we developed the CISpaces (Collaborative Intelligence Spaces) decision support tool in collaboration with professional intelligence analysts. CISpaces combines multiple AI-based methods including argumentation theory, crowdsourced Bayesian analysis, and provenance recording. We show that CISpaces is able to provide support to analysts by facilitating the interpretation of di↵erent types of evidence through argumentation-based reasoning, provenance analysis and crowdsourcing. We undertook an experimental analysis with intelligence analysts which highlights three key points. (1) The
novel, principled AI methods implemented in CISpaces advance performance in
intelligence analysis. (2) While designed as a research prototype (at TRL 3),
analysts benchmarked it against their existing software tools, and we provide
results suggesting intention to adopt CISpaces in analysts’ daily activities. (3)
Finally, the evaluation highlights some drawbacks in CISpaces. However, these
are not due to the technologies underpinning the tool, but rather its lack of
integration with existing organisational standards regarding input and output
formats. Our evaluation with intelligence analysts therefore demonstrates the
potential impact that an integrated tool building on state-of-the-art AI techniques can have on the process of understanding complex situations, and on how such a tool can help focus human e↵ort on identifying more credible interpretations of evidence.
Original languageEnglish
Article number200151
JournalIntelligent Systems with Applications (ISWA)
Early online date8 Dec 2022
Publication statusPublished - Feb 2023

Bibliographical note

We would like to thank the professional analysts from the UK, US and international agencies for their support in developing this research. This research was
sponsored by the U.S. Army Research Laboratory and the U.K. Ministry of Defence and was accomplished under Agreement Number W911NF-06-3-0001. The
views and conclusions contained in this document are those of the author(s) and
should not be interpreted as representing the social policies, either expressed or
implied, of the U.S. Army Research Laboratory, the U.S. Government, the U.K.
Ministry of Defence or the U.K. Government. The U.S. and U.K. Governments
are authorized to reproduce and distribute reprints for Government purposes
notwithstanding any copyright notation hereon.


Dive into the research topics of 'Human-Machine Collaboration in Intelligence Analysis: An Expert Evaluation'. Together they form a unique fingerprint.

Cite this