Multimodal Image and Spectral Feature Learning for Efficient Analysis of Water-Suspended Particles

Tomoko Takahashi* (Corresponding Author), Zonghua Liu, Thangavel Thevar, Nicholas Burns, Dhugal Lindsay, John Watson, Sumeet Mahajan, Satoru Yukioka, SHUHEI TANAKA, YUKIKO NAGAI, Blair Thornton

*Corresponding author for this work

Research output: Contribution to journalArticlepeer-review


We have developed a method to combine morphological and chemical information for the accurate identification of different particle types using optical measurement techniques that require no sample preparation. A combined holographic imaging and Raman spectroscopy setup is used to gather data from six different types of marine particles suspended in a large volume of seawater. Unsupervised feature learning is performed on the images and the spectral data using convolutional and single layer autoencoders. The learned features are combined, where we demonstrate that non-linear dimensional reduction of the combined multimodal features can achieve a high clustering macro F1 score of 0.88, compared to a maximum of 0.61 when only image or spectral features are used. The method can be applied to long-term monitoring of particles in the ocean without the need for sample collection. In addition, it can be applied to data from different types of sensor measurements without significant modifications.
Original languageEnglish
Pages (from-to)7492-7504
Number of pages13
JournalOptics Express
Issue number5
Publication statusPublished - 27 Feb 2023

Bibliographical note

apan Science and Technology Agency SICORP and Natural Environment Research Council (JST-NERC SICORP Marine Sensor Proof of Concept Grant JPMJSC1705, NE/R01227X/1); JSPS KAKENHI Grant (18K13934 and 18H03810); Sumitomo Foundation: Grant for environmental Research Project (203122).
Acknowledgments. The authors thank Dr. T. Fukuba for the support for building the experimental setup. The authors also thank Dr. H. Sawada for providing samples for this work.

Data Availability Statement

Data underlying the results presented in this paper are not publicly available at this time but may be obtained from the authors upon reasonable request.


Dive into the research topics of 'Multimodal Image and Spectral Feature Learning for Efficient Analysis of Water-Suspended Particles'. Together they form a unique fingerprint.

Cite this