Capsule networks as recurrent models of grouping and segmentation

Adrien Doerig; Lynn Schmittwilken; Bilge Sayim; Mauro Manassi; Michael H herzog

doi:10.1371/journal.pcbi.1008017

Capsule networks as recurrent models of grouping and segmentation

Adrien Doerig^* (Corresponding Author), Lynn Schmittwilken, Bilge Sayim, Mauro Manassi, Michael H herzog

^*Corresponding author for this work

Psychology

Research output: Contribution to journal › Article › peer-review

23 Citations (Scopus)

6 Downloads (Pure)

Abstract

Classically, visual processing is described as a cascade of local feedforward computations. Feedforward Convolutional Neural Networks (ffCNNs) have shown how powerful such models can be. However, using visual crowding as a well-controlled challenge, we previously showed that no classic model of vision, including ffCNNs, can explain human global shape processing. Here, we show that Capsule Neural Networks (CapsNets), combining ffCNNs with recurrent grouping and segmentation, solve this challenge. We also show that ffCNNs and standard recurrent CNNs do not, suggesting that the grouping and segmentation capabilities of CapsNets are crucial. Furthermore, we provide psychophysical evidence that grouping and segmentation are implemented recurrently in humans, and show that CapsNets reproduce these results well. We discuss why recurrence seems needed to implement grouping and segmentation efficiently. Together, we provide mutually reinforcing psychophysical and computational evidence that a recurrent grouping and segmentation process is essential to understand the visual system and create better models that harness global shape computations.

Original language	English
Article number	e1008017
Number of pages	19
Journal	PLoS Computational Biology
Volume	16
Issue number	7
DOIs	https://doi.org/10.1371/journal.pcbi.1008017
Publication status	Published - 21 Jul 2020

Bibliographical note

Funding: AD was supported by the Swiss National Science Foundation grant n.176153 “Basics of visual processing: from elements to figures”. The funders had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript.

Data Availability: The human data for experiment 2 and the full code to reproduce all our results are available here: https://github.com/adriendoerig/Capsule-networks-as-recurrent-models-of-grouping-and-segmentation.

Keywords

crowding
visions
psychophysics
visual system
Boats
deep learning
recurrent neural networks
Neural Networks, Computer
Reproducibility of Results
Humans
Computational Biology
Male
Normal Distribution
Algorithms
Models, Biological
Vision, Ocular
Computer Simulation
Image Processing, Computer-Assisted/methods
Female
Pattern Recognition, Visual
FEEDFORWARD
OBJECT RECOGNITION
INTERFERENCE

Access to Document

10.1371/journal.pcbi.1008017Licence: CC BY

Doerig_et_al_PCB_CapsuleNetworksAsRecurrentModels_VoR
Copyright: © 2020 Doerig et al. This is an open access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited. https://creativecommons.org/licenses/by/4.0/
Final published version, 2.59 MBLicence: CC BY

Cite this

@article{b08c9c0d6c554df390fc6aad76709c5a,

title = "Capsule networks as recurrent models of grouping and segmentation",

abstract = "Classically, visual processing is described as a cascade of local feedforward computations. Feedforward Convolutional Neural Networks (ffCNNs) have shown how powerful such models can be. However, using visual crowding as a well-controlled challenge, we previously showed that no classic model of vision, including ffCNNs, can explain human global shape processing. Here, we show that Capsule Neural Networks (CapsNets), combining ffCNNs with recurrent grouping and segmentation, solve this challenge. We also show that ffCNNs and standard recurrent CNNs do not, suggesting that the grouping and segmentation capabilities of CapsNets are crucial. Furthermore, we provide psychophysical evidence that grouping and segmentation are implemented recurrently in humans, and show that CapsNets reproduce these results well. We discuss why recurrence seems needed to implement grouping and segmentation efficiently. Together, we provide mutually reinforcing psychophysical and computational evidence that a recurrent grouping and segmentation process is essential to understand the visual system and create better models that harness global shape computations.",

keywords = "crowding, visions, psychophysics, visual system, Boats, deep learning, recurrent neural networks, Neural Networks, Computer, Reproducibility of Results, Humans, Computational Biology, Male, Normal Distribution, Algorithms, Models, Biological, Vision, Ocular, Computer Simulation, Image Processing, Computer-Assisted/methods, Female, Pattern Recognition, Visual, FEEDFORWARD, OBJECT RECOGNITION, INTERFERENCE",

author = "Adrien Doerig and Lynn Schmittwilken and Bilge Sayim and Mauro Manassi and herzog, {Michael H}",

note = "Funding: AD was supported by the Swiss National Science Foundation grant n.176153 “Basics of visual processing: from elements to figures”. The funders had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript. Data Availability: The human data for experiment 2 and the full code to reproduce all our results are available here: https://github.com/adriendoerig/Capsule-networks-as-recurrent-models-of-grouping-and-segmentation.",

year = "2020",

month = jul,

day = "21",

doi = "10.1371/journal.pcbi.1008017",

language = "English",

volume = "16",

journal = "PLoS Computational Biology ",

issn = "1553-734X",

publisher = "Public Library of Science",

number = "7",

}

TY - JOUR

T1 - Capsule networks as recurrent models of grouping and segmentation

AU - Doerig, Adrien

AU - Schmittwilken, Lynn

AU - Sayim, Bilge

AU - Manassi, Mauro

AU - herzog, Michael H

N1 - Funding: AD was supported by the Swiss National Science Foundation grant n.176153 “Basics of visual processing: from elements to figures”. The funders had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript. Data Availability: The human data for experiment 2 and the full code to reproduce all our results are available here: https://github.com/adriendoerig/Capsule-networks-as-recurrent-models-of-grouping-and-segmentation.

PY - 2020/7/21

Y1 - 2020/7/21

N2 - Classically, visual processing is described as a cascade of local feedforward computations. Feedforward Convolutional Neural Networks (ffCNNs) have shown how powerful such models can be. However, using visual crowding as a well-controlled challenge, we previously showed that no classic model of vision, including ffCNNs, can explain human global shape processing. Here, we show that Capsule Neural Networks (CapsNets), combining ffCNNs with recurrent grouping and segmentation, solve this challenge. We also show that ffCNNs and standard recurrent CNNs do not, suggesting that the grouping and segmentation capabilities of CapsNets are crucial. Furthermore, we provide psychophysical evidence that grouping and segmentation are implemented recurrently in humans, and show that CapsNets reproduce these results well. We discuss why recurrence seems needed to implement grouping and segmentation efficiently. Together, we provide mutually reinforcing psychophysical and computational evidence that a recurrent grouping and segmentation process is essential to understand the visual system and create better models that harness global shape computations.

AB - Classically, visual processing is described as a cascade of local feedforward computations. Feedforward Convolutional Neural Networks (ffCNNs) have shown how powerful such models can be. However, using visual crowding as a well-controlled challenge, we previously showed that no classic model of vision, including ffCNNs, can explain human global shape processing. Here, we show that Capsule Neural Networks (CapsNets), combining ffCNNs with recurrent grouping and segmentation, solve this challenge. We also show that ffCNNs and standard recurrent CNNs do not, suggesting that the grouping and segmentation capabilities of CapsNets are crucial. Furthermore, we provide psychophysical evidence that grouping and segmentation are implemented recurrently in humans, and show that CapsNets reproduce these results well. We discuss why recurrence seems needed to implement grouping and segmentation efficiently. Together, we provide mutually reinforcing psychophysical and computational evidence that a recurrent grouping and segmentation process is essential to understand the visual system and create better models that harness global shape computations.

KW - crowding

KW - visions

KW - psychophysics

KW - visual system

KW - Boats

KW - deep learning

KW - recurrent neural networks

KW - Neural Networks, Computer

KW - Reproducibility of Results

KW - Humans

KW - Computational Biology

KW - Male

KW - Normal Distribution

KW - Algorithms

KW - Models, Biological

KW - Vision, Ocular

KW - Computer Simulation

KW - Image Processing, Computer-Assisted/methods

KW - Female

KW - Pattern Recognition, Visual

KW - FEEDFORWARD

KW - OBJECT RECOGNITION

KW - INTERFERENCE

UR - http://www.scopus.com/inward/record.url?scp=85089127596&partnerID=8YFLogxK

U2 - 10.1371/journal.pcbi.1008017

DO - 10.1371/journal.pcbi.1008017

M3 - Article

C2 - 32692780

SN - 1553-734X

VL - 16

JO - PLoS Computational Biology

JF - PLoS Computational Biology

IS - 7

M1 - e1008017

ER -

Capsule networks as recurrent models of grouping and segmentation

Abstract

Bibliographical note

Keywords

Access to Document

Other files and links

Fingerprint

Mauro Manassi

Cite this

Capsule networks as recurrent models of grouping and segmentation

Abstract

Bibliographical note

Keywords

Access to Document

Other files and links

Fingerprint

Profiles

Mauro Manassi

Cite this