Towards Embodied Cognition in Robots via Spatially Grounded Synthetic Worlds

Joel Currie; Gioele Migno; Enrico Piacenti; Maria Elena Giannaccini; Patric Bach; Davide De Tommaso; Agnieszka Wykowska

doi:10.48550/ARXIV.2505.14366

Towards Embodied Cognition in Robots via Spatially Grounded Synthetic Worlds

Joel Currie
, Gioele Migno
, Enrico Piacenti
, Maria Elena Giannaccini
, Patric Bach
, Davide De Tommaso
, Agnieszka Wykowska

Italian Institute of Technology

Research output: Working paper › Preprint

33 Downloads (Pure)

Abstract

We present a conceptual framework for training Vision-Language Models (VLMs) to perform Visual Perspective Taking (VPT), a core capability for embodied cognition essential for Human-Robot Interaction (HRI). As a first step toward this goal, we introduce a synthetic dataset, generated in NVIDIA Omniverse, that enables supervised learning for spatial reasoning tasks. Each instance includes an RGB image, a natural language description, and a ground-truth 4X4 transformation matrix representing object pose. We focus on inferring Z-axis distance as a foundational skill, with future extensions targeting full 6 Degrees Of Freedom (DOFs) reasoning. The dataset is publicly available to support further research. This work serves as a foundational step toward embodied AI systems capable of spatial understanding in interactive human-robot scenarios.

Original language	English
Publisher	ArXiv
Number of pages	3
DOIs	https://doi.org/10.48550/ARXIV.2505.14366
Publication status	Published - 20 May 2025

Bibliographical note

Accepted to: Intelligent Autonomous Systems (IAS) 2025 as Late Breaking Report

Dataset availability: We release our synthetic dataset of minimal 3D scenes, each containing an RGB image, a natural language prompt, and a ground-truth 4×4 pose matrix. The dataset [6] is available at: https://huggingface.co/datasets/jwgcurrie/synthetic-distance.

Version History

[v1] Tue, 20 May 2025 13:49:09 UTC (213 KB)

Funding

This work has received support from the Project ”Future Artificial Intelligence Research (hereafter FAIR)”, PE000013 funded by the European Union - NextGenerationEU PNRR MUR - M4C2 - Investimento 1.3 - Avviso Creazione di ”Partenariati estesi alle università, ai centri di ricerca, alle aziende per il finanziamento di progetti di ricerca di base” CUP J53C22003010006.

Funders	Funder number
European Commission	PE000013

Keywords

Visual Perspective Taking
Visual Language Models
Spatial Reasoning
Embodied-AI
Human-Robot Interaction

Access to Document

10.48550/ARXIV.2505.14366Licence: CC BY

2505.14366v1Submitted manuscript, 653 KB

https://huggingface.co/datasets/jwgcurrie/synthetic-distanceLicence: Other

Cite this

@techreport{ddcf4a1c5dd24ab29b2d19bfc7ba5c94,

title = "Towards Embodied Cognition in Robots via Spatially Grounded Synthetic Worlds",

abstract = "We present a conceptual framework for training Vision-Language Models (VLMs) to perform Visual Perspective Taking (VPT), a core capability for embodied cognition essential for Human-Robot Interaction (HRI). As a first step toward this goal, we introduce a synthetic dataset, generated in NVIDIA Omniverse, that enables supervised learning for spatial reasoning tasks. Each instance includes an RGB image, a natural language description, and a ground-truth 4X4 transformation matrix representing object pose. We focus on inferring Z-axis distance as a foundational skill, with future extensions targeting full 6 Degrees Of Freedom (DOFs) reasoning. The dataset is publicly available to support further research. This work serves as a foundational step toward embodied AI systems capable of spatial understanding in interactive human-robot scenarios.",

keywords = "Visual Perspective Taking, Visual Language Models, Spatial Reasoning, Embodied-AI, Human-Robot Interaction",

author = "Joel Currie and Gioele Migno and Enrico Piacenti and Giannaccini, \{Maria Elena\} and Patric Bach and Tommaso, \{Davide De\} and Agnieszka Wykowska",

note = "Accepted to: Intelligent Autonomous Systems (IAS) 2025 as Late Breaking Report Dataset availability: We release our synthetic dataset of minimal 3D scenes, each containing an RGB image, a natural language prompt, and a ground-truth 4{\texttimes}4 pose matrix. The dataset [6] is available at: https://huggingface.co/datasets/jwgcurrie/synthetic-distance. ",

year = "2025",

month = may,

day = "20",

doi = "10.48550/ARXIV.2505.14366",

language = "English",

publisher = "ArXiv",

type = "WorkingPaper",

institution = "ArXiv",

}

TY - UNPB

T1 - Towards Embodied Cognition in Robots via Spatially Grounded Synthetic Worlds

AU - Currie, Joel

AU - Migno, Gioele

AU - Piacenti, Enrico

AU - Giannaccini, Maria Elena

AU - Bach, Patric

AU - Tommaso, Davide De

AU - Wykowska, Agnieszka

N1 - Accepted to: Intelligent Autonomous Systems (IAS) 2025 as Late Breaking Report Dataset availability: We release our synthetic dataset of minimal 3D scenes, each containing an RGB image, a natural language prompt, and a ground-truth 4×4 pose matrix. The dataset [6] is available at: https://huggingface.co/datasets/jwgcurrie/synthetic-distance.

PY - 2025/5/20

Y1 - 2025/5/20

N2 - We present a conceptual framework for training Vision-Language Models (VLMs) to perform Visual Perspective Taking (VPT), a core capability for embodied cognition essential for Human-Robot Interaction (HRI). As a first step toward this goal, we introduce a synthetic dataset, generated in NVIDIA Omniverse, that enables supervised learning for spatial reasoning tasks. Each instance includes an RGB image, a natural language description, and a ground-truth 4X4 transformation matrix representing object pose. We focus on inferring Z-axis distance as a foundational skill, with future extensions targeting full 6 Degrees Of Freedom (DOFs) reasoning. The dataset is publicly available to support further research. This work serves as a foundational step toward embodied AI systems capable of spatial understanding in interactive human-robot scenarios.

AB - We present a conceptual framework for training Vision-Language Models (VLMs) to perform Visual Perspective Taking (VPT), a core capability for embodied cognition essential for Human-Robot Interaction (HRI). As a first step toward this goal, we introduce a synthetic dataset, generated in NVIDIA Omniverse, that enables supervised learning for spatial reasoning tasks. Each instance includes an RGB image, a natural language description, and a ground-truth 4X4 transformation matrix representing object pose. We focus on inferring Z-axis distance as a foundational skill, with future extensions targeting full 6 Degrees Of Freedom (DOFs) reasoning. The dataset is publicly available to support further research. This work serves as a foundational step toward embodied AI systems capable of spatial understanding in interactive human-robot scenarios.

KW - Visual Perspective Taking

KW - Visual Language Models

KW - Spatial Reasoning

KW - Embodied-AI

KW - Human-Robot Interaction

UR - https://arxiv.org/abs/2505.14366

U2 - 10.48550/ARXIV.2505.14366

DO - 10.48550/ARXIV.2505.14366

M3 - Preprint

BT - Towards Embodied Cognition in Robots via Spatially Grounded Synthetic Worlds

PB - ArXiv

ER -

Towards Embodied Cognition in Robots via Spatially Grounded Synthetic Worlds

Abstract

Bibliographical note

Version History

Funding

Keywords

Access to Document

Other files and links

Fingerprint

Vision-language models gain spatial reasoning skills through artificial worlds and 3D scene descriptions

Cite this

Towards Embodied Cognition in Robots via Spatially Grounded Synthetic Worlds

Abstract

Bibliographical note

Version History

Funding

Keywords

Access to Document

Other files and links

Fingerprint

Press/Media

Vision-language models gain spatial reasoning skills through artificial worlds and 3D scene descriptions

Cite this