
Truth is in the eyes of the solver: an eye-tracking study of three-dimensional chemical structure problems
InfraVis User
Karolina Broman (UmU),
Anders Hofverberg(UmU)
InfraVis Application Expert
Emin Zerman (MIUN),
Liang Zhou (MIUN),
Henrik Garde (LU),
Mattis Lindmark (UmU)
InfraVis Node Coordinator
Mårten Sjöström (MIUN)
Open data
N/A
Tools & Skills
Unreal Engine, Scientific Visualization, Eye Tracking
Keywords
Eye Tracking, Chemistry Education, Human-Computer Interaction, 3D Representation
about
Understanding how students visually inspect and mentally manipulate molecular structures is important for research on spatial competence and learning in chemistry. Organic chemistry in particular requires learners to move between two-dimensional representations and three-dimensional molecular structures, while recognising spatial relationships that may be difficult to communicate using conventional displays. Virtual reality (VR) provides an opportunity to present and manipulate molecular structures directly in three dimensions, while integrated eye tracking can provide information about where learners direct their visual attention during problem solving.
This project focuses on the development of a VR-based experimental environment for studying visual attention and interaction with three-dimensional molecular structures. The goal was to combine molecular visualization, participant-controlled manipulation, and eye-tracking data collection in a common environment suitable for research in chemistry education. The system was designed around an Unreal Engine-based VR environment, with support for molecular structure import, controller-based rotation, eye-tracking logging, fixation analysis, and subsequent visualization of gaze behaviour. A central objective was to provide researchers with access to sufficiently detailed gaze and interaction data for subsequent analysis rather than relying solely on predefined eye-tracking visualizations. This enables investigation of questions such as which atoms or parts of a molecule attract attention, in what order they are inspected, and how visual attention changes when participants rotate and explore molecular structures.
Challenges and opportunities
The main challenge was integrating eye tracking with interactive three-dimensional molecular visualization. Unlike eye tracking on a conventional static display, VR experiments involve objects that exist in three-dimensional space and can be rotated by the participant. Eye-tracking observations therefore need to remain interpretable in relation to both the molecular structure and its changing orientation. Self-occlusion, overlapping atoms, depth, and the movement of molecular models further complicate identification and visualization of gaze targets.
Calibration and reliability were also important practical challenges. Testing demonstrated that eye-tracking calibration can differ between participants, while occasional tracking failures can produce missing gaze data. Because problems may not be automatically apparent to the application, researchers need an efficient way of checking calibration before an experiment and determining whether recalibration is necessary. At the same time, continuous gaze visualization can distract participants during the actual experimental tasks. An additional challenge was designing the system for real experimental conditions. Participant identifiers, task stages, recording, output files, molecular models, and visualization parameters need to be managed without interrupting the experiment. Interaction also needs to be predictable and comfortable: molecular rotation must be sufficiently smooth and responsive, while fixation visualizations must remain connected to a molecule when its orientation changes.
These challenges created an opportunity to develop an experimental platform in which molecular visualization, manipulation, gaze logging, fixation processing, and replay are integrated while keeping the resulting data accessible for further analysis. The approach can support research into both the potential of VR for chemistry education and the visual problem-solving processes involved in interpreting molecular structures.
Project Process
The project progressed iteratively from requirements gathering and technical exploration to prototype integration and pilot testing. Early development focused on establishing the core workflow:
- importing molecular structures into an Unreal Engine-based VR environment,
- enabling participants to manipulate the molecules, and
- integrating eye tracking to record visual attention during experimental tasks.
Separate developments for molecular visualization, interaction, and eye-tracking were subsequently integrated into a common prototype. Gaze data could be logged and processed to identify fixations and associate visual attention with individual atoms, while molecular rotations were recorded to support analysis of gaze behaviour as participants manipulated the structures. The experimental workflow was also refined to include training and calibration, clearer management of participants and tasks, and more reliable data collection.
Pilot testing at Umeå was used to evaluate the complete workflow and identify practical improvements. This led to refinements in eye-tracking calibration, rotation controls, fixation visualization, and replay. Particular attention was given to ensuring that fixation markers correctly followed rotated molecules and that recorded gaze and rotation data could be reviewed after an experiment.
Results
The project resulted in an integrated VR prototype for studying how participants visually explore and interact with three-dimensional molecular structures. The system combines molecular visualisation and participant-controlled rotation with eye-tracking, fixation processing, and object-based association of gaze with individual atoms.
Recorded gaze and rotation data can be used for quantitative analysis as well as visual replay of participant behaviour. Fixation markers provide a spatial representation of where participants focused their attention, while the replay functionality allows researchers to revisit how a molecule was explored and manipulated during an experiment.
Pilot testing demonstrated the feasibility of the overall approach and helped identify practical considerations around calibration, interaction, fixation display, and experimental operation. The resulting platform provides a foundation for future studies of visual attention, spatial competence, and problem-solving processes in VR-based chemistry education.
Acknowledgement
Prior to this project, thanks to Infravis work completed back in 2024 (ticket T2024-2321) by UmU and LU nodes (Matttis Lindmark, Jens Nirme and Henrik Garde), several options ware already explored, solutions narrowed down and research questions sharpened. Also a platform for logging, analysing and visualising 3D eye tracking data (Jens Nirme) was a great asset when implementation of the final solution was initiated. The users empathise the added value of stretching a complex project with novel research challenges like this, over a long time and across two tickets with time in between to improve the requirement specifications before applying for further Infravis support and funding from Vetenskapsrådet.


