SIMON IS AN IMMERSIVE ON-CALL SIMULATOR THAT FAMILIARISES JUNIOR DOCTORS WITH ON-CALL ENVIRONMENTS AND REAL-TIME PROCEDURAL DECISION-MAKING. IT AIMS TO EASE THE STEEP LEARNING CURVE THEY FACE DURING THEIR FIRST ON-CALL SHIFTS, MARKED BY MINIMAL ONBOARDING, PRIORITISATION OVERLOAD, INCONSISTENT SUPERVISION, AND EMOTIONAL STRAIN. THROUGH SPATIAL SIMULATION AND INTERACTIVE SCENARIOS, SIMON RECREATES THE COMPLEXITY OF HOSPITAL SETTINGS, ALLOWING DOCTORS TO PRACTISE PRIORITISATION, PROCEDURAL RESPONSES, AND COGNITIVE DECISION-MAKING IN A SAFE, REPEATABLE ENVIRONMENT. THE SYSTEM AIMS TO STRENGTHEN CONFIDENCE THROUGH EXPOSURE, IMPROVE COGNITIVE DECISION-MAKING SKILLS, REDUCE COGNITIVE OVERLOAD, AND SUPPORT A SMOOTHER TRANSITION INTO REAL-WORLD ON-CALL DUTIES. SUPERVISED BY ASSISTANT PROFESSOR GABRIEL LIPKOWITZ FOR THE SPATIAL COMPUTING: DESIGN & DEVELOPMENT PLATFORM.
Client
NUS Centre for Healthcare Simulation & NUH Department of Emergency Medicine
Team members
ANG SZE ERN, ALEX ONG LI HONG, DANIKH AQIB BIN MOHD NIZAM, RYKA NOUVIN BINTE MOHAMAD AZHAR
Year
2025
Figma, SwiftUI, Xcode, Spatial Media, Apple Foundation Model
Design Brief:
This project focuses on creating an Apple Vision Pro application built in Swift and SwiftUI, incorporating spatial media to demonstrate the potential of spatial computing in healthcare contexts. The aim is to support workflows for both patients and clinicians by leveraging immersive interaction and spatialised information. Each team is to develop a distinct solution informed by the pain points identified by the clinical collaborators and healthcare partners they will be working with.
DEMO VIDEO
The design were first done in Figma before we started building it on Xcode and SwiftUI
Pre-filmed scenarios are paired with pre-crafted questions that pause the video at key moments, prompting you to respond by typing or voice. As you answer, the system analyses your reasoning, resumes the scene, and delivers instant, tailored guidance on your clinical decision-making.
Digital twins of NUH wards familiarise junior doctors with different layouts and environments, while the Apple Foundation Model generates adaptive scenarios and questions based on the ward they are in, analyses their responses, and provides tailored feedback with follow-up prompts.
Live Demo & Expert Feedback
We presented Simon at both the Apple Developer Centre and the NUS Division of Industrial Design, engaging members of the Apple team, doctors, healthcare professionals, HCI researchers, and academics working in emerging technologies such as Spatial Computing, AR, VR, and XR. Through live demonstrations and discussions, participants experienced the simulation firsthand and provided feedback on its relevance to clinical training and decision-making.
A recurring point of praise was Simon's use of a digital twin of the hospital environment within Apple Vision Pro. By generating context-aware scenarios and questions based on the user's location, the system supports cognitive decision-making in a realistic setting. Clinicians noted its potential as both a training tool for junior doctors and a refresher for more experienced practitioners, enabling users to rehearse critical thinking and clinical reasoning within a familiar spatial context.