Perception 2.0

Surgical Extended Reality (SXR) Training System 

ROLE: Project Manager & Lead Developer


SKILLS UTILIZED: 3D Modeling, Virtual Environment Design, Sensor Data Collection, Data Processing, Data Analysis, Coding (C++, Python), Project Management 


SOFTWARE: Unreal Engine 4, AutoCAD, SolidWorks, LeapMotion Orion SDK, Numenta NuPic (HTM Systems), OpenAI Stable Baselines, Microsoft Excel

HARDWARE: HTC Vive Pro, Pupil Labs EyeTracker, LeapMotion Hand Tracker, Shimmer Consensys Sensor Kit




Extended reality (XR) and it's three primary forms - Virtual Reality (VR), Augmented Reality (AR), and Mixed Reality (MR) offer an incredible opportunity to visualize data in innovative ways. Many occupations can benefit from these technologies, especially those in the medical field. Surgeons, who must simultaneously navigate complex mental maps of human anatomy and perform precise cut operations can optimize their performance with XR. 


The Perception 2.0 project began with a literature review of present surgical XR applications and systems. Through this process, the flaws in current surgical extended reality (SXR) became evident and this insight was leveraged to design a novel SXR system that integrates VR, AR, and MR modalities. Preliminary Perception 2.0 experiments are currently being conducted with the HTC Vive Pro, Pupil Labs EyeTracker, and LeapMotion Hand Tracker, and Shimmer Consensys Sensor Kit