Dr. Andy – Surgeon in a Box

Robotic tele-surgery platform enabling skilled surgeons to operate remotely using VR hand tracking and robotic arms.

  • Built at HooHacks 2025

Dr. Andy – The Surgeon in a Box combines a pair of robotic arms with a stereoscopic camera rig in the field and a VR control station anywhere in the world. Surgeons don a standard OpenXR‑compatible headset and see through the robot’s eyes while their real‑time hand movements manipulate the instruments with sub‑50 ms end‑to‑end latency. High‑fidelity hand tracking and haptic feedback preserve the dexterity required for delicate procedures, making expert care possible in war zones, disaster areas, and underserved regions.

System overview

How We Built It

We fused the Meta Quest 3 hand‑tracking SDK with a custom ROS2 control stack that drives Dynamixel‑powered arms. A real‑time kinematic model translates headset‑space hand poses into joint commands, while a lightweight ML filter suppresses micro‑tremors. The pipeline is written in C++ and Python and streams over WebRTC to keep latency under 50 ms.

Challenges We Ran Into

  • Latency: Even a 30 ms spike can be dangerous in surgery. We profiled every stage—sensor capture, pose estimation, network transport, and actuator response—and trimmed copy operations to hit consistent sub‑50 ms budgets.
  • Smooth kinematics: Human wrists have more degrees of freedom than our servos. We iteratively tuned inverse‑kinematics constraints to avoid singularities and jerky motion.
  • UX for non‑gamers: Many surgeons had never used VR. We built context‑aware overlays and a tutorial mode to flatten the learning curve.

Accomplishments We’re Proud Of

  • Achieved real‑time control with 46 ms median latency measured end‑to‑end.
  • Demonstrated tasks such as tying a surgical knot and threading a needle on camera.
  • Built a reusable ROS2 driver that the UVA Robotics Lab has begun adopting for other medical‑robotics projects.

What We Learned

Robotics, networking, UI, and medicine are all unforgiving of delay or ambiguity. The project forced us to think holistically—from firmware to human factors—and appreciate the power of interdisciplinary teamwork.

What’s Next

  1. Harden the hardware for sterilizable clinical environments.
  2. Integrate vision‑based AI assistance to dampen hand tremor automatically.
  3. Test over 5G and Starlink to evaluate global reach.
  4. Seek partnerships with Médecins Sans Frontières and field hospitals for pilot trials.

Built With

arduino • C • C# • C++ • HLSL • Python • Unity • ROS2 • WebRTC

Links