The Project
The project is about utilising the power of Mixed Reality (MR) on the HoloLens platform to enhance the teaching and learning experience of Junior doctors. Specifically, we created three scenes which provide a complete experience inside a hospital so as to aid junior doctors to identify details quickly, be able to extract medical entities and see definitions and diagrams from speech while a senior doctor is talking. Lastly, we created a notes scene which allows junior doctors to use the walls of the hospital to write virtual notes and attach them inside the hospital surfaces making learning medical information much easier than using traditional paper methods.
The Solution
Any medical terminology recorded in a conversation would now make sense as the glasses give you a real-time interaction with a medical dictionary and relevant diagrams. The application’s sleek and simplistic design has an intuitive approach to the user ensuring a comfortable and productive experience. Furthermore, the patient information scene presents patient information in a minimalist yet informative way, and the junior doctors can also view brain scans or MRIs. Lastly, the note scene allows medical data to be pinned inside the Hospital and even persists and stays in the same room even if the application is exited and reopened.
Achievements
The greatest achievement of this project is designing and developing a product off the beaten track since AR/MR are still very new. What we accomplished is easing junior doctors into the digital world and make them feel in control despite their inexperience. We provide invaluable information by performing NLP on speech and extracting medical terms as well as utilising the actual physical environment to attach notes and help make learning even more efficient.
Key Features
What makes medical training and teaching easier
Real-Time display of medical data
We display medical entities which were extracted in real-time while a senior doctor was or is talking. We also provide definitions and relevant diagrams and give the junior doctor the opportunity to export everything in a file for further study.
User-Friendly Interface
We utilise components and best practices as specified by the Microsoft Mixed Reality Academy hence we created an interactive and dynamic yet still not distracting user interface, with the use of solvers to make the menus available only when the user needs them without being all the time in their FOV.
Note taking on physical surfaces
We utlise the physical space of the hospital to allow junior doctors to place "physical" notes on surfaces so as to remember dosologies/drugs or definitions easier and thus make their learning more efficient. Our modular code also allows the note taking scene to be made into a shared experience in the future as we already use "World Anchors" the feature needed for sharing.
Displaying Patient Data
We created an elegant interface to allow junior doctors to view patient information easily without to much movement, we utilised voice commands and the gaze input to decrease tapping as their hands might be busy with examining patients.
The Team
The team behind this Proof Of Concept
Amy Jeffcoate
UX Engineer
Amy is responsible for researching potential solutions for junior doctors and use cases with the HoloLens such as real-time voice input or the note annotation scene. She also aids in editing reports and proofreading.
Petros Xen
Backend Engineer / Team Leader
Petros is responsible for what happens behind the scenes of the project. He is doing research on mixed reality practices and how to make it as easy as possible to integrate the medical back-end with the HoloLens.(He does wear glasses, but not the ones in the picture.)
Elena Aleksieva
UI/UX Engineer / Tester
Elena is responsible for building interactions that happen on the actual HoloLens using Unity and C#. While building the UI she is also reporting on bugs and potential fixes or improvements as the project moves along.