Team 7 Systems Engineering

We are using the HoloLens platfrom to utilise the power of Mixed Reality (MR) to aid clinical teaching and training.
Our application gives young medical professionals the unique experience to jump from the hospital straight into the world of Mixed Reality. We give them the chance to access interactive medical information just in front of their eyes without disturbing any patient consultation. Annotate a room with notes such as dosologies or patient information as well as extract medical entities from speech to see definitions and diagrams.

The Project

The project is about utilising the power of Mixed Reality (MR) on the HoloLens platform to enhance the teaching and learning experience of Junior doctors. Specifically, we created three scenes which provide a complete experience inside a hospital so as to aid junior doctors to identify details quickly, be able to extract medical entities and see definitions and diagrams from speech while a senior doctor is talking. Lastly, we created a notes scene which allows junior doctors to use the walls of the hospital to write virtual notes and attach them inside the hospital surfaces making learning medical information much easier than using traditional paper methods.

The Solution

Any medical terminology recorded in a conversation would now make sense as the glasses give you a real-time interaction with a medical dictionary and relevant diagrams. The application’s sleek and simplistic design has an intuitive approach to the user ensuring a comfortable and productive experience. Furthermore, the patient information scene presents patient information in a minimalist yet informative way, and the junior doctors can also view brain scans or MRIs. Lastly, the note scene allows medical data to be pinned inside the Hospital and even persists and stays in the same room even if the application is exited and reopened.


The greatest achievement of this project is designing and developing a product off the beaten track since AR/MR are still very new. What we accomplished is easing junior doctors into the digital world and make them feel in control despite their inexperience. We provide invaluable information by performing NLP on speech and extracting medical terms as well as utilising the actual physical environment to attach notes and help make learning even more efficient.

Key Features

What makes medical training and teaching easier

Real-Time display of medical data

We display medical entities which were extracted in real-time while a senior doctor was or is talking. We also provide definitions and relevant diagrams and give the junior doctor the opportunity to export everything in a file for further study.

User-Friendly Interface

We utilise components and best practices as specified by the Microsoft Mixed Reality Academy hence we created an interactive and dynamic yet still not distracting user interface, with the use of solvers to make the menus available only when the user needs them without being all the time in their FOV.

Note taking on physical surfaces

We utlise the physical space of the hospital to allow junior doctors to place "physical" notes on surfaces so as to remember dosologies/drugs or definitions easier and thus make their learning more efficient. Our modular code also allows the note taking scene to be made into a shared experience in the future as we already use "World Anchors" the feature needed for sharing.

Displaying Patient Data

We created an elegant interface to allow junior doctors to view patient information easily without to much movement, we utilised voice commands and the gaze input to decrease tapping as their hands might be busy with examining patients.

The Team

The team behind this Proof Of Concept


Amy Jeffcoate

UX Engineer

Amy is responsible for researching potential solutions for junior doctors and use cases with the HoloLens such as real-time voice input or the note annotation scene. She also aids in editing reports and proofreading.


Petros Xen

Backend Engineer / Team Leader

Petros is responsible for what happens behind the scenes of the project. He is doing research on mixed reality practices and how to make it as easy as possible to integrate the medical back-end with the HoloLens.(He does wear glasses, but not the ones in the picture.)


Elena Aleksieva

UI/UX Engineer / Tester

Elena is responsible for building interactions that happen on the actual HoloLens using Unity and C#. While building the UI she is also reporting on bugs and potential fixes or improvements as the project moves along.