Our Prototype

Our architecture is based mostly on the Microsoft Design and Programming guidelines, as we are developing for the Microsoft HoloLens.

A very useful resource was the Microsoft Mixed Reality Academy which provided us with lots of examples, regarding how to code for the HoloLens platform as well as very useful design guidelines for Mixed Reality.

We use numerous technologies, all working in unison in order to deliver an immersive yet educating experience for medical personnel using our product.

Our Tech Stack

• C# inside Unity for Logic Control
• Microsoft Mixed Reality Toolkit to make coding for the HoloLens easier
• The powerful Lexigram API to extract medical terms and provide information about virtually any medical content (including contextual information!)
• The Open-Source and supported by the U.S Food & Drug Administration (FDA) OpenFDA API to show possible side-effect of drugs or devices found in the context

---

Architecture Diagram

Below is an outline of our Architecture Diagram

An overview of the architecture of our Application

At the core of our architecture is ofcourse Microsoft HoloLens. We use the microphone for speech-to-text as well as the sensors at the top of the HoloLens for gesture recognition.

All the holographic manipulation regarding the visuals, i.e showing buttons and text is handled by the Unity3D game engine which is supported by the Microsoft Mixed Reality Toolkit written in C#.

Once the medical personnel selects to start recording speech of a doctor, the microphone start recording, and the input is converted in real-time to text to be passed to the Lexigram API

Sample Usage of Application

1. The user selects an option from the provided menu from the HoloLens, currently the one calling the APIs for medical extraction is the Dictionary option, which is used to extract medical data from an unstructured text such as during a Hospital Ward.
2. The voice input is converted by the HoloLens into text and we pass it using our back-end to the Lexigram API which uses a custom engine called “Lexipedia” to find medical concepts relating to a specific entity.
3. After we get the concepts, we classify depending on whether they are a drug or a device mention. Then we pass each of those to the respective endpoint to the OpenFDA API and we receive all the publicly available information the FDA holds about those drugs or devices.
4. The user can select each extracted medical term for further examination, such as view the FDA papers or synonyms of the device,drug or medical entity.

---

Implementation of Finished Functionalities

At the current time of writing, both our Front-End and Back-End work, but individually, due to some Library errors from Unity, which although it is written in C# does not support some normal .NET libraries otherwise supported by C# we will have to re-write the API calls using a different library which is supported by Unity. The library has been found to re-write everything and we hope that by the time of demonstration everything will work as expected.

Front-End Completed Features

---

Back-End Completed Features