Research

Potential Devices / Front-End Solutions Review

Prior to beginning out project, we were uncertain as to which platform our AR project would be based on. Our brief mentioned two devices: Vusix and Hololens. Aside from these, we also looked into the other mixed-reality headsets.

Summary of final Front-End decision

Ultimately, we decided upon designing for the Hololens as we could do this via Unity and the Hololens emulator relatively easily. Hololens offered much more in terms of design potential and interactivity, whereas our other considered device, Vusix, did not. The Hololens is also much more widely used, meaning there is a lot more available in terms of support and resources. We were able to utilise both the Microsoft Academy tutorials as well as video tutorials found on YouTube and Linda. Moreover, designing for two platforms would take far too much time so it made much more sense to focus on one.

---

Master Project Review

We were introduced to two projects done by two master students, Olivia Phillips and John Wadolowski which created "Doctor's basecode" and "Holographic Patient Dialogue" respectively.

Jan’s Project

Jan developed the backend for a Natural Language processing app intended for the Hololens. His project used an API to connect to a cTakes sever which then connect to a medical dictionary. This would then return a JSON file which could be read and manipulated within AR. The issue with this project is that it required a server to be created and run simultaneously to the Hololens for terms to be extracted and processed. This proved to be complex as there were very little resources written about deploying an API endpoint with a cTAKES server.

The only available resource online, which was also used in Jan's report was a github page by Dirk Weisenborn [10], but the last commit was done 2 years ago. When we tried to deploy this option, following Jan's report instruction, we could not deploy it due to breaking changes in the cTAKES server dependencies. Hence we had to find an alternative solution of extracting medical entities from speech.

Olivia's Project

Olivia design a similar project; however, her data was presented within a WebApp, and allowed for interaction of the extracted terms from speech-to-text. She was able to save and delete terms, while also viewing a description and an image relating to the term. However, since we are developing for Hololens we could not make use of the Web Tools she used.

She made extensive use of Javascript and utilised the Web Speech API [9] which can be argued that is slightly more powerful than the default HoloLens one due to the fact that it recongnise pauses and still not finish its dictation. It can also record continuously with a really long time limit of 30 minutes. However, due to the limitation of our platform, which is ground-breaking but very new, we could not utilise these resources. Nevertheless, we still used her idea of using the MediaWiki API and extracting medical term definition and images from Wikipedia.

---

Back-End Medical Extraction Solutions Review

cTAKES

cTAKES is a natural language processing system which allows for the extraction of information from electronic medical documents. (5. Ctakes.apache.org, 2019) We initially worked with cTAKES, by deploying the server on an Azure Linux Virtual machine, But communicating with it and deploying it as a REST service using the instructions provided by the above reports proved to be extremely difficult as the REST server used hasn't been updated for atleast two years resulting in errors to the build. Hence we needed to find other solutions to extract medical data. (1. Ctakes.apache.org, 2019)

UMLS

UMLS We initially used the Unified Medical Language System library which provides medical content for the server to extract data from. The Unified Medical Language System (UMLS) is a compendium of vocabularies within biomedical sciences.(2. Nlm.nih.gov, 2019) Later we realised that as we were using the Lexigram API there was no need for the UMLS database as the Lexigram API had its own connection with it and each extracted term had also a UMLS reference.

Clinical Named Entity Recognition system (CliNER)

Encountered as an alternative to cTAKES this medical extraction engine proved to be initially exactly what we wanted. However, it later was found that this project used data to train a AI model so it could extract medical entities. The problem was that this data was not publicly available and hence training the model to read any unstructure medical text was virtually impossible. (3. github.com/text-machine-lab/CliNER)

Lexigram API

After we lost all hope on deploying cTAKEs and with limited resources about it online, we did a last search in a hope to find a publicly available medical extraction API which can process unstructured text. A rather picky search we must say, however, a few weeks ago version 1 of the Lexigram API was released! We literally, could not believe it, the only limitation of the API is that we can only perform 2 requests per second, which is anyway very decent for what we want. We therefore chose this API as is the easiest to use, it has a beautiful RESTful interface which we can process unstructured text and as well as get feedback on medical concepts. (4. docs.lexigram.io/v1/welcome/getting-started)

OpenFDA

We ultimately wanted to use the OpenFDA API to get information about drugs & devices as to whether they had any potential side-effects or published papers that indicated they were not suitable for use. However, the API did not provide an easy to use response format, and we could not parse it without substantial additional performance overhead. Hence, we decided to not use the OpenFDA API as it was beyond the scope of our project, and the details provided coudl be unneccessary to junior doctors.

---

Back-End Medical Definitions Solutions Review

Wikimedia API

We also reviewed the Wikimedia API [6], which is based on the all-purpose and powerful Wikipedia online free encyclopedia for various definitions and in-depth explanations. The end-points for this API are provided by Wikimedia [6] which is the foundation responsible for running Wikipedia. This API exposes almost all the features found in Wikipedia, such as the ability to extract text from a specific part or specific images from an article. The response is in a JSON format, which for us means that we already have the foundations for it as we already use an api with a JSON response. Nevertheless, this API is not seen as the most accurate in terms of medical terms nor the most trustworthy as anyone can edit it.

Marriam-Webster Medical Dictionary API

Another option for our medical definitions was the marriam-webster dictionary API. At the time during our research, the documentation of the dictionary was obselete, and was being re-written, many parts of the documentation were signed as deprecated, and the API had an XML response which responded with diagrams as well, however, the XML [8] response was being depracated on September 2019, which was would've been a huge flaw in application if we chose to use this, as it would mean that our application would cease to work on September 2019. Furthermore, while many of the responses included medical diagrams, they were in XML format, which was very difficult to find a parser for inside of Unity. As a result this API was far from ideal for our application.

Summary of Medical Extraction Back-End Solution

We concluded to use the Lexigram API and write our own C# scripts inside Unity to allow us to call the end-points we needed with the speech-to-text result so as to get the required response. As cTAKES was extremely complicated to set-up and the solutions provided by both Jan's and Olivia's project where now obsolete at the time of our writing (last commit to cTAKES server is 2 years ago) we had no other (free) option. Furthermore, we thought of creating our own cTAKES server using docker and writing an API using Express and NodeJS but it would be very time consuming and would deviate greatly to what our client requested our project to do. Hence the most sensible, time-efficient and performant solution was the Lexigram API. In terms of performance, the only limitation is the speed of the internet connection, however the term extraction is almost instantaneous and we had no issues with bottle-necks i.e. sending to many terms, not getting response.

Summary of Medical Definition Back-End Solution

After careful consideration of the above two solutions, we realised that these were the only two free options we could use without running into legal issues I.e. scrape other sites for definitions using html/css. First of all, we wanted to make sure our client was okay with using the Wikipedia API, as we raised the flaws of the Webster dictionary and it was clear that our client was not fond of the idea of an obsolete application after mere months of its creation. Given also the fact that Olivias' project utilised Wikipedia as to provide definitions and images for the medical terms we were further encourage to use it as a definition extraction for our solution. Our client agreed and told us it is absolutely fine to use it due to the fact that our application is a proof-of-concept and not indented (yet) to be used in production. Nevertheless, we tried our best to make our application as production-ready as possible, and comply with all the HoloLens comfort requirements as they are discussed in the HCI page.

---

References