Mixed Reality – VR and AR visualization:

Suddenly, Holograms is the new buzz word when a little more than 3 years ago Augmented Reality was a marketing mantra. Last year, it was VR and now thanks to devices such as the ‘Hololens’ every industry wants to see if holograms can be used to some benefit.

While it takes a little getting used to, if holograms is the preferred term, so be it! We prefer calling it mixed reality – because really, it’s a person in the real world, interacting with computer generated visuals (augmented) over their field of view of the world, either aided by a camera – called Digital see-through – or the computer generated visuals are projected/superimposed over said persons field-of-view, via specially coated glasses (optical see-through).

AR and VR research, once the domain of well funded institutions and private organizations, is now getting democratized, and along with it so will the terminology.

Medical VR: Training and Mode of Action scenarios:

We’ve already completed two “mode of action” films and while they were not narrative cinematic VR projects, we still tried to creatively tell a “story” that would engage audiences. In this case the audience were Doctors, Pharmacists and Nurses.

The ‘hero’ in these films, was a brand of medication for diabetes, and we followed ‘it’ at a cellular level – in look around VR – as a battle ensued between insulin inhibitors, glucagon receptors and DPP-4 enzymes. An otherwise dreary topic of training seminars, suddenly held the attention of everyone in the room as they watched this mode-of-action video on the GearVR, with many of them wanting a copy for their cellphones and Google Cardboards.

There is a more serious use for VR and AR in the Medical field, however. Virtual Cadavers in education for instance, have already been talked about in the slickly produced Hololens video:


Teaching Empathy through Augmented Reality:

MedicalAugmentedReality.com has a nice article on the topic, and to quote [verbatim] from it: “Simulating complex working situations that require a combination of social, technical and team skills is a highly interesting application domain for Augmented Reality. Nurses and further members of the clinical staff being in charge of patient care are confronted with this type of scenarios on a daily bases. Augmented Reality gives trainers full design freedom to generate endless training scenarios in order to reflect real working conditions.”

Medical mannequins are brought to life using Augmented Reality, which is invaluable in training, as the video below shows.


Creating Mixed Reality solutions for the Medical Industry:

The current project we’re working on is via a grant by a client, wanting to test the feasibility of using Mixed reality in a Clinical Training setting as well as for Mechanisms of action VR videos.

Some considerations and food for thought:

  • In the Medical Industry, there is at times the need for very high resolution imagery. In addition, there might be the need to visualize large 3D datasets (scans)
  • Are devices such as the Hololens, able to meet these specifications. The videos shown so far are a mix of  high fidelity imagery, yet seem to have a marketing angle, contrasted by other cruder (but more, believable) capability.
  • Fiducial tracking markers are better at eliminating tracking jitter and would be recommended in a medical scenario, over more cosmetic / fancy image or natural world feature tracking, which are desirable by the marketing and advertising market.
  • Should Optical see-through or Digital see-through be employed? This brings us again to the Hololens and Google Glass v/s wearable VR devices such as the GearVR. Both have their pros and cons. Optical see-through naturally creates a less alienating experience, affording the wearer to see the real world, but we yet do not know how high fidelity these so-called “light points” and holographic density are in comparison to the resolution of camera captured AR via a digital see-through AR solutions. An article by PC World has more related info on the topic of Hololens resolution
  • Digital see-through mixed reality, does have the problem of associated lag that if it occurs, can be more nauseating than any lag from an optical see-though device, as the whole “world” will be out of sync with the user, which can lead to VR sickness. In our tests with a moderately high polygon model (for a mobile device) as seen in the video above, there was noticeable lag in update. Since this was a stress test for a mobile Mixed Reality solution, we aim to optimize models and assets more.
  • Desktop driven VR such as the Oculus rift and HTC Vive, will of-course not have these resolution and lag limitations, but at the expense of being tethered to the headset.

One advantage Digital see-through Mixed Reality has over optical see-through, is that the user can seamlessly ‘travel’ between worlds. This is important in a training simulation, for instance:

If the training is for care of victims in an epidemic or trauma – The Mixed reality simulation can comprise: A mannequin placed on a green-screen mat, which then gets inserted into a VR world, complete with chaos, as would be evident during a situation to medical first responders. Such training simulations are done most effectively in virtual reality.

Digital see-through affords the ability to merge AR with VR for a mixed reality experience. Another example can be of administrating a drug to a patient (an augmented reality 3D model) seen superimposed on a hospital bed, and then seamlessly following the course the drug takes into the patients blood vessels and organs. This is a VR world, that the trainee then get’s immersed in.

Mixed reality offers some exciting potential to the Medical Industry, while serving a true real-world benefit.


For the technically inclined: 

  • The Mixed Reality study project being done by Realvision was created to stress test the feasibility of a mobile, self-contained and non tethered solution for visualization and training.
  • The project is being tested on the Unity game engine, via the Vuforia AR system, and runs on a Samsung GearVR with S6Edge phone. 
  • The slight lag in the video, is due to high polygon count models and textures running on the S6 along with MirrorOP(a screen recording solution for android) on the same phone, transmitting the S6’s screen a PC in the next room over wifi. The picture-in-picture window is the recorded output.
  •  The S6 does fire a thermal throttle warning, due to the demands mentioned above.