Science

Watch the world’s first 360° VR surgery live stream on Thursday

VR operations: it’s all about stitch ‘em up, not shoot ’em up.

Bob Dormon – Apr 13, 2016 9:12 am | 15

Update: The 360-degree VR surgery will be live streamed today, Thursday, at 1pm BST (2pm CEST, 8am EDT). You can watch it from within the free VRinOR app (iOS/Android), or there should also be a live stream on the Medical Realities website.

Original story

On May 22 2014, Mr Shafi Ahmed, consultant general colorectal, and laparoscopic surgeon at Barts NHS trust was in the news for bringing his craft a good deal closer to the gaze of his medical students.

By donning Google Glass as he removed tumours from the liver and bowel of a 78-year-old British man, he allowed 13,000 trainees and clinicians to witness a surgeon’s eye view of the procedure as it happened.

Now, the streaming surgeon is going live again. The Royal London hospital’s operating theatre—with the surgery at centre stage—will be available for all to see live in virtual reality at lunchtime on Thursday, April 14.

On the day, anyone with a compatible iOS or Android mobile or tablet will be able to immerse themselves in the cut and thrust of 21st-century surgery. The streaming content isn’t exclusive to the app, and will be available to view directly from the website of Medical Realities—the company driving this unique event. If the 360º demo footage is anything to go by, it’s going to be quite an eyeful.

Audio and video will be captured using a Mativision system that features an intricate array of six different cameras, and stitches the content on the fly. Mativision is better known for delivering immersive entertainment from live concerts. Its clients include Guns N’ Roses axeman Slash, British rock group Muse, and Samsung for its S7 launch. Indeed, this sort of content goes down a treat with Samsung’s Gear VR goggles.

For a live gig setup, Mativision typically operates three of its cameras and, while there might be a couple used during the surgery on Thursday, apparently only one will be viewable during streaming.

Medical Realities is a collaborative venture that was founded in London by Shafi Ahmed and AR/VR evangelist Steve Dann, and draws upon a mix of clinical, educational, and CGI authoring talent.

The company hasn’t always relied on Mativision cameras, and has produced its own 360º surgical content, which was captured with a rig utilising six GoPro cameras. The main difference here being that the stitching wasn’t performed in real time.

There are several examples of this work on the Medical Realities YouTube channel, and you don’t need the VRinOR app to view this content, but the YouTube app will come in handy for mobiles. Although a stereoscopic viewing option is available for Android, you can navigate the scene without needing a VR viewer by simply moving your iOS or Android device. Alternatively, you can use a compatible browser (alas, not Safari) from a computer and drag the scene around as it plays.

An example of VR surgery, from last year. Thursday’s surgery should be better than this. Don’t forget to drag/move the video around to change your viewport.

While these videos capture the operating theatre environment, there’s a fair bit of burn out in the area where the surgery takes place, but you can still make out what’s going on by turning to view the video monitor.

The Mativision rig has greater control over exposure and is expected to deliver a much better result on Thursday.

The next page contains some graphic images of intestines.

Tech plays lead role for wannabe surgeons across the globe

The event itself isn’t just about being the first in streaming surgery again. Shafi Ahmed’s motivation is to improve training and education. As he talks about his use of technology, he highlights how difficult it has been for students to witness live what takes place in theatre. He seized upon Google Glass as a means to deliver a better experience to trainees and clinicians, making it a more interactive process. Indeed, during his live Google Glass operation back in 2014, he was also answering questions appearing as texts in the eyewear.

Surgeon uses Google Glass during cancer operation

He hasn’t given up on this approach either, and has a separate training company called Virtual Medics that has established itself to produce point-of-view videos and to explore the use of wearable technology in healthcare that’s not just limited to Google Glass, but supports other augmented reality eyewear brands.

“We have two glasses. You can see the surgeon’s view and the assistant’s view and you’ve got ChatBox. It’s about how you use this technology to change education for the better. Our students now, don’t come to theatre, they can be anywhere in the world. My students watch this remotely. They communicate with me live and that’s going to change the way we do things.” says Ahmed.

Introducing technological change into a institution such as the National Health Service (NHS) requires evidence of cost benefits, as well as efficacy. Yet, in so many ways, Shafi Ahmed is a hands-on kind of guy and, rather than just talk about methods to improve training using technology, he demonstrates its effectiveness.

Who could argue that reaching 13,000 students and answering questions on the job, as he did with the Google Glass live streaming event, would not have some training and cost benefits?

“My view is that if you want technology to be part of healthcare, you make that change yourself. You control it, you show which way it’s going to go.”

Having had separate meetings with Apple, Google, and Microsoft, Ahmed declares that while these companies have the technology, they need input from people such as healthcare professionals to show what can be done with it.

“They want us to challenge it and move it on. It’s actually up to all of us. To sit at the back and go, ‘Oh, I’m not going to change…’ well, you’ll be forced to change at some stage. The cost of admission is cheaper, it’s what patients want, so it’s going to happen within five or 10 years, regardless.”

His vision extends into primary care and, again, it’s about connectivity, sharing knowledge and lowering costs. A way of dealing with the lack of beds in the NHS is to keep the patients in the community and manage their needs using a telemedicine—mobile kit with Google Glass-style eyewear, and a phone connection that enables remote viewing of patients by specialists.

Ahmed has been working with Advanced Medical Applications (AMA), a Ubisoft company whose Xpert Eye Mobile system has found uses in advising paramedics on the scene of emergencies, as well as less critical outpatient monitoring.

“In the community, you put the glasses on a district nurse, or a physician. They look at a condition and they can type and communicate with a whole load of specialists in the hospital, chat with the specialist, get some advice on dermatology, chronic wounds, or general management of patients themselves. You can do it remotely, instead of say, admit to A&E or go to casualty to be admitted, waste a bed, and waste money. We can save but not by compromising healthcare.”

Saving money for the NHS and delivering on clinical needs for the patients in the UK is only a small part of the prize for Ahmed, as he envisages remote training and advice being available to developing countries where access to safe, affordable surgery and anesthesia is limited or non-existent.

He quotes a paper commissioned by medical journal The Lancet that examines global access to healthcare featuring a world map with vast areas in darkening shades of red, indicating populations deprived of these procedures.

“Look at the dark red—the areas that are really poor that don’t have access to healthcare. We are penetrating the areas of concern just by using a smart device, a simple 3G connection and nothing else. So you can see the value of technology in the work place. Imagine Google Loon; providing Wi-Fi access to remote parts of Africa, South America and Asia. Suddenly, the world becomes a smaller place. And that’s what we think about—exponential medicine. It’s not about here and now, it’s about the world as a whole. How do you make that better? And that’s where technology has a huge role to play.”

Blood ‘n’ guts: The cut and thrust of online surgery

So how does the prospect of an immersive 360º surgical procedure being live streamed this week make the world better?

Shafi Ahmed has already become the poster boy for virtual reality by donning a Zeiss VR One headset during a surgical procedure. He confesses that he wasn’t wearing it while performing the actual operation, it was just a rather cool photo opportunity. Yet having worked with the AR capabilities of Google Glass, he sees VR as an antidote to conventional simulators that seem rather tired these days.

“Simulation has reached its plateau. A simple laparoscopic setup uses a cardboard box and an iPad. You can do that for low cost or go the whole hog and have an expensive simulator with haptic feedback. We have two or three of these simulators in the hospital, and they remain untouched. You’re in a hospital, they’re on a corner, no-one sees them because they’re not sexy. They’re not good, they’ve reached their sell by date, we need to move on. Simulations have changed,” he says.

“At the moment, it’s about technical skills. We’re obsessed with technical skills. Can you do that to that, and stitch that to this…? Surgery is much more than that. In an operating theatre you have noises going on, you have stress levels, you have things going wrong, you have people passing things to you. Everything’s around you and it’s hard to train people in that. It really is because, unless you’re in that environment, you don’t know how to behave.

“With immersion you learn how to behave professionally with your colleagues and how a team functions. Suddenly that whole learning environment becomes much greater than it would have been with a [conventional] simulation, which you can’t create unless you’re in a VR immersive world.”

To create this immersive world, the surgical 360º video recordings are combined with CGI overlays—elements created by the team at Medical Realities that will include navigational aids for interactivity as well as life support monitoring panels and instructional modules. Steve Dann, co-founder of Medical Realities, describes the process to Ars:

“One of the things we’re doing is creating a lot of spheres within spheres—a little bit like a Russian doll. So there will be the core sphere in the middle which will be, say, the initial operation and then wrapped around that will be lots of other spheres which would contain different types of information which, as a viewer, you can go in or come back from. You can switch them off, or add them, and see everything as well.”

“We use a lot of gaming technology. One of our favourite tools is Unity. Most of the CGI that we create is using software such as Maya. But for the actual control systems, we find Unity is a great tool. Really and truly we’re going to be taking a lot of the control systems and the control language or usage language from the gaming community, and putting it into Virtual Reality.”

Surgical precision: gamifying the system

As more operations take place and are recorded as 360º content, Medical Realities will begin the task of identifying the educational aspects that need to be added as well as the relevant CGI elements to enhance the overall immersive experience. The finished products will be rolled out for three different viewing scenarios, as Dann explains:

“We have a professional version that’s designed to go into training hospitals. It’s based around Oculus Rift and pretty heavy weight computers to make sure that everything functions correctly. That will be leased and we’ll supply upgraded operations on a regular basis.”

The ongoing development process has involved working with medical students and trainee surgeons to get feedback, with the most frequently requested feature being able to experience the immersive content beyond the hospital environs. This has led to the creation of mobile alternatives.

“There are two versions.” says Dann. “A subscription model based on the Samsung Gear VR, which has about 80 percent of what you can do inside the hospital on the tethered Oculus Rift setup. This includes technical feedback with questions to see how well they understand what’s gone on which could feature as credits towards their work.”

“The other is designed to work on Google Cardboard, or a less sophisticated headset, and that will be free but will be funded by advertising or promotion from pharmaceutical companies or medical suppliers. This will have 50-60 percent of the interactivity, and it’s partly so people can experience a try-before-you-buy scenario.”

If the rather basic demos of the CGI elements are anything to go by, then this is very much a roadmap for the interactive aspects of VR surgical training. Dann even reckons that Medical Realities will be transitioning to create a completely CGI operating theatre featuring avatar surgeon, anesthetist, and patient. The idea is to provide complete control to see how well trainee surgeons cope when things go wrong, much like a flight simulator.

“We will be ‘gamifying’ the operation, so that everything will be CGI.” says Dann.

“We’re very conscious of what’s technically possible and deliverable right now, and what will be possible, and deliverable within two and a half, or five years’ time. What we can do right now, it’s based around filming real operations. Then we’ll transition over to creating a completely believable CGI environment you can inhabit. Finally, when the technology has caught up with what we want to do, we can add haptic feedback.”

Adding that element of touch is another dimension in the immersive experience that surgeon Shafi Ahmed is keen to explore.

He talks of tactile feedback, exoskeleton development, and the use of balloons and bubbles to create the feel of the human body. He looks forward to this level of simulation becoming available within five to 10 years, or perhaps sooner, and states that the only way to advance this technology is for people to get involved with these innovators.

If you’ve got the stomach for it, we’re promised that the live-streamed VR surgery on Thursday—courtesy of Barts NHS trust—is just the beginning of an immersive educational future for medical and trainee surgeons. A future that allows lessons to be learned and mistakes to be made, rather than, as Mr Ahmed puts it, “training on our poor patients.”

***
You can view the event live on Thursday, April 14, at 1pm BST (2pm CEST, 8am EDT) directly from the VRinOR app (iOS, Android), or by using Samsung VR, Google Cardboard, and similar VR headsets. The content will also be streamed on the Medical Realities website. In other words, it can be viewed from a conventional Web browser, too.

***

Bob Dormon’s technological odyssey began as a teenager working at GCHQ, yet his passion for music making took him to London to study sound recording. During his studio days he regularly contributed to music technology and Mac magazines for over 12 years. Fascinated by our relationship with technology he eventually turned to journalism full-time, and for over six years was part of The Register’s senior editorial team. Bob lives in London with far too many gadgets, guitars, and vintage MIDI synths.

15 Comments