Google Develops Augmented Reality Microscope To Detect Cancer In Real Time

  • Google builds Augmented Reality Microscope for detecting cancer in real time. 
  • It uses a combination of machine learning algorithm and head-up display in the microscope. 
  • To better assist the doctor, digital projection is directly superimposed on the original specimen’s image.  

Medical technology has advanced exponentially in the last couple of decades, but cancer diagnosis is still a tedious and time-consuming process. For most aspects of medical testing, we rely on light microscopy, which is a gold standard for cancer diagnostics in anatomic pathology. They have been effective for more than a century across a wide range of specimens and stains.

Now researchers at Google have developed an alternative (or you can say, a better) approach that involves using deep learning techniques to detect breast, prostate and other types of cancers.

We’ve already been using deep learning in medical fields like pathology, dermatology and radiology; and so far, it’s has shown promising results in both speed and accuracy. This time Google researchers have developed an Augmented Reality (AR) Microscope prototype that can speed-up the adoption of AI tools for pathologists.

What’s AR Microscope and How It Works?

It’s a modified version of light microscope that allows doctors to analyze image/samples in real time, and machine learning algorithms present the result into the field of view. The best thing is there is no need to buy a dedicated system for this; it can be integrated into existing microscopes (using low-cost modules) commonly found in healthcare facilities.

To enable machine learning algorithms on conventional microscope, it requires 3 novel technologies working together –

  1. Convolutional network for precise detection and classification.
  2. Tightly integrated sw/hw to implement algorithms in real time for better interactive user experience.
  3. Parallax-free head-up display in the microscope for projecting high-resolution data.

Like conventional microscope, doctors view that specimen via eyepiece. Then the algorithms projects its result back into the viewing path in real time. To better assist the doctor, this digital projection is directly superimposed on the original specimen’s image.

Schematic sketch and actual implementation | Credit: Google 

The current system runs at 10 fps (frame per second). Modern deep learning models and advanced computational modules — like those developed upon TensoreFlow — will enable various trained models to run on AR Microscope.

It provides numerous helpful visual feedback such as heatmaps, arrows, text, animations, etc. Also, it can run different AI algorithms to solve various issues like detection of target, classification and quantification.

So far, researchers used this platform to execute two different algorithms to detect two different types of cancers. The one was configured to detect prostate cancer, while the other was set to detect breast cancer.

Source: Google Research | Accompanying Paper


Detection of prostate cancer at 4x, 10x and 20x. 

The platform can magnify the viewing samples up to 40 times, highlighting regions of tumor (green contour). Both machine learning algorithms were trained on sample images, and they performed exceptionally well on the AR Microscope.

More specifically, algorithms configured for prostate and breast cancer detection had an area-under-the-curve of 0.96 and 0.98, respectively, when executed on the AR Microscope.

What’s Next?

According to the developers, the performance can be further enhanced by introducing more digital pictures in training phase, and by images taken directly from AR Microscope itself.

Read: Google Develops AI That Predicts Heart Disease By Scanning Your Eyes

Furthermore, the technology has potential for a huge impact on medical field, especially for infectious disease diagnosis like malaria and tuberculosis, in developing nations. In coming years, AR Microscope could be used along with the digital workflow for a wide range of applications.

Written by
Varun Kumar

Varun Kumar is a professional technology and business research analyst with over 10 years of experience. He primarily focuses on software technologies, business strategies, competitive analysis, and market trends.

Varun received a Master's degree in computer science from GGSIPU University. To find out about his latest projects, feel free to email him at [email protected]

View all articles
Leave a reply