Google’s clinical search-and-answer tool can now query images

Dive Brief:

• Google’s artificial intelligence-backed search-and-answer tool for clinicians can now scrub tables, charts and other images for information, with the goal of giving doctors and nurses a broader view of patients’ health than what’s in clinical notes alone.
• The tool, called Vertex AI Search for healthcare, is also newly backed by Google’s latest AI model, Google announced Monday. The model, called Gemini 2.0, is faster and more accurate than past iterations, according to the company.
• Tech giants and developers have announced a number of new AI tools and initiatives in the past few days, hoping to get their products in front of prospective buyers at the massive HIMSS health IT conference in Las Vegas this week.

Dive Insight:

Google has been working to expand Vertex AI Search for healthcare’s capabilities since launching the product last March. The tool uses generative AI to let clinicians search for information across patient notes and other clinical data, and can also answer questions, like queries about a patient’s medical history.

It’s an offshoot of Vertex AI Search, a platform that helps developers build search engines for their websites and data stores, that was tuned specifically on medical information.

With Google’s new update, Vertex AI Search for healthcare is expanding into what’s known as multimodal AI, when algorithms can understand and pull data from a variety of sources beyond text.

Multimodal AI could be particularly helpful in healthcare, an industry in which almost 90% of data is in image form, like x-rays or scans, according to research.

With the new multimodal functionality Google is calling “Visual Q&A,” Vertex AI Search can receive images like diagrams directly as an input, without having to convert the image into text first. If the information is relevant to an end user’s search, Vertex AI will include it in its findings, according to Google.

Vertex AI, which already integrates with two of Google’s large language models — Gemini 1.5 Flash and MedLM — will now also be backed by Gemini 2.0, which was unveiled in December.

A spokesperson for Google did not respond to questions about how many healthcare organizations are currently using Vertex AI Search by time of publication.

But it has been used by health systems including Community Health Systems and Highmark Health, and EHR vendor Meditech has woven Google’s AI into its health records software for hospitals. AI documentation company Suki also recently announced its own clinical assistant tool would use Google’s patient summarization and Q&A technology.

Google also announced Monday that it was partnering with physician enablement platform Counterpart Health, a subsidiary of health insurer Clover, to integrate Vertex AI Search into Counterpart’s clinical decision support tool.

Clover, which sells Medicare Advantage plans in five states, launched Counterpart Assistant last spring to peddle its technology platform to third-party clients, targeted toward markets where it doesn’t operate an MA plan.

Amid rising interest in AI tools from healthcare organizations, cloud giants like Google, Microsoft and Amazon are racing to build out their healthcare products and ink partnerships with health systems holding the keys to valuable patient data.

Along with clinical search, Google has been focusing on tuning large language models specifically on healthcare data, and unveiled a platform in December to help organizations craft their own semi-autonomous AI agents.

Yet some are concerned that healthcare, an industry with unique privacy and security concerns, may be moving too quickly to adopt AI, given a lack of federal regulation and the technology’s sticky tendency to make mistakes.

And errors could be compounded when non-text inputs like images are introduced into the equation. Last July, researchers at the National Institutes of Health found that an AI model solved questions on a medical quiz with high accuracy, but made mistakes when describing images.