Jul 14

Jul 14 Harvard study: Mayo Clinic's online symptom checker gets a failing grade

Mayo Clinic's online symptom checker is less than 60 percent accurate, yet still one of the most helpful in the industry, according to a newly-released study by Harvard Medical School.

Researchers tested 23 online symptom checkers — from Mayo to WebMD to Drugs.com — by using realistic patient scenarios and examining the results. Here's how the Washington Post explains the algorithm:

The researchers' evaluation, which was published in June in the BMJ, the former British Medical Journal, consisted of running 45 patient scenarios (or as many as made sense on specialty sites focused on certain types of conditions or demographics) on each of the symptom checkers. Fifteen of the cases required emergency care, 15 required non-emergency care, and 15 may have required self care but did not necessarily require a medical visit. Of the 45 cases, 26 described common diagnoses while 19 described uncommon diagnoses.

An accurate answer meant that the site listed the correct diagnosis within the top three possibilities. Using that data, they found that sites ranged in accuracy from 29 to 71 percent.

"People who use these tools should be aware of their inaccuracy and not see them as gospel," Dr. Ateev Mehrotra, who led the research, told NPR news. "They shouldn't think that whatever the symptom checker says is what they have."

Results of 23 online #DigitalHealth symptom checkers tested by Harvard researchers http://t.co/dJin1paP2b #mHealth pic.twitter.com/AIqcvAO5hb
— Paul Sonnier (@Paul_Sonnier) July 12, 2015

Mayo Clinic finished sixth overall on the list. Researchers found that its first online diagnosis was right about 17 percent of the time, compared to an average of 34 percent. However, the clinic's tool had the correct diagnosis on a list of 20 in 76 percent of cases.

Dr. John Wilkinson, who works on Mayo’s symptom checker, told Kaiser Health News that the point of the tool is to direct patients in the right direction. “We’re always trying to improve but if most of the time the correct diagnosis is included in the list of possibilities, that’s all we’re attempting to do,” he said.