Tapping the Brakes on Machine Learning

February 12, 2018

Amid a great deal of current excitement, shading into hype, surrounding biomedical applications for machine learning, Vanderbilt University professor of statistics Frank Harrell sounds a timely note of caution.  At his Statistical Thinking blog, Dr. Harrell examines a recent study by Stanford researchers who used a deep-learning approach to develop an algorithm for predicting patient mortality. The ultimate aim of the predictive tool, which draws on data from millions of electronic health records, is to identify patients for whom palliative care might represent an appropriate option.

In the post, Dr. Harrell painstakingly dissects some of the key limitations and sources of potential bias in the Stanford researchers’ approach and contrasts them with other mortality prediction models that rely on methods that, while less eye-catching than newer machine learning approaches, yield results that may be both more reliable and more amenable to inspection and evaluation.

“In the rush to use ML and large EHR databases to accelerate learning from data, researchers often forget about the advantages of statistical models and of using more compact, cleaner, and better defined data.”

--Frank Harrell, “Is Medicine Mesmerized by Machine Learning”?

Discussions such as the one prompted by the Stanford paper and Dr. Harrell’s critique in my view represent an essential conversation that needs to be taking place constantly as we apply powerful (and sometimes inscrutable) computational techniques to enormous, complex data sets. The essential point here is not that machine learning approaches are inherently flawed or suspect—far from it. Rather, we need to keep in mind that as we pursue our ultimate goal of improving care and outcomes for patients and populations, we must continuously subject novel methods and toolsets to intelligent inspection and constructive criticism that allow the entire field to progress.

Regardless of the analytical method used, there’s no substitute for transparency, empirical validation, and review by the community of experts.  I still love the saying: “In God we trust; all others must bring data.”

See more blog posts by Dr. Califf