Fifteen years ago, when someone I loved was first diagnosed with breast cancer the first response from the hospital doctors and specialists was to run a series of tests and scans. The tests and scans looked grim and she was told—abruptly and with no bedside manner—there was an 80% likelihood she had metastatic disease. When she finally met with an oncologist, he did something different. He put the test slides and charts aside and conducted a physical exam; he looked at her actual body. He concluded, that based on his experience, she did not have mets.
And indeed, the information from touching and looking at her combined with years of experience turned out to be correct; her cancer had not, in fact, spread to other organs. And it gives me—and countless other people—pleasure that she is still on this planet despite the predictions generated by technology.
The underlying assumption of the San Antonio Breast Cancer Symposium, the largest breast cancer conference in the world, seems to be that we are just around the corner from some breakthrough generated by scientific and technical advance. There are endless papers searching for the molecular drivers of cancer and ways to shut these down; biomarkers to predict treatment response; developing technology and tests to probe and predict; and other ways to generate and analyze data to improve treatment. The truth is, researchers are better at generating data than turning it into meaningful results.
It can all feel like a rabbit hole of information and attempts at analysis, with each new insight leading to more questions than it answers—while 40,000 women are dying every year.
And the fact remains, as many researchers seeking a biomarker or genomic test confessed throughout the conference, that clinical information continues to be both useful and the best way to make decisions currently. [Clinical information includes information readily available to doctors currently, such as an individual’s tumor characteristics, demographics, and specific health history.]
So, in this context, it was more than a little strange to learn about the IBM Watson for Oncology (WFO), which is a machine learning system trained by oncologists at Memorial Sloan Kettering Cancer Center. It uses natural language processing to search the medical literature and a patient’s records and make treatment recommendations for cancer patients.
The last oral session on Friday was Dr. Somashekhar presenting the performance of Watson’s treatment recommendations against those of Manipal multidisciplinary tumor board in Bangalore, India. Physician treatment recommendations of 638 breast cancer were compared to Watson’s report, and when there were differences they went back to the tumor board. The average time of analysis was 11-15 minutes per patients.
Most of the time, Watson affirmed the physician’s treatment recommendations and Dr. Somashekhar reported that initially the two recommendations agreed 73% of the time (what researchers called “overall concordance”). The greatest differences were in cases of metastatic disease and HER2+ disease. In those cases, where there is disagreement, it appears that a majority of the time (63%) the doctors changed their mind after reviewing Watson’s report so that there was agreement 90% of the time after the second review (what they call “final concordance”). [Note: one other possible explanation for the changes after further review is that instead of doctors changing their mind, additional data and new guidelines were put into the computer to change Watson’s recommendation.]
Although the presenter clearly stated this should be viewed as an “assisting tool” which can never replace a live human—hopefully working to build a strong patient-provider relationship—the presumption seems to be that the computer will help improve treatment choices. Not only do we know that there is no single “right” treatment choice and that each treatment plan must be customized and developed with each patient, there is no data about the quality of the computer’s recommendations.
And so I asked the presenter if Watson’s advice positively benefits patients’ health at the end of the day. Are the outcomes for patients actually better factoring in computer-generated treatment recommendations? The answer is that they are not currently looking at the real-world impact on patients and are currently focused on bringing the computer and physician recommendations into alignment.
Only about 40 of the 1,150 papers presented at SABCS are presented in the oral session. And it says a lot about the dominant mindset to fetishize technology that this paper, with no proven benefit to patients, made it to the “main stage”.
Going back to my initial story, perhaps what is really needed to improve treatment decisions is investing in well-trained, culturally-competent, compassionate and respectful providers who have time to build a trusting patient-provider relationship. It may not make headlines, but it will likely make a bigger difference to patients.