Man or machine: who is better at assessing risks? A calculator tool might be able to correctly assess complications before surgery – and be more reliable than surgeons’ gut feeling.
Not enough time to read the full article? Skip to the end for a TL;DR – a new feature we are currently testing. Let us know what you think in the comments!
Experience, knowledge and plain gut feeling: these factors often play a key role in surgical decision-making. A practicing surgeon’s experience influences how they estimate 30-day postoperative complications. That’s why a study looked at whether the estimates of experienced surgeons could compete with those of a specially created risk calculator. The results were published in the Journal of the American College of Surgeons.
Nowadays, there seems to be an AI or a similar tool for everything. But how decisive is the missing human factor? Particularly in the case of serious decisions that need to consider many unpredictable factors and variables – and are therefore not always based exclusively on logic –, many artificial decision-making aids still have a hard time. One of these multifactorial areas is the assessment of postoperative complication risk. Surgeons weigh different factors differently and include circumstances that a risk calculator may not know or does not perceive as important. On the other hand, two surgeons often will not make the same decision based on their differences in experience. For example, one study shows that patients who get a second opinion from another doctor get the same diagnosis only 12 % of the time.
“The integration of the explicit, the intangible and experience together form what we call surgeon intuition. Surgeons with a certain level of training and experience will have relatively similar intuition in certain cases,” said senior study author and trauma surgeon Gabriel A. Brat, MD. “However, intuition is dynamic. It depends on the characteristics of the provider. One surgeon can see one patient and believe one thing about that patient’s outcome, and another surgeon can see the same patient and predict a different outcome.” Can risk calculators help to make better decisions, or do experienced surgeons usually have the right gut feeling?
The researchers asked surgeons before operating on 216 patients how they would assess the risk of complications up to 30 days after each patient’s surgery. The small number of patients is one of the main limitations of the study. In addition, most of the participating surgeons were trauma and emergency surgeons – so a bias toward these specialties cannot be ruled out.
The personal assessments of the surveyed surgeons were then compared with the results of a risk calculator, derived from the American College of Surgeons (ACS) National Surgical Quality Improvement Program (NSQIP). To make the data comparable, the researchers developed an algorithm based on the collected surgeon data that predict postoperative outcomes. This data was then compared with data from 9,182 patients whose risk for postoperative complications was determined by a risk calculator.
An important factor for the surgeons’ accuracy was their experience – the more experienced the surgeon, the better their risk assessment. Nevertheless, the results show that overall surgeons’ estimates were inferior to the risk calculator’s estimates. “A binomial regression model trained on clinical data alone had an AUC of 0.83 (95 % CI: 0.80–0.85) in predicting any complication. A model trained on only preoperative surgeon intuition had an AUC of 0.70 (95 % CI: 0.63–0.78),” the authors conclude. The researchers also found that a combined model of the risk calculator and human intuition did not lead to a better prediction than that of the risk calculator alone.
“The value of surgical intuition for preoperative prediction was not improved by including human intuition in the model and this suggests that, at least for most presurgical prediction, the information that is gathered by the NSQIP Risk Calculator is better at predicting those outcomes than the gut feeling that surgeons have when looking at patients“, said Brat.
Image generated with: Dall-E
