Minimizing Diagnostic Error: The Importance of Follow-up & Feedback

diagnosticindexI lost my composure recently after reading David Brooks column “Beyond the Brain” in the June 17, 2013, New York Times.  I tried to write a post in response to it, but it got too political and personal quick.  I decided that is not what I want to do here.  But this post is prompted by something personal.

In the spring of 2011, my then 57 year old sister went to her internal medicine doctor after experiencing a large amount of blood in her urine.  She was referred to a urologist who did several tests and scans and found no issues. In August 2012 after experiencing back and leg pain, she was diagnosed with bladder cancer of the renal pelvis-Stage IV.  She has been through chemotherapy, radiation, surgery, and now more chemotherapy.  She has not seen the same urologist or his group since the diagnosis.

It seems likely that had the cancer been diagnosed in 2011, the prognosis would be better. According to Radiopaedia “transitional cell carcinoma of the renal pelvis is uncommon compared to renal cell carcinoma, and can be challenging to identify on routine imaging when small.”  This is not about whether the cancer should have been diagnosed then, but about whether someone will ever get a chance to look at an old image and possibly improve their skill at identification or find out about her later diagnosis and learn something. More importantly, how can a system evolve that would facilitate this.

Gordon Schiff is currently one of the experts in examining medical diagnostic error. I will be looking at his paper: “Minimizing Diagnostic Error: The Importance of Follow-up and Feedback” in The American Journal of Medicine (2008) vol 121, 538-542 and also his paper: “Can Electronic Clinical Documentation Help Prevent Diagnostic Errors?” N Engl J Med 2010; 362:1066-1069.

Schiff indicates that to an unacceptably large extent, clinical diagnosis is an open loop system.  Clinicians only learn about their diagnostic accuracy in ad hoc ways:

  1. A malpractice lawsuit
  2. A medical resident bumping into a surgical resident and learning that a patient that he had cared for has been readmitted.
  3. A radiologist stumbling upon an earlier chest x-ray of a patient with lung cancer and noticing a nodule that was overlooked.

Schiff says that physicians lack systematic methods for calibrating diagnostic decisions based on feedback from their outcomes.  Worse yet the organizations that employ these physicians have no way to learn about the thousands of diagnostic decisions.

Schiff does not agree that overconfident physicians are the problem. He says that they know the thin ice on which they are skating, but that they have no consistent and reliable systems to get feedback. Table 1 from Schiff’s first noted paper lists some of the factors that mitigate against systematic feedback.  Schiff believes that heroic effort by genius diagnosticians is not the answer, but that mundane mechanisms to ensure adequate follow-up are. Schiff believes that missing unsuspected new diagnoses is only a small part of the problem, while the biggest problem is the complexity of weighing and pursing diagnostic considerations that are either obvious, have been previously considered, or simply represent “dropped balls.” Table 2 provides a listing of paradigms often more important than affixing a label.  The most basic of these is whether, the patient is “sick” and needs to be hospitalized or sent home.


The traditional “test of time” has some validity if systematically tracked, but basing diagnostic accuracy on whether or not a patient responds successfully to treatment is according to Schiff fraught with nuances and complexities.  Nevertheless, such feedback is critical.

Schiff notes that building dialogue into the clinical diagnostic process, whereby the patient tells the practitioner how she is doing is important. Schiff recommends following up by calling patients after an appointment since it acknowledges the extremely important role patients have in making the best possible diagnosis. Schiff believes that such coproduction of diagnosis goes beyond sending the patient home to google the suggested diagnosis or obtaining a second opinion.  He says that it should mean that the patient is a partner in thinking through and testing the diagnostic hypothesis. This according to Schiff could include:

  1. Confirming or refuting a diagnostic hypothesis based on timing.
  2. Noting relieving or exacerbating factors.
  3. Assessing the response to treatment.
  4. Feeding back the nuances of the comments of a specialist referral.
  5. Triggering past historical clues.

Schiff says this points out that the diagnosis should be a relationship rather than a label and that the doctor should be pulled back by communication of downstream outcomes to the patient to adjust.  This sounds difficult and time consuming, but such communication might have been helpful with my sister.  She, of course, wanted to be told that she was okay, and her limited relationship with the urologist limited the credibility of her description of her symptoms.

Of course, feedback can increase bias, notably availability bias. Schiff provides the example that upon learning that a patient with a headache that was initially dismissed as benign was found to have a brain tumor, the physician subsequently works up imaging studies for all future headache patients. Thus, feedback on the patient with the brain tumor is given undue weight thereby biasing future decisions. Conversely, if each time the physician learned of a fruitless negative workup for a rare diagnosis, he decided to never do so many tests, the cherished continuous feedback loop could be making the quality of diagnosis worse.  Schiff points to other researchers who make the point that feedback that inappropriately leads to bolstering or shaking the confidence of the physician in future diagnostic decision making is not a good thing.

Learning and feedback are inseparable, and Schiff says that we cannot rely on lucky feedback, or a doctor’s memory, or a patient initiating a follow-up appointment. The Electronic Health Record (EHR) has the potential to help by:

  1. Filtering, organizing, and providing access to information.  Too much information may be the new problem, but hopefully system designers will provide graphic or visual ways to accumulate information in an understandable way.
  2. Serving as a place to note unanswered questions an nuances of diagnoses.
  3. Insure fail-safe communication and action in the areas of ordering tests and tracking the results.
  4. Incorporate checklist prompts to make sure that key questions are asked and relevant diagnoses considered. Despite renewed interest in safety checklists, diagnostic checklists have so far been neither clinically helpful nor widely used.  Artificial intelligence software may eventually be helpful.
  5. Should do more to help with follow-up and the systematic oversight of feedback on diagnostic accuracy. Clinicians need a reliable, automatic follow-up system.