Evidence Background

Before delineating the evidentiary data review process, a conceptual framework for evidentiary review of imaging used by Intermountain is given. Several rating systems have been established to rank the strength of medical evidence. The 2011 revision of the Oxford Centre for Evidence-Based Medicine (OCEBM) 2011 Levels of Evidence standard includes categorical levelling grades relevant to diagnostic studies (OCEBM Levels of Evidence Working Group*). This system can be used to rate individual sources of evidence (published papers or other research data) on a 5-point scale. OCEBM levels for diagnostic studies are given in the following table.

Question Step/Level 1  Step/Level 2  Step/Level 3 Step/Level 4 Step/Level 5
Diagnosis Systematic review of cross sectional studies with consistently applied reference standard and billing  Individual cross sectional studies with consistently applied reference standard and billing Non-consecutive studies, or studies without consistently applied reference standards Case-control studies, or poor or non-independent reference standard Mechanism-based reasoning

Imaging Value Chain

The performance of an imaging test should never be viewed as an isolated event. It always takes place in a clinical context and will have an impact on downstream events. A simple depiction of the Imaging Value Chain is given in the following graphic.


In this model, patients present in common clinical contexts where their medical condition is associated with a default plan for therapy. Where there are unanswered key questions, diagnostic tests may be ordered before instituting the default therapy. Imaging tests are ordered with the primary objective of answering the key questions in the anticipation that the answers will change the therapeutic plan.

Further, it is anticipated that this change in therapy will ultimately improve patient outcome. The decision to order an imaging test must, however, include consideration of the possible harms that could negatively impact outcome. In the aggregate, based on the use of the test in a population of patients, the aggregated outcomes will have a global societal impact.

Efficacy of Diagnostic Imaging

Fryback and Thornbury (Fryback DG, Thornbury JR. The Efficacy of Diagnostic Imaging. Med Decis Making 1991;11:88-94) published a conceptual framework for the assessment of the efficacy of diagnostic imaging that has been used extensively. Levels of efficacy in their model are given in the following table. Fryback & Thornbury levels of efficacy can also be linked to discrete steps in the imaging value chain and are annotated in red on the value chain graphic above.

Level Type of Efficacy   Meaning
1 Technical efficacy Physical capabilities to produce images of adequate quality (i.e. contrast resolution, spatial resolution)
2 Diagnostic accuracy efficacy Efficacy in making a diagnosis (i.e. sensitivity/specificity)
3 Diagnostic thinking efficacy  Efficacy in changing clinicians subject assessment of the probability of disease or appropriate therapeutic course (i.e. expert opinion)
4 Therapeutic efficacy Efficacy in changing the course of treatment (i.e. changes in drug therapy, surgical therapy, need for additional diagnostic tests)
5 Patient outcome efficacy  Improvement or decline in morbidity, mortality, or quality of life attributable to the test 
6 Societal efficacy  Benefit-cost or cost-effectiveness benefit from societal viewpoint 

Most diagnostic imaging research has focused on levels 1 through 3. Levels 4, 5, and 6 studies are much harder to conduct. They require multidisciplinary engagement and the reality is that it is unusual to rare for specific imaging questions to be included within what are primarily therapeutic studies. Consequently, when viewed from within the framework of the imaging value chain, evidentiary support for the use of imaging is often weak.

Appropriateness of Imaging Model

A recent publication by Gazelle et.al. (Gazelle, GS, et.al. A Framework for Assessing the Value of Diagnostic Imaging in the Era of Comparative Effectiveness Research, Radiology 2011;261(3):692- 698) details an approach to assessing the aggregate of available evidence to yield an overall assessment of appropriateness of imaging for a clinical context.

In this model, the clinical context is classified in three primary domains or pillars, including:

  • Size of the at-risk population
  • Anticipated clinical impact
  • Economic impact

In this framework, the impacts of the imaging in the three pillar domains are each classified on a 3-point scale (large, medium and small). This assessment is then used as a basis to designate the appropriate Fryback & Thornbury level of evidence that must be satisfactorily demonstrated by scientific data in order for the imaging exam to be deemed “appropriate” as depicted in the following graphic.


Evidentiary Review Procedure

Intermountain’s evidentiary review process uses the concepts described above to determine whether the use of an imaging test is adequately supported by medical evidence. Although clerical and administrative staff resources are used to compile and organize inputs, the development teams are charged to assess the quality, relevance, and to ultimately objectively determine the evidentiary support for the appropriate use of imaging by completing the following process steps.

  1. Performance of a literature search by use of medical search engines, identifying references within key review articles, identifying reference sources from other published AUCs and imaging guidelines.
    • Every effort is made to assure that this search includes publications from the imaging literature and from the literature from the relevant clinical services.
    • Assure that evidence on harms has been included.

  2. Collection of applicable internal Intermountain data. This can include both clinical data and financial data.

  3. Identification of consensus statements of national professional medical societies. Search for recommendations from all relevant medical societies.

  4. From the assembled evidence, identify key sources that are most relevant and that will be thoroughly reviewed.

  5. For each key evidence source determine the:
    • Rate of evidence level using “The Oxford 2011 Levels of Evidence” methodology.
    • Assign evidence source to the most relevant step(s) in imaging value chain and designate the Fryback & Thornbury level for the evidence source.

  6. Review the aggregated quality of evidence and Fryback & Thornbury levels of the key evidence sources. Determine the aggregate Fryback & Thornbury level of evidentiary support for the use of imaging.

  7. Determine the required level of Fryback & Thornbury efficacy required to confirm appropriateness of imaging by:
    • Applying the 3-point scale of grading for each of the 3 Gazelle pillars for the clinical context.
    • Determining the required Fryback & Thornbury level using the Gazelle framework.

  8. Make a final assessment of the appropriateness of the imaging test use in the clinical context (appropriate, inappropriate).