J Gen Intern Med. 2012 Jun;27 Suppl 1:76-82.

Abstract

Limited by what is reported in the literature, most systematic reviews of medical tests focus on “test accuracy” (or better, test performance), rather than on the impact of testing on patient outcomes. The link between testing, test results and patient outcomes is typically complex: even when testing has high accuracy, there is no guarantee that physicians will act according to test results, that patients will follow their orders, or that the intervention will yield a beneficial endpoint. Therefore, test performance is typically not sufficient for assessing the usefulness of medical tests. Modeling (in the form of decision or economic analysis) is a natural framework for linking test performance data to clinical outcomes. We propose that (some) modeling should be considered to facilitate the interpretation of summary test performance measures by connecting testing and patient outcomes. We discuss a simple algorithm for helping systematic reviewers think through this possibility, and illustrate it by means of an example.

J Gen Intern Med. 2012 Jun;27 Suppl 1:67-75.

Abstract

The classical paradigm for evaluating test performance compares the results of an index test with a reference test. When the reference test does not mirror the “truth” adequately well (e.g. is an “imperfect” reference standard), the typical (“naïve”) estimates of sensitivity and specificity are biased. One has at least four options when performing a systematic review of test performance when the reference standard is “imperfect”: (a) to forgo the classical paradigm and assess the index test’s ability to predict patient relevant outcomes instead of test accuracy (i.e., treat the index test as a predictive instrument); (b) to assess whether the results of the two tests (index and reference) agree or disagree (i.e., treat them as two alternative measurement methods); (c) to calculate “naïve” estimates of the index test’s sensitivity and specificity from each study included in the review and discuss in which direction they are biased; (d) mathematically adjust the “naïve” estimates of sensitivity and specificity of the index test to account for the imperfect reference standard. We discuss these options and illustrate some of them through examples.

J Gen Intern Med. 2012 Jun;27 Suppl 1:28-32.

Abstract

This article discusses techniques that are appropriate when developing search strategies for systematic reviews of medical tests. This includes general advice for searching for systematic reviews and issues specific to systematic reviews of medical tests. Diagnostic search filters are currently not sufficiently developed for use when searching for systematic reviews. Instead, authors should construct a highly sensitive search strategy that uses both controlled vocabulary and text words. A comprehensive search should include multiple databases and sources of grey literature. A list of subject-specific databases is included in this article.

J Gen Intern Med. 2012 Jun;27 Suppl 1:20-7.

Abstract

In this chapter of the Evidence-based Practice Centers Methods Guide for Medical Tests, we describe how the decision to use a medical test generates a broad range of outcomes and that each of these outcomes should be considered for inclusion in a systematic review. Awareness of these varied outcomes affects how a decision maker balances the benefits and risks of the test; therefore, a systematic review should present the evidence on these diverse outcomes. The key outcome categories include clinical management outcomes and direct health effects; emotional, social, cognitive, and behavioral responses to testing; legal and ethical outcomes, and costs. We describe the challenges of incorporating these outcomes in a systematic review, suggest a framework for generating potential outcomes for inclusion, and describe the role of stakeholders in choosing the outcomes for study. Finally, we give examples of systematic reviews that either included a range of outcomes or that might have done so. The following are the key messages in this chapter: Consider both the outcomes that are relevant to the process of testing and those that are relevant to the results of the test. Consider inclusion of outcomes in all five domains: clinical management effects, direct test effects; emotional, social, cognitive and behavioral effects; legal and ethical effects, and costs. Consider to which group the outcomes of testing are most relevant. Given resource limitations, prioritize which outcomes to include. This decision depends on the needs of the stakeholder(s), who should be assisted in prioritizing the outcomes for inclusion.

J Gen Intern Med. 2012 Jun;27 Suppl 1:11-9.

Abstract

Topic development and structuring a systematic review of diagnostic tests are complementary processes. The goals of a medical test review are to identify and synthesize evidence to evaluate the impacts alternative testing strategies on health outcomes and to promote informed decisionmaking. A common challenge is that the request for a review may state the claim for the test ambiguously. Due to the indirect impact of medical tests on clinical outcomes, reviewers need to identify which intermediate outcomes link a medical test to improved clinical outcomes. In this paper, we propose the use of five principles to deal with challenges: the PICOTS typology (patient population, intervention, comparator, outcomes, timing, setting), analytic frameworks, simple decision trees, other organizing frameworks and rules for when diagnostic accuracy is sufficient.

J Gen Intern Med. 2012 Jun;27 Suppl 1:39-46.

Abstract

Use of medical tests should be guided by research evidence about the accuracy and utility of those tests in clinical care settings. Systematic reviews of the literature about medical tests must address applicability to real-world decision-making. Challenges for reviews include: (1) lack of clarity in key questions about the intended applicability of the review, (2) numerous studies in many populations and settings, (3) publications that provide too little information to assess applicability, (4) secular trends in prevalence and the spectrum of the condition for which the test is done, and (5) changes in the technology of the test itself. We describe principles for crafting reviews that meet these challenges and capture the key elements from the literature necessary to understand applicability.

J Gen Intern Med. 2012 Jun;27 Suppl 1:4-10.

Abstract

Evaluation of medical tests presents challenges distinct from those involved in the evaluation of therapies; in particular, the very great importance of context and the dearth of comprehensive RCTs aimed at comparing the clinical outcomes of different tests and test strategies. Available guidance provides some suggestions: 1) Use of the PICOTS typology for clarifying the context relevant to the review, and 2) use of an organizing framework for classifying the types of medical test evaluation studies and their relationship to potential key questions. However, there is a diversity of recommendations for reviewers of medical tests and a proliferation of concepts, terms, and methods. As a contribution to the field, this Methods Guide for Medical Test Reviews seeks to provide practical guidance for achieving the goals of clarity, consistency, tractability, and usefulness.

J Gen Intern Med. 2012 Jun;27 Suppl 1:33-8.

Abstract

Assessing methodological quality is a necessary activity for any systematic review, including those evaluating the evidence for studies of medical test performance. Judging the overall quality of an individual study involves examining the size of the study, the direction and degree of findings, the relevance of the study, and the risk of bias in the form of systematic error, internal validity, and other study limitations. In this chapter of the Methods Guide for Medical Test Reviews, we focus on the evaluation of risk of bias in the form of systematic error in an individual study as a distinctly important component of quality in studies of medical test performance, specifically in the context of estimating test performance (sensitivity and specificity). We make the following recommendations to systematic reviewers: 1) When assessing study limitations that are relevant to the test under evaluation, reviewers should select validated criteria that examine the risk of systematic error, 2) categorizing the risk of bias for individual studies as “low,” “medium,” or “high” is a useful way to proceed, and 3) methods for determining an overall categorization for the study limitations should be established a priori and documented clearly.