Critically Appraising Medical Evidence

The Critical Appraisal Skills Programme (CASP) is a non-profit UK organization based in Oxford, and provides resources and education aiming to improve the critical appraisal of medical evidence.

The most important CASP resources are eight critical appraisal tools to apply to medical research, covering Systematic Reviews, Randomised Controlled Trials, Cohort Studies, Case Control Studies, Economic Evaluations, Diagnostic Studies, Qualitative Studies and Clinical Prediction Rule.

CASP provides these freely under a Creative Commons license on their website as downloadable PDFs. Below are web-versions of the critical appraisal checklist for four primary research types, reformatted to make them more useful on laptops, tablets and smartphones.

Why Appraise Critically?

Critical Appraisal Skills enable you to assess the trustworthiness, relevance and results of published papers so that you can decide if they are believable and useful.

We evaluate scientific articles in three steps:

Is the study valid?

The first step is to decide whether the study was unbiased by evaluating its methodological quality. Different criteria for validity of articles are used for different types of question: questions on treatment, diagnosis, prognosis and economic evaluation. Depending on the validity of an article we can classify it within a scale of levels of evidence and degrees of recommendation.

What are the results?

If we decide that the study is valid, we can go on to looking at the results.  At this step we consider whether the study’s results are clinically important.  For example, did the experimental group show a significantly better outcome compared with the control group?  We also consider how much uncertainty there is about the results, as expressed in the form of p values, confidence intervals and sensitivity analysis.

Are the results applicable to my needs?

Once you have decided that your evidence is valid and important, you need to think about how it applies to your clinical question.  It’s likely, for example, that your patient or population may have different characteristics to those in the study.  CASP provides a framework within which to consider these issues in an explicit, transparent way.

Appraising a randomised controlled trial

(A) Are the results of the review valid?

Screening Questions

1. Did the trial address a clearly focused issue?

 Yes  Can’t tell  No

HINT: An issue can be ‘focused’ In terms of

  • The population studied.
  • The intervention given.
  • The comparator given.
  • The outcomes considered.

2. Was the assignment of patients to treatments randomised?

 Yes  Can’t tell  No

HINT: Consider

  • How was this carried out?
  • Was the allocation sequence concealed from researchers and patients?

3. Were all of the patients who entered the trial properly accounted for at its conclusion?

 Yes  Can’t tell  No

HINT: Consider

  • Was the trial stopped early?
  • Were patients analysed in the groups to which they were randomised?

Is it worth continuing?

Detailed questions

4. Were patients, health workers and study personnel ‘blind’ to treatment?

 Yes  Can’t tell  No

HINT: Think about

  • Patients?
  • Health workers?
  • Study personnel?

5. Were the groups similar at the start of the trial?

 Yes  Can’t tell  No

HINT: Look at

  • Other factors that might affect the outcome such as age, sex, social class

6. Aside from the experimental intervention, were the groups treated equally?

 Yes  Can’t tell  No

(B) What are the results?

7. How large was the treatment effect?

HINT: Consider

  • What outcomes were measured?
  • Is the primary outcome clearly specified?
  • What results were found for each outcome?

8. How precise was the estimate of the treatment effect?

HINT: Consider

  • What are the confidence limits?

(C) Will the results help locally?

9. Can the results be applied in your context? (or to the local population?)

 Yes  Can’t tell  No

HINT: Consider whether

  • Do you think that the patients covered by the trial are similar enough to the patients to whom you will apply this?, if not how to they differ?

10. Were all clinically important outcomes considered?

 Yes  Can’t tell  No

HINT: Consider

  • Is there other information you would like to have seen?
  • If not, does this affect the decision?

11. Are the benefits worth the harms and costs?

 Yes  Can’t tell  No

HINT: Consider

  • Even if this is not addressed by the review, what do you think?

Appraising a systematic review

(A) Are the results of the review valid?

Screening Questions

1. Did the review address a clearly focused question?

 Yes  Can’t tell  No

HINT: An issue can be ‘focused’ In terms of

  • The population studied
  • The intervention given
  • The outcome considered

2. Did the authors look for the right type of papers?

 Yes  Can’t tell  No

HINT: ‘The best sort of studies’ would

  • Address the reviews question
  • Have an appropriate study design (usually RCTs for papers evaluating interventions)

Is it worth continuing?

Detailed questions

3. Do you think the important, relevant studies were included?

 Yes  Can’t tell  No

HINT: Look for

  • Which bibliographic databases were used.
  • Follow up from reference lists.
  • Personal contact with experts.
  • Search for unpublished as well as published studies.
  • Search for non-English language studies.

4. Did the review’s authors do enough to assess the quality of the included studies?

 Yes  Can’t tell  No

HINT: The authors need to consider the rigour of the studies they have identified. Lack of rigour may affect the studies’ results. (“All that glisters is not gold” Merchant of Venice – Act II)

5. If the results of the review have been combined, was it reasonable to do so?

 Yes  Can’t tell  No

HINT: Consider whether

  • The results were similar from study to study
  • The results of all the included studies are clearly displayed
  • The results of the different studies are similar
  • The reasons for any variations in results are discussed

(B) What are the results?

6. What are the overall results of the review?

 Yes  Can’t tell  No

HINT: Consider

  • If you are clear about the review’s ‘bottom line’ results?
  • What these are (numerically if appropriate)?
  • How were the results expressed (NNT, odds ratio etc)?

7. How precise are the results?

HINT: Look at the confidence intervals, if given.

(C) Will the results help locally?

8. Can the results be applied to the local population?

 Yes  Can’t tell  No

HINT: Consider whether

  • The patients covered by the review could be sufficiently different to your population to cause concern.
  • Your local setting is likely to differ much from that of the review.

9. Were all important outcomes considered?

 Yes  Can’t tell  No

HINT: Consider whether

  • Is there other information you would like to have seen

10. Are the benefits worth the harms and costs?

 Yes  Can’t tell  No

HINT: Consider

  • Even if this is not addressed by the review, what do you think?

Appraising a cohort study

(A) Are the results of the study valid?

Screening Questions

1. Did the study address a clearly focused issue?

 Yes  Can’t tell  No

HINT: A question can be ‘focused’ In terms of

  • The population studied
  • The risk factors studied
  • The outcomes considered
  • Is it clear whether the study tried to detect a beneficial or harmful effect?

2. Was the cohort recruited in an acceptable way?

 Yes  Can’t tell  No

HINT: Look for selection bias which might compromise the generalisibility of the findings:

  • Was the cohort representative of a defined population?
  • Was there something special about the cohort?
  • Was everybody included who should have been included?

Is it worth continuing?

Detailed questions

3. Was the exposure accurately measured to minimise bias?

 Yes  Can’t tell  No

HINT: Look for measurement or classification bias:

  • Did they use subjective or objective measurements?
  • Do the measurements truly reflect what you want them to (have they been validated)?
  • Were all the subjects classified into exposure groups using the same procedure

4. Was the outcome accurately measured to minimise bias?

 Yes  Can’t tell  No

HINT: Look for measurement or classification bias:

  • Did they use subjective or objective measurements?
  • Do the measures truly reflect what you want them to (have they been validated)?
  • Has a reliable system been established for detecting all the cases (for measuring disease occurrence)?
  • Were the measurement methods similar in the different groups?
  • Were the subjects and/or the outcome assessor blinded to exposure (does this matter)?

5. (a) Have the authors identified all important confounding factors?

List the ones you think might be important, that the author missed.

 Yes  Can’t tell  No

HINT: Look for restriction in design, and techniques e.g. modelling, stratified-, regression-, or sensitivity analysis to correct, control or adjust for confounding factors

6.  (a) Was the follow up of subjects complete enough?

      (b) Have they taken account of the confounding factors in the design?

      (b) Was the follow up of subjects long enough?

 Yes  Can’t tell  No

HINT: Consider

  • The good or bad effects should have had long enough to reveal themselves.
  • The persons that are lost to follow-up may have different outcomes than those available for assessment.
  • In an open or dynamic cohort, was there anything special about the outcome of the people leaving, or the exposure of the people entering the cohort?

(B) What are the results?

7. What are the results of this study?

HINT: Consider

  • What are the bottom line results?
  • Have they reported the rate or the proportion between the exposed/unexposed, the ratio/the rate difference?
  • How strong is the association between exposure and outcome (RR)?
  • What is the absolute risk reduction (ARR)?

8. How precise are the results?

HINT: Look for the range of the confidence intervals, if given.

9. Do you believe the results?

HINT: Consider

  • Big effect is hard to ignore!
  • Can it be due to bias, chance or confounding?
  • Are the design and methods of this study sufficiently flawed to make the results unreliable?
  • Bradford Hills criteria (e.g. time sequence, dose-response gradient, biological plausibility, consistency)

 Yes  Can’t tell  No

(C) Will the results help locally?

10. Can the results be applied to the local population?

 Yes  Can’t tell  No

HINT: Consider whether

  • A cohort study was the appropriate method to answer this question?
  • The subjects covered in this study could be sufficiently different from your population to cause concern?
  • Your local setting is likely to differ much from that of the study?
  • You can quantify the local benefits and harms?

11. Do the results of this study fit with other available evidence?

 Yes  Can’t tell  No

12. What are the implications of this study for practice?

HINT: Consider

  • One observational study rarely provides sufficiently robust evidence to recommend changes to clinical practice or within health policy decision making.
  • For certain questions observational studies provide the only evidence.
  • Recommendations from observational studies are always stronger when supported by other evidence.

Appraising a case control study

(A) Are the results of the study valid?

Screening Questions

1. Did the study address a clearly focused issue?

 Yes  Can’t tell  No

HINT: A question can be focused in terms of

  • The population studied.
  • The risk factors studied.
  • Whether the study tried to detect a beneficial or harmful effect?

2. Did the authors use an appropriate method to answer their question?

HINT: Consider

  • Is a case control study an appropriate way of answering the question under the circumstances? (Is the outcome rare or harmful)
  • Did it address the study question?

 Yes  Can’t tell  No

Is it worth continuing?

Detailed questions

3. Were the cases recruited in an acceptable way?

 Yes  Can’t tell  No

HINT: We are looking for selection bias which might compromise validity of the findings

  • Are the cases defined precisely?
  • Were the cases representative of a defined population? (geographically and/or temporally?)
  • Was there an established reliable system for selecting all the cases?
  • Are they incident or prevalent?
  • Is there something special about the cases?
  • Is the time frame of the study relevant to disease/exposure?
  • Was there a sufficient number of cases selected?
  • Was there a power calculation?

4. Were the controls selected in an acceptable way?

 Yes  Can’t tell  No

HINT: We are looking for selection bias which might compromise the generalisibilty of the findings.

  • Were the controls representative of defined population (geographically and/or temporally)
  • Was there something special about the controls?
  • Was the non-response high? Could non-respondents be different in any way?
  • Are they matched, population based or randomly selected?
  • Was there a sufficient number of controls selected?

5. Was the exposure accurately measured to minimise bias?

 Yes  Can’t tell  No

HINT: We are looking for measurement, recall or classification bias

  • Was the exposure clearly defined and accurately measured?
  • Did the authors use subjective or objective measurements?
  • Do the measures truly reflect what they are supposed to measure? (Have they been validated?)
  • Were the measurement methods similar in the cases and controls?
  • Did the study incorporate blinding where feasible?
  • Is the temporal relation correct? (Does the exposure of interest precede the outcome?)

6. (a) What confounding factors have the authors accounted for?

 Yes  Can’t tell  No

HINT: List the ones you think might be important, that the author missed.

  • Genetic
  • Environmental
  • Socio-economic

6. (b) Have the authors taken account all of the potential confounding factors in the design and/or in their analysis?

 Yes  Can’t tell  No

HINT: Look for

  • Restriction in design, and techniques e.g. modelling stratified-, regression-, or sensitivity analysis to correct, control or adjust for confounding factors.

7. What are the results of this study?

HINT: Consider

  • What are the bottom line results?
  • Is the analysis appropriate to the design?
  • How strong is the association between exposure and outcome (look at the odds ratio)?
  • Are the results adjusted for confounding, and might confounding still explain the association?
  • Has adjustment made a big difference to the OR?

(B) What are the results?

8. How precise are the results? How precise is the estimate of risk?

HINT: Consider

  • Size of the P-value
  • Size of the confidence intervals
  • Have the authors considered all the important variables?
  • How was the effect of subjects refusing to participate evaluated?

9. Do you believe the results?

 Yes  No

HINT: Consider

  • Big effect is hard to ignore!
  • Can it be due to chance, bias or confounding?
  • Are the design and methods of this study sufficiently flawed to make the results unreliable?
  • Consider Bradford Hills criteria (e.g. time sequence, dose-response gradient, strength, biological plausibility)

(C) Will the results help locally?

10. Can the results be applied to the local population?

 Yes  Can’t tell  No

HINT: Consider whether

  • The subjects covered in the study could be sufficiently different from your population to cause concern?
  • Your local setting is likely to differ much from that of the study
  • Can you quantify the local benefits and harms?

11. Do the results of this study fit with other available evidence?

 Yes  Can’t tell  No

HINT: Consider all the available evidence from RCT’s, systematic reviews, cohort studies and case-control studies as well for consistency.

Remember:

One observational study rarely provides sufficiently robust evidence to recommend changes to clinical practice or within health policy decision making. However, for certain questions observational studies provide the only evidence. Recommendations from observational studies are always stronger when supported by other evidence.

 
CASP checklists are copyright CASP and shared under a Creative Commons Attribute-Noncommercial-Share-alike license.

Back to Top ↑

Skip to toolbar