Critical appraisal
Critical appraisal overview
Before using studies in your review, you need to critically appraise them for quality and risk of bias.
Move through the slide deck below to learn more about critical appraisal. Alternatively, download the PDF document at the bottom of this box.
| The interactive element below requires you to enable cookies in your web browser |
-
Critical appraisalThis document is a printable version of the slide deck above.
Critical appraisal tools
Use a formal Critical Appraisal Tool ("CAT") to assess your papers. The tool must be applied without adaptation to the appropriate study design.
- AMSTARAssessment of Multiple Systematic Reviews (AMSTAR) is a 37-item assessment tool used to assess the methodological quality of systematic reviews.
- Critical appraisal skills programme (CASP)Tools for systematic reviews, randomised controlled trials, cohort studies, and case control studies
- JBI: Critical appraisal toolsCATs for a range of study types: case control, case reports, cohort, diagnostic test , economic evaluations, prevalence, quasi-experimental, RCTs, SRs and more.
- PEDro – Physiotherapy Evidence DatabaseDeveloped to support evidence-based practice in physiotherapy, PEDro can be used to find trials, reviews and guidelines evaluating physiotherapy interventions. Trials are assessed for quality using the PEDro scale. Aimed at a global audience and produced in Australia by the Institute for Musculoskeletal Health.
The application of a CAT ('CASP')
The Cochrane Common Mental Disorders group have produced 7 videos demonstrating the application of the CASP checklist to different study designs.
Guidelines and standards
- Cochrane Handbook, part 2, chapter 7: Considering bias and conflicts of interest amongst the included studies
This chapter provides requirements for authors performing critical appraisal for SRs in health sciences. The relevant MECIR are embedded. - Cochrane Handbook, part 2, chapter 8: Assessing risk of bias in a randomised trialThis chapter outlines how to critically appraise randomised trails using of the revised version of the Cochrane Risk of Bias tool, known as RoB 2.
'Specify the methods used to assess risk of bias in the included studies, including details of the tool(s) used, how many reviewers assessed each study and whether they worked independently, and, if applicable, details of automation tools used in the process.' - PRISMA 2020 Explanation and Elaboration, p. 11
Other standards
- Overview of systematic reviewsSee the overview page (of this guide) for additional guidelines and standards.
Hierarchy of evidence variations
Many hierarchies have been developed to show the different levels of evidence, and to 'rank' different study designs. See some common ones below:
-
NHMRC Evidence HierarchyEvidence Hierarchy used by Australia's National Health & Medical Research Council.
-
Brian Haynes 6S Hierarchy of Evidence Based ResourcesDeveloped in 2009 this is a useful model for guiding clinical decision making.
-
Levels of Evidence FAMEJBI, Collaborating Centres, and Evidence Translation Groups assign a level of evidence to all conclusions drawn in JBI Systematic Reviews.
-
CEBM 2020 Levels of Evidence SystemIn addition to traditional critical appraisal, this enables clinicians/patients to answer clinical questions quickly and without pre-appraised sources.
Further resources
- JBI: Critical appraisal toolsCATs for a range of study types: case control, case reports, cohort, diagnostic test , economic evaluations, prevalence, quasi-experimental, RCTs, SRs and more.
- Quality of reporting of observational longitudinal researchThis study developed and tested a checklist of criteria related to threats to the internal and external validity of observational longitudinal studies.
- Study designs (Centre for Evidence-Based Medicine)This short article gives a brief guide to the different study types and a comparison of the advantages and disadvantages.
- Using preprints in evidence synthesis: Commentary on experience during the COVID-19 pandemicClyne, B., Walsh, K. A., O'Murchu, E., et al. (2021). Journal of Clinical Epidemiology, 138, 203-210. https://doi.org/10.1016/j.jclinepi.2021.05.010