Systematic Review Service

This step is closely tied to Step 7: Collect the Data. Parts of these steps are intertwined or simultaneous.

This step can be commonly referred to as quality appraisal, critical appraisal, quality assessment, risk of bias assessment, or validity assessment.

While there are some nuances to these terms, in general, they are used to refer to the same process: using a tool/instrument to assess the quality of the individual studies you have included in your review.

This is a key part of the systematic review process, and like with the two phases of screening, at least two reviewers must be involved.

There are a large number of tools to choose from. Your choice will depend on the types of studies you are looking at (e.g., randomized controlled trials, diagnostic studies, mixed methods studies, intervention studies, etc.).

You can find lists of tools here:

• Quality Assessment and Risk of Bias Tool Repository
• Latitudes Network

Some journals specify the tool you should use in the author instructions, or require the use of a validated tool. Always check your target journal's requirements prior to tool selection.

Note that scoping reviews do not need to complete this step.

Like with quality appraisal above, this step is referred to with many names, including certainty assessment, confidence assessment, certainty of evidence, quality of evidence, confidence in evidence, strength of evidence, and strength of recommendations. Assessing can also be referred to as grading or rating.

In short, this involves assessing the certainty you have in the body of evidence found and the strength of recommendations you can make. Quality appraisal seems similar to certainty assessment, but these are generally separate, as the first is about risk of bias in individual studies and the second goes further to consider the overall quality of evidence for each outcome assessed.

Many systematic review teams use the Grading of Recommendations Assessment, Development and Evaluation (GRADE) approach for certainty assessment, as it looks at risk of bias at the outcome level, inconsistency, indirectness, imprecision, and publication bias. GRADE can also be used to report the results of certainty assessment into an evidence summary table. Within GRADE this is known as a Summary of Findings table, where the evidence is graded one outcome at a time and presented in a clear format.

• GRADE Working Group
• GRADE Handbook

Your MSK librarian can point you to resources for selecting an appropriate quality appraisal tool. They can also assist with setting up a pilot assessment project in Covidence prior to finalizing tool selection and implementation.

Once within the extraction stage in Covidence, you will see the option to create a quality assessment template.

Cochrane's original risk of bias (RoB) tool was designed for randomized trials, and is the default within the template, but the template is fully customizable. Covidence does not support multiple quality appraisal tools within one review, but you can add multiple tools (or just the additional fields you need to assess multiple study types) to the same template.

Learn more: