Systematic Review Service

This step is closely tied to Step 7: Collect the Data. Parts of these steps are intertwined or simultaneous.

This step can be commonly referred to as quality appraisal, critical appraisal, quality assessment, risk of bias assessment, or validity assessment.

While there are some nuances to these terms, in general, they are used to refer to the same process: using a tool/instrument to assess the quality of the individual studies you have included in your review.

This is a key part of the systematic review process, and like with the two phases of screening, at least two reviewers must be involved.

There are a large number of tools to choose from. Your choice will depend on the types of studies you are looking at (e.g., randomized controlled trials, diagnostic studies, mixed methods studies, intervention studies, etc.).

You can find lists of tools here:

• Quality Assessment and Risk of Bias Tool Repository
• Latitudes Network

Some journals specify the tool you should use in the author instructions, or require the use of a validated tool. Always check your target journal's requirements prior to tool selection.

Note that scoping reviews do not need to complete this step.

Like with quality appraisal above, this step is referred to with many names, including certainty assessment, confidence assessment, certainty of evidence, quality of evidence, confidence in evidence, and strength of recommendations. Assessing can also be referred to as grading or rating.

In short, this involves assessing the certainty you have in the body of evidence found and the strength of recommendations you can make.

Quality appraisal seems similar to certainty assessment, but these are generally separate, as the first is about risk of bias in individual studies and the second goes further to consider the overall quality of evidence for each outcome assessed. One tool where they are linked is Grading of Recommendations Assessment, Development and Evaluation (GRADE), which many systematic review teams use for this assessment, as it looks at risk of bias, inconsistency, indirectness, imprecision, and publication bias. GRADE is a part of completing summary of findings tables.

Your MSK librarian can point you to resources for selecting an appropriate appraisal tool. They can also assist with setting up a pilot assessment project in Covidence prior to finalizing tool selection and implementation.

Covidence includes a customizable module for quality assessment. The Cochrane Risk of Bias tool is the default tool given, but the module is fully customizable. 

Learn more about customizing your Covidence Quality Assessment template: