Systematic Review Service

Artificial intelligence (AI) is an evolving field with a lot of interest. There is the promise that it can cut down on the amount of time it takes to complete a review, and it can! But you have to know the strengths and limits of the tool you are using.

If you ever have any questions about using AI in your research or are curious about a specific AI tool, please reach out! We would be happy to explore this with you.

GenAI can be helpful for generating ideas that you vet and adapt for your review. For example, something like ChatGPT might be beneficial to inform your research question and help you brainstorm inclusion and exclusion criteria and what terms you want in your search strategy. 

GenAI cannot:

• Give you a list of the most important articles for your topic.
• Build a systematic search strategy.

GenAI tools have a difficult time being comprehensive and will often suggest fake articles or Medical Subject Headings (MeSH). These issues are popularly known as hallucinations (i.e., it is making things up).

Keep in mind that while ChatGPT and other similar AI tools are often described with human-like terms, what they are designed to do is process your question by predicting the next word. This is not thinking, and they are not able to actually understand what you are researching.

AI can be built into review software. For example, Covidence, an online software platform used for systematic and other related reviews, uses machine learning to sort records by relevance and classify randomized controlled trials. Covidence is also beta testing using team-entered inclusion and exclusion criteria to auto-remove records.

Before using an AI tool, explore its parameters, and be skeptical about anything that seems too good to be true or removes all human intervention from a task.

The MSK Library has experience with AI companies promising they can generate fully-automated systematic reviews for you in a short amount of time. Systematic reviews are very time and labor intensive, but if less time is taken and corners are cut, quality is lost.

Here are a few of the limitations we have compiled via exploration and demos; these AI tools generally:

• Cannot generate a reproducible search strategy.
• Can only pull from open access articles freely available online.
• Have limits to the number of articles included.
• Are not able to show you if an article has been retracted.
• Do not include a risk of bias assessment in their methodology.
• Lack transparency.

You cannot follow systematic review reporting guidelines such as PRISMA if you use these tools. Until the standards change, using them will not adhere to best practices.