RobotReviewer at the Global Evidence Summit #GESummit17

RobotReviewer (Iain and Joël) is at the Global Evidence Summit in Cape Town! 

We are looking forward to our workshop this afternoon (which is fully booked).

It also would be great to see you at our talk, or poster session, or around the summit :)

Thursday 12:30-14:00: Poster presentation (number 2040)

Saturday 14:00-15:30: Presentation in Long oral session 25: Tools for evidence production and synthesis (Room 1.44)

 

file.jpeg

RobotReviewer demo — links for ACL 2017 paper

Welcome to RobotReviewer! We have collected some links here associated with our demo at ACL 2017. RobotReviewer takes clinical trial reports (journal articles in PDF format), and automatically produces evidence summaries.

Try out the demonstration site

You may try out the system here if you have some clinical trial PDFs.

Example reports

Alternately, we have saved some example reports (using open access clinical trial PDFs) on a few clinical topics, which can be accessed from the following links:

Source code

Our full system is open source, released under the GPL v3.0, and is available on our github page.

Demonstration video

Finally, we give a demonstration of the use of RobotReviewer, and its key features in the short video below.

Come work with us on RobotReviewer

We're delighted to announce that we're recruiting a Research Associate to work with us on RobotReviewer.

This is a two year post based at King's College London for someone with background in health sciences or computer science.

Full details and the application form can be found on the King's College London jobs site.

Introducing the new version of RobotReviewer, first steps to auto-synthesis!

The new version of RobotReviewer has been live on our website for a couple of months now, but we've just released all the code as open source, which is a good opportunity to write a post!

The goal of the RobotReviewer project is to speed up evidence-based medicine, by automating, or semi-automating the extraction of data from clinical trial reports.

To date we can to automatically extract information about bias (using the Cochrane Risk of Bias tool), and additionally can extract information about the participants, interventions, and outcomes studied in a trial. We are actively researching and adding new machine learning models to the system, and aim to extract all the pieces of information from a clinical trial needed to produce a systematic review.

The first version of RobotReviewer processed one paper at a time. The most important change is in how RobotReviewer can process multiple trial PDFs at once. This will allow us in future to move beyond summarising individual trials, and towards synthesis.

The first screen of the new RobotReviewer interface. Simply drag a bunch of PDFs of clinical trial reports to the dashed box to start the process

The first screen of the new RobotReviewer interface. Simply drag a bunch of PDFs of clinical trial reports to the dashed box to start the process

Thinking time... takes a minute or two

Thinking time... takes a minute or two

Then comes our new Report View, which is the main user interface. We've taken small steps towards synthesising the results from individual studies, with the Risk of Bias table being one example.

 

Automatic Risk of Bias tables! Note in the top right we now have download buttons, you can download the report in HTML, MS Word, or JSON formats.

Automatic Risk of Bias tables! Note in the top right we now have download buttons, you can download the report in HTML, MS Word, or JSON formats.

In the background, RobotReviewer now automatically recognises the identity of an uploaded trial, and is able to retrieve relevant related information from PubMed, and ICTRP. For the moment, we simply use this to make nicely formatted citations, but clearly lots more is possible. 

Further down the report, characteristics of the study population, interventions, and outcomes is extracted, plus the justifications for the judgements in the Risk of Bias table.

Further down the report, characteristics of the study population, interventions, and outcomes is extracted, plus the justifications for the judgements in the Risk of Bias table.

One key feature is that we retain links to the source PDF. One important problem in research synthesis is that the provenance of data is often not clear. By contrast, by clicking any of the PDF icons next to the extracted data, RobotReviewer will take you to the exact place in the PDF where the data came from.

the source PDF with the text describing the trial population highlighted in green

the source PDF with the text describing the trial population highlighted in green

We hope you enjoyed this tour of our new RobotReviewer! We're excited by the possibilities our new framework gives us, and will post updates as they happen.

Our code is available here on GitHub.

RobotReviewer + Trip

Automatic bias assessment for articles in the Trip Database

We are excited to announce that the Trip Database now uses RobotReviewer to automatically identify trials that are likely to have the lowest biases in conduct and reporting. Bias, when present, leads to over- or under-estimates of true intervention effects, in turn complicating treatment choice. Bias can be introduced in several ways in the context of clinical trials (e.g., improper blinding or poor randomization); these domains have been codified in the Cochrane risk of bias tool. RobotReviewer predicts the risk of bias for each of these.

   

 

 

RobotReviewer's risk of bias estimates are based on the words and short phrases used in the titles and abstracts of papers. RobotReviewer has ingested thousands of such articles and identified statistical correlations between word use and manually assessed risks of bias. This general approach is known as “machine learning”.

The full RobotReviewer system (shown above) performs a risk of bias assessment using the Cochrane Risk of Bias tools. In general, RobotReviewer accepts as input the full text of clinical trial reports (as PDFs). For the version deployed within Trip, RobotReviewer performs a more limited assessment based on the title and abstract only. Moreover, in this case bias predictions are limited to the Random sequence generation, Allocation concealment, and Blinding domains.

The full RobotReviewer system (shown above) performs a risk of bias assessment using the Cochrane Risk of Bias tools. In general, RobotReviewer accepts as input the full text of clinical trial reports (as PDFs). For the version deployed within Trip, RobotReviewer performs a more limited assessment based on the title and abstract only. Moreover, in this case bias predictions are limited to the Random sequence generation, Allocation concealment, and Blinding domains.

How does RobotReviewer work?

RobotReviewer has ‘learned’ how to assess bias by examining the titles and abstracts of tens of thousands of articles describing clinical trials that are also included in the Cochrane Database of Systematic Reviews. These trials have all been manually assessed for bias by systematic review authors, using the Cochrane Risk of Bias tool. As mentioned above, this tool is used to assess various common biases in Randomized Controlled Trials, including whether a trial used robust randomization, and whether participants were adequately blinded to which intervention they received. Specifically, the version of RobotReviewer used in Trip focuses on biases in random sequence generation, allocation concealment, and blinding.

While the predictions are reasonably accurate, we caution that they are not perfect, especially because they rely only on the titles and abstracts of articles (rather than full-texts). Nonetheless, in the context of Trip searches, the predictions can help quickly identify what is likely to be the highest quality research for a given search topic. If you require very high accuracy bias assessments (for example if you are conducting a systematic review) you will of course need to consult the full text of the paper. You may wish to use the full RobotReviewer tool, which provides detailed information on all the domains of the Cochrane Risk of Bias tool from full text papers and can aid in your assessments.

For more information on the algorithm used in RobotReviewer, please see the references below.

[1] Iain J. Marshall, Joël Kuiper, and Byron C. Wallace. "RobotReviewer: evaluation of a system for automatically assessing bias in clinical trials." Journal of the American Medical Informatics Association (2015): ocv044.

[2] Iain J. Marshall, Joël Kuiper, and Byron C. Wallace. "Automating risk of bias assessment for clinical trials." IEEE journal of biomedical and health informatics 19.4 (2015): 1406-1412.