Systematic review ethical considerations

It is inevitable that, if a type of study method sits at the top of the pyramid of evidence hierarchy, everybody will want to either conduct such a study or publish them. Because systematic reviews generally do not need ethics committee or institutional review board approval, nor are faced with any of the other multiple obstacles to the conduct of clinical research, they would seem an ideal endeavor for anyone seeking to improve their CV or impact score.

The result is the proliferation of so-called systematic reviews in the literature as has been recently highlighted in a key article by John Ioannidis1 and reflected in a commentary by Page and Moher in Milbank Quarterly.2 Ioannidis points to an exponential increase in the number of systematic reviews being published and highlights key flaws in the process—particularly poor methods and conflicted interests but also much duplication and redundancy.

The effect of this is to downgrade the actual level of evidence many systematic reviews provide. The strictest intellectual critique should be applied to any article claiming to be systematic and actually this is easily done for readers and journal editors by using the PRISMA3,4 checklists for systematic reviews.

“Is this so-called systematic review actually systematic?” should be the paramount question in the mind of the critical appraiser. What makes the difference between a systematic review and a narrative literature review?

It is the difference between a subjective and essentially personal view of the literature, usually on quite a broad subject, and an objective description [and when appropriate—meta-analysis] of the state of knowledge for a specific question. A systematic review is a reproducible piece of observational research and should have a protocol that sets out explicitly objective methods for the conduct of the review, particularly focusing on the control of error, both from bias and the reduction of random error through meta-analysis. Especially important in a systematic review is the objective, methodologically sound and reproducible retrieval of the evidence using sensitive and specific search strategies devised by a trained and experienced information scientist. This must ensure the retrieval of all relevant evidence in a process that is free from any selection bias- See Table 1 for a list of the components of a sysematic review.

All Cochrane reviews have a protocol which is peer reviewed and published before the review work commences. The peer review part of this process is essential to ensure:

• The review is needed and has not already been done—in other words it is not going to be redundant;

• The Population[s], Intervention[s], Comparison[s] and Outcome[s] are clearly defined including whether or not patients or their caregivers have been involved in identifying these;

• The methods are clearly specified including searching for studies, including and excluding studies, data extraction from studies, meta-analytic methods and the quantification and management of heterogeneity and risk of bias. Any planned subgroup or sensitivity analyses are set out in detail;

• Transparent methods are used to assess the overall certainty of the evidence, for example GRADE5 which provides an explicit method for doing this.

The requirement to write a protocol, and have it peer reviewed, inevitably prolongs the timelines to final publication and is probably why many authors choose to use Cochrane methods but seek to get their reviews published elsewhere.

A systematic review is more than just a description of what is out there. It should provide a filter to avoid biased interpretation of the evidence while reducing imprecision through meta-analysis when appropriate. This is done with explicit methods for the control of bias [and confounding for non-randomized studies] defined a priori in the protocol. That protocol should be published or at least registered online [e.g. Prospero6] so that it can be accessed and compared to what is finally published in the completed review. The protocol is critically important in differentiating a systematic review from a narrative review which can so easily drift by being influenced by what is found in the searches rather than remaining focused on the defined question.

Control of bias and confounding is inherently more straightforward for randomized controlled trials because their design, when properly conducted, limits the influence of both. However, assessment of bias is still needed in reviews of trials for four main domains:

• Selection bias [controlled by generating a random allocation sequence and concealing this from people recruiting participants to the trial];

• Performance and detection bias [controlled by masking];

• Attrition bias [controlled by high follow-up and intention-to-treat analysis];

• Selective outcome reporting bias [all pre-specified trial outcomes be reported].

All of these sources of bias are assessed systematically in Cochrane reviews and the risk of bias assessment then informs the summary of findings using the GRADE approach to judge the certainty of the evidence. For non-randomized observational studies, control of bias and confounding is much more challenging in systematic reviews.

Serious bias comes from the well-known propensity for positive findings to be published and published quickly while negative ones more slowly or not at all. This remains a huge challenge to the publishers of biomedical literature. Trials registries provide a means of dealing with this but the need to register trials has to be enforced by ethics committees and the registries themselves monitored and maintained.

For all clinicians to have instant access to the most up-to-date findings from research is an ideal towards which we will continually strive and was Archie Cochrane’s challenge. Such improvement can happen if the will is there to make it so. Our patients naturally expect health experts to be indeed expert and know about all the relevant and latest developments in the treatment of their condition.

Out-of-date syntheses are potentially dangerous especially when underpowered [despite meta-analysis] to safely draw conclusions about effectiveness. An example of this is the story about steroids for acute brain injury. An old review of a number of small and unreliable studies suggested that the standard treatment with the use of high-dose corticosteroids in severe head injury was probably life-saving [though inconclusive]. This uncertainty lead to a properly powered multinational randomized control trial CRASH 17 which found that the standard treatment was harmful and that giving corticosteroids increased the risk of death. The original review was now out of date. It would, if not corrected quickly, be still out there giving entirely the wrong message.

Textbooks can only teach us methods now. It is no longer possible for a book to provide up-to-date and reliable information about effectiveness of treatments. This is why the Cochrane model of online publication and updating is imperative. Hard copy journals have a role in making readers aware of what is happening online and of course for publishing primary research. Though they too may eventually be replaced by more efficient, affordable and potentially less biased means of disseminating evidence.

High quality systematic reviews provide a vital resource for trainees as long as they are easily accessible and quality controlled. So editors of journals which publish systematic reviews have a serious responsibility to ensure the reviews they publish meet the highest standards.

This is perhaps the most urgent and challenging issue and is well described by Ioannides. Declarations of interest are notoriously feeble in dealing with this cancer at the heart of scientific endeavor but are all we have at present. Producers of clinical guidelines are well aware of their vulnerability to the undue influence of those who may benefit from decisions reached or recommendations made in their guideline groups. Agencies like the National Institute for Health and Clinical Excellence in the UK have tough standards for declaration that are rigorously applied. Conflicts of interest are not just restricted to financial conflicts and the pharmaceutical industry. Academic and intellectual biases are common, as well as the heartfelt beliefs of charities and NGOs who in some instances might be tempted to campaign and fundraise using biased or exaggerated estimates of effect or magnitude.

Identification and managing this kind of bias needs to involve authors and editors alike. Interests that would be considered in conflict must be clearly and openly stated. A balanced and transparent editorial process can pick out biased interpretation of evidence and conclusions being drawn that are unsupported by the finding of the review.

To undertake a truly systematic review is challenging and these articles are very far from a quick fix for an easy publication. But there are some simple solutions to help the reader appraise published systematic reviews, such as the PRISMA checklist. Many journals do publish protocols and PROSPERO is there to provide a home for those that are not. Journals should only publish a completed systematic review if there is a previously published or registered protocol.

Perhaps the most important component of the PRISMA checklist for a protocol for a systematic review is the requirement to clearly justify the need for a systematic review on the topic—what other reviews have been done and what this one might add, how it will further our knowledge and if not, how will it generate a new and well specified research priority?

And finally, the use of the GRADE approach can improve the transparency of the interpretation of the findings and may be presented in a Summary of Findings table that also includes estimates of absolute and relative risks. Assessment of the certainty of the evidence, taking into account risk of bias, ensures conclusions drawn can only be based on what has actually been found in the review. This provides clarity on certainty or the lack of it, whether more research is needed or not, and the strength of the evidence for incorporation into guidelines and practice.

Table 1. Components of a systematic review.

The authors report no conflicts of interest. The authors alone are responsible for the writing and content of this article.

Richard Wormald is funded by financial support from the Department of Health through the award made by the National Institute for Health Research [NIHR] to Moorfields Eye Hospital NHS Foundation Trust and UCL Institute of Ophthalmology for a Specialist Biomedical Research Centre for Ophthalmology. Jennifer Evans is funded by a grant from NIHR which supports Cochrane Eyes and Vision

The views expressed in this publication are those of the authors and not necessarily those of the Department of Health or NIHR.