Cochrane Reviews of Psychiatric Drugs Are Untrustworthy

The Cochrane Collaboration was established in 1993 as an idealistic grassroots organisation with the aim of publishing updated systematic reviews of the benefits and harms of interventions in healthcare. It started in Oxford in the UK, spread rapidly throughout the world, and became very successful. Unfortunately, like most successful grassroots organisations, it came to suffer from institutional corruption early on. This moral decline accelerated when a new CEO, journalist Mark Wilson, was appointed in 2012. I co-founded Cochrane in 1993 and have described its moral and scientific downfall in two books. This sad story will also be told in a documentary and in my upcoming podcast series, Broken Medical Science. Cochrane reviews are published in the Cochrane Library, once at the forefront of science, but now an anachronism, as is Cochrane’s marketing logo, “Trusted evidence.” This is particularly true with Cochrane reviews of psychiatric treatments; they can’t be trusted. Today, Cochrane is keen to protect guild interests and to comply with official views from governments even when they contradict the evidence in Cochrane reviews, as illustrated by its recent review of face masks. Cochrane is too close to industry, and Cochrane does not care much about financial conflicts of interest for its review authors. When I was elected to the Cochrane Governing Board in 2017, I called for a strengthening of our conflicts of interest policy. This was agreed to, and I rewrote the policy in an afternoon. But the Cochrane leadership ignored it and spent over two years consulting with Cochrane members before its “strengthened” policy came out. Today, 33% of new review authors can be on company payroll. Before, it was 50%. Semmelweis would not have liked this “revolutionary” new policy. He never told doctors to reduce contamination by washing one hand. He told them to wash both hands. The game for Cochrane seems to be over. Cochrane’s CEO Mark Wilson showed no interest at all when, a decade ago, Tom Jefferson demonstrated that Cochrane reviews of drugs against influenza based on published trial reports are totally misleading. I demonstrated that this was also the case for psychiatric drugs. Wilson suddenly left Cochrane in the middle of a month, in April 2021, with no explanation, seven days before the UK funder announced that a major budget cut was likely. The current leadership shows no interest either in this vitally important issue of whether its reviews are misleading. In April 2021, Professor Ken Stein, Director of the Evidence Synthesis Programme at the UK National Institute for Health and Care Research (NIHCR), the major funder of Cochrane, criticised Cochrane at a webinar for much the same reasons as I had when I joined the Cochrane Governing Board. He said the writing had been on the wall for eight years, which was exactly the period when Mark Wilson ruled the organisation and destroyed it. About the failing scientific integrity, Stein noted that, “This is a point raised by people in the Collaboration to ensure that garbage does not go into the reviews; otherwise, your reviews will be garbage.” And as Tom Jefferson said in an interview in 2018, such reviews are published with a nice little Cochrane logo on the garbage. There were, at the start of 2023, 52 Cochrane review groups in the world, together covering all aspects of healthcare. However, in March 2023, all 24 with a base in the UK lost their government funding. Apart from four groups that have other funding sources, they have all now closed down. In July 2023, the NIHCR announced that it will end its contract with the UK Cochrane Centre in March 2024, a year before it was due to expire. The reason? Inefficiency and lack of relevance of the Cochrane reviews. Cochrane has become a self-serving institution, a hugely ineffective bureaucracy, which is succumbing under its own weight, as illustrated by the size of Cochrane reviews. The median size was 26 pages in 2001, 41 in 2012 and 79 in 2023. A few  are so big that they correspond to several books. If you have a collection of, say, 30 drug trials that are all deeply flawed, which is typically the case in psychiatry, you are not allowed to say this in a short review. Instead, you are required to write a very long review with details about every single trial even though it is impossible to conclude anything based on them. Yet, even given this meticulous attention to detail, most Cochrane authors and editors pay far too little attention to all the flaws in the trials and draw conclusions about drug benefits and a lack of harms that are unjustified. Cochrane reviews of psychiatric drugs should generally be ignored. Virtually all of them are based on untrustworthy published trial reports, rather than on the lengthy clinical study reports the manufacturers have submitted to drug agencies to get approval for their drugs. In particular, reviews of placebo-controlled trials are a garbage in, garbage out exercise.

Cochrane editorial misconduct in relation to our review of safe withdrawal of depression drugs

In 2016, I contacted psychiatrist Rachel Churchill, editor of the Cochrane Common Mental Disorders group, who showed great interest in my proposal to do a review on safe withdrawal of depression drugs. The story of how Cochrane treated this proposal provides a case study of how Cochrane protects guild interests. This was a very important review. Hundreds of millions of patients have become dependent on depression drugs, just like patients become dependent on benzodiazepines. The withdrawal symptoms are very similar; about half have difficulty stopping; and doctors don’t know how to taper the drugs safely and therefore usually just renew the prescriptions, for many years on end. But we quickly faced a roadblock. The Cochrane group was extremely slow to respond and imposed ever increasing demands to our protocol. At the same time, they negotiated secretly with another author group on the same issue. This violated two of the ten Cochrane key principles: Collaboration, which involves open and transparent communication and decision-making, and avoiding duplication of effort, which is about avoiding that two reviews cover more or less the same ground. The Cochrane group rejected our protocol two years and four months after we first submitted it, while it accepted the other authors’ protocol and published a very embarrassing Cochrane review, which is full of misleading statements and marketing messages of great value for the drug industry but irrelevant for doctors and patients (see below). On 28 March 2023, I sent a complaint to Karla Soares-Weiser, Cochrane’s Editor-in-Chief, about editorial misconduct in an open letter. Later, I also sent a complaint to Cochrane’s current CEO, Catherine Spencer. I asked some simple questions, which they refused to answer. I have described the bizarre interactions I had with the Cochrane leadership elsewhere. Briefly, they beat about the bush, just like the drug industry does when you have a really good case against them. I couldn’t see the difference. As Jefferson once said in relation to Roche’s unwillingness to provide the data from its unpublished trials of Tamiflu for influenza, If you have nothing to hide, then hide nothing. Cochrane hid everything in relation to our review. It also turned out that Cochrane has no mechanism for handling allegations of editorial misconduct in an impartial manner, something all reputable journals have. My translation of the message I got from Cochrane’s CEO was this: “We don’t give a damn. We are beyond reproach.”

Cochrane abused the peer review process to the extreme

Our protocol for reviewing drug withdrawal studies was rejected in November 2019. However, in February 2020, a similar protocol about withdrawing depression drugs was published in the Cochrane Library. It takes a very long time to get a Cochrane protocol approved and published, which means that this project must have been underway secretly for many months while the review group increased their demands regarding our protocol, which I see as an effort to wear us out without their being seen as uncooperative. Cochrane abused the peer review process to the extreme. Four editors and three peer reviewers provided individual comments, and, including our replies to earlier comments, the peer review document took up 12,044 words, seven times the number of words in our original protocol. My co-author wrote to me that our review was quite simple and that we just wanted to help people who wished to come off their drugs, but we weren’t allowed to do so. He asked: “What kind of world is this?” The 8^th peer review was as long as our protocol. It was clear to us that the final reviewer’s mission was to protect psychiatry’s guild interests, providing an excuse for the editor to reject our protocol. This was done by denying a long array of scientific facts; by using strawman arguments accusing us of things we had never claimed; and by requesting us to discuss issues that were totally irrelevant for our review and to insert text that was blatantly wrong. In contrast to the 7 previous reviewers, this one was anonymous. We asked for the identity of the “hangman,” but this was not granted. I noted that very few changes to the protocol were needed and submitted a new version and a rebuttal with solid scientific arguments against the many errors and misguided opinions in the peer review, but this didn’t matter the slightest bit for Cochrane. The 8^th peer review is one of the worst I have ever seen. I published a detailed account of it four years ago and shall only summarise the main issues here. The reviewer accused us of “painting a picture” about antidepressants being “bad medications” to be avoided, which did not represent the scientific consensus, and wanted us to “Start with a statement as to why antidepressants are considered by the scientific community to be beneficial … in treating a broad range of highly disabling and debilitating mental health problems.” The reviewer found it “unscientific, and unacceptable in the context of the current evidence base” that we had not mentioned the beneficial effects. We responded that our review was not an advertisement for the drugs and that it was not relevant to discuss their effect in a review about stopping using them. The reviewer believed that a chemical imbalance in the brain was the cause of depression and wanted us to write about this. We responded that our review was not the place for such discussions and that the hypothesis of a lack of serotonin being the cause of depression had been discredited by many convincing studies. We were asked to explain the concept of ongoing prophylactic antidepressant treatment, “a well-accepted clinical strategy,” but this was outside the scope of our review. Furthermore, the randomised trials comparing maintenance therapy with drug withdrawal are seriously flawed by their cold turkey design. Some of the withdrawal symptoms in the placebo group mimic depression. The reviewer wrote that we conflated relapse with withdrawal symptoms, which wasn’t true. But many psychiatrists do, which is a major reason why many patients are treated for decades or for life. The reviewer argued that most people who had taken antidepressants for extended periods could stop safely without either rebound of the disease or withdrawal symptoms. This is untrue, which we had documented and referenced in our protocol. The UK Royal College of Psychiatrists reported in 2012 that 63% of 817 people who had stopped taking depression pills experienced withdrawal symptoms. The reviewer wanted us to remove this sentence: “the patients’ condition is best described as drug dependence” arguing, with reference to the DSM-IV drug dependence criteria, that it is an unreasonable misappropriation of a term. We responded: “The official definitions of dependence are ridiculous and self-serving, in addition to serving the drug companies that have benefitted hugely from the false perception that only benzodiazepines cause dependence, not the SSRIs. Craving larger and larger doses as a criterion for dependence is absurd, as it means that no one who smokes 20 cigarettes every day is dependent on smoking cigarettes!”

The published Cochrane review of withdrawal was of poor quality and filled with industry-like marketing messages

While we were being blocked from conducting a drug withdrawal review for Cochrane, the other group that had submitted a protocol for doing so was given the green light to proceed, and in 2021 Cochrane published it. The review was restricted to adults with depression or anxiety, which is irrational. The drugs are used for many conditions, and the withdrawal symptoms are not dependent on why the drugs are prescribed or the age of the patient. Moreover, the review did not include trials comparing different withdrawal strategies, which we did, whereas it included many flawed studies comparing abrupt discontinuation (cold turkey design) with continuation. We only included trials that had at least one treatment arm that aimed to help patients withdraw from a depression drug. The published Cochrane review included 33 studies (4995 participants). In our review, which we first published as a preprint (likely to come out in a medical journal soon), we included 13 studies (2085 participants). The Cochrane review is 209 pages (110,770 words), the length of a full book, 23 times as long as our review of 9 pages. I studied the Background section in the Cochrane review, which, with its 4239 words, is longer than most scientific papers. It is full of irrelevant marketing messages and misleading statements, which I noted in my complaint to Cochrane about editorial misconduct. It states that “Maintenance treatment is provided to prevent recurrence.” There is no mention that all maintenance trials are deeply flawed and that the patients’ withdrawal symptoms are erroneously interpreted as a relapse of the depression. “Antidepressants have been shown to be efficacious in adults … (Cipriani 2018) … between seven and eight people needed to be treated with an SSRI … for one person to experience improvement.” These praises of drug benefits are invalid. The effect on the Hamilton scale, a difference of 2 over placebo, also in Cipriani 2018 (an effect size of 0.30 corresponds to a Hamilton difference of 2.3), is far below what is clinically relevant, as the least difference on the Hamilton scale that can be detected is about 5-6. I have shown why the number needed to treat (NNT) with a psychiatric drug to benefit one patient is largely an illusion. One reason is that more patients are harmed than those who benefit. When patients decide whether it is worthwhile to continue in a trial, they make a judgement on whether the benefits they perceive exceed the harms. Based on clinical study reports we obtained from drug regulators, we found that 12% more patients drop out on a depression pill than on placebo (P < 0.00001). Thus, the patients consider placebo more useful than a depression drug, which means there cannot be an NNT, only a number needed to harm (NNH). Our meta-analysis showed that this number is about 25. The published trial reports do not reveal this devastating finding. When the top among UK psychiatrists in 2014 tried to convince their readers that depression pills are highly effective, they mentioned an impressive effect on recurrence, with an NNT of around three. But these trials did not assess recurrence but withdrawal symptoms in the placebo group. As only two patients are needed to get one with withdrawal symptoms when a drug is stopped, there cannot exist an NNT to prevent recurrence, only an NNH, which is two. Since depression pills harm the sex life in about half the patients, the NNH is two. Thus, by not using depression pills, we will preserve the normal sex life in one out of every two patients we do not treat. The published Cochrane review is unbalanced. It gives precise but misleading estimates of the benefit in the form of NNT but does not offer similar estimates for the most serious harms. This goes against the very ethos of Cochrane, which always was to focus similarly on the benefits and harms of interventions. The Cochrane review mentions the worst harm, suicidality and suicide, in many places, but it does not say that depression drugs double the suicide risk, both in children and adults. The Cochrane review states that “evidence suggests that continuation of antidepressant treatment is effective, as it reduces risk of relapse and recurrence by 50% to 70%.” The authors should not have propagated this horrible misinformation, but rather should have stated that there is no reliable evidence that the drugs reduce recurrence. The review continues: “The effect of most antidepressants fully develops after some weeks, indicating that neurophysiological changes in brain tissue … are necessary for improvement in depressive symptoms.” This neuro mumbo jumbo is also highly misleading. One cannot say that the effect fully develops after some weeks when there is a very gradual and slow separation of the Hamilton scores on drug and on placebo and when the effect even after seven weeks is so small that it is not clinically relevant: The authors note that “suggesting that a single biochemical deficiency is the cause of depression and that antidepressants work by correcting chemical deficiency is not correct.” Interesting that they were allowed to say this. We were heavily criticised by the hangman for having written that the hypothesis about a chemical imbalance in the brain being the cause of depression had been discredited. One of the main aims in establishing the Cochrane Collaboration in 1993 was to assist patients in their decision making. However, the whole Background section is about what doctors think and the review is highly paternalistic. There is no mention that many patients want to come off the drugs, which should have been the key motivation for the authors to do their review. There is no mention in the Background section that the tapering should be hyperbolic, which is essential if one wishes to minimise withdrawal symptoms, whereas the authors quote a NICE guideline from 2009 without criticising it for recommending a fast, non-hyperbolic tapering that is outright dangerous, as it can cause akathisia, suicide and violence. When we started Cochrane in 1993, we were willing to criticise the authorities. The current leadership wants to please the authorities and the drug industry, which this Cochrane review demonstrates. The abstract of the Cochrane review is 915 words, four times as long as ours of 236 words. But length is not a substitute for quality. The Cochrane abstract states that “We cannot make any firm conclusions about effects and safety of the approaches studied to date” and that “Future studies should report key outcomes such as successful discontinuation rate.” Our little abstract is far more informative. We included fewer but more relevant studies and found a median successful discontinuation rate of 50%. More importantly, we noted: “A meta-regression showed that the length of taper was highly predictive for the risk of relapse (P = 0.00001). All the studies we reviewed confounded withdrawal symptoms with relapse; did not use hyperbolic tapering; withdrew the depression drug too fast in a linear fashion; and stopped it entirely when receptor occupancy was still high.” We concluded that “The true proportion of patients on depression drugs who can stop safely without relapse is likely considerably higher than the 50% we found.” The lengthy Cochrane abstract did not mention any of these essential issues.

Cochrane is in deep trouble

Currently, faced with an ongoing crisis caused by multiple missteps, Cochrane’s Editor-in-Chief, Karla Soares-Weiser, has hired the pricey consulting firm Envoy to address scientists’ concerns about her lack of transparency, leadership, and communication skills. The crisis came to a head in March, when Soares-Weiser rushed out an apology even though there was nothing to apologize for, which undermined Jefferson’s and colleagues’ Cochrane review that showed that masks did not reduce respiratory viral infection. Soares-Weiser even claimed, totally falsely, that “the review is not able to address the question of whether mask-wearing itself reduces people’s risk of contracting or spreading respiratory viruses.” This was one of the aims of the review! When investigative journalist Paul D Thacker asked for insight into how Cochrane had handled his request for comment and answers in relation to this scandal, he received heavy redacted documents: So much for Cochrane transparency, which appears in the first of Cochrane’s 10 key principles: “Collaboration by fostering global co-operation, teamwork, and open and transparent communication and decision-making.” What is the difference to the drug industry? Thacker called his article: “Cochrane: world’s preeminent medical information resource goes into tailspin.” It should come as no surprise that many people who were previously very supportive of Cochrane now see it as a slowly dying organisation.