All the previous sections were written prior to, and then updated in, 2021 (with a key few numbers updated again in 2024). At this time, we were considering publishing the story and wrote our reflections about the case at that point. We’ve included an update in 2024 in the next section.
The eight-year anniversary of the first journal notification, 2021, seemed a good time to reflect on our experience.
The case can be summarised fairly simply: a highly prolific research group with >300 publications has committed scientific misconduct that has led so far to >130 retractions but affects an unknown number of unretracted papers. At least eight incomplete and opaque institutional investigations have occurred, along with countless and equally opaque investigations by individual journals and publishers. To our knowledge, there has been no co-operation in investigations between affected journals or between affected parties. While the various parties have been inefficiently and inadequately assessing and (sometimes) resolving the concerns, the research continued to be cited.
The group and its research were influential. As stated earlier (Part 2), 12 now-retracted trials reported data on hip fractures were cited by >1100 papers, including 68 systematic reviews, meta-analyses, and guidelines. The findings of 20% of these 68 papers would change if the retracted trials were removed [6]. 14 of the group’s own reviews and two by other groups have been retracted because they were dependent on the retracted papers. Related to this issue, in April 2020 we identified 6926 references in 4764 articles to any of 376 articles published by Sato or Iwamoto. Of these, 3925 references are to articles with an EOC or that have been retracted. 198 of the references occurred after the EOC/retraction (ie should not have occurred) but 3688 references cited articles by Sato or Iwamoto that subsequently have been retracted or have an EOC [13]. It is unknown how many of these citing articles depend on discredited research from Sato or Iwamoto, but even if the proportion affected is only small, the absolute numbers of papers involved could be sizeable. However, reassessing all these articles is not practical, though we have widened our investigation to evaluate the influence of all retracted clinical trials on systematic reviews and clinical guidelines (see Section 6).
From our perspective, it has been extremely laborious, time-consuming and overwhelmingly frustrating. We estimate we have spent >10,000 hours (attention JAMA!) on attempting to resolve our concerns. It is ironic to receive replies from journals and publishers bemoaning the time it has taken them to conduct their (frequently minimal) assessments of the product from which they profit. The same issues arose continually: failure to respond to emails or to address specific concerns, slow responses, inconsistent responses, and lack of transparency. Quite often, our correspondence with affected parties was met with dismissive, trite or arguably unprofessional responses (eg copying us in on internal emails complaining about our persistence, misrepresenting our infrequent periodic attempts to obtain updates as bullying, questioning our motivation and implying we were personally targeting the authors rather than trying to correct the literature, misconstruing the raising of concerns as ‘allegations’ etc), despite our attempts to be systematic and objective throughout. But most disappointing is that the integrity of the literature has not been maintained or restored: a typical reader who comes upon an uncorrected paper by this group will read and possibly cite it without knowing that major concerns exist about the work of this group. Even when an article has an expression of concern or retraction, systems are inadequate to promptly alert researchers who are potentially affected.
The inconsistencies and deficiencies occurred despite many journals and publishers professing to follow the COPE guidelines. Notably, COPE does not provide guidance on how to assess publication integrity. It often seemed that the innumerable responses that claimed that COPE guidance was being followed were a licence to take a long time to do not much, and were a convenient way of avoiding or downplaying the concerns, and/or fobbing us off. If indeed journals and publishers were following COPE guidelines, the resulting inconsistencies call into question whether those documents are fit for the purpose of protecting publication integrity. In addition, COPE has been, at best, of slight assistance to us in this case, and we wonder what value it adds for journal readers. It may be of value as an industry body to support publishers and journals, but for readers it simply provides the illusion of an independent body to consider their concerns when in reality it only considers the process(es) used in handling concerns rather than their substance.
What started out as a simple, fairly quick review of a group of trials, not too dissimilar to a standard systematic review that many researchers, including ourselves, do regularly, has snowballed into something much more complex. A lot of interesting academic things have taken place along the way (doing an interesting RCT [5], designing and publishing a checklist we think will be very helpful for anyone assessing publication integrity [9], adapting, developing and validating statistical techniques to detect unexpected distributions of data in publications [3, 14-18], developing techniques for assessing overlapping and duplicated data [19], raising concerns, and assessing and highlighting the deficiencies in institutional and journal/publisher responses [7, 8, 12, 20-25], and exploring publication and citations rates in individuals with multiple retractions [13, 26, 27], and offering suggestions to improve a broken system [11]. But there has been a lot more that has been dispiriting. If other people are thinking of pursuing a research integrity investigation, we would suggest taking it on as a team and not as an individual. The support from colleagues helps with the frustration and despondency that you will inevitably experience. Informal discussions about what you are doing and why with senior colleagues can also be very helpful.
One of the biggest frustrations is not knowing the detailed story behind the misconduct, particularly the motivations of the researchers, their relationships with their employing institutions, learned societies and perhaps pharmaceutical companies. Kai Kupferschmidt, a journalist working for Science, tried to investigate and wrote a fascinating account entitled “Tide of lies” [28]. But even with his efforts, we still do not know what of the group’s output was fabricated, what was falsified, what was real (if any), or why they did it.
We have been aware throughout of the potential impact of our actions on the authors involved. Consequently, we have attempted to ensure that the focus of our efforts and correspondence with the various parties (journals, publishers, institutions, special societies and regulatory bodies, colleagues) has been on ensuring the integrity of literature and have strenuously avoided making accusations toward the behaviour of the researchers involved. As we have written previously, publication integrity should be assessed independently from the reasons for any compromised integrity, by those with independent expertise in this area who do not have conflicts of interest [9]. The uncoordinated process must also be extremely frustrating from the perspective of the various other parties involved. Numerous investigations have already taken place, but despite this, each journal or publisher continues to attempt to contact authors and institutions about issues that have been repeatedly investigated by others but left unresolved. It is perhaps not surprising that the authors have opted to disengage, with the most recent retraction notices indicating that the authors did not respond to the journal’s emails. Institutions, despite conducting investigations that left concerns unaddressed or unresolved, have also disengaged. This disengagement was foreshadowed in a quote from Dr Iwamoto’s lawyer in Kai Kupferschmidt’s article: “From his [Dr Iwamoto’s] perspective, he is a victim. Avenell’s team … is now giving Iwamoto’s papers a level of scrutiny that is unfair and is causing his client a great deal of distress” [28]. While we do not think scrutiny of publications with integrity issues on the background of multiple existing retractions for misconduct and other issues is unwarranted, we do think that anyone whose work has had concerns about publication integrity raised should expect that the concerns are addressed independently, comprehensively and transparently within a reasonable timeframe by people with suitable expertise. Such a process clearly has not happened in this case, which is unfortunately by no means an isolated example of flawed investigations. Repeated reinvestigations, as a result of inadequate prior investigations, are in nobody’s best interests.
You cannot spend so much time and energy on trying to correct publications without reflecting on the underlying system. In our view, it is broken [11]. Papers with compromised publication integrity should not be in the literature, because they mislead readers, researchers, clinicians and ultimately lead to harm for patients. This point often seems to get overlooked when journals, publishers, and institutions do not conduct timely, transparent, and detailed investigations. Harms do result directly from the current broken system, which is why it must change [11]. It is probably not surprising but since this case, we have become involved with many other cases where concerns regarding publication integrity have been raised, about half the time by us and half by others. Sadly, our experiences in those cases are entirely consistent with that for the Sato/Iwamoto case, with no evidence of improvement over time. Which only reinforces to us the urgent need for change.
Concerns about publication integrity 