By Leo
Every time that I am teaching evidence-based practice to my students on how to conduct randomised controlled trials, I heard questions like: Why trial quality is not getting better over time?
Don’t get me (or my students) wrong here: trial quality IS improving over time, but in a very (veeeeeeeery) slow pace. Although we can observe a slow improvement over time, it is hard to understand why initiatives like the CONSORT Statement, Cochrane Reviews or PEDro database did not push this improvement further (or faster). There is a lot of data showing that this slow improvement is happening in all fields of health-related research, so there is no winner here. Incidentally, what trial quality is, and how we go about measuring it is another beast all together, and the topic for next week’s post.
My concern is that clinicians don’t have to fully understand research methods in such depth, and dodgy trials make their lives much more complicated. Maybe that is the reason that clinicians sometimes just ignore evidence! I don’t expect a perfect world on this issue, but I think that is unacceptable to see so many biased trials been published these days.
Rob Herbert’s excellent book1 gives us some insights on what would be the reasons that bad trials are still being published:
- Research is not always carried out by experts in the field, actually much of the clinical research is conducted by people with little or no training in research.
- The peer review process is not always performed by good researchers/editors.
- Some journals are desperate to publish something, in this case poorly design studies are used to fill the pages of their journals.
- Sometimes authors want to make their point rather than test a specific hypothesis.
- Finally some good researchers can have practical problems in conducting their trials (for example, to recruit a reasonable number of patients for a specific trial).
I was wondering if the ICECREAMers have more ideas on this issue, I would love to hear your opinions on it.
1. Herbert R, Jamtvedt G, Mead J, Hagen KB. Practical Evidence-Based Physiotherapy ed. London: Elsevier’s Health Sciences, 2005.
8 Comments
Good post Leo. I am going to play the devil’s advocate here – do you think we are too hard on ourselves? I think trial quality IS improving over time, and I would argue that we only have empirical evidence on whether some the risk of bias items actually influence internal validity. That is, I think for most randomised controlled trials it is important to have concealed allocation, true randomisation, blinding where possible, and not excluding patients from analysis. But I don’t think we know how other items on some of the risk of bias items, for example compliance, contamination or co-interventions, influence internal validity.
On the other hand we know that some other features, such as sample size, influence the size of treatment effect but sample size is not currently considered a quality item per se. Just some thoughts!
Hi Chris,
I think you are right mentioning that we are sometimes too hard on ourselves… however my point is with regards to the real bad trials, for example the average PEDro score for physiotherapy trials written in Portuguese is 3.2 (on a 11-point scale)…this is really bad studies that should never been published (IMO)!
In my view, one of the best ways to improve this is by changing editorial policies and educating authors… things are improving, but we still need better studies to inform clinicians about their interventions.
You point regarding sample size is great and I totally agree with it… Nick Henschke is working on a delphi study on “fatal flaws in RCTs” which may address these issues further… hopefully he will stop taking holidays for a while and will finish the study soon
(I heard that he is on holidays again!)
Cheers,
Leo
Dear all,
Interesting discussion! You can look at this from all kind of perspectives, lets take the one of pt’s in my surrounding!
Recently I asked PT’s in a discussion on linkedin about who’s responsibility it is to keep pt’s up to date. Do pt’s need to search, interpret ate, etc. them selves or is it the task of researchers to provide information in for example best evidence statements?
I completely understand the importance of good quality trials. On the website everybody agreed pt’s should take also care of their knowledge by reading articles. Some even said: You can read English, so just type your term and read the article/abstract. But as we know reading only a conclusion (or only the abstract) can cause generalization and misinterpretation of the actual results found. By reading the article, without further knowledge of the consequences of a bad method and bias, it looks easy to set up a trail.
Trials are set up because of the questions in the field and their/our ‘inner will’ to seek an answer. If for outsiders trails are so easy to do, why spend more time, energy, money on extra researchers?
Maybe if we, as researchers, could explain in a simple way why methodology is so important for pt’s them selves. Methodology might be more considered when preparing a trial.
… and if we are already working on it any why, let’s also explain the importance of a fitting statistic method..
(hope my statement was clear in English..)
Hi Marlous,
Thanks for your comment. You make an interesting point: to educate clinicians about the beauty and the importance of critically appraise a study. That would be the best way to disseminate research findings better.
Here in Brazil, I have been teaching Evidence-Based practice to undergraduates as well as to clinicians (in continuous education programs) with good results. These clinicians have been sending feedback to my classes commenting on RCTs and systematic reviews that they decided to read after these classes… which is pretty cool in my opinion!
If we don’t do this, clinicians will keep ignoring research findings and researchers will keep not believing in clinicians!
Another point is that researchers should be more open to clinicians opinions, we have a lot to learn from them!
Cheers,
Leo
Hey Leo… i don’t think that clinicians should waste their time reading trials, even the most elegantly designed trial only provides a small part of an answer. I reckon that if clinicians want to read anything research related to inform their practice then it should be a good systematic review, or as Marious suggests some best evidence guidance written by researchers. With the number of SR around these days there must be an answer on most things – even if the answer is ‘we don’t know, the evidence isn’t good enough’. Then you can do what you want!
I really worry that clinicians somehow feel pressured to change or adapt their practice on the basis of the results of single RCTs, or even a couple of RCTs and that is never a good idea – not in the work that we do anyway. As we all know there was a HUGE revolution in physio practice on the basis of one or two early small studies on Motor-Control Exercises (MCE) for low back pain. It wasn’t until much later that the SRs started to appear that suggested that the effects were much smaller than were originally thought and perhaps MCE for many people was not much more than a waste of time.
Exactly the same thing is starting to happen in applying ‘neuroscience’ methods for treating pain patients. A couple of very impressive early trials, followed up by less impressive results. Lets see what the SRs tell us, though I know where my money is.
James
Hey James!
Although I agree with you… my concern that most of these dodgy trials are the ones that populate our “high quality SR”… so a SR full of dodgy trials can also be considered by some people as a bad source of information… anyway this is a great debate and I am sure that all researchers aim the same: to do their best!
With regards to MCE…hummmmmmm well… I think we have already talked enough about this, don’t we?
Cheers,
Leo
Whether or not clincians should read RCTs to inform their practice (personally I’m not convinced clinicians ‘shouldn’t waste their time reading trials’) people in practice still need to be able to interpret evidence from RCTs. It is not possible to appropriately interpret research evidence without undestanding something about research methods. I think that if there is sufficient understanding of same then people are less likely to be misled about the clinical implications of published papers, whether they be case reports, RCTs, SRs or articles in Hello magazine.
That said, all the education in the world won’t stop some humans trying to justify their actions by misinterpreting any type of ‘evidence’, although I would stress to say that I don’t think this applies to most clinicians.
Steve
[...] posted recently on the state of research quality (here). While there is a bit of overlap between the issues, in this post (and the next) I want to explore [...]