How to evaluate manual therapy: value and pitfalls of randomized clinical trials

Article Outline

• Copyright

There are many possibilities to investigate the effects of manual therapy varying from case studies, case series, non- or quasi-experimental studies with control groups and finally the true experimental designs (in medical research approached by the design of a randomized clinical trial (RCT)). All designs are frequently used and all have their relative strengths and weaknesses. I argue that the RCT is the design of first choice in manual therapy research when it comes to answering questions like ‘is the new (or old) intervention more effective than no therapy at all, a placebo treatment or another existing therapy?’

The RCT is characterized by the use of one or more control groups and random allocation of the patients. Often blinding of therapists, patients and outcomes assessment is included. Due to its potential to control for different forms of bias, the RCT is by many people considered to be the ‘gold standard’ for intervention research.

There already have been carried out and published quite a number of RCTs within the field of manual therapy. It is no secret however, that many trials carried out in the past showed important methodological shortcomings, thereby threatening the internal validity of the study and making it difficult to interpret the study results. For many manual therapeutic interventions it holds that currently there is insufficient evidence to draw conclusions regarding their effects. Clearly, there seems to be a need for more and better RCTs in this area.

There exist, however, a number of potential drawbacks of RCTs. They usually are time consuming. Three to four years seem to be quite normal periods for such trials, making them quite costly. The mean costs for a (medium sized; n=100–200) randomized clinical trial will be about US$500.000. Other comments on RCTs include that randomizing patients to, for example, a placebo treatment group or a no-treatment group is unethical. However, only after the conduct of RCTs do we have valid information that the intervention under study is doing more good than harm. So, we may ask ourselves whether it is ethical to treat patients with therapies with unknown effects. Another criticism is that in previously conducted RCTs often isolated parts of a manual therapeutic intervention have been investigated which may not reflect the situation in daily clinical practice. This may indeed be a problem. However, this is not a characteristic of an RCT in which it certainly is possible to investigate a ‘full’ manual therapeutic treatment for example in comparison with a drug treatment. Of course before starting a RCT one must always consider whether the research question is clinically relevant. Finally, the conduct of a randomized clinical trial is not an easy job. Frequently encountered methodological problems of randomized clinical trials in the field of manual therapy concern the following topics.

1. Prognostic homogeneous study populations

In randomized clinical trials one usually attempts to include a prognostic homogeneous study group, which will likely respond to the experimental intervention(s) under study. However, in the available randomized clinical trials it appears that often rather heterogeneous study groups were included. This may hamper finding a treatment effect if, for instance, an intervention is effective only for a subset of the population. In this case the positive effect in this subgroup will be diluted due to the absence of effect in the complementary subgroups.

2. Standardization of interventions

On occasions one may read that the experimental or control treatments in a study consisted of manual therapy/physiotherapy at the discretion of the participating therapist. It is obvious that, with a description like this, readers of are not well informed of the kind of therapy that was investigated in the study at issue. For generalizibility of study findings it is essential that the type, the intensity, the frequency and duration of the treatment is sufficiently described in order to make it possible to replicate the therapy elsewhere. It is not always necessary and/or feasible to develop a strict treatment protocol. In such cases, it is certainly permissable to work with some kind of treatment algorithm, in which the steps in the treatment path depend on the outcome of a previous step. In any case in the absence of a clear treatment protocol or algorithm, a clear description of the actual treatment applied in the study should be recorded and presented.

3. Blinding of patients, and therapists and outcome measurement

Blinding at several levels is one of the important features in a randomized clinical trial. In trials in manual therapy blinding is often problematic.

3a. Blinding of patients is hampered by the fact that, from the content of the manual therapy, the patient in most cases will know which treatment he/she receives. Blinding patients by including a placebo treatment may not always be possible because a good and trustworthy placebo cannot be developed. For example, placebo exercise therapy, or placebo spinal manipulation appear to be difficult to develop. Since knowledge by the patient may influence the outcome of a trial some measures may need to be taken. During the selection process of patients, one may ask the potential participant about their treatment preferences, and may decide only to include patients with no strong preferences for or against the treatments included in the study. In the same way, patients with extensive previous experiences with one of the investigated treatments may also be excluded.

3b. Blinding of therapists will not be possible in most trials evaluating manual therapy. Due to the nature of the manual treatment the participating therapist will have actual knowledge of the treatment that he/she applies. Also in this case some attention to treatment preferences of therapist may be needed. Ideally, the therapists of the various study treatments should be equally positive, etc., regarding the delivery of the treatment. Especially, in studies in which a placebo therapy has to be convincingly delivered this may cause problems for the therapists. This delivery may have to be practised extensively before the start of the trial.

3c. Blinding of outcome measurement has also been problematic in previous randomized clinical trials. Blinding of patients and therapist is often not satisfactorily feasible as stated above. In addition, the outcome assessment often includes a subjective rating of pain and functioning. In these cases, the only available method to include at least partial blinded outcome measurement is to use a blinded independent observer. This observer should assess the patient without knowledge of the assigned therapy. In randomized clinical trials published in the last few years this method seems to have become more common.

4. Small sample sizes

Although sample size deals more with the precision of the estimation of effect and not necessarily with the validity of the study, it remains an important aspect of a trial. In manual therapy, the randomized clinical trials usually include small sample sizes (i.e. less than 50 patients per study group).

Because of this there might be a problem in detecting a (statistically significant) small but true treatment effect. This is sometimes called a type II-error and is caused by the low (statistical) power of small studies to detect small but clinically relevant treatment effects.

Another problem with small sample sizes is that the comparability of the study groups at baseline may be in danger. Only with increasing numbers of randomized patients do we have some assurance that known, but also unknown, prognostic factors will be evenly distributed over the study groups.

5. Drop outs/loss to follow-up

The last methodological flaw relates to the description of dropouts of a study. If patient dropout occurs it is essential to know and report the reason for this dropping out. Patients may drop out because they are completely recovered or because they feel worse than ever. Knowledge of this is especially needed if the dropout rate is selective (i.e. occurs mainly in one of the study groups). Loss to follow-up may also be substantial in trials of physiotherapy. Loss to follow-up relates to the number of patients participating in the outcome assessment. It is obvious that if there are large numbers of loss to follow-up (>20%) that the outcome of the study can be much influenced. Again, this is even more problematic if the loss to follow-up is selective. It is possible, however, to deal with selective follow-up in the analysis phase of a study. Additional analysis, for example a ‘worst case analysis’ could be carried out.

In conclusion, randomized clinical trials clearly have their strengths and weaknesses. They appear to be a powerful research tool for answering questions on the efficacy of interventions. Despite some problems with the conduct of randomized clinical trials, which mainly relate to problems with blinding of patients, therapists (and consequently the outcome assessment), it is certainly possible to carry out high-quality studies in this area.

It will take, however, an open mind and courage of the manual therapeutic professions to go (further) into the direction of evidence-based-manual therapy and to conduct RCTs. Not all interventions, which currently are applied, will be shown to be effective. In this case the profession itself must be prepared to stop applying those treatments and focus on the interventions, which do turn out to be effective.

Of course, other types of research aimed at increasing ‘the body of knowledge’ of manual therapy also should be carried out. Basic sciences, including animal studies and biomechanical work are needed to develop new therapies, improve old therapies, etc. At the end of the day however, only by conducting high-quality RCTs will we be able to collect evidence whether the new therapies are (cost-) effective or not.