- Open Access
Real world research: a complementary method to establish the effectiveness of acupuncture
BMC Complementary and Alternative Medicine volume 15, Article number: 153 (2015)
Acupuncture has been widely used in the management of a variety of diseases for thousands of years, and many relevant randomized controlled trials have been published. In recent years, many randomized controlled trials have provided controversial or less-than-convincing evidence that supports the efficacy of acupuncture. The clinical effectiveness of acupuncture in Western countries remains controversial.
Acupuncture is a complex intervention involving needling components, specific non-needling components, and generic components. Common problems that have contributed to the equivocal findings in acupuncture randomized controlled trials were imperfections regarding acupuncture treatment and inappropriate placebo/sham controls. In addition, some inherent limitations were also present in the design and implementation of current acupuncture randomized controlled trials such as weak external validity. The current designs of randomized controlled trials of acupuncture need to be further developed. In contrast to examining efficacy and adverse reaction in a “sterilized” environment in a narrowly defined population, real world research assesses the effectiveness and safety of an intervention in a much wider population in real world practice. For this reason, real world research might be a feasible and meaningful method for acupuncture assessment.
Randomized controlled trials are important in verifying the efficacy of acupuncture treatment, but the authors believe that real world research, if designed and conducted appropriately, can complement randomized controlled trials to establish the effectiveness of acupuncture. Furthermore, the integrative model that can incorporate randomized controlled trial and real world research which can complement each other and potentially provide more objective and persuasive evidence.
Acupuncture has been used in China over 2500 years, and is increasingly practiced in Western countries, for a variety of medical conditions, including chronic pain, hypertension, nausea and vomiting [1, 2]. The efficacy and safety profile of acupuncture has been increasingly investigated with randomized controlled trials (RCTs) over the past decades [3, 4]. Clinical research of acupuncture, and particularly the RCT, has been increasingly conducted presumably due to ongoing development of evidence based medicine. Several RCTs and relevant systematic reviews provided reasonably strong evidence that supports the efficacy of acupuncture on the treatment of some medical conditions, including pain , hypertension , seasonal allergic rhinitis , radiotherapy and chemotherapy induced side effects , etc. Other studies, including large-scale multi-center RCTs, however do not support the efficacy of acupuncture in treating pain [9–12], hypertension , and radiotherapy-induced emesis , relative to placebo or sham acupuncture. A systematic review of 13 RCTs enrolling 3025 patients evaluating acupuncture as a treatment for pain yielded inconclusive conclusion because the evidence was insufficient . In general, the efficacy of acupuncture remains controversial. Table 1 is a list of examples of RCTs that did not support the efficacy of acupuncture over placebo or sham controls.
The authors agree that evaluating acupuncture as a viable treatment modality using modern technologies, as typified by RCTs, is a major advance but believe that the current RCTs of acupuncture need to be further developed. Recently, real world research (RWR) has been conducted to estimate the effectiveness and safety of interventions in real world settings. In the current paper, we discussed the features unique to acupuncture, the existing problems that plagued clinical evaluation of acupuncture, and the feasibility of RWR for therapeutic evaluation on acupuncture. We also give some suggestions for study design on acupuncture assessment and call for high-quality RWR as a complementary method to establish the effectiveness of acupuncture.
Features unique to acupuncture
Acupuncture treatment includes three aspects: needling, specific non-needling components drove by acupuncture theory, and generic components not unique to acupuncture treatment . In addition, acupuncture treatment should be performed on the basis of the patient condition and traditional Chinese medicine (TCM) theory (Fig. 1).
Pattern identification refers to syndrome classification of a variety of unbalance conditions based on TCM theories. Individuals with the same disease can have distinct syndrome classification that can dictate different treatment strategies . In this case, acupoint selection and regimen of treatments will be tailored to the individual patient. This situation is analogous to individualized treatment of the same condition (e.g., essential hypertension) based on the varying pathological bases in Western medicine. Generally, pattern identification and acupoint selection vary considerably across different physicians due to different skill and experience.
Within this context, instead of perceiving acupuncture as an act of merely needling, acupuncture treatment is a highly complex intervention that mainly involves pattern identification, acupoint selection, style of acupuncture, regimen of treatments, patient-practitioner interaction, and patient belief/expectation [18–20]. Moreover, acupuncture is an individualized treatment rather than a standardized needle manipulation. Clinical trials of acupuncture, therefore, should clearly define/clarify the treatment (acupuncture) characteristics (Fig. 1) to achieve objective and persuasive results.
Benefits and limitations of RCTs on acupuncture
In evidence-based medicine, high quality RCTs and systematic reviews provide the best evidence for health care intervention . In RCTs, the investigators control the factor(s) of primary interest (typically the therapeutic intervention), and therefore, could theoretically reach a reasonable conclusion with regard to cause-effect relationship. Indeed, the efficacy of acupuncture has been established with reasonable confidence in certain trials [5, 6]. However, the internal validity of RCTs is largely dependent on appropriate randomization, allocation concealment, blinding, and strict adherence to eligibility criteria. Without rigorous design and performance, results from RCTs can be unreliable. What’s more, due to the essential characteristics of acupuncture (complex intervention and individualized treatment), designing appropriate acupuncture treatments in RCTs is challenging, let alone designing and implementing appropriate placebo/sham controls for acupuncture. The following is a list of major problems in these aspects, based on the opinions of the authors as well as other scholars .
In many RCTs, acupuncture was not considered as a complex intervention that involves different components. Generally, only one or two of the whole components (typically not including pattern identification) were considered, and often in isolation . Due to this imperfection, the effectiveness of verum acupuncture may be less than optimal.
Acupuncture in RCTs often differ considerably from that used in real world practice. A prominent problem here is lack of adjustment of the intervention based on assessment of varying pattern identification over the trial duration. This is important considering the fact that the underlying disease is a dynamic process which could be influenced by the intervention. To achieve maximum therapeutic effects, physicians often need to modify acupuncture during the trial.
Placebo/sham controls in acupuncture RCTs are often inappropriate. Placebo/sham controls should have similar appearance and administration, but lack the essential components of acupuncture, as stated before. However, many placebo/sham controls in acupuncture RCTs have similar characteristics such as sensory stimulation, and patient-practitioner interaction to verum acupuncture . The absence of difference between acupuncture and placebos/shams could lead to underestimation of the efficacy of acupuncture. In addition, a RCT published in British Medical Journal reported higher efficacy of sham acupuncture than placebo pill on pain outcomes . With this context, a paradox might emerge that verum acupuncture was superior to no treatment, but didn’t significantly outperform placebo/sham acupuncture.
Randomization is often difficult and sometimes not practical. Patients with a strong personal belief and preference for a specific treatment tend to refuse randomization, and are prone to dropout even after randomization.
Blinding is also difficult, for both investigators and patients. Experienced physicians may be able to ascertain group assignment based on their experience. Similarly, patients with prior experience to acupuncture treatment may also be able to indentify whether they receive verum acupuncture or a sham procedure.
Even with good internal validity, external validity (applicability to the general patient population) is often highly problematic due to unique sample characteristics and “sterilized” settings of the RCTs (relative to the real world settings) .
In the six RCTs listed in Table 1, the major limitations that contributed to the equivocal results supporting acupuncture actually involve imperfections regarding acupuncture treatment and inappropriate placebo/sham controls.
Feasibility of RWR for therapeutic evaluation on acupuncture
RWR, also called observational research, is the recent name given to assess the effectiveness and safety of interventions and post-approval of pharmaceutical drugs in a real world setting . In contrast to examining efficacy and adverse events in a “sterilized” environment in a narrowly defined population, RWR assesses the effectiveness and safety of a particular intervention in a much wider population in real world practice. In RWR, patients are treated with interventions in accordance with their actual conditions and preferences, rather than a random assignment. The intervention is decided by, and fully transparent to both the physicians and the patients. With regard to the study aim, RWR typically do not attempt to reach simplified/rigid treatment protocols. Another important difference (relative to RCT) is the emphasis on the patients rather than the disease; as a result, study outcomes tend to be patient-oriented or even patient-reported.
In 2002, “patient registries”, a gold standard for RWR was proposed . A number of registries with large number of patients and data points, some involving international collaboration, have been developed [29–32]. For instance, the global registry of acute coronary events (GRACE) is an RWR that included 102,341 patients receiving medical care during a period between 1999 and 2009 across 154 hospitals in the world . The GRACE study covered patients with acute coronary syndromes but varying co-morbidity. On the basis of clinical practice, with broad inclusion criteria, large sample size, and long term follow-up, registry studies have achieved considerable results. For example, the GRACE study in 2008 found that following presentation with a high-risk acute coronary syndrome, advanced age alone was not a contraindication to invasive management .
Reputable RWR on acupuncture is scarce. Due to the limitation of RCTs on acupuncture assessment, as illustrated before, we believe that RWR is much more feasible and meaningful:
In RWR, interventions are tailored to the patients’ specific conditions, in contrast to standardized treatment. As a result, conclusions based on RWR consider all aspects of acupuncture that affect the effectiveness.
At an operational level, patients’ choice of the treatment(s) decreases the difficulties in recruiting and retaining patients during the data collection period.
The study sample in RWR is much more representative of the real world situation (similar to the section of the population that receives the treatment). The study, therefore, has higher external validity.
RWR tends to have a larger sample size and longer follow-up period than RCT, and thus is more appropriate for assessing the safety of acupuncture.
By no means do we argue against RCTs for acupuncture, but we strongly believe that RWR has an important role in acupuncture research.
General limitations of RWR also apply for acupuncture RWR. First, it is not possible to firmly establish a cause-effect relationship with RWR since the intervention is not under the control of the researchers. Second, any biases and influences by confounding factors tend to be larger in RWR due to lack of control and unbalanced design. Third, effective management of a large data set is also a practical challenge.
Suggestions for study design on acupuncture assessment
Usually upon an unexpected or unwelcome research finding, the first “culprit” tends to be poor research design and/or protocol implementation . This may be particularly true for acupuncture RCTs due to the complex nature of acupuncture treatment as described above. Some efforts, e.g., the revised STandards for Reporting Interventions in Clinical Trials of Acupuncture (STRICTA) , have been devoted to improving reporting quality of acupuncture clinical trials. In addition, a novel statistical method (individual patient data meta-analyses) has been conducted to reanalyze data from RCTs of acupuncture and got expected and welcome findings . The following is a list of additional suggestions based on the opinions of the authors as well as other scholars:
The study aim (whether to assess the efficacy of acupuncture needling or the effectiveness of acupuncture treatment) should be clearly defined and differentiated;
The treatment protocol should have some degree of flexibility to allow for individualization ;
The use of qualitative research (e.g., interview) to explore some missing areas (e.g., experience of practitioners and patient-practitioner relationship) in acupuncture research .
RWR has critical importance in acupuncture study, but need to be properly designed and conducted. A suggested flow diagram of RWR on acupuncture is presented in Fig. 2.
The choice of the condition(s) should be clinically significant. We believe that high-quality RWR is particularly needed for medical conditions where there are controversial evidence for acupuncture treatment, such as chronic low back pain.
Here, we recommend the GRACE approach . Ideal RWR should be prospectively designed, and involve an international and multicenter study. In addition, information of all individual participants should be recorded in detail (including intervention, outcomes, safety and costs, as well as potential covariates) using a format that could be easily tracked.
To safely guard data credibility, data collection should be prospectively designed to capture sufficient data to support analysis if possible. This is particularly important for acupuncture studies due to the complex nature of acupuncture treatment. State-of-the-art but validated technologies (e.g., data mining) for information acquisition and management of mega-database should be used .
Multiple factor analysis is almost invariably required for the complex nature of acupuncture studies. In acupuncture RWR, without rigorous inclusion criteria, randomization, and standardized treatments, potential confounding factors may often exist, and therefore, influence validity and reliability of findings. In this case, a variety of statistical techniques, such as the propensity score and instrumental variable analysis can be used to reduce/minimize the effects of confounding factors .
Acupuncture is a complex and individualized treatment. Currently, the effect of acupuncture based on RCTs remains controversial. Common problems associated with acupuncture RCTs are imperfections regarding acupuncture treatment and inappropriate placebo/sham controls. RCTs are important in verifying the efficacy of acupuncture treatment, but the authors believe that RWR, if designed and conducted properly, will make a unique contribution. If taking an integrative approach, the RCT and RWR can complement each other and give full play to their advantages in acupuncture assessment. This will provide more objective and persuasive evidence in favor of acupuncture treatment.
Randomized controlled trials
Real world research
Traditional Chinese medicine
Global registry of acute coronary events
STandards for reporting interventions in clinical trials of acupuncture
World health organization: WHO traditional medicine strategy: 2014–2023. [http://apps.who.int/iris/bitstream/10665/92455/1/9789241506090_eng.pdf?ua=1]
NIN Consensus Development Panel on Acupuncture. NIH consensus conference. Acupuncture. JAMA. 1998;280:1518–24.
Kaptchuk TJ. Acupuncture: theory, efficacy, and practice. Ann Intern Med. 2002;136:374–83.
Park J, Linde K, Manheimer E, Molsberger A, Sherman K, Smith C, et al. The status and future of acupuncture clinical research. J Altern Complement Med. 2008;14:871–81.
Thomas KJ, MacPherson H, Thorpe L, Brazier J, Fitter M, Campbell MJ, et al. Randomised controlled trial of a short course of traditional acupuncture compared with usual care for persistent non-specific low back pain. BMJ. 2006;333:623.
Flachskampf FA, Gallasch J, Gefeller O, Gan J, Mao J, Pfahlberg AB, et al. Randomized trial of acupuncture to lower blood pressure. Circulation. 2007;115:3121–9.
Brinkhaus B, Ortiz M, Witt CM, Roll S, Linde K, Pfab F, et al. Acupuncture in patients with seasonal allergic rhinitis: a randomized trial. Ann Intern Med. 2013;158:225–34.
Lian WL, Pan MQ, Zhou DH, Zhang ZJ. Effectiveness of acupuncture for palliative care in cancer patients: A systematic review. Chin J Integr Med. 2014;20:136–47.
Brinkhaus B, Witt CM, Jena S, Linde K, Streng A, Wagenpfeil S, et al. Acupuncture in patients with chronic low back pain: a randomized controlled trial. Arch Intern Med. 2006;166:450–7.
Haake M, Müller HH, Schade-Brittinger C, Basler HD, Schäfer H, Maier C, et al. German Acupuncture Trials (GERAC) for chronic low back pain: randomized, multicenter, blinded, parallel-group trial with 3 groups. Arch Intern Med. 2007;167:1892–8.
Cherkin DC, Sherman KJ, Avins AL, Erro JH, Ichikawa L, Barlow WE, et al. A randomized trial comparing acupuncture, simulated acupuncture, and usual care for chronic low back pain. Arch Intern Med. 2009;169:858–66.
Vas J, Aranda JM, Modesto M, Benítez-Parejo N, Herrera A, Martínez-Barquín DM, et al. Acupuncture in patients with acute low back pain: a multicentre randomised controlled clinical trial. Pain. 2012;153:1883–9.
Macklin EA, Wayne PM, Kalish LA, Valaskatgis P, Thompson J, Pian-Smith MC, et al. Stop Hypertension with the Acupuncture Research Program (SHARP): results of a randomized, controlled clinical trial. Hypertension. 2006;48:838–45.
Enblom A, Johnsson A, Hammar M, Onelöv E, Steineck G, Börjeson S. Acupuncture compared with placebo acupuncture in radiotherapy-induced nausea–a randomized controlled study. Ann Oncol. 2012;23:1353–61.
Madsen MV, Gøtzsche PC, Hróbjartsson A. Acupuncture treatment for pain: systematic review of randomised clinical trials with acupuncture, placebo acupuncture, and no acupuncture groups. BMJ. 2009;338:a3115.
Langevin HM, Wayne PM, Macpherson H, Schnyer R, Milley RM, Napadow V, et al. Paradoxes in acupuncture research: strategies for moving forward. Evid Based Complement Alternat Med. 2011;2011:180805.
Ferreira AS, Lopes AJ. Chinese medicine pattern differentiation and its implications for clinical practice. Chin J Integr Med. 2011;17:818–23.
Paterson C, Baarts C, Launsø L, Verhoef MJ. Evaluating complex health interventions: a critical analysis of the ‘outcomes’ concept. BMC Complement Altern Med. 2009;9:18.
MacPherson H, Nahin R, Paterson C, Cassidy CM, Lewith GT, Hammerschlag R. Developments in acupuncture research: big-picture perspectives from the leading edge. J Altern Complement Med. 2008;14:883–7.
Colagiuri B, Smith CA. A systematic review of the effect of expectancy on treatment responses to acupuncture. Evid Based Complement Alternat Med. 2012;2012:857804.
The Medical Research Library of Brooklyn: The Evidence Pyramid. January 2004 [http://library.downstate.edu/EBM2/2100.htm]
Kaptchuk TJ, Chen KJ, Song J. Recent clinical trials of acupuncture in the West: responses from the practitioners. Chin J Integr Med. 2010;16:197–203.
Nahin RL, Straus SE. Research into complementary and alternative medicine: problems and potential. BMJ. 2001;322:161–4.
Lundeberg T, Lund I, Näslund J, Thomas M. The Emperors sham - wrong assumption that sham needling is sham. Acupunct Med. 2008;26:239–42.
Kaptchuk TJ, Stason WB, Davis RB, Legedza AR, Schnyer RN, Kerr CE, et al. Sham device v inert pill: randomised controlled trial of two placebo treatments. BMJ. 2006;332:391–7.
Kaptchuk TJ. The double-blind, randomized, placebo-controlled trial: gold standard or golden calf? J Clin Epidemiol. 2001;54:541–9.
Reinke T. Real-world research expands beyond comparing medications. Manag Care. 2012;21:12–3.
Trotter JP. Patient registries: a new gold standard for “real world” research. Ochsner J. 2002;4:211–4.
Fox KA, Eagle KA, Gore JM, Steg PG, GRACE and GRACE2 Investigators, Anderson FA. The global registry of acute coronary events, 1999 to 2009—GRACE. Heart. 2010;96:1095–101.
Douglas PS, Brennan JM, Anstrom KJ, Sedrakyan A, Eisenstein EL, Haque G, et al. Clinical effectiveness of coronary stents in elderly persons: results from 262,700 medicare patients in the American College of Cardiology-National Cardiovascular Data Registry. J Am Coll Cardiol. 2009;53:1629–41.
Gale CP, Manda SO, Batin PD, Weston CF, Birkhead JS, Hall AS. Predictors of in-hospital mortality for patients admitted with ST-elevation myocardial infarction: a real-world study using the Myocardial Infarction National Audit Project (MINAP) database. Heart. 2008;94:1407–12.
Tada T, Kimura T, Morimoto T, Ono K, Furukawa Y, Nakagawa Y, et al. Comparison of three-year clinical outcomes after sirolimus-eluting stent implantation among insulin-treated diabetic, non-insulin-treated diabetic, and non-diabetic patients from j-Cypher registry. Am J Cardiol. 2011;107:1155–62.
Devlin G, Gore JM, Elliott J, Wijesinghe N, Eagle KA, Avezum A, et al. Management and 6-month outcomes in elderly and very elderly patients with high-risk non-ST-elevation acute coronary syndromes: The Global Registry of Acute Coronary Events. Eur Heart J. 2008;29:1275–82.
Kaptchuk TJ. Effect of interpretive bias on research evidence. BMJ. 2003;326:1453–5.
MacPherson H, Altman DG, Hammerschlag R, Youping L, Taixiang W, White A, et al. Revised STandards for Reporting Interventions in Clinical Trials of Acupuncture (STRICTA): extending the CONSORT statement. PLoS Med. 2010;7:e1000261.
Vickers AJ, Cronin AM, Maschino AC, Lewith G, MacPherson H, Foster NE, et al. Acupuncture for chronic pain: individual patient data meta-analysis. Arch Intern Med. 2012;172:1444–53.
Witt CM, Aickin M, Baca T, Cherkin D, Haan MN, Hammerschlag R, et al. Effectiveness Guidance Document (EGD) for acupuncture research - a consensus document for conducting trials. BMC Complement Altern Med. 2012;12:148.
Witt CM. Clinical research on acupuncture - Concepts and guidance on efficacy and effectiveness research. Chin J Integr Med. 2011;17:166–72.
Witt CM. Efficacy, effectiveness, pragmatic trials–guidance on terminology and the advantages of pragmatic trials. Forsch Komplementmed. 2009;16:292–4.
Qiu Y, Xu H, Zhao DY. Therapeutic evaluation on complex interventions of integrative medicine and the potential role of data mining. Chin J Integr Med. 2010;16:466–71.
Stukel TA, Fisher ES, Wennberg DE, Alter DA, Gottlieb DJ, Vermeulen MJ. Analysis of observational studies in the presence of treatment selection bias: effects of invasive cardiac management on AMI survival using propensity score and instrumental variable methods. JAMA. 2007;297:278–85.
This work was supported by Beijing Project of Science and Technology Plan (No.D0805703020801). We are extraordinarily grateful to Dr. Kehong Zhang for his helpful advice and comments as well as elaborative English review of this manuscript.
The authors declare that they have no competing interests.
JL: Wrote the outline of the argument and the first manuscript, participated in redrafting and rewriting. HX: Conceived the general idea, critically read the manuscript and helped redrafting the manuscript, adding information about study design. BL: Conceived the general idea, critically read the manuscript, added argumentative content. All authors read and approved the final manuscript.
HX and BL have many years’ experience of researching complementary therapies, including acupuncture, and BL is the chairman of The World Federation of Acupuncture-Moxibustion Societies. All authors have a keen interest in complementary therapies and research methodology.
About this article
Cite this article
Luo, J., Xu, H. & Liu, B. Real world research: a complementary method to establish the effectiveness of acupuncture. BMC Complement Altern Med 15, 153 (2015). https://doi.org/10.1186/s12906-015-0676-6
- Real world research
- Randomized controlled trials