A systematic review of the clinical effectiveness of acupuncture for allergic rhinitis
BMC Complementary and Alternative Medicine volume 8, Article number: 13 (2008)
Allergies cause a considerable burden to both sufferers and the National Health Service. There is growing interest in acupuncture as a treatment for a range of conditions. Since acupuncture may modulate the immune system it could be a useful treatment for allergic rhinitis (AR) sufferers. We therefore assessed the evidence for the clinical effectiveness of acupuncture in patients with AR by performing a systematic review of the literature.
Searches (to 2007) were conducted in all major databases for randomised controlled trials (RCTs) evaluating the clinical effectiveness of acupuncture in the treatment of AR. No limits were placed on language. Studies were included if they compared acupuncture to a sham or inactive acupuncture treatment (placebo) with or without standard care. Meta-analysis was performed where feasible.
Seven relevant RCTs were included after screening and application of inclusion and exclusion criteria. The trials were generally of poor quality as assessed by a modified Jadad scale, with the exception of two studies which scored highly. A wide variety of outcomes was measured but most assessed symptom severity on a visual analogue scale. A meta-analysis failed to show any summary benefits of acupuncture treatment for symptom severity scores or serum IgE measures which could not have been accounted for by chance alone. Acupuncture was not associated with any additional adverse events in the trials.
There is currently insufficient evidence to support or refute the use of acupuncture in patients with AR. A large well conducted RCT, which overcomes identified methodological problems in the existing RCTs, would be required to resolve this question.
Allergies are responsible for an estimated annual expenditure of £1 billion in the National Health Service (NHS) . Allergic Rhinitis (AR) is a complex of symptoms associated with atopic hypersensitivity reactions to common allergens including house dust mite, animal dander and pollens from grasses, trees and weeds. The prevalence of AR is highest in developed countries. In Western Europe, AR is estimated to affect up to 23% of the population [1–3].
Acupuncture developed from the traditional Chinese medicine techniques that can trace recorded origins back to the 2nd century BC . Acupuncture involves the stimulation of acupoints that are located at lines of meridians that correspond to the flow of energy through the body. Modern acupuncture has evolved other methods of stimulating acupoints including the use of an electrical current, by applying pressure to the acupoint (acupressure) or using a low intensity laser [5, 6]. Complementary and alternative medicine (CAM) treatments are becoming increasingly popular in the UK. It is estimated the UK spends an average of £1.6 billion per year on CAM with acupuncture the fourth most common treatment behind aromatherapy, homeopathy and herbal medicines .
There is some biological plausibility to the use of acupuncture for allergies. A number of small studies suggest that acupuncture can modulate levels of cytokines and other anti-inflammatory mediators, although the net effect of these changes is not necessarily anti-inflammatory nor would they predictably attenuate allergic disease . Acupuncture can stimulate the release of β endorphin, which, coupled to the release of adrenocorticotrophic hormone (ACTH)  acts on the adrenal cortex to stimulate the release of cortisol, offering another possible anti-inflammatory effect [5, 8–10].
If it were proven to be effective, acupuncture would be an attractive alternative to conventional symptomatic treatment for some patients. Previous studies have shown that side effects are rare in acupuncture and generally only minor, such as irritation at the needle site . The cost of acupuncture sessions is likely to be comparable to that of symptomatic medication. In addition many patients dislike, and therefore do not adhere to, daily prophylactic medication use, and a drug-free, safe treatment option therefore has considerable attractions. In contrast many health policy makers remain sceptical about acupuncture and reluctant to commission acupuncture services while its effect remains unsubstantiated
The aim of this systematic review is to evaluate the evidence of effectiveness of acupuncture for AR.
Literature searches were conducted in MEDLINE (1966-August 2007), EMBASE (1988-August 2007), the Cochrane library (including DARE and CENTRAL), the British Library Allied and Complementary Medicines database (AMED 1985-August 2007) and the National Research Register for ongoing trials. The specialist acupuncture library Acubriefs  was searched as was the Chinese literature via the database provided by the China National Knowledge Infrastructure (CNKI) website . Index terms for 'acupuncture' 'allergic rhinitis' 'hay fever' and 'clinical trial' were used (see Additional file 1). The bibliographies of included studies were searched for additional references. All studies were collected and organised using the Refman software package (Version 11, Thompson ISI ResearchSoft).
Randomised controlled trials (RCTs) of acupuncture treatment for patients with AR were included in this review. RCTs were the only study design chosen to minimise any potential for bias where non-objective or patient reported outcomes are used in studies. Inclusion criteria were kept as broad as possible to include subjects with persistent AR (seasonal or perennial) with or without other allergies such as asthma. Patients of any age were included. Any form of acupuncture treatment that stimulates an acupoint was included such as solid needles, electro-acupuncture and laser acupuncture. Sham or fake acupuncture treatment with or without standard care was the comparator. Any outcome measure relating to the effect of treatment was sought including quality of life (QoL), days off work or school, rhinitis symptom scores, medication usage score and adverse effects. Studies published in any language were included.
Non- RCT evidence, studies investigating patients with non-allergic rhinitis or without a placebo (sham or inactive acupuncture) control were excluded.
Assessment of methodological quality
A modification of the Jadad scoring system  was used to asses the quality of the evidence found (Table 1). The scoring system enabled a maximum score of eight for each study and assessed quality parameters such as baseline characteristics, blinding, allocation concealment and drop out rates. A score of 1 was given for each parameter fulfilled, (with an additional point for adequate description of the method) and 0 for those not fulfilled or where this could not be assessed.
Data extraction and statistical methods
Data from included studies was extracted using a structured pro-forma (see Additional file 2). Where possible the results from studies were summarised quantitatively using the Revman software package (Version 4.2, Cochrane Collaboration, Oxford). Data of interest were continuous measures, so the relevant effects were differences between the means, summarised as weighted mean differences or where different scales were used to measure the same general attribute, standardised mean differences. Heterogeneity was assessed using the chi2 test and I2 test for heterogeneity. A random effects model was preferred where there was marked heterogeneity (I2 > 50%) and a fixed effects model in other circumstances. A funnel plot was the intended method of detecting small study and publication bias if sufficient included studies had been detected.
Quality assessment and data extraction were completed independently by two reviewers for English language studies and singly for non-English language studies. Any discrepancies were discussed and resolved between reviewers, but inter-rater reliability was not formally assessed.
Seven trials met the inclusion criteria (details of study flow are given in Figure 1). Five trials were published in English language, two were published in German. Of the excluded studies two were in English, two in German, two in Chinese and one in Czech (see Additional file 3).
The majority of trials were conventional RCT's, one trial was conducted in an experimental setting using a chamber with a regulated flow of pollen. One trial adopted a cross-over RCT design (full details of included studies are in Table 2).
Participants in the studies were recruited either via advertisements in GP practices or in the local media. This volunteer population may not be a true representation of the general AR suffering population. The participants in trials were generally adults with the exception of one study which was set in a paediatric respiratory clinic but included some adults
The diagnosis of AR was variable: Two studies included patients with perennial rhinitis, while six studied exclusively patients with seasonal symptoms. Details of diagnosis were poorly reported, with no details given in some studies and others simply confirming that patients had positive tests for allergies (skin prick tests or specific IgE measurement) without further details.
The intervention varied considerably across the trials. One study used pre-seasonal treatment, whereas the others treated patients when symptomatic. The frequency of treatment sessions ranged from one to five per week with a total between two and 16 sessions overall. One trial evaluated the use of laser acupuncture compared to needle acupuncture, but also included a no treatment group allowing comparison between acupuncture and a control group. The acupoints selected by practitioners were detailed in five of the studies. There were 24 different acupoints used in these trials. A wide range of outcome measures was used, with the most common being symptom severity score measured on a visual analogue scale (VAS) and medication use.
The comparator in trials was a placebo acupuncture treatment that generally involved needling away from the active site or needling at a more shallow depth with shorter needles (to avoid stimulating the acupoint).
Only three studies achieved quality scores of greater than four from a maximum of score of eight (Table 1). All of the trials were described as randomised but only three of the seven trials gave details of the method used. Only two of the trials included information on allocation concealment. Due to the nature of the intervention it is difficult to tell if the intervention was distinguishable from the control treatment or not. Blinding of patients and/or outcome assessors was mentioned in four of the trials. An intention to treat (ITT) analysis was stated in only one trial, but even here it was not possible to assess if this had really been carried out. Losses to follow-up were described in all trials and were minimal. In three reports the baseline characteristics of the study groups were not comparable, and in two further studies comparability could not be adequately assessed from the details given. This may be due to the low numbers of participants in some trials, but raises quality concerns. Only one study addressed the issue of sample size and power.
Symptom severity scores
Five trials reported symptom severity in a way that could be used in meta-analysis, and in one further study the authors kindly provided the raw data for meta-analysis . The study by Wolkenstein et al [16–18] could not be incorporated as it reported baseline VAS scores with means and median values, but only reported the outcomes as medians. One study did not report symptom severity scores  but investigated percentage change in symptoms from a pre-season level set to 100%. This study reported a significant change (at the 2% level) in nose swelling and conjunctivitis after treatment for both acupuncture and placebo groups, but did not report a comparison between the groups.
There was a high degree of heterogeneity between the studies when combined in the meta-analysis (Chi2 104.06 p = 0.00001 I2 96.2%) with a total of 160 subjects in the intervention arm and 188 in the control arm, which was associated with a non-significant trend in favour of the intervention, -1.09 (95%CI -2.33 to 0.10) (Figure 2). Two of the studies have shown a positive treatment effect in contrast to the other studies and the results are dominated by one study that has shown a strong treatment effect in favour of the intervention . This study however was of uncertain quality; when this study was excluded as a sensitivity analysis, the degree of heterogeneity dropped (Chi2 16.77 p = 0.002 I2 76.1%) leaving 172 patients in the intervention arm and 169 in the control arm. The overall effect estimate continued to show a non-significant trend in favour of the treatment of -0.43 (95% CI -0.89 to 0.02).
Clinical significance of improved symptom severity
It is difficult to interpret the clinical significance of any magnitude of change in symptom score on a VAS. One trial correlated the improvement in daily rhinitis score with a significant increase in the number of symptom free days (from baseline 3.2 to 12.7 after follow-up in the acupuncture group compared to baseline 1.38 to 2.4 in the control group, p = 0.0001) . The pre-treatment baseline outcome values of the groups in this trial, although similar for rhinitis score, were different for symptom free days and this might confound this conclusion .
The results of the Xue et al  study were also broken down to changes in nasal and non-nasal symptoms, which were both significant. However, this cross-over trial did not have a wash out phase between the cross-over of treatments. This design also does not take account of variable pollen counts and resultant symptom severities that may occur during the course of a pollen season. The authors did not comment on possible carry-over of treatment effect between phases.
Serum and plasma biomarkers
Two studies included measurement of serum IgE levels [15, 21]. The studies were combined in a meta-analysis (Figure 3) and included a total of 55 patients in the intervention arm and 57 in the control arm. There was very little heterogeneity between these studies (Chi2 0.61 p = 0.32 I2 0%) and the effect estimate was modestly in favour of the intervention (-0.19) but was not significant (95% CI -0.56 to 0.18).
One study investigated plasma cytokine levels and found a reduction in plasma IL-10 levels 24 hours after treatment in the active acupuncture arm . Plasma IL-10 levels were shown to be significantly reduced in the intervention group from baseline 138.1 +/- 28.5 ng/ml to 74.9 +/- 21.4 ng/ml (p < 0.05).
Changes in Medication use
One trial reports a significant reduction in drug usage in the acupuncture group . Pre-season use was set at 100% and in the acupuncture group medication use fell from 550% to 75% over the pollen season compared to the placebo group (263% to 88%). Magnusson et al  showed no differences between intervention and control groups in medication use. Williamson et al  showed no difference in the unit use of medication between groups.
Adverse events reported in trials
None of the studies reported any severe adverse events associated with the use of acupuncture.
This is, to our knowledge, the first comprehensive review that has specifically investigated the use of acupuncture in AR that has followed the standards guidelines of the Quality of Reporting of Meta-analyses (QUOROM) recommendations for the reporting of systematic reviews and meta-analysis . There were no restrictions applied to language and a number of literature databases were searched using a comprehensive search strategy. It was not possible to combine the results of all the trials due to the variety of outcome measures used. Data was obtained from authors where possible allowing pooling of some results into a meta-analysis which has not previously been published in this field. We cannot assess publication bias statistically in such a small collection of studies.
The quality scores of the studies were generally poor, although two scored very highly. While some of the studies pre-date the CONSORT guidelines for reporting RCTs, omission of such quality data from later studies is a serious concern. We nonetheless included these in the analysis since they fulfilled our inclusion criteria. Given the difficulty of performing RCTs in acupuncture all available data should contribute to a final analysis.
The meta-analysis of AR symptom scores showed no overall effect of acupuncture. This conclusion may be confounded by the generally low patient numbers and variable reporting of outcomes. Inclusion of Wolkenstein's data would have increased the number of patients analysed, but this was not available in a form that could be included in our meta-analysis. While there is a trend in favour of a positive effect of acupuncture, this is largely attributable to one study which scored poorly on quality assessment (19). The higher quality studies tended to show lower overall effects.
Our inclusion criteria stipulated that studies had to compare acupuncture with a control sham or inactive acupuncture (placebo intervention). Interestingly one of the excluded studies  compared acupuncture with standard care alone. Outcomes measured in this study included symptom severity and serum IgE, with IL-4 and Interferon gamma (IFNγ) levels. There was a significant improvement in long term cumulative score of symptom before and after treatment and a significant decrease of IgE and IL-4 for the acupuncture and standard care groups, but this did not reach significance between groups. The effects of acupuncture may be more marked when compared to a non-blinded comparator, highlighting the importance of sham control in acupuncture studies .
Other outcome measures such as drug use or QoL were either poorly reported or not reported across the studies. Likewise, the measurement of serum and plasma biomarkers such as IgE and cytokines was performed in only a few studies. While some individual studies have reported significant changes, the biological significance of these observations is questionable. The report of active acupuncture reducing serum levels of IL-10 contradicts evidence from allergen-specific immunotherapy, where active treatment is associated with an increase in IL-10 production . Overall these measures have not contributed usefully to the summative assessment of the evidence for acupuncture in AR. Any future use of such indices in trials of acupuncture in AR should focus on those which have been shown to be helpful in trials of proven therapeutic modalities such as allergen-specific immunotherapy.
A previously published systematic review investigating a number of CAM treatments for rhinitis and asthma was identified . The review was published by the international board of the Allergic Rhinitis and its Impact on Asthma (ARIA) collaboration in 2006 . It included studies on acupuncture, herbal medicines, homeopathy and physical techniques (including yoga, chiropractic and educational programs). This large review has attempted to answer a very broad question. It is limited in the fact that it only included studies published in the English language and the searches were only conducted in MEDLINE and the Cochrane library . The review contained four trials relevant to acupuncture in rhinitis, three of which are included in this review (one study included patients with non-allergic rhinitis so was not included in this review). The authors note the few RCT data available for AR and that the majority of studies outside of RCTs are 'not randomised, controlled or descriptive' . By focussing specifically on acupuncture for AR and using comprehensive search strategies across all publication languages we have been able to add to this evidence base by including four additional RCTs not identified in this previous review, and combine the results in a meta-analysis. The results of our review further confirm the previous work and strengthen the conclusion that there is currently insufficient high quality evidence to support or refute the use of acupuncture for AR .
It is not possible to recommend acupuncture as a proven treatment for AR on the basis of published evidence. However this evidence is derived from inadequate clinical trials. Studies in AR frequently demonstrate marked improvements in the placebo group, and this means that large, well-controlled studies are required to demonstrate true effects. The quality of future trials could be improved by utilizing the proposed standardised reporting of acupuncture studies, similar to the CONSORT statement [26, 29, 30] and use of standardised outcome measures for conventional studies in AR . Given the high prevalence of AR, and the popularity of CAMs such as acupuncture, recruitment to such a study should be limited only by the availability of suitable study centres with the required expertise in both allergy and acupuncture.
After completing this review the following article was published: Xue CC et al Acupuncture for persistent allergic rhinitis: a randomised, sham-controlled trial. MJA 187 (6): 337-341, 2007
This well conducted trial evaluated the use of acupuncture vrs a sham control acupuncture in patients with persistent AR over an eight week treatment program and 12 week follow-up period. The authors randomised 42 patients to active acupuncture and 38 to a sham control group.
The result of this trial was an improvement in cumulative seven day symptom score (sum of individual scores for nasal obstruction, sneezing, rhinorrhoea and nasal itching) between the active and sham acupuncture groups after treatment (effect size -1.85 in favour of treatment p = 0.01) and at follow-up (effect size -1.87 in favour of treatment p = 0.001) this however did not correlate with any difference in the secondary outcome measure which was relief medication use.
This trial follows the trend in results highlighted in the previously published literature with a moderate improvement in symptom severity score which is difficult to interpret in regard to clear clinical benefit. This therefore does not change the overall conclusions of this review but is highlighted here in the interest of including the most up to date information available.
Gupta R, Sheikh A, Strachan DP, Anderson HR: Time trends in allergic disorders in the UK. Thorax. 2007, 62: 91-96. 10.1136/thx.2004.038844.
Bauchau V, Durham SR: Prevalence and rate of diagnosis of allergic rhinitis in Europe. Eur Respir J. 2004, 24: 758-764. 10.1183/09031936.04.00013904.
Bauchau V, Durham SR: Epidemiological characterization of the intermittent and persistent types of allergic rhinitis. Allergy. 2005, 60: 350-353. 10.1111/j.1398-9995.2005.00751.x.
Ma KW: Acupuncture: Its place in the History of Chinese medicine. Acupuncture in Medicine. 2004, 18: 88-99.
Stux G, Berman B, Pomeranz B: Basics of Acupuncture. 2003, Springer London UK
Hopwood V, Lovesey M, Mokone S: Acupuncture & related techniques in physical therapy. 2003, Elsevier Science Ltd, London UK
Ernst E: The BBC survey of complementary medicine use in the UK. Complement Ther Med. 2000, 8: 32-36.
Zijlstra FJ, Lange B, Huygen FJ, Klein J: Anti-inflammatory actions of acupuncture. Mediators Inflamm. 2003, 12: 59-69. 10.1080/0962935031000114943.
Han JS: Acupuncture and endorphins. Neurosci Lett. 2004, 361: 258-261. 10.1016/j.neulet.2003.12.019.
Lee SC, Yin SJ, Lee ML, Tsai WJ, Sim CB: Effects of acupuncture on serum cortisol level and dopamine beta-hydroxylase activity in normal Chinese. Am J Chin Med. 1982, 10: 62-69. 10.1142/S0192415X82000117.
Ernst G, Strzyz H, Hagmeister H: Incidence of adverse effects during acupuncture therapy-a multicentre survey. Complement Ther Med. 2003, 11: 93-97. 10.1016/S0965-2299(03)00004-9.
Website title : 2007, [http://www.acubriefs.com]
Website title : 2007, [http://www.cnki.net]
Jadad AR, Moore RA, Carroll D, Jenkinson C, Reynolds DJ, Gavaghan DJ, McQuay HJ: Assessing the quality of reports of randomised clinical trials: Is blinding necessary?. Controlled Clinical Trials. 1996, 17: 1-12. 10.1016/0197-2456(95)00134-4.
Magnusson AL, Svensson RE, Leirvik C, Gunnarsson RK: The effect of acupuncture on allergic rhinitis: a randomized controlled clinical trial. Am J Chin Med. 2004, 32: 105-115. 10.1142/S0192415X04001813.
Wolkenstein E: Acupuncture and allergiology. Was leistet die akupunktur in der allergologie?. Deutsche Zeitschrift fur Akupunktur. 1996, 39: 124-126.
Wolkenstein E, Horak F: Protective effect of acupuncture on allergen provoked rhinitis. Wien Med Wochenschr. 1998, 148: 450-453.
Wolkenstein E, Horak F: Protekitiver Effekt von Akupunktur gegenuber einer mittels Allergen-provokation induzierten rhinitis. Dtsch Zschr Akup. 1993, 36: 132-137.
Langer H, Hauswald B: Die therapeutische Wirkung der Akupunktur undLaserpunktur bei Patienten mit Rhinopathia pollinosa. Dtsch Zschr Akup. 1989, 32: 109-111.
Petti FB, Liguori A, Ippoliti F: Study on cytokines IL-2, IL-6, IL-10 in patients of chronic allergic rhinitis treated with acupuncture. J Tradit Chin Med. 2002, 22: 104-111.
Ng DK, Chow PY, Ming SP, Hong SH, Lau S, Tse D, Kwong WK, Wong MF, Wong WH, Fu YM, Kwok KL, Li H, Ho JC: A double-blind, randomized, placebo-controlled trial of acupuncture for the treatment of childhood persistent allergic rhinitis. Pediatrics. 2004, 114: 1242-1247. 10.1542/peds.2004-0744.
Xue CC, English R, Zhang JJ, Da CC, Li CG: Effect of acupuncture in the treatment of seasonal allergic rhinitis: a randomized controlled clinical trial. Am J Chin Med. 2002, 30: 1-11. 10.1142/S0192415X0200020X.
Williamson L, Yudkin P, Livingstone R, Prasad K, Fuller A, Lawreance M: Hay Fever treatment in General Practice: A randomised controlled trial comparing standardised Western acupuncture with sham acupuncture. Acupuncture in Medicine. 1996, 14: 6-10.
Moher D, Cook DJ, Eastwood S, Olkin I, Rennie D, Stroup DF: Improving the quality of reports of meta-analysis of randomised controlled trials: the QUOROM statement. Quality of Reporting of Meta-analyses. Lancet. 1999, 354: 1896-1900. 10.1016/S0140-6736(99)04149-5.
Rao YQ, Han NY: Therapeutic effect of acupuncture on allergic rhinitis and its effects on immunologic function. Zhonggou Zhen Jiu. 2006, 26: 557-560.
White AR, Filshie J, Cummings TM: Clinical trials of acupuncture: consensus recommendations for optimal treatment, sham controls and blinding. Complement Ther Med. 2001, 9: 237-245. 10.1054/ctim.2001.0489.
Nouri-Aria KT, Wachholz PA, Francis JN, Jacobson MR, Walker SM, Wilcock LK, Staple SQ, Aalberse RC, Till SJ, Durham SR: Grass Pollen Immunotherapy Induces Mucosal and Peripheral IL-10 Responses and Blocking IgG Activity. J Immunol. 2004, 172: 3252-3259.
Passalacqua G, Bousquet PJ, Carlsen KH, Kemp J, Lockey RF, Niggemann B, Pawankar R, Price D, Bousquet J: ARIA update: Systematic review of complementary and alternative medicine for rhinitis and asthma. J Allergy Clin Immunol. 2006, 117: 1054-1062. 10.1016/j.jaci.2005.12.1308.
White A: Conducting and reporting case series and audits--author guidelines for acupuncture in medicine. Acupunct Med. 2005, 23: 181-187.
Akerlund A, Andersson M, Leflein J, Lidholdt T, Mygind N: Clinical trial design, nasal allergen challenge models and considerations of relevance to pediatrics, nasal polyposis and different classes of medication. J Allergy Clin Immunol. 2005, 115: S450-S482. 10.1016/j.jaci.2004.12.016.
Xue CC, An X, Cheung TP, Costa CD, Lenon GB, Thien FC, Story DF: Acupuncture for persistent allergic rhinitis: a randomised, sham-controlled trial. MJA. 2007, 187: 337-341.
The pre-publication history for this paper can be accessed here:http://www.biomedcentral.com/1472-6882/8/13/prepub
The authors would like to thank information specialists Sue Bayliss and Anne-Fry Smith for advice with literature searches. We would also like to thank Ann Massey and Natalie Rowles for document retrieval and administrative assistance.
This work was supported by the West Midlands purchasers levy to the Department of Public Health and Epidemiology, University of Birmingham. The sponsor played no role in the conduct, production or conclusions of this article.
The authors declare that they have no competing interests.
JR performed the literature search, applied inclusion criteria, data extracted, performed the meta-analysis and wrote the first draft of the manuscript. AH applied inclusion criteria, data extracted and edited the manuscript. JD translated German studies, applied inclusion criteria, data extracted and edited the manuscript. DW translated Chinese studies, applied inclusion criteria, gave statistical advice and edited the manuscript. CH gave methodological advice, participated in the design of the review and edited the manuscript. All authors read and approved the final manuscript.
Electronic supplementary material
Additional file 1: Search Strategy. Basic search strategy and terms used. This was adapted depending on the database being searched. (DOC 20 KB)
Additional file 3: Included and excluded studies. Details of included and excluded studies reason for exclusion and references. (DOC 88 KB)
About this article
Cite this article
Roberts, J., Huissoon, A., Dretzke, J. et al. A systematic review of the clinical effectiveness of acupuncture for allergic rhinitis. BMC Complement Altern Med 8, 13 (2008). https://doi.org/10.1186/1472-6882-8-13