Comparing physicians’ and patients’ reporting on adverse reactions in randomized trials on acupuncture—a secondary data analysis

Background We aimed to compare patients’ and physicians’ safety reporting using data from large acupuncture trials (44,818 patients) and to determine associations between patient characteristics and reporting of adverse reactions. Methods Six pragmatic randomized trials with an additional non-randomized study arm that included those patients who refused randomization were evaluated. Patients received acupuncture treatment for osteoarthritis of the hip or knee, chronic neck pain, chronic low back pain, chronic headache, dysmenorrhea, or allergic rhinitis or asthma. Safety outcomes were evaluated by questionnaires from both the physicians and the patients. To determine level of agreement between physicians and patients on the prevalence of adverse reactions, Cohen’s kappa was used. With multilevel models associations between patient characteristics and reporting of adverse reactions were assessed. Results Patients reported on average three times more adverse reactions than the study physicians: for bleeding/haematoma, 6.7% of patients (n = 2458) vs. 0.6% of physicians (n = 255) and for pain, 1.7% of patients (n = 636) vs. 0.5% of physicians (n = 207). We found only minor agreements between patients and physicians (maximum Cohen’s kappa: 0.50, 95% confidence interval [0.49;0.51] for depressive mood). Being a female and participation in the randomization were associated with higher odds of reporting an adverse reaction. Conclusions In our study, patients’ and physicians’ reports on adverse reactions of acupuncture differed substantially, possibly due to differences in patients’ and physicians’ questionnaires and definitions. For the assessment of safety, we strongly support the inclusion of patients’ and physicians’ reports while ensuring standardization of data collection and definitions. Electronic supplementary material The online version of this article (10.1186/s12906-019-2638-x) contains supplementary material, which is available to authorized users.

The rate of patient consultations for CAM treatments has been increasing [15]. As a result, the number of reported adverse reactions might increase, especially those which are not always avoidable, such as haematomas, nausea, vomiting, and aggravation of symptoms [13,16]. Previous observational studies showed that acupuncture can be considered a safe therapy [13,17,18], although some case reports might give another impression [12,19]. However, serious life-threatening adverse reactions of acupuncture, such as pneumothorax, are very rare [12,13] but have been published in some case reports [20]. Acupuncture treatment for chronic low back and knee pain had been included in routine reimbursements by statutory health insurances in Germany since 2007 [21][22][23][24].
Patients who are interested in receiving needle acupuncture treatment should be informed about possible adverse reactions for ethical reasons [12] and patient safety should have a greater priority in acupuncture training [14]. However, the type and frequency of adverse reactions are difficult to compare between the various studies evaluating acupuncture safety [12].
A limitation in many trials is that only health care professionals, especially the physicians, have the responsibility to document (serious) adverse events or adverse reactions [25,26]. The health care professionals have the professional competence regarding the evaluation of adverse events or adverse reactions, whereas patients have good individual knowledge about their own safety in healthcare [27]. Only a few studies document the adverse reactions by both physician and patients, e.g. [28] with the result that frequency and severity can differ between physicians and patients self-reports.
The aim was to compare patients' safety reporting with physicians' safety reporting regarding the safety of acupuncture using data from several large acupuncture trials. Furthermore, associations between patient characteristics and reporting of adverse reactions were evaluated.

Methods
The present secondary data analysis is based on the Acupuncture in Routine Care (ARC) studies that evaluated the effectiveness of the addition of needle acupuncture treatment [23] compared to usual care only. In those trials, patients and physicians had to complete questionnaires to document safety parameters.

Study design
The ARC studies were part of the German model project on acupuncture ('Modellvorhaben Akupunktur') funded by the German statutory health insurances [8,23,29]. The project was performed to evaluate the effectiveness of acupuncture treatment in routine medical care, as well as its safety and cost effectiveness [23].
The ARC studies were large pragmatic randomized trials with an additional non-randomized study arm including those patients who refused randomization. The recruitment period was from December 2000 to July 2004, and patients in both acupuncture treatment arms (randomized and non-randomized) received 10-15 sessions of needle acupuncture. Patients were eligible if they were at least 18 years of age and had been suffering from one of the following diseases for more than 6 months: osteoarthritis pain of the knee or hip, low back pain, neck pain, headache, allergic rhinitis/asthma, or dysmenorrhea. For each study, more detailed eligibility criteria were employed [8,[30][31][32][33][34].

Randomization for needle acupuncture
Of the 50,473 pooled patients over all trials who were asked if they agreed to be randomized, 11,486 agreed and were randomized either to the acupuncture treatment group (ACU, n = 5831) or to the control group (CON, n = 5655). Participants who did not agree to the randomization were part of the non-randomized acupuncture group (NR-ACU, n = 38,987). Participants in the ACU and the NR-ACU group started with the acupuncture treatment immediately, whereas the CON group received acupuncture treatment after 3 months. Needle acupuncture was performed by study physicians with at least 140 h of acupuncture training [8].

Data collection
At baseline, patient age, gender, school graduation, highest educational degree, occupational status, living situation, diagnosis, health insurance status, and health insurance type were assessed. Data collection regarding safety parameters was performed by questionnaires for patients and their study physicians after a complete treatment cycle.
If either the patient or the physician reported the presence of any side effect caused by acupuncture (adverse reaction) in a short first questionnaire, both received a detailed second questionnaire to report additional information about it, including frequency, duration, time between needle acupuncture and reaction, and treatment need because of adverse reaction. All questions of the questionnaires regarding the safety outcomes in the ARC studies are listed in the supplementary material (see additional file 1).
In the present analysis, only the acupuncture treatment groups ACU and NR-ACU that received the immediate acupuncture (n = 44,818) were considered because the CON group received different types of questionnaires regarding the safety parameters.

Safety parameters
We used the following definitions of the CONSORT statement to differentiate the safety parameters: adverse events are 'harmful events that occur during a trial.'; in contrast, adverse reactions are defined as 'events for which a causality link to the tested intervention is well established and strong enough (sensitive and specific)' [35]. Several other institutions in the healthcare sector such as the Food and Drug Administration (FDA), Europeans Medicine Agency (EMA), World Health Organization (WHO), or the German Federal Institute for Drugs and Medical Devices (BfArM) define adverse events and adverse reactions, including drug reactions, in a similar, but not identical way. The definitions are listed in the supplementary material (see additional file 2). In a review by Edwards and Aronson, the differentiation was explained as follows: 'The terms adverse effect and adverse reaction are interchangeable' and 'must be distinguished from adverse event.' [36].
In addition, the agreement between patients' and physicians' reports was assessed using Cohen's kappa (κ), a coefficient that measures inter-rater agreement corrected for agreement [38]. Kappa can take values from − 1 to 1 and can be interpreted in accordance with the five levels by Landis and Koch: less than 0.00, poor; 0.00 to 0.20, slight; 0.21 to 0.40, fair; 0.41 to 0.60, moderate; 0.61 to 0.80, substantial; and 0.81 to 1.00, almost perfect agreement [39]. Note that the observed agreement is prevalence-dependent, but the agreement by chance is not.
To assess the association between self-reported adverse reactions by patient or physician (yes/no) and patients' characteristics, a logistic regression approach was used. Because different participants were treated by the same study physician, the data are clustered. The effect of clustered data was estimated with the intraclass correlation coefficient (ICC) and the design effect (DE) [40]. The ICC estimates the correlation (similarity) of patients' and physicians' reports for patients of the same physician based on a null model, which represents a regression model with the variable for clustering only but no further covariates. If the ICC is near 1 and the design effect is much higher than 1, a clustered data structure is present. In the multilevel model developed by Laird and Ware [41], the clustered data structure can be taken into account in the model with physicians as random effects and patient characteristics as fixed effects. In a sensitivity analysis, generalized estimated equation (GEE) models by Liang and Zeger were used [42].
All analyses were performed with the statistics software R (The R Foundation for Statistical Computing, Vienna, Austria, version 3.1.1.) and the packages lme4 and geepack for clustered data regression based on the data set in SPSS format (IBM SPSS Statistics 19). An explorative significance level of 0.05 was used, and multiple test corrections were not applied. Note that the significance of all results (or confidence intervals) should be interpreted only exploratively. Furthermore, it should be noted that for all the following results, missing data were not imputed, and the analyses were based on the respective available data.

Patient characteristics
In the ARC studies, n = 44,818 patients received immediate acupuncture treatment in the ACU and NR-ACU treatment groups performed by 6727 physicians. Patients who received acupuncture treatment were on average 48.5 ± 14.1 (mean ± standard deviation) years old, and 67.5% were women (Table 1). Of the included patients, 37.1% had at least a high school degree, 59.5% were employees, and 83.8% live in a multi-person household. Characteristics were similar in the ACU and NR-ACU groups. The most common diagnoses for inclusion in the study were headache and neck pain in both groups (ACU: 26.9%, 30.1%; NR-ACU: 29.6%, 26.7%, respectively). On average, approximately seven patients were treated by one study physician (6.6 ± 9.1, median = 4), with a range of only one to more than 50 patients per physician.
Based on available data, 79% (n = 696) of the patients stated that they had informed their physician about their adverse reaction, whereas only 25% (n = 426) of the physicians reported they had learned this from their patients. Most of the physicians (88.5%, n = 1512) had not observed the adverse reactions themselves.
The agreements between patient-and physician-reported adverse reactions as measured by Cohen's kappa differed for the various categories (Table 2). Depressive mood had the highest kappa value of 0.50, which represents a moderate agreement between patient and physician. Anxiety (Cohen's kappa 0.35), tinnitus (0.29), and diarrhoea (0.29) also showed higher values. Many kappa values, however, represented only slight agreement with Table 1 Baseline characteristics for the acupuncture in routine care (ARC) patients by the treatment groups randomized acupuncture (ACU) and non-randomized acupuncture (NR-ACU) Patient  kappa values between 0.0 and 0.2 (not unexpected due to small prevalence). For several adverse reactions, the kappa value was estimated as agreement by chance (kappa = 0), e.g., for local infection, generalized muscle pain, increase in blood pressure, nerve injury, even though these adverse reactions had above-average prevalence. Specific questions on serious adverse events were included in the physicians' questionnaires (see additional file 1), but frequencies were too low to be analysed meaningfully.

Association between reported adverse reactions and baseline characteristics
The ICC, estimating the similarity of patients' and physicians' reports for patients of the same physician, was 0.12 based on patients' reports and 0.90 for physicians' reports. Therefore, 12 and 90% of the total variability is between the patients treated by the same study physicians, and this effect will be considered in the subsequent logistic regression analyses. To assess associations between patients' characteristics and patient-reported or physician-reported adverse reactions (yes/no), a multivariable multilevel logistic regression was applied. Female patients showed higher odds of reporting adverse reactions than males (

Discussion
We evaluated the reporting of adverse reactions in a secondary data analysis of a large semi-randomized controlled clinical trial on acupuncture for chronic pain patients. We compared patients' and physicians' reports regarding the frequency of adverse reactions and evaluated their agreement. Overall, the patients reported on average three times more adverse reactions than their physicians. The most commonly reported adverse reaction was bleeding/haematoma for both patients and physicians, similar to a study by Witt et al. [13]. Despite this, many types of adverse reactions were seldom reported, especially lifethreatening adverse reactions such as pneumothorax [13,20,43]. No or only slight chance-corrected agreements existed. However, differences in actual frequency did not necessarily result in differences regarding frequency category commonly used in product descriptions [37]. Moreover, we observed that the chance of reporting an adverse reaction either by the patient or the physician was higher for patients who had agreed to be randomized at baseline, i.e., who were willing to participate in an RCT, were female, and had a higher education degree. Various reasons might explain the difference between patients' and physicians' reporting. In general, the physician is equipped with more medical knowledge than the patient due to long-term medical training and professional experience, which can impact the reporting of adverse reactions, especially when the causality is vague. Indeed, it is feasible that the patient is best positioned to report his or her own symptoms [44].
The communication about the treatment and about its potential adverse reactions, the motivation and time for  question. This and the fact that these reactions might not be considered side effects by acupuncturists from the perspective of traditional Chinese medicine have possibly contributed to the differences in these categories. Although we included a comparably large number of cases and treatments with more than 44,000 chronic pain patients, conclusions regarding the specific frequency of  For the evaluation of acupuncture safety, other even larger studies were specifically designed, performed and published [13,17,45]. This study has some limitations. Firstly, we used secondary data from December 2000 to July 2004 which were not primarily designed to evaluate the differences between physicians' and patients' reporting of adverse reactions. The primary aim of the ARC trials was the evaluation of efficacy, while evaluating safety was only one of many secondary outcomes. We did not adjust for multiple comparisons.
A further problem is that the patients and physicians do not rate completely independent because many adverse reactions are invisible to the study physician and have to be reported by the patient to the physician first, and the physician might explain the definition of adverse reactions to the patient. Hence, physician's reports based on the patient's reports could validate the patient's report, assuming the physician's assessment can serve as the gold standard. However, this might cause underreporting, whereas over-reporting of too many unjustified adverse reactions could cause difficulties when explaining the safety characteristics of the intervention in a real life setting. A further limitation of our study is that the assessment of adverse reactions was based on retrospective self-reports, which can be influenced by recall bias [46]. The lack of differentiation between adverse events and adverse reactions caused by acupuncture could be an additional reason for the differential reporting. Definitions according to WHO, FDA or EMA differ by nuance [35,[47][48][49]. In the literature, these terms are sometimes used synonymously (e.g., [50,51]). The exact definitions of reputable institutions are listed in the supplemental material (see additional file 2).
In the physicians' questionnaire in this study, we included a definition for an adverse reaction that referred to its noxious and unintended character to separate it from an adverse event. In contrast, the patient's questionnaire did not include any explanation to improve clarity and usability of the questionnaire and because only adverse reactions and not adverse events had to be reported by patients. The difference in the questionnaires may to some extent explain differences in reporting of some adverse reactions, such as bleeding/haematoma that is sometimes intended by acupuncturists or pain, but not the differences for adverse reactions such as vertigo or fatigue. For future studies, we recommend a similar application of written definitions for both the physician and the patient questionnaire. However, tick boxes or free text should also be applied in a similar way. Not only who assesses but also how the assessment is performed can cause large differences in reported rates as shown in an RCT by Bent et al. [52]. This study compared three methods (1. an openended question, 2. an open-ended, defined question, and  Table 2) 3. a checklist of 53 common side effects) to assess adverse events experienced by study participants. The percentage of patients reporting any adverse events was much higher in the group using the checklist (77%) than in the first (14%) or second group (13%). This demonstrates the complexity of reporting and standards.
Strikingly, most of the studies on the safety characteristics of acupuncture are either based on therapists'/ Table 3 Association between patient characteristics and reporting of adverse reaction (yes/no) from patients and physicians (multivariable multilevel logistic regression, yielding adjusted* odds ratios and 95% confidence intervals (CI))  [57]. In the study, 37 men with prostate cancer reported their adverse events, and the agreements with the study physicians using Cohen's kappa was determined. The total Cohen's kappa value was 0.15, which represents slight agreement, and was similar to our results. For rheumatoid arthritis, the reporting of adverse drug events between patients (n = 4246) and physicians differed; even for serious adverse events, the agreement was only 37% [58], whereas patients reported more events, which is similar to our results. In a study comparing adverse events reported in postdischarge patient interviews with adverse events detected by medical record review, the agreement for adverse events (kappa = 0.20) and serious adverse events (kappa = 0.33) was low and comparable to our agreement results [59]. In contrast, in an oncology study in 2005, the agreement of 400 patients with their clinicians was higher (kappa up to 0.5) [60]. Especially, for observable reactions, the agreement was higher than for subjective ones [60].
A standardized reporting and documentation of both adverse events and reactions is essential [35,61,62]. For drug safety, the FDA developed a reporting system in 1998 [63]. For non-interventional pain studies, there are guidelines as CONSORT or ACTTION to document adverse reactions, but even these guidelines do not provide a high degree of detail [64]. In oncology, there are currently some documentation tools to combine the analytical and professional side of physicians and individual patient's side using quality of life, symptoms, and patient-reported outcomes to enhance patient-clinician communication and to enable early detection of toxicities [60,65]. In a British acupuncture study, the adverse events were monitored in a standardized way with self-reports by patient at each acupuncture session [66]. However, the documentation of safety by patient reports is still not standardized in clinical trials and health care. Even for the obligatory adverse drug reaction documentation, various information systems are used [67]. Standardized electronic web-based documentation software or intuitive mobile apps in contrast to classical methods (phone, questionnaire) could support the complete and harmonized documentation of adverse reactions [67,68]. Further, it is important to differentiate between adverse events and adverse reactions and to evaluate a possible causal link to the intervention.
We think that both patients' and physicians' reports should be included when evaluating safety aspects of a medical intervention while electronic documentation tools might support this. Patients (or their relatives) can play an important role in signalling safety aspects in clinical trials as well as in routine care [62,69] and can help the patient-centred approach in the future. Regular communication between the physicians, other clinical staff and patients and the standardization of documents, including clarification of definitions, might help to minimize differences.

Conclusions
In our study, patients' and physicians' reports of adverse reactions of acupuncture differed substantially, possibly due to differences in patients' and physicians' questionnaires and definitions. The use of frequency categories has been shown to be useful and able to compensate for reporting differences. For the assessment of safety parameters, we strongly support the inclusion of both patients' and physicians' reports while ensuring standardization of data collection and definitions.