Problems with the outcome measures in randomized controlled trials of traditional Chinese medicine in treating chronic heart failure caused by coronary heart disease: a systematic review

Background Traditional Chinese medicine (TCM) has gained widespread application in treating chronic heart failure (CHF) secondary to coronary heart disease (CHD). However, the sound clinical evidence is still lacking. Corresponding clinical trials vary considerably in the outcome measures assessing the efficacy of TCM, some that showed the improvement of clinical symptoms are not universally acknowledged. Rational outcome measures are the key to evaluate efficacy and safety of each treatment and significant elements of a convincing clinical trial. We aimed to summarize and analyze outcome measures in randomized controlled trials (RCTs) of TCM in treating CHF caused by CHD, subsequently identify the present problems and try to put forward solutions. Methods We systematically searched databases including Embase, PubMed, Cochrane Library, CBM, CNKI, VIP and Wanfang from inception to October 8, 2018, to identify eligible RCTs using TCM interventions for treating CHF patients caused by CHD. Cochrane Database of Systematic Reviews (CDSR) was searched to include Cochrane systematic reviews (CSRs) of CHF. Two authors independently assessed the risk of bias of the included RCTs according to the Cochrane Handbook. Outcome measures of each trial were extracted and analyzed those compared with the CSRs. We also evaluated the reporting quality of the outcome measures. Results A total of 31 RCTs were included and the methodology quality of the studies was generally low. Outcome measures in these RCTs were mortality, rehospitalization, efficacy of cardiac function, left ventricular ejection fraction (LVEF), 6 min’ walk distance (6MWD) and Brain natriuretic peptide (BNP), of which mortality and rehospitalization are clinical end points while the others are surrogate outcomes. The reporting rate of mortality and rehospitalization was 12.90% (4/31), the other included studies reported surrogate outcomes. As safety measure, 54.84% of the studies reported adverse drug reactions. Two trials were evaluated as high in reporting quality of outcomes and that of the other 29 studies was poor due to lack of necessary information for reporting. Conclusions The present RCTs of TCM in treating CHF secondary to CHD did not concentrate on the clinical end points of heart failure, which were generally small in size and short in duration. Moreover, these trials lacked adequate safety evaluation, had low quality in reporting outcomes and certain risk of bias in methodology. For objective assessment of the efficacy and safety of TCM in treating CHF secondary to CHD, future research should be rigorous designed, set end points as primary outcome measures and pay more attention to safety evaluation throughout the trial. Supplementary Information The online version contains supplementary material available at 10.1186/s12906-021-03378-z.


Introduction
Heart Failure (HF), a clinical syndrome of the dysfunction of ventricular filling or ejection led by the abnormality of cardiac structure and function, affects about 26 million people around the world [1]. The prevalence of HF is 1-2% of the adult population in developed countries [2,3] and in China there are about 4.5 million patients of HF [4]. Although the treatment of Chronic Heart Failure (CHF) has made great progress, the mortality and rehospitalization of CHF remain high, only half of patients could survive for more than 5 years [5,6]. The mortality of hospitalized patients with CHF was 4.1% according to the China-HF registry study [7]. From 2000 to 2010, the cardiovascular hospitalization of CHF has not decreased [8].
Coronary Heart Disease (CHD) is the first cause of CHF among all the primary diseases [9,10], so that the prevention and treatment of CHF caused by CHD is a significant part of cardiovascular health decisions. Traditional Chinese medicine has been widely used to treat all kinds of CHF which could effectively reduce the levels of N-terminal pro-brain natriuretic peptide (NT-proBNP) [11]. However, evidence from TCM clinical trials has not been universally acknowledged in the international medical system nor been included in clinical practice guidelines. The available randomized controlled trials (RCTs) are suboptimal with diverse outcome measures, many of which only showed the improvement of symptoms. To understand the status quo of outcome measures in RCTs of TCM in treating CHF caused by CHD, we conducted a systematic review to evaluate the outcome measures, identify relevant problems and try to put forward solutions.

Eligibility criteria
We included RCTs meeting the following criteria: (1) performed in CHF patients with CHD as primary disease (2) assessing TCM treatment compared with a control group (without restriction). Exclusion criteria were: (1) duplicate publication (2) studies without full text.

Information sources
Electronic databases including Embase, PubMed, Cochrane Central Register of Controlled Trials (CEN-TRAL) and China National Knowledge Infrastructure (CNKI), Chinese Scientific Journal Database (VIP), Wanfang and Chinese Biomedicine Literature Database (CBM) were searched from inception to October 8, 2018. Bibliographies of selected articles were also consulted in search of additional trials not detected in the initial searches.
We also searched Cochrane Database of Systematic Reviews (CDSR) to collect Cochrane systemic reviews (CSRs) of CHF for comparative analysis.

Search
We conducted a systematic search. "Medicine, Chinese Traditional [MeSH]", "Heart Failure [MeSH]", "Randomized Controlled Trial [Publication Type]" were applied as search terms and free words were used according to the characteristics of each database. The detailed search strategy was shown in Additional file 1.

Study selection
Two reviewers (JY H and RJ Q) independently selected the eligible studies, first through title and abstract and afterwards through the full text. Any disagreements of the selection period were discussed, and if the discussion could not resolve the problem, we consulted the third author (M L) and reached consensus.

Data collection process and data items
Reviewers JY H and CY L independently extracted information of the studies using a standardized data extraction form including the first author, year of publication, disease type, sample size, interventions in the treatment and control group and outcome measures.

Risk of bias in individual studies
We used the Cochrane Handbook for Systematic Reviews of Interventions version 5.1.0 [12] to assess the risk of bias of the included RCTs. Two reviewers (JY H and RJ Q) individually assessed the risk of bias and if there existed any disagreements, we resolved it through discussion with a third author (HC S).

Summary measures
We calculated the reporting rate of each outcome measure in the included RCTs and conducted comparative analysis with that in the CSRs of CHF. On account of the aim to analyze outcome measures, we did not synthesize data of the trials nor conduct a meta-analysis.

Additional analyses
Two authors (JY H, RJ Q) independently evaluated the reporting quality of outcome measures in the included RCTs based on the Management of Otitis Media with Effusion in Cleft Palate (MOMENT) criteria [13], considering the following 6 items: 1) Is the primary outcome clearly stated? 2) Is the primary outcome clearly defined so that another researcher would be able to reproduce its measurement? Where appropriate, this should include clear descriptions of time points, the person measuring the outcome, how the outcome was measured (for example, tools and methods used) and where the outcome was measured. 3) Are the secondary outcomes clearly stated? 4) Are the secondary outcomes clearly defined? 5) Do the authors explain the use of the outcomes they have selected? 6) Are methods used to enhance the quality of outcome measurement (for example, repeated measurement, training) if appropriate?

Study characteristics
Thirty-one included studies were all conducted in China and 29 were published in Chinese, two were published in English [39,40]. The main information of each study is shown in Table 1 and the information of 16 CSRs of CHF in Table 2.

Risk of bias within studies
Among the 31 RCTs, only seven studies [21,28,30,39,40,43,44] used "random number table" or statistical software to generate the random sequence, the others just mentioned "random" but no description of specific methods. Two studies [39,40] described allocation concealment and the blinding methods. Three studies [30,39,40] reported the case abscission and withdrawal. Generally, the risk of bias within the included RCTs was classified as high (See Fig. 2).  , 100%), focused on the end points and safety measures and analyzed the all-cause and specificcause mortality or hospitalization respectively. The overall reporting of outcome measures is shown in Table 3 and Fig. 3.

Additional analysis
Reporting quality of outcome measures All 31 RCTs reported the specific definition of outcomes, while only two [39,40] clearly stated the primary and secondary outcome measures which were considered as high reporting quality of outcomes. Eight studies [14,17,29,31,32,34,35,41] explained the use of the outcomes they had reported and five [19,21,23,28,40] adopted methods to enhance the quality of the outcome  Tables 4  and 5 shows the assessment of outcome reporting quality [13].

Discussion
This systematic review mainly analyzed outcome measures in RCTs which assessed the efficacy of TCM in treating CHF caused by CHD. We included 31 trials meeting the eligibility criteria and extracted outcome measures from these studies. The outcome measures were mortality, rehospitalization, efficacy of cardiac function, LVEF, 6MWD and BNP, of which mortality and rehospitalization are end points for patients with CHF while the others are surrogate outcomes [61]. Only four studies (4/31, 12.90%) reported mortality or rehospitalization, and in comparison, all 16 CSRs of CHF analyzed all-cause mortality. This difference indicated that present TCM trials mostly assessed the surrogate outcomes and lacked evaluation of CHF end points. In this review, nearly half of the included studies (14/ 31, 45.16%) did not mention any ADRs or adverse events, which apparently affected the safety assessment.
Apart from the problems of selecting outcome measures, the reporting quality of outcome measures was generally low, twenty-nine (93.55%) trials did not define the primary and secondary outcomes, which would confuse readers about major objectives of the trials and what the interventions really can improve.
In terms of methodology of the included RCTs, there were only two RCTs [39,40] considered as high-quality. In general, the risk of bias of these trials was classified as high. We considered that the design and implementation of most studies were far away from an optimal RCT in random sequence, allocation concealment, blinding, statistics and reporting.
The selection of outcome measures is a critically important step in clinical trials. Scientifically rigorous outcomes could show significant and comprehensive information about the efficacy and safety of specific intervention [62], which would produce positive impact on clinical choices and decisions for physicians. In largescale trials of heart failure, end points like mortality and hospitalization, were mostly set as primary outcomes [63,64] and treatments that could reduce mortality or morbidity would be recommended in influential clinical guidelines [65,66]. We did comparative analysis with CSRs, which are commonly agreed as high-quality information for making health decisions, to identify the present problems with outcome measures in studies conducted by TCM researchers. It was found that evaluation of improving clinical symptoms without robust evidence of clinical end points might be the primary reason why TCM interventions have not been widely recognized [67]. A European Society of Cardiology (ESC) consensus on the outcomes of HF trials [61], which was included in the Core Outcome Measures in Effectiveness Trials (COMET) database, highlighted that clinical end points could support the consolidation of therapeutic strategies. Whilst surrogate outcomes reflecting manifestations are typically applied in earlier phases of drug or device development to support proof-of-concept (Fig. 4). We recommended that the future TCM trials could refer to this consensus to select outcome measures.
The assessment of safety is indispensable for any clinical trial. In the included RCTs, CHF patients secondary to CHD, mostly hade one or more comorbid conditions that would potentially cause treatment conflict [68].   x Yunyou Cheng 2012 [21] x x Lihong Gong 2012 [22] x Junli Lu 2012 [23] x x Xu Gu 2003 [24] x Dongmin Liu 2011 [25] x Hua Zhou 2007 [26] x Renkui Lai 2015 [27] x Na Lin 2017 [28] x x Deyu Zhao 2011 [29] x x Wei Zhang 2010 [30] x He Li 2013 [31] x  Researchers should attach great importance to ADRs, adverse events or other safety outcomes throughout the studies and have the responsibility to estimate whether the intervention has a negative impact on patients or aggravates heart failure subsequently affecting mortality or hospitalization [69]. It is strongly recommended that TCM researchers should pay enough attention to the evaluation and reporting of safety in each trial. Through this review, we proposed that TCM clinical trials should focus on the assessment of clinical endpoints when evaluating TCM interventions in treating CHF. Whereas, we were aware that the included trials were all too small to assess clinical endpoints. Whether the quantity of participants, the duration of the trial or the involved areas, these trials cannot be regarded as large-scale trials. The shortest duration of the included trials was 1 week [39] in which it seemed to be impossible to record mortality, rehospitalization or other endpoints. Actually, there might be difference of the endpoints between treatment and control group when the follow-up time was longer than or equal to 6 months in clinical trials [20,22].
It is indeed difficult to conduct a TCM trial with certain size and duration to evaluate endpoints of heart failure, which would need appropriate organization and funding. We need high-quality prospective, multicenter x Yongzhi Wang 2018 [43] x Junfang Lv 2018 [44] x Notes: x in the column represents the study reported the corresponding item; empty columns indicate the study did not mention the item; studies with clear definitions of outcomes which did not preset primary or secondary outcomes were noted x in item 2 RCTs [11,70] rather than the present repetitive trials within a limited scale to promote the benign development of TCM [71]. We recommend collaboration among hospitals, research institutes and enterprises of TCM to conduct multicenter clinical trials to assess endpoints and generate convincing evidence which could guide the TCM clinical practice in a real sense. This review has several limitations. First, Thirty-one trials might not be enough to analyze various outcome measures. Second, neither our review nor the included trials distinguished heart failure with reduced ejection fraction or preserved ejection fraction, which would affect the selection and evaluation of corresponding outcome measures. Third, the proportion and reporting quality of the outcomes we analyzed in the review cannot involve comprehensive information about outcome measures in RCTs. The methods to measure the outcomes, timing of measurement, how to enhance the quality of outcome measurement, follow-up of the primary outcomes and the assessment of composite outcomes are all significant factors discussing outcome measures and our future research will focus on these problems. Fourth, due to the aims of the review, we did not conduct meta-analysis within the 31 RCTs. In the future, we would include trials without or with low heterogeneity, comprehensively analyze outcomes and evaluate the efficacy and safety of TCM treatments.

Conclusions
Several problems with the outcomes existed in present trials of TCM in treating CHF caused by CHD, including the lack of concentration on the clinical end points of HF, adequate safety evaluation, together with the low reporting quality. Moreover, the risk of bias was classified as high. In order to produce robust and convincing evidence for TCM in treating CHF caused by CHD, further studies should be rigorous and well-designed, set clinical end points as the primary outcome measures and strengthen evaluation of safety.