Methodological flaws on “manual therapy for the pediatric population: a systematic review” by Prevost et al. (2019)
BMC Complementary Medicine and Therapies volume 21, Article number: 4 (2021)
Prevost et al. published a systematic review evaluating the use of manual therapy for clinical conditions in the pediatric population in 2019. However, several methodological flaws in the conduct of the review limit the internal validity of its conclusions. We caution readers about the validity of the recommendations and suggest that the review not be used to inform the clinical management of pediatric patients.
We reviewed the systematic review by Prevost et al  and are concerned about its methodological quality. We critically appraised the systematic review using the Scottish Intercollegiate Guidelines Network (SIGN) criteria for systematic reviews. Our appraisal of the review suggests that it suffers from numerous methodological flaws, and that it does not meet the minimal quality criteria of a valid systematic review. The methodological limitations refer to the formulation of its research question, the search strategy, the classification of the quality of evidence, and the evidence synthesis. These limitations restrict the interpretation and generalizability of this review.
Specifically, our first concern is with the stated objectives of the review: “…to evaluate the use of manual therapy for clinical conditions in the pediatric population, assess the methodological quality of the studies found, and synthesize findings based on health condition.” These objectives are vague and do not comply with accepted frameworks to formulate research questions to assess the effectiveness of interventions [2, 3] In particular, the clinical conditions should be identified in the research question since other conditions (e.g., cardiovascular disease, renal disease) were not included in the review. “Assess the methodological quality of the studies found” is not a research question/objective. Furthermore, “Synthesize findings based on health condition” is not clear, specific or representative, or reproducible.
Second, the search strategy presented in Table 1 is not comprehensive and does not meet the current standards for systematic review literature searching, as defined in the Peer Review of Electronic Search Strategies (PRESS)  Specifically, important search terms that could identify studies conducted in the pediatric populations were not included (e.g., newborn, infant, youth). Moreover, according to PRESS, the search strategy should be developed with the assistance of a health sciences librarian and reviewed by a second health sciences librarian using the PRESS checklist  Although the authors mention that three librarians were used, it is unclear whether and how the librarians reviewed each other’s work using the PRESS checklist. Furthermore, the authors claimed that they searched several databases (as listed in Table 1). However, we are concerned about the validity of this statement because ScienceDirect and McCoy Press are journal publishers, not databases. Therefore, we cannot exclude that the search strategy led to significant selection bias in the included literature.
Third, this review used cut-off points to classify the quality of evidence (i.e., “low quality study if the score was between 0 and 33.3%, medium quality if the score was between 33.4 and 66.6%, and high quality if the score was above 66.6%”). Although cut-points are commonly used to classify levels of evidence, it is important to understand the limitations of this methodology. One important limitation is that impact of biases identified by the items is not weighted (i.e., each item is given the same weight regardless of their impact on study results)  Therefore, a study with a high score can be deemed to be of high quality even if it suffers from a fatal methodological flaw (e.g., differential attrition rates leading to differential ascertainment of outcomes).
Fourth, the results of each study were categorized to indicate whether a treatment led to: 1) improvement; 2) no improvement; or 3) no difference in participants’ outcomes. However, the decisions used to make the judgements are not clear and cannot be critically appraised nor replicated. Specifically, what consideration was given to statistical significance versus clinical importance and to within- versus between-group differences? Also, the main descriptive results presented in data extraction tables (Tables 6–10) only provide a general summary of the results and the necessary information to interpret the results are lacking (e.g., control interventions, effect size, statistical significance, precision of estimates, clinical importance, etc.). We could not identify any supplemental tables providing such information about control interventions and effect size.
Based on our concerns, we caution readers about the poor internal validity of the review and suggest that its recommendations should not be used to inform the clinical management of pediatric patients.
Availability of data and materials
The Scottish Intercollegiate Guidelines Network
The Peer Review of Electronic Search Strategies
Parnell Prevost C, Gleberzon B, Carleo B, Anderson K, Cark M, Pohlman KA. Manual therapy for the pediatric population: a systematic review. BMC Complement Altern Med. 2019;19(1):60 https://bmccomplementalternmed.biomedcentral.com/articles/10.1186/s12906-019-2447-2.
Riva JJ, Malik KMP, Burnie SJ, Endicott AR, Busse JW. Commentary – what is your research question? An introduction to the PICOT format for clinicians. J Can Chiropr Assoc. 2012;56(3):167–71 https://www.ncbi.nlm.nih.gov/pmc/articles/PMC3430448/.
Moher D, Liberati A, Tetzlaff J, Altman DG, The PRISMA Group (2009). Preferred reporting items for systematic reviews and meta-analyses: the PRISMA statement. BMJ. 2009;339:b2535.
McGowan J, Sampson M, Salzwedel DM, Cogo E, Foerster V, Lefebvre C. PRESS peer review of electronic search strategies: 2015 guideline statement. J Clin Epidemiol. 2016;75:40–6 https://www.jclinepi.com/article/S0895-4356(16)00058-5/pdf.
McGowan J, Sampson M, Salzwedel DM, Cogo E, Foerster V, Lefebvre C. PRESS Peer Review of Electronic Search Strategies: 2015 Guideline Statement. J Clin Epidemiol. 2016;75:40–6.
van der Velde G, van Tulder M, Côté P, Hogg-Johnson S, Aker P, Cassidy JD, et al. The sensitivity of review results to methods used to appraise and incorporate trial quality into data synthesis. Spine. 2007;32(7):796–806 https://www.ncbi.nlm.nih.gov/pubmed/17414916.
Ethics approval and consent to participate
Consent for publication
The authors declare that they have no competing interests.
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
About this article
Cite this article
Yu, H., Shearer, H., Taylor-Vaisey, A. et al. Methodological flaws on “manual therapy for the pediatric population: a systematic review” by Prevost et al. (2019). BMC Complement Med Ther 21, 4 (2021). https://doi.org/10.1186/s12906-020-03145-6
- Manual therapy
- Systematic review