How are meta-analyses being conducted and reported in dentistry ? a meta-research study

Aim: This study aimed to evaluate how meta-analyses are conducted and reported in dentistry. Methods: We conducted a search to identify dentistry-related Systematic Reviews (SRs) indexed in PubMed in 2017 (from January 01 until December 31) and published in the English language. We included only SRs reporting at least one meta-analysis. The study selection followed the 4-phase flow set forth in the Preferred Reporting Items for Systematic Reviews and Meta-Analyses Statement (PRISMA), and it was independently conducted by two researchers. Data extraction was performed by one of three reviewers, and data related to conducting and reporting of the meta-analysis were collected. Descriptive data analysis was performed summarizing frequencies for categorical items or median and interquartile range for continuous data. Results: We included 214 SRs with meta-analyses. Most of the studies reported in the title that a meta-analysis was conducted. We identified three critical flaws in the included studies: Ninety (90) meta-analyses (43.1%) did not specify the primary outcome; most of the meta-analyses reported that a measure of statistical heterogeneity was used to justify the use of a fixed-effect or random-effects meta-analysis model (n=114, 58.5%); and a great part did not assess publication bias (n=106, 49.5%). Conclusion: We identified deficiencies in the reporting and conduct of meta-analysis in dentistry, suggesting that there is room for improvement. Educational approaches are necessary to improve the quality of such analyses and to avoid biased and imprecise results.


Introduction
Meta-analyses are an important component of systematic reviews and are a statistical method to combine results from two or more independent studies 1 . This method enables improving the precision of estimates and answering conflicting questions or questions not discussed in individual studies 2 . However, meta-analyses might be misused and biased in a similar way to other research methods.
Ioannidis 3 (2016) demonstrated that 9,135 meta-analyses were published in PubMed in 2014, corresponding to an increase in the publication rate of 2,635% between 1991 and 2014. The author highlighted that many meta-analyses are redundant, unnecessary, or have methodological flaws. Page et al. 4 investigated flaws in the application and interpretation in a sample of meta-analyses of therapeutic interventions. The findings demonstrated problems in aspects such as interpretation of the model used, subgroup analyses, and the minimum number of studies recommended to test funnel plot asymmetry.
In dentistry, Saltaji et al. 5 assessed 1,118 systematic reviews (SRs) published between 1991-2012; the findings demonstrated that almost 50% performed a meta-analysis, and the median of included studies in the largest meta-analysis was 9. Moreover, several articles assessed the quality of meta-analyses performed in different oral health specializations, demonstrating that the overall quality varies from low to medium 6 . However, there are no studies in dentistry evaluating reporting and conducting meta-analysis characteristics, which would enable highlighting areas for future improvement. Thus, this study aimed to evaluate how meta-analyses are conducted and reported in dentistry.

Materials and Methods
We used a dataset of SRs in dentistry indexed in PubMed in 2017. The reporting and conduct characteristics of these SRs were previously published 7 . We have only evaluated data from SRs with meta-analysis and assessed how meta-analyses are conducted and reported.

Search and Eligibility criteria
A full description of the search strategy and eligibility criteria is available in the study of Bassani et al. (2019) 7 . First, we conducted a search in PubMed to identify dentistry-related SRs published in the English language indexed in 2017 (from January 01 until December 31). We considered the article as a SR based on the Preferred Reporting Items for Systematic reviews and Meta-Analysis Protocols (PRISMA-P) definition 8 . In this article, we have only included SRs reporting at least one meta-analysis. In addition, we included articles regardless of questions answered and study designs included (clinical studies, in vitro, in situ, etc.). The search strategy used is presented in Table 1 and was based on MeSH terms of PubMed and a specific filter (U.S. National Library of Medicine).

Screening
We selected the studies using a reference manager software (EndNote X7, Thomson Reuters, New York, USA). Details about the pilot test screening and study selection are available in the study by Bassani et al. 7 . The study selection followed the 4-phase flow set forth in the Preferred Reporting Items for Systematic Reviews and Meta-Analyses Statement 9 . Two researchers independently identified articles by reviewing titles and abstracts, and then screening the full text in a second phase.

Data extraction
A standardized form was created using Microsoft Excel based on the study by Page et al. 10 (2016). A pilot data extraction was performed to ensure consistency, and the details are available in the study by Bassani et al. 7 . Data related to the conduct and reporting of meta-analyses were extracted by one of three reviewers. Data regarding the journal category (general or specialized journal), number of authors, SR focus (Epidemiology, Diagnosis, Prevention, Prognosis, Treatment/ Therapeutic, Other, Unclear, or Mixed), dental specialization, funding (reported or not reported), and data related to conducting and reporting of meta-analysis such as details about the primary outcome, the model used, number of studies in the largest meta-analysis, statistical heterogeneity, publication bias, additional analysis and software used to conduct the meta-analyses were extracted. The form used is available in the supplementary material. One author subsequently verified the data consistency, and the data were extracted again in the case of any doubt or inconsistency.

Data analysis
Descriptive analysis of the data was performed by summarizing frequencies for categorical items and calculating median and interquartile range for continuous data. All analyses were performed using Stata 14.2 software. Characteristics of the meta-analyses were assessed considering all included SRs and meta-analyses. Results Figure 1 presents a flow diagram outlining the study selection process. The initial search in PubMed yielded 1375 records, and we included 214 SRs with meta-analysis after study screening of the title/abstract and full-text analysis. Table 2 presents the epidemiological characteristics of the SRs with meta-analyses included in the study. Most of the meta-analyses were published in specialized journals (n=159, 74,3%). Considering the country where those meta-analyses were produced, 4 countries produced 64.3% of SR with meta-analysis published, in which Brazil had the greatest contribution with 61 (28.5%). The main specialization was Oral and Maxillofacial Surgery (n=32, 14.9%), followed by Implantology (n=31, 14.5%) and Periodontics (n=29, 13.5%). The main focus of most of the SRs with meta-analyses was treatment/therapeutic (n=101, 47.2%), and a great number of the included meta-analyses reported no funding (n=80, 37.4%). The median number of authors was 5 (IQR 4-6).   Table 3 presents the conduct and reporting characteristics of meta-analyses. Most of the SRs reported that a meta-analysis was conducted in the title (n=187, 87.4%). The "continuous variable" was the type of outcome most used (n=41, 34.4%), and the unit of measure of the first reported result of the primary outcome was "mean difference" (n=45, 38.5%). Most of the meta-analyses reported the statistical significance of the first reported result of the primary outcome as "favorable, statistically significant" (n=60, 50.8%), and the majority of studies used a random effect model for all meta-analyses (n=130, 62.5%). Regarding the statistical heterogeneity of included studies in the meta-analysis, the majority of meta-analyses described some method to formally evaluate the statistical heterogeneity of included studies (n=180, 87.0%), and the method most used was the I 2 test (n=161, 75.2%). When additional analyses were conducted, the most used was subgroup (n=65, 30.4%); however, most of the meta-analyses did not conduct additional analyses (n=121, 56.6%). The software most used to conduct meta-analyses was RevMan (n=73, 34.1%), followed by Stata (n=34, 15.9%). The median of studies included in the largest meta-analyses was 9 (IQR 5-16). Forty (40) meta-analyses (18.7%) presented a funnel plot graph; however, 10 did not report that the analysis was performed, and a further 3 did not report that publication bias was assessed.

Did the authors report assessing (or an intent to assess) publication bias?
No, publication bias was not assessed, and the authors did not report an intention to assess it 106 49.5% No, publication bias was not assessed, but the authors reported that they intended to assess it if they identified a sufficient number of studies 29 13.5% Yes, publication bias was assessed 79 36.9%

Continue Continuation
Which methods did the authors report using (or intending to use) to assess publication bias? (*considering the number total of studies)

Median of studies included in the largest meta-analysis 9 (IQR =16-5)
For the primary outcome, 90 meta-analyses (43.1%) did not specify the primary outcome, and most of the meta-analyses did not incorporate the risk of bias assessment into them (n=119, 59.9%). Most of the meta-analyses reported that a measure of statistical heterogeneity was used to justify the use of a fixed-effect or random-effects meta-analysis model (n=114, 58.5%) and did not assess publication bias (n=106, 49.5%).

Discussion
Our study is the first in dentistry to assess how meta-analyses are conducted and reported. We identified that the reporting and conduct characteristics of meta-analyses are varied. Most of the studies reported that a meta-analysis was conducted in the title and formally evaluated the statistical heterogeneity of included studies. However, most of the meta-analyses reported that a measure of statistical heterogeneity was used to justify the use of a fixed-effect or random-effects meta-analysis model and did not assess publication bias, thus demonstrating that there is room for improvement in conducting meta-analyses in dentistry. In addition, our estimates showed an increase rate of SRs with meta-analyses published in recent years compared to a previous study 5 .
Several studies in dentistry assessed the quality of meta-analysis published in different dental specializations 11,12 . Both publications demonstrated that the meta-analysis quality can vary from medium to low and highlighted that important aspects such as publication bias assessment were not conducted in most meta-analyses; these findings were also corroborated by our study. Page et al. 4 demonstrated that flaws in meta-analyses are not restricted to dentistry. The authors evaluated 110 statistical analyses in systematic reviews of therapeutic interventions in the biomedical field and pointed out that the involvement of statisticians in conducting systematic reviews could improve the quality of meta-analyses.
Three important flaws in the conduct/report of meta-analyses were identified. First, almost 50% of the included studies did not assess the impact of publication bias. Publication bias occurs when there is a predilection for publishing particular results, such as in studies reporting high effect sizes which are more likely to be published than studies reporting lower effect sizes, and this type of bias can affect systematic reviews. Failure to assess the possible impact of publication bias on systematic reviews results can generate an invalid estimate, which could guide interventions or actions based on biased conclusions 1,13 . Koletsi et al. 14 (2016) assessed 162 systematic reviews in dentistry, with the findings demonstrating a low rate of publication bias analysis performed and a high rate of improperly conducted publication bias assessment.
The second important flaw is the use of a statistical heterogeneity measure to justify the use of a fixed-or random-effects meta-analysis model. This could be considered a shortcoming since the choice of each effect only based on a statistical measure could exacerbate the effect of bias, thus producing a spurious estimate and an inappropriate conclusion. Recent recommendations clearly affirm that the decision between the use of the model is a topic of much debate. An alternative to the topic is a pragmatic approach planned to perform both analyses (fixed and random models), while the random effect model can be present in cases of not identifying asymmetry in funnel plot, even knowing that funnel plot asymmetry suggests that both methods are problematic. In addition, the author pointed out that "the choice between a fixed-effect and a random-effects meta-analysis should never be made based on a statistical test for heterogeneity" 1 .
Lastly, almost 50% of the meta-analyses included did not specify the primary outcome. Tricoo et al. 15 (2016) evaluated a sample of 96 systematic reviews and demonstrated that one-third of the studies did not specify or change the primary outcome. We did not assess if the studies included in our study were registered or did not compare the information reported in protocols and in the final publication, however it could be possible that study results were related to outcome reporting bias, generating imprecise results.
There are some undeniable limitations of our study. We only considered one database and only studies published in English, and we performed a cross-sectional analysis only considering 2017. Also, our data extraction was based on the study report, and it is possible that some SRs were conducted more rigorously than was specified in the report. Additionally, we did not include meta-analyses without a systematic review, which could have led to selecting studies with high methodological quality, and we did not judge the study quality and if a correct methodology was used to perform the meta-analyses because this was not the aim of this study.
Various approaches are encouraged to minimize the flaws identified in our analysis: 1) graduate program students should be receiving adequate training to perform and report meta-analyses; 2) involvement of statisticians on the systematic review team could improve analysis quality; 3) dissemination of PRISMA Statement extensions such as PRISMA for Individual Patient Data and Network Meta-Analyses could help in reporting adequate meta-analyses.
In conclusion, we identified deficiencies in the reporting and conduct of meta-analyses in dentistry, suggesting that there is room for improvement. Educational approaches are necessary to improve the quality of such analyses and to avoid unbiased and imprecise results.