5 - Power Simulation Program: Application to the Center for Neuroscience and Regenerative Medicine (CNRM) Sleep Healthy Using the Internet (SHUTi) Study

Abstract Text: Efforts to treat traumatic brain injury (TBI) and its comorbidities include the development of analytic tools for use in the design and analysis of TBI-related studies. Power analyses play a critical role in clinical trial design. Given the assumptions in the planning of a trial, these analyses provide a quantitative assessment to detect significant study results. Initial assumptions may change over the course of a trial. Changes in participant recruitment levels or variability of the outcome measure of interest may occur. Reassessment of power given these changes can be helpful for assessing whether adequate power is maintained or if the study sample size may need to be adjusted. While different statistical packages can calculate power under a variety of assumptions, we are proposing a method with a more tailored approach for assessing power reflective of more specific assumptions and questions investigators may have about their study data. The program tool we have developed uses R and available libraries in R software.

For the purposes of this program tool, the user would conduct a series of steps. First, data are generated based on protocol assumptions and/or information from comparable clinical trials where available–i.e., mean differences between treatment and placebo groups, variability of the study outcome measure, correlation in subjects’ outcome measures over time, and study sample size. Second, an assessment of the treatment effect is conducted where a statistical model with a term for the treatment effect (e.g., mean change over time in outcome between treatment and control groups) is fit to the data, as well as other criteria (e.g., two-sided test, significance level=0.05). Given the fitted model, a t-statistic is obtained for the treatment effect. Third, data are resampled from the original sample, the same model is refit, and t-statistics are obtained. Lastly, based on the pool of t-statistics, the program finds the proportion of treatment effects that are significant to represent power.

We applied this tool to a clinical trial that uses SHUTi to treat insomnia in participants with TBI. SHUTi is a program developed at the University of Virginia that utilizes cognitive behavioral therapy for insomnia (CBT-I) through an online portal. SHUTi was invented to solve the problem of limited accessibility to trained individuals providing CBT-I. In 2017, Ritterband et al found that 56.6% of adult participants who self-referred to participate in a study achieved insomnia remission status with SHUTi.

For purposes of SHUT-I, we applied initial protocol assumptions regarding treatment effects, the variability of the Insomnia Severity Index (ISI), and the correlation of the ISI over time, while assessing different recruitment levels. The original study was powered for N=200 participants. Power was assessed under smaller N (=160,140,120,110,100) given different treatment effect sizes (1 to 5), variability of the ISI (SD=5.5, 8.5), and correlation in individuals’ ISI between baseline and the first post-treatment follow-up visit (0.6 to 0.8), reflective of the variation in any given patient’s responses. Re-assessment of power based on the simulation indicated that for a mean difference = 3 in the ISI between treatment and control groups, power at all sample sizes was maintained (>0.85). For a mean difference >3, the study was more than optimally powered. A mean difference < 3 and a correlation less than 0.8 in patients’ ISI resulted in a less-than-optimally powered study. The power was substantially lower for N=120, where the power assuming different correlations (rho=0.8, 0.7, 0.6) was 0.86, 072 and 0.62, respectively. If previous assumptions with regard to SHUT-I in terms of treatment effect and study variability hold, the study power should be adequate for varying study recruitment levels.

This program tool is user-friendly, can be applied to a variety of study designs, not only clinical trials, and may prove to be a promising planning and pre-data analysis tool for researchers. It provides enhanced flexibility, given actual study data are simulated. Additionally, this flexibility allows for examination of power based on demographic or diagnostic information (e.g., severe vs. mild insomnia) (i.e., differential treatment effects) and allows for re-assessment of power under potentially varying study assumptions over time.

Keywords: power, research design, simulation, statistical analysis, sample size