Integrity of Evidence-Based Practice: Are Providers Modifying Practice Content or Practice Sequencing?

This study examined patterns of evidence-based treatment (EBT) implementation within community settings by evaluating integrity along separate dimensions of practice content (PC; a session included the prescribed procedure) and practice sequencing (a session occurred in the prescribed sequence) within a recent randomized effectiveness trial. We measured whether sessions showed integrity to PC and to flexible or linear practice sequences. Findings revealed that providers tended to incorporate content from the EBT protocol in most treatment sessions, but that the sequencing of the sessions was often modified, suggesting that providers are amenable to evidence-based procedures, but not necessarily their prescribed arrangement.

Keywords: service use, randomized trial, implementation, community mental health Service Use Findings from the Child STEPs Effectiveness Trial:

Additional Support for Modular Designs
For more than a decade, research, training, and policy efforts aiming to bridge the gap between clinical science and clinical service have focused on the effective dissemination and implementation of evidence-based treatments (EBTs) within public mental health settings. Yet, a major concern for those attempting to integrate EBTs into community mental health services has been that the efficacy of EBTs found within research contexts has not universally translated to effectiveness within service contexts (e.g., Southam-Gerow et al., 2010;Weisz & Gray, 2008).
To address this issue, researchers have begun to consider modifying the design of such treatments in order to enhance the fit between EBTs and the contexts in which they will ultimately be applied (Chorpita, Daleiden, & Weisz, 2005).
Given that many of the youths receiving public mental health services appear to have a higher degree of complexity and comorbidity than those in typical research trials (Southam-Gerow, Chorpita, Miller, & Gleacher, 2008), one strategy for enhancing the fit between science and service has been to increase the flexibility of EBT protocols, which traditionally prescribe a predetermined sequence and set of practices for a single disorder, to allow providers to make real-time, informed adjustments in response to clinical events that might otherwise adversely affect treatment (Chorpita, Korathu-Larson, Knowles, & Guan, in press). For example, researchers have recommended treatment designs that involve the application of efficacious practices commonly cited within the clinical literature using a data-driven, guiding algorithm that allows for adaptations to the selection and sequencing of practices to address the client's unique needs (Chorpita & Daleiden, 2009;Chorpita & Daleiden, 2014;. Such STEPS SERVICE UTILIZATION 4 designs can expand providers' catalogs of potential practices beyond simply those included in individual EBT protocols by tapping into the broader evidence base (e.g., interventions determined to be efficacious via randomized clinical trials)-in turn, allowing providers to address a wider array of clinical disorders and concerns-and allow providers to modify their treatment approach in response to incoming data (e.g., emergence of comorbid anxiety; poor outcomes with the current intervention). In a recent randomized trial examining the effectiveness of a modular EBT that allowed for adaptations to the selection and sequencing of practices as care was delivered with anxiety-, depression-, and conduct-disordered youths , the modular EBT was found to have better clinical outcomes, including steeper trajectories of improvement and fewer diagnoses at post-treatment, relative to standard treatment manuals and usual care, which has been characterized as featuring a wide variety of therapeutic practices (Garland et al., 2010) (Weisz et al., 2012). Additionally, when clinical outcomes from the same effectiveness trial were examined over a two-year period, Chorpita et al. (2013) found that youths who received the modular EBT improved at a significantly faster rate than youths who received usual care.
Yet, despite improved clinical outcomes for youths in the modular condition, results from the above study showed that youths' long-term service utilization was not significantly different across conditions-with roughly 40% to 50% of all youths having sought additional outpatient mental health services one year after entering the study, and approximately one-third of youths accessing additional services two years after study entry (Chorpita et al., 2013). Although these findings suggest a lack of an association between treatment condition and subsequent service use, the service use analyses from Chorpita et al. (2013) are limited in two major ways. First, the previous study examined youths' service utilization dichotomously (i.e., whether or not a youth STEPS SERVICE UTILIZATION 5 received additional services) and, therefore, did not account for the extensiveness of youths' service use, which may be an important consideration given that youths may access multiple services during an episode of care (Farmer, Burns, Phillips, Angold, & Costello, 2003). Second, the study examined youths' use of only one type of service (i.e., outpatient), which underrepresented the scope of available resources within the service system. Given that use of additional mental health services following an initial treatment episode may be an indicator of service quality (e.g., clients who have continued psychopathology or functional impairment may be more likely to seek subsequent services) and place burdens on both the individual and service system, further investigation into the impact of treatment modality on subsequent service use is necessary.
In the present study, we build upon the findings from the Chorpita et al. (2013) study by investigating whether youths' treatment protocol influenced their service utilization up to two years after beginning an episode of care in the same effectiveness trial. The study aims were to: (1) assess the association between treatment condition and the number of service settings (e.g., residential treatment center, community mental health center) youths accessed following an episode of care, and (2) examine whether service use varied within each condition over time.
Given that youths who received the modular EBT exhibited significantly better clinical outcomes relative to those who received usual care (Chorpita et al. 2013;Weisz et al., 2012), it was hypothesized that youths in the modular condition would utilize fewer subsequent mental health services than those in the usual care condition. Additionally, since the design of the modular protocol allows providers to address a wider array of clinical concerns, it was hypothesized that youths who received the modular EBT would also utilize fewer subsequent services than those who received a traditional standardized EBT. Furthermore, because treatment gains tend to be STEPS SERVICE UTILIZATION 6 most potent immediately after completing an intervention and diminish over time (Weisz, McCarty, & Valeri, 2006), we anticipated differences in service use to be greater during youths' first year of study involvement-as clients may be more likely to seek additional (or adjunct) services if they are unsatisfied with their previous (or current) therapy-and increasingly subside during the second year-as their psychopathologies should have hopefully resolved, thus alleviating their need to access additional services.

Method
This study used data from the Child System and Treatment Enhancement Projects (STEPs) effectiveness trial (Weisz et al., 2012), a longitudinal randomized trial involving the application of three treatment protocols within community mental health settings. All study procedures were approved by the institutional review boards of Judge Baker Children's Center (affiliated with Harvard Medical School), the University of Hawaii at Manoa, and the University of California, Los Angeles.

Participants
Youths (N = 174) averaged 10.59 years (SD = 1.76) in age and were predominately male (70%); 45% were Caucasian, 32% were of mixed ethnicity, 9% were African American, 6% were Latino or Hispanic, and 4% were Asian American or Pacific Islander. Annual family income was less than $40,000 for 55% of the sample, $40,000 to $79,000 for 28% of the sample, $80,000 to $119,000 for 12% of the sample, and more than $120,000 for 6% of the sample.

Measures
The Services Assessment for Children and Adolescents -Parent Version (SACA) is a semi-structured interview designed to assess youths' use of inpatient (e.g., residential treatment center, drug or alcohol treatment unit), outpatient (e.g., community mental health center, day STEPS SERVICE UTILIZATION 7 treatment program), and school (e.g., special classroom for students with behavior problems, school counseling) services. The SACA, which contains questions relating to 30 different service settings, prompts caregivers to endorse yes or no as to whether their child utilized services within a particular setting during the past 3 months. Due to limited power to detect effects for any one type of service setting, a total score representing an overall index of the number of accessed service settings was calculated by summing the yes responses aggregated across inpatient, outpatient, and school services. The SACA has demonstrated good test-retest reliability (κ = . 75-.86; Horwitz et al., 2001), parent-child agreement (κ = .45-.77; Stiffman et al., 2000), and concordance with service records (κ = .76; Hoagwood et al., 2000).

Procedure
Families participating in the Child STEPs effectiveness trial were recruited via referrals from providers across 10 community mental health centers in Honolulu, HI and Boston, MA.
Eligibility criteria required that children and adolescents had clinically elevated problem levels in the areas of anxiety, depression, or conduct (see Weisz et al., 2012 for details on clinical diagnoses, comorbidity, and the CONSORT flowchart). The study used a cluster randomized design, such that youths who met the inclusion criteria for the study then began treatment with their referring provider, who was previously randomized to one of three treatment conditions: Standard, Modular, or usual care (UC). The Standard condition employed three standardized treatment manuals [one for anxiety (Kendall, 1994), one for depression , and one for conduct (Barkley, 1997)], which each feature a linear sequencing structure and a prescribed order and number of treatment sessions. The Modular condition employed a modular EBT manual , which contained similar content to that of the Standard treatment manuals, but utilized a decision-making algorithm that encouraged providers to focus STEPS SERVICE UTILIZATION 8 on the primary presenting problem while allowing for flexibility in the sequence, content, and number of sessions to address the client's individual needs. Lastly, providers in the UC condition were instructed to utilize the treatment practices that they would typically deliver in their routine care. Prior to enrolling in the effectiveness trial, caregivers and youths reviewed and signed consent forms. The SACA was distributed upon study entry and again one and two years later.

Analytic Strategy
Given that subsequent service use is likely associated with youths' level of psychopathology at posttreatment and that treatment effects may be limited for youths who prematurely drop out of therapy (e.g., no show for their remaining treatment sessions; discontinue services due to a change in health insurance) (e.g., Pellerin, Costa, Weems, & Dalton, 2010), this study examined service use for a subsample of youths who had a routine termination from therapy (n = 82) in addition to the full sample. Youths in the Standard condition were classified as having a routine termination once their assigned treatment protocol was completed; youths in the Modular condition were classified as having a routine termination if agreed upon by the Principal Investigators, who reviewed and discussed the youths' therapeutic progress and outcomes on a weekly basis; and youths in UC were classified as having a routine termination when agreement to end the episode of care was reached amongst the family, provider, and study investigators. Youths who had a routine termination from therapy did not significantly differ from those who prematurely dropped out on age, gender, ethnicity, annual family income, or baseline service use. Additionally, the number of youths who had a routine termination from therapy did not significantly differ across conditions. In order to assess whether treatment condition influenced youths' use of subsequent services, analyses of covariance tested for differences in the number of service settings accessed STEPS SERVICE UTILIZATION 9 at the one and two year assessments across conditions using the number of service settings accessed at baseline as a covariate. To determine whether service use varied within each condition over time, we initially conducted a 3 x 3 analysis of variance (ANOVA) to assess for a condition (Standard, Modular, UC) by time (baseline, one year, two years) interaction; however, given the pattern of service use identified in this analysis as well as our hypothesis that service use at the two-year assessment would be limited, we chose to conduct and report a 3 x 2 ANOVA that focused on changes in service use between the baseline and one year assessment.
Paired samples t-tests then assessed for main effects of time within each condition. Table 2 summarize the average number of total, inpatient, outpatient, and school service settings accessed by youths over their two-year participation in the effectiveness trial for the full sample and subsample of youths who had a routine termination, respectively.

Full Sample
Although results found significant differences in the number of service settings that youths accessed in the 3 months preceding study entry across conditions [F(2,173) = 3.65; p = .
028]-specifically, that youths allocated to the Modular condition used a significantly wider variety of services relative to those allocated to UC [t(113) = 2.51; p = .013]-youths' baseline clinical severity was not significantly different across conditions (Weisz et al., 2012). After controlling for baseline service use, results revealed no significant differences in service there were no significant changes in service use for those who received the Standard EBT or UC.

Routine Termination Subsample
For the subsample of youths who had a routine termination from therapy, there were no significant differences in baseline service use across conditions [F(2,81) = 2.45; p > .05]. Results revealed a significant difference in service use at the one-year assessment across conditions [F(2,67) = 4.28; p = .018]. Subsequent post-hoc comparisons showed that clients in the Modular condition reported accessing significantly fewer service settings at the one-year assessment relative to Standard [Least Significant Difference (p = .032)] and UC clients (p = .009). No significant difference in service utilization at the one-year assessment was found between Standard and UC. Service utilization at the two-year assessment for this subsample was not significantly different across conditions. A significant Condition x Time interaction was also found for this subsample. Similar to the findings with the full sample, youths allocated to the Modular protocol utilized significantly fewer service settings at the one-year assessment compared to baseline [t(28) = 3.36; p = .002] while youths allocated to the Standard or UC protocols had no significant changes in service use.

Discussion
The present study examined youths' service utilization up to two years after starting an episode of care with one of three mental health protocols (i.e., Standard, Modular, or UC). These findings build upon the service use analyses reported in Chorpita et al. (2013), which explored the impact of treatment modality on whether or not youths subsequently accessed services, but did not account for the quantity of service settings that youths accessed nor consider inpatient or school service settings. By further exploring the association between treatment protocol and STEPS SERVICE UTILIZATION 11 service use, we hoped to better understand the design characteristics that may promote more effective and efficient community mental health services.
Findings from the present study showed that treatment condition influenced the number of service settings accessed by youths, one year after starting an episode of care. Specifically, youths who received the Modular EBT utilized a significantly smaller variety of service settings at their one-year assessment relative to those who received a Standard EBT or UC-but only when they had a routine termination from therapy. Additionally, youths in the Modular condition accessed significantly fewer service settings at their one-year assessment relative to their baseline assessment; whereas, youths in the Standard and UC conditions maintained the same level of service use during their first year of study involvement. One possibility for why these effects were more potent for the subsample of youths who had a routine termination as opposed to the full sample may be related to youths' level of psychopathology at posttreatment. That is, youths who prematurely dropped out of therapy may have had an increased need to seek subsequent mental health services in order to resolve their existing clinical concerns relative to those who completed an episode of care. Furthermore, given that much of the individualization allowed by the Modular protocol is likely to occur in the mid-to late-stages of treatment (e.g., after the provider has a better understanding of the client's unique clinical needs; Chorpita & Daleiden, 2014), it is possible that clients who prematurely terminated from the Modular EBT may not have received this aspect of care-and thus, may have had similar service use trajectories to clients who received the Standard protocol, which contained similarly efficacious practices, but lacked protocol adaptability. On the other hand, the limited effectiveness previously demonstrated by UC  in addition to the service use outcomes reported in this study suggest that some level of protocol structure may be beneficial for STEPS SERVICE UTILIZATION 12 treatment durability. These findings provide support for the longer term clinical benefits of the Modular protocol and highlight the promise of EBT protocols that involve structured adaptability for use within service settings.
One limitation of this study relates to the distribution schedule of the SACA in the Child STEPs effectiveness trial. Specifically, since the instrument was only distributed up to two years after youths entered the study, it is possible that youths may have sought additional services at a later time. In addition, this sample had a low representation of ethnic minority youths, and thus, these findings may not generalize to other mental health populations. Furthermore, this study collapsed inpatient, outpatient, and school services into one overall service use construct in order to enhance the statistical power of the analyses. Yet, the effects of treatment modality on type of service (e.g., inpatient versus outpatient) and other factors, such as format (e.g., individual versus group therapy) or dosage (e.g., one hour per week versus one hour per day), may have important implications for service systems, and thus these distinctions should be explored in future research.
These results highlight the value of treatment design considerations when contemplating effective treatment options for youths seeking public mental health services (e.g., Daleiden, Chorpita, Donketvoet, Arensdorf, & Brogan, 2006). Specifically, given that the Standard and Modular conditions featured similar content, but allowed for varied adaptability in the selection and sequencing of prescribed practices, there may be utility in disseminating and implementing EBTs that allow for real-time, structured adaptations to strategically address youths' full array of needs. Although further investigation is necessary to better understand which treatment design characteristics can promote successful mental health outcomes within service settings, our STEPS SERVICE UTILIZATION 13 findings strengthen the support for protocols that allow for structured adaptation as a means towards greater effectiveness and efficiency within public mental health settings.