I FAMAGU’ON-TA (Our Children)Guam System of Care Final Evaluation Report2002-2009 December 2009 University of Guam Center for Excellence in Developmental Disabilities Education, Research, and Service (Guam CEDDERS) Inside cover: ©2009 University of Guam CEDDERS Children’s Mental Health Initiative (I Famagu’on-ta) Final Evaluation Report Bonnie Brandt, M.A., Mariles Benavente, A.C.S.W., and Keith Villaluna, B.B.A. Data analysis was provided by Ranilo Laygo, Ph.D. December 2009 University of Guam Center for Excellence in Developmental Disabilities Education, Research, and Service (Guam CEDDERS) Office of Academic and Student Affairs UOG Station Mangilao, Guam 96923 One hundred percent (100%) funding for this report was provided by the Child Mental Health Initiative Cooperative Agreement #5U79SM054487-06 between the U.S. Substance Abuse and Mental Health Services Administration (SAMHSA)/Center for Mental Health Services (CMHS) and the Guam Department of Mental Health and Substance Abuse (DMHSA). Table of Contents 3 Introduction 4 What Is The System of Care Initiative? 6 Timeline of Key Milestones & Activities 8 Overview of the Evaluation Design and Methodology 11 Evaluation Findings 11 Who Were The Children & Families Served by Guam’s System of Care? 15 What Outcomes Did Children & Families Experience? 18 What Do We Know About Services & Service Providers? 22 How Do We Compare to Other System of Care Sites? 23 What Do Stakeholders Have to Say? 25 Summary and Recommendations Footer: Guam CMHI 2002-2009 Final Report | 1 List of Tables and Figures TABLES Table 1 Ethnicity page 12 Table 2 Legal Custody Status page 12 Table 3 Family Income page 12 Table 4 Child & Family History page 12 Table 5 Mental Health DSM-IV Diagnosis page 13 Table 6 Reasons Caregivers Gave for Their Child Having an IEP page 14 Table 7 Youth Report on Behavioral & Emotional Strengths, Intake to 24 Months page 16 Table 8 Caregiver Report of Behavioral & Emotional Strengths, Intake to 24 Months page 16 Table 9 Most Frequent Delinquent Behaviors and Illegal Activities at Intake & 24 Months page 17 Table 10 Outpatient Service Use Over Time page 19 Table 11 Support Service Use Over Time page 19 Table 12 Caregiver Perspective on Services Over Time page 20 Table 13 Youth Perspective on Services Over Time page 20 Table 14 Summary of Significant Differences Between the National Sample & Guam page 22 Table 15 Stakeholders Priority Areas and Challenges and Needs page 24 FIGURES Figure 1 Juvenile Justice Involvement at Intake page 14 Figure 2 Severe Clinical Impairment at Intake & After 24 months page 15 Figure 3 High Levels of Depression at Intake & After 24 months page 15 Figure 4 Suicide Ideation Over Time page 15 Figure 5 Suicide Attempts Over Time page 15 Figure 6 Change in Attendance After 24 months page 16 Figure 7 Change in Grades After 24 Months page 16 Figure 8 Arrests, Crime Convictions, & Incarcerations at Intake & After 24 Months page 17 Footer: Guam CMHI 2002-2009 Final Report | 2 Introduction In 2002, the Guam Department of Mental Health and Substance Abuse (DMHSA) was awarded a six year Cooperative Agreement from the Center for Mental Health Services (CMHS) of the Substance Abuse and Mental Health Services Administration (SAMHSA) to develop and implement I Famagu’on-ta (Our Children), a system of care for children and youth ages 5 – 21 with serious emotional disturbances (SED). A system of care is an approach to meeting the complex and changing needs of children and their families that is grounded in the belief that mental health services should be individualized, accessible and flexible, family driven and youth guided, culturally and linguistically competent, coordinated among agencies and providers, and community-based. To meet the evaluation requirements of the Cooperative Agreement, DMHSA contracted with the University of Guam Center for Excellence in Developmental Disabilities Education, Research, and Service (Guam CEDDERS) from 2002 - 2009 to develop and manage the evaluation component for I Famagu’on-ta. With input from I Famagu’on-ta staff, families and youth, and other community stakeholders, the Guam CEDDERS’ Children’s Mental Health Initiative (CMHI) Evaluation Team designed and implemented a comprehensive evaluation plan that addressed three levels of outcomes: (1) child and family; (2) service/ program; and (3) system. A significant focus of the Guam CEDDERS’ CMHI Evaluation Project was on the successful implementation of the national evaluation component. All funded system of care sites are required to participate in the five national evaluation research studies which were designed and managed by ICF Macro. Results of these studies are reported annually to Congress and document the positive outcomes associated with systems of care across the country. As part of the evaluation of I Famagu’on-ta, the Guam CEDDERS CMHI Evaluation Team also designed and conducted local evaluation activities. These included: * Satisfaction surveys for training events and conferences * Focus groups with families, youth, service providers, and other stakeholders * Structured interviews with agency and community partners * Development of family and youth personal stories * Customized data analysis to answer local evaluation questions * Development and ongoing implementation of a Continuous Quality Improvement (CQI) process for I Famagu’on-ta staff Throughout the term of its contract with DMHSA, the Guam CEDDERS’ CMHI Evaluation Team regularly reported its findings to I Famagu’on-ta, to families and youth, and community partners locally through public forums and nationally through conference presentations. A series of three annual evaluation reports were developed and disseminated. In 2008, the Guam CEDDERS CMHI Evaluation Project received SAMHSA’s Excellence in Evaluation Award in the category of Family and Youth Involvement in recognition of its efforts in implementing the national evaluation. In September 2009, a Children’s Mental Health Evaluation Summit was conducted with policy makers, families and youth, and agency/program representatives during which the Guam CEDDERS CMHI Evaluation Team reported a summary of the findings from its evaluation of I Famagu’on-ta over the past seven years. The findings presented during the Summit are included in, and expanded upon, in this report. For more information about any of the findings presented, or to obtain this report in an alternative format, please contact Bonnie Brandt at bonnie.brandt@guamcedders.org. Footer: Guam CMHI 2002-2009 Final Report | 3 What is the System of Care Initiative? Disparities in children’s mental health services were reported as early as 1969 by the Joint Commission on the Mental Health of Children, but little attention was directed towards addressing these disparities until the Children’s Defense Fund issued “Unclaimed Children: The Failure of Public Responsibility to Children and Adolescents in Need of Mental Health Services” in 1982. In this seminal report, Jane Knitzer noted that over 67% of children with serious emotional disturbances in the U.S. were not receiving the services they needed and many more were receiving inappropriate care. This report gained widespread attention and set the stage for children’s mental health policy reform. In 1983, the National Institute of Mental Health created the Child and Adolescent Service System Program (CASSP) to support the development of more accessible and appropriate children’s mental health services through a systems of care approach. As defined by Stroul and Friedman (1986), a system of care is: “A comprehensive spectrum of mental health and other necessary services which are organized into a coordinated network to meet the multiple and changing needs of children and their families.” Over the next decade, children’s mental health services continued to transform and in 1992, Congress authorized the Comprehensive Community Mental Health Services Program for Children and their Families. As of September 2009, this program, which is administered by the Child, Adolescent, and Family Branch of the Center for Mental Health Services, has funded 144 system of care sites across the states, territories, and tribal communities. A system of care is not just a “program.” It is a systems reform initiative that is guided by a set of core values and principles that were first articulated by Stroul and Friedman in 1986. To be effective, communities must embrace these values and principles and change the fundamental ways in which they provide services across programs and mandates. True systems of care development requires changes in policies, finance mechanisms, workforce development, and infrastructure. System of Care Core Values * Child-Centered & Family-Focused * Community-Based * Culturally & Linguistically Competent System of Care Guiding Principles 1. Children should have access to a comprehensive array of services, including formal services and informal supports. 2. Children should receive individualized services using a strengths based approach. 3. Children should receive services in the least restrictive environment. 4. Families should be full, authentic participants in all aspects of the system of care. Footer: Guam CMHI 2002-2009 Final Report | 4 5. Services should be integrated with linkages among agencies and programs. 6. Children should receive care coordination using collaboration among agencies and families. 7. Early identification and intervention should be available and promoted within the community. 8. Children should be ensured smooth transitions to adult services. 9. The rights of children should be protected and promoted. 10. Services should be nondiscriminatory, culturally appropriate, and responsive to special needs. Guam’s System of Care Milestones The system of care initiative on Guam can be traced back to the 1980’s when concern was raised over the off-island residential placement of children and youth with serious mental health needs. The Interagency Case Review Committee (IACRC) was formed, bringing together representatives from Guam’s child serving agencies to review cases and to make recommendations for off-island placement. The 1990’s were a time of local awareness building about the needs of children with mental illness. The Department of Mental Health and Substance Abuse (DMHSA) established the Child and Adolescent Services Division and the Guam System of Care Council (GSOCC) was established by law. Throughout this period, the community came together to discuss how the needs of this population could be better met and how to secure federal funding to develop a system of care on Guam. The theme for the 2000’s has been on building partnerships. Guam was fortunate to receive training and technical assistance from Hawaii’s Ohana Project, as well as from many nationally recognized experts in wraparound and systems of care. In May 2000, Guam CEDDERS successfully applied for a community action grant and Project Filak was launched to build awareness of the wraparound approach. Policy makers, service providers, and families participated in the national Systems of Care Policy Academy, and in 2002, I Famagu’on-ta became 1 of 29 Phase IV systems of care sites funded by SAMHSA. Over the past seven years, Guam has witnessed many changes in how services are provided to children with mental health needs. The community has achieved some significant milestones, including funding for I Famagu’on-ta in the 2008 and 2009 local government budgets. Footer: Guam CMHI 2002-2009 Final Report | 5 Local Evaluation Activities During the first year of funding, the Guam CEDDERS CMHI Evaluation Team facilitated the development of a logic model for Guam’s system of care with I Famagu’on-ta staff, families, and agency/program partners. A logic model is a tool that is used to describe how a program or initiative is supposed to work and what it hopes to achieve. I Famagu’on-ta’s logic model included a description of the population to be served, desired outcomes for the system of care initiative, and strategies that would be implemented to achieve the outcomes. Strategies, indicators, and outcomes were identified for three levels: (1) child and family; (2) program or services; and (3) system, and an evaluation plan based on the logic model was developed to answer locally identified evaluation questions. A variety of local evaluation activities were designed and implemented, in conjunction with data collection for the national evaluation, to answer these evaluation questions. National Evaluation Every CMHS funded system of care community participates in the national evaluation which is required by Federal law and organized into phases, based on sites funding cycles. I Famagu’on-ta was a Phase IV site and part of a cohort of 29 sites who received their initial federal funding in 2002, 2003, and 2004. Phase IV sites participated in five primary national evaluation studies: (1) Cross-sectional Descriptive Study; (2) Longitudinal Child and Family Outcome Study; (3) Service Experience Study; (4) System of Care Assessment Study; and (5) Services and Costs Study. The Guam CEDDERS CMHI Evaluation Team was primarily responsible for the implementation of the Cross-sectional Descriptive Study, Longitudinal Child and Family Outcome Study, and Service Experience Study, and collaborated with ICF Macro to obtain data for the other two studies. Cross-sectional Descriptive Study The purpose of the Cross-sectional Descriptive Study is to describe the characteristics of children and families served by CMHS funded system of care communities. All children and their families who were referred to I Famagu’on-ta were eligible to participate in the study, and of the 393 eligible children and families, 374 agreed to participate. Data for the Cross-sectional Descriptive study was collected from October 2004 – August 2009 using the Enrollment and Demographic Information Form (EDIF). Longitudinal Child and Family Outcome and Service Experience Studies The purpose of the Longitudinal Child and Family Outcome Study is to determine the clinical and functional outcomes of children and families served by the system of care. The Service Experience Study is designed to assess the kinds of services children and families receive, their satisfaction with these services, and caregiver’s perceptions of cultural competency in service delivery. Only children who were determined eligible for I Famagu’on-ta services and their caregivers participated in these two studies. A total of 239 children and their families were eligible to participate, and of these, 167 agreed to be interviewed. The Guam CEDDERS CMHI Evaluation Team Family Interviewers collected data through highly structured interviews using a standardized protocol of 11 caregiver instruments and eight youth instruments. A baseline interview was conducted within 30 days of enrollment into I Famagu’on-ta, with follow-up interviews being conducted every six months for up to 36 months. Interviews were scheduled at times and locations convenient for families and participants were provided with incentives for continued participation in the studies. Interviews were conducted in either English or Chamorro, depending upon caregivers’ preferences, or through an interpreter for some of the participating Chuukese families. Data for the Longitudinal Child and Family Outcome and Service Experience studies was collected from December 2004 – August 2009. Over the course of the evaluation, Guam CEDDERS’ CMHI Evaluation Team completed: 167 baseline interviews 135 six-month follow-up interviews 122 twelve-month follow-up interviews 120 eighteen-month follow-up interviews 108 twenty-four-month follow-up interviews 76 thirty-month follow-up interviews 70 thirty-six-month follow-up interviews Since data collection began in 2004, only 36 youth/families were considered lost to the evaluation studies. Data Analysis and Reporting for the National Evaluation Data analysis for the Cross-sectional Descriptive, Longitudinal Child and Family Outcome, and Service Experience studies was conducted by ICF Macro and Data Profile Reports were disseminated at regular intervals. These reports were provided to the Guam CEDDERS CMHI Evaluation Team who reviewed the data in light of the local context, and developed local presentations of the findings to I Famagu’on-ta staff, the Guam System of Care Council, and other stakeholder groups on a regular basis. Local analysis of data collected for the national evaluation studies was also conducted by the Guam CEDDERS CMHI Evaluation Team to address specific areas of interest and needs identified by stakeholders, such as a more in depth analysis of ethnicity data and analysis of descriptive and outcome data for special cohorts of youth enrolled in I Famagu’on-ta, such as youth who reported involvement in Guam’s juvenile justice system at intake. Numbers Served and Referral Sources A total of 3931 children were referred to I Famagu’on-ta between October, 2004 and August, 2009. Referrals to Guam’s system of care came from: Caregivers.................................... 33.0% Schools...................................... 27.7% Mental Health Providers (DMHSA & Private Clinicians) .................... 14.0% Courts, Probation, & Department of Youth Affairs . . . . 10.1% Child Welfare.................................. 8.9% Other ........................................ 6.4% Although caregivers were the largest source of referrals, this percentage may be somewhat inflated and reflect instances in which a judge, teacher, or some other individual told the caregiver to refer their child to I Famagu’on-ta. 1 Data is only reported for children and families who agreed to participate in the evaluation studies and for whom documentation on the Enrollment and Demographic Information Form was complete. As a result, data is not available on all children referred to I Famagu’on-ta and the actual number of children served by the program is larger than what is reported here. Guam CMHI 2002-2009 Final Report | 11 Demographic Characteristics of Children Served Although mental health problems impact the lives of children of both genders and of all ages, 74% of the children referred to I Famagu’on-ta were males and the average age of all referred children was 11.6 years. Data shows the following distribution of referrals by age group: 4 – 6 years ..................... 10.3% 7 – 11 years .................... 36.6% 12 – 14 years ................... 28.2% • 15 – 18 years ................... 24.6% Table 1 presents ethnicity data for referred children and demonstrates that the majority of children were of Chamorro or Chamorro mix ethnicity. This is an overrepresentation of this ethnic group based on Guam census data. Chuukese children and youth are also overrepresented, and Filipinos are underrepresented. Family Characteristics and History The majority of children referred to I Famagu’on-ta (83.2%) live with their biological families, and of these, 43.5% live with a non-parent relative such as a grandmother, aunt or uncle, or older sibling. Table 2 provides information on children’s legal custody status. At intake, 51.1% of caregivers reported being employed during the six months prior to their children entering I Famagu’on-ta and working an average of 39 hours a week. Although children enrolled in I Famagu’on-ta come from all socio-economic levels, Table 3 demonstrates that over 50% live in a household with an annualincome of less than $15,000. Families of children referred to I Famagu’on-ta often face multiple, complex challenges including homelessness, medical problems, lack of health insurance and transportation, as well as child and family histories, which place children at increased risk for mental health problems (see Table 4). When asked about their family history, 43% of caregivers reported a family history of depression; 29% of caregivers reported a family history of mentalillness, other than depression; and 50% of caregivers reported a family history of substance abuse. Characteristics of Children and Youth at Intake Mental health diagnosis Children referred to I Famagu’on-ta completed an intake process and received a diagnostic evaluation to determine their eligibility for services. Children eligible for system of care services met the following criteria: 1. Age: 5 – 21 years 2. Diagnosis: DSM-IV or its ICD-10 equivalent emotional, behavioral or mental disorder (except a primary DSM-IV diagnosis of “V” codes, substance use disorders, or developmental disorders are not eligible for the program unless these diagnosis co-occur with another diagnosable disorder). 3. Functioning Level: unable to function effectively in their homes, schools or community, or in a combination of these settings; or, whose level of functioning is such that the child’s needs requires multiagency involvement. 4. Duration of Problem: the disability has been present for a year or is expected to last more than one year. Table 5 provides information on eligible children’s mental health diagnoses. Because youth may have more than one diagnosis, percentages for diagnoses may sum to more than 100%. Presenting Problems reported at intake At intake, caregivers and youth are asked about the types of problems the child is experiencing that led to their referral to I Famagu’on-ta. The most commonly reported problems were: Conduct/Delinquency ................76.6% Hyperactivity & Attention . . . . . . . . . . . . . .58.3% School Performance. . . . . . . . . . . . . . . . . .52.4% Depression. . . . . . . . . . . . . . . . . . . . . . . . . .40.3% Adjustment .........................37.5% educational Status of Children at intake Ninety-five percent (95%) of youth had attended school within the six months prior to entering I Famagu’on-ta and 92.1% of youth attended a regular public school program. Children with serious emotional and behavioral disorders are at risk for school disciplinary actions, and caregivers reported that during the six months prior to intake, 42.2% of youth had been suspended from school. Children with mental illness are also often eligible for special education services if their disability negatively impacts their school performance. At intake, 41.4% of children were reported to have an Individualized Education Plan (IEP). Table 6 summarizes reasons caregivers gave for their child having an IEP. Juvenile Justice involvement at intake National data suggests that approximately 70% of youth in the juvenile justice system meet the criteria for at least one mental health disorder, and over 60% meet the criteria for multiple disorders (Shufelt and Cocozza, 2006). On Guam, 74% of youth who received services from I Famagu’on-ta reported delinquent or illegal behavior in the six months prior to entry into the program. Figure 1 provides information about the type of juvenile justice involvement youth experienced in the six months prior to enrollment into the program. What Outcomes did Children and Families experience? A total of 167 families participated in the Child and Family Longitudinal Outcome Study which was the primary data source to answer the question, “What outcomes did children and families experience?” The data presented below are based on an analysis of data for a cohort of children and youth for whom data was available at all of the following data collection points: intake, 6 months, 12 months, 18 months, and 24 months2. Emotional & Behavioral Health Changes in children’s emotional and behavioral health from intake to 24 months later were assessed along several dimensions. Positive outcomes include a reduction in the number of children experiencing severe clinicalimpairment (Figure 2) and a significant decrease in the number of children who reported serious depression (Figure 3). After 24 months, 52.2% of children were reported to have High Total Problem Scores on the Columbia Impairment Scale compared to 76.8% at intake, and the percentage of youth who reported high levels of depression dropped from 20.8% at intake to only 12.5% at 24 months. Suicide ideation & Attempts Over Time Nationally, suicide is the third leading cause of death among youth ages 10 -24 years and children with serious emotional and behavioral disorders are at increased risk for suicide. At intake, youth enrolled in I Famagu’on-ta reported nearly twice the level of suicide ideation and self injury as youth enrolled in system of care sites nationally. However, after 24 months, Guam youth reported significant reductions in both suicide ideation (Figure 4) and suicide attempts (Figure 5). 30% 25% 20% 15% 10% 5% 0% 2 Due to the small sample size of the cohort of children and youth with data available through the 36 months data collection point and issues of validity associated with such a small sample size, a decision was made to use the intake to 24 months cohort as the group for data analysis and reporting. Functioning at School After 24 Months Youth with serious emotional disorders often experience multiple school related problems including high levels of absenteeism, poor academic performance, and frequent suspensions. Nationally, these youth also have the highest drop out rate of any disability group. Locally, school performance was identified as a significant presenting problem for over half of all children referred to I Famagu’on-ta. After 24 months, 51.6% of youth improved their school attendance (Figure 6) and 43.9% improved their grades (Figure 7). In a separate analysis of youth grades over time, the percentage of youth with a grade average of “C” or better improved from intake (62.9%) to 81.4% after 18 months and then decreased to 67.% after 24 months. No data is available on the reason for the drop at 24 months which may reflect the increasing difficulty of school work over time or changes in the types and/or amount of services children received the longer they remained in the study. No children enrolled in the evaluation studies reported being expelled from school at intake or in subsequent interviews. However, at intake, 42.2% of children were reported to have been suspended at least once in the six months prior to enrollment in I Famagu’on-ta. The percentage of children who were reported to have been suspended at each subsequent data collection interview decreased from 42.2% at baseline to: 28.9% after six months; 13.3% after twelve months; 17.8% after eighteen months; and 28.9% after twenty-four months. Functioning at Home & in the Community Behavioral and emotional strengths are associated with resiliency and increased functioning across environments. I Famagu’on-ta children and youth experienced increases in measures of their strengths as self-reported in Table 7 and reported by their caregivers in Table 8. Although both youth and caregivers reported increased strengths over time, youth rated their behavioral and emotional strengths higher at each data collection point. Juvenile Justice Involvement Intake & After 24 Months On Guam, 74% of youth who received services from I Famagu’on-ta reported delinquent or illegal behavior in the six months prior to entry into the program. Due to the large number of youth with juvenile justice involvement at intake, a special data analysis was conducted for this cohort. As noted in Figure 8 and Table 9, after 24 months, youth reported fewer arrests, convictions, and incarcerations, as well as decreased rates of delinquent behaviors and illegal activities. Off-Island Residential Placements Prior to the funding of I Famagu’on-ta, children and youth in need of residential treatment and therapeutic group home services were being sent off-island for placement. The reduction in off-island residential placements was considered a priority outcome for Guam’s system of care. Prior to I Famagu’on-ta, 8-12 children were being sent off-island for residential placement each year. Since the funding of I Famagu’on-ta, only one youth has been sent off-island for placement in a treatment program for sexual offenders. This youth has since returned to Guam. Additionally, five youth who were in off-island residential treatment at the time I Famagu’on-ta was funded have returned to Guam, and three of these youth have graduated from high school and transitioned back home. A critical element of a system of care is an accessible array of community-based mental health services and supports. One outcome for I Famagu’on-ta was to develop a full service array that included: 1. Diagnosis/Evaluation 2. Case Management 3. Individualized Service Planning 4. Outpatient Counseling 5. 24/7 Emergency Services and Crisis Outreach 6. Intensive Home-based 7. Intensive Day Treatment 8. Respite 9. Therapeutic Foster Care 10. Transition to Adult Life 11. Therapeutic Group Home. Data from the 2008 national evaluation System of Care Assessment, as well as from structured interviews and focus groups conducted by the Guam CEDDERS CMHI Evaluation Team in 2009, indicate that some new services were successfully established through Guam’s system of care initiative, and that there are many services that still need to be developed and/or made more accessible. Strengths in the area of service array development include the provision of intensive case management and individualized service planning using a wraparound approach; intensive day treatment through the Rays of Hope program; and the establishment of Latte Treatment Center, a therapeutic group home. The recent pilot of the Transitioning Adolescents Successfully to Independence (TASI) which has ten I Famagu’on-ta transition age youth is another promising development. There has also been a recent increase in diagnostic and evaluation services, through the hiring of additional psychiatrists by DMHSA. Services that have been identified as high priorities are more transition services; intensive home-based services to prevent more restrictive placements and to serve as a “step down” to Latte Treatment Center; 24/7 emergency services and crisis outreach; therapeutic foster care; respite; and increased access to outpatient counseling. There are still significant barriers for the development of the service array. In 2007, comprehensive training was provided to prospective therapeutic foster parents. However, since the training, no additional therapeutic foster care providers have been certified due to administrative barriers families encountered in their certification process. Additional barriers to the development of a comprehensive service array include: lack of trained providers; problems with the Government of Guam procurement process and lack of timely payment of vendors; and “turf issues” among agencies and programs and unwillingness on the part of agencies to pool resources to establish new services. Service Utilization Children and youth in Guam’s system of care require multi-agency involvement to meet their needs, as well as a range of traditional and non-traditional services and supports. Traditional mental health services include outpatient services such as assessment/evaluation, therapy or counseling, medication management, and crisis stabilization, as well as inpatient services such as hospitalization, residential treatment, therapeutic foster care, and therapeutic group home. Support services are broader and focus on providing other things that the child and family needs in order to achieve the goals included in their individualized service plan. Support services include: care coordination, flex funds, peer-to-peer support, transportation, recreational services, transition planning, etc. Information about children’s and families’ use of outpatient, support, and inpatient services was collected every six months and analyzed to identity the most frequently utilized services within each service category, as well as to reveal changes in service use over time. The three most frequently utilized outpatient mental health services were: Assessment or evaluation ....................61.5% Individual Therapy.......................... 43.5% Medication Monitoring ...................... 36.3% Families reported that support services, such as Wraparound Care Coordination and informal supports, were especially useful and helped them and their children achieve positive outcomes. The three most frequently utilized support services were: Case management/wraparound . . . . . . . . . . . . . . 88.7% Informal supports .......................... 59.3% Transportation .............................33.1% Informal supports included peer-to-peer support provided by the Family Partners. Table 11 provides information about families service use over time for the six most frequently used support services. Unlike the use of outpatient services which declined over time (See Table 10), families continued to access several support services at relatively high rates at all data collection points. In systems of care, the focus is on meeting children’s needs through community-based services. However for some children, more intensive inpatient services will be required at times. Data on inpatient service utilization showed that over time, inpatient hospitalization increased from 4.3% at six months to 6.5% after 24 months. The reason for this increase is unclear and may be related to Latte Treatment Center being at capacity and/or the lack of Intensive Home Based services. If children with intensive needs were not able to access these community-based services, they may have had to be treated in a more restrictive setting. Youth & Caregiver Service Experience & Satisfaction with Services Satisfaction with services received in the six months prior to each data collection interview was assessed across several domains using the Youth Services Survey for Families (YSS–F), Abbreviated Version and the Youth Services Survey (YSS), Abbreviated Version. Scores on each domain range from 1 to 5. Higher scores indicate a positive service experience in that domain. Tables 12 and 13 summarize findings related to satisfaction for both caregivers and youth and demonstrates that levels of satisfaction across all domains increased over time. Factors Contributing to Youth Satisfaction A series of youth focus groups with I Famagu’on-ta youth, ages 15-19, were held in 2008. Members of the Youth Evaluation Team co-facilitated the focus groups and participated in analyzing and interpreting the data collected. During the focus groups, youth were asked: “With great achievement comes true happiness... I finally graduated from high school, and I am proud and happy with myself! College, here I come!” – Esther-Marie Santos, Southern High School Class of 2008 how do We Compare to Other System of Care Sites? In 2009, the Guam CEDDERS CMHI Evaluation Team collaborated with ICF Macro to conduct a preliminary comparison study of youth served by Guam’s system of care to a national sample. A random sample of equal size to Guam’s national evaluation cohort was drawn from the other 28 Phase IV sites for use in the comparison analysis. The study utilized the data collected for each participating youth/family from the Cross-Sectional Descriptive, Longitudinal Child and Family Outcome, and Service Experience studies using the standardized national evaluation instrument protocols. The comparison study was conducted in two parts. Part I compared the sample populations across a variety of dimensions at intake (i.e. demographics, service use and clinical histories, living situations, educational status, juvenile justice involvement, substance use, and clinical measures). Part II investigated service experience over time and included: number of services received, access to services, participation in treatment, cultural sensitivity, satisfaction with services, and satisfaction with outcomes. ICF Macro provided the statistical analysis for the study and results indicate that the two samples are significantly different across several dimensions which are summarized below. 3 Brandt, B., Laygo, R., Xu, Y., and Dewey, J. Comparing Youth Served by a System of Care in Guam to a National Sample. Presented at the Hawaii Pacific Evaluators Conference, 2009. To learn more about the impact of the system of care initiative in our community and to identify strengths, unmet needs, and priorities for sustaining Guam’s system of care, the Guam CEDDERS CMHI Evaluation Team conducted focus groups with service providers, and the Guam System of Care Council, as well as structured interviews with Directors and managers from system of care agency/program partners and the Executive Director of Guam Identifies Families Terrific Strengths (GIFTS). In general, stakeholders felt that I Famagu’on-ta benefited the community in several significant ways. One was that the needs of many children with mental illness who were previously unserved, or inappropriately served, were now being met. Agency partners also noted that by I Famagu’on-ta serving this population of children, they were able to focus more on meeting their own agency’s mandates. Stakeholders also commented on the increased awareness in the community about the needs of children with emotional and behavioral disorders and the effectiveness of wraparound when implemented with fidelity. At the child and family level, stakeholders felt that through I Famagu’on-ta, youth and family involvement and advocacy were strengthened. Respondents also indicated that the wraparound process modeled an approach to problem solving for families to replicate. Further, the vast majority of respondents noted that the system of care initiative resulted in positive outcomes for children & families. At the program/service level, stakeholders identified the wraparound process as a significant strength. The process facilitated appropriate service delivery and the wrap efforts were youth/family focused and driven. The role of the Family Partner was also identified as a program/service level strength. This new role facilitated cross-agency collaboration and ensured outreach to families and advocacy. At the systems level, both the Guam System of Care Council (GSOCC) and GIFTS were identified as strengths. GSOCC was a forum for collaboration and accountability, helped actualize family involvement in the governance of the system of care, promoted the system of care approach and wraparound, and supported training and policy development. GIFTS was established as a local chapter of the National Federation of Families, entered into a collaborative role with the family court, and continues to provide training to the public school system. Stakeholders were also asked to identify and prioritize significant challenges and unmet needs. Table 15 provides a summary of their responses. Stakeholder Input at the Children’s Mental Health Evaluation Summit During the Children’s Mental Health Evaluation Summit, a total of 56 policy makers, families and youth, and agency/program Directors and managers gathered on September 15 & 16, 2009 to review the data presented in this report. As part of the Summit, caregivers, youth, and service providers also shared their personal experiences as system of care partners. From these testimonials and panel discussions, we learned more about family and youth resiliency and the complex challenges both families and providers face on a day-to-day basis. Wrap Coordinators, Family Partners, and agency partners shared their concerns about increasing caseloads and how that is impacting their ability to effectively implement a system of care approach with the children and families they serve. On Day Two, participants were given the opportunity to engage in a facilitated discussion about the future of I Famagu’on-ta and further development of Guam’s system of care for children and youth with serious emotional and behavioral disorders. Common concerns voiced by participants included lack of clarity about the local funding of I Famagu’on-ta as of October 1, 2009; concern over relationships between agencies and providers; and the need to focus on the development of the service array as our community’s number one priority. At the conclusion of the Summit, the Guam System of Care Council agreed to take the lead in convening follow-up discussions about “next steps.” The system of care initiative on Guam began as a grassroots movement in the 1980’s in response to the recognition that the needs of children with serious mental health problems were being inadequately addressed, with many children being sent off-island for residential treatment. For more than 20 years, awareness of the unmet needs of these children and their families continued to grow, as did the community’s efforts to better respond to their needs. A significant milestone in Guam’s system of care initiative occurred in 2002 when the Department of Mental Health and Substance Abuse (DMHSA) applied for, and was awarded by the Substance Abuse and Mental Health Services Administration (SAMHSA), a six year Cooperative Agreement for I Famagu’on-ta, Guam’s system of care for children and youth with serious emotional disorders. From 2002 – 2009, this program focused significant financial and human resources on developing and implementing Guam’s system of care. Findings from the evaluation of I Famagu’on-ta suggest that these efforts resulted in many positive outcomes. This is especially true at the child and family level. Data demonstrates that many of the children served by I Famagu’on-ta showed improvements in their emotional and behavioral health, and experienced positive outcomes in school, at home, and in the community. Positive outcomes and family satisfaction appear to be related to the provision of individualized service planning and care coordination provided by Wrap Coordinators, as well as the provision of informal supports primarily delivered through flexible funds and Family Partners. Positive outcomes were also achieved at the service and program level. Service providers across Guam’s child serving agencies, as well as the community at large, are now more aware of the needs of children with mental health disorders and have increased knowledge of system of care values, principles, and practices. Collaborative partnerships between front line providers have been strengthened; resulting in more coordinated and integrated service delivery for children and families. There has been increased involvement by the Courts and the Department of Youth Affairs in the wraparound process and outcome data for youth in Guam’s juvenile justice system demonstrates the positive impact of this increased cross-agency collaboration on the lives of children and families. Through the system of care initiative, new services were developed, including: individualized service planning using the wraparound approach; intensive care coordination; day treatment through the Rays of Hope program; and the opening of Latte Treatment Center, an on-island therapeutic group home for children with more intensive needs. The establishment of Latte Treatment Center was especially significant in addressing the community’s concerns over off-island residential treatment and facilitating the return of children who were placed off-island prior to the grant. However, Guam’s service array is still very much incomplete and many services remain difficult to access. Significant barriers to sustaining services developed under this initiative, and to developing additional services, exist in the “ Once a family engaged in the services of the i Famagu’on-ta program, the minor’s contacts with the juvenile justice system diminished. i Famagu’on-ta served as a diversion, usually keeping the youth from more serious delinquent acts. As there was less delinquent behavior the court was able to “ The program allowed for better treatment placements of children with Sed (serious emotional disturbances). Through i Famagu’on-ta, children with special needs were served in appropriate and non-punitive settings instead of inappropriate, juvenile correctional facilities such as dYA. The program facilitated the identification and proper treatment of children with Sed and helped to distinguish them and prevent treatment as a typical juvenile delinquent. The program helped to heighten staff awareness and sensitivity towards this target group, resulting in more appropriate referrals for mental health services, better monitoring of cases for referrals, as well as enhancing advocacy for educational opportunities for these children.” –Chris Duenas, Former Director Guam Department of Youth Affairs (DYA) areas of Government of Guam personnel policies and job classifications, workforce development, Government of Guam procurement procedures, and the timely payment of private contractors. At the systems level, a notable outcome was the establishment of the Guam System of Care Council (GSOCC) in statute in 2003 as the advisory body for Guam’s system of care. Consistent with system of care values and principles, the majority of Council members are families, and in the past year, Council membership was expanded to include youth representatives. Unfortunately, funding to support the operations of GSOCC was exhausted in 2008 and additional funds have not been made available. This has impacted the Council’s ability to perform to its functions. Authentic family participation in all aspects of the system of care is vital to systems reform. Over the past seven years, Guam’s families of children with serious mental health needs have become increasingly empowered and organized. In 2004, Guam Identifies Families Terrific Strengths (GIFTS) was established as a 501c3 and as a local chapter of the Federation of Families. In 2005, GIFTS, in collaboration with the Federation of Families, hosted the annual national Federation of Families Conference, and in 2007, was awarded the Statewide Family Network grant. During the past year, GIFTS has played an active role in providing training to Guam’s schools and families and, in collaboration with the Family Court, produced a video to educate families about what to expect when they are required to appear before the judge. While collaboration among front line providers improved over the grant period, this outcome was not achieved at the level of Directors and program managers. Early in the grant period, two collaborative bodies were established to address and facilitate system level coordination (strategic and sustainability planning, cross-agency policies and procedures, funding strategies, etc.): the Executive Commission, comprised of agency Directors and administrators from Guam’s child serving agencies, and the Ombudsmen, comprised of the program managers/administrators. Initially, these groups met regularly. Over time, participation in meetings decreased and eventually these groups ceased to meet on a regular basis. As a result, the cross-agency structures and processes crucial to supporting true systems change and the sustainability of Guam’s system of care were not developed. This appears to be related to several factors: lack of clearly defined roles and responsibilities; competing priorities within and between agencies, including DMHSA’s involvement in a Permanent Injunction for adult services; changes in leadership within DMHSA and other agencies; concerns over the impact of the initiative on individual agencies’ operating budgets; and a perceived lack of understanding and buy-in related to individual agency roles within the system of care initiative. Despite the absence of a strong governance body, some key policy makers, including members of the Guam Legislature and Judiciary, increased their awareness of the needs of children with serious emotional and behavioral disorders and became educated about the importance of a systems of care approach to meeting their needs. Increased awareness among members of the Legislature resulted in local appropriations of $750,000 specifically to I Famagu’on-ta in fiscal years 2008 and 2009, although I Famagu’on-ta had difficulty accessing the funds. Within the Judiciary, two of Guam’s judges who handle cases involving youth with serious mental health needs have become strong system of care advocates, applying the wraparound process and a therapeutic approach with the youth and families that appear in their courtrooms. Over the past seven years, Guam has experienced many positive outcomes as a result of the establishment of I Famagu’on-ta and the collaborative efforts of its families, youth, partnering agencies, and other community stakeholders. Many challenges and barriers were also encountered and there is much uncertainty about the future of Guam’s system of care and the sustainability of I Famagu’on-ta. To assist in identifying next steps, the following three priority recommendations are offered. “ System of Care has changed the way i operate, the approaches and actions i take. i have no problem delivering consequences, but what system of care has taught me is to measure those consequences against the culpability…rather than punitive consequences i also order treatment and restitution for kids with mental illness. (For youth with serious emotional and behavioral disorders) i abandon a delinquency focus and shift to a therapeutic approach.” –Honorable Arthur R. Barcinas, Judge Superior Court of Guam Continue to Develop and Sustain Guam’s System of Care. Continued development and sustainability of Guam’s system of care is essential but cannot be achieved by a single agency or program. It is crucial that the cross-agency and family driven collaborative structures (i.e. Guam System of Care Council, Executive Commission, and Ombudsmen) be revitalized and/or restructured. Members of these collaborative bodies need to re-commit to system of care values and principles, rebuild trust and collaborative relationships, and examine how respective agency mandates could serve as an impetus for coordination instead of fragmentation to create true responsiveness to the needs of children and families served. System of care values and principles should become the foundation for decision-making, especially when decisions are challenging and seemingly put agencies at odds with each other. Cross-agency policies and procedures to increase access and flexibility in the provision of services, as well as innovative ways (e.g. blended funded, cross-training and assignment of staff, etc.) for sustaining Guam’s system of care need to be explored and commitments formalized through an interagency Memorandum of Understanding. Ensure Availability and Access to High Fidelity Wraparound Care Coordination and Peer-to-Peer Supports. Providing high fidelity wraparound and peer-to-peer supports to children with the most complex mental health needs is crucial, and these services should be sustained and expanded. It is recommended that the Wrap Coordinators and Family Partners become locally funded classified employees and that existing vacancies both within I Famagu’on-ta and the Children’s and Adolescent Services Division (CASD) of DMHSA be filled. It is critical that DMHSA management, and others with decision-making authority related to DMHSA personnel and budgetary issues, increase their understanding of how wraparound care coordination differs from less intensive case management and that budget allocations, position descriptions, and caseloads assignments respond to these differences. If, due to budgetary and/or personnel shortages, DMSHA cannot meet the needs for high fidelity wraparound care coordination, cross-agency alternatives (e.g. cross-training of social workers from other child serving agencies) should be explored and policies and processes should be established for assigning providers. These discussions and decision making should occur at the level of the Executive Commission with input from families. While it is essential to continue to reform service delivery for children with serious emotional disorders and their families via continuance of wraparound services, it is also crucial that the range of mental health services is also strengthened and sustained for all other children who require other levels of mental health care in the community. 3. Develop a Comprehensive Service Array Through Collaborative Partnerships. “Children’s Mental Health” needs to become a prioritized agenda for Guam that is reflected in public policies and budgetary decisions. A cross agency collaborative approach, that also includes public/private partnerships, should be used to plan and fund the on-going development of a comprehensive service array. Families and community stakeholders need to be authentically involved in decision-making about the development of the service array, and priority should be given to developing community-based services, particularly those directed at reducing the need for more restrictive placements. In order for a comprehensive service array to be fully realized, the establishment of public/private partnerships is essential. Systemic issues related to the Government of Guam procurement process and timely payment of venders must be addressed at the highest levels of the government with the involvement of agency Directors, the Department of Administration, and the Bureau of Budget Management Research. Issues related to agencies’ ability to access federal monies in a timely manner also needs to be addressed. Families are an under-utilized resource and may be able to play key roles in the provision of peer-to-peer and other informal supports. Resources should be invested in building capacity within both GIFTS and the GSOCC to provide these types of services to families. References Knitzer, J. (1982). Unclaimed Children: The failure of public responsibility to children and adolescents in need of mental health services. Washington, DC: Children’s Defense Fund. Shufelt, J.L., & Cocozza, J.J. (2006). Youth with Mental Health disorders in the juvenile justice system: results from a multi-state prevalence study. Retrieved on May 15, 2009 from http://www.ncmhjj.com. Stroul, B., & Friedman, R. (1986). A system of care for children and youth with severe emotional disturbances (rev. ed.) Washington, DC: Georgetown University Child Development Center, National Technical Assistance Center for Children’s Mental Health. For More Information Contact: Children’s Mental Health Initiative (CMHI) I Famagu’on-ta Evaluation Project Center for Excellence in Developmental Disabilities Education, Research, & Service Office of Academic & Student Affairs University of Guam, House #12 Dean Circle Mangilao, Guam 96923 Tel: (671) 735-2397/2398/2497/2618 TTY: (671) 734-6531 Fax: (671) 734-2308 Email: bonnie.brandt@guamcedders.org One hundred percent (100%) funding for this report was provided by the Child Mental Health Initiative Cooperative Agreement #5U79SM054487-06 between the U.S. Substance Abuse and Mental Health Services Administration (SAMHSA)/Center for Mental Health Services (CMHS) and the Guam Department of Mental Health and Substance Abuse (DMHSA).