Front Cover, Page 1 Project Kariñu, Loving Our Babies Guam’s Early Childhood System of Care FINAL EVALUATION REPORT 2009-2016 Project Karinu Vision Our children and families will have healthy minds, bodies, and spirits for lifelong success Smiling young boy dressed in a blue and black plaid shirt, black pants, and green shoes. Page 2 Mother kissing her baby. Page 3 Table of Contents 4 Executive Summary 6 Introduction 7 The Importance of Early Childhood Mental Health 8 Origins of Project Kariñu 10 Overview of Project Kariñu 12 Timeline: Key Milestones October 2009 – September 2016 14 Overview of the Evaluation Design and Methodology 17 Summary of Evaluation Findings 18 Characteristics of Children and Families Served by Project Kariñu 24 Service Utilization 25 Child and Family Outcomes 30 Systems Change Outcomes 32 Training Timeline: October 2011 – August 2016 36 Challenges and Recommendations 38 Data Sources 38 References Smiling young girl dressed in a blue stripped shirt, light brown pants, and pink shoes. Page 4 Executive Summary Smiling young boy riding on a toy car. Smiling young boy with his arms outstretched. Smiling young girl climbing monkey bars. Project Kariñu is Guam’s early childhood system of care for young children from birth to age five with challenges in their social, emotional, and/or behavioral development, as well as for children who are “at risk.” A system of care is a special way of meeting the mental health needs of children and their families based on a set of core values and principles. In systems of care, families and service providers work in partnership and services are coordinated across various agencies and programs. Young children’s social emotional development and mental health is the foundation for future learning and school readiness, and health and well-being throughout the lifespan. Yet, many of Guam’s young children are at risk for poor outcomes. Prior to Project Kariñu, mental health services for young children were virtually non-existent. Most people were not aware of the importance of social emotional development and providers were not trained to work with young children. In 2009, the Department of Public Health and Social Services (DPHSS) was awarded a six year cooperative agreement from the Substance Abuse and Mental Health Services Administration (SAMHSA) for Project Kariñu and things began to change. From 2009 – 2016, Project Kariñu worked to develop and implement an early childhood system of care. Drawing from both the science of early childhood and lessons learned from the system of care movement, Project Kariñu developed and implemented a full continuum of mental health promotion, prevention, and intervention services and worked to bring about systems change. Throughout the grant cycle, the University of Guam Center for Excellence in Developmental Disabilities Education, Research, and Service (Guam CEDDERS) provided evaluation services for Project Kariñu which included implementation of four national studies and numerous local evaluation activities. Findings from the evaluation reveal that many positive outcomes were achieved. Project Kariñu established a highly visible presence across the island. Through outreach activities and social marketing, they increased community awareness of the importance of young children’s social emotional development and early childhood mental health. Since Project Kariñu is located within DPHSS, which is often the first point of contact with Guam’s service delivery system for families, caregivers could easily access services in a non-stigmatizing setting. Children and families served by Project Kariñu experienced positive outcomes. Children showed increased behavioral and emotional strengths and many showed significant improvement in their acting out (externalizing) and holding in (internalizing) behaviors. Caregivers learned new parenting skills and nearly one-third reported decreased levels of stress. The majority of caregivers reported satisfaction with the services they received and reported positively about their experience getting services. Page 5 Guam’s workforce increased its capacity to meet the social emotional and mental health needs of young children. Fifty-three (53) trainings were conducted, building providers’ knowledge and skills in screening and assessment, best practice approaches to service delivery and evidence-based and evidence-informed practices and interventions. Providers and stakeholders increased their knowledge of system of care values and principles and interpreters were trained to provide language access services in early childhood and behavioral health settings. Collaboration among providers also increased and was most notable in the areas of outreach, workforce development, development of a universal referral process, and service planning. The return on investment in early childhood programs is estimated to be between 6 – 10% (Heckman, 2008) so sustaining Project Kariñu is a sound investment in the future of Guam. Multiple strategies were pursued to ensure sustainability after the federal funding ends. Most significant were the inclusion of Project Kariñu in the FY 2017 DPHSS budget and federal funding of Guam LAUNCH, a mental health promotion and prevention grant which will assume some of Project Kariñu’s services. Over the seven years of federal funding, early childhood mental health evolved significantly and there is still more to do to ensure that the mental health needs of Guam’s young children are fully met. Based on the evaluation findings, the following three recommendations are priorities. 1. Ensure sustainability of Project Kariñu by: a. increasing awareness of early childhood mental health among policy makers; b. guaranteeing access to local funding; and c. reclassifying staff as permanent Government of Guam employees. 2. Expand service delivery through: a. continued workforce development; b. further development of Project Kariñu’s service delivery model; and c. revising the age for eligibility to birth – eight years. 3. Increase service coordination and integration through cross-agency referral, assessment and service planning processes. Toddler crawling on the floor. Two smiling children sitting on chairs. Mother and child sitting on a slide. Page 6 Introduction Smiling young girl with her hands on her hips. Project Kariñu (Loving Our Babies) is Guam’s early childhood system of care for children from birth to five years of age who are experiencing challenges in their social, emotional, and/or behavioral development, as well as for those who are considered “at risk.” A system of care is a special way of meeting the mental health needs of children and families. It is based on the belief that services should be community-based, child-centered and family driven, and culturally and linguistically competent. In a system of care, families and service providers work in partnership and services are coordinated across the various agencies and programs. Project Kariñu is located within the Department of Public Health and Social Services (DPHSS). It was initially funded in 2009 through a six year Cooperative Agreement from the Center for Mental Health Services (CMHS), Substance Abuse and Mental Health Services Administration (SAMHSA) under the program commonly known as the Children’s Mental Health Initiative (CMHI). In fulfillment of the goals of the grant, Project Kariñu has provided mental health services to Guam’s young children, increased the knowledge and skills of service providers, and contributed to systems reform. From 2009 – 2016, the University of Guam Center for Excellence in Developmental Disabilities Education, Research, and Service (Guam CEDDERS) provided evaluation services to Project Kariñu through a contract with DPHSS. The primary focus of the evaluation contract was to: 1. manage and implement data collection for the national evaluation studies; 2. design and implement local evaluation activities; 3. design and implement a continuous quality improvement process; and 4. disseminate evaluation findings through a series of annual evaluation reports and presentations to families and stakeholders. This report provides a synthesis of data collected from the evaluation of Project Kariñu and reflects the outcomes and experiences of children and families, as well as the impact of the initiative on service providers, cross agency collaboration, and policy reform. For more information about the evaluation methodology and/or findings presented, or to obtain the report in an alternative format, please contact Bonnie Brandt at bonnie.brandt@guamcedders.org. Page 7 The Importance of Early Childhood and Mental Health Project Kariñu’s vision is “Our children and families will have healthy minds, bodies, and spirits for lifelong success” and the science of early childhood tells us that early childhood development is the foundation for children’s overall health, school readiness, and long term well-being (Shonkoff and Phillips, 2000). Research in the area of neuroscience (National Scientific Council on the Developing Child, 2007) shows that children’s early experiences literally shape the architecture of the developing brain. Exposure to extreme or repeated stressors and/or disruptions in the child-parent relationship can affect young children’s ability to self-regulate, experience and manage emotions, and form relationships (Thompson, 2008). Children exposed to maltreatment, parental depression and/or mental illness, domestic violence and poverty are at increased risk. Problems in children’s social emotional development are often indicators of early, as well as future, mental health problems, which can result in poor health and outcomes across their lives. The good news is that early intervention works. The developing brain is incredibly resilient and there are proven strategies to address young children’s social emotional development and mental health. Project Kariñu’s approach to helping children and families is grounded in this knowledge and reflects best practice strategies in early childhood mental health. What is Early Childhood Mental Health? . The social-emotional and behavioral well-being of infants, toddlers, young children, and their families. . The developing capacity to experience, regulate, and express emotion. . The ability to form close, secure relationships. . The capacity to explore the environment and learn. 9.5% to 14% of children, birth to 5 years of age, experience social, emotional, and/or behavioral challenges which impact their or their families' functioning. Smiling young boy hugging a stuffed animal. Page 8 Origins of Project Kariñu Young boy. Baby playing with toys. Young girl laughing. Project Kariñu was a natural outcome of two separate, but related, systems reform initiatives: Guam’s Early Childhood Comprehensive System (ECCS) and the adoption of a system of care approach by Guam’s mental health system. Both of these initiatives were aligned with national reform efforts and local milestones are highlighted below. Guam’s Early Childhood Comprehensive System In 2004, the Executive Committee on Early Childhood was established and resulted in the development of Guam’s Comprehensive Plan for Early Care and Education of Young Children. Among priorities included in the plan was a focus on improving the coordination of care among early childhood programs and services. Building on this in 2005, Guam received funding for Project Tinituhon (The Beginning), Guam’s Early Childhood Comprehensive System (ECCS). A significant accomplishment of Project Tinituhon was the creation of Guam’s Early Childhood State Plan which articulated goals and strategies to ensure young children have access to comprehensive and coordinated health and education services. In 2008, the Guam Early Learning Council (GELC) was established via Executive Order (and later formalized in statute via P.L. 31-62). The GELC was tasked to support and enhance coordination and collaboration among agencies and programs serving young children birth – 8 years of age. Through the work of Project Tinituhon and the GELC, awareness of the importance of young children’s social emotional development and mental health increased. In 2008, Dr. Ross Thompson, Professor of Psychology at the University of California, reinforced this awareness through his presentation on the impact of early brain development and life experiences on young children at the Governor’s Summit on Early Childhood. Over 70 stakeholders were present and provided input for the refinement of the Guam Early Childhood State Plan. Stakeholder input from the Summit included the recommendation that Guam develop and implement an early childhood system of care to address young children’s mental health needs. In response to this recommendation, the Department of Public Health and Social Services, in collaboration with Guam CEDDERS, submitted a grant proposal to SAMHSA in January 2009 and DPHSS received federal funding for Project Kariñu in October of that year. System of Care Approach In 1982, Jane Knitzer published "Unclaimed Children: The Failure of Public Responsibility to Children and Adolescents in Need of Mental Health Services" bringing attention to the fact that over 67% of children with serious mental health needs were not receiving the help they needed. This report set in motion the reform of children’s mental health and development of the system of care approach. A system of care is defined as: “A comprehensive spectrum of mental health and other necessary services which are organized into a coordinated network to meet the multiple and changing needs of children and their families.” (Stroul and Friedman, 1986). It is both a philosophy and a systems change initiative that is guided by a set of core values and Page 9 principles. In 1993, the Comprehensive Community Mental Health Services Program for Children and their Families, referred to as the Children’s Mental Health Initiative (CMHI), was authorized by Congress, providing funding to states, territories, and tribal communities to develop and implement systems of care to address children’s mental health. A key milestone in children’s mental health reform on Guam occurred in 1994 with the establishment of the Child and Adolescent Services Division within the Department of Mental Health and Substance Abuse (DMHSA). From 1994 – 2000, there was growing concern about the state of children’s mental health on Guam. Children with significant mental health needs were being sent off island for residential placement and locally, little was known about systems of care. Then in 2000, Hawaii’s system of care, the Ohana Project, provided technical assistance to Guam which resulted in the formation of a collaborative working group committed to developing a local system of care. That same year, the Guam System of Care Council (GSOCC) was established and Project Filak, a community action grant, was awarded to Guam CEDDERS to increase awareness of system of care values and the wraparound approach. Policy makers, service providers and families also participated in a national Systems of Care Policy Academy and work began on seeking federal funding to bring the system of care approach to Guam. In 2002, SAMHSA awarded DMHSA a six-year cooperative agreement for I Famagu’on-ta (Our Children), Guam’s system of care for children 5 – 21 years of age with serious emotional disturbances. Federal funding for I Famagu’on-ta ended in 2009 and the program has been sustained with local funding. Early Childhood System of Care Core Values 1. Child-centered and family-driven 2. Community-based 3. Culturally and linguistically competent System of Care Guiding Principles 1. Children should have access to a comprehensive array of services. 2. Children should receive strengths based individualized services. 3. Children should receive services in the least restrictive environment. 4. Families should be full, authentic participants in all aspects of the system of care. 5. Services should be integrated with linkages among agencies and programs. 6. Care coordination with collaboration among agencies and families should be provided. 7. Children should be ensured smooth transitions to other programs and services. 8. The rights of children should be protected and promoted. 9. Services should be non-discriminatory and culturally appropriate. 10. Mental health services should use a public health approach with a focus on wellness promotion, prevention, early identification and intervention. 11. Services should be grounded in developmental knowledge with an understanding of the importance of the caregiver-child relationship. 12. Services for young children should be infused into natural settings. Young boy standing and smiling. Page 10 Overview of Project Kariñu Target Population, Purpose, and Goals Project Kariñu’s target population is children ages birth to five years with challenges in their social, emotional, and/or behavioral development and their families. The initiative was initially funded in the fall of 2009 to develop, implement, and sustain an early childhood system of care that is guided by system of care values and principles. It is noteworthy that in addition to the systems of care approach, Project Kariñu also adopted a public health framework. This expanded the scope of the early childhood system of care from strictly focusing on mental health interventions for children with diagnosable behavioral health problems, to include an emphasis on promoting positive social emotional development and preventing mental health problems for all children, including those who are considered “at risk.” Over the seven years of federal funding, Project Kariñu worked to achieve five goals: 1. create and sustain systems change; 2. expand capacity to serve young children and their families; 3. provide a comprehensive array of services and supports; 4. authentically involve families in all aspects of the system of care; and 5. provide family-driven, culturally competent, individualized service planning and care coordination. Context and Statement of Need The Department of Public Health and Social Services (DPHSS) was purposely selected as the lead agency and “home” for Project Kariñu. It is the primary entry point into Guam’s service system for families with young children. Families apply for birth certificates at DPHSS and many routinely access well baby visits through the Community Health Centers. Nearly a third of Guam’s population receives some sort of public assistance administered by DPHSS. By locating Project Kariñu within DPHSS, families had easy access to early childhood mental health services in a non-stigmatizing setting. At the time Project Kariñu was initially funded, few people on Guam were aware of the importance of early childhood mental health and no formal mental health services existed for children under the age of five. A survey of mental health providers conducted prior to the grant application revealed that only a few had knowledge of early childhood development and/or would consider working with young children. Yet, many children under the age of five were in need of services or were considered “at risk” due to exposure to poverty, maltreatment, parental health or behavioral health problems and substance abuse, domestic violence, etc. Young boy holding a toy panda. Page 11 Overview of Service Delivery and System Change Strategies As a system of care, Project Kariñu is a fundamentally a systems reform initiative. While providing direct services to children and families is clearly part of its mandate, its focus is much broader and includes working to create changes in policies, funding streams, workforce development, and infrastructure. Through its service delivery component, Project Kariñu developed and implemented a continuum of services and supports, specifically geared to young children’s social emotional development and mental health. They provided: . screening of all children related to developmental and social emotional milestones; . parent education, training, and family strengthening to support “at risk” children; . evaluation, individualized service planning, wraparound and care coordination for those children with more significant needs; and . individualized clinical services to improve parent and child well-being. Project Kariñu also worked to bring about systemic change through collaboration with Guam’s Early Childhood Comprehensive System (including the Guam Early Learning Council and Project Tinituhon, the Guam System of Care Council, and Guam’s System of Care Expansion grant). Systems change efforts focused on: . increasing the capacity of Guam’s workforce to meet young children’s social, emotional, and behavioral needs; . development of cross-agency referral and service delivery protocols; . increasing awareness of Culturally and Linguistically Appropriate Services (CLAS) standards and related practices; and . developing the infrastructure, including local funding mechanisms, to ensure Project Kariñu’s sustainability. Timeline of Implementation of Strategies and Key Milestones Federal funding for Project Kariñu began on October 1, 2009 and the first 18 months of the grant were devoted to planning and establishing the management structure and service delivery protocols. On June 1, 2011, Project Kariñu officially opened its doors and began conducting outreach activities and providing direct services to children and families. A timeline depicting the implementation of core strategies and key milestones achieved from October 2009 – September 2016 is provided on the following pages. Project Kariñu staff. Page 12-13 Timeline: Key Milestones October 2009-September 2016 2009 Guam received CMHI grant award to fund Project Kariñu (Loving Our Babies), Guam’s early childhood system of care for children birth – 5 years of age •Acting Project Director appointed by DPHSS Director •Contracts formalized between DPHSS & Guam CEDDERS for CLC, Training, & Evaluation components •Logic Model workgroup established •4 trainings & 1 technical assistance activity conducted 2010 •Strategic Planning Summit & follow up session •Project Kariñu office established •Workgroups established & strategic plans drafted •Project Director, Lead Wrap Coordinator, Administrative Officer, Social Marketing Coordinator, 2 Nurse Aides, Family Partner, Attachment Specialist, and Clinical Director hired •Social marketing began •Referral & service delivery protocols developed •Universal Referral Form drafted •1 training conducted •Staff attended national conference 2011 •2 Family Partners, Nurse Aid, and 2 Wrap Coordinators hired •Logic Model, Cultural and Linguistic Competency, and Social Marketing Plans completed by workgroups •Community Evaluation Advisory Team (CEAT) formed •Meet and Greet events conducted to facilitate collaboration and service coordination •5 trainings conducted •3 presentations conducted locally •Staff attended national conference Service Delivery Activities for 2011 • Outreach activities conducted • Project Kariñu began service delivery: assessment, service planning, care coordination, wraparound, & clinical services • Family needs assessment focus groups conducted 2012 •Lead Family Partner hired •Governance structure revised •MOU: Development of Universal Referral Process •MOA: Nene Directory •Project Kariñu website launched •SAMHSA Site Visit •Participation in National Children’s Mental Health Awareness Day activities in collaboration with I Famagu’on-ta •12 trainings & 1 technical assistance activity conducted •Staff attended national conferences •Staff presented at national conference Service Delivery Activities for 2012 • Meet, Greet, and Mobilize family event • Incredible Years program implemented at the Alee Shelter • Peer Family Support Group began • Assessment, service planning, care coordination, wraparound & clinical services Mother and son smiling at each other. 2013 •Wrap Coordinator hired •2 Wrap Coordinators completed requirements for clinical licensure •Universal Referral & Intake Form Standard Operating Procedures developed through cross-agency collaboration •Participation in National Children’s Mental Health Awareness Day activities in collaboration with I Famagu’on-ta •CLC Coordinator helped establish CLASP, a coalition to promote language access services •3 MOAs executed to enhance collaboration •System of Care Assessment Site Visit •Publication of annual evaluation brief •9 trainings conducted •Staff attended national conference Service Delivery Activities for 2013 • 11 outreach activities • Nene Directory disseminated to families • 8 New Family Orientation sessions • 7 Peer Family Support Group meetings • 7 Play Dates & Family Respite Days • 2 Parent Cafés • Assessment, service planning, care coordination, wraparound & clinical services 2014 •Early Childhood Summit, Part I •CLASP Strategic Planning •MOU: Universal Referral and Intake Process •SAMHSA Site Visit •Legislative Round table •Publication of annual evaluation brief •1 Wrap Coordinator completed requirements for clinical licensure •11 trainings conducted •Staff attended national conferences •1 presentation conducted locally •Staff presented at national conference Service Delivery Activities for 2014 • 7 New Family Orientation sessions • 5 Peer Family Support Group meetings • 3 Parent Cafés • 7 Play Dates & Respite days • Assessment, service planning, care coordination, wraparound & clinical services • Early childhood mental health consultation in child care & classroom settings 2015 •Early Childhood Summit, Part II •Project Kariñu Stakeholders Summit •Project Kariñu Project Director sworn in as member of GSOCC •DPHSS Early Childhood Strategic Planning Retreat •System of Care Assessment Site Visit •Publication of Notice of Free LAS Poster •Completion of draft CLC Language Access Plan & CLC Policies and Procedures •Publication of annual evaluation brief & Family Support Activities Report •Local funding included in approved FY 2016 DPHSS budget •Project Kariñu received MagPro award •9 trainings conducted •Staff attended national conference Service Delivery Activities for 2015 • 15 outreach activities conducted • 6 New Family Orientation sessions • 2 Peer Family Support Group meetings • 4 Parent Cafés • 4 Play Dates • Power of Play • Assessment, service planning, care coordination, wraparound & clinical services Service Delivery Activities for 2015 • 15 outreach activities conducted • 6 New Family Orientation sessions • 2 Peer Family Support Group meetings • 4 Parent Cafés • 4 Play Dates • Power of Play • Assessment, service planning, care coordination, wraparound & clinical services 2016 •Legislative Round table •Publication of “Beyond Mandates: LAS on Guam” •Implementation of LAS Survey •Publication of “Let’s Play” •Pacific Daily News supplement featuring Project Kariñu •Cultural Conversations •Chamorro translation of a Behavioral Health Glossary •Publication of annual evaluation brief & Project Kariñu Final Evaluation Report •Local funding included in FY 2017 DPHSS budget •Project Kariñu Stakeholder Evaluation Summit •7 trainings conducted •Staff attended national conferences Service Delivery Activities for 2016 • 14 outreach activities • 4 New Family Orientation sessions • 2 Peer Family Support Group meetings • 1 Parent Café • Power of Play • Assessment, service planning, care coordination, wraparound & clinical services Page 14 Overview of the Evaluation Design and Methodology Approach and Evaluation Questions The evaluation component of Project Kariñu was designed and implemented by Guam CEDDERS Evaluation Team using a participatory approach. Throughout the evaluation, a Community Evaluation Advisory Team (CEAT) comprised of Project Kariñu staff, families, and representatives from Guam’s child serving agencies and non-profit organizations provided input into all aspects of the evaluation. This included reviewing the evaluation plan and data collection instruments, identifying local evaluation needs, reviewing and interpreting data, and providing input into evaluation reports. The Project Kariñu Evaluation Plan focused on evaluation questions that were of interest to SAMHSA, Project Kariñu staff, and the community. A variety of local evaluation activities were designed and implemented, in conjunction with data collection for the national evaluation, to answer these questions. Evaluation Questions 1. What are the characteristics of children and families served by Project Kariñu? 2. Are children and families improving and in what ways? 3. What services did children and families receive through the early childhood system of care? 4. What are families’ experiences with Project Kariñu? Are they satisfied with the services they received? 5. To what extent, and in what ways, have system level outcomes been achieved? Five young boys smiling. Page 15 National Evaluation All system of care sites funded by the Children’s Mental Health Initiative (CMHI) are required by federal law to participate in the national evaluation. The national evaluation is comprised of multiple studies and provides information to Congress on impact of the CMHI program. For over 20 years, findings from the national evaluation have been instrumental in demonstrating the importance of systems of care, resulting in 300 system of care grants and cooperative agreements being awarded to states, counties, territories, and tribal communities. Project Kariñu participated in four national evaluation studies: (1) the Cross Sectional Descriptive study; (2) the Longitudinal Child and Family Outcomes study; (3) the Service Experience study; and the (4) System of Care Assessment study. Guam CEDDERS assumed primary responsibility for data collection and management of the Cross Sectional Descriptive, the Longitudinal Child and Family Outcome, and the Service Experience studies, and facilitated data collection for the System of Care Assessment in collaboration with Project Kariñu and ICF International, the federal contractor for the national evaluation. Cross Sectional Descriptive Study The purpose of the Cross-sectional Descriptive study was to describe the characteristics of children served by the system of care. All families whose children were referred to Project Kariñu were eligible to participate. Between June 2011 and February 2016, 324 families were eligible, and a total of 315 agreed to participate in the study. Data for the Cross Sectional Descriptive study was collected using the Enrollment and Demographic Information Form (EDIF). Longitudinal Child and Family Outcome and Service Experience Studies The Longitudinal Child and Family Outcome study assessed changes in children’s strengths, social emotional development, and behavioral concerns over time. The study also assessed characteristics of families, living situations, and outcomes related to caregiver stress. The Service Experience study provided information on the number and kinds of services children and families received through the system of care, caregivers’ satisfaction with services, and their perceptions of cultural competency in service delivery. Only caregivers of children who were eligible for Project Kariñu system of care services participated in these studies. Of the 261 eligible caregivers who were invited to participate, 206 agreed to be interviewed. Data for these studies was collected by the Guam CEDDERS Family Interviewers using a standardized interview protocol of 8 to 12 data collection instruments depending on the age of the child. Families were first interviewed within 30 days of enrollment into Project Kariñu (baseline) and then every six (6) months for up to 24 months. Family Interviewers met one-on-one with children’s primary caregivers in locations and at times that were convenient for them, including in their homes and in the evenings and weekends. Interviews were conducted in English or through an interpreter, depending upon the caregivers’ needs and preferences. Interviews were conducted from June 2011 – August 2016. Over the course of the evaluation, the Guam CEDDERS Evaluation Team completed: Interview Type Number Completed Baseline: 206 6 month follow-up: 167 12 month follow-up: 129 18 month follow-up: 97 24 month follow-up: 75 Young boy wearing red polo shirt and blue jeans. Page 16 Local Evaluation Activities Local evaluation activities were planned in collaboration with CEAT and provided additional data to answer the key evaluation questions. They utilized qualitative methods to provide opportunities for families, and others, to share their experiences in their own words, as well as locally developed surveys. Caregivers provided information about their service experiences during two focus groups and an evaluation café. Focus groups were conducted in February 2013 with a total of 12 participants. The evaluation café was conducted in August 2015 with 31 additional caregivers. The Evaluation Team collaborated with Project Kariñu to gather additional data on family experiences related specifically to family support activities (i.e., New Family Orientation, Peer Family Support Groups, and Parent Cafés). After these events, families were asked to complete surveys which assessed their satisfaction with the events and perceived outcomes. Data was collected from 2013 – 2016 for a total of 32 events. To gather information about collaboration and the impact of Project Kariñu at a systems level, stakeholders with a history of collaboration with Project Kariñu participated in structured interviews. A total of six program managers and administrators from child serving programs were interviewed in August and September 2015. Additionally, the Guam CEDDERS Evaluation Team developed and implemented evaluation surveys for all Project Kariñu sponsored training activities. Data Analysis and Reporting Data analysis for the national evaluation studies was conducted by ICF International and reported three times a year through Data Profile Reports (DPRs). The Evaluator reviewed the DPRs to identify any potential data errors and interpreted the data within the local context. Data from local evaluation activities was analyzed by the Guam CEDDERS Evaluation Team. Local analysis of data collected for the national evaluation studies was also conducted by the Evaluation Team to address specific requests by Project Kariñu staff and interests of the community and to compare special cohorts of children (i.e., children in the child welfare system and those with physical health problems). Findings from both national and local evaluation activities were regularly reported to Project Kariñu staff, CEAT, and other stakeholders through localized presentations and reports. Beginning in 2013, the Guam CEDDERS Evaluation Team developed, and widely disseminated, a series of annual evaluation briefs summarizing key findings. Two mothers holding their children in their laps. Page 17 Summary of Evaluation Findings Project Kariñu worked to achieve positive outcomes for children and families and systems change. Key outcomes are highlighted below, followed by a more in-depth presentation of the findings and supporting data. Characteristics of Children and Families Served by Project Kariñu •From June 2011 – July 2016, a total of 324 children were referred to Project Kariñu •Most children (nearly 50%) were referred by their caregivers •Children were referred for a wide range of presenting problems •76% of children were Chamorro or Chamorro mix •Over a third of children were involved with Child Protective Services at intake •Many families face multiple, complex challenges and have family histories which put them and their children at risk Service Utilization •94% of caregivers participated in the development of their individualized service plan •At intake, families were receiving, on average, 3 types of services •Children and families enrolled in Project Kariñu used both mental health and support services; support services were more widely accessed •From 2013 – 2016, families participated in a total of 67 Project Kariñu family support activities •Nearly 200 caregivers participated in Parent Cafés which are designed to build family resiliency Child and Family Outcomes •Over 1/3 of children showed significant improvement in their acting out and holding in behaviors •Over time, fewer children displayed challenging behaviors in all areas •Caregivers learned new parenting skills •Nearly 1/3 of caregivers reported significantly less stress after 12 months •Most families are satisfied and reported positively about services Systems Change Outcomes •A total of 53 local trainings were offered for providers •3 social workers gained licensure as clinicians with specific training in diagnosis and clinical interventions for young children •Collaboration increased and partnerships strengthened •49 interpreters increased their skills through Behavioral Health Interpreter trainings •Project Kariñu developed a Language Access Plan and cultural and linguistic competency (CLC) policies and procedures •Local funding for Project Kariñu included in the FY2016 and FY2017 DPHSS local budgets Page 18 Characteristics of Children and Families Served by Project Kariñu From June 2011 – February 2016, a total of 3241 children were formally referred to Project Kariñu and eligible to participate in the evaluation. Of those eligible for the evaluation, descriptive data was collected from 315 families and is reported below. Project Kariñu touched the lives of many more children and families through its outreach events and public awareness activities. Referral Sources Fifty percent (50%) of children were referred to Project Kariñu by their caregivers. Concerns about children’s development, adjustment, and/or behavior problems were common reasons why families sought help. Bar graph: Most children referred by Caregivers Caregivers 50% Child Protective Services 22% Head Start 9% Primary Health Providers 8% Mental Health Providers 4% Other 4% Other Department of Education Programs 3% Demographic Characteristics of Children Referred The majority referred were boys (63%). Children as young as two months through six years of age were referred to Project Kariñu with the average age being 3.2 years. Pie Chart: More Boys than Girls Boys 63% Bar graph: Children Ages 2 Months – 6 years Under a year old 10% 1 to under 3 years old 20% 3 to under 5 years old 45% 5 years and older 25% Footnote 1. Data is only reported for children and families who completed the referral process and for whom documentation on the Enrollment and Demographic Information Form (EDIF) was complete. As a result, the actual number of children referred is larger than reported here. Young boy holding a flower. Page 19 The majority of children referred were of Chamorro or Chamorro mix ethnicity (76%) with the next largest groups being Filipino/Filipino mix (9%) and Chuukese (7%). Based on the 2010 Census, Chamorro/Chamorro mix children are overrepresented and Filipinos are underrepresented. Bar graph: Most Children Were Chamorro/Chamorro Mix Chamorro or Chamorro Mix 76% Filipino or Filipino Mix 9% Chuukese 7% Other race/ethnicity 8% Presenting Problems Children were referred to Project Kariñu for a wide range of presenting problems and the ten most frequent problems are depicted in the graph below. Because children were often referred for more than one problem, percentages exceed 100%. Additional concerns not reflected in the graph were: anxiety, conduct problems, sleeping and feeding problems, school performance, separation problems, and others. It is noteworthy that six (6) children were referred to Project Kariñu as a result of having been excluded from preschool/childcare due to their behaviors. Bar graph: 10 Most Common Problems Specific Developmental Disabilities 42% Adjustment 36% Disruptive Behaviors 28% Maltreatment 27% Hyperactivity/Attention 27% Parental Mental Health 19% Other Problems of Child’s Health 15% Pervasive Developmental Disability 14% Other Parental/Family Problems 12% Excessive Crying/Tantrums 8% Young boy hugging his mother. Young girl smiling. Page 20 Other Agency/Program Involvement Older children in systems of care typically receive services from multiple agencies due to the complexity of their needs. While this appears to be true for some families of young children in Guam’s early childhood system of care, it should be noted that for the majority of families, Project Kariñu was their first point of contact with the service delivery system. At the time of initial referral, families were most frequently involved with: •Child Protective Services (32%); •Department of Education programs (28%); and •Primary care providers (25%). Upon enrollment into Project Kariñu, 40% or, 84 children, ages 3 years and older were participating in an early education programs, the majority of which were either enrolled in Head Start or a preschool program2. Only 17 children had Individual Educational Plans (IEPs), with 11 of these children receiving services through the Department of Education Preschool Special Education Program. When asked the reasons for children’s IEP, only two caregivers responded that special education was provided to address their children’s behavioral and/or emotional problems. Most children’s IEPs targeted speech impairments. About half (51%) of the children enrolled in Project Kariñu had a routine physical health examination in the 6 months prior to their involvement with the program. Since wellchild visits are opportunities for parents to get information about their children’s development, this provides support for the importance of Project Kariñu’s outreach efforts and its presence in Guam’s Community Health Centers. Caregivers also reported that almost a third (27%) of children’s regular activities had been disrupted due to a reoccurring or chronic health problem. At Referral, 1/3 of children were involved with Child Protective Services Of these children, 43% came from homes under investigation for child abuse and/or neglect. 47% were living in court ordered out-of-home placements Footnote 2. Data related to children’s participation in early education was collected only on children who, upon completion of the referral and intake process, were 3 years of age and determined eligible for further services from Project Kariñu, and whose caregivers agreed to participate in the Longitudinal Child and Family Outcome study. Young boy with paint on his face and hands. Young boy making a silly face. Page 21 Children in Guam’s Child Welfare System Nearly one-third (32%) of all children referred to Project Kariñu were involved with Child Protective Services (CPS) at intake. A special analysis of the data was conducted to compare the characteristics of these children to those who were not in the child welfare system. Not surprisingly, significant differences were evident in the children’s presenting problems. In addition to maltreatment, presenting problems for children with child welfare involvement focused on their caregivers’ mental or physical health and family problems. Children who were not involved with CPS were more likely to have presenting problems that related to their own development and self-regulation: hyperactivity and attention; developmental disabilities; and disruptive behaviors. Other findings included: over twice as many children in the child welfare system (58%) came from families with a history of substance abuse and significantly more of these children also: witnessed domestic violence (39%); lived with someone who committed a crime (31%), and experienced sexual assault (11%). Differences between the two groups for having experienced physical assault were not significant; both children in the child welfare system (16%) and those who were not (11%) had been victims of abuse. Also, both groups total protective factor scores and total problem behaviors scores at intake fell within the same range. Children in the child welfare system were less likely to be enrolled in Head Start, preschool special education, and early care settings. Family Characteristics and Risk Factors Families served by Project Kariñu reflect the diversity of Guam’s island community. They came from different cultures, economic conditions, and educational backgrounds. Most children lived with their biological families (83%), and this includes children who lived with one or both of their biological parents, as well as grandparents and other relatives. With regard to custody and guardianship, the majority of children’s biological parents had legal custody (77%), but 12% were wards of the state. Given the age of children enrolled in the initiative, as well as local cultural values related to family and children, this was an unexpected finding and reinforces the need for programs such as Project Kariñu which address the needs of Guam’s most vulnerable children. Higher levels of parental education are often associated with positive outcomes for children and greater access to resources which support healthy development. Caregivers of children enrolled in Project Kariñu ranged in their levels of educational attainment, but a significant number (29%) had not completed high school or their GED. Bar graph: Nearly 1/3 Caregiver Did Not Finish High School College Degree 18% Associate Degree/Some College 22% High School Diploma or GED 31% Below High School 29% Young girl putting on sunglasses. Smiling boy on the playground. Young girl crying with hands on her face. Page 22 Family Income The average household size for Project Kariñu families was six (6) family members, inclusive of adults and children. According to the Guam Bureau of Statistics and Plans, the poverty threshold for a family of six is $29,405. Forty-seven percent (47%) of Project Kariñu families reported annual incomes below $25,000, placing many children at risk due to economic conditions. Table Header: Family Income Less Than $5,000 8.7% $5,000–$9,999 5.1% $10,000–$14,999 12.8% $15,000–$19,999 10.8% $20,000–$24,999 9.7% $25,000–$34,999 15.9% $35,000–$49,999 17.4% $50,000–$74,999 8.2% $75,000–$99,999 4.1% $100,000 and Over 7.2% Risk Factors Many families served by Project Kariñu face multiple complex challenges. Caregivers reported problems related to: accessing healthcare, housing, legal issues, economic conditions, unavailability of social services, transportation, and others. These types of challenges, along with family and child histories, can place children at increased risk for disruptions in their social emotional development and mental health. Upon enrollment into Project Kariñu, caregivers were asked about their family and child histories and many children experienced significant risk factors. Table header: percentage of Children with Risk Factors Lived with someone who was depressed 43% Witnessed domestic violence 27% Lived with someone with a substance abuse problem 27% Lived with someone convicted of a crime 21% Lived with someone w/mental illness other than depression 18% Ran away 18% Experienced physical assault 13% Experienced sexual assault 5% Mothers hugging her children. Page 23 System of Care Eligibility Project Kariñu elected to expand system of care eligibility criteria beyond what is standard practice for systems of care serving older children and youth. Rather than just serving children with diagnosable mental health problems, they also served children birth – 5 years who were considered at “imminent risk.” This was consistent with their adoption of a public health framework with its focus on the promotion of positive mental health and prevention. Seventy-six percent (76%) of children were eligible for system of care services due to a socio-emotional, behavioral, or mental health disorder and 24% were considered at “imminent risk3.” The most frequent DSMIV Axis I and Axis II diagnoses are presented below. Because children may have more than one diagnosis, percentages sum to more than 100%. Table header: Diagnosis. Percentages. Other: Neglect, Sexual Abuse & Physical Abuse of Child 39.8% Adjustment Disorders 35.5% Pervasive Developmental Disorders 34.4% V code[c] 28.9% Attention-Deficit/Hyperactivity Disorders 22.7% Learning, Motor Skills, and Communication Disorders 19.4% PTSD and Acute Stress Disorder 4.7% Mental Retardation 4.0% Disruptive Behavior Disorder 2.3% Anxiety Disorders 2.3% Young boy wearing a cultural headdress. Young boy holding a soccer ball. Footnote 3. Children were diagnosed using the DSM-IV or DSM-V or its ICD-9-CM equivalents, or the Diagnostic Classification of Mental Health and Developmental Disorders of Infancy and Early Childhood (DC:0-3R). Children were determined at imminent risk through an assessment using the Ages and Stages Questionnaire (ASQ-3) and Ages and Stages Social Emotional Questionnaire (ASQ-SE). Page 24 Service Utilization In systems of care, children and families should also have access to a comprehensive array of community-based services. System of care services include traditional mental health services, such as assessment or evaluation and clinical interventions. They also include what are known as support services. Case management or care coordination, peer-to-peer support, parent education and training, transportation, and respite are examples of support services. The actual services children and families receive are determined by the goals of their individualized service plan and families should have an active role in developing the plan. Ninety-four percent (94%) of Project Kariñu caregivers participated in the development of their individualized service plan. During interviews for the national evaluation, caregivers are asked what types of services they received from the system of care. At intake, families reported receiving, on average, 3 types of services. Children and families enrolled in Project Kariñu used both mental health and support services, although support services were, by far, more widely accessed. This is most likely due to the age of the children receiving services and nature of their presenting problems and diagnoses, as well as the fact that Project Kariñu did not limit services to children with diagnosable mental health disorders. For those families with young children who were considered “at risk,” parent training, peer-to-peer support, and home visitation, all types of support services, were ways of promoting healthy development and preventing mental health problems. Table: Type of Services Used Intake 6 Months 12 Months Family Preservation 10% 7% 10% Case Management 37% 71% 66% Family Support 29% 51% 41% Transportation 59% 71% 68% Informal Support 32% 61% 49% Assessment or Evaluation 98% 68% 57% Clinical Services 5% 20% 20% A particular strength noted by families during the focus groups and evaluation café were Project Kariñu’s family support activities. These included: New Family Orientation, Peer Family Support Groups, Play Dates and Respite, and Parent Cafés. From 2013 – 2016, families participated in a total of 67 family support activities. Nearly 200 caregivers participated in Parent Cafés which are designed to build family resiliency. Bar graph Family Orientation 10 Peer Family Support 16 Play Day/ Respite 16 Parent Café 25 Young boy playing with a toy car. Page 25 Child and Family Outcomes Behavioral and Emotional Strengths Although many of the children enrolled in Project Kariñu experienced numerous challenges and risk factors, they also appeared to be resilient, displaying strengths across multiple areas which increased slightly over time. At intake, on the Behavioral and Emotional Rating Scale (BERS), children’s average scores on the Strength Index was 94. This falls within the average range of 90 – 110 for the survey. Children’s behavioral and emotional strengths scores from intake to 12 months are displayed below. Scores from 8 – 12 are considered within the average range. Table Children’s Strengths Increase (Higher Scores Indicate Greater Strengths) Strength Subscale Intake 12 months Interpersonal Strength 7.7 8.6 Family Involvement 9.2 9.8 Intrapersonal Strength 10.1 10.2 School Functioning 7.9 9.0 Affective Strength 9.3 9.8 The Reliable Change Index compares children’s scores on the subscales of the BERS at two different time points and indicates whether the change in their scores show significant improvement, stability or worsening. After 12 months, the majority of children’s scores remained stable and only a small percentage of children’s scores worsened (4% - 11% depending on the subscale). Some children showed significant improvement: 26% in their Overall Strength; 22% in their Interpersonal Strength; 21% in School Functioning; 11% in Affective Strength; 8% in Intrapersonal Strength; and 6% in Family Involvement. Pie chart: Children Showing Improvement Overall Strength Improved 26% Interpersonal Strength Improved 22% Young boy on a swing. Page 26 Behavioral and Emotional Challenges Children with behavioral and emotional challenges are less likely to enter school ready to learn, more likely to be bullied, and less likely to establish positive relationships. Upon enrollment into Project Kariñu, approximately half of all children were exhibiting challenging behaviors within the clinical range on the Child Behavioral Checklist 1 . - 5 (CBCL 1 . - 5). Over time the number of children displaying significant challenging behaviors decreased across all areas. Bar graph: Children’s Challenging Behaviors Decrease Intake Attention problems 28% Withdrawn 40% Sleep Problems 25% Aggressive Behavior 31% Emotionally Reactive 24% 12 Months Attention Problems 20% Withdrawn 19% Sleep Problems 15% Aggressive Behavior 15% Emotionally Reactive 9% After 12 months, about 1/3 of children showed significant improvement in their externalizing or “acting out” behaviors and in their internalizing or “holding in” behaviors. Pie chart: 1/3 of Children Show Improvement Acting Out Behaviors 34% Improved Holding In Behaviors 34% Improved Caregivers Share About Positive Changes in their Children “My child is now listening to me…He came out and shared his emotions…” “My child now interacts with others and is more social when she was not social before.” “No more tantrums. More friends, a better playmate; now peers like him.” Page 27 Family Resiliency and Parenting Skills The quality of the caregiver/child relationship is the foundation for healthy social emotional development and many Project Kariñu families face challenges and risk factors which may compromise their ability to engage in supportive nurturing relationships with their children. Using the Strengthening Families approach which builds the five protective factors during Parent Cafés, Project Kariñu helped families become more resilient and improve their parenting skills. From 2013 – 2016, Project Kariñu conducted 10 Parent Cafés which touched the lives of nearly 200 families. Ninety-nine percent (99%) of caregivers that participated in the Parent Cafés agreed that they learned something new. Most planned to make changes in how they interact with their children based on what they learned. Bar graph: Caregivers Plan to Make Changes I plan to change something about my parenting 98% Agree I plan to change how I discipline my child(ren) 97% Agree I plan to make sure I understand my children's feelings 100% Agree Families Share About Their Parent Café Experiences “Talking to different parents and finding out that you’re not alone in the rocky journey that you’re in. Just brought tears to my eyes. It is an awesome experience.” “I loved and enjoyed the Parent Café! As always, I have learned new ways to work w/my children and new ways to work on myself.” “Parent Café structure is always empowering. I leave feeling I can take on the world!” “I learned that I need to be more flexible. I also saw how I need to be a better listener.” Mother holding her child. Page 28 Smiling father and his child. Family Satisfaction with Services Family satisfaction was assessed as part of the national evaluation and through focus groups, a local survey, and the evaluation café. Consistently across all evaluation activities, the majority of caregivers reported positively about their experiences with Project Kariñu. National Evaluation The national evaluation assessed satisfaction with service experience along multiple domains using the Youth Services Survey for Families (YSS-F). A local survey was also administered to families as part of the focus groups and included similar domains and one additional domain, Interactions with Staff. Table: Caregiver Perspective on Services % Reporting Positively after 12 months (YSS-F) Access to Services 78% Participation in Treatment 83% Cultural Sensitivity 95% Satisfaction with Services 72% Outcome 73% Functioning 76% Social Connectedness 77% Interactions with Staff N/A % Reporting Positively at Focus Groups Access to Services 78% Participation in Treatment N/A Cultural Sensitivity 100% Satisfaction with Services 81% Outcome 83% Functioning N/A Social Connectedness N/A Interactions with Staff 92% Focus Group and Evaluation Café During focus groups and an evaluation café, caregivers shared their experiences related to: (1) accessing services; (2) service quality and satisfaction; and (3) child and family outcomes. As noted above, the vast majority of caregivers reported positively about their experiences across all three areas. Caregiver experiences related to child and family outcomes were incorporated into previous sections of this report and service access and satisfaction is discussed below. Most caregivers talked about how “easy” it was to access services and many noted the helpfulness of Project Kariñu staff and their willingness to be flexible and accommodating to families’ needs. For a very small number of families, their experience was less favorable. One caregiver noted that while the evaluation was done quickly, she did not receive the results in a timely manner and several families expressed frustration over the lack of follow up. In terms of service quality and satisfaction, caregivers described receiving a wide range of services, including: play therapy, service coordination, home visits, outreach, respite, parent education, family support activities, advocacy, support transitioning to other services, and informal supports. Families were extremely positive about the services they received. Very few caregivers expressed dissatisfaction with Project Kariñu and/or reported not receiving services. Those that did, noted the following reasons for their dissatisfaction: frustration about the amount of paperwork, services being discontinued too soon and/or without their Page 29 involvement; lack of transition services; and staff shortages and changes. For families whose children were “aging out” of Project Kariñu, there was concern about what would happen next. Several families suggested extending the age of Project Kariñu. Across the focus groups and evaluation café, caregivers expressed appreciation and gratitude for Project Kariñu staff. One caregiver described Project Kariñu as the “backbone” for her family and went on to share that staff provided supplies to meet basic needs such as diapers and food, as well as training and education, support, information, and resources. Staff were repeatedly described as: caring, supportive, helpful, understanding, flexible, patient, respectful, and helpful. Caregiver Stress Many of the caregivers of children enrolled in Project Kariñu face multiple and complex challenges of their own, in additional to the stress of raising a young child with behavioral health needs. At intake, over half of all caregivers reported levels of parenting stress in the clinical range. During parent cafés, focus groups and the evaluation café, caregivers talked about the stress they feel related to parenting a child with behavioral health needs. One mother shared that “parents are scared about the reality of having a child with a disability.” And another simply stated, “it’s hard.” By implementing a system of care approach and building the protective factors, Project Kariñu helped families address the stress associated with raising their children. Caregivers Share How Project Kariñu Helped “They help families relieve stress.” (I) “am being more patient; being more understanding.” (I) “learned how to adapt to my son’s needs.” “I’m not as reactive as before and I am better able to hand(le) situations in a calmer manner. Pie chart: Nearly 1/3 of Caregivers Reported Feeling Less Stress Global Strain 31% Improved Results from the Caregiver Strain Questionnaire (CGSQ) indicated that after 12 months, nearly 1/3 of caregivers reported significant improvements in their levels of stress associated with raising a child with emotional and behavioral challenges. One Caregiver Shared (Project Kariñu) “helps all families no matter what nationality.” She went on to say it was the first time accessing services that she did not feel discriminated against. Mother playing with her son with a sock puppet. Page 30 Systems Change Outcomes To achieve systems change outcomes, Project Kariñu partnered with Guam’s child serving agencies and programs and actively participated in the governance/advisory groups for both the early childhood system and the system of care for older children and youth. Interviews were conducted with stakeholders from these groups to learn more about the impact of Project Kariñu and identify positive systems change outcomes, unmet needs, and priorities for continued system building. Stakeholders agreed that Project Kariñu had achieved the following positive system change outcomes: •increased community awareness of the importance of early childhood mental health; •increased awareness of cultural and linguistic competence; •increased capacity of Guam’s workforce to meet the needs of young children and their families; and •improved collaboration and service coordination across child serving programs. Increased Community Awareness Stakeholders unanimously agreed that Project Kariñu is highly visible in the community and has increased awareness through its outreach activities and social marketing. “Project Kariñu has been proactive in educating the public about system of care and early childhood mental health.” Consistently, those interviewed commented positively about breadth of Project Kariñu’s outreach and their collaboration with other programs during outreach activities. One person shared: “Project Kariñu is at every outreach and their presence is a plus.” Many of those interviewed acknowledged the range (e.g., radio and television commercials, pamphlets, posters, etc.) and quality of social marketing materials. Increased Awareness of Cultural and Linguistic Competence Cultural and linguistic competency (CLC) is a core system of care value and Project Kariñu achieved positive CLC outcomes within its service delivery component and at a systems level. Under the leadership of its CLC Coordinator, Project Kariñu was the first early childhood program on Guam to have a Language Access Plan and formal CLC policies and procedures. Numerous trainings (including training on the Culturally and Linguistically Appropriate Services standards) and CLC awareness activities were sponsored, co-sponsored and/or conducted by Project Kariñu. In 2012, the first local Behavioral Health Interpreter training was conducted and a second training for interpreters was conducted in 2014. To date, a total of 49 interpreters were trained in providing language access services in behavioral health settings. A listing of trained interpreters is now part of the NENE Directory, a directory of services which is widely disseminated throughout the community. During the interviews, stakeholders spoke positively about Project Kariñu’s CLC component, specifically identifying having a list of trained interpreters, translated materials, training for staff, and trained interpretersas positive outcomes. One person shared, “Project Kariñu brought CLC understanding to agencies” and another said the “training made us look at everything we do and how we do it.” Group posing for a photo. Page 31 In 2013, the CLC Coordinator helped establish CLASP (Culture and Language Access Service Partners), a coalition to promote language access services. Through CLASP, a Language Access Services (LAS) selfassessment was developed and administered to agencies and programs; an Executive Order requiring agencies to develop Language Access Plans was established; “Beyond Mandates: LAS in Guam,” was published and disseminated documenting the local issues and strategies used to address LAS on Guam; and the Guam Community College was supported in establishing nationally trained local LAS trainers. At this writing, Chamorro Language Specialists are working on translations for a Behavioral Health Glossary in Chamorro spearheaded by Project Kariñu. Throughout the term of the grant, Project Kariñu engaged in numerous other activities to advance culturally responsive service delivery. These included Cultural Conversations to learn more about different cultures' views related to behavioral health and early childhood concepts as well as the development of language access products which helped to acknowledge diversity of languages and cultures among those served. As one stakeholder shared, “Project Kariñu is very sensitive to cultural beliefs and practices.” Increased Capacity of Guam’s Workforce Project Kariñu accomplished a great deal in the area of workforce development beginning in 2009 – 2010, when five trainings were conducted to build awareness of the goals of the grant and the system of care approach primarily within DPHSS and Project Kariñu’s newly hired staff. Then in 2011, Project Kariñu began actively collaborating with other initiatives and child serving programs to systematically increase the capacity of Guam’s workforce. Through joint planning, braiding of funds, a commitment of in-kind resources, and a shared vision, a total of 53 local trainings were offered for providers from 2011 – 2016. All stakeholders interviewed agreed that increasing local training opportunities and the cross agency training of staff were significant outcomes. One administrator noted that the cross-agency trainings increased collaboration and coordination between programs and supported the actual implementation of training content. Providers received training in three key areas related to the early childhood system of care: (1) use of validated screening and assessment tools (ASQ-3, ASQ:SE, and the DC:0-3R), (2) best practice approaches to service delivery (system of care approach, wraparound, and cultural and linguistic competency), and (3) evidence based and evidence-informed practices (Incredible Years, Pyramid Model/CSEFEL, and Strengthening Families, play therapy). A timeline of training activities supported by Project Kariñu from 2011 to 2016 appears on the following page. Project Kariñu also provided opportunities for students enrolled in higher education programs to participate in field experiences and provided supervision for those seeking clinical licensure. Through partnerships with Guam Community College, the University of Guam, and an off island institution, 74 students participated in field experiences or internships with Project Kariñu. Working under the supervision of the Clinical Director, eight (8) providers received clinical supervision and three (3) social workers achieved licensure as mental health clinicians with specific training in diagnosis and clinical interventions for young children. Smiling family. Mother with her two children. Page 32 Project Kariñu Loving Our Babies Training Timeline: October 2011-August 2016 2011 •Ages & Stages Questionnaire, Version 3 (ASQ-3) •Ages & Stages Questionnaire: Social Emotional (ASQ-SE) •Incredible Years Basic Preschool Modules •Incredible Years Infant & Toddler Modules •Pyramid Model (CSEFEL): Infant & Toddler Modules 2012 •Ages & Stages Questionnaire: Social Emotional Online •Behavioral Health Interpreter Training •Cultural & Linguistic Competency Awareness •Cultural & Linguistic Competency & Language Assistance Services •Incredible Years refresher training •Strengthening Families: Parent Café •Wraparound: 6 sessions for Staff, Families & Stakeholders 2013 •Behavioral Health Interpreter Training •Building an Early Childhood System of Care Planning Retreat •Early Childhood Mental Health: Diagnostic Framework for the Clinician’s Toolkit . •Early Childhood Mental Health Consultation •Pyramid Model (CSEFEL): Train the Trainer Sessions •Strengthening Families: Parent Café •Universal Referral Process •Wellness Coaching: Balancing Body, Mind, & Spirit •Kariñu Wraparound Training Mother with her four children. Page 33 2014 •Access to Language Equality Public Forum •Ages & Stages Questionnaire, Version 3 (ASQ-3) •Cultural & Linguistic Competency & CLAS Standards •Early Childhood Mental Health: Diagnostic Framework for the Clinician’s Toolkit, DC:0-3R •Healthy Moms, Healthy Babies, Train the Trainer •Pyramid Model (CSEFEL): Developing Behavioral Support Plans •Pyramid Model (CSEFEL): Early Childhood Mental Health Consultation •Pyramid Model (CSEFEL): Family Coaching •Pyramid Model (CSEFEL): Positive Solutions for Families •Pyramid Model (CSEFEL): Train the Trainer Preschool Modules •Strengthening Families: Parent Café, Bringing the Protective Factors to Life 2015 •Addressing Disparities through Organizational Cultural & Linguistics Competency (CLC) •Early Childhood Mental Health: Diagnostic Framework for the Clinician’s Toolkit, DC:0-3R •Pyramid Model (CSEFEL): Family Coaching •Pyramid Model (CSEFEL): Parents Interacting with Infants (PIWI) •Pyramid Model (CSEFEL): Positive Solutions for Families •Screening Young Children and Their Families •Social-Emotional Assessment/Evaluation Measure (SEAM) Training •Strengthening Families: Parent Café •Strengthening Families: Train-the Trainer 2016 •Ages & Stages Questionnaire, Version 3 (ASQ-3) •ASQ and SEAM Family Profile Training •Brain Development in Young Children •DC:03R and DSM V Training Part II •Play Therapy in Early Childhood Mental Health •Strengthening Families: Bringing the Protective Factors to Life in Your Work •Strengthening Families: Parent Café Mother with her child, both smiling happily. Page 34 Improved Collaboration Improved collaboration was identified as a positive outcome by all stakeholders who were interviewed. As one person put it, “Collaborating with Project Kariñu means (our program) isn’t doing all the work alone.” When asked about the specific areas where collaboration was most effective, stakeholders identified: outreach, training, referral, and service planning. Programs frequently collaborated to refer children to each other’s programs as well as to improve the referral, screening and assessment process for families that were referred to multiple programs. Most stakeholders identified the development of the Universal Referral and Intake Form (URIF) as a key systems change outcome. Development of the URIF began in 2010 with the formation of Project Kariñu’s service delivery workgroup. Over the next four years the group worked to develop and pilot the URIF. In 2014, it was officially adopted by the Guam Early Learning Council. As one program administrator shared, “The URIF has been really helpful; we don’t have to repeat what was done.” Several stakeholders mentioned the increased ease of making referrals between programs. Several programs noted that for the first time, they had an appropriate place to refer children who were experiencing social emotional and behavioral challenges. Stakeholders also discussed increased collaboration in service planning (i.e., involvement of their staff as members of Project Kariñu’s wrap teams, as well as Project Kariñu’s involvement in the development of their programs’ mandated service plans). Several stakeholders mentioned that while there has been progress in improving collaboration, more work is needed to ensure true care coordination and service integration. One person shared, “Seems like crisis drives more coordination and most cases aren’t at a crisis level….need to identify critical areas for coordination.” Another program manager shared, (We are) “still trying to figure out how to collaborate earlier to avoid separate assessments and service plans.” With respect to collaboration in providing direct services to families, most stakeholders shared that their staff participated in cross-agency teams that sponsored and hosted Parent Cafés to the benefit of both their staff and the families they served. One administrator specifically discussed Project Kariñu’s willingness to collaborate on-site to address crisis situations and two mentioned Project Kariñu staff directly assisting their staff in working with children’s challenging behaviors. Formalized collaboration across agencies is also an indicator of systems change and from 2010 to 2016, Project Kariñu entered into six (6) memoranda of understanding/agreements with other early childhood programs and initiatives. These MOUs/MOAs addressed different aspect of the early childhood system with the goal of increasing access, collaboration, and service coordination. Group posing for a photo. Page 35 Group posing at the Legislature. Sustainability James Heckman, Nobel laureate in economics, estimates the return on investment in early childhood programs is between 6 – 10%. Early intervention promotes school success and completion, reduces crime, and promotes adult wellness; resulting in a healthier, more productive society. As such, sustaining Project Kariñu is an important investment in the children of Guam and in the future of our island community. In 2013, Project Kariñu engaged in strategic planning related to sustainability and began pursuing multiple sustainability strategies. Despite significant fiscal challenges within the Government of Guam, local funding for Project Kariñu was made available in the FY 2016 DPHSS budget and was included again in the FY 2017 DPHSS budget which became law in September 2016. While this is a significant outcome, it should be noted that the program remains vulnerable. Systems of care, traditionally focus only on children with diagnosable behavioral health problems. However, by adopting a public health framework, Project Kariñu expanded the scope of its services to include universal access to screening and wellness promotion activities and targeted access to prevention services for children considered “at risk.” The majority of children served by Project Kariñu fell within these levels of care, making fewer resources available for children with more intensive needs. To address the need for a more balanced allocation of resources as well as sustainability, DPHSS, in collaboration with Guam CEDDERS, submitted a grant proposal to SAMHSA for Project LAUNCH (Linking Actions for Unmet Needs in Children’s Health). In September 2014, DPHSS was awarded a five-year cooperative agreement for Guam LAUNCH and, since then, DPHSS has worked to integrate Project Kariñu and Guam LAUNCH staff and service delivery components. Guam LAUNCH provides developmental (including social emotional development) screening, home visitation, training and support for families, early childhood mental health consultation, with Project Kariñu providing wraparound and more intensive mental health interventions. The funding of Guam LAUNCH is an interim sustainability strategy which will enable DPHSS to continue pursuing solid local funding for the early childhood system of care. Additionally, Project Kariñu collaborated with I Famagu’on-ta and Guam’s Medicaid Office to amend the Medicaid State Plan to include reimbursement for some system of care services. Technical assistance was provided by Georgetown University resulting in draft language for the amendment that included descriptions of system of care services, provider qualifications, and proposed fees. Challenges to billing Medicaid for services remain and are discussed in the next section but the groundwork has been established. Critical to the ability to bill for services, is having a system to collect and track service data. Although not yet fully operational, a ChildLink database is under development which could be used for this purpose. ChildLink is currently being used by other DPHSS programs and the long term goal is to be able to share data for service integration and for interoperability with the Community Health Centers’ electronic health record. Lastly in February 2014, the Guam Legislature passed P.L. 32-120 which designated how revenues from the re-valuation of real property will be allocated across multiple Government of Guam entities. The bill designates 3.5% of the remaining balance (after funds are distributed across five other entities) for I Famagu’on-ta and Project Kariñu. However, to date no funding has been made available and several critical points remain unclear: (1) how much money could potentially be made available, (2) how programs access the funds, and (3) how the 3.5% is to be distributed between I Famagu’on-ta and Project Kariñu. Page 36 Challenges and Recommendations Sustainability At the time of this report, it appears Project Kariñu will be sustained through local funding and the shifting of some service delivery components to Guam LAUNCH. The Government of Guam is facing many fiscal challenges, so it is important that stakeholders remain vigilant to ensure Project Kariñu can actually access the funds that were appropriated and staff are retained. The majority of Project Kariñu staff are unclassified employees. Renewal of their personnel actions is not automatic and is based on the availability of funds. Unclassified employees are often the first to go when budget constraints require downsizing. Significant time and resources have been devoted to developing a competent, skilled workforce and the retention of staff is essential. It is recommended DPHSS work to establish permanent positions within the Government of Guam for Project Kariñu staff. It is also recommended that stakeholders engage policy makers to keep Project Kariñu’s sustainability (and growth) on their agendas. Members of the Guam Early Learning Council, Guam System of Care Council, the Governor’s cabinet and advisors, and Guam Legislature need to be informed about the importance of early childhood mental health and encouraged to commit to investing in Guam youngest citizens. This can be accomplished through annual briefings with policy makers and presentations during meetings of the two Councils. The use of Medicaid funds and private insurance to pay for some system of care services is a common sustainability strategy across system of care sites in the U.S. mainland. However, significant barriers exist for their implementation locally. On Guam, Medicaid funding is capped with a local match of 45%. Given that there is a limited pot of available Medicaid dollars, a high local match, and large percentage of Guam residents who rely on Medicaid for primary care and hospital services, it remains to be seen if Project Kariñu will be able to access Medicaid dollars. To date, little has been done to educate local private health insurance providers about early childhood mental health services and pursue third party reimbursement for assessment and clinical services. It is recommended that Project Kariñu collaborate with the Guam Early Learning Council and Guam System of Care Council to establish a workgroup to further investigate how to access these funding streams. Additionally, efforts need to continue to establish the infrastructure required for billing and accountability. Expand Service Delivery and Eligibility Significant resources have been expended to develop Guam’s workforce and the number of mental health providers with knowledge and skills in early childhood mental health has increased as a result of Project Kariñu. In order to expand service delivery and meet the needs of more children and families, it is recommended that Project Kariñu establish public/private partnerships with qualified providers to further enhance its current workforce. Two children smiling. Page 37 Group posing for a photo at the Governor’s Complex Conference Room. The majority of caregivers expressed satisfaction with services, but some noted frustration and disappointment that they did not receive more services or services over a more extended period of time. There are many possible explanations and families themselves shared that they felt Project Kariñu was challenged by not having enough staff and by staff turnover. This was compounded by the need for Project Kariñu to provide a full continuum of mental health promotion, prevention, and intervention services, rather than just intensive services for children with diagnosable mental health conditions. With the establishment of Guam LAUNCH and its focus on mental health promotion and prevention, Project Kariñu can now devote their resources to providing wraparound and clinical interventions for children with more intensive needs which should result in enhanced service delivery. It is recommended that DPHSS continue to integrate Guam LAUNCH and Project Kariñu to create a comprehensive array of mental health promotion, prevention, and intervention services. This is likely to require some restructuring of staff positions and responsibilities and additional workforce development. The Guam Early Childhood Comprehensive System (ECCS) includes all programs on Guam for young children ages birth to eight years and Project Kariñu was an important addition to the system. However, due to federal requirements at the time of the original grant application, Project Kariñu currently only serves children from birth – five years. When children served by Project Kariñu turn six, they must be referred and transitioned to I Famagu’on-ta, a mental health program for children and youth within Guam Behavioral Health and Wellness Center (GBHWC). During focus groups and interviews, families expressed concern about this and requested that the eligibility age for Project Kariñu be extended. It should also be noted that the target population for Guam LAUNCH is birth – eight years and the other programs that most frequently collaborate with Project Kariñu are those within DPHSS and the ECCS (e.g., child welfare, Special Kids Clinic, Bisita I Familia, Head Start, child care centers, Guam Early Intervention Services, and Preschool Special Education, etc.). To better support children ages six – eight years and build on existing partnerships within DPHSS and the ECCS, it is recommended that eligibility for Project Kariñu be extended to age eight. A workgroup that includes representation by families, DPHSS, GBHWC, the Guam Early Learning Council, and the Guam System of Care Council should be formed to develop an implementation strategy, including an analysis of related staffing and budgetary issues. Regardless of the age of eligibility for Project Kariñu, staff from Project Kariñu and I Famagu’on-ta should collaborate to develop family friendly protocols for transitioning children between programs. Joint transition meetings, formalized transition plans, and monitoring services for children who do not meet I Famagu’on-ta eligibility requirements are recommended strategies that should be explored. Service Coordination and Integration Under the leadership of Project Tinituhon, Project Kariñu and its agency partners have established systems change goals and formed Strategic Management Team (SMT) workgroups to further enhance service coordination and integration. Unfortunately, federal funding for Project Tinituhon ended in 2016 although some support to the SMTs continues under a no cost extension. It remains to be seen if the SMT workgroups will continue without the support provided by Project Tinituhon. It is recommended that Project Kariñu, assume leadership for the SMT for Early Childhood Social Emotional Development and Mental Health and collaborate with workgroup members to address goals related to service coordination and integration. Protocols should be developed to establish universal screening of young children, including screening children ages 6 – 8 years; for the sharing assessment information; and for developing joint service plans with the ultimate goal of “one child, one plan.” Page 38 Data Sources Much of the data for this report was collected using protocols and instruments selected by and/or developed by ICF Macro. Information related to the Characteristics of Children and Families was collected using the: Enrollment and Demographic Information Form (EDIF); the Caregiver Information Questionnaire (CIQ); and the Education Questionnaire – Revised 2 (EQ-R2). Data related to Child and Family Outcomes was collected using the: Child Behavioral Checklist (CBCL 1 . - 5); the Behavioral and Emotional Rating Scale-Second Edition, Parent Rating Scale (BERS-2C) and the Preschool Behavioral and Emotional Rating Scale (PreBERS); and the Devereaux Early Childhood Assessment (DECA-C); the Caregiver Strain Questionnaire (CGSQ); and Youth Service Survey for Families (YSS-F). Service Utilization data was collected using the Multi-Sector Service Contacts Revised (MSSC-R). Data analysis support was provided by Dr. Ranilo Laygo. Locally developed surveys and qualitative data was used to supplement data collected for the national evaluation. More information about these data sources is available by contracting Bonnie Brandt at bonnie.brandt@guamcedders.org. References Heckman, J.J. (2008). Return on Investment: Cost vs. Benefit. University of Chicago. Retrieved from http://www.heckmanequations.org. Knitzer, J. (1982). Unclaimed Children: The failure of public responsibility to children and adolescents in need of mental health services. Washington, DC: Children's Defense Fund. National Council on the Developing Child (2007). The Science of Early Childhood Development. Retrieved from http://www.developingchild.net. Shonkoff, J.P. & Phillips, D.A. (Eds.) (2000). From Neurons to Neighborhoods: The Science of Early Childhood Development. Washington, DC: National Academy Press. Stroul, B.A. & Friedman, R. M. (1986). A System of Care for Children and Youth with Severe Emotional Disturbances (rev. ed.). Washington, DC: Georgetown University Child Development Center, National Technical Assistance Center for Children’s Mental Health. Thompson, R. (2008). Investing in Our Future. Unpublished data. Page 39 A full-page photo of a family of thirteen smiling. Page 40 Project Karinu Final Evaluation Report 2009- 2016 ©2016 University of Guam CEDDERS Project Kariñu Final Evaluation Report Bonnie Brandt, M.A. and Keith Villaluna, B.B.A. University of Guam Center for Excellence in Developmental Disabilities Education, Research, and Service (Guam CEDDERS) Office of Graduate Studies, Sponsored Programs, & Research UOG Station Mangilao, Guam 96923 One hundred percent (100%) funding for this publication was provided by the Child Mental Health Initiative (CMHI) Cooperative Agreement 5U79SM059022-06 between the U.S. Department of Health and Human Services/Substance Abuse and Mental Health Services Administration/Center for Mental Health Services and the Guam Department of Public Health and Social Services. The University of Guam is an Equal Opportunity Employer and Provider.