Photo by Kate Lord
MASN coordinates a variety of data collection and evaluation services for afterschool programs. Although the framework was designed for 21st CCLC and School Age Community (SAC) grantees, the framework has been used with the St. Louis Mental Health Board’s Afterschool for All Partnership sites. The data collection and quality improvement tools can be used for individual sites.
History
In 2012, MASN contracted with Dr. Wayne Mayfield at the Office of Social and Economic Data Analysis (OSEDA) to develop an evaluation plan for measuring the impact of the Network’s training and technical assistance provided to the 21st CCLC grantees. The report “Recommendations for Program- and Site-Level Data Collection” outlined the need for more standardization in the local evaluations, a consistent observable measure of program quality, and a way of measuring youth outcomes related to college and career readiness.
Also in 2012, MASN’s Quality committee and the statewide Program Assessment Scoring/System (PASS) committee met jointly and selected the Program Quality Assessments (PQAs) as the instruments to be used for state-funded quality improvement efforts. During this time period, the Quality committee continued to work with Dr. Mayfield to create a logic model and “best practice” survey recommendations for all afterschool programs throughout the state. When the workgroup became aware of the Weikart Center’s Leading Indicators framework, MASN partnered with OSEDA and the Weikart Center to develop one set of afterschool surveys that contained both the Leading Indicators and MO College and Career Readiness questions. Meanwhile, DESE and MASN began conversations about using these new resources to revise the 21st CCLC Statewide Evaluation process.
In 2013 and 2014, MASN, DESE, and OSEDA worked closely to develop the new evaluation framework (MO Goals and Objectives) that would apply to both the statewide and local external evaluations. Although the data collection and evaluation system was first designed to meet the needs of the 21st CCLC and SAC programs, the it can be used in part or whole with any afterschool program.
Data collection started in the 2013-14 academic year and the first 21st CCLC Statewide Evaluation report was completed by OSEDA and the Weikart Center for Youth Program Quality. In 2014-15, the Local External Evaluation system changes went into effect and the grantee-level evaluations for new 21st CCLC Cohorts were parallel to the state evaluation. In addition to being used for the statewide and local evaluations, the same data PQA and survey data was also provided back to the sites during Planning with Data sessions to create site-level Quality Action Plans.
Using the Data at All Levels
As we developed this data collection and improvement framework, we have been very intentional about using the data at all levels.
Sites
Individual sites receive a copy of their PQA results following the on-site observation and they receive the survey results during the following school year in a Planning with Data session. Each year, site teams participate in a Planning with Data session. Depending on the needs and experience of the site staff, there are two Planning with Data options: 1) Advanced Planning with Data – a six hour training where site teams become part of a professional learning community by learning about the tools, practicing reading generic data, reviewing their reports, and developing site-level Quality Action Plans; and 2) Mini Planning with Data – a three hour session where site teams review the previous year’s Quality Action Plans, review the new data, and develop or refine their Quality Action Plans. This is part of the overall Training and Technical Assistance for 21st CCLC and SAC grantees provided through the Missouri Afterschool Resource Center (MOARC).
Grantees
Grantees, and their external evaluators, receive a copy of their reports mid-August. Based on the same goals and objectives as the Statewide Evaluation, the grantees and external evaluators meet to discuss the grantee’s site-level data and complete a Guided Reflection document.
MASN provides certification training to the external evaluators so that they are familiar with the goals and objectives, data collection protocols, and analysis and reports being provided to all grantees. The certified external evaluators role focuses on meeting with the program director to discuss the local context in that community and helping the program director interpret the data as they looked for trends across sites. The certified external evaluator documents the unique strengths and challenges of the grantee, highlighting the impact of the local context, in a Guided Reflection summary document. This new standardized data collection, analysis/reporting, and Guided Reflection process allows DESE to have an “apples to apples” comparison of the grantees, while giving the external evaluator a framework to focus on how the local context impacts the grantee’s site-level data. See below for more information about becoming a certified external evaluator.
Statewide
At the state level, the same survey, PQA, and Kids Care Center data (attendance and grades) is used by OSEDA and the Weikart Center for the Statewide Evaluation. Click the icons to download each year’s Statewide Evaluation Report.
Certified External Evaluators
The local 21st CCLC External Evaluation plan includes consistent evaluation methods, data collection, and reporting for all 21st CCLC grantees. As part of the process, each grantee selects a certified external evaluator to complete the Guided Reflection process.
MASN provides the training and certification for 21st CCLC external evaluators so that they are familiar with the consistent goals and objectives, data collection protocols, and analysis and reporting provided to all grantees. The role of the certified external evaluators is to meet with the program director to discuss the local context in that community and to help the program director interpret the data as they looked for trends across sites. The certified local/external evaluator documents the unique strengths and challenges of the grantee, highlighting the impact of the local context, in a Guided Reflection summary document.
In order to become a certified external evaluator, new evaluator candidates must attend a required training session and submit an application and supporting documentation (if not previously submitted). Returning evaluators who were certified for the previous program year may attend an online “refresher” training to become certified for the upcoming year.