Afterschool Data Collection and Evaluation
MASN coordinates a variety of data collection and evaluation services for afterschool programs. Although the framework was designed for 21st CCLC and School Age Community (SAC) grantees, the framework has been used with the St. Louis Mental Health Board’s Afterschool for All Partnership sites. The data collection and quality improvement tools can be used for individual sites.
In 2012, MASN contracted with Dr. Wayne Mayfield at the Office of Social and Economic Data Analysis (OSEDA) to develop an evaluation plan for measuring the impact of the Network’s training and technical assistance provided to the 21st CCLC grantees. The report “Recommendations for Program- and Site-Level Data Collection” outlined the need for more standardization in the local evaluations, a consistent observable measure of program quality, and a way of measuring youth outcomes related to college and career readiness.
Also in 2012, MASN’s Quality committee and the statewide Program Assessment Scoring/System (PASS) committee met jointly and selected the Program Quality Assessments (PQAs) as the instruments to be used for state-funded quality improvement efforts. During this time period, the Quality committee continued to work with Dr. Mayfield to create a logic model and “best practice” survey recommendations for all afterschool programs throughout the state. When the workgroup became aware of the Weikart Center’s Leading Indicators framework, MASN partnered with OSEDA and the Weikart Center to develop one set of afterschool surveys that contained both the Leading Indicators and MO College and Career Readiness questions. Meanwhile, DESE and MASN began conversations about using these new resources to revise the 21st CCLC Statewide Evaluation process.
In 2013 and 2014, MASN, DESE, and OSEDA worked closely to develop the new evaluation framework (MO Goals and Objectives) that would apply to both the statewide and local external evaluations. Although the data collection and evaluation system was first designed to meet the needs of the 21st CCLC and SAC programs, the it can be used in part or whole with any afterschool program.
Data collection started in the 2013-14 academic year and the first 21st CCLC Statewide Evaluation report was completed by OSEDA and the Weikart Center for Youth Program Quality. In 2014-15, the Local External Evaluation system changes went into effect and the grantee-level evaluations for new 21st CCLC Cohorts were parallel to the state evaluation. In addition to being used for the statewide and local evaluations, the same data PQA and survey data was also provided back to the sites during Planning with Data sessions to create site-level Quality Action Plans.
Program Quality Assessments
The Program Quality Assessments (PQAs) from the Weikart Center for Youth Program Quality are reliable and valid tools for looking at the quality of afterschool programs. The School Age Program Quality Assessment (SAPQA) provides feedback for programs serving younger youth (K-6) and the Youth Program Quality Assessment (YPQA) provides feedback for programs serving older youth (4-12). Additional protocols and supplemental scales have been developed for the tools. In Missouri, the SAPQA Walk Through methodology is used to look at the full variety of programming including snack and transitions. For the older youth, the STEM supplemental scales used on the YPQA provide additional feedback related to the grant’s science objectives (the STEM supplement is only available for the older youth YPQA instrument).
The 4-H Center for Youth Development manages the process of scheduling trained observers and ensures that the proper protocols are followed so that the data is collected and presented in a consistent manner. Typically, each site has one SAPQA or YPQA completed each academic year.
Surveys provide the opportunity to collect feedback from a variety of stakeholders of the afterschool program, including youth, families, program coordinators, front line staff, school administrators, and community partners. This information can help to guide local and statewide program improvement efforts, and influence the future direction of afterschool in Missouri. MASN distributes online survey links with a unique Org ID for each site. If the site chooses to use paper scantron forms, the same Org ID will be used to match the data. The Weikart Center and OSEDA receive de-identified data sets from MASN to use in the preparation of the 21st CCLC Statewide Evaluation, Site-Level Leading Indicators, and MO College and Career Readiness reports.
Kids Care Center
The Kids Care Center (KCC) data system was designed to facilitate data reporting to meet federal and state requirements for the DESE afterschool grantees. Therefore, all grantees that are awarded funds must use this KCC systems so that data can be entered in to APR21. KCC is also available to any afterschool program free of charge. A KCC student ID is used on the Youth Surveys so that the grades and attendance data can be matched between datasets.
Four reports are generated by OSEDA and the Weikart Center for use at the local or grantee-level:
- Site Summary Report – This report calculates whether the site has met or not met the benchmarks in each objective. This site-level report is given to the Certified External Evaluator and Program Director for use in the Guided Reflection process.
- College and Career Readiness Report – This report contains the specific items that make up each of the scales in Goals 1 and 3, along with additional scales for administrator feedback, community partner feedback, and Strengthening Families. This report is used for the grantee external evaluation and the site-level Planning with Data.
- Leading Indicators Report – This report contains the specific items that make up each of the scales in Goal 2, along with the additional Weikart Center Leading Indicators items not included in Goal 2. This report is used for the grantee external evaluation and the site-level Planning with Data.
- PQA Report – This report includes the item level scores and comments. It is provided to sites and TA providers for use in Planning with Data sessions and is provided to the certified external evaluator for inclusion in the guided reflection process.
In addition to the site-level reports, OSEDA and the Weikart Center use the same data to complete the 21st CCLC Statewide Evaluation report, and OSEDA prepares an additional 21st CCLC and CCDF Grantees comparison report.
Using the Data at All Levels
As we developed this data collection and improvement framework (link to chart), we have been very intentional about using the data at all levels.
Individual sites receive a copy of their PQA results following the on-site observation and they receive the survey results during the following school year in a Planning with Data session. Each year, site teams participate in a Planning with Data session. Depending on the needs and experience of the site staff, there are two Planning with Data options: 1) Advanced Planning with Data – a six hour training where site teams become part of a professional learning community by learning about the tools, practicing reading generic data, reviewing their reports, and developing site-level Quality Action Plans; and 2) Mini Planning with Data – a three hour session where site teams review the previous year’s Quality Action Plans, review the new data, and develop or refine their Quality Action Plans. This is part of the overall Training and Technical Assistance for 21st CCLC and SAC grantees provided through the Missouri Afterschool Resource Center (MOARC).
Grantees, and their external evaluators, receive a copy of their reports mid-August. Based on the same goals and objectives as the Statewide Evaluation, the grantees and external evaluators meet to discuss the grantee’s site-level data and complete a Guided Reflection document.
MASN provides certification training to the external evaluators so that they are familiar with the goals and objectives, data collection protocols, and analysis and reports being provided to all grantees. The certified external evaluators role focuses on meeting with the program director to discuss the local context in that community and helping the program director interpret the data as they looked for trends across sites. The certified external evaluator documents the unique strengths and challenges of the grantee, highlighting the impact of the local context, in a Guided Reflection summary document. This new standardized data collection, analysis/reporting, and Guided Reflection process allows DESE to have an “apples to apples” comparisons of the grantees, while giving the external evaluator a framework to focus on how the local context impacts the grantee’s site-level data.
At the state level, the same survey, PQA, and Kids Care Center data (attendance and grades) is used by OSEDA and the Weikart Center for the Statewide Evaluation.210