Carnegie has made considerable progress on developing an information system that integrates real-time faculty and student data with institutional records to inform continuous improvement as we address problems of educational practice and constantly improve our ability to serve developmental mathematics students at community colleges.

For example, we are designing a system that will incorporate institutional records going back to 2008 on the longitudinal performance of cohorts of students designated for developmental mathematics at each of the 30 colleges participating in our community college mathematics pathways initiative. These data constitute a baseline for understanding institutional performance over time, for establishing college-by-college improvement targets, and for exploring the antecedents and conditions of performance going forward.

Building off of this, these two pathways — Statway™ and Quantway™ — draw additional detailed data from weekly faculty facilitator calls, faculty forum discussions, two-minute faculty and student surveys reported on a lesson-by-lesson basis, supplemental background data on students, dispositions toward success and math tests of fundamental concepts, follow-up student surveys, data feeds from the out-of-class digital platform that include student homework and other out-of-class activities, common topical and end-of-module assessments, as well as end-of-course assessments and course grades. It is all data for the improvement of teaching and of the materials.

Analytics is now prototyping continuous data feed reports to faculty on their classroom context and individual student progress. Identifying places where rapid interventions might occur is one driver in our overall plan for advancing student success.

Specifically, analytics address four broad purposes:

1. To inform faculty so they can more effectively address the needs of students

Routine reports describing the “classroom ecology,” including students’ language background, interest and engagement in mathematics, and entering levels of basic mathematics skills can inform faculties’ curricular and instructional planning from the very outset of the course. Predictive models developed through analytics can forecast students’ needs (including real time indicators of engagement or disassociation) to enable faculty to tailor interventions and support services for individual students.

2. To identify needed improvements in the instructional system

Data from numerous sources, especially those that track student engagement and achievement, but also including frequent faculty and student feedback, are analyzed to determine where instructional improvements may be needed. This includes possible changes to the instructional kernel itself — curricular materials, the out-of-class platforms and assessments. It also includes identifying areas of faculty knowledge and skill that may require attention in the advancing quality teaching strand.

3. To test proposed changes as potential improvements

As modifications to the instructional system are proposed, they are prototyped and rapidly tested using the techniques and methodologies of improvement research to ascertain whether these changes can be warranted as improvements. In this manner, Carnegie aims to ensure effective implementation across locations and contexts.

4. To examine and test for impact and accomplishment of the whole effort

The data that are gathered to enable improvement also allow us to examine the overall impact of the Pathways initiative. We are now in the process of developing value-added estimates for each classroom, college and sub-group of students and will compare these against benchmarks established this first year and against results for similar students not enrolled in the pathways in each college. We intend to continue to track the success of students post-pathway through the following academic year. Understanding “what happens next” for students is another key to improving what we do with them while they are actually enrolled in a pathway.

Our overarching goal is to enhance the capacities of participating faculty to engage in systematic program improvement. We pursue this both for the sake of their fullest participation in the Statway™ and Quantway™ Networked Improvement Communities and to develop the overall capacity of member institutions to use the approaches, tools, and techniques of quality improvement to enhance the diverse program implementation efforts in which they may engage. Ultimately, the biggest payoff of all may reside there.

————

This post was written with contributions from Carnegie colleagues.