Austin’s Experience Using Multiple Measures of SEL

By Lindsay Lamb, Program Evaluation Coordinator, Austin Independent School District

Measures of students’ SEL competence can be used alongside other data to support decisions around program implementation. Currently, the Austin Independent School District’s (AISD) Department of Research and Evaluation (DRE) uses multiple measures to assess effective SEL implementation, including an SEL Implementation Survey; AISD Student Climate Survey; Teaching Empowering, Leading, Learning (TELL) AISD Staff Climate Survey; Family Survey; PK-6th grade students’ personal development skills (report card ratings); and academic performance, attendance, and discipline data.

Elementary school teachers’ assessment of their students’ SEL skills

Every 9 weeks, teachers assess the frequency with which their PK-6th grade students engage in SEL-related skills using the personal development skills report card ratings. These ratings are captured on students’ report cards along with their academic progress. Although these data allow for researchers and program staff to monitor students’ SEL skill acquisition during the course of an academic school year and beyond, as research shows, they cannot be separated from teachers’ own biases in ratings (see the following report, for example).

Given that teachers’ ratings of their students’ skills are biased, researchers in DRE also evaluate students’ SEL skill acquisition as part of the annual AISD Student Climate Survey.

Student Climate Survey

As part of this survey, students are asked to indicate the degree to which they engage in specific SEL skills. These reports are published online in an interactive format, allowing for school administrators, school staff, parents of students, and lay people to explore students’ responses based on student characteristics such as race, gender, economic disadvantage, and others (click here to access the online Student Climate Survey reports). Once again, it is important to note that students’ self-perceptions of their SEL skills are also biased and could vary on the day, time of year, or mood of the student at the time of survey administration. This is one reason we do not use survey data to make high stakes decisions regarding SEL.

To address this issue, and to provide program staff with a more objective measure of effective SEL implementation, DRE researchers worked with SEL program staff to develop a school level SEL implementation survey.

School-level SEL implementation survey

The SEL implementation survey is composed of four broad goals related to SEL implementation in AISD: 1) empowering campus leadership, 2) coordination with family and community partners, 3) coordination with school climate and pedagogy, and 4) explicit SEL instruction. Currently, district-level SEL specialists work with schools to complete the survey in collaboration with school administrators to assess how well the school is meeting these goals, and to set specific goals for a given academic school year.

How we are using our SEL and climate survey data

DRE staff have used SEL implementation data to determine if how well a school has implemented SEL (as measured by the SEL implementation survey) is more related to program outcomes (i.e., school climate, student achievement, attendance, discipline) than length of time participating in SEL (schools began implementing SEL in 2011-2012 with the final schools joining in 2015-2016).

Results from a recent report found that SEL implementation seems to matter more than longevity in SEL:

• Schools with more frequent opportunities to engage parents and community members in SEL (goal 2 on the implementation survey) predicted students’ reading performance on the State of Texas Assessments for Academic Readiness (STAAR), regardless of length of time in SEL
• The percentage change in students receiving discretionary removals was greater at secondary schools in the top quartile of total school-level SEL implementation ratings than at secondary schools in the bottom quartile of school-level SEL implementation ratings.  Note:This finding is important because it suggests that students attending secondary schools where SEL is implemented with greater fidelity experience more positive outcomes than students who attend secondary schools where SEL is implemented with less fidelity.
• Elementary schools where SEL strategies were integrated into school climate and pedagogy had students with higher Student Climate Survey ratings of “I like to come to school” and “I feel safe at school” in 2016–2017 than did schools with less integrated SEL, regardless of years in SEL.
• Secondary school staff felt more confident in their ability to manage students’ behavior (measured on the TELL AISD Staff Climate Survey) in 2016–2017 at schools where SEL explicit instruction was more frequent than at schools where it was less frequent, regardless of longevity in SEL.

On a practical level, SEL specialist have used the implementation and school climate data (both from the Student Climate Survey and the TELL AISD Staff Climate Survey) to work with school-based SEL facilitators and administrative staff to set specific goals for a given academic year. For example, a goal might be based on students’ responses to a specific question on the student climate survey (i.e., African American girls’ responses to the question, “It is easy for me to talk about my problems with the adults at my school”). Specialists also use responses on the SEL implementation survey to guide school goals, such as improving campus leadership in SEL skills.

Additionally, SEL program staff have developed a way to recognize strong SEL implementation through the Seed Model Campuses program. As part of this program, schools set specific goals based on the SEL implementation survey and document their progress toward that goal.

Finally, the percentage of schools meeting or maintaining their school-level (i.e., elementary, middle, high) implementation survey rating has become part of AISD’s superintendent’s scorecard. In this way, moving toward strong SEL implementation is not only a goal for each school, but also for AISD as a whole.

Using multiple measures to assess students’ SEL skill acquisition and fidelity of SEL implementation allows for AISD researchers and practitioners to use data to improve the quality of SEL implementation. By not relying on a single measure of success, AISD builds on the strengths of each school, recognizing areas of celebration, and areas needing growth.