In Defense of Self-Report Surveys
By Kent Pekel, Eugene C. Roehlkepartain, and Amy K. Syvertsen, Search Institute
Our organization recently interviewed more than fifty leaders and practitioners in schools and youth programs to launch a major applied research project focused on strengthening young people’s social and emotional skills by building developmental relationships between young people and adults and between young people and their peers. As part of these interviews, leaders were asked if and how they are measuring social-emotional skills and relationships. Most said they are measuring SEL in some way, usually by asking students to complete self-report surveys on which they report on their attitudes, skills, and behaviors.
Despite the relatively widespread use of self-report surveys, several of the leaders told us that they are beginning to doubt the value of that approach to measuring social and emotional competencies. One interviewee anticipated that her organization will soon discontinue using self-report surveys in favor of new ways of measuring SEL, such as identifying other people’s feelings as they are portrayed in videos and having young people solve puzzles to measure persistence.
Diversifying the ways SEL is measured holds great promise for shedding new light on social-emotional competencies in young people’s lives. At the same time, we think reports of the death of self-report surveys as measures of SEL have been greatly exaggerated. Even as new approaches to measuring SEL are being developed and tested, self-report surveys can and should continue to be used and continuously improved.
Briefly stated, the argument for self-report surveys rests upon at least three pillars:
- As researchers Angela Duckworth and David Yeager have written, “self-report questionnaires are arguably better suited than any other measure for assessing internal psychological states, like feelings of belonging.” A large body of research has shown that those psychological states can powerfully influence important educational outcomes such as grades, high school graduation rates, and college enrollment rates.
- Self-report surveys are efficient and cost-effective when compared to other ways of measuring SEL.
- Youth voice and perspective matter. Youth self-report surveys are an important tool for capturing and understanding young people’s lived experiences from their own perspectives.
That said, self-report SEL measures have several limitations. Researchers must address these in the design of survey instruments, and practitioners should be aware of them when using data from those instruments. One of the most important of those limitations is “reference bias,” or the fact that people often judge themselves and the world based upon their own experiences and understanding. As a result, responses can be highly subjective, making it difficult to interpret the data when comparing across participants. For example, what one student considers working hard, another might consider doing the bare minimum.
In surveys we design, we use several approaches to minimize or mitigate reference bias. For example, we conduct cognitive interviews (or survey “think alouds”) with youth who are similar to those we seek to include in the study. In these interviews, young people help us understand how young people may differ in their interpretation of a survey item. We then use this information to refine the survey. In addition, we rarely ask youth to compare themselves to others.
Looking ahead, we continue to explore other ways to reduce reference bias. One promising approach is to integrate anchoring vignettes into surveys. Anchoring vignettes ask survey participants to assess the attitudes and behaviors of people described in short scenarios. Understanding variation in the ways that participants assess the vignettes makes it possible to interpret and sometimes recalibrate participants’ responses. An anchoring vignette might make it possible, extending the example above, to understand and adjust for the differing ways that young people define working hard.
To date this technique has mostly been utilized to improve the comparability of surveys in research studies (in part because adding anchoring vignettes to a survey can increase its length, which decreases its usability in the real worlds of schools and youth programs). However, with further research and input from practitioners, anchoring vignettes could help to reduce reference bias and further improve the value of self-report surveys for measuring students’ social-emotional skills.
While addressing reference bias through improvements in survey design is an important priority, it is not likely to be possible or even advisable to eliminate such bias completely. As noted above, one of the benefits of measuring SEL using a self-report survey is that the instrument captures the students’ perspectives on their own competencies. Although the student’s perception may not be objectively accurate, perception is reality in important ways, especially when the goal is to understand and strengthen students’ social-emotional competencies. Or, as Thomas’ Theorem from the 1920s reminds us: What people perceive as real is real in its consequences.
It is an exciting time for the field of SEL as new measures are being developed. But let’s not throw out the self-report baby with the proverbial bathwater.
Disclaimer: The Assessment Work Group is committed to enabling a rich dialogue on key issues in the field and seeking out diverse perspectives. The views and opinions expressed in this blog are those of the authors and do not necessarily reflect the official policy or position of the Assessment Work Group, CASEL or any of the organizations involved with the work group.