Simply put, the best surveys yield the best, most useful data. No matter the style, the length, or purpose, strategic and purposeful planning are important to solid survey design. To further highlight this importance of the process, we can examine survey design through five lenses:
- The Foundational Lens
- The Research Lens
- The Critic Lens
- The Useful Lens
- The Inclusive Lens
The Foundational Lens
This lens epitomizes nearly every Survey Design 101 course and survey design textbooks. For folks who have taken any formal training or course work related to survey design, this lens will seem very familiar. This lens considers many basics of good survey design. A few of the lessons from this lens include avoiding questions that are double-barreled, contain jargon, or lead respondents to particular answers. A foundational approach to survey design will also focus on the structure of the survey, for instance putting the most important questions first.
But, while this lens is the most common approach to thinking through survey design, it is not always the most engaging and accessible. And, approaching the topic from an alternate lens can help to uncover potential problems that a foundational approach might overlook.
The Research Lens
This lens focuses on using research and theory as a foundation for survey design and also emphasizes using research methodology to continually test survey content. When “testing” survey content, Skyfactor Director of Analytics & Research Sherry Woosley suggests ensuring the survey adheres to and is underpinned by research. For example, a survey question that asks students, “To what degree are you struggling with homesickness?” seems useful and relevant on the surface, and may check many boxes when viewed from a foundational lens. But, it actually ignores the research, theories, and literature about the different types of homesickness and misses concepts that relate to student outcomes.
The Critic Lens
Getting pushback on your well thought out survey can be slightly annoying, but it is ultimately extremely helpful. The snarky participants who question survey language or offer alternative thoughts test not only the limits of patience but also the strength and validity of your survey. Their critical feedback provides marginal perspectives that might be unintentionally or even historically overlooked. Embrace the critics in the spirit of continuous improvement and a solid survey.
The Useful Lens
A question may be well-designed and produce valid results, but it may not be possible to affect change with the results. This lens weights the concept of “interesting vs useful” in regards to survey design. When designing a survey, it is important to not just ask, “What do you want to know?” Good survey design means considering what you will do with the information collected. Whether it’s acting on it unilaterally or being able to pass the results to someone else who is able to take action, consider how the results will be used when designing a survey.
The Inclusive Lens
This final lens, inclusivity, can support a participant’s full and authentic engagement with the survey. Higher engagement with fidelity leads to richer and more useful data. Inclusivity allows the participant to see themselves accurately portrayed and identified, which in turn allows for a greater confidence in the usefulness of the survey. Inclusive language and response options related to concepts like gender and race signal that there is value in a broader voice. Inclusivity also takes into account other demographic information about the potential participants that might affect their reaction or understanding of a survey question. For example, questions about summer vacations might be inappropriate for participants who are not privy to such luxuries. And poorly-worded questions about family can be isolating for participants whose situations are not represented.
Fowler, J., Floyd J. (1995). Improving survey questions: Design and evaluation. (Vol. 38). Thousand Oaks, CA: Sage Publications.
Montenegro, E., & Jankowski, N. A. (2017, January). Equity and assessment: Moving towards culturally responsive assessment (Occasional Paper No. 29). Urbana, IL: University of Illinois and Indiana University, National Institute for Learning Outcomes Assessment (NILOA).
Suskie, L.A. (1996). Questionnaire survey research: What works (2nd ed.). Tallahassee, FL: The Association for Institutional Research.
