Forced Questions (Will Hurt Your Data)

In this episode of Total Survey Design, Dr. Azdren Coma sits down with Dr. Rubén Ángel Arias Rueda, Project Manager and Lead Researcher at Washington State University's Social and Economic Sciences Research Center, for the second part of a two-part series on multilingual surveys. Dr. Arias Rueda specializes in mixed-methods equity research, bilingual data collection, and community-engaged fieldwork, and has led statewide qualitative and survey studies for Washington State agencies and commissions—including research on opportunity gaps for Hispanic and Latino students and on agricultural labor conditions. Before joining the SESRC, he spent 15 years teaching Spanish and Spanish literature at all levels of the curriculum.The conversation explores Rubén's approach to designing surveys that work in both English and Spanish, beginning with the core goal that guides his bilingual instruments. He weighs in on whether tools like Google Translate and generative AI are sufficient for translating a survey into a second language, and discusses the strengths and limits of back-translation as a quality check. From there, the discussion turns to the practical challenges of designing and administering multilingual surveys in the field, the methodological questions involved in combining data across language versions, and the considerations researchers should keep in mind when interpreting results from multilingual samples.The episode closes with Rubén's broader advice for researchers and practitioners taking on multilingual survey work, drawing on lessons from his experience of bilingual data collection with Spanish-speaking communities across Washington State.Support the showContact us at: [email protected] us online at: instagram.com/totalsurveydesign/https://taplink.cc/totalsurveydesign

In this episode of Total Survey Design, Dr. Azdren Coma explores what it takes to design surveys for populations that speak two or more languages. The episode explains why measurement equivalence (ensuring responses mean the same thing across languages) matters more than literal translation, and why the goal should be conceptual equivalence rather than word-for-word matching. Drawing on a personal case study of the 2021 North Macedonia Census, Azdren walks through real-world translation pitfalls, including untranslated Cyrillic dropdown menus and grammatically awkward translations from Macedonian into Albanian, and closes with practical takeaways for reducing cognitive burden and bias across language groups. This is the first of a two-part series on surveying multilingual populations.Support the showContact us at: [email protected] us online at: instagram.com/totalsurveydesign/https://taplink.cc/totalsurveydesign

In this episode, I explore research by Lena Le and Thom Allen at Washington State University on the use of personal appeals in survey contact letters — the persuasive statements that encourage people to participate, like explaining how results will be used or telling respondents their opinion matters.Analyzing 190 surveys, their data shows response rates climb from 21.6% with no appeals to 51.6% with nine. I also ran my own statistical analysis on the data, finding each additional appeal is associated with a 4.1 percentage point increase in response rate (p = 0.010). Ideal for survey researchers and anyone responsible for designing contact strategies to maximize participation.Support the showContact us at: [email protected] us online at: instagram.com/totalsurveydesign/https://taplink.cc/totalsurveydesign

In this episode, Dr. Azdren Coma explores how the Dunning-Kruger effect distorts self-reported skill ratings in hiring and surveys. A seemingly straightforward question like “On a scale of 0 to 10, how skilled are you in Microsoft Excel?” can produce misleading data because undefined scale points allow respondents to interpret them differently. Beginners often overestimate their abilities due to limited awareness of complexity, while more skilled individuals rate themselves more conservatively as they recognize what they don’t know—potentially reversing the ranking of candidates.This episode breaks down the psychological mechanism behind the bias, illustrates it with real-world examples, and offers practical fixes for better data quality, including fully labeling scales, using clearer anchors, and shifting to task-specific questions. Ideal for HR professionals, hiring managers, and anyone relying on self-assessments in talent acquisition or people operations.Support the showContact us at: [email protected] us online at: instagram.com/totalsurveydesign/https://taplink.cc/totalsurveydesign

In this episode, I will be talking about open-ended questions. Along with closed-ended questions, open-ended questions are the other major type of question used in surveys. This episode covers the purpose of open-ended questions, their pros and cons, the different types of open-ended questions, several examples, and finally some tips and best practices for implementing them effectively in surveys.The episode also includes an important ethical cautionary tale, as well as a warning about the dangers of over-relying on generative AI to analyze open-ended questions. Support the showContact us at: [email protected] us online at: instagram.com/totalsurveydesign/https://taplink.cc/totalsurveydesign

What happens when you are faced with a list of options but do not have a strong preference? In this episode, I explore how the simple order of a list can nudge a person's choice. I move beyond the idea of favorite flavors to look at the undecided voter and the hungry diner. I examine why being first on a ballot can sway an election by as much as 10 percent in some local races.I discuss why these order effects are most dangerous in non-partisan elections where voters may not recognize every name on the list. Drawing on the extensive research of Jon Krosnick and others, I explain the difference between the Primacy Effect in written surveys and the Recency Effect in spoken lists. I provide practical solutions for researchers who want to ensure their data is a true reflection of the respondent rather than a fluke of the list order. I also cover randomization techniques and the best way to handle no opinion responses to maintain high data quality.Support the showContact us at: [email protected] us online at: instagram.com/totalsurveydesign/https://taplink.cc/totalsurveydesign

This episode is a live guest lecture recorded for a Marketing Research class at Washington State University. The guest lecture is meant to introduce complete newcomers to general ideas and considerations for survey design. I also offer a few general tips on survey questionnaire design.Support the showContact us at: [email protected] us online at: instagram.com/totalsurveydesign/https://taplink.cc/totalsurveydesign

In this episode of the Total Survey Design Podcast, I explore whether to include a neutral midpoint option in survey questions or force respondents to pick a side. I define forced-choice formats, from even-numbered scales and binary picks to pairwise comparisons. I discuss key evidence like the 2019 Pew Research experiment, which found forced yes/no questions increased reporting on sensitive topics compared to select-all-that-apply lists. I present strong arguments on both sides before explaining why I usually favor neutral options for attitude and opinion questions. I end by offering an elegant solution that combines the presence of a neutral option with the need for respondents to pick a side. Support the showContact us at: [email protected] us online at: instagram.com/totalsurveydesign/https://taplink.cc/totalsurveydesign

Should you ever force respondents to answer a survey question? In this episode of Total Survey Design, I examine forced questions. While these features are often used under the assumption that more complete data means better data, forcing responses frequently leads to worse outcomes by increasing survey breakoffs, encouraging satisficing, and undermining respondents’ trust. I outline more effective and respectful alternatives, encouraging responses without coercion. Support the showContact us at: [email protected] us online at: instagram.com/totalsurveydesign/https://taplink.cc/totalsurveydesign

On this episode of the Total Survey Design podcast, I put generative AI to the test by asking it to critique and improve the Microsoft Word Feedback survey: a short, everyday questionnaire that many of us have seen after using the program. I compare the suggestions from Grok, ChatGPT, and Gemini, highlight where they agree, where they differ, and share my own thoughts on what they got right and what they missed.I recap the original survey (which takes just 50 seconds to describe), dive into the strengths and weaknesses of the AI outputs, and explain why I still think human expertise in survey methodology matters most. Along the way, I discuss the pitfalls of the Net Promoter Score scale, the importance of balanced question wording, building trust with privacy statements, and more.There is still time to have your voices heard! Share your thoughts about this podcast at: https://www.tinyurl.com/totalsurveydesignSupport the showContact us at: [email protected] us online at: instagram.com/totalsurveydesign/https://taplink.cc/totalsurveydesign