Over the past couple of years, a few of us at Fathom have been experimenting with a new method to gather insights from large groups of users by using an AI agent trained to conduct qualitative user research via chat. If you’ve done user research, you know the usual tradeoff: go wide with a large sample or go deep with qualitative data. This method, at least on paper, appears to offer both; we’ve been calling it “Qual-at-Scale.”
Four healthcare-related studies into our testing, we wanted to share some of what’s working, what’s not, and where we’ve been surprised. We fully expect that our thinking will change over time as we get smarter and the tools evolve, but here are some early takeaways from the team.
What is Qual-at-Scale and how does it work?
Qualitative research at scale is designed to collect a high volume of user perspectives through an AI‑moderated chat experience rather than one-to-one human-moderated interviews or a fixed‑form survey.
To get started, the Fathom team consults with our client to define and sharpen their primary areas of inquiry—it’s important they’re sufficiently discrete, clear, and truly aligned with the goals of the project. As the areas of inquiry are loaded into platform, the AI chatbot starts connecting with participants, using adaptive phrasing and follow-ups based on how each individual responds.
Participants respond in their own words, in a conversational format, allowing them to explain how they think, act, and make decisions in real-world contexts. Research participants can choose to chat in real time or leave the chat for a bit and come back later when they have time.
After the chat agent conducts a predetermined volume of “interviews”—in our projects, we reached between 75 and 205—a synthesis agent running on the same platform analyzes the transcripts for insights, generating several reports and a full logic tree. A dashboard is generated for the research team to read and interact with in order to further refine their inquiry using a different AI research agent trained to answer questions about the data set.
How can research teams use the insights from Qual-at-Scale?
While AI-powered qualitative research at scale cannot match the depth of understanding that comes from human-led one-on-one interviews, it appears capable of surfacing clear, directional signals across large groups of participants. This approach offers quick takes on patterns in behavior, experiences, and even sentiment. Further, the scale and breadth of roles and settings can reduce the risk of over-indexing on an “outlier” participant, a common pitfall in small-scale qual.
The insights also differ from those of traditional quantitative research. Because participants are neither asked standardized questions, nor limited to responding with pre-built answer sets and scales, the findings do not reliably speak to frequency or prevalence or represent statistical significance.
We have seen our clients lean into these types of insights in order to:
- Inform product strategy and prioritization
- Align teams around shared user realities
- Stress‑test assumptions about current practice
- Guide hypothesis formation and research
We counsel our clients not to use them as standalone evidence for precise sizing, forecasting, or performance measurement.
When is Qual-at-Scale a good approach?
In our experience so far, we see Qual-at-Scale as a good way to reinforce or build confidence in something that is already known or suspected from other qualitative research.
- The larger volume may provide broader context of understanding
- The qualitative nature of the questions can open the door to learning more about the "why" or "how" behind something
- The chatbot model seems particularly good at gathering sentiments, opinions, or descriptions of current state
- The transcripts allow you to see how participants describe things in their own words
As with traditional quant methods, trusting the results means the research team needs to know enough to “check” the AI insights. AI is really good at making things that sound right, so it’s important that teams are familiar enough with their areas of inquiry to spot anything that feels “off” and be able to dig in and validate or invalidate it.
The tool we’ve used at Fathom allows research teams to drill into the parts of transcripts that the synthesis agent used to generate an insight, so this validation is relatively seamless.
When is Qual-at-Scale NOT a good approach?
Like any method, Qual-at-Scale has its blind spots. Here's where we've seen it struggle.
- Hard numbers or percentages are difficult to get to and not guaranteed
- Nuanced topics with non-standard language may generate false insights (e.g., a “rep” in healthcare can mean many things)
- A text-based format may not work for sharing deep thinking (someone’s inner reasoning, mental model, or multi-step process may take too long to type out)
- Broad, “everything under the sun” approaches don't work well—AI does better with narrow areas and homogeneous audiences
What to look for in a partner for Qual-at-Scale
If you feel like you have a user research need where Qual-at-Scale might be a good fit, it can be beneficial to work with an experienced research partner to get the most from your study.
Finding a research partner who understands the method and has familiarity with a particular tool will allow you to focus on getting the most insights from your study instead of learning a new methodology. In addition, the right partner can add value by:
- Helping determine whether a Qual-at-Scale study is the right approach, or whether another method would serve you better
- Interviewing stakeholders and subject-matter experts to shape the research objectives and areas of exploration so the AI agent can follow along
- Recommending which audiences to include and writing effective screening questions to find the best participants
- Using the AI research platform to verify insights by digging into confusing, surprising, or ambiguous conclusions the model may produce
- Creating insights presentations at the right level and in a digestible format, with ties back to your prior research, hypotheses, and research goals
- Providing actionable recommendations for your project and for further research
The future of AI-assisted research
While Qual-at-Scale still has room to improve, there’s no doubt AI-assisted research will continue to evolve—bringing with it new value and, likely, new risks. We're still learning, and we'd love to compare notes.If you're experimenting with AI-moderated research yourself, or wondering whether it's right for your next project, reach out. We're always happy to talk shop.
