Fireside Product Management podkast

I Tested 5 AI Tools to Write a PRD—Here's the Winner

0:00
52:07
Do tyłu o 15 sekund
Do przodu o 15 sekund

TLDR: It was Claude :-)When I set out to compare ChatGPT, Claude, Gemini, Grok, and ChatPRD for writing Product Requirement Documents, I figured they’d all be roughly equivalent. Maybe some subtle variations in tone or structure, but nothing earth-shattering. They’re all built on similar transformer architectures, trained on massive datasets, and marketed as capable of handling complex business writing.

What I discovered over 45 minutes of hands-on testing revealed not just which tools are better for PRD creation, but why they’re better, and more importantly, how you should actually be using AI to accelerate your product work without sacrificing quality or strategic thinking.

If you’re an early or mid-career PM in Silicon Valley, this matters to you. Because here’s the uncomfortable truth: your peers are already using AI to write PRDs, analyze features, and generate documentation. The question isn’t whether to use these tools. The question is whether you’re using the right ones most effectively.

So let me walk you through exactly what I did, what I learned, and what you should do differently.

The Setup: A Real-World Test Case

Here’s how I structured the experiment. As I said at the beginning of my recording, “We are back in the Fireside PM podcast and I did that review of the ChatGPT browser and people seemed to like it and then I asked, uh, in a poll, I think it was a LinkedIn poll maybe, what should my next PM product review be? And, people asked for ChatPRD.”

So I had my marching orders from the audience. But I wanted to make this more comprehensive than just testing ChatPRD in isolation. I opened up five tabs: ChatGPT, Claude, Gemini, Grok, and ChatPRD.

For the test case, I chose something realistic and relevant: an AI-powered tutor for high school students. Think KhanAmigo or similar edtech platforms. This gave me a concrete product scenario that’s complex enough to stress-test these tools but straightforward enough that I could iterate quickly.

But here’s the critical part that too many PMs get wrong when they start using AI for product work: I didn’t just throw a single sentence at these tools and expect magic.

The “Back of the Napkin” Approach: Why You Still Need to Think

“I presume everybody agrees that you should have some formulated thinking before you dump it into the chatbot for your PRD,” I noted early in my experiment. “I suppose in the future maybe you could just do, like, a one-sentence prompt and come out with the perfect PRD because it would just know everything about you and your company in the context, but for now we’re gonna do this more, a little old-school AI approach where we’re gonna do some original human thinking.”

This is crucial. I see so many PMs, especially those newer to the field, treat AI like a magic oracle. They type in “Write me a PRD for a social feature” and then wonder why the output is generic, unfocused, and useless.

Your job as a PM isn’t to become obsolete. It’s to become more effective. And that means doing the strategic thinking work that AI cannot do for you.

So I started in Google Docs with what I call a “back of the napkin” PRD structure. Here’s what I included:

Why: The strategic rationale. In this case: “Want to complement our existing edtech business with a personalized AI tutor, uh, want to maintain position industry, and grow through innovation. on mission for learners.”

Target User: Who are we building for? “High school students interested in improving their grades and fundamentals. Fundamental knowledge topics. Specifically science and math. Students who are not in the top ten percent, nor in the bottom ten percent.”

This is key—I got specific. Not just “students,” but students in the middle 80%. Not just “any subject,” but science and math. This specificity is what separates useful AI output from garbage.

Problem to Solve: What’s broken? “Students want better grades. Students are impatient. Students currently use AI just for finding the answers and less to, uh, understand concepts and practice using them.”

Key Elements: The feature set and approach.

Success Metrics: How we’d measure success.

Now, was this a perfectly polished PRD outline? Hell no. As you can see from my transcript, I was literally thinking out loud, making typos, restructuring on the fly. But that’s exactly the point. I put in maybe 10-15 minutes of human strategic thinking. That’s all it took to create a foundation that would dramatically improve what came out of the AI tools.

Round One: Generating the Full PRD

With my back-of-the-napkin outline ready, I copied it into each tool with a simple prompt asking them to expand it into a more complete PRD.

ChatGPT: The Reliable Generalist

ChatGPT gave me something that was... fine. Competent. Professional. But also deeply uninspiring.

The document it produced checked all the boxes. It had the sections you’d expect. The writing was clear. But when I read it, I couldn’t shake the feeling that I was reading something that could have been written for literally any product in any company. It felt like “an average of everything out there,” as I noted in my evaluation.

Here’s what ChatGPT did well: It understood the basic structure of a PRD. It generated appropriate sections. The grammar and formatting were clean. If you needed to hand something in by EOD and had literally no time for refinement, ChatGPT would save you from complete embarrassment.

But here’s what it lacked: Depth. Nuance. Strategic thinking that felt connected to real product decisions. When it described the target user, it used phrases that could apply to any edtech product. When it outlined success metrics, they were the obvious ones (engagement, retention, test scores) without any interesting thinking about leading indicators or proxy metrics.

The problem with generic output isn’t that it’s wrong, it’s that it’s invisible. When you’re trying to get buy-in from leadership or alignment from engineering, you need your PRD to feel specific, considered, and connected to your company’s actual strategy. ChatGPT’s output felt like it was written by someone who’d read a lot of PRDs but never actually shipped a product.

One specific example: When I asked for success metrics, ChatGPT gave me “Student engagement rate, Time spent on platform, Test score improvement.” These aren’t wrong, but they’re lazy. They don’t show any thinking about what specifically matters for an AI tutor versus any other educational product. Compare that to Claude’s output, which got more specific about things like “concept mastery rate” and “question-to-understanding ratio.”

Actionable Insight: Use ChatGPT when you need fast, serviceable documentation that doesn’t need to be exceptional. Think: internal updates, status reports, routine communications. Don’t rely on it for strategic documents where differentiation matters. If you do use ChatGPT for important documents, treat its output as a starting point that needs significant human refinement to add strategic depth and company-specific context.

Gemini: Better Than Expected

Google’s Gemini actually impressed me more than I anticipated. The structure was solid, and it had a nice balance of detail without being overwhelming.

What Gemini got right: The writing had a nice flow to it. The document felt organized and logical. It did a better job than ChatGPT at providing specific examples and thinking through edge cases. For instance, when describing the target user, it went beyond demographics to consider behavioral characteristics and motivations.

Gemini also showed some interesting strategic thinking. It considered competitive positioning more thoughtfully than ChatGPT and proposed some differentiation angles that weren’t in my original outline. Good AI tools should add insight, not just regurgitate your input with better formatting.

But here’s where it fell short: the visual elements. When I asked for mockups, Gemini produced images that looked more like stock photos than actual product designs. They weren’t terrible, but they weren’t comp…



This is a public episode. If you would like to discuss this with other subscribers or get access to bonus episodes, visit firesidepm.substack.com

Więcej odcinków z kanału "Fireside Product Management"