Picking Between Claude and GPT for a Feature: Our Actual Process

Clients ask us which model to use more often than they ask almost anything else about an AI feature, and the honest answer is always the same: it depends on the task, and we do not know until we test it against the actual thing we are building. We keep both OpenAI and Anthropic as default providers specifically because neither one wins every task, and picking one upfront on reputation alone is how you end up rebuilding the integration layer six months later.

The eval set comes before the model choice, not after

We do not evaluate models in the abstract. Before we pick a provider for a feature, we build a small evaluation set from real examples of the task — for a natural-language analytics feature like the one underlying DataPulse, that means real questions a revenue team would actually ask, with a known-correct answer or acceptable answer range for each. Fifty to a hundred examples is usually enough to see a meaningful difference between models on the specific task, even though it is nowhere near enough to make a claim about general model quality.

Every model gets run against the same eval set with the same prompt structure, and we compare on the axes that actually matter for the feature: task accuracy against the known answers, latency at the percentile that matters for the product experience, cost per request at expected volume, and failure mode — does it fail by refusing, by hallucinating confidently, or by asking a clarifying question. That last one matters more than people expect. A model that fails by asking for clarification is often easier to build a good product experience around than one that fails by guessing.

Where each has actually won for us

In our experience, and this changes as both providers ship new models, GPT models have tended to edge ahead on tasks with well-defined structured output requirements and function calling reliability, which matters for agentic workflows with many tool calls in sequence. Claude models have tended to edge ahead on tasks requiring longer context retention and on instruction-following when the prompt has many specific constraints layered together, which matters for the kind of nuanced analytics questions DataPulse handles.

Neither of those patterns is a permanent law. We rerun the eval set against new model releases from both providers before assuming last quarter's winner is still the right choice, because it frequently is not.

What this looks like in practice

• Build the task-specific eval set first, from real examples, before touching a model API
• Run every candidate model against the same set with the same prompt
• Weight accuracy, latency, cost, and failure mode by what the actual product needs, not by a generic benchmark leaderboard
• Re-run the comparison when either provider ships a new model generation

The teams that get burned are the ones who pick a provider in a kickoff meeting based on which one they personally prefer using, then discover eight weeks into the build that the model is failing in a way their eval set — which did not exist yet — would have caught immediately. Build the eval harness first. The model choice becomes almost mechanical once you have it.