The aidnn planner agent is the brain behind the team. It turns a question into steps and decides which agents and tools to use.
- 01PlanA planner turns the question into steps, using your definitions.
- 02RunCode runs in a sealed sandbox next to your data. Only a summary comes back.
- 03CritiqueA separate critic can veto the plan and send it back with the reason.
- 04VerifyA verifier checks the result against the source before anyone sees it.
- 05AnswerYou get the answer and the trail of how every number was reached.
Choosing the model behind aidnn’s planner is a decision we keep coming back to. We currently use GPT-5.5, but new models arrive regularly, each promising better answers, shorter waits or lower costs. We want to know which of those gains hold up in the work our customers actually ask aidnn to do.
That takes more than trying a new release. Each model needs to work well with the rest of the team, across different tasks and longer conversations. We make that evaluation part of building aidnn, so our customers can benefit from better models without having to keep up with every release themselves.
As we explored in our comparison of decision models, better benchmark numbers don’t always translate into better performance in an enterprise workflow. For aidnn’s planner, the model has to turn a question into sensible steps, choose the right agents and tools, and leave results the team can verify. As the conversation develops, it needs to carry forward earlier findings and requirements. And when a step fails or a result doesn’t check out, it needs to work out what went wrong and adjust the plan without losing sight of the user’s goal. Those are the behaviors we need to preserve when we switch models.
With that in mind, we tested five models on the same 136 questions, from simple lookups and multi-step analysis to longer conversations with follow-ups. We measured response time and total cost, checked the answers, and reviewed selected cases to understand how each model planned the work. We wanted to know whether another model could match GPT-5.5’s answer quality and speed at a lower cost, while following the planning instructions we rely on.
GPT-6 Sol is our preferred candidate to replace GPT-5.5. It cost 34.8% less at almost the same average response time. In selected planning traces, it also separated independent work and validation steps more explicitly, which fits the planning discipline we want.
Sonnet 5.5 was the fastest overall. Its response times were similar to GPT-6 Sol on simple and medium questions, and faster on complex questions and in longer conversations. It remains a good alternative when speed matters most or customers request Anthropic BYOK. All five models passed the evaluated answer checks.
Click a legend to show or hide a model. Hover, tap or focus a point for its recorded value.
Sonnet is fastest; GPT-6 Sol cuts cost
All 136 questions per model · 680 total interactions · Mean per question
See the numbers
| Model | Questions | Mean response time | Total cost / question |
|---|---|---|---|
| GPT-5.5 | 136 | 26.96s | $0.2206 |
| GPT-6 Sol | 136 | 26.84s | $0.1438 |
| GPT-6.1 Sol | 136 | 41.41s | $0.1207 |
| Sonnet 5.5 | 136 | 21.05s | $0.1541 |
| Opus 5.5 | 136 | 25.61s | $0.2580 |
What we found
Each model answered 136 questions: 15 simple, 17 medium and 19 complex, plus 85 questions across separate 10-, 25- and 50-question conversations. That gave us 680 interactions, covering direct lookups, multi-step analysis, workflows that produce tables or charts, and follow-ups as conversation context accumulated.
GPT-6 Sol averaged 26.84 seconds and $0.1438 per answer. GPT-5.5 averaged 26.96 seconds and $0.2206. The response times were nearly identical, with a clear cost advantage for GPT-6 Sol.
Sonnet averaged 21.05 seconds and $0.1541. It was 21.6% faster overall, while GPT-6 Sol was 6.7% cheaper. Their response times were similar on simple and medium questions; Sonnet pulled ahead on complex questions and in deep sessions.
GPT-6.1 Sol had the lowest cost at $0.1207, but the longest wait at 41.41 seconds. Opus averaged 25.61 seconds and $0.2580, with no speed, cost or measured accuracy advantage over Sonnet.
| Model | Questions | Mean latency | Total response time | Mean USD/query | Total USD |
|---|---|---|---|---|---|
| GPT-5.5 | 136 | 26.96s | 61.11 min | $0.2206 | $30.00 |
| GPT-6 Sol | 136 | 26.84s | 60.84 min | $0.1438 | $19.55 |
| GPT-6.1 Sol | 136 | 41.41s | 93.85 min | $0.1207 | $16.41 |
| Sonnet 5.5 | 136 | 21.05s | 47.72 min | $0.1541 | $20.95 |
| Opus 5.5 | 136 | 25.61s | 58.05 min | $0.2580 | $35.08 |
Costs include the main agent and its helpers. Total response time sums individual answer timers. Inference settings were fixed for this benchmark. See methodology.
Five models selected.
136 questions per model: 15 simple + 17 medium + 19 complex + 85 deep-session (10 + 25 + 50).
Where the extra wait comes from
Simple questions usually need one tool; harder questions can require code, several steps and a chart or table. This chart groups the 51 single-question cases by difficulty.
Response time by question difficulty
See the numbers
| Model | Difficulty | Seconds |
|---|---|---|
| GPT-5.5 | simple | 8.23 |
| GPT-6 Sol | simple | 7.06 |
| GPT-6.1 Sol | simple | 17.46 |
| Sonnet 5.5 | simple | 6.69 |
| Opus 5.5 | simple | 8.40 |
| GPT-5.5 | medium | 20.85 |
| GPT-6 Sol | medium | 15.88 |
| GPT-6.1 Sol | medium | 31.55 |
| Sonnet 5.5 | medium | 15.82 |
| Opus 5.5 | medium | 20.75 |
| GPT-5.5 | complex | 41.84 |
| GPT-6 Sol | complex | 46.66 |
| GPT-6.1 Sol | complex | 78.19 |
| Sonnet 5.5 | complex | 34.88 |
| Opus 5.5 | complex | 43.81 |
What we paid for
One question can take several calls to the model: the planner chooses a tool, reads its result and decides what to do next. Each call to that main model is a responder request. The number of calls, the context sent each time and the output generated all affect cost.
Supporting agents also make their own model calls. The cost charts include those calls too; the token charts below cover only the planner’s model.
Input tokens: cached and fresh
See the numbers
| Model | Cache reads K | Cache writes K | Fresh K |
|---|---|---|---|
| GPT-5.5 | 173.42 | 0.00 | 8.14 |
| GPT-6 Sol | 181.42 | 5.55 | 0.02 |
| GPT-6.1 Sol | 176.09 | 5.37 | 0.02 |
| Sonnet 5.5 | 246.68 | 0.42 | 29.66 |
| Opus 5.5 | 241.28 | 0.42 | 30.64 |
Why is Claude’s input higher? Part of the difference comes from tokenization. Sonnet 5.5 and Opus 5.5 use Claude’s newer tokenizer, which Anthropic says produces about 30% more tokens for the same text than its previous tokenizer, depending on the content. That comparison is with earlier Claude models, not GPT. Providers also encode tool definitions differently, so higher token counts do not necessarily mean more text was sent. Our costs already reflect the actual provider-reported usage.
Output tokens cover more than the final answer. They include tool arguments and billed reasoning across all main-agent requests needed to answer a question. They should not be read as the length of the answer a user sees.
Claude generated more output per question
See the numbers
| Model | Output tokens / question |
|---|---|
| GPT-5.5 | 573.39 |
| GPT-6 Sol | 484.63 |
| GPT-6.1 Sol | 463.10 |
| Sonnet 5.5 | 1022.05 |
| Opus 5.5 | 1142.26 |
Most input was cached. The rate below is cached reads divided by total main-agent input, including cache writes.
Prompt-cache hit rate
See the numbers
| Model | Cached reads / input (%) |
|---|---|
| GPT-5.5 | 95.51 |
| GPT-6 Sol | 97.02 |
| GPT-6.1 Sol | 97.03 |
| Sonnet 5.5 | 89.13 |
| Opus 5.5 | 88.60 |
Sonnet took fewer steps to get to an answer. Across the evaluation, it made 509 tool calls compared with GPT-6 Sol’s 623. It inspected data previews less often (41 versus 125 calls) and used fewer execution calls (146 versus 178). In some traces we reviewed, Sonnet also combined independent tasks into one step, rather than separating them as the planning prompt asks. That helped keep the workflow shorter, but GPT-6 Sol followed that decomposition rule more closely in those cases. Both returned correct answers in those examples. These totals include unsuccessful calls.
How their plans differed
For a sales/products join, GPT first checked product matches, then aggregated revenue. Sonnet joined and aggregated in one step. Both returned correct totals. For three independent dataset analyses, GPT used three coding steps while Sonnet used one.
Separating independent work makes results easier to check and failures easier to retry. Related operations can still belong in one step, so fewer calls alone do not prove weaker planning.
These examples favor GPT’s separation of independent work, but do not establish better instruction following across the whole workload. Some single-step plans were appropriate, and Sonnet preserved earlier chart requirements better in one follow-up.
Tool calls per question
See the numbers
| Model | Tool calls |
|---|---|
| GPT-5.5 | 4.34 |
| GPT-6 Sol | 4.58 |
| GPT-6.1 Sol | 4.37 |
| Sonnet 5.5 | 3.74 |
| Opus 5.5 | 3.70 |