OpenClaw Eval#
Challenge#
OpenClaw went viral and people set it up using their Claude subscriptions as the AI model for their OpenClaws. Anthropic then blocked subscriptions making calls with third-party tools registered (i.e. blocked OpenClaw); their proposed alternative is to use an API Key.
I had the same set up. My OpenClaw was named “Kuh” … there were several iterations in the lineage: kuh, kuh2, nanokuh. I was using Kuh often with the $100 per month Claude subscription, if I switched to API-based usage my cost would be closer to $1,000 per month. There are other AI models available, but which one strikes the balance of cost and performance for my use cases?
Action#
Plan#
- Web search for best agentic models and costs
- Define a task / prompt to evaluate performance
- Get OpenRouter API key, configure different models for Kuh, run the prompt for each model
Agentic Models May 2026#
My baseline was Sonnet 4.6. So I want to review cheaper models only. My intuition is to go with either DeepSeek or ChatGPT. DeepSeek because it’s cheap / opensource. ChatGPT becausee they explicitly allow/encourage OpenClaw bots to be used under their subscription plan … subsidizing usage.
| Model | Input /1M | Output /1M | Context | BenchLM Agentic Score |
|---|---|---|---|---|
gpt-5.4 | $2.50 | $15.00 | 1M | 89 |
gpt-4o | $2.50 | $10.00 | 128K | 43 |
claude-sonnet-4-6 | $3.00 | $15.00 | 1M | 83 |
deepseek-v4-pro | $0.44 | $0.87 | 1M | Unpublished |
deepseek-v3.2 | $0.25 | $0.38 | 128K | 58 |
Note: this was not a comprehensive eval. GPT-4.1 should have be evaluated instead of GPT-4o.
Task#
I measured performance based on ability to pull my finance data using a “Kuh”-defined skill. This skill uses the Teller API, categorizes finances, looks for reimbursements.
Noteably, this finance skill was written with Claude, so I should account for some bias in performance with Claude.
I ran the same prompt against each model via Telegram.
The task required the model to: authenticate with Teller, pull transaction data across a date range (hitting the API multiple times due to a per-call transaction limit), deduplicate, categorize, classify reimbursements, and output a clean CSV. Subjectively, I’d subjectively rate the task a 5/10 in difficulty.
OpenRouter#
OpenRouter is very easy to use. $20 worth of credits, one line in my Kuh config file to change the model.
Results#
GPT-5.4 is the clear winner. The table below summarizes my qualitative analysis along with measured costs and request calls on OpenRouter. The requests column is a rough proxy for how simply each model decomposed the problem — fewer requests suggests cleaner reasoning. GPT-5.4 wins on that too: 13 requests, the lowest count, the best grade, and the lowest cost. DeepSeek V4 Pro made 25 requests and still needed a nudge.
| Model | Grade | Cost | Time | Requests | Notes |
|---|---|---|---|---|---|
| GPT-5.4 | A | $0.49 | 2 min | 13 | All data, no hallucinated columns, caught reimbursements |
| DeepSeek V4 Pro | B+ | $1.78 | 15 min | 25 | Best reasoning, caught extra reimbursement, missing rows, needed re-prompt |
| Claude Sonnet 4.6 | B | $4.45 | 3 min | 16 | All data, added unrequested status column |
| DeepSeek V3.2 | C | $0.22 | 10 min | — | Verbose, missed reimbursed transactions, no personality |
| GPT-4o | F | $1.17 | 2 min + retries | 17 | Incomplete data, bad self-evaluation; cost includes 2 validation prompts |
GPT-4o — Failed#
The CSV came back about 60% complete. One column had incorrect data, and the last row was cut off mid-entry. The root cause was likely pagination: Teller enforces a transaction limit per API call, so the model needed to make multiple calls with a smaller date range and stitch the results together. GPT-4o didn’t recognize it had to do this and stopped early without flagging the gap.
When I asked it to evaluate its own output, it didn’t read the CSV file it had just created. It summarized what it thought it did based on tokens still in session. It had no idea it had failed.
Claude Sonnet 4.6 — Passed (B)#
Pulled all the data in one shot, handled pagination correctly, and formatted the CSV cleanly. It identified two reimbursed transactions by cross-referencing other data in the project, this was incomplete as there were more reimbursed. It added a status column marking each transaction as pending or settled. I didn’t ask for that either, and it wasn’t right to add it. Hallucinating schema is a meaningful error.
GPT-5.4 — Passed (A)#
Got all the data. Correct columns — nothing added, nothing missing. Also caught all reimbursed transactions. No hallucinated schema. The clear winner.
DeepSeek V3.2 — Passed (C)#
Passed in the sense that it completed the task and returned a CSV — but the output had meaningful gaps. It didn’t mark any reimbursed transaction in the CSV. The path it took to get there was verbose: lots of narration, restating what it was about to do. It got there, but it didn’t feel like it understood what it was doing.
DeepSeek V4 Pro — Passed (B+)#
Required a re-prompt to complete the task, which is a mark against it. It missed several reimbursements, but it caught a more nuanced reimbursement that GPT 5.4 missed; exptectedly, the quality of reasoning was noticeably higher than V3.2.
Decision#
I will keep Kuh running with GPT-5.4 via OpenRouter. If my costs exceed $20 / month, I will sign up for an OpenAI subscription.