OpenAI’s July 17 post, *A scorecard for the AI age*, is not a model launch. It is a procurement argument. The company is telling CFOs and operators to stop measuring AI the way they measured SaaS: seats bought, licenses renewed, tokens burned. The better scorecard, OpenAI says, is **useful intelligence per dollar** — whether the value of completed work grows faster than the full cost of producing it. That framing is late, obvious, and still useful. Most enterprise AI dashboards still track activity instead of outcomes. ## The old metrics are lying to you Token price is a vendor metric. Seat count is a sales metric. Weekly active users is a product metric. None of them answer the only question that matters in production: **Did the system complete work we would otherwise pay a human to finish?** A cheap model that fails three times and needs heavy review can cost more than a expensive model that lands a usable result on the first pass. OpenAI is finally saying this out loud because the market is full of teams with rising AI bills and unclear ROI stories. The scorecard OpenAI proposes has four questions: 1. Is AI completing work that matters? 2. What does each successful task cost? 3. Can people depend on the result? 4. Does each AI dollar produce more value as usage grows? That is a better operating system than “we adopted Copilot.” ## 1. Count finished work, not prompts OpenAI’s first point is blunt: measure outcomes in the system where the work happens. - Support: issues resolved - Engineering: code changes that pass tests and ship - Legal: contracts reviewed accurately and on time - Finance: forecast packages prepared for decision, not just summarized This is the difference between AI as autocomplete and AI as labor substitution. If your metric is “messages sent,” you can look busy while producing nothing durable. A practical rule for teams: ```text Define “done” before you measure the model. If you cannot define done, you are not ready to measure ROI. ``` For a finance workflow, “done” is not “AI made a chart.” Done is “the forecast deck is reconciled, sourced, and ready for review.” For coding, done is not “the agent wrote code.” Done is “the PR is reviewed, tested, and merged.” ## 2. Cost per successful task beats cost per token OpenAI’s second measure is the one most buyers still get wrong. Full task cost includes: - model inference - tool calls and retries - human review time - rework - latency waiting for a usable answer - escalation when the system fails So the equation is simple: ```text cost_per_success = total_cost_of_attempts / successful_tasks ``` Not: ```text cost_per_token * tokens_used ``` This is why OpenAI is pushing the GPT-5.6 family as a tiered system: - **Sol** — flagship reasoning when one-pass quality matters - **Terra** — balanced depth and cost - **Luna** — fast and cheap for high-volume work The correct routing decision is not “always use the cheapest model.” It is “use the model that minimizes total cost for the required quality bar.” OpenAI claims GPT-5.6 Sol reached 72.7% on DeepSWE v1.1 long-horizon engineering tasks, above Claude Fable 5’s 69.9%, at lower estimated API cost and with fewer output tokens on coding-agent evaluations. Treat vendor benchmarks as directional, not gospel. The useful idea survives even if the exact ranking moves: **efficiency is outcome-normalized, not token-normalized.** ## 3. Dependability is an economic feature The third measure is where agentic systems usually die. OpenAI breaks outcomes into three buckets: - **Ready to use** - **Needs correction** - **Needs escalation** That is a better dashboard than accuracy alone. A model can be “smart” and still expensive if every answer requires cleanup. As systems move from drafting into action — browsing, editing files, calling tools, changing tickets, touching production systems — dependability becomes the product. Without it, organizations freeze AI in the draft box forever. Before expanding autonomy, define: - what data the system can access - what systems it can change - when a human must approve - what failure looks like in the workflow, not just the model log Safety and governance are not side quests here. They are prerequisites for putting AI into high-value paths. ## 4. Value should improve as usage grows The fourth measure is the compounding test. Track one workflow over time: - successful tasks completed - total cost - cost per successful task - quality hold rate If completed work grows faster than cost while quality holds, each AI dollar is getting better. If usage rises while quality falls or review burden explodes, you do not have leverage. You have subsidized busywork. OpenAI ties this to compute economics: better models, better routing, better infrastructure, higher utilization. The customer-facing version is simpler. Every generation should make the same work cheaper or unlock work that was previously too expensive to automate. ## What this means for buyers right now If you are evaluating AI spend in 2026, stop asking vendors for seat discounts first. Ask for: 1. **Workflow definition** — which task is being completed end-to-end? 2. **Success criteria** — what counts as ready-to-use? 3. **All-in cost** — model + tools + human review + retries 4. **Escalation rate** — how often people still finish the job 5. **Time series** — is cost per success falling over 30/60/90 days? A useful internal scorecard looks like this: | Metric | Bad signal | Good signal | |---|---|---| | Useful work completed | Prompt volume up, output unused | Resolved tickets / merged PRs / closed reviews up | | Cost per success | Cheap tokens, expensive rework | Higher unit price, lower total completion cost | | Dependability | Constant human cleanup | Ready-to-use rate rising | | Scale economics | More usage, worse quality | More usage, lower cost per success | ## The cynical read OpenAI has a commercial reason to push this narrative. If buyers keep optimizing for lowest token price, cheaper or open models win more often. If buyers optimize for completed work and reduced review burden, frontier models and full product surfaces like ChatGPT Work become easier to defend. That does not make the framework wrong. It makes it self-serving and correct at the same time. Most companies are still buying AI like software licenses and measuring it like chat apps. OpenAI is arguing they should buy and measure it like labor infrastructure. For any team running agents across real systems, that is the right fight. ## Practical takeaways - Replace “AI adoption” dashboards with workflow completion dashboards. - Route models by total cost of success, not sticker price. - Track ready-to-use / needs-correction / needs-escalation on every production workflow. - Do not expand autonomy until dependability is explicit and measurable. - Re-evaluate model choice quarterly against the same task definition; brand loyalty is not a strategy. Source: [OpenAI — A scorecard for the AI age](https://openai.com/index/a-scorecard-for-the-ai-age/) (July 17, 2026).