A token count answers one question
How much text did the model process? That is a useful question, but it is not the same as asking what the task cost. A workflow can process less text without reducing its invoice by the same percentage. The difference matters when a business compares AI architectures or budgets for recurring work.
Separate the categories before comparing the totals
Input, cached input and generated output can have different prices. Under a provider's reporting rules, cached input may be a subset of input rather than another quantity to add on top. Check the current provider documentation and the actual usage record. Cache eligibility, retention and charges depend on the model and service; one blended token total hides those differences.
Use an acceptable answer as the unit of work
A comparison should include the work required to finish the business task, including retries and tool calls. Define the required facts before running the test. If one response misses a related invoice and another finds it, the two responses are not interchangeable simply because both sound plausible. Quality checks belong beside the cost record.
Keep architecture and pricing separate
Holiday's internal efficiency study compares retrieval through separate MCP tools with retrieval through its connected business environment. The study describes fixed model and underlying records, with access architecture as the changed variable. It reports token use, agent-facing calls and answer completeness. Those measures help explain retrieval work. They should not be relabelled as a universal percentage reduction in actual bills.
A cache changes the calculation
The expanded traditional workflow in the study includes cached input. Cached tokens still contribute to processed text, but may be charged at a reduced provider rate. For a bill comparison, record the relevant prices and charges for each category. Measure latency separately as well. Neither a token reduction nor a shorter tool sequence establishes an identical saving in elapsed time.
Make the result reproducible
Keep the question, records, model configuration, permissions and quality standard consistent. Record the run conditions and any missing information. Repeat the comparison across tasks before treating it as a planning assumption. An internal benchmark is evidence about its tested conditions, not an independently audited customer outcome.
Read the commercial terms too
Holiday states that AI usage is passed through at provider cost without markup. That is a statement about usage charging, not free software or implementation. When choosing an approach, separate usage from implementation, hosting and other agreed charges. A useful budget connects the full cost to correctly completed work.