← All posts

article

Evaluate retrieval before choosing another AI model

Test how business information reaches AI, with fixed questions, records and answer requirements, before blaming the model for every weak result.

An answer can fail before reasoning begins

A model cannot reconcile an invoice it never received. Business questions often cross email, customer records, job history and accounting. A weak answer can come from the retrieval path as well as the model. Changing the model without checking that path may leave the original problem untouched.

Write the answer requirements first

Choose one repeatable question and list the facts a correct response must contain. Use a hypothetical customer-context review: identify the correct business, recover a changed requirement, find related work and check the payment position. Include the source of a conflict rather than forcing records to agree. This is a test design example, not a customer result.

Make both approaches answer the same question

Hold the underlying records, access permissions and required information fixed. Record how each workflow identifies the customer and obtains the related records. An email address alone may not be enough to match accounting and CRM entries. If the information differs between runs, explain that difference rather than presenting the result as a controlled comparison.

Inspect the material returned to the model

Check whether the response includes only relevant records or large unrelated lists. Also check whether it loses a relationship that changes the answer. Smaller context is not automatically better. The aim is enough connected evidence to answer the question completely, without avoidable retrieval and repetition.

Score useful context transparently

Holiday's internal study distinguishes required data points from additional relevant linked context. Its completeness rubric can exceed 100% when a response includes every required fact and useful related information. That does not mean an answer is more than completely correct, or that a 115% score is a 115% improvement. Publish the rubric alongside a score so readers know what was counted.

Record boundaries, not only wins

Test missing records, stale information and contradictory sources. A retrieval system should preserve permission checks and indicate uncertainty. Record tokens, calls and charges separately, then inspect answer quality. Holiday's study is an internal comparison, not proof that its results will hold across every workload or model.

Use the result to choose the next change

If retrieval fails to bring back the right evidence, improve the context path first. If the evidence is complete but the response still misses the task, investigate model choice, instructions and evaluation. Holiday's Environment connects approved business information and relationships in a per-company environment. The test remains the business question and the answer it needs, rather than a demonstration of how many tools can be called.

Sources and further reading

Questions answered

Why test retrieval before changing an AI model?

An answer can be incomplete because relevant records or relationships never reach the model, even if the model could reason over them.

Can answer completeness exceed 100%?

Under Holiday's internal rubric, it can when all required data points and extra relevant linked context are returned. The rubric must be stated.

What should remain fixed in a retrieval comparison?

Keep the question, records, permissions and answer requirements consistent, and record model configuration and run conditions.