A useful AI demonstration shows what the assistant can do. A security evaluation also needs to show what it cannot do, who makes that decision and what evidence remains afterwards.
That is the question I would put ahead of a feature checklist: when an AI requests an action outside its authority, what actually stops it?
A prompt that tells an assistant to respect company policy is not the same as a software control that rejects an unauthorised operation. The difference becomes visible when you test a request the model is willing to make but the business has not allowed.
Start with a permission matrix, not a clever prompt
For an evaluation, write down a small set of permitted and prohibited operations. Identify the requesting person, the AI role, the records involved and the action being attempted.
Use test records rather than live customer information. The aim is to compare the intended permission decision with the actual result, not to prove a point by exposing data.
Here is a hypothetical example. A person may read a customer account and ask an AI role to prepare an internal summary. That person is not allowed to change billing details. The AI role may read approved records but has no authority to make financial changes.
Reading the approved account should succeed. Changing its billing details should not. Neither a confident model response nor an urgent instruction changes that boundary.
Test the person and the AI role separately
Do not test only the case where both identities have matching access. That can hide which permission set the system actually enforces.
- Person allowed, AI role restricted: ask for an operation the person may perform but the AI role may not. The person's access should not silently expand the AI role's authority.
- AI role allowed, person restricted: ask through a person who lacks access to the record. The AI role's broader responsibilities should not become a shortcut around that person's permissions.
- Both allowed: confirm that the intended operation works and uses the approved connection.
- Neither allowed: confirm that the operation is rejected without returning the protected information.
In each case, check the underlying result. An assistant saying it refused is not evidence that no operation occurred. An assistant saying it completed something is not evidence that the change succeeded.
Find the enforcement point
Ask the supplier to show where the authorization decision happens. Is the model deciding whether to obey a policy, or does application code independently check the requested operation before allowing it?
Holiday is a secure business knowledge platform. Its trust architecture places that decision outside the model: every tool call is checked against both the AI identity's authority and the requesting user's permissions. The model can request an action, but it cannot grant itself access.
AI operating inside Holiday uses approved connections assigned to its role rather than inheriting a person's credentials. Those identities and scoped connections matter because the business needs to know what authority was used, not simply which employee happened to be signed in.
The independent knowledge environment matters too. Holiday brings approved business information into a separate, single-tenant environment, with permissions enforced in application code. Connecting useful context does not mean giving every AI role unrestricted access to each source application.
Check the next operation, not just the first one
A task can begin within scope and then request something different. Reading a record, updating it and sending information elsewhere are separate operations.
In your test, let an approved read succeed. Then request an unapproved change. Confirm that the earlier success does not carry forward as blanket authority for the next tool call.
Repeat the relevant checks when responsibilities or access change. The point is to test the deployed boundary, not to rely forever on a demonstration performed with different roles and settings.
Agree what evidence the review needs
For each test, capture the expected decision, the actual decision and the verified outcome. Ask how the operating record connects the requesting person, AI identity, approved credential, records accessed, attempted action and any required human review.
Holiday's governance reporting follows work from instruction to outcome. Its standard customer export includes audit logs at no charge. Active retention periods and any SIEM delivery requirements are agreed for the customer environment during evaluation, so ask for those details rather than assuming them.
These are evaluation questions, not a claim that a short checklist proves an entire deployment secure. Authentication settings, model providers, processing locations and other controls still need their own review. Holiday's Trust & Security page separates implemented controls from deployment-specific options and work in progress.
AI should be able to use business knowledge without becoming the authority that decides its own access. Test the denied operation as carefully as the successful one. That is where the boundary becomes more than a promise.
Bring your permission scenarios to Holiday and we can walk through the controls for your proposed environment.