What an AI penetration test actually covers

An AI security assessment can cover administration services, data retrieval, agent actions, prompt injection and ordinary application controls. These are related but distinct workstreams. Define which are in scope and which evidence each will produce, rather than assuming one label fixes the depth or method.

AI-powered testing and testing AI applications

These describe different things. AI-powered penetration testing uses AI to investigate software; the target might have no AI features at all. An AI security assessment examines an application that contains AI: its permissions, data paths, administration services and agent actions. The assessment may itself use manual, automated or combined techniques.

Pixel’s service covers the second question. Agree which AI components and accounts are in scope and what evidence you need. If you are comparing delivery methods, read our guide to automated and specialist testing.

Why "test our AI" is an ambiguous request

Whoever asks almost always has one specific worry in mind and almost never says which. The same handful come up in our scoping calls, and they call for genuinely different work.

"Can it be talked into saying something embarrassing?" A question about the model's behaviour — refusals, tone, bias, unsafe output. It is answered by safety red-teaming against a content policy: a content exercise, not a security one.

"Is the thing any good?" Accuracy, hallucination rate, retrieval quality — an evaluation problem, measured against a labelled dataset with a scoring harness by the team that built the feature.

"Are we allowed to ship this?" Governance and obligation: documentation, risk classification, human oversight, record-keeping. The reference points are the NIST AI Risk Management Framework, ISO/IEC 42001 and, where it applies, the EU AI Act. Assurance work, usually with an auditor or legal counsel involved.

"Could someone use this feature to get data or control they should not have?" That one is a penetration test — authorisation, trust boundaries, credentials and reachable attack surface, answered by someone attempting the attacks and showing you what worked.

These are not degrees of the same thing. A provider can do an excellent job on one and leave the other three untouched, and because all four get sold as "AI security testing", you can buy one believing you received another.

The first risk is usually the administration plane

Almost all public AI security conversation is about prompt injection. It is a real attack class, genuinely hard to fix, and it demonstrates beautifully in a meeting. But an AI feature inside a business application is not a model. It is a small distributed system with its own administration surface: something stores the prompt configuration, something holds the credentials the pipeline uses to reach object storage and the surrounding business systems, something creates and trains and deletes models, something triggers the jobs that pull your data in. All of it is configuration, all of it is exposed over HTTP, and we see it reachable by users never meant to hold it.

Prompt injection can expose data or cause unsafe tool use when application controls permit it. Administrative writes can provide another direct route to configuration and credentials, depending on their scope. Test both according to reachable capabilities and intended permissions. Our admin-plane analysis explains why we include that surface early; it is not a reason to exclude prompt-injection risks.

One engagement makes it concrete. On an enterprise SaaS platform with embedded generative-AI features, testing as an ordinary non-administrator tenant user, we found the AI administration services accepting privileged writes from a plain business user. The token asserted non-admin, and the admin interface correctly hid its screens — client-side only. The AI services behind it enforced no server-side administrator check and accepted writes on coarse service-level scopes alone. That user could write and read back the AI pipeline's object-storage credential set, effectively choosing where the pipeline's training and inference data was written; write the generative-AI prompt configuration; write a separate business-system credential store; trigger the data-ingestion job; and operate the full model lifecycle.

The comparison with the neighbouring identity service showed that the platform could enforce an administrator gate. The AI services required their own effective permission policy. A test confined to model responses would not cover those API operations; this does not establish that prompt injection was absent or that one blanket administrator role is appropriate for every AI operation.

What is actually in scope

Six areas. The first two produce most severe findings.

The AI administration plane

Enumerate every administrative operation the AI component exposes — prompt and template configuration, extension points where tenant-specific instructions get spliced in, credential and connection records, model creation and training and deletion, deployment, ingestion triggers — then attempt each as the least-privileged user who can authenticate. Read and write are tested separately, because they routinely differ. What you want back is an operation-by-operation record of what the server enforced, not an assurance that the area is role-protected.

Authorisation and tenant isolation of AI data

AI features generate and store a lot: conversation threads, ingested documents, embeddings, generated artefacts. These usually live in a new service with its own datastore and identifier space, wired to the session for authentication but often not into the platform's existing object-authorisation model. So the tester creates content as two users in the same role and as users in two tenants, swaps identifiers both ways, and tests retrieval, metadata, export and deletion separately. Derived artefacts deserve their own attention: a generated summary can be more sensitive than any record behind it, because the analysis is the value.

The agent and tool authorisation boundary

If the feature can do things — call internal APIs, read a record, raise a ticket, run a query — the critical question is whose authority it acts under. Does each tool call carry the end user's identity and get authorised against it, or does the agent hold one service identity broad enough to serve everybody? The second pattern is common, and it makes the agent a confused deputy: a user entitled to one region's data asks a question and the tool answers from all of it. Testing it means enumerating the tool list, then attempting through the agent what the requesting user could not do directly. The OWASP Agentic Security Initiative is a reasonable public reference here.

Prompt injection, direct and indirect

Direct injection is the user overriding instructions in their own turn. Indirect injection is the interesting one: instructions planted in content the system ingests — a document, a ticket comment, a web page, a file name — and later treated as authoritative. That is where injection stops being a party trick, because the attacker needs no account to reach it.

Insist that injection be tested for consequence. "The model produced disallowed text" is a weak finding; "content planted in an ingested document made the agent call a tool and return another user's data" is a real one. The payload is not the finding; what it reached is.

The data path and retention

Follow the data. Where prompts, conversations, uploads and outputs go, which components and third parties see them, what is logged and in what detail, how long each store keeps it, what happens on deletion. Logs and traces are a recurring problem, because observability tooling for AI pipelines captures full prompt and completion bodies and then sits somewhere with far broader access than the application. Check whether the storage behind artefacts is reachable with no session, and whether its credentials are long-lived and broadly scoped.

The ordinary web and API security of the AI endpoints

The unglamorous part, and skipping it is a mistake. AI endpoints are web endpoints: authentication on every route including the streaming ones, authorisation on each, rate limiting and cost controls, input size limits, error handling that leaks no stack traces or internal hostnames, CORS configuration, no secrets in the client bundle. AI features are often built fast and beside the main application rather than inside it, so they miss the framework-level protections the rest of the platform gets free. Our approach to this layer is on our methodology page; the AI work sits on top of it.

What an AI penetration test is not

The other three kinds of work are real, and someone should be doing them.

Not model safety red-teaming. Whether the assistant can be induced to produce hateful, biased or off-policy content needs a written content policy to measure against, a large volume of attempts and reviewers with the right background. We report what we stumble into, but if that is your real concern you want a safety red-team exercise, usually with your trust-and-safety or legal function involved.

Not a model evaluation. Accuracy, hallucination rate, retrieval relevance and regression between model versions are measured with a benchmark harness and a labelled dataset, continuously, by the team that owns the feature.

Not a compliance certification. A test produces technical evidence that can support an assurance activity. It does not make you compliant with anything, and we are not an auditor, assessor or certification body. If an obligation applies to you, agree what your evidence must show with whoever signs it off before scoping.

If the feature calls a hosted model, distinguish your application, orchestration and data paths from provider-managed infrastructure. Your authorisation does not automatically cover the provider’s systems. Any testing of those systems needs the provider’s applicable permission and testing rules; an authorised research programme may define a separate scope.

What you have to give the testers

AI engagements are unusually sensitive to preparation, because so much of the value sits in authorisation testing — impossible without accounts.

Two accounts per role, plus one administrator. Two users at the same privilege level in one tenant is what makes horizontal access-control testing possible — one reaching the other's conversations and documents. One account per role tests only the vertical boundaries, the easier half. The administrator account feels counter-intuitive, but a tester must see the administration surface working as intended to enumerate it and then attempt it from below.

Two tenants, if you are multi-tenant. Cross-tenant isolation cannot be tested from inside one tenant. If provisioning a second is slow, start before booking.

A populated environment. An empty knowledge base and no conversation history means there is nothing to reach across a boundary. Clearly-marked synthetic data in the corpus is worth more than production access, and avoids a whole category of difficult conversation.

The agent's tool list, with parameters. Every tool the model can invoke, what it does, what it takes and what identity it runs as. If no such document exists, writing it is valuable on its own, and its absence is usually a finding in waiting.

A written statement of what the feature is allowed to do. Most often missing, and the item that most changes the report: which roles may use it, which data domains it may touch, which actions it may take on a user's behalf, what it must refuse. Without that, a tester can report what the system does but not where it diverges from what you intended — and that gap is what an authorisation finding is.

Architecture notes and a reachable engineer. Where the orchestration layer sits, which stores hold prompts, embeddings and artefacts, which credentials the pipeline holds, whether any part is vendor-managed. Then decide in writing whether the tester may trigger training runs and ingestion jobs — not read-only operations — and who to call when something sticks.

How to tell a real AI test from a prompt-injection demo

You can usually tell from the proposal.

What does the scoping conversation ask about? A real one asks about roles, tenants, the tool list, where prompts and credentials live, which operations the feature performs and under whose identity. A thin one asks for a URL and a login, then quotes.

Does the scope name the administration surface? If prompt configuration, credential records, model lifecycle and ingestion are absent from the scope document, they will be absent from the report.

How many accounts and tenants do they ask for? The quickest signal available. A provider who does not ask for two users per role has not planned to test horizontal access control, whatever the methodology section promises.

Are findings stated as consequences or outputs? "The model can be jailbroken with this string" is a demonstration. "An unprivileged user could replace the pipeline's storage credentials, and here is the request" is a finding. Ask to read a redacted one before you sign — ours are in the sample report.

Does the retest cover the authorisation findings? Access-control fixes get applied to the endpoint reported and not to its siblings, so a retest re-attempting the whole administrative surface is worth far more than one re-checking a single request. And a provider who says plainly what they are not covering is easier to trust on what they do claim.

A scoping checklist

Work through this before you approach anyone.

DECIDE FIRST
  [ ] Which question am I paying to answer?
      security / safety / accuracy / governance
  [ ] Worst realistic outcome if the feature is abused
  [ ] Which environment, and how closely it matches production

SCOPE — name these, or they will not be tested
  [ ] Admin plane: prompt and template configuration, extension
      points, credential and connection records, model lifecycle,
      deployment, ingestion triggers
  [ ] Authorisation and tenant isolation for conversations, documents,
      embeddings, generated artefacts (read, write, export, delete)
  [ ] Agent/tool boundary: whose identity each tool runs as
  [ ] Prompt injection, direct and indirect, tested for consequence
  [ ] Data path, logging and retention, incl. observability stores
  [ ] Web/API security of the AI endpoints: auth, authorisation,
      rate limiting, input limits, error disclosure, CORS
  [ ] Out of scope, stated: third-party model infrastructure,
      accuracy evaluation, content red-teaming, certification

PROVIDE
  [ ] Two accounts per role, same tenant; plus one administrator
  [ ] Two tenants, if multi-tenant
  [ ] A populated corpus of clearly-marked synthetic data
  [ ] The tool list: parameters, and the identity each runs as
  [ ] What the feature may do, and must refuse, in writing
  [ ] Architecture notes and a named, reachable engineer
  [ ] Written agreement on training and ingestion calls

AGREE BEFORE SIGNING
  [ ] Findings carry the request, the account and the consequence
  [ ] Retest re-attempts the whole administrative surface
  [ ] Coverage mapped to a public reference on request
  [ ] Handling and retention of data the testing produces

The through-line is simple enough to carry into a meeting. An AI feature is a new system with a new administration surface, bolted onto an application whose permission model nobody revisited when it was switched on. The test you want establishes who can reach that surface, whose authority the feature acts under, and whether the data it generates respects the boundaries the rest of your product does. Prompt injection belongs in the scope and we test it — but it is a route into those questions, not the question. Our AI penetration testing page describes how we run it.