The penetration testing buyer's checklist

Not everything sold as a penetration test is one. A fair number are a vulnerability scan with a cover page, and the difference is almost never visible in the proposal — it surfaces months later, in a report nobody can act on. What follows is the long version of the buyer's checklist: what to settle before you approach anyone, what to ask, how to read the answers, and how to arrive on day one ready.

Decide what question you need answered

Most disappointing engagements were mis-specified before anyone was contacted. The buyer knew they needed "a pen test" and nothing more precise, so they received whatever the provider's default product happened to be — and defaults vary enormously between firms. Before you write to anyone, write down the question you need answered. In practice it is one of four.

• Do we have known, tool-detectable weaknesses? Answered by scanning with human validation. It will not find broken access control between roles, nor anything requiring an understanding of what your application is for.
• What would a competent attacker find within a bounded effort? What most people mean by a penetration test: a methodology worked against an agreed scope for an agreed period, following what looks interesting and stopping when the time is spent. Methodical but finite by construction.
• Can a specific consequence be achieved? Reach customer records, take administrative control of the tenant, cause a payment to settle incorrectly. Defined by goals rather than coverage, so the finding count may be small while the result is severe.
• Would we notice, and could we stop it? An exercise against the organisation rather than the asset — detection, escalation, response — carrying deconfliction overhead a scoped application test does not.

Then settle the audience. A report that satisfies an auditor is not automatically one your engineers can work from. Decide whether the output must stand up in front of an assessor, a customer's security review, an insurer, your own developers, or several at once — and say so when you ask for proposals. Retro-fitting an audience onto a finished report is the commonest source of rework.

Work out what you can actually give the testers

What you hand over changes both the cost and the value of the work — and the relationship runs opposite to most buyers' instincts. Withholding information does not reduce the work; it relocates it. A tester given no credentials spends the opening stretch inferring what you could have provided on the first morning: enumerating roles, reverse-engineering a token format, establishing which service actually enforces authorisation. That time comes out of testing, not out of the budget. The realism argument is weaker than it sounds, too — a real attacker is not time-boxed, so simulating their ignorance inside a bounded engagement mostly simulates their first week. Where that is the objective, choose it deliberately; as a default it is a poor trade.

Establish early which of these you can produce, and how long each takes:

• One working account per role, verified by someone who has personally logged in with it at the right privilege level.
• Two tenants, or two accounts per role, where isolation matters. Access-control testing without a second subject to reach is not testing.
• Realistic seeded data. An empty environment is close to untestable: with no records to reach, there is nothing to attempt unauthorised access to.
• Documentation — architecture notes, an API specification, a role and permission matrix even if it is a hand-maintained spreadsheet.
• Source or configuration access, if policy permits. It turns "this behaves oddly, is it exploitable?" into a question answered by reading.
• A named technical contact who can act, not relay — someone able to reset a locked account the same day.

If some of this is impossible, that is a legitimate constraint. Say so in the request rather than discovering it in week one, and expect the proposal to account for it.

Scoping inputs: what to bring to the first call

A provider who quotes from a count of URLs and IP addresses alone has not scoped anything. Effort tracks distinct behaviour, not inventory: forty endpoints behind one gateway with one authorisation model are a fraction of the work of two genuinely different applications. Your job on the call is to describe distinctness, and the things that make a nominally small system expensive.

• The asset list, grouped: which items share a framework, gateway, authentication mechanism and authorisation model, and which are bespoke.
• The role list, flagging roles that are cosmetic variations on the same grants. Access control is a matrix traversed, not a feature checked; each extra role adds a pass over every privileged operation plus new pairings to probe in both directions.
• Where the bespoke logic lives — anything that prices, moves money, grants entitlements or enforces a regulatory rule. The findings that matter cluster here, and it must be learned before it can be attacked.
• Multi-tenancy, and whether isolation is enforced in one place or per query, per service, per cache key.
• Anything unusual in authentication — custom tokens, home-grown single sign-on, federation, step-up, device binding, signed client-side state.
• Stateful workflows. Where outcomes depend on sequence, testers must reach each state before testing it and rebuild it after every destructive attempt — time invisible in a scope document and very visible in a schedule.
• Third-party integrations, and which may be exercised against a live counterparty.
• Environment: production or staging, and every known difference — authentication, feature flags, infrastructure, integrations.
• Platform obstacles: rate limiting, bot protection, geo-blocking, firewall rules, and whether source addresses can be allow-listed.
• Constraints and framework: windows, change control, blackout periods; and which control the test must evidence, if the output has a compliance audience.

The questions to ask every provider

Ask all of these, ask them of everyone, and record the answers side by side. The value is less in any single answer than in the comparison.

Who performs the testing, and what hands-on qualifications do they hold? You want names, or at least seniority and evidence of practical rather than multiple-choice assessment. "Assigned to the team" means you are buying capacity, and capacity is not a skill level.

May I see an anonymised sample report? The most informative request you can make, because the report is the thing you are buying. Nobody who writes good reports minds showing one. To see what a complete finding looks like before you compare, read a full report end to end.

How much of this is manual, and how is the effort split? Ask how many days are hands-on-keyboard testing as distinct from scanning, triage, reporting and project management. A proposal that will not separate those is hiding the ratio.

Will you test business logic and access control, or only known vulnerability classes? Ask for a concrete example of a logic flaw found in a comparable system, described without identifying the client. Vague answers here are the most reliable predictor of a scanner-shaped report.

How do you scope, and is the price fixed before work starts? Scope and fee in writing, with a stated mechanism for anything found to be larger than described.

How are critical findings communicated during the test? Same-day notification by an agreed channel, with enough detail to act on — not a line in a report four weeks later. Ask too about the rules of engagement: permitted and prohibited techniques, source addresses, hours, escalation contacts, what happens if something breaks, and how evidence containing your data is handled and destroyed.

Can you map findings to our framework? ISO 27001, SOC 2, PCI DSS, DORA, NIS2 — if the report has an audit audience, the mapping belongs in the deliverable, not reconstructed by you afterwards.

What will you tell us you did not have time to test? A provider who plans to state coverage limits distinguishes "we tested this and found nothing" from "we did not reach this" — the difference between assurance and a gap you do not know about.

Reading the answers: red flags and green flags

Certain responses are reliably bad news. A quote derived solely from counts of IPs or URLs with no conversation about goals — the provider has priced inventory, which means they intend to sample. No named tester. A sample report that is a severity table with no reproduction steps or business impact, the single clearest indicator of scan output. No retest, or a retest sold as an expensive add-on, which says the engagement is designed to end at delivery. Vague answers on business logic and access control. Unwillingness to put scope and price in writing. And a finding count used as a selling point, which rewards volume over severity and usually means informational noise padding the total.

The inverse is equally legible: senior, named testers with practical certifications; a sample report showing chained findings, real evidence and concrete fixes rather than generic advice; a fixed scope and price in writing with the retest included; clear rules of engagement and same-day reporting of critical issues. Add one more — questions asked of you during scoping that you had not considered. A provider probing your authorisation model before quoting is demonstrating method.

How to compare proposals fairly

Three proposals for "a penetration test of our platform" can differ by a wide multiple and all be honest, because they are offers to do different things. Normalise before comparing: put them on one page and align four columns — scope (asset by asset, roles, environment), depth (which of the four questions it answers), manual effort (days hands-on, excluding reporting), and deliverables (report, mapping, debrief, retest, re-issued report). Differences then read as scope and depth differences rather than price differences.

Then check the exclusions, where the cheapest proposal usually reveals itself: roles not covered, authenticated areas out of scope, the API excluded because only the web interface was named, denial-of-service testing excluded (reasonable), business-logic testing excluded (not), retest or debrief excluded. Price per day is a poor comparator; price per day of manual testing on the scope you care about is a usable one.

PENETRATION TESTING BUYER'S CHECKLIST

BEFORE YOU APPROACH ANYONE
[ ] The question this report must answer
[ ] The audience: engineers / auditor / customer / insurer / board
[ ] Depth: validated scan / time-boxed manual / objective-driven /
    adversary simulation
[ ] Framework and control the test must evidence, if any
[ ] What we can hand over: credentials, docs, API spec, source
[ ] Constraints we cannot remove: windows, change control, blackouts

SCOPING INPUTS TO BRING
[ ] Asset list grouped by shared framework / gateway / auth model
[ ] Role and permission matrix, duplicates flagged
[ ] Where bespoke logic lives (money, entitlements, regulated rules)
[ ] Multi-tenancy model and where isolation is enforced
[ ] Non-standard authentication
[ ] Stateful workflows that must be reached before testing
[ ] Third-party integrations, and which may be exercised
[ ] Environment choice + known staging/production differences
[ ] Platform obstacles: rate limits, WAF, bot protection, geo-blocking

ASK EVERY PROVIDER
[ ] Who tests, and what hands-on qualifications do they hold?
[ ] May we see an anonymised sample report?
[ ] How many days are manual testing, excluding reporting?
[ ] Will you find business-logic and access-control flaws?
[ ] How do you scope, and is the price fixed in writing?
[ ] How are critical findings communicated during the test?
[ ] Rules of engagement and data-handling terms?
[ ] Is a remediation retest included, and on what terms?
[ ] Can you map findings to our framework?
[ ] What will you tell us you did not have time to test?

DELIVERABLE MUST CONTAIN
[ ] Executive summary readable without jargon
[ ] Scope and methodology, including what was NOT tested
[ ] Per finding: severity, component, evidence, reproduction steps,
    business impact, specific fix
[ ] Root-cause grouping, not just an instance list
[ ] Remediation plan ranked by real risk, not raw CVSS
[ ] Framework mapping, if the audience requires it
[ ] Live debrief with the tester who did the work

RETEST TERMS, IN WRITING
[ ] Included at no extra cost
[ ] Window after delivery
[ ] All findings, or only high and critical?
[ ] Re-issued report, or signed confirmation?
[ ] Performed by the original tester?
[ ] What happens if a fix is incomplete

BEFORE DAY ONE
[ ] One working account per role, logged into by a human
[ ] Two tenants / accounts per role, data distinguishable
[ ] Realistic seeded data in every account
[ ] Source addresses allow-listed; WAF and rate limits handled
[ ] Named technical contact who can act, not relay
[ ] Channel agreed for critical findings
[ ] Remediation owners identified
[ ] Retest date provisionally in the calendar

What the deliverable must contain

The report is the product. Specify it in the contract rather than hoping for it.

An executive summary stating overall risk and themes in language a non-technical reader can act on. A scope and methodology section recording what was tested, how, and explicitly what was not — coverage gaps stated as gaps. Findings each carrying a severity with its reasoning, the affected component, evidence, reproduction steps precise enough for your engineer to confirm the issue unaided, business impact in your terms, and a specific fix rather than a textbook paragraph. Root-cause grouping, because one fix frequently closes several findings and a flat instance list hides that. A remediation plan ordered by real risk — severity combined with the value of what is exposed and how reachable it is — not by raw CVSS. Framework mapping where the audience needs it. And a live debrief with the person who did the testing, which is where the questions you did not know to ask get answered.

Insist on two further things: that chained findings are described as chains, with the sequence set out, and that plausible attack paths the tester investigated and disproved are recorded. Knowing something was tested and did not work is assurance you paid for.

Retest terms, in writing

The retest is the only part of the engagement that proves the money achieved anything. An unretested report is a list of things you believe you fixed.

Settle six points before signing. Inclusion, at no additional cost. The window — how long after delivery you may use it, and whether that clock is realistic against your release cycle. Coverage — all findings, or only high and critical? A retest limited to criticals leaves the mediums permanently unverified. Output — a re-issued report with findings marked resolved, or a separate confirmation you can hand to an auditor or customer. Who performs it: the original tester retests faster and recognises a fix that merely moves the problem. And what happens when a fix is incomplete — whether a second attempt is covered, and whether partial remediation is reported as resolved, mitigated or open, because those three words mean very different things to an assessor.

What to prepare so the engagement is not wasted

Engagements rarely fail because the tester was weak. They fail because the first part was spent waiting. Testing is cumulative — a tester holds a model of the system in their head and loses time rebuilding context after every interruption — so lost time is not recovered later, and a narrow daily window does not yield a proportionate fraction of the testing. It yields less, because the overhead recurs.

Finish the following before the start date: accounts provisioned and tested by a human at every role, with enrolment and multi-factor setup completed; two tenants with distinguishable seeded data; source addresses allow-listed through the firewall, bot protection and rate limiter, with someone able to adjust them; a staging environment that is actually up, with no release landing mid-test; documentation sent, not promised; a named contact reachable during testing hours; and an agreed channel for critical findings, carrying people who can authorise an emergency change.

Then prepare for the part after delivery, where most of the value is realised or lost. Decide who owns remediation for each system, and how findings will be tracked — your existing ticketing system with the report reference on each ticket, not a spreadsheet that goes stale. Book a provisional retest date before the test begins; it is the simplest mechanism for making sure fixes land, and it converts a document into a change in your security position. Keep the report and the retest confirmation together: your auditor and your customers' security reviewers will ask for both.

If you want the condensed version to take into a meeting, the one-page buyer's checklist puts the questions, red flags and green flags on a single sheet. For how we scope and run engagements, see our penetration testing service.