Skip to main content
Security templateReviewed

Penetration testing provider evaluation: a weighted scorecard for choosing who tests your application

Check each provider against pass or fail gates for accreditation and independence, then score the rest on eight criteria, from relevant experience and sample-report quality to retest terms and exclusions.

The weights are yours to set, every score carries its evidence, and a moderation sheet settles differences when several people score.

Weighted provider scorecard with gates and a moderation sheet

Copy the scorecard into a spreadsheet or shared document. Fill in Part A and sign off the weights in Part C before any proposal is opened, then check each provider against the Part B gates.

How to use it

Score the providers that pass with the Part D anchors, total them in Part E, and settle differences between evaluators with the Part F moderation sheet. Part G is an invented worked example.

penetration-testing-provider-scorecard.txt

Introduction2 lines

PENETRATION TESTING PROVIDER SCORECARD Copy this into a spreadsheet or a shared document, one copy per evaluation. Fill in Part A and sign off the weights in Part C before anyone opens a proposal. Send every provider the "Ask for" lines in Part D, in the same words and with the same reply date: together they are your supplier questionnaire.

What a penetration testing provider evaluation has to settle

Choosing a penetration testing company decides who will look at your application, how closely, and what you will hold at the end.

Two providers can quote for the same scope and return very different things: one a report your developers can reproduce and a customer can follow, the other a scan export with a cover page. The UK National Cyber Security Centre (NCSC) gives the reason in its penetration testing guidance: penetration tests cannot be entirely procedural, an exhaustive set of test cases cannot be drawn up, and so the quality of a penetration test is closely linked to the abilities of the testers involved.

A scorecard turns that judgement into something you can write down, share and defend later. This one works in four steps. Pass or fail gates remove any provider that cannot do the job you were asked to buy. Weights, set before any proposal is opened, record what matters most for this application.

Each remaining provider is scored from 0 to 5 on eight criteria against written anchors, with the evidence noted beside every score. Where more than one person scores, a moderation sheet settles the differences on evidence rather than on seniority.

It compares providers, not prices. Score first, then line up the quotes from the providers still standing on the same written scope, which is what the budget worksheet in the cost guide is for. A sample report is one of the eight criteria here; the annotated report example reads one in depth. If what you are buying is software rather than testing, the vendor security questionnaire is the better tool.

Pass or fail gates come before any score

Some requirements are not a matter of degree.

If the person who asked for the test named an accreditation, required an independent tester or set conditions on where your data may go, a provider either meets that or it does not, and a strong score elsewhere cannot make up for it.

Read the exact wording of the request first, because the gates come from that wording rather than from anyone's marketing. Then check every provider against every gate, and do not score one that fails.

Accreditations: check with the body that grants them

CREST awards accreditation to organisations that evidence compliance with its accreditation standards through an application, evidence submission and review, and accredited companies undergo periodic reassessment to keep it. Its accreditation disciplines include penetration testing and vulnerability assessment, so confirm that the accreditation covers the work you are buying, and check it with CREST rather than from a logo on a proposal.

Card payments bring two roles defined by the PCI Security Standards Council. An Approved Scanning Vendor (ASV) conducts external vulnerability scanning to validate adherence with the external scanning requirements of PCI DSS Requirement 11.3.2, and its scan solution is tested and approved by the Council before the vendor is added to the Council's list.

Qualified Security Assessor (QSA) companies are independent security organisations qualified by the Council to validate an entity's adherence to PCI DSS. On both pages, the Council advises a client, each time it engages one, to check the list and confirm the company has kept its status.

A government buyer may require IRAP. Where one does, ask that buyer which providers it will accept before you score anyone.

Clever Ops is not CREST accredited, so we refer that work to a provider that is. Clever Ops is not a PCI Approved Scanning Vendor or a Qualified Security Assessor, so a PCI scan or assessment goes to a provider that is one. Where a government buyer requires IRAP, Clever Ops does not meet that requirement, and the work goes to a provider that does.

Independence and the other gates

Independence becomes a gate when the request asks for an independent test. A provider that built, hosts or maintains the system is checking its own work, however capable it is. When Clever Ops built or hosts a system, we call the work a security review rather than an independent test, and the report says who built and who hosts it.

That review can be the quicker route to fixes before a release, but it does not answer a customer or auditor who asked for independence.

Four more gates apply to every provider:

  • Authorisation: testing starts only after the owner of each target has signed an authorisation, and runs to written rules of engagement.
  • Service fit: the provider tests the kind of target you have, such as a web application and its API rather than an office network, and offers what the requester asked for rather than a substitute such as an unattended automated scan.
  • Evidence conditions: any rule your contract or policy sets on where test evidence is stored or processed is met, in writing.
  • Insurance minimum: if your contract sets minimum cover for suppliers, the provider's certificates meet it.

Where Clever Ops fails the service-fit gate

Clever Ops tests what customers reach through a browser or an API. Internal networks, Active Directory, red teaming, social engineering, physical testing and load or denial-of-service testing fail the service-fit gate for us, and belong with a specialist in that work.

Setting the weights before anyone opens a proposal

Weights state what matters most for this particular test, so they belong to the buyer and are set before any proposal is read.

Set after the scores are visible, they can be bent, without anyone meaning to, towards a provider someone already prefers. Write them down, have the person who owns the decision sign and date them, and keep them with the sheets.

The starting spread gives 20 of 100 points to sample-report quality, because the report is what you keep: your developers fix from it, a customer reads it and the retest updates it. Relevant application experience, methodology, and exclusions and scope fit take 15 each, since together they decide whether testing reaches the parts of your application that carry the risk.

Independence, data handling and retest terms take 10 each. Insurance evidence takes 5, because it is closer to a yes or no than a scale; where your contract sets minimum cover, that minimum is already a gate.

Move points between criteria rather than adding them, so the total stays at 100 and every increase is paid for somewhere else. Some reasons to move them:

  • Health, financial or other sensitive records inside the scope: more points to data handling.
  • A customer, insurer or auditor wants evidence that findings were fixed: more points to retest terms.
  • Several customer organisations in one database, payment flows, partner APIs or an AI agent that can take actions: more points to experience and methodology.
  • Independence required by the request: it is already a gate, so its weight now scores what remains, such as products the provider resells or an offer to fix its own findings.

Changing a weight after proposals arrive

If the team later agrees a weight was wrong, change it in the open. Record the reason, then re-score every provider under the new weight, not only the provider the change would help. The worked example further down shows how far a change of weights can move a decision.

The eight criteria, and what earns a high score

The template gives anchors for scores of 0, 1, 3 and 5 on each criterion, and a red-flag question to ask when an answer falls short.

The notes below explain what each criterion measures and where the evidence for it comes from.

Relevant application experience

The NCSC says third-party penetration tests should be performed by qualified and experienced staff only. 1 min read

Sample-report quality

The sample report is the nearest thing to what you will receive, which is why it carries the most weight in the starting spread. 1 min read

Methodology

A method earns its score when it is written down and the report can be checked against it. 1 min read

Independence

Past the gate, independence comes in degrees. 1 min read

Data handling

Test evidence is sensitive in its own right. 1 min read

Insurance evidence

Ask every provider for a current certificate of currency for each type of cover your contract asks of suppliers, rather than accepting a line in a proposal that says cover is held. 1 min read

Retest terms

The retest is where a finding becomes a fixed finding someone else can rely on. 1 min read

Exclusions and scope fit

Every test leaves something out, and a written exclusions list shows you what. 1 min read

Moderating the scores when several people evaluate

Two evaluators reading the same proposal can land two points apart, and the gap is information: one of them has seen something the other missed, or has read a promise as a commitment.

Moderation finds out which before the scores become a decision, and the template's moderation sheet records each step.

  • Score alone first. Each evaluator fills in a sheet without seeing anyone else's, with one line of evidence beside every score: a page of the proposal, a finding in the sample report or a written answer.
  • Declare relationships before scoring. A former employer, a friend at the firm or a current supplier does not disqualify anyone, but the moderator records it and may ask that person to sit out scoring that provider.
  • Discuss any spread of two points or more. Each evaluator shows the evidence behind their score, and the agreed score follows the evidence, not a vote or the most senior person present.
  • Settle a one-point spread cautiously: take the lower score unless an evidence line supports the higher one.
  • Record dissent. An evaluator who still disagrees adds one line with their name and reason.
  • Keep everything. The signed weights, the moderated scores and the evidence lines explain the decision to anyone who asks later, including the provider you did not choose.

The moderator runs the process

The moderator collects the sheets, chairs the discussion and records the agreed scores, and does not score. In a small business the moderator may be the owner. What matters is that evaluators commit their scores before they hear the owner's view.

Worked example: three providers and two sets of weights

This is an illustration.

The business, the three providers and every score are invented, and the providers are named only Provider A, Provider B and Provider C. Part G of the template sets out the same sheet line by line.

A software business runs a booking and records platform for allied health clinics: a web application, its API, three roles and several clinics sharing one database. A hospital group considering the platform sent a supplier review asking for an independent penetration test of the application and API, and evidence that high-risk findings were fixed. It named no accreditation.

Provider C is the agency that built the platform. Its offer of a security review could serve the business well before a release, but the hospital group asked for independence, so Provider C failed that gate and was not scored. Providers A and B passed every gate.

Before any proposal was opened, the business moved points to data handling, now 20, because the platform holds clinical notes, and to retest terms, now 15, because the hospital group wanted evidence of fixes. Experience, methodology and sample-report quality gave up 5 points each. Three evaluators then scored, and moderation settled these scores out of 5:

  • Provider A:experience 5, sample report 5, methodology 4, independence 5, data handling 2, insurance evidence 3, retest terms 2, exclusions and scope fit 4.
  • Provider B:experience 3, sample report 3, methodology 3, independence 5, data handling 5, insurance evidence 5, retest terms 5, exclusions and scope fit 3.

How the weights moved the decision

Under the business's own weights, Provider B totals 80 and Provider A 72. Under the starting spread, the same scores give Provider A 80 and Provider B 74. Neither total is wrong. The weights record that, for this platform and this customer, how evidence is handled and how fixes are proven matter more than the depth of the sample report, and they were signed before anyone knew which provider that would favour.

What moderation and follow-up changed

Moderation changed one score. On Provider A's retest terms the evaluators gave 4, 2 and 1. The evaluator who gave 4 had read "retest included" in the proposal summary; the terms section limited the retest to Critical and High findings and left the request window unwritten. The agreed score was 2, with the terms section recorded as the evidence.

Evidence can still move a provider. Had Provider A answered the red-flag questions with written data handling and retest terms earning a 4 on each, it would have totalled 86 under the business's weights and led again. The sheet rewards what a provider can show, not what it says it does.

Turning the totals into a decision

The totals rank the providers.

These rules turn the ranking into a choice you can explain.

  • Treat a gap of five points or fewer as a tie. Break it on the criterion with the highest weight, or put the same follow-up question to both providers.
  • Hold on any score of 0 or 1. A leading total with a 1 on data handling still carries that gap, so ask the red-flag question for that criterion and re-score on the written answer.
  • Read the leading provider's weakest criterion before you sign. It is where the engagement has the least evidence behind it.
  • Compare price last, for the providers still standing, on the same written scope. A lower quote for a narrower scope buys less testing, not cheaper testing.
  • Write the choice and the reason down in one sentence, and keep it with the sheets and the contract.

What the total does not tell you

A scorecard does not make the decision, and a high total is not a certificate for the provider. It makes the reasons visible, so the person who asked for the test can check them and the next evaluation can start from them.

Putting Clever Ops through the same scorecard

If Clever Ops is on your list, score it on the same sheet as everyone else, from what we can show you in writing.

Where a gate rules us out, that is the right result. For the criteria, this is what you can ask Clever Ops for and check:

  • Sample report: a report on a lab application Clever Ops controls, which shows the format. It is not client work.
  • Methodology: the method page sets out how testing runs, phase by phase, with checks drawn from WSTG v4.2 and requirements from ASVS 5.0. A web application test runs from test accounts in each role the scope names and calls the API directly as well as through the screens.
  • Data handling: evidence has session tokens, passwords, keys and personal details redacted, is stored encrypted in Australia and deleted on the schedule set in the data handling terms, and is processed outside Australia only where your statement of work approves it.
  • Retest terms: one consolidated retest of the reported findings, requested within the window set in your statement of work, after which each finding is marked fixed, partially fixed, still reproducible or not retested.
  • Exclusions and scope fit: a written scope and quote naming the targets, environment, routes, roles and exclusions, and what the retest covers, with testing starting once the owner of each target has signed an authorisation.

If another provider scores higher

Score what we send against the anchors like any other proposal. If another provider comes out ahead on your weights, choose them; the scorecard is built so that you can.

Frequently asked questions

Start with pass or fail gates: any accreditation, independence or data condition the person who asked for the test has set, plus signed authorisation and a service that fits your target. Then score each remaining provider from 0 to 5 on relevant experience, sample-report quality, methodology, independence, data handling, insurance evidence, retest terms, and exclusions and scope fit, using weights you set before reading any proposal. Note the evidence beside each score, moderate when several people score, and compare price last.

Who will do the testing, and what have they tested that resembles your application? Can you see a complete sample report, and the written method? Did they build, host or sell into your system? Where is evidence stored, and when is it deleted? Can they supply certificates of currency for the cover your contract requires? What does the retest cover, and what does the scope leave out? The "Ask for" lines in Part D of the scorecard put these into a supplier questionnaire you can send as it stands.

No, it is a gate. If a tender, contract or customer requires an accreditation such as CREST, or requires a PCI Approved Scanning Vendor or Qualified Security Assessor, a provider either holds it for the work in scope or is not scored at all. If nobody requires one, leave it off the sheet and score the evidence each provider can show.

It can review it, and that review can be useful before a release. It is not an independent test, because the tester is checking its own work, so it fails the independence gate whenever a customer, auditor or contract asks for one. When Clever Ops built or hosts a system, we call the work a security review, and the report says who built and who hosts it.

After the scoring. Price is left off the scorecard so that a low quote cannot hide a thin report or a narrow scope. Once the gates and scores have narrowed the list, compare the remaining quotes on the same written scope, including what each retest covers.

Three gives a useful spread. Each extra provider adds a full sample report to read and a set of follow-up questions to chase, so add one only when it brings something the others lack, such as an accreditation a gate requires.

Yes. Send each one the "Ask for" lines from Part D, in the same words and with the same reply date. Identical questions make the answers comparable, and a provider that answers a different question from the one asked has told you something the sheet can record.

Scope the right test for your application

Tell us what you have built, who uses it and when it needs to launch. You will get a plain-English recommendation: a website audit, a web application test or a referral to a specialist.