Penetration testing provider evaluation: a weighted scorecard for choosing who tests your application
Check each provider against pass or fail gates for accreditation and independence, then score the rest on eight criteria, from relevant experience and sample-report quality to retest terms and exclusions.
The weights are yours to set, every score carries its evidence, and a moderation sheet settles differences when several people score.
On this page
Weighted provider scorecard with gates and a moderation sheet
Copy the scorecard into a spreadsheet or shared document. Fill in Part A and sign off the weights in Part C before any proposal is opened, then check each provider against the Part B gates.
How to use itHide the instructions
Score the providers that pass with the Part D anchors, total them in Part E, and settle differences between evaluators with the Part F moderation sheet. Part G is an invented worked example.
penetration-testing-provider-scorecard.txt
Introduction2 lines
Part A. Before any proposal is opened8 lines
Part B. Pass or fail gates7 lines
Part C. Weights (total 100)11 lines
Part D. Scoring anchors and red-flag follow-ups86 lines
Part E. Score sheet14 lines
Part F. Moderation sheet (two or more evaluators)9 lines
Part G. Worked example (illustration only)12 lines
Whole template163 lines
PENETRATION TESTING PROVIDER SCORECARD
Copy this into a spreadsheet or a shared document, one copy per evaluation. Fill in Part A and sign off the weights in Part C before anyone opens a proposal. Send every provider the "Ask for" lines in Part D, in the same words and with the same reply date: together they are your supplier questionnaire.
PART A. BEFORE ANY PROPOSAL IS OPENED
A1. What is to be tested (application, API addresses, environment): ____
A2. Who asked for the test, and their exact wording (a customer, insurer, auditor, tender, or your own decision): ____
A3. Accreditation that wording requires: CREST / PCI Approved Scanning Vendor / PCI Qualified Security Assessor / other ____ / none. If a government buyer requires IRAP, note it here and ask that buyer which providers it accepts: ____
A4. Does the wording require a tester independent of whoever built, hosts or maintains the system? yes / no
A5. Conditions on where test evidence may be stored or processed: ____
A6. Insurance cover your contract or policy requires of suppliers, by type and limit: ____
A7. Evaluators: ____ Moderator: ____
A8. Weights in Part C signed off by ____ on ____ (date)
PART B. PASS OR FAIL GATES
Check every provider against every gate and write pass or fail in each cell. A provider that fails one gate is not scored, however well it would score elsewhere.
B1. Accreditation. If A3 names one, the provider holds it for the work in scope, checked with the accrediting body itself (for the two PCI roles, the PCI Security Standards Council's published list) on ____ (date). Provider A ____ Provider B ____ Provider C ____
B2. Independence. If A4 is yes, the provider did not build the system and does not host or maintain it. A ____ B ____ C ____
B3. Authorisation. The provider starts only after the owner of each target has signed an authorisation, and works to written rules of engagement. A ____ B ____ C ____
B4. Service fit. The provider tests the kind of target in A1, such as a web application and its API rather than an office network, and offers the service A2 asked for rather than a substitute such as an unattended automated scan. A ____ B ____ C ____
B5. Evidence conditions. The provider meets every condition in A5, in writing. A ____ B ____ C ____
B6. Insurance minimum. If A6 sets a minimum, the provider's certificates meet it. A ____ B ____ C ____
PART C. WEIGHTS (TOTAL 100)
The starting spread is on the left. Write your own on the right before any proposal is opened.
C1. Relevant application experience: 15 Yours ____
C2. Sample-report quality: 20 Yours ____
C3. Methodology: 15 Yours ____
C4. Independence: 10 Yours ____
C5. Data handling: 10 Yours ____
C6. Insurance evidence: 5 Yours ____
C7. Retest terms: 10 Yours ____
C8. Exclusions and scope fit: 15 Yours ____
Total: 100 Yours ____ (must equal 100)
Move points between criteria; never add them. Sensitive records in scope: move points to C5. Someone outside the business wants proof that findings were fixed: move points to C7. Several customer organisations in one database, payment flows, partner APIs or an AI agent that can act: move points to C1 and C3.
PART D. SCORING ANCHORS AND RED-FLAG FOLLOW-UPS
Score each criterion from 0 to 5. The anchors define 0, 1, 3 and 5; a 2 or a 4 sits between them. Write one line of evidence beside every score: the proposal page, the sample report finding or the written answer it rests on. A score with no evidence line is capped at 1.
D1. RELEVANT APPLICATION EXPERIENCE (C1)
Ask for: the names of the people who will test, what they have tested that resembles your application (its stack, roles, customer organisations, payments and APIs), and one redacted finding from comparable work.
5: Names the testers, describes comparable work in specifics, and asked how your roles, data and payments work before quoting.
3: Relevant experience described in general terms; testers named only at booking.
1: Experience given only as a count of tests, years in business or client logos, with nothing about applications like yours.
0: No answer.
Red flags, and the question to ask next:
- The person on the scoping call will not be the person testing. Ask: who tests, and can that person join the call?
- Part of the work is subcontracted. Ask: to whom, and does the subcontractor pass the same gates?
- No questions about how your application works. Ask: where would you start in an application like this one, and why?
D2. SAMPLE-REPORT QUALITY (C2)
Ask for: a complete sample report, labelled as a lab, a demonstration or redacted client work. Read one finding from description to fix.
5: Scope and exclusions stated; each finding shows the request, the response and the account that sent it; every CVSS score carries its vector string; a business rating with a written reason wherever it departs from the technical one; checks that found nothing recorded with their limits; retest statuses defined in advance.
3: Clear findings and fixes, but thin on scope, limits or retest status.
1: Scanner output under a cover page, or findings with no evidence.
0: Neither a sample nor an extract offered.
Red flags, and the question to ask next:
- Scores with no vector string. Ask: which inputs produced this score?
- Nothing in the sample is marked as unconfirmed. Ask: how does a report of yours show a finding you could not fully prove?
- The sample is not labelled. Ask: is this a lab, a demonstration or client work, and who approved sharing it?
D3. METHODOLOGY (C3)
Ask for: the written method, the published guides it references (such as the OWASP Web Security Testing Guide v4.2 or ASVS 5.0), how each role and the API are tested, how findings are confirmed and rated, and whether a record of the checks run comes with the report.
5: A written method mapped to a published guide; testing as every role in scope and against the API directly; each finding confirmed; a coverage record returned with the report.
3: Method explained on a call and tied to a guide, but no coverage record.
1: "Best practice" named with no method behind it, or a single automated scan.
0: Will not describe the method.
Red flags, and the question to ask next:
- Talk of an OWASP certification or an ASVS compliance certificate. Ask: which requirements were checked, and what did each result show?
- Outside-only testing when your customers sign in. Ask: which roles will you test as, and with how many accounts in each?
- Mostly an unattended automated scan. Ask: how much of the window is a tester working through the application while signed in?
- No activity log. Ask: will you keep a step-by-step record of what was sent, and can it be seen if something unexpected happens?
D4. INDEPENDENCE (C4)
Ask for: a written statement of any part the provider had in building, hosting, maintaining or supplying the system, any product it resells, whether it will also be paid to fix the findings, and whether any work is subcontracted.
5: No part in the system and no product to sell on the back of the findings; relationships disclosed in writing and stated in the report.
3: A past or minor relationship, disclosed in writing and stated in the report.
1: A relationship that came out only when asked, or that the report will not state.
0: Built, hosts or maintains the system and calls the work independent.
Red flags, and the question to ask next:
- A finding whose fix is a product the provider sells. Ask: would the fix be different if that product were never bought?
- The provider will fix its own findings and retest them. Ask: who retests the fixes, and how will the updated report show who made each change?
D5. DATA HANDLING (C5)
Ask for: written terms covering where evidence is stored and in which country, encryption, who can reach it, any subcontractor or processing elsewhere, how the report is delivered, how long evidence is kept, and how deletion is confirmed.
5: All of these in writing before testing, the report delivered encrypted, and deletion confirmed in writing on request.
3: Most of them in writing on request, the rest only said on a call.
1: "Data is kept safe" with nothing specific behind it.
0: Will not say where evidence goes.
Red flags, and the question to ask next:
- The report travels as an ordinary email attachment. Ask: how will only the intended readers be able to open it?
- No retention period. Ask: when is the evidence deleted, and how will that be confirmed?
- Passwords, session tokens or customer records visible in the sample report. Ask: what is redacted, and what is kept?
D6. INSURANCE EVIDENCE (C6)
Ask for: a current certificate of currency for each type of cover in A6, showing the insurer, the insured entity, the type of cover, the policy period and the limit.
5: Certificates for every type in A6, in the name of the entity signing the contract, current for the whole testing window.
3: Certificates supplied only after a reminder, or one detail, such as the policy period, needing a follow-up.
1: Cover stated, no certificate.
0: No cover, or no answer.
Red flags, and the question to ask next:
- The insured entity differs from the one signing the contract. Ask: which entity is insured, and which one signs?
- The policy period ends before testing finishes. Ask: when will the renewed certificate arrive?
D7. RETEST TERMS (C7)
Ask for: whether a retest is included, which findings it covers, how the window for requesting it is set, which build it runs on, the statuses each finding can take, and the form of the updated report.
5: A retest in writing for every reported finding; statuses defined in advance, including not retested; an updated report that keeps findings not retested apart from fixed ones.
3: A retest for some severities only, or statuses left undefined.
1: A retest "if needed", with nothing written.
0: No retest available.
Red flags, and the question to ask next:
- An "all clear" letter in place of a status for each finding. Ask: how is each finding marked, and how is one nobody retested shown?
- The window or the coverage left out of the contract. Ask: can both go in writing before testing starts?
D8. EXCLUSIONS AND SCOPE FIT (C8)
Ask for: the written scope naming targets, environment, roles and exclusions; the assumptions it depends on; and what happens when something outside the scope turns up during testing.
5: Every exclusion listed, each one acceptable to the requester in A2 or to be recorded as a limitation in the report, and a stated way of putting scope changes to the risk owner.
3: Exclusions listed, but one touches something A2 asks about.
1: No exclusions listed, or "full coverage" claimed.
0: The scope leaves out the thing that prompted the test.
Red flags, and the question to ask next:
- "Everything is in scope" with no list. Ask: what will not be tested, and where will the report say so?
- An API, a role or a production-only integration missing from the scope. Ask: is it excluded or forgotten?
PART E. SCORE SHEET
Points for a criterion = weight x score / 5. Each provider's total is out of 100.
C1 Experience (weight ____): A ____ / ____ pts B ____ / ____ pts C ____ / ____ pts
C2 Sample report (weight ____): A ____ / ____ pts B ____ / ____ pts C ____ / ____ pts
C3 Methodology (weight ____): A ____ / ____ pts B ____ / ____ pts C ____ / ____ pts
C4 Independence (weight ____): A ____ / ____ pts B ____ / ____ pts C ____ / ____ pts
C5 Data handling (weight ____): A ____ / ____ pts B ____ / ____ pts C ____ / ____ pts
C6 Insurance evidence (weight ____): A ____ / ____ pts B ____ / ____ pts C ____ / ____ pts
C7 Retest terms (weight ____): A ____ / ____ pts B ____ / ____ pts C ____ / ____ pts
C8 Exclusions and scope (weight ____): A ____ / ____ pts B ____ / ____ pts C ____ / ____ pts
Total out of 100: A ____ B ____ C ____
E1. A gap of five points or fewer between two totals is a tie. Break it on the criterion with the highest weight, or put the same follow-up question to both providers.
E2. A 0 or 1 on any criterion is a hold, whatever the total. Ask that criterion's red-flag question and re-score on the written answer before choosing the provider.
E3. Price is not on this sheet. Compare quotes for the providers still standing, on the same written scope, after scoring.
E4. Decision: ____ Reason, in one sentence: ____ Signed: ____ Date: ____
PART F. MODERATION SHEET (TWO OR MORE EVALUATORS)
F1. Each evaluator scores alone, with an evidence line for every score, before seeing any other sheet.
F2. Each evaluator declares any relationship with a provider before scoring, such as a former employer, a friend at the firm or a current supplier. The moderator records it and may ask that evaluator not to score that provider.
F3. Copy this line once for each provider and criterion:
Provider ____ Criterion ____ Evaluator 1 ____ Evaluator 2 ____ Evaluator 3 ____ Spread (highest minus lowest) ____ Agreed score ____ Evidence relied on ____ Reason for any change ____
F4. Spread of 2 or more: discuss it. Each evaluator points to the evidence behind their score. The agreed score follows the evidence, not a vote or the most senior person present.
F5. Spread of 0 or 1: take the lower score, unless an evidence line supports the higher one.
F6. Weights do not change once proposals are opened. If the team agrees a weight was wrong, record the reason and re-score every provider under the new weight.
F7. Dissent (evaluator name and one line of reason): ____
F8. Moderated scores signed off by ____ on ____ (date)
PART G. WORKED EXAMPLE (ILLUSTRATION ONLY)
The business, the providers and the scores below are invented. They show the sheet in use and describe no real firm, client or quote.
Situation: a booking and records platform for allied health clinics, with a web application, its API, three roles (clinic owner, practitioner and client) and several clinics sharing one database.
A2: a hospital group's supplier review asks for "an independent penetration test of the application and API, and evidence that high-risk findings were fixed".
A3: none. A4: yes. A5: evidence stored in Australia. A6: professional indemnity cover at the limit the hospital group's contract sets.
Gates: Provider C built the platform and fails B2, so it is not scored. Its security review could still suit a check before a release, but it cannot answer A2. Providers A and B pass every gate.
Weights signed off before proposals were opened: C1 10, C2 15, C3 10, C4 10, C5 20, C6 5, C7 15, C8 15. Points moved to C5 because the platform holds clinical notes, and to C7 because A2 asks for evidence of fixes.
Moderated scores, Provider A then Provider B: C1 5 and 3. C2 5 and 3. C3 4 and 3. C4 5 and 5. C5 2 and 5. C6 3 and 5. C7 2 and 5. C8 4 and 3.
Points under these weights: Provider A 10 + 15 + 8 + 10 + 8 + 3 + 6 + 12 = 72. Provider B 6 + 9 + 6 + 10 + 20 + 5 + 15 + 9 = 80. Provider B leads by 8, outside the tie band.
The same scores under the starting spread: Provider A 15 + 20 + 12 + 10 + 4 + 3 + 4 + 12 = 80. Provider B 9 + 12 + 9 + 10 + 10 + 5 + 10 + 9 = 74. Provider A would lead by 6.
Moderation line: Provider A, C7. Evaluators 4, 2 and 1; spread 3. Evaluator 1 had read "retest included" in the proposal summary; the terms section limits the retest to Critical and High findings and leaves the request window unwritten. Agreed score 2. Evidence: the proposal's terms section.
Follow-up: had Provider A put data handling and retest terms in writing that earned a 4 on C5 and on C7, its total under these weights would be 86, and it would lead.
Decision: Provider B. Reason: it leads on data handling and retest terms, the two criteria this business moved points to, and none of its scores is a hold.What a penetration testing provider evaluation has to settle
Choosing a penetration testing company decides who will look at your application, how closely, and what you will hold at the end.
Two providers can quote for the same scope and return very different things: one a report your developers can reproduce and a customer can follow, the other a scan export with a cover page. The UK National Cyber Security Centre (NCSC) gives the reason in its penetration testing guidance: penetration tests cannot be entirely procedural, an exhaustive set of test cases cannot be drawn up, and so the quality of a penetration test is closely linked to the abilities of the testers involved.
A scorecard turns that judgement into something you can write down, share and defend later. This one works in four steps. Pass or fail gates remove any provider that cannot do the job you were asked to buy. Weights, set before any proposal is opened, record what matters most for this application.
Each remaining provider is scored from 0 to 5 on eight criteria against written anchors, with the evidence noted beside every score. Where more than one person scores, a moderation sheet settles the differences on evidence rather than on seniority.
It compares providers, not prices. Score first, then line up the quotes from the providers still standing on the same written scope, which is what the budget worksheet in the cost guide is for. A sample report is one of the eight criteria here; the annotated report example reads one in depth. If what you are buying is software rather than testing, the vendor security questionnaire is the better tool.
Pass or fail gates come before any score
Some requirements are not a matter of degree.
If the person who asked for the test named an accreditation, required an independent tester or set conditions on where your data may go, a provider either meets that or it does not, and a strong score elsewhere cannot make up for it.
Read the exact wording of the request first, because the gates come from that wording rather than from anyone's marketing. Then check every provider against every gate, and do not score one that fails.
Accreditations: check with the body that grants them
CREST awards accreditation to organisations that evidence compliance with its accreditation standards through an application, evidence submission and review, and accredited companies undergo periodic reassessment to keep it. Its accreditation disciplines include penetration testing and vulnerability assessment, so confirm that the accreditation covers the work you are buying, and check it with CREST rather than from a logo on a proposal.
Card payments bring two roles defined by the PCI Security Standards Council. An Approved Scanning Vendor (ASV) conducts external vulnerability scanning to validate adherence with the external scanning requirements of PCI DSS Requirement 11.3.2, and its scan solution is tested and approved by the Council before the vendor is added to the Council's list.
Qualified Security Assessor (QSA) companies are independent security organisations qualified by the Council to validate an entity's adherence to PCI DSS. On both pages, the Council advises a client, each time it engages one, to check the list and confirm the company has kept its status.
A government buyer may require IRAP. Where one does, ask that buyer which providers it will accept before you score anyone.
Clever Ops is not CREST accredited, so we refer that work to a provider that is. Clever Ops is not a PCI Approved Scanning Vendor or a Qualified Security Assessor, so a PCI scan or assessment goes to a provider that is one. Where a government buyer requires IRAP, Clever Ops does not meet that requirement, and the work goes to a provider that does.
Independence and the other gates
Independence becomes a gate when the request asks for an independent test. A provider that built, hosts or maintains the system is checking its own work, however capable it is. When Clever Ops built or hosts a system, we call the work a security review rather than an independent test, and the report says who built and who hosts it.
That review can be the quicker route to fixes before a release, but it does not answer a customer or auditor who asked for independence.
Four more gates apply to every provider:
- Authorisation: testing starts only after the owner of each target has signed an authorisation, and runs to written rules of engagement.
- Service fit: the provider tests the kind of target you have, such as a web application and its API rather than an office network, and offers what the requester asked for rather than a substitute such as an unattended automated scan.
- Evidence conditions: any rule your contract or policy sets on where test evidence is stored or processed is met, in writing.
- Insurance minimum: if your contract sets minimum cover for suppliers, the provider's certificates meet it.
Where Clever Ops fails the service-fit gate
Clever Ops tests what customers reach through a browser or an API. Internal networks, Active Directory, red teaming, social engineering, physical testing and load or denial-of-service testing fail the service-fit gate for us, and belong with a specialist in that work.
Setting the weights before anyone opens a proposal
Weights state what matters most for this particular test, so they belong to the buyer and are set before any proposal is read.
Set after the scores are visible, they can be bent, without anyone meaning to, towards a provider someone already prefers. Write them down, have the person who owns the decision sign and date them, and keep them with the sheets.
The starting spread gives 20 of 100 points to sample-report quality, because the report is what you keep: your developers fix from it, a customer reads it and the retest updates it. Relevant application experience, methodology, and exclusions and scope fit take 15 each, since together they decide whether testing reaches the parts of your application that carry the risk.
Independence, data handling and retest terms take 10 each. Insurance evidence takes 5, because it is closer to a yes or no than a scale; where your contract sets minimum cover, that minimum is already a gate.
Move points between criteria rather than adding them, so the total stays at 100 and every increase is paid for somewhere else. Some reasons to move them:
- Health, financial or other sensitive records inside the scope: more points to data handling.
- A customer, insurer or auditor wants evidence that findings were fixed: more points to retest terms.
- Several customer organisations in one database, payment flows, partner APIs or an AI agent that can take actions: more points to experience and methodology.
- Independence required by the request: it is already a gate, so its weight now scores what remains, such as products the provider resells or an offer to fix its own findings.
Changing a weight after proposals arrive
If the team later agrees a weight was wrong, change it in the open. Record the reason, then re-score every provider under the new weight, not only the provider the change would help. The worked example further down shows how far a change of weights can move a decision.
The eight criteria, and what earns a high score
The template gives anchors for scores of 0, 1, 3 and 5 on each criterion, and a red-flag question to ask when an answer falls short.
The notes below explain what each criterion measures and where the evidence for it comes from.
The NCSC says third-party penetration tests should be performed by qualified and experienced staff only. It also says to ensure the external team has the relevant qualifications and skills to perform testing on your IT estate, and to highlight any unusual systems, such as mainframes or bespoke hardware, in the bid process so the teams know what skill sets will be required.
For a web application, tell bidders about the features that change what the test needs: several customer organisations in one database, payment and refund flows, an API that partners call with their own keys, or an AI agent that can act on a user's behalf.
Score the experience of the people who will test, with applications like yours, rather than the size of the firm. The questions a provider asks you count as evidence too. Questions about who can see which records, and how money moves through the application, show a test being planned around your application rather than around a URL.
The sample report is the nearest thing to what you will receive, which is why it carries the most weight in the starting spread. The introduction to the OWASP Web Security Testing Guide (WSTG) v4.2 says the test report should allow another security tester to reproduce the results, and its reporting chapter suggests each finding include enough information to understand, replicate and resolve it, with possibly sensitive data such as passwords, personal information or credit card details masked.
Scores need their inputs. FIRST, in its CVSS v3.1 specification, makes it a condition of use that anyone who publishes scores provides both the score and the scoring vector, so others can understand how the score was derived. A score without its vector is a number you cannot check. Ask, too, how the sample was made: a lab, a demonstration or redacted client work should each be labelled as such.
A method earns its score when it is written down and the report can be checked against it. The WSTG lists several published penetration testing methodologies, among them the OWASP testing guides and the Penetration Testing Execution Standard, so the name matters less than what the provider can show: which checks were run, as which roles, and against the API directly as well as through the screens.
The WSTG reporting chapter suggests appendices that include the test methodology used and a checklist of all the tests conducted. A record like that tells you what a clean result covered.
Two other things are worth checking. NIST SP 800-115 says assessors should keep a log with a step-by-step record of their activities, which provides an audit trail and allows the organisation to distinguish between the actions of assessors and those of true adversaries.
And certification language around OWASP deserves a second look: ASVS 5.0 says OWASP, as a vendor-neutral nonprofit, does not certify vendors, verifiers or software, and that any assurance, trust mark or certification claiming ASVS compliance is not officially endorsed by OWASP.
Past the gate, independence comes in degrees. A provider that resells a security product has a reason to find problems that product solves. A provider that will also be paid to fix its findings will then be retesting its own work. Neither rules a provider out on its own.
Both should be disclosed in writing and stated in the report, so anyone who reads it later knows where the tester stood. Ask whether any part of the testing or reporting is subcontracted, and to whom, because the gates apply to the subcontractor too.
Test evidence is sensitive in its own right. NIST SP 800-115 says data results from any type of information security test identify vulnerabilities an adversary can exploit and should be considered sensitive. Its section on assessment plans lists the data handling requirements a plan includes: storage of organisational data on the assessors' systems during the assessment, storage when the assessment concludes, transmission during or after it, and removal of data from systems when it concludes.
The WSTG reporting chapter advises securing the report and encrypting it so that only the receiving party is able to use it.
Ask for each of these in writing before testing starts, along with the country where evidence is kept and how deletion will be confirmed.
Ask every provider for a current certificate of currency for each type of cover your contract asks of suppliers, rather than accepting a line in a proposal that says cover is held. Check four things on each one: the insured entity is the entity signing your contract, the cover is the type you asked for, the policy period runs past the end of the testing window, and the limit meets your contract.
Which cover to require is a question for your contract owner and, where needed, your broker or solicitor. The scorecard records only whether the evidence matches.
The retest is where a finding becomes a fixed finding someone else can rely on. Ask whether one is included, which findings it covers, how the window for requesting it is set, which build it runs on, and how each finding is marked afterwards.
The WSTG reporting chapter suggests that a re-test report summarise the findings of the previous test and the updated status of previously identified vulnerabilities. A status list that keeps not retested apart from fixed stops an untouched finding being read as a passed one.
Every test leaves something out, and a written exclusions list shows you what. Set each exclusion beside the request: an excluded API, role or production-only integration may be the very thing the requester cares about. The NCSC says that when the testing team finds systems or components outside the scope that could affect the security of in-scope systems, it may suggest a change to the scope or recommend that their exclusion be recorded as a limitation on testing, and that the choice will generally be down to the risk owner. Score highest the provider whose proposal already says how that choice will be put to you.
Moderating the scores when several people evaluate
Two evaluators reading the same proposal can land two points apart, and the gap is information: one of them has seen something the other missed, or has read a promise as a commitment.
Moderation finds out which before the scores become a decision, and the template's moderation sheet records each step.
- Score alone first. Each evaluator fills in a sheet without seeing anyone else's, with one line of evidence beside every score: a page of the proposal, a finding in the sample report or a written answer.
- Declare relationships before scoring. A former employer, a friend at the firm or a current supplier does not disqualify anyone, but the moderator records it and may ask that person to sit out scoring that provider.
- Discuss any spread of two points or more. Each evaluator shows the evidence behind their score, and the agreed score follows the evidence, not a vote or the most senior person present.
- Settle a one-point spread cautiously: take the lower score unless an evidence line supports the higher one.
- Record dissent. An evaluator who still disagrees adds one line with their name and reason.
- Keep everything. The signed weights, the moderated scores and the evidence lines explain the decision to anyone who asks later, including the provider you did not choose.
The moderator runs the process
The moderator collects the sheets, chairs the discussion and records the agreed scores, and does not score. In a small business the moderator may be the owner. What matters is that evaluators commit their scores before they hear the owner's view.
Worked example: three providers and two sets of weights
This is an illustration.
The business, the three providers and every score are invented, and the providers are named only Provider A, Provider B and Provider C. Part G of the template sets out the same sheet line by line.
A software business runs a booking and records platform for allied health clinics: a web application, its API, three roles and several clinics sharing one database. A hospital group considering the platform sent a supplier review asking for an independent penetration test of the application and API, and evidence that high-risk findings were fixed. It named no accreditation.
Provider C is the agency that built the platform. Its offer of a security review could serve the business well before a release, but the hospital group asked for independence, so Provider C failed that gate and was not scored. Providers A and B passed every gate.
Before any proposal was opened, the business moved points to data handling, now 20, because the platform holds clinical notes, and to retest terms, now 15, because the hospital group wanted evidence of fixes. Experience, methodology and sample-report quality gave up 5 points each. Three evaluators then scored, and moderation settled these scores out of 5:
- Provider A:experience 5, sample report 5, methodology 4, independence 5, data handling 2, insurance evidence 3, retest terms 2, exclusions and scope fit 4.
- Provider B:experience 3, sample report 3, methodology 3, independence 5, data handling 5, insurance evidence 5, retest terms 5, exclusions and scope fit 3.
How the weights moved the decision
Under the business's own weights, Provider B totals 80 and Provider A 72. Under the starting spread, the same scores give Provider A 80 and Provider B 74. Neither total is wrong. The weights record that, for this platform and this customer, how evidence is handled and how fixes are proven matter more than the depth of the sample report, and they were signed before anyone knew which provider that would favour.
What moderation and follow-up changed
Moderation changed one score. On Provider A's retest terms the evaluators gave 4, 2 and 1. The evaluator who gave 4 had read "retest included" in the proposal summary; the terms section limited the retest to Critical and High findings and left the request window unwritten. The agreed score was 2, with the terms section recorded as the evidence.
Evidence can still move a provider. Had Provider A answered the red-flag questions with written data handling and retest terms earning a 4 on each, it would have totalled 86 under the business's weights and led again. The sheet rewards what a provider can show, not what it says it does.
Turning the totals into a decision
The totals rank the providers.
These rules turn the ranking into a choice you can explain.
- Treat a gap of five points or fewer as a tie. Break it on the criterion with the highest weight, or put the same follow-up question to both providers.
- Hold on any score of 0 or 1. A leading total with a 1 on data handling still carries that gap, so ask the red-flag question for that criterion and re-score on the written answer.
- Read the leading provider's weakest criterion before you sign. It is where the engagement has the least evidence behind it.
- Compare price last, for the providers still standing, on the same written scope. A lower quote for a narrower scope buys less testing, not cheaper testing.
- Write the choice and the reason down in one sentence, and keep it with the sheets and the contract.
What the total does not tell you
A scorecard does not make the decision, and a high total is not a certificate for the provider. It makes the reasons visible, so the person who asked for the test can check them and the next evaluation can start from them.
Putting Clever Ops through the same scorecard
If Clever Ops is on your list, score it on the same sheet as everyone else, from what we can show you in writing.
Where a gate rules us out, that is the right result. For the criteria, this is what you can ask Clever Ops for and check:
- Sample report: a report on a lab application Clever Ops controls, which shows the format. It is not client work.
- Methodology: the method page sets out how testing runs, phase by phase, with checks drawn from WSTG v4.2 and requirements from ASVS 5.0. A web application test runs from test accounts in each role the scope names and calls the API directly as well as through the screens.
- Data handling: evidence has session tokens, passwords, keys and personal details redacted, is stored encrypted in Australia and deleted on the schedule set in the data handling terms, and is processed outside Australia only where your statement of work approves it.
- Retest terms: one consolidated retest of the reported findings, requested within the window set in your statement of work, after which each finding is marked fixed, partially fixed, still reproducible or not retested.
- Exclusions and scope fit: a written scope and quote naming the targets, environment, routes, roles and exclusions, and what the retest covers, with testing starting once the owner of each target has signed an authorisation.
If another provider scores higher
Score what we send against the anchors like any other proposal. If another provider comes out ahead on your weights, choose them; the scorecard is built so that you can.
Frequently asked questions
Start with pass or fail gates: any accreditation, independence or data condition the person who asked for the test has set, plus signed authorisation and a service that fits your target. Then score each remaining provider from 0 to 5 on relevant experience, sample-report quality, methodology, independence, data handling, insurance evidence, retest terms, and exclusions and scope fit, using weights you set before reading any proposal. Note the evidence beside each score, moderate when several people score, and compare price last.
Who will do the testing, and what have they tested that resembles your application? Can you see a complete sample report, and the written method? Did they build, host or sell into your system? Where is evidence stored, and when is it deleted? Can they supply certificates of currency for the cover your contract requires? What does the retest cover, and what does the scope leave out? The "Ask for" lines in Part D of the scorecard put these into a supplier questionnaire you can send as it stands.
No, it is a gate. If a tender, contract or customer requires an accreditation such as CREST, or requires a PCI Approved Scanning Vendor or Qualified Security Assessor, a provider either holds it for the work in scope or is not scored at all. If nobody requires one, leave it off the sheet and score the evidence each provider can show.
It can review it, and that review can be useful before a release. It is not an independent test, because the tester is checking its own work, so it fails the independence gate whenever a customer, auditor or contract asks for one. When Clever Ops built or hosts a system, we call the work a security review, and the report says who built and who hosts it.
After the scoring. Price is left off the scorecard so that a low quote cannot hide a thin report or a narrow scope. Once the gates and scores have narrowed the list, compare the remaining quotes on the same written scope, including what each retest covers.
Three gives a useful spread. Each extra provider adds a full sample report to read and a set of follow-up questions to chase, so add one only when it brings something the others lack, such as an accreditation a gate requires.
Yes. Send each one the "Ask for" lines from Part D, in the same words and with the same reply date. Identical questions make the answers comparable, and a provider that answers a different question from the one asked has told you something the sheet can record.
Scope the right test for your application
Tell us what you have built, who uses it and when it needs to launch. You will get a plain-English recommendation: a website audit, a web application test or a referral to a specialist.