Skip to main content
Security comparisonReviewed

Penetration testing vs vulnerability scanning: what each one shows, and which one your situation calls for

A vulnerability scan, a vulnerability assessment and a penetration test answer different questions and leave different gaps.

This page sets out what each can show, how each deals with false positives, and a decision table for eight situations a business with a website or web application runs into, from a launch to a suspected hack.

Decision table: scan, assessment or penetration test in eight situations

Find the row closest to your situation and read across. Each row sets out what a vulnerability scan, a vulnerability assessment and a penetration test would give you, how false positives are handled, what you receive at the end and where the work stops, then gives a plain recommendation.

How to use it

The rows describe situations a business with a website or web application can meet; none of them is a client. Where a row says what Clever Ops does or refers elsewhere, that is in the last column. Use it before a scoping call, or to hold a proposal against what you need.

8 rows

Decision table: scan, assessment or penetration test in eight situations

SituationWhat a vulnerability scan givesWhat a vulnerability assessment givesWhat a penetration test givesHow false positives are handledWhat you receiveLimits to keep in mindWhat fits
1. Before launch. A new customer portal, booking product or app is about to take real accounts, payments or personal details for the first time.A fast pass over the release build for known issues: components with published advisories, missing security headers, files that should not be reachable. Each item it finds is cheapest to fix now.Breadth across the whole build, signed out and signed in, with every result confirmed and ranked. A good fit for a conventional build, or when you want a ranked list before deciding how deep to go.A tester working through the application in each role, trying to read other customers' records, reach staff functions, skip payment steps and break the rules the application is meant to enforce.Scan: you sort the output yourself. Assessment and test: a tester reproduces each result on the build under test before it is reported, and anything that will not reproduce stays out of the findings.Findings tied to the build you will release, each with evidence, a fix and the criteria a retest will use, then a retest once the fixes are deployed, so the launch rests on fixes that were checked.Each method covers the build, accounts and dates in its scope. A feature added after the scope is agreed, or a payment provider's hosted pages, sit outside it.A penetration test when customers sign in, pay or see data that belongs only to them; a website security check when only your own staff sign in. Clear the scan results first, so the test time goes where a scan cannot reach.
2. A customer asks for evidence. A security questionnaire, contract or tender asks how the application is tested, or asks for a penetration test by name.Shows that you run routine checks for known issues, which some questionnaires ask about. Sent in place of a requested penetration test, it describes different work from what was asked for.Evidence that the application was examined for known kinds of weakness, with confirmed findings ranked and fixed. It answers questions about how you find and prioritise vulnerabilities.A record of a tester trying to defeat the application's controls: what was tested, from which roles, what worked and what was fixed. This is the work the term penetration test describes.Raw scan output can put alarming items that do not apply to you in front of a customer's reviewer. A report listing only confirmed findings, with negative results recorded, reads as evidence.A report you can share or summarise, with scope, dates, findings and their status. After the retest, each finding shows as fixed, partially fixed, still reproducible or not retested.A report is evidence for the customer's decision, not a certificate. If the request requires an accreditation or a named scheme, a report from a provider without it does not meet that requirement.Match the request's exact wording: a penetration test, with its retest finished, when one is named; an assessment when the questions are about how vulnerabilities are found and fixed. A request that requires CREST goes elsewhere: Clever Ops is not CREST accredited, so we refer that work to a provider that is.
3. Routine checks between releases. Your developers ship changes regularly, and the components the application uses gain new published advisories between tests.The right tool for this job. An unattended automated scan run with each release, or on a schedule, picks up new advisories for components you use and settings that have drifted. OWASP's testing guide suggests building such checks into the release workflow.Too heavy to repeat with every release. Worth repeating when the application has changed enough that the last ranked list no longer describes it.Not a routine check. The NCSC calls penetration testing a way of gaining assurance in your own vulnerability assessment and management, not a primary method for identifying vulnerabilities.Someone on your side reads each new result, decides whether it applies, and records the decision with a reason, so the same false positive is not investigated again after the next run.Whatever the scan produces, usually a list of matches with the scanner's own severity labels. Its value depends on the person who reads it and turns real items into tickets.A scan works from signatures of known issues that have to be kept current, and OWASP says business logic flaws cannot be detected by a vulnerability scanner. A quiet run means nothing matched, not that nothing is there.A scan run by your developer or host, with a named person who owns its results. Clever Ops does not offer a scanner-only subscription; scans are a step inside a check or test.
4. Following up fixes. Your team has deployed fixes for findings from an earlier report and needs to know whether they hold.Can confirm a fix for something a scan detected, such as an updated component version. It cannot confirm a fix to access control or business logic that it did not find in the first place.Not needed on its own. A fresh assessment is worth it only if the application changed enough during the fixing that new weaknesses are likely.A retest: each reported finding tried again on the fixed build, the way it was first shown. OWASP's testing guide notes that focused penetration testing, aimed at vulnerabilities found in earlier reviews, can be useful in detecting whether specific ones are actually fixed in the deployed code.The risk shifts from false positives to false comfort: a fix can close one route and leave the same flaw open on another. A retest tries the original evidence and every affected route listed under the finding.An updated report in which each finding carries one of four statuses: fixed, partially fixed, still reproducible or not retested. A finding left out of the retest is shown as not retested, not as fixed.A retest covers the reported findings within the agreed scope. New features and unrelated changes are new work, and the retest says nothing about them.The retest that comes with the original test. Clever Ops includes one consolidated retest with every test, and can retest findings from another provider's report as a separate piece of work.
5. A suspected compromise. Pages you did not write, admin accounts nobody created, or a warning from your host or a search engine suggest the site has been hacked.Not now. Google's recovery guidance warns that vulnerability scanners can be far more invasive than antivirus scanners, with greater potential to cause unwanted damage to a site, and says to follow all directions, such as backing up the site, before running one. Leave that decision to the specialist handling the incident.Later. Once the site has been cleaned, an assessment can look for other weaknesses before you rely on the site again.Later, and only once the site is clean. A test can then look for the weakness the attacker may have used, if it is still open, and for whatever else the site exposes.Not the question yet. Keeping evidence and stopping harm to visitors come before any list of findings.The incident specialist's own findings, under their terms. From a later check or test: a report on the cleaned site, recording what was checked and what was not.None of the three is incident response, forensic investigation or malware cleanup, and none of them tells you what the attacker did while they had access.Your hosting provider and an incident response specialist first. Clever Ops does not provide incident response, forensic investigation, malware cleanup or monitoring; an audit or test can follow once the site is clean.
6. A significant change. A new payment option, a new role such as customer administrator, or a second customer organisation sharing the same database.Picks up components and settings the change brought in. It cannot tell whether the new role can reach functions meant for another, or whether one organisation can see another's records.Useful when the change is broad but conventional, such as a redesign that touched many pages, to confirm and rank what is new.Suits any change to who can see or do what, and any change to money. Scoped to the new routes, roles and rules plus the existing areas they touch, the test stays in proportion to the change.As in any test, results are reproduced before they are reported. Scan results from the new build are useful input: clear the real ones before testing, so the tester's time goes on the new logic.A report scoped to the change, naming the new routes, roles and rules tested, the existing areas they touch, and anything left out as an exclusion.A change-scoped test does not cover the rest of the application again. If much else has changed since the last full test, raise it at scoping.A penetration test scoped to the change when it affects access, payments or data shared between customers; an assessment when the change is broad and conventional.
7. An application with no security history. You took it over from an agency or from a developer who has left, or bought a business that runs it, and nobody can say what has been checked.A first inventory of which components and versions are running and which carry published advisories. A quick way to see how far behind the application has fallen.Usually the best first step: the whole application examined, signed out and signed in, with confirmed findings ranked, so you get a work list and a picture of where the risk sits.Better as a second step, scoped from what the assessment found, unless the application already takes payments or keeps each customer's records apart from the others'.An application nobody has maintained can produce long scan output. An assessment merges results that share a cause, confirms which are real and records what could not be checked, so the backlog reflects real work.A ranked queue of confirmed findings, each with an owner, a fix and retest criteria, and a map of which areas were examined signed out, signed in or not at all.An assessment ranks weaknesses one at a time. It does not chain them to see how far they lead, which is the question a penetration test answers.A vulnerability assessment first, then a penetration test scoped from its queue where the application has several roles, takes payments or holds customer data.
8. A request for PCI DSS scans. Your acquirer or payment provider asks for external vulnerability scans under PCI DSS.This is a specific kind of scan. The PCI Security Standards Council says an Approved Scanning Vendor conducts external vulnerability scanning to validate adherence with the external scanning requirements of PCI DSS.Does not stand in for those scans. It can sit beside them as evidence about the web application itself.Also not a substitute for those scans. A web application test answers different questions; ask whoever made the request what else they need from you.A matter for the vendor. Ask how it deals with a result you believe is a false positive.Scan reports from the Approved Scanning Vendor, which you pass to whoever asked for them.An external scan looks at what can be reached from the internet. What signed-in users can do inside the application is a question for a test of the application.An Approved Scanning Vendor from the Council's list. Clever Ops is not a PCI Approved Scanning Vendor, so we refer that work to one; a web application test can sit beside those scans.

Situation

What a vulnerability scan gives

  1. 1. Before launch. A new customer portal, booking product or app is about to take real accounts, payments or personal details for the first time.
    What a vulnerability scan gives
    A fast pass over the release build for known issues: components with published advisories, missing security headers, files that should not be reachable. Each item it finds is cheapest to fix now.
    What a vulnerability assessment gives
    Breadth across the whole build, signed out and signed in, with every result confirmed and ranked. A good fit for a conventional build, or when you want a ranked list before deciding how deep to go.
    What a penetration test gives
    A tester working through the application in each role, trying to read other customers' records, reach staff functions, skip payment steps and break the rules the application is meant to enforce.
    How false positives are handled
    Scan: you sort the output yourself. Assessment and test: a tester reproduces each result on the build under test before it is reported, and anything that will not reproduce stays out of the findings.
    What you receive
    Findings tied to the build you will release, each with evidence, a fix and the criteria a retest will use, then a retest once the fixes are deployed, so the launch rests on fixes that were checked.
    Limits to keep in mind
    Each method covers the build, accounts and dates in its scope. A feature added after the scope is agreed, or a payment provider's hosted pages, sit outside it.
    What fits
    A penetration test when customers sign in, pay or see data that belongs only to them; a website security check when only your own staff sign in. Clear the scan results first, so the test time goes where a scan cannot reach.
  2. 2. A customer asks for evidence. A security questionnaire, contract or tender asks how the application is tested, or asks for a penetration test by name.
    What a vulnerability scan gives
    Shows that you run routine checks for known issues, which some questionnaires ask about. Sent in place of a requested penetration test, it describes different work from what was asked for.
    What a vulnerability assessment gives
    Evidence that the application was examined for known kinds of weakness, with confirmed findings ranked and fixed. It answers questions about how you find and prioritise vulnerabilities.
    What a penetration test gives
    A record of a tester trying to defeat the application's controls: what was tested, from which roles, what worked and what was fixed. This is the work the term penetration test describes.
    How false positives are handled
    Raw scan output can put alarming items that do not apply to you in front of a customer's reviewer. A report listing only confirmed findings, with negative results recorded, reads as evidence.
    What you receive
    A report you can share or summarise, with scope, dates, findings and their status. After the retest, each finding shows as fixed, partially fixed, still reproducible or not retested.
    Limits to keep in mind
    A report is evidence for the customer's decision, not a certificate. If the request requires an accreditation or a named scheme, a report from a provider without it does not meet that requirement.
    What fits
    Match the request's exact wording: a penetration test, with its retest finished, when one is named; an assessment when the questions are about how vulnerabilities are found and fixed. A request that requires CREST goes elsewhere: Clever Ops is not CREST accredited, so we refer that work to a provider that is.
  3. 3. Routine checks between releases. Your developers ship changes regularly, and the components the application uses gain new published advisories between tests.
    What a vulnerability scan gives
    The right tool for this job. An unattended automated scan run with each release, or on a schedule, picks up new advisories for components you use and settings that have drifted. OWASP's testing guide suggests building such checks into the release workflow.
    What a vulnerability assessment gives
    Too heavy to repeat with every release. Worth repeating when the application has changed enough that the last ranked list no longer describes it.
    What a penetration test gives
    Not a routine check. The NCSC calls penetration testing a way of gaining assurance in your own vulnerability assessment and management, not a primary method for identifying vulnerabilities.
    How false positives are handled
    Someone on your side reads each new result, decides whether it applies, and records the decision with a reason, so the same false positive is not investigated again after the next run.
    What you receive
    Whatever the scan produces, usually a list of matches with the scanner's own severity labels. Its value depends on the person who reads it and turns real items into tickets.
    Limits to keep in mind
    A scan works from signatures of known issues that have to be kept current, and OWASP says business logic flaws cannot be detected by a vulnerability scanner. A quiet run means nothing matched, not that nothing is there.
    What fits
    A scan run by your developer or host, with a named person who owns its results. Clever Ops does not offer a scanner-only subscription; scans are a step inside a check or test.
  4. 4. Following up fixes. Your team has deployed fixes for findings from an earlier report and needs to know whether they hold.
    What a vulnerability scan gives
    Can confirm a fix for something a scan detected, such as an updated component version. It cannot confirm a fix to access control or business logic that it did not find in the first place.
    What a vulnerability assessment gives
    Not needed on its own. A fresh assessment is worth it only if the application changed enough during the fixing that new weaknesses are likely.
    What a penetration test gives
    A retest: each reported finding tried again on the fixed build, the way it was first shown. OWASP's testing guide notes that focused penetration testing, aimed at vulnerabilities found in earlier reviews, can be useful in detecting whether specific ones are actually fixed in the deployed code.
    How false positives are handled
    The risk shifts from false positives to false comfort: a fix can close one route and leave the same flaw open on another. A retest tries the original evidence and every affected route listed under the finding.
    What you receive
    An updated report in which each finding carries one of four statuses: fixed, partially fixed, still reproducible or not retested. A finding left out of the retest is shown as not retested, not as fixed.
    Limits to keep in mind
    A retest covers the reported findings within the agreed scope. New features and unrelated changes are new work, and the retest says nothing about them.
    What fits
    The retest that comes with the original test. Clever Ops includes one consolidated retest with every test, and can retest findings from another provider's report as a separate piece of work.
  5. 5. A suspected compromise. Pages you did not write, admin accounts nobody created, or a warning from your host or a search engine suggest the site has been hacked.
    What a vulnerability scan gives
    Not now. Google's recovery guidance warns that vulnerability scanners can be far more invasive than antivirus scanners, with greater potential to cause unwanted damage to a site, and says to follow all directions, such as backing up the site, before running one. Leave that decision to the specialist handling the incident.
    What a vulnerability assessment gives
    Later. Once the site has been cleaned, an assessment can look for other weaknesses before you rely on the site again.
    What a penetration test gives
    Later, and only once the site is clean. A test can then look for the weakness the attacker may have used, if it is still open, and for whatever else the site exposes.
    How false positives are handled
    Not the question yet. Keeping evidence and stopping harm to visitors come before any list of findings.
    What you receive
    The incident specialist's own findings, under their terms. From a later check or test: a report on the cleaned site, recording what was checked and what was not.
    Limits to keep in mind
    None of the three is incident response, forensic investigation or malware cleanup, and none of them tells you what the attacker did while they had access.
    What fits
    Your hosting provider and an incident response specialist first. Clever Ops does not provide incident response, forensic investigation, malware cleanup or monitoring; an audit or test can follow once the site is clean.
  6. 6. A significant change. A new payment option, a new role such as customer administrator, or a second customer organisation sharing the same database.
    What a vulnerability scan gives
    Picks up components and settings the change brought in. It cannot tell whether the new role can reach functions meant for another, or whether one organisation can see another's records.
    What a vulnerability assessment gives
    Useful when the change is broad but conventional, such as a redesign that touched many pages, to confirm and rank what is new.
    What a penetration test gives
    Suits any change to who can see or do what, and any change to money. Scoped to the new routes, roles and rules plus the existing areas they touch, the test stays in proportion to the change.
    How false positives are handled
    As in any test, results are reproduced before they are reported. Scan results from the new build are useful input: clear the real ones before testing, so the tester's time goes on the new logic.
    What you receive
    A report scoped to the change, naming the new routes, roles and rules tested, the existing areas they touch, and anything left out as an exclusion.
    Limits to keep in mind
    A change-scoped test does not cover the rest of the application again. If much else has changed since the last full test, raise it at scoping.
    What fits
    A penetration test scoped to the change when it affects access, payments or data shared between customers; an assessment when the change is broad and conventional.
  7. 7. An application with no security history. You took it over from an agency or from a developer who has left, or bought a business that runs it, and nobody can say what has been checked.
    What a vulnerability scan gives
    A first inventory of which components and versions are running and which carry published advisories. A quick way to see how far behind the application has fallen.
    What a vulnerability assessment gives
    Usually the best first step: the whole application examined, signed out and signed in, with confirmed findings ranked, so you get a work list and a picture of where the risk sits.
    What a penetration test gives
    Better as a second step, scoped from what the assessment found, unless the application already takes payments or keeps each customer's records apart from the others'.
    How false positives are handled
    An application nobody has maintained can produce long scan output. An assessment merges results that share a cause, confirms which are real and records what could not be checked, so the backlog reflects real work.
    What you receive
    A ranked queue of confirmed findings, each with an owner, a fix and retest criteria, and a map of which areas were examined signed out, signed in or not at all.
    Limits to keep in mind
    An assessment ranks weaknesses one at a time. It does not chain them to see how far they lead, which is the question a penetration test answers.
    What fits
    A vulnerability assessment first, then a penetration test scoped from its queue where the application has several roles, takes payments or holds customer data.
  8. 8. A request for PCI DSS scans. Your acquirer or payment provider asks for external vulnerability scans under PCI DSS.
    What a vulnerability scan gives
    This is a specific kind of scan. The PCI Security Standards Council says an Approved Scanning Vendor conducts external vulnerability scanning to validate adherence with the external scanning requirements of PCI DSS.
    What a vulnerability assessment gives
    Does not stand in for those scans. It can sit beside them as evidence about the web application itself.
    What a penetration test gives
    Also not a substitute for those scans. A web application test answers different questions; ask whoever made the request what else they need from you.
    How false positives are handled
    A matter for the vendor. Ask how it deals with a result you believe is a false positive.
    What you receive
    Scan reports from the Approved Scanning Vendor, which you pass to whoever asked for them.
    Limits to keep in mind
    An external scan looks at what can be reached from the internet. What signed-in users can do inside the application is a question for a test of the application.
    What fits
    An Approved Scanning Vendor from the Council's list. Clever Ops is not a PCI Approved Scanning Vendor, so we refer that work to one; a web application test can sit beside those scans.

Penetration testing vs vulnerability scanning: three different purchases

A vulnerability scan and a penetration test can be quoted side by side, even under the same name, but they answer different questions.

A scan asks which known weaknesses a system appears to have. A penetration test asks what someone could actually do to the system, and how far one foothold would carry them. A vulnerability assessment sits between the two, and it may be the purchase you need when you are comparing the other two.

NIST's glossary defines vulnerability scanning, taking the definition from its Technical Guide to Information Security Testing and Assessment (SP 800-115), as a technique used to identify hosts and host attributes and associated vulnerabilities. The guide explains the mechanism. The system's behaviours and outputs in response to the scanner's attack patterns are compared with the signatures of known vulnerabilities, and the tool reports any matches.

It calls a scanner a relatively fast and easy way to measure exposure to surface vulnerabilities, which it describes as weaknesses that exist in isolation, independent of other vulnerabilities.

The same guide defines penetration testing as security testing in which assessors mimic real-world attacks to identify methods for circumventing the security features of an application, system or network. It adds that most penetration tests search for combinations of weaknesses on one or more systems that together give more access than a single vulnerability could.

The UK's National Cyber Security Centre (NCSC) describes it as a method for gaining assurance in the security of an IT system by attempting to breach some or all of that system's security, using the same tools and techniques as an adversary might.

SP 800-115 draws the line in one sentence: vulnerability scanners check only for the possible existence of a vulnerability, while the attack phase of a penetration test exploits the vulnerability to confirm its existence. An assessment, which NIST's glossary calls a systematic examination, does the confirming and ranking that a scan leaves undone, one weakness at a time.

A penetration test goes further, following one weakness into the next and testing the rules your application is meant to enforce.

  • Vulnerability scan: an unattended automated comparison against known issues. Fast and repeatable, and each result is a possibility until someone checks it.
  • Vulnerability assessment: broad coverage, with a tester confirming each result, combining results that share a cause and ranking what is real, so you receive a work list rather than raw output.
  • Penetration test: a tester working through the application in each role, trying to defeat its controls and recording what worked, what did not, and how far it led.

What each method can show, and where it stops

Each method has a blind spot, and the blind spots differ.

Knowing them is more useful than knowing the names, because the label on a proposal tells you less than its description of the work.

A scan: known issues, quickly, without judgement

SP 800-115 says vulnerability scanning can help identify outdated software versions, missing patches and misconfigurations. On a website or web application, that means components with published advisories, settings that have drifted, and files or headers that should not be there. Because a scan is cheap to repeat, it is good at noticing when something changes between other reviews.

Its limits come from how it works. A scanner relies on a repository of signatures that has to be kept up to date for it to recognise the latest vulnerabilities, and the guide says scanners are unable to detect vulnerabilities revealed only as the result of potentially unending combinations of attack patterns.

The OWASP Web Security Testing Guide v4.2 says business logic flaws cannot be detected by a vulnerability scanner.

A scan has no way to know that one customer should not open another customer's invoice, or that a quote should not be accepted after it expires, because no signature describes your business rules.

An assessment: breadth, confirmed and ranked

An assessment covers the same broad ground as a scan and the signed-in areas too, given test accounts. The difference is the tester between the raw results and your backlog: each result is reproduced or ruled out, its priority comes from what the affected asset holds and who can reach it, and it reaches you with an owner and a fix.

It still treats weaknesses one at a time. It notes combinations it happens to see, but it does not chain one weakness into the next to find out how far they lead.

A penetration test: depth where the risk sits

A penetration test spends its time where a scan cannot go: signed in as each role, calling the API directly, and trying to make the application do what its owners did not intend. Because a test confirms a weakness by using it, within rules of engagement agreed before testing starts, each finding comes with evidence of what was done and what it exposed.

Depth costs breadth. OWASP's testing guide quotes Gary McGraw's Software Penetration Testing: "In practice, a penetration test can only identify a small representative sample of all possible security risks in a system." The NCSC adds that a penetration test can only validate that systems are not vulnerable to known issues on the day of the test.

No method finds every weakness. A test is a sample taken from one build at one time, and a good report says so.

False positives, false negatives and what a clean result means

A false positive is a reported weakness that is not really there.

A false negative is a real weakness that went unreported. The useful question for a buyer is who deals with each, and what the report says about the results it could not settle.

With a scan, the sorting falls to you. SP 800-115 calls vulnerability scanning a somewhat labour-intensive activity that needs a high degree of human involvement to interpret results. When nobody does that interpreting, the cost arrives later: a developer chases a result that was not real, or a customer's reviewer reads a list of alarming items that do not apply to your application.

In an assessment or a penetration test, the sorting happens before the report is written. OWASP's testing guide says it is vital to review the findings and weed out any false positives that may remain, and that reporting an incorrect finding can often undermine the valid message of the rest of a security report.

At Clever Ops, a person confirms each finding before it is reported. In an assessment, a result that would not reproduce is recorded among the checks that found nothing, together with the requests that were tried.

False negatives are harder, because nothing on the page marks them. SP 800-115 says scanners suffer from high false negative rates, and that what one scanner misses can potentially be caught by more scanners or by other forms of testing.

A clean scan therefore means nothing on that run matched what the scanner checks for. A clean area in a penetration test means the tester tried the listed cases on that build, with those accounts, and none of them worked.

Neither one says the system is sound. A report that records its negative results shows you exactly how far a clean result reaches.

How scanning, assessment and testing fit together over a year

The three methods are not rivals.

SP 800-115 says organisations should consider less labour-intensive testing on a regular basis to keep the security posture they need, and that a well-designed program of regularly scheduled network and vulnerability scanning, interspersed with periodic penetration testing, can help prevent many types of attacks and reduce the potential impact of successful ones.

The NCSC is direct about the order of things. It says penetration testing should be viewed as a method for gaining assurance in your organisation's vulnerability assessment and management processes, not as a primary method for identifying vulnerabilities, and that in an ideal world you should know what the testers are going to find before they find it.

It also notes that it is not uncommon for a year or more to pass between penetration tests, so vulnerabilities could exist for long periods without you knowing about them if testing is your only means of validating security.

That gives each method a job. Scans watch for known issues as the application and its components change. An assessment builds the ranked picture when an application is new to you or has changed a great deal. A penetration test examines the places a scan cannot reach, and checks your own picture against what a tester can actually do.

Tie each one to events rather than to the calendar: how often each is worth running depends on how often your application changes and what it holds.

One year for an invented quoting portal

An illustration written for this page, not a client. The business runs a quoting portal where its trade customers sign in, accept quotes and pay deposits online. Its year might run like this.

  • With each release: automated checks in the developer's release workflow look for known issues in components and configuration, and a named person reads the results and turns real ones into tickets. OWASP's testing guide says consideration should be given to building security tests into continuous integration and deployment workflows, to identify low-hanging-fruit weaknesses.
  • When an advisory lands for a component the portal uses: the version is confirmed and updated. A new test is worth considering only if that component guards sign-in, access or payments.
  • Before the portal takes its first real customers: a penetration test of the release build, signed in as each role, then fixes and the retest before launch.
  • When a large customer's security questionnaire arrives: the penetration test report with its retest status, plus a short description of how scan results are reviewed between tests.
  • When a new role, such as a customer administrator, or a new payment option ships: a penetration test scoped to the change and the existing areas it touches.
  • When the portal has changed so much that the last report no longer describes it: a fresh assessment or test of the whole application, starting from the old report's open items.
  • Throughout: each scan result reviewed once, false positives recorded with the reason they were dismissed, and confirmed issues tracked until fixed.

Feed each method into the next

The NCSC suggests sharing a current vulnerability assessment with the testers at scoping, so testing can be designed to support a reasonable opinion on its accuracy and completeness. It lists, among what a test report should include, an opinion on the accuracy of your vulnerability assessment and advice on improving that process.

A finding the test turns up that your own checks missed deserves a second look: the NCSC says such findings should be given special attention, with the aim of identifying ways to spot such issues in future.

How to tell which one a proposal is really offering

Read a proposal for what will be done, not for what it is called.

These details usually settle it.

  • What the provider asks you for. A web address and nothing more fits a scan. A test needs test accounts in each role, a list of the routes or features in scope and the rules your application enforces.
  • Who confirms each result. Ask whether a tester reproduces every finding before it is reported, and what happens to results that will not reproduce.
  • Which roles the work signs in as, and whether one account will try to reach another account's records.
  • Whether the API is called directly, or only through the screens a browser shows.
  • What the sample report looks like. Severity labels with no evidence, generic fix text and no record of checks that found nothing point to scan output. A test report shows the request, the response and the account behind each finding.
  • Whether a retest is included, and which findings it covers.
  • Whether the proposal states its limits: what is excluded, what the work cannot show, and the build and dates it applies to.

When the request came from someone else

None of this makes a scan a poor purchase; it makes it a different one. If a customer or a contract asked for the work, hold each proposal against their exact wording before you sign. A request for a penetration test, in NIST's sense of the term, asks for an attempt to circumvent the application's security features, which a list of scan matches does not show.

Where Clever Ops fits, and what goes to another provider

Clever Ops is a small Australian consultancy based in Gippsland, Victoria, working with businesses earning $1M to $50M in annual revenue.

It tests what customers reach through a browser or an API, and its services line up with the three methods like this.

  • Website security check: an external review of what anyone can reach without logging in, with a person confirming every finding. It stays outside the login, so it is not a penetration test.
  • WordPress security audit: follows each role that can log in, from subscriber to administrator, and checks installed plugin and theme versions with evidence.
  • Vulnerability assessment: breadth across a website or web application, signed out and signed in, with each finding validated by a tester and ranked by asset, role and exploit context.
  • Web application and API penetration test: a signed-in test of a custom application from test accounts in every role, with the API called directly and the business rules the application enforces put to the test.
  • Retests and fixes: every test includes one consolidated retest, findings from another provider's report can be retested as a separate piece of work, and a security fix sprint can handle remediation under its own statement of work.

Scans on their own

Clever Ops does not offer a scanner-only subscription. Scans are a step inside a check or test, not a service of their own. Routine scanning between tests is usually best run by your developer or host, who can act on the results directly.

Requests that go to another provider

Some requests that arrive as a scan or a test need a different kind of provider, and it is better to say so on the first call.

  • PCI DSS external scans. The PCI Security Standards Council says an Approved Scanning Vendor conducts external vulnerability scanning to validate adherence with the external scanning requirements of PCI DSS. Clever Ops is not a PCI Approved Scanning Vendor, so we refer that work to one, and a web application test does not stand in for those scans.
  • A tender, panel or contract that requires a CREST-accredited provider. CREST lists Penetration Testing and Vulnerability Assessment among the disciplines its accreditation covers. Clever Ops is not CREST accredited, so we refer that work to a provider that is.
  • Network vulnerability scanning of servers, firewalls, VPN gateways or office networks, which goes to a network security specialist.
  • A site that may be compromised now. An incident response specialist comes first. Clever Ops does not provide incident response, forensic investigation or malware cleanup.
  • A website or application Clever Ops built or hosts. The same work then runs as a security review rather than an independent test, and the report states who built and who hosts it.
  • A certificate. A scan, assessment or test report is evidence for a decision someone else makes, such as a customer or an auditor, and it is not a certification of your site.

Frequently asked questions

No. In NIST's terms, a scan checks for the possible existence of known vulnerabilities, while a penetration test confirms a weakness by exploiting it and looks for combinations that lead further. A scan report lists matches. A penetration test report shows what a tester working through the application could actually do, with evidence for each finding. If someone asked you for a penetration test, a scan report answers a different question.

Start from what you need to learn. If you need a complete, ranked list of what is wrong across the application, that is an assessment: broad, confirmed and ordered for fixing. If you need to know whether someone could reach data or actions they should not, and how far one weakness leads into the next, that is a penetration test. They suit different stages, and an assessment's list makes a good starting scope for a later test.

Judgement. A scan hands you raw matches labelled with the scanner's own severity scale, and SP 800-115 notes that scanners often define risk levels by their own proprietary methods. An assessment puts a tester between those matches and your backlog: each result reproduced or ruled out, related results combined into one item, and each confirmed item ranked by what it exposes in your application and given to the person who can fix it.

No. It means nothing on that run matched what the scanner checks for. SP 800-115 says scanners suffer from high false negative rates, and OWASP says business logic flaws cannot be detected by a vulnerability scanner. Treat a clean scan as one negative result for the checks it ran, on that date, and keep it beside the other evidence you have rather than in place of it.

In most cases, yes. A test is a sample of one build at one time, and the components your application uses can gain new published advisories after it ends. SP 800-115 says a well-designed program of regularly scheduled network and vulnerability scanning, interspersed with periodic penetration testing, can help prevent many types of attacks. The scans catch known issues as they appear, and the next test examines what scans cannot reach.

Deal with the known issues first. Run a scan or your platform's own security checks and fix whatever turns out to be real before the test starts, so the tester's time goes on access between accounts, business rules and the other areas a scan cannot reach. For an application nobody has examined before, an assessment is often the better first step, because it gives you a confirmed, ranked list to fix and to scope the test from.

Read the request's exact wording first. If it names a penetration test, a scan report describes different work, because it lists possible issues rather than showing what a tester could do to the application. If the wording is looser, such as asking how you find and fix vulnerabilities, an assessment report or a description of how your scan results are reviewed may answer it. When in doubt, ask the customer what they will accept before you buy anything.

Scope the right test for your application

Tell us what you have built, who uses it and when it needs to launch. You will get a plain-English recommendation: a website audit, a web application test or a referral to a specialist.