Skip to main content
Security comparisonReviewed

Black box penetration testing compared with grey box and white box: what each one can see

What a tester is told decides what a penetration test can find.

This page sets out how NIST, the UK NCSC and OWASP draw the lines between black box, grey box and white box testing, where they disagree, what each approach asks of your team, and how one invented booking application looks when it is tested all three ways.

Three-lens walkthrough: one booking application, black box to white box

An illustration written for this page, not a client or a real test. book.lab.test is an invented booking application for a fictional mobile dog-grooming business, with two roles, customer and staff. Customers register through open sign-up and request grooms at home, and each booking holds an address, a phone number, the dog's name and a note for the groomer, such as a side-gate code.

How to use it

Staff confirm bookings, waive deposits and export the customer list, and a nightly job sends that list to a mailing service. Here, black box means the production address and public sign-up only; grey box adds test accounts, documentation and written rules, tested on staging; and white box adds read-only code, with code findings taken back to staging. NIST SP 800-115 calls a mix of white box and black box techniques grey box testing, so by its wording the white box column could be called grey box too, which is one reason to ask a provider what each label includes. Read down a column for one approach, or across a row to compare all three. To reuse it, replace the four flaws with risks in your own application and keep the rows for what is given, what you set up, what is tested, what stays out of view and what evidence results. Every hostname, account, route and result is invented.

10 rows

Three-lens walkthrough: one booking application, black box to white box

LensBlack box: the production address onlyGrey box: accounts, rules and documentation, on stagingWhite box: the grey box view plus read-only code
What the tester is givenThe production address, book.lab.test, and nothing else. Public sign-up is open, and the rules of engagement allow the tester to register customer accounts through it, listed for removal afterwards.The production and staging addresses. Two customer accounts (cust-a1, cust-a2) and two staff accounts (staff-1, staff-2), each with sample bookings. The API documentation the developers keep. The written booking rules: customers see and change only their own bookings, only staff confirm bookings or waive deposits, a customer can cancel free of charge until 24 hours before the appointment, and only staff export the customer list.Everything in the grey box column, plus read-only access to the repository at the commit deployed on staging, and the names, not the values, of the environment variables each environment uses.
What your team sets up firstA signed authorisation naming the production hostname, the testing window and the production risks you accept. A check of the host's testing policy. A decision on whether accounts may be created through public sign-up. A technical contact who can be reached during testing. No accounts or documents to prepare.The black box items, with staging added to the authorisation. The four test accounts and their sample bookings, the written rules and the current API documentation. A staging copy linked to a non-live payment account, so deposit checks charge no real card, and a check that staging matches production. Here that check covered the code but not every setting.The grey box items, plus repository access for the agreed period, revoked once the retest is complete. The list of environment variable names. A developer who can answer questions about the code. Data handling terms that treat code excerpts as evidence.
Paths the tester takesOn production: public pages, sign-up, sign-in, password reset and the public availability route. The JavaScript served to a visitor and to a signed-in customer. Customer routes, using two accounts the tester registered. Staff paths guessed from the naming of the customer API.Every customer and staff screen and the API calls behind it, each replayed from the other customer's session, and each staff request replayed from a customer session. Each documented route called directly. Each written rule tried on test records, including a cancellation inside the 24-hour window, which was refused as the rules require.The routing files, for every route the application answers, scheduled jobs included. The check each group of routes passes through. The shared booking lookup and every route that calls it. How settings are read. Each suspected flaw then taken back to staging as a pair of requests where the scope allows.
Flaw 1: the booking lookup checks the id but not the ownerFound, because sign-up is open. The second registered account read the first account's booking, with its address and gate-code note, and cancelled another booking the first account had made. Both accounts and all their bookings were the tester's own. With sign-up closed, this column would have no way to the finding.Found on three documented routes: booking detail, cancel and receipt. cust-a2's session read and cancelled cust-a1's test bookings, each request shown beside the owner's own request for the same booking.Found at the root cause: one shared lookup that filters by booking id alone. The code shows a fourth caller, a reschedule route behind a feature flag that is off on staging, so that route is reported as code only, with the reason.
Flaw 2: staff routes check that someone is signed in, not that they are staffPartly found. A guessed path, GET /api/staff/calendar, returned the week's staff calendar to a self-registered customer. The response held real customers' details, so the tester stopped at one request, kept the status and field names with every personal detail redacted, and told the technical contact. The other staff routes were not guessed and stayed untested.Found on four routes. Requests captured from staff-1's screens for the calendar, confirming a booking, waiving a deposit and the customer export all succeeded from cust-a1's session on staging. A fifth route, for editing service prices, lost its screen in a redesign but still answers. It is missing from the documentation, so it was not tested.Found at the root cause: the staff route group applies the signed-in check where the staff check was intended. All five routes are listed, price editing included, and each was confirmed on staging from a customer session, the price change on a test service only.
Flaw 3: a fallback key in the repository opens the customer exportNot found. No page, script or response mentions the export route the nightly job uses.Not found. The route is absent from the documentation, and no screen calls it.Found. The export route reads its key from an environment variable and, when the variable is unset, falls back to a fixed value written in the repository. On staging the variable was unset, and the fixed value returned the staging customer list. On production it was not tried, because success would download real customers' details. The report marks it confirmed on staging and asks your team to check the production variable.
Flaw 4: production shows full error detailsFound. A malformed date sent to the public availability route on production returned a stack trace with file paths and the framework version.Not found. Staging returns a generic error page, because its debug setting is off. Staging matched production in its code but not in this setting.Seen as a condition, not a finding. The error handler prints full details when a debug variable is true, but the production value lives outside the repository, so the report records an observation for your team to check.
What stays out of viewRoles you issue yourself, here staff, apart from paths that happen to be guessed. Every customer route, had sign-up been closed. Routes nothing public mentions. Business rules nobody told the tester, such as who may waive a deposit.Routes missing from the documentation. Flaws visible only in code, such as the fallback key. Settings that differ between staging and production. Any role without test accounts.Values set outside the repository, such as production environment variables and the host's own settings. Any difference between the commit read and what is deployed. Code outside the areas the scope names, when code access is targeted rather than whole.
Evidence the report can carryRequests and responses from production, sent with no session and from self-registered accounts, with dates. The guessed paths and what each returned. Negative results for the public routes. Together they show what a stranger, or anyone who signs up, could do on production on those dates.A pair of requests for each finding, naming the account, role and staging build. Negative results for every documented route and role, and for each written rule. Together they show what each role can do on that build, within the documentation supplied.Everything in the grey box column, plus the file, line and commit of each root cause, the sibling routes that share it, and a label on each finding: confirmed on staging, or code only with the reason. Together they show why each flaw exists and where else it reaches, for the commit read.
Where this lens fitsYou want to know what an outsider, or anyone who can register, can do to the system customers use today, or you want a first look before deeper work. A weak fit on its own when staff functions or private records sit behind logins you issue.Customers sign in and own records, staff have functions customers must not reach, and you can provide accounts and written rules. Only as strong as the information supplied, so out-of-date documentation leaves gaps.The riskiest logic sits in code your developers can point to, an earlier finding may have siblings, or routes exist that no screen calls, such as jobs and callbacks. Pair it with signed-out checks on production for settings that code cannot show.

Lens

Black box: the production address only

  1. What the tester is given
    Black box: the production address only
    The production address, book.lab.test, and nothing else. Public sign-up is open, and the rules of engagement allow the tester to register customer accounts through it, listed for removal afterwards.
    Grey box: accounts, rules and documentation, on staging
    The production and staging addresses. Two customer accounts (cust-a1, cust-a2) and two staff accounts (staff-1, staff-2), each with sample bookings. The API documentation the developers keep. The written booking rules: customers see and change only their own bookings, only staff confirm bookings or waive deposits, a customer can cancel free of charge until 24 hours before the appointment, and only staff export the customer list.
    White box: the grey box view plus read-only code
    Everything in the grey box column, plus read-only access to the repository at the commit deployed on staging, and the names, not the values, of the environment variables each environment uses.
  2. What your team sets up first
    Black box: the production address only
    A signed authorisation naming the production hostname, the testing window and the production risks you accept. A check of the host's testing policy. A decision on whether accounts may be created through public sign-up. A technical contact who can be reached during testing. No accounts or documents to prepare.
    Grey box: accounts, rules and documentation, on staging
    The black box items, with staging added to the authorisation. The four test accounts and their sample bookings, the written rules and the current API documentation. A staging copy linked to a non-live payment account, so deposit checks charge no real card, and a check that staging matches production. Here that check covered the code but not every setting.
    White box: the grey box view plus read-only code
    The grey box items, plus repository access for the agreed period, revoked once the retest is complete. The list of environment variable names. A developer who can answer questions about the code. Data handling terms that treat code excerpts as evidence.
  3. Paths the tester takes
    Black box: the production address only
    On production: public pages, sign-up, sign-in, password reset and the public availability route. The JavaScript served to a visitor and to a signed-in customer. Customer routes, using two accounts the tester registered. Staff paths guessed from the naming of the customer API.
    Grey box: accounts, rules and documentation, on staging
    Every customer and staff screen and the API calls behind it, each replayed from the other customer's session, and each staff request replayed from a customer session. Each documented route called directly. Each written rule tried on test records, including a cancellation inside the 24-hour window, which was refused as the rules require.
    White box: the grey box view plus read-only code
    The routing files, for every route the application answers, scheduled jobs included. The check each group of routes passes through. The shared booking lookup and every route that calls it. How settings are read. Each suspected flaw then taken back to staging as a pair of requests where the scope allows.
  4. Flaw 1: the booking lookup checks the id but not the owner
    Black box: the production address only
    Found, because sign-up is open. The second registered account read the first account's booking, with its address and gate-code note, and cancelled another booking the first account had made. Both accounts and all their bookings were the tester's own. With sign-up closed, this column would have no way to the finding.
    Grey box: accounts, rules and documentation, on staging
    Found on three documented routes: booking detail, cancel and receipt. cust-a2's session read and cancelled cust-a1's test bookings, each request shown beside the owner's own request for the same booking.
    White box: the grey box view plus read-only code
    Found at the root cause: one shared lookup that filters by booking id alone. The code shows a fourth caller, a reschedule route behind a feature flag that is off on staging, so that route is reported as code only, with the reason.
  5. Flaw 2: staff routes check that someone is signed in, not that they are staff
    Black box: the production address only
    Partly found. A guessed path, GET /api/staff/calendar, returned the week's staff calendar to a self-registered customer. The response held real customers' details, so the tester stopped at one request, kept the status and field names with every personal detail redacted, and told the technical contact. The other staff routes were not guessed and stayed untested.
    Grey box: accounts, rules and documentation, on staging
    Found on four routes. Requests captured from staff-1's screens for the calendar, confirming a booking, waiving a deposit and the customer export all succeeded from cust-a1's session on staging. A fifth route, for editing service prices, lost its screen in a redesign but still answers. It is missing from the documentation, so it was not tested.
    White box: the grey box view plus read-only code
    Found at the root cause: the staff route group applies the signed-in check where the staff check was intended. All five routes are listed, price editing included, and each was confirmed on staging from a customer session, the price change on a test service only.
  6. Flaw 3: a fallback key in the repository opens the customer export
    Black box: the production address only
    Not found. No page, script or response mentions the export route the nightly job uses.
    Grey box: accounts, rules and documentation, on staging
    Not found. The route is absent from the documentation, and no screen calls it.
    White box: the grey box view plus read-only code
    Found. The export route reads its key from an environment variable and, when the variable is unset, falls back to a fixed value written in the repository. On staging the variable was unset, and the fixed value returned the staging customer list. On production it was not tried, because success would download real customers' details. The report marks it confirmed on staging and asks your team to check the production variable.
  7. Flaw 4: production shows full error details
    Black box: the production address only
    Found. A malformed date sent to the public availability route on production returned a stack trace with file paths and the framework version.
    Grey box: accounts, rules and documentation, on staging
    Not found. Staging returns a generic error page, because its debug setting is off. Staging matched production in its code but not in this setting.
    White box: the grey box view plus read-only code
    Seen as a condition, not a finding. The error handler prints full details when a debug variable is true, but the production value lives outside the repository, so the report records an observation for your team to check.
  8. What stays out of view
    Black box: the production address only
    Roles you issue yourself, here staff, apart from paths that happen to be guessed. Every customer route, had sign-up been closed. Routes nothing public mentions. Business rules nobody told the tester, such as who may waive a deposit.
    Grey box: accounts, rules and documentation, on staging
    Routes missing from the documentation. Flaws visible only in code, such as the fallback key. Settings that differ between staging and production. Any role without test accounts.
    White box: the grey box view plus read-only code
    Values set outside the repository, such as production environment variables and the host's own settings. Any difference between the commit read and what is deployed. Code outside the areas the scope names, when code access is targeted rather than whole.
  9. Evidence the report can carry
    Black box: the production address only
    Requests and responses from production, sent with no session and from self-registered accounts, with dates. The guessed paths and what each returned. Negative results for the public routes. Together they show what a stranger, or anyone who signs up, could do on production on those dates.
    Grey box: accounts, rules and documentation, on staging
    A pair of requests for each finding, naming the account, role and staging build. Negative results for every documented route and role, and for each written rule. Together they show what each role can do on that build, within the documentation supplied.
    White box: the grey box view plus read-only code
    Everything in the grey box column, plus the file, line and commit of each root cause, the sibling routes that share it, and a label on each finding: confirmed on staging, or code only with the reason. Together they show why each flaw exists and where else it reaches, for the commit read.
  10. Where this lens fits
    Black box: the production address only
    You want to know what an outsider, or anyone who can register, can do to the system customers use today, or you want a first look before deeper work. A weak fit on its own when staff functions or private records sit behind logins you issue.
    Grey box: accounts, rules and documentation, on staging
    Customers sign in and own records, staff have functions customers must not reach, and you can provide accounts and written rules. Only as strong as the information supplied, so out-of-date documentation leaves gaps.
    White box: the grey box view plus read-only code
    The riskiest logic sits in code your developers can point to, an earlier finding may have siblings, or routes exist that no screen calls, such as jobs and callbacks. Pair it with signed-out checks on production for settings that code cannot show.

What black box penetration testing means

Black box penetration testing puts the tester where an outsider starts: with the address of your application and little or no knowledge of how it is built.

There is no source code, no design document and no list of routes, so everything the tester learns comes from what the application reveals as it is used. NIST, the UK National Cyber Security Centre (NCSC) and OWASP agree on that starting point, and part ways after it, as the next section shows.

Grey box testing gives the tester part of the picture, and white box testing gives the internals as well. Between them, the three approaches trade a realistic starting point against how much of the application a test can reach in the time it has. The NCSC states the trade plainly.

It says testing with no information about the internals more accurately models the risk faced from attackers who are unknown or unaffiliated to the target organisation, and that the lack of information can also leave vulnerabilities undiscovered in the time allocated for testing.

The box labels describe how much the tester is told. They leave out what decides the result for a web application: which accounts the tester holds, which environment is tested, and whether any code is read. This page compares the published definitions, shows where they disagree, sets out what each approach asks of your team, and walks one invented booking application through all three, so you can see which flaws each approach would find and which it would miss.

How NIST, the NCSC and OWASP draw the lines

Each body draws its lines in a different place, and some use different words for the same idea.

Which definition a provider has in mind changes what a quote means, so it is worth knowing the four views below before you compare proposals.

NIST: no knowledge, some knowledge, substantial knowledge

The NIST Computer Security Resource Center glossary gives one set of definitions that turns on how much is known about the internal structure and implementation detail of the assessment object, the thing being assessed. 1 min read

NIST again: what the test examines

The same glossary pages carry a second definition of black box and white box, from NIST SP 800-192, that turns on what is examined rather than what is known. 1 min read

The UK NCSC: open box and closed box, with nothing between

The NCSC's penetration testing guidance sorts tests by how much information about your system the testers are armed with, in a list it calls illustrative rather than comprehensive. 1 min read

OWASP: penetration testing as black box

The OWASP Web Security Testing Guide (WSTG) v4.2 ties the labels to activities. 1 min read

Where the definitions disagree, and what to ask instead

Set side by side, the definitions differ on four points that matter when you read a quote.

  • What the label measures. The SP 800-53A wording in NIST's glossary, the NCSC and OWASP measure what the tester knows. The SP 800-192 wording measures what the test examines, and SP 800-115 measures which techniques are used. A provider can describe one engagement accurately by one measure and differently by another.
  • Where grey box sits. NIST's glossary and OWASP describe partial knowledge. SP 800-115 describes a mix of white box and black box techniques. The NCSC's list names no middle category at all.
  • Whether the tester signs in. None of the definitions settles it. The OWASP passage that calls penetration testing black-box testing also says the team is typically able to access an application as if they were users, and OWASP's own horizontal access test, WSTG-ATHZ-02, has the tester register or generate two users with identical privileges. On that reading, accounts and box colour are separate questions.
  • What your own staff know. The boxes describe what the tester knows. Whether your administrators know a test is under way is a different matter. The NCSC notes that how much knowledge of the test is given to system administrators can vary within the same test regime, and SP 800-115 calls testing with the knowledge and consent of the IT staff overt, and testing without their knowledge but with the full knowledge and permission of upper management covert.

Three questions that pin a label down

Because the labels move, ask for the substance. Three answers settle what a quote means, whatever box it names.

  • Which accounts will the tester hold, in which roles, and will they be accounts you create for the test or accounts the tester registers through public sign-up?
  • What will the tester read: none of your documents, your API documentation and business rules, or your source code? For code, which areas?
  • Which environment will be tested, production or a staging copy, and who has checked that the copy matches production in its settings as well as its code?

One booking application through three lenses

The walkthrough on this page takes one invented application and tests it three ways. book.lab.test is a booking application for a fictional mobile dog-grooming business, written for this page and not drawn from any client.

It has two roles. Customers register through open sign-up and request a groom at home, and each booking holds an address, a phone number, the dog's name and a note for the groomer, such as a side-gate code. Staff confirm bookings, waive deposits for regulars and export the customer list, and a nightly job sends that list to the business's mailing service.

In the black box column the tester has the production address and whatever public sign-up allows. In the grey box column the tester has two test accounts in each role, the developers' API documentation and the business's written booking rules, and works on a staging copy.

The white box column adds read-only access to the code and takes each code finding back to staging. Four flaws run through all three columns:

  • The booking lookup checks the id but not the owner. All three columns find it, the black box column only because sign-up is open. The white box column traces it to one shared function and finds a further route that uses it.
  • Staff routes check that someone is signed in, not that they are staff. The black box column finds one route by guessing its path. The grey box column confirms four from the staff screens. The white box column traces them all to the check applied to the whole staff route group, and finds a fifth route whose screen was removed but which still answers.
  • A fallback key written in the repository opens the customer export the nightly job uses. Only the white box column finds it, because nothing public or documented mentions the route.
  • Production shows full error details because of a setting staging does not share. Only the black box column finds it, because it is the only one pointed at production.

What the walkthrough shows

No column finds all four flaws, and the column with the least information is the only one to find the fourth. The point is not that one approach wins. Each approach answers a narrower question than its label suggests, and a scope can combine them, for example signed-out checks on production alongside signed-in or code-assisted work on staging.

What each approach asks of you, and where its coverage stops

Each step from black box towards white box asks more of your team before testing starts and gives the tester a wider view.

None of the three is the right choice for every engagement, and each has a blind spot that another one covers.

Black box: least to prepare, narrowest view

A black box test of production asks the least of you: a signed authorisation that names the production hostnames, the testing window and the production risks you accept, a check of your host's testing policy, a decision on whether the tester may register accounts through public sign-up, and a technical contact. 1 min read

Grey box: accounts and rules widen the view

A grey box test asks for test accounts, sample records, the rules your application is meant to enforce and whatever documentation you keep. 1 min read

White box: code access reaches routes no screen calls

A white box test asks for read-only access to the repository at the commit deployed on the test environment, the names of the settings each environment uses, and a developer who can answer questions. 1 min read

Choosing an approach for your application

Start from what sits behind your login and what you can share, not from the label.

Whatever you choose, the report should record the accounts and roles used, the environment, the build and any code read, so a reader can see what the test covered without relying on the box it names. These questions usually settle the choice:

  • Does anyone other than your own editors sign in? If not, most of what you expose can be seen from outside, and a signed-out test covers it. The website security check, which stays outside the login, is built for that case.
  • Do customers hold accounts and records of their own, or do staff have functions customers must not reach? Then the access rules that protect those records sit behind the login. A black box test reaches them only if sign-up is open, and reaches roles you issue yourself only by chance, so test accounts in every role belong in the scope.
  • Does the application have routes no screen calls, such as scheduled jobs, exports or callbacks from payment and messaging services, or logic concentrated in a few files your developers can name? Code access reaches those directly.
  • Was an earlier finding fixed in one place? Reading the code shows whether the same mistake sits elsewhere, which requests alone can only establish route by route.
  • Is the question what an outsider can do to production today? Then keep the production hostnames in scope for at least the signed-out checks, with the production risks recorded in the authorisation, because a staging copy answers a slightly different question.
  • Does a tender, panel or contract require a CREST-accredited provider? Clever Ops is not CREST accredited, so we refer that work to a provider that is.

How Clever Ops describes the same choices

Clever Ops scopes web application tests by depth rather than by box: unauthenticated, authenticated and code-assisted.

Against the definitions on this page, unauthenticated testing sits closest to black box, authenticated testing with test accounts to grey box, and code-assisted testing moves towards white box without reaching it, because code access is targeted at the areas the scope names rather than a line-by-line read of the whole repository.

A full review of the code is a separate piece of work, the secure code review. The methodology guide explains what each depth reaches.

A web application and API penetration test is authenticated by default, with two test accounts in each role in scope. When code access is added, it is read-only, a weakness seen in the code is confirmed against a running build where the scope allows, and a finding that rests on the code alone says so.

Testing starts only after the owner of each target signs an authorisation and your hosting provider's testing policy has been checked. The authorisation lists the source IP addresses test traffic will come from, so your team and your host can tell it apart from an attack.

Destructive actions are off by default for real people's accounts and records that existed before the test, and reset, lockout and deletion checks run only on test accounts and test records where the rules of engagement allow them.

If Clever Ops built or hosts the application, the same work runs as a security review rather than an independent test, and the report says who built it and who hosts it.

Frequently asked questions

A penetration test in which the tester starts with little or no knowledge of the application's internals and learns only what the running application reveals. NIST's glossary, using the SP 800-53A Rev. 5 wording, calls black box testing a test methodology that assumes no knowledge of the internal structure and implementation detail of the assessment object. The OWASP testing guide describes its own method as based on the black box approach, where the tester knows nothing or has very little information about the application.

Testing in which the tester has explicit and substantial knowledge of the application's internals, in the definition NIST's glossary takes from SP 800-137. SP 800-115 ties white box techniques to direct analysis of the source code and says they tend to be more efficient and cost-effective for finding defects in custom applications, but cannot detect problems in interfaces between components or in installation-time configuration. Code findings are strongest when they are confirmed against a running build.

Testing with part of the picture. NIST's glossary, quoting CNSSI 4009-2022, defines it as assuming some knowledge of the internal structure and implementation detail, and OWASP describes the tester as having some partial knowledge about the application. SP 800-115 instead uses the term for tests that combine white box and black box techniques. NIST and OWASP spell it the US way, with an a. Because the definitions differ, ask what the tester will be given: test accounts, documentation, written business rules or some of the code.

Neither in general, because they find different things. OWASP lists among the strengths of penetration testing that it tests the code actually exposed, and among the drawbacks of source code review that the code deployed might differ from the code analysed. In the booking walkthrough on this page, only the white box column found the fallback key in the repository, and only the black box column found the error setting that existed on production alone. Choose by the risks in your application, or combine the approaches.

Its starting point is more realistic for an outsider who knows nothing about your application. The NCSC says testing with no information about the internals more accurately models attackers who are unknown or unaffiliated to the organisation. A test has a fixed window, though, and the same guidance warns that the lack of information can leave vulnerabilities undiscovered in the time allocated. An outsider who registers through your public sign-up also ends up holding a customer account, which is the grey box tester's starting point for that role.

No. Without it, a test still covers what the running application does for each role the tester holds accounts in. Code access adds the routes and root causes that requests alone cannot show, and OWASP's testing principles advise giving source code to the security staff where it is available. If you share it, read-only access to the areas in scope is enough, you can revoke the access once the retest is complete, and code excerpts in a report are handled as evidence like any other, with secrets redacted.

No. The tester works from test accounts created for the test, two in each role, each with sample records, so nobody's real login is shared. Lockout and reset checks run only on those test accounts, where the rules of engagement allow them. Keep the accounts until the retest is complete, so the same requests can be replayed, and remove them afterwards.

Scope the right test for your application

Tell us what you have built, who uses it and when it needs to launch. You will get a plain-English recommendation: a website audit, a web application test or a referral to a specialist.