Black box penetration testing compared with grey box and white box: what each one can see
What a tester is told decides what a penetration test can find.
This page sets out how NIST, the UK NCSC and OWASP draw the lines between black box, grey box and white box testing, where they disagree, what each approach asks of your team, and how one invented booking application looks when it is tested all three ways.
On this page
Three-lens walkthrough: one booking application, black box to white box
An illustration written for this page, not a client or a real test. book.lab.test is an invented booking application for a fictional mobile dog-grooming business, with two roles, customer and staff. Customers register through open sign-up and request grooms at home, and each booking holds an address, a phone number, the dog's name and a note for the groomer, such as a side-gate code.
How to use itHide the instructions
Staff confirm bookings, waive deposits and export the customer list, and a nightly job sends that list to a mailing service. Here, black box means the production address and public sign-up only; grey box adds test accounts, documentation and written rules, tested on staging; and white box adds read-only code, with code findings taken back to staging. NIST SP 800-115 calls a mix of white box and black box techniques grey box testing, so by its wording the white box column could be called grey box too, which is one reason to ask a provider what each label includes. Read down a column for one approach, or across a row to compare all three. To reuse it, replace the four flaws with risks in your own application and keep the rows for what is given, what you set up, what is tested, what stays out of view and what evidence results. Every hostname, account, route and result is invented.
10 rows
Lens
Black box: the production address only
What the tester is given
- Black box: the production address only
- The production address, book.lab.test, and nothing else. Public sign-up is open, and the rules of engagement allow the tester to register customer accounts through it, listed for removal afterwards.
- Grey box: accounts, rules and documentation, on staging
- The production and staging addresses. Two customer accounts (cust-a1, cust-a2) and two staff accounts (staff-1, staff-2), each with sample bookings. The API documentation the developers keep. The written booking rules: customers see and change only their own bookings, only staff confirm bookings or waive deposits, a customer can cancel free of charge until 24 hours before the appointment, and only staff export the customer list.
- White box: the grey box view plus read-only code
- Everything in the grey box column, plus read-only access to the repository at the commit deployed on staging, and the names, not the values, of the environment variables each environment uses.
What your team sets up first
- Black box: the production address only
- A signed authorisation naming the production hostname, the testing window and the production risks you accept. A check of the host's testing policy. A decision on whether accounts may be created through public sign-up. A technical contact who can be reached during testing. No accounts or documents to prepare.
- Grey box: accounts, rules and documentation, on staging
- The black box items, with staging added to the authorisation. The four test accounts and their sample bookings, the written rules and the current API documentation. A staging copy linked to a non-live payment account, so deposit checks charge no real card, and a check that staging matches production. Here that check covered the code but not every setting.
- White box: the grey box view plus read-only code
- The grey box items, plus repository access for the agreed period, revoked once the retest is complete. The list of environment variable names. A developer who can answer questions about the code. Data handling terms that treat code excerpts as evidence.
Paths the tester takes
- Black box: the production address only
- On production: public pages, sign-up, sign-in, password reset and the public availability route. The JavaScript served to a visitor and to a signed-in customer. Customer routes, using two accounts the tester registered. Staff paths guessed from the naming of the customer API.
- Grey box: accounts, rules and documentation, on staging
- Every customer and staff screen and the API calls behind it, each replayed from the other customer's session, and each staff request replayed from a customer session. Each documented route called directly. Each written rule tried on test records, including a cancellation inside the 24-hour window, which was refused as the rules require.
- White box: the grey box view plus read-only code
- The routing files, for every route the application answers, scheduled jobs included. The check each group of routes passes through. The shared booking lookup and every route that calls it. How settings are read. Each suspected flaw then taken back to staging as a pair of requests where the scope allows.
Flaw 1: the booking lookup checks the id but not the owner
- Black box: the production address only
- Found, because sign-up is open. The second registered account read the first account's booking, with its address and gate-code note, and cancelled another booking the first account had made. Both accounts and all their bookings were the tester's own. With sign-up closed, this column would have no way to the finding.
- Grey box: accounts, rules and documentation, on staging
- Found on three documented routes: booking detail, cancel and receipt. cust-a2's session read and cancelled cust-a1's test bookings, each request shown beside the owner's own request for the same booking.
- White box: the grey box view plus read-only code
- Found at the root cause: one shared lookup that filters by booking id alone. The code shows a fourth caller, a reschedule route behind a feature flag that is off on staging, so that route is reported as code only, with the reason.
Flaw 2: staff routes check that someone is signed in, not that they are staff
- Black box: the production address only
- Partly found. A guessed path, GET /api/staff/calendar, returned the week's staff calendar to a self-registered customer. The response held real customers' details, so the tester stopped at one request, kept the status and field names with every personal detail redacted, and told the technical contact. The other staff routes were not guessed and stayed untested.
- Grey box: accounts, rules and documentation, on staging
- Found on four routes. Requests captured from staff-1's screens for the calendar, confirming a booking, waiving a deposit and the customer export all succeeded from cust-a1's session on staging. A fifth route, for editing service prices, lost its screen in a redesign but still answers. It is missing from the documentation, so it was not tested.
- White box: the grey box view plus read-only code
- Found at the root cause: the staff route group applies the signed-in check where the staff check was intended. All five routes are listed, price editing included, and each was confirmed on staging from a customer session, the price change on a test service only.
Flaw 3: a fallback key in the repository opens the customer export
- Black box: the production address only
- Not found. No page, script or response mentions the export route the nightly job uses.
- Grey box: accounts, rules and documentation, on staging
- Not found. The route is absent from the documentation, and no screen calls it.
- White box: the grey box view plus read-only code
- Found. The export route reads its key from an environment variable and, when the variable is unset, falls back to a fixed value written in the repository. On staging the variable was unset, and the fixed value returned the staging customer list. On production it was not tried, because success would download real customers' details. The report marks it confirmed on staging and asks your team to check the production variable.
Flaw 4: production shows full error details
- Black box: the production address only
- Found. A malformed date sent to the public availability route on production returned a stack trace with file paths and the framework version.
- Grey box: accounts, rules and documentation, on staging
- Not found. Staging returns a generic error page, because its debug setting is off. Staging matched production in its code but not in this setting.
- White box: the grey box view plus read-only code
- Seen as a condition, not a finding. The error handler prints full details when a debug variable is true, but the production value lives outside the repository, so the report records an observation for your team to check.
What stays out of view
- Black box: the production address only
- Roles you issue yourself, here staff, apart from paths that happen to be guessed. Every customer route, had sign-up been closed. Routes nothing public mentions. Business rules nobody told the tester, such as who may waive a deposit.
- Grey box: accounts, rules and documentation, on staging
- Routes missing from the documentation. Flaws visible only in code, such as the fallback key. Settings that differ between staging and production. Any role without test accounts.
- White box: the grey box view plus read-only code
- Values set outside the repository, such as production environment variables and the host's own settings. Any difference between the commit read and what is deployed. Code outside the areas the scope names, when code access is targeted rather than whole.
Evidence the report can carry
- Black box: the production address only
- Requests and responses from production, sent with no session and from self-registered accounts, with dates. The guessed paths and what each returned. Negative results for the public routes. Together they show what a stranger, or anyone who signs up, could do on production on those dates.
- Grey box: accounts, rules and documentation, on staging
- A pair of requests for each finding, naming the account, role and staging build. Negative results for every documented route and role, and for each written rule. Together they show what each role can do on that build, within the documentation supplied.
- White box: the grey box view plus read-only code
- Everything in the grey box column, plus the file, line and commit of each root cause, the sibling routes that share it, and a label on each finding: confirmed on staging, or code only with the reason. Together they show why each flaw exists and where else it reaches, for the commit read.
Where this lens fits
- Black box: the production address only
- You want to know what an outsider, or anyone who can register, can do to the system customers use today, or you want a first look before deeper work. A weak fit on its own when staff functions or private records sit behind logins you issue.
- Grey box: accounts, rules and documentation, on staging
- Customers sign in and own records, staff have functions customers must not reach, and you can provide accounts and written rules. Only as strong as the information supplied, so out-of-date documentation leaves gaps.
- White box: the grey box view plus read-only code
- The riskiest logic sits in code your developers can point to, an earlier finding may have siblings, or routes exist that no screen calls, such as jobs and callbacks. Pair it with signed-out checks on production for settings that code cannot show.
What black box penetration testing means
Black box penetration testing puts the tester where an outsider starts: with the address of your application and little or no knowledge of how it is built.
There is no source code, no design document and no list of routes, so everything the tester learns comes from what the application reveals as it is used. NIST, the UK National Cyber Security Centre (NCSC) and OWASP agree on that starting point, and part ways after it, as the next section shows.
Grey box testing gives the tester part of the picture, and white box testing gives the internals as well. Between them, the three approaches trade a realistic starting point against how much of the application a test can reach in the time it has. The NCSC states the trade plainly.
It says testing with no information about the internals more accurately models the risk faced from attackers who are unknown or unaffiliated to the target organisation, and that the lack of information can also leave vulnerabilities undiscovered in the time allocated for testing.
The box labels describe how much the tester is told. They leave out what decides the result for a web application: which accounts the tester holds, which environment is tested, and whether any code is read. This page compares the published definitions, shows where they disagree, sets out what each approach asks of your team, and walks one invented booking application through all three, so you can see which flaws each approach would find and which it would miss.
How NIST, the NCSC and OWASP draw the lines
Each body draws its lines in a different place, and some use different words for the same idea.
Which definition a provider has in mind changes what a quote means, so it is worth knowing the four views below before you compare proposals.
The NIST Computer Security Resource Center glossary gives one set of definitions that turns on how much is known about the internal structure and implementation detail of the assessment object, the thing being assessed. Its black box entry, from NIST SP 800-53A Rev. 5, where the same wording defines basic testing, is a test methodology that assumes no knowledge of that structure and detail.
Its grey box entry, from CNSSI 4009-2022, assumes some knowledge and gives focused testing as another name, and SP 800-53A Rev. 5 uses the same definition under that name.
Its white box entry, from NIST SP 800-137, where it defines comprehensive testing, assumes explicit and substantial knowledge. The glossary files the middle term under the US spelling, with an a.
The same glossary pages carry a second definition of black box and white box, from NIST SP 800-192, that turns on what is examined rather than what is known. There, black box testing examines the functionality of an application without peering into its internal structures or workings, and white box testing tests those internal structures or workings, as opposed to the functionality.
The glossary lists clear box, glass box, transparent box and structural testing as other names for white box.
NIST SP 800-115, the Technical Guide to Information Security Testing and Assessment, dated September 2008, draws the line by technique. Its appendix on application security testing says white box techniques involve direct analysis of the application's source code, black box techniques are performed against the application's binary executable without source code knowledge, and many tests use both, a combination it calls grey box testing.
A footnote adds that many web applications have no compiled executable, so black box techniques may not be applicable to analysing their code.
The NCSC's penetration testing guidance sorts tests by how much information about your system the testers are armed with, in a list it calls illustrative rather than comprehensive. In transparent or open box testing, full information about the target is shared with the testers, and the NCSC says this confirms the efficacy of internal vulnerability assessment and management controls by identifying known software vulnerabilities and common misconfigurations.
In opaque or closed box testing, no information about the internals of the target is shared, and the test is performed from an external perspective.
The list names no middle category. The guidance says each type of test it describes can be run as either, including vulnerability identification in bespoke or niche software, which it says is most commonly used in web applications.
The OWASP Web Security Testing Guide (WSTG) v4.2 ties the labels to activities. Its introduction says penetration testing is also commonly known as black-box testing, describes it as testing a system or application remotely to find security vulnerabilities without knowing the inner workings of the target, and adds that the penetration test team is typically able to access an application as if they were users.
It pairs white-box testing with source code analysis, and describes grey box testing as testing in which the tester is assumed to have some partial knowledge about the application.
Its example: with some knowledge of how the application manages sessions, a tester can better understand whether log out and timeout are properly secured.
The guide's testing chapter says its own method is based on the black box approach, in which the tester knows nothing or has very little information about the application to be tested.
Where the definitions disagree, and what to ask instead
Set side by side, the definitions differ on four points that matter when you read a quote.
- What the label measures. The SP 800-53A wording in NIST's glossary, the NCSC and OWASP measure what the tester knows. The SP 800-192 wording measures what the test examines, and SP 800-115 measures which techniques are used. A provider can describe one engagement accurately by one measure and differently by another.
- Where grey box sits. NIST's glossary and OWASP describe partial knowledge. SP 800-115 describes a mix of white box and black box techniques. The NCSC's list names no middle category at all.
- Whether the tester signs in. None of the definitions settles it. The OWASP passage that calls penetration testing black-box testing also says the team is typically able to access an application as if they were users, and OWASP's own horizontal access test, WSTG-ATHZ-02, has the tester register or generate two users with identical privileges. On that reading, accounts and box colour are separate questions.
- What your own staff know. The boxes describe what the tester knows. Whether your administrators know a test is under way is a different matter. The NCSC notes that how much knowledge of the test is given to system administrators can vary within the same test regime, and SP 800-115 calls testing with the knowledge and consent of the IT staff overt, and testing without their knowledge but with the full knowledge and permission of upper management covert.
Three questions that pin a label down
Because the labels move, ask for the substance. Three answers settle what a quote means, whatever box it names.
- Which accounts will the tester hold, in which roles, and will they be accounts you create for the test or accounts the tester registers through public sign-up?
- What will the tester read: none of your documents, your API documentation and business rules, or your source code? For code, which areas?
- Which environment will be tested, production or a staging copy, and who has checked that the copy matches production in its settings as well as its code?
One booking application through three lenses
The walkthrough on this page takes one invented application and tests it three ways. book.lab.test is a booking application for a fictional mobile dog-grooming business, written for this page and not drawn from any client.
It has two roles. Customers register through open sign-up and request a groom at home, and each booking holds an address, a phone number, the dog's name and a note for the groomer, such as a side-gate code. Staff confirm bookings, waive deposits for regulars and export the customer list, and a nightly job sends that list to the business's mailing service.
In the black box column the tester has the production address and whatever public sign-up allows. In the grey box column the tester has two test accounts in each role, the developers' API documentation and the business's written booking rules, and works on a staging copy.
The white box column adds read-only access to the code and takes each code finding back to staging. Four flaws run through all three columns:
- The booking lookup checks the id but not the owner. All three columns find it, the black box column only because sign-up is open. The white box column traces it to one shared function and finds a further route that uses it.
- Staff routes check that someone is signed in, not that they are staff. The black box column finds one route by guessing its path. The grey box column confirms four from the staff screens. The white box column traces them all to the check applied to the whole staff route group, and finds a fifth route whose screen was removed but which still answers.
- A fallback key written in the repository opens the customer export the nightly job uses. Only the white box column finds it, because nothing public or documented mentions the route.
- Production shows full error details because of a setting staging does not share. Only the black box column finds it, because it is the only one pointed at production.
What the walkthrough shows
No column finds all four flaws, and the column with the least information is the only one to find the fourth. The point is not that one approach wins. Each approach answers a narrower question than its label suggests, and a scope can combine them, for example signed-out checks on production alongside signed-in or code-assisted work on staging.
What each approach asks of you, and where its coverage stops
Each step from black box towards white box asks more of your team before testing starts and gives the tester a wider view.
None of the three is the right choice for every engagement, and each has a blind spot that another one covers.
A black box test of production asks the least of you: a signed authorisation that names the production hostnames, the testing window and the production risks you accept, a check of your host's testing policy, a decision on whether the tester may register accounts through public sign-up, and a technical contact.
In return it answers a question no staging test can, which is what a stranger can do to the system your customers use, on the dates it was tested.
Its reach ends at the login for every role you issue yourself, at routes that nothing public reveals, and at business rules nobody has told the tester. ASVS 5.0 is direct about the limit: even its Level 1 is not necessarily penetration testable by an external tester without internal access to documentation or code, and it gives black box testing as the example.
A grey box test asks for test accounts, sample records, the rules your application is meant to enforce and whatever documentation you keep. Two accounts in each role is what the access tests in WSTG v4.2 are built around: its horizontal test has the tester register or generate two users with identical privileges, and its insecure direct object reference test says the best way to test is with at least two users who own different objects.
With those in place, the test can show what each role can do to every other account's records, which is where the private data of an application with customer accounts sits.
It is only as good as what you supply. Documentation that has fallen behind the code hides routes, and a staging copy that differs from production in its settings hides problems that only production has. SP 800-115 warns that there are usually inconsistencies between test and production environments, which can result in missed vulnerabilities when non-production systems are used.
A white box test asks for read-only access to the repository at the commit deployed on the test environment, the names of the settings each environment uses, and a developer who can answer questions. In return it can see what requests alone cannot: routes no screen or document mentions, such as scheduled jobs and callbacks from other services, the root cause behind a finding, and every other place the same mistake was made.
SP 800-115 says white box techniques still tend to be more efficient and cost-effective for finding security defects in custom applications than black box techniques.
OWASP's testing principles advise giving the source code, where it is available, to the security staff, noting that vulnerabilities can be discovered in the source that a black-box engagement would miss.
Code has blind spots of its own. SP 800-115 says white box techniques cannot detect security defects in interfaces between components, nor identify problems caused during compilation, linking or installation-time configuration, and OWASP lists among the drawbacks of source code review that the code actually deployed might differ from the code analysed.
A white box finding is strongest once requests to a running build confirm it, and settings that live outside the repository need a running system to show them.
ASVS 5.0, which calls documentation or source code-led testing hybrid, strongly encourages it and says it will be necessary to verify many ASVS requirements.
Choosing an approach for your application
Start from what sits behind your login and what you can share, not from the label.
Whatever you choose, the report should record the accounts and roles used, the environment, the build and any code read, so a reader can see what the test covered without relying on the box it names. These questions usually settle the choice:
- Does anyone other than your own editors sign in? If not, most of what you expose can be seen from outside, and a signed-out test covers it. The website security check, which stays outside the login, is built for that case.
- Do customers hold accounts and records of their own, or do staff have functions customers must not reach? Then the access rules that protect those records sit behind the login. A black box test reaches them only if sign-up is open, and reaches roles you issue yourself only by chance, so test accounts in every role belong in the scope.
- Does the application have routes no screen calls, such as scheduled jobs, exports or callbacks from payment and messaging services, or logic concentrated in a few files your developers can name? Code access reaches those directly.
- Was an earlier finding fixed in one place? Reading the code shows whether the same mistake sits elsewhere, which requests alone can only establish route by route.
- Is the question what an outsider can do to production today? Then keep the production hostnames in scope for at least the signed-out checks, with the production risks recorded in the authorisation, because a staging copy answers a slightly different question.
- Does a tender, panel or contract require a CREST-accredited provider? Clever Ops is not CREST accredited, so we refer that work to a provider that is.
How Clever Ops describes the same choices
Clever Ops scopes web application tests by depth rather than by box: unauthenticated, authenticated and code-assisted.
Against the definitions on this page, unauthenticated testing sits closest to black box, authenticated testing with test accounts to grey box, and code-assisted testing moves towards white box without reaching it, because code access is targeted at the areas the scope names rather than a line-by-line read of the whole repository.
A full review of the code is a separate piece of work, the secure code review. The methodology guide explains what each depth reaches.
A web application and API penetration test is authenticated by default, with two test accounts in each role in scope. When code access is added, it is read-only, a weakness seen in the code is confirmed against a running build where the scope allows, and a finding that rests on the code alone says so.
Testing starts only after the owner of each target signs an authorisation and your hosting provider's testing policy has been checked. The authorisation lists the source IP addresses test traffic will come from, so your team and your host can tell it apart from an attack.
Destructive actions are off by default for real people's accounts and records that existed before the test, and reset, lockout and deletion checks run only on test accounts and test records where the rules of engagement allow them.
If Clever Ops built or hosts the application, the same work runs as a security review rather than an independent test, and the report says who built it and who hosts it.
Frequently asked questions
A penetration test in which the tester starts with little or no knowledge of the application's internals and learns only what the running application reveals. NIST's glossary, using the SP 800-53A Rev. 5 wording, calls black box testing a test methodology that assumes no knowledge of the internal structure and implementation detail of the assessment object. The OWASP testing guide describes its own method as based on the black box approach, where the tester knows nothing or has very little information about the application.
Testing in which the tester has explicit and substantial knowledge of the application's internals, in the definition NIST's glossary takes from SP 800-137. SP 800-115 ties white box techniques to direct analysis of the source code and says they tend to be more efficient and cost-effective for finding defects in custom applications, but cannot detect problems in interfaces between components or in installation-time configuration. Code findings are strongest when they are confirmed against a running build.
Testing with part of the picture. NIST's glossary, quoting CNSSI 4009-2022, defines it as assuming some knowledge of the internal structure and implementation detail, and OWASP describes the tester as having some partial knowledge about the application. SP 800-115 instead uses the term for tests that combine white box and black box techniques. NIST and OWASP spell it the US way, with an a. Because the definitions differ, ask what the tester will be given: test accounts, documentation, written business rules or some of the code.
Neither in general, because they find different things. OWASP lists among the strengths of penetration testing that it tests the code actually exposed, and among the drawbacks of source code review that the code deployed might differ from the code analysed. In the booking walkthrough on this page, only the white box column found the fallback key in the repository, and only the black box column found the error setting that existed on production alone. Choose by the risks in your application, or combine the approaches.
Its starting point is more realistic for an outsider who knows nothing about your application. The NCSC says testing with no information about the internals more accurately models attackers who are unknown or unaffiliated to the organisation. A test has a fixed window, though, and the same guidance warns that the lack of information can leave vulnerabilities undiscovered in the time allocated. An outsider who registers through your public sign-up also ends up holding a customer account, which is the grey box tester's starting point for that role.
No. Without it, a test still covers what the running application does for each role the tester holds accounts in. Code access adds the routes and root causes that requests alone cannot show, and OWASP's testing principles advise giving source code to the security staff where it is available. If you share it, read-only access to the areas in scope is enough, you can revoke the access once the retest is complete, and code excerpts in a report are handled as evidence like any other, with secrets redacted.
No. The tester works from test accounts created for the test, two in each role, each with sample records, so nobody's real login is shared. Lockout and reset checks run only on those test accounts, where the rules of engagement allow them. Keep the accounts until the retest is complete, so the same requests can be replayed, and remove them afterwards.
Scope the right test for your application
Tell us what you have built, who uses it and when it needs to launch. You will get a plain-English recommendation: a website audit, a web application test or a referral to a specialist.