Exposed API keys: where secrets leak, how to check your own site, and what to fix first
API keys and other secrets leak into source files, git history, the JavaScript every visitor downloads, log files, and environment files or .git folders a web server hands out.
This guide shows how to confirm a leak on your own property, why the old key is revoked or rotated before anything is cleaned, and how to prove it no longer works. It comes with a fixture repository you can run on your own computer, taking four planted canary keys from detection through to proof.
In this guide
Canary fixture repository and leak response sequence
An illustration you can run on your own computer: a local git repository with four canary keys planted in current source, history, a client bundle and a log, a local key service that can revoke and replace them, and one script for each step from detection to proof, with a regression test. Every output below was copied from one recorded run of these files on Node.js 26.0.0, git 2.53.0 and git filter-repo 2.47.0.
How to use itHide the instructions
The canaries use a format invented for this page.
secrets-exposure.txt
Introduction40 lines
Part 1: The lab files775 lines
Part 2: The recorded run266 lines
Part 3: The same sequence on your own systems31 lines
Whole template1,130 lines
CANARY FIXTURE REPOSITORY AND LEAK RESPONSE SEQUENCE
ILLUSTRATION ONLY. The shop app, its keys, the key service and every result below were written for this page and run on one computer. Nothing here is a client's code or a client's result. Every API key in the lab is either a canary in a format invented for this lab, "lab-canary-" followed by a label, a hyphen and 16 hexadecimal characters, or a key the lab's own key service makes up while it runs, "lab-key-" followed by a name, a version and 32 hexadecimal characters. No provider issues keys in either format. The key service's admin token is random, made when the service starts and never printed. Run the lab on your own computer, and use what it teaches only on systems you own or are authorised in writing to test.
WHAT IT NEEDS
Node.js, git and git filter-repo. The scripts import only node:assert/strict, node:child_process, node:crypto, node:fs, node:fs/promises, node:http, node:os, node:path, node:test, node:timers/promises and node:url, and call the global fetch() function, so the lab installs nothing from npm. git filter-repo is a separate tool you install yourself; GitHub's documentation gives Homebrew as one way. The run below was recorded on macOS with Node.js 26.0.0, git 2.53.0 and git filter-repo 2.47.0, the only versions and the only system it was run on. GitHub's documentation says the --sensitive-data-removal option needs git filter-repo 2.47 or later. Timings, timestamps, process ids and the new keys differ from run to run. The commit ids did not: three complete runs on this computer produced the same ones.
WHAT IT DOES NOT DO
Every server in the lab listens on 127.0.0.1, on a port the system picks, and no port is printed. Nothing calls a real provider, and no real key is created, used or revoked. The fixture is a new git repository that make-fixture.mjs creates inside the lab folder. Every git command names it with -C, and the script stops if git resolves to any other repository. The scripts set an invented author, Fixture Developer at fixture.invalid, and point GIT_CONFIG_GLOBAL and GIT_CONFIG_SYSTEM at /dev/null, so the fixture's commits use neither your git settings nor your name and email address. The search in search.mjs looks for this lab's canary format and for exact values it is given. It is not a general secret scanner, and it would pass straight over a real key in any other format.
THE FOUR CANARIES
name value planted in
orders lab-canary-orders-3f9a61c0d27e4b85 current source: config/credentials.json
mailer lab-canary-mailer-8c2d04e7b61f3a95 history only: .env, committed then deleted
search lab-canary-search-51e7c9a0f2d84b36 client bundle: dist/client.js
payments lab-canary-payments-a04f7d2c9e81b653 application log: logs/app.log
The key service starts with all four active, the way a provider still accepts a key that nobody revoked.
THE SEQUENCE, AND THE SOURCE BEHIND EACH STEP
Set-up
Scripts: node service.mjs start, then node make-fixture.mjs. They build the leaky repository, its bundle and its log.
Source: git's documentation of GIT_AUTHOR_NAME, GIT_AUTHOR_DATE, GIT_COMMITTER_DATE and GIT_CONFIG_GLOBAL, for the fixed author, dates and settings.
Step 1, detect
Script: detect.mjs. It searches the working tree; every commit (git rev-list --all, then git grep over each commit's tree); every object git holds, reachable or not (git cat-file --batch-all-objects); the bundle; and the log. Then it requests five paths from a server whose web root is the project folder.
Sources: git's grep, rev-list and cat-file documentation; the latest edition of OWASP WSTG-CONF-04 for .env and .git/.
Step 2, contain
Script: contain.mjs. It revokes every key the search found, before anything is cleaned.
Sources: GitHub, Removing sensitive data from a repository; OWASP Secrets Management Cheat Sheet, section 9.2.
Step 3, rotate
Script: rotate.mjs. It issues replacements into the host environment, deletes the key file, changes the app to read every key from the environment, and rebuilds the bundle without the secret.
Sources: Supabase, Rotate a leaked or compromised API key; GitHub's advice to use environment variables or a secret management service.
Step 4, clean up
Script: clean.mjs. It removes .env and config/credentials.json from every commit with git filter-repo, then redacts the revoked keys in the log and keeps every line.
Sources: GitHub's procedure and git filter-repo's documentation; OWASP Secrets Management Cheat Sheet, section 9.2, on removing secrets from logs while keeping log integrity.
Step 5, prove
Script: prove.mjs. Old keys refused, new keys accepted, refused attempts recorded, the first changed commit gone, every search empty, sensitive paths not served.
Sources: git filter-repo's documentation for the git cat-file -t check; OWASP Secrets Management Cheat Sheet, section 8.3, on logging attempts to use a revoked secret.
Regression test
Script: leak.test.mjs. It fails on the leaky fixture and passes on the cleaned one.
Source: Node.js test runner documentation, which says a failing test sets the exit code to 1.
================================================================
PART 1: THE LAB FILES
================================================================
Make an empty folder named secrets-lab with a folder named fixture-files inside it. Save the fifteen files below under the names shown, then run the commands in Part 2 from the secrets-lab folder. The three files in fixture-files are the fixture's own application, which make-fixture.mjs and rotate.mjs copy into the fixture.
================================================================
FILE 1 OF 15: lab-common.mjs
================================================================
// lab-common.mjs: settings and helpers shared by every script in the lab.
// Illustration only. Every key in the lab is either an invented canary or a
// key issued by the lab's own key service, and nothing leaves 127.0.0.1.
import { execFileSync } from 'node:child_process'
import { mkdirSync, readFileSync, writeFileSync } from 'node:fs'
import { dirname, join } from 'node:path'
import { fileURLToPath } from 'node:url'
export const LAB_DIR = fileURLToPath(new URL('.', import.meta.url))
export const FIXTURE = join(LAB_DIR, 'fixture')
export const STATE_DIR = join(LAB_DIR, 'lab-state')
// The four canaries. The format is invented for this lab: "lab-canary-",
// a label, a hyphen and 16 hexadecimal characters. No provider issues keys
// in this format. Each canary is planted in exactly one place.
export const CANARIES = [
{ name: 'orders', plantedIn: 'current source: config/credentials.json', value: 'lab-canary-orders-3f9a61c0d27e4b85' },
{ name: 'mailer', plantedIn: 'history only: .env, committed then deleted', value: 'lab-canary-mailer-8c2d04e7b61f3a95' },
{ name: 'search', plantedIn: 'client bundle: dist/client.js', value: 'lab-canary-search-51e7c9a0f2d84b36' },
{ name: 'payments', plantedIn: 'application log: logs/app.log', value: 'lab-canary-payments-a04f7d2c9e81b653' },
]
// The search pattern. It matches this lab's canary format and nothing else.
// It is not a general secret scanner: real keys in other formats pass it by.
export const CANARY_PATTERN = 'lab-canary-[a-z]+-[0-9a-f]{16}'
export function findCanaries(text) {
return text.match(new RegExp(CANARY_PATTERN, 'g')) || []
}
// Shows a key by its label and last four characters, never in full.
export function mask(value) {
return value.slice(0, value.lastIndexOf('-') + 1) + '...' + value.slice(-4)
}
// git writes a newline character at the end of every line.
const NEWLINE = String.fromCharCode(10)
export function lines(text) {
return text.split(NEWLINE).filter((line) => line.length > 0)
}
// A fixed author identity and no global or system git settings, so the
// commit ids do not depend on who runs the lab or how their git is set up.
export const GIT_ENV = {
GIT_AUTHOR_NAME: 'Fixture Developer',
GIT_AUTHOR_EMAIL: 'developer@fixture.invalid',
GIT_COMMITTER_NAME: 'Fixture Developer',
GIT_COMMITTER_EMAIL: 'developer@fixture.invalid',
GIT_CONFIG_GLOBAL: '/dev/null',
GIT_CONFIG_SYSTEM: '/dev/null',
}
// Runs git against the fixture only: -C names the fixture on every call.
export function git(args, extraEnv) {
const env = Object.assign({}, process.env, GIT_ENV, extraEnv || {})
return execFileSync('git', ['-C', FIXTURE].concat(args), { env: env, encoding: 'utf8', stdio: ['ignore', 'pipe', 'pipe'] })
}
export function writeFixtureFile(path, text) {
mkdirSync(dirname(join(FIXTURE, path)), { recursive: true })
writeFileSync(join(FIXTURE, path), text)
}
export function copyIntoFixture(labFile, fixturePath) {
writeFixtureFile(fixturePath, readFileSync(join(LAB_DIR, 'fixture-files', labFile), 'utf8'))
}
// Commits everything with a fixed date, then prints the new commit.
export function commit(message, date) {
git(['add', '--all'])
git(['commit', '--quiet', '-m', message], { GIT_AUTHOR_DATE: date, GIT_COMMITTER_DATE: date })
console.log('commit ' + git(['log', '-1', '--format=%h %ad %s', '--date=short']).trim())
}
// The host environment: lab-state/host-env.json stands in for the settings
// screen or secrets store of your hosting platform. It is outside the
// fixture, so it is never committed, built or served.
export function readHostEnv() {
return JSON.parse(readFileSync(join(STATE_DIR, 'host-env.json'), 'utf8'))
}
export function writeHostEnv(values) {
mkdirSync(STATE_DIR, { recursive: true })
writeFileSync(join(STATE_DIR, 'host-env.json'), JSON.stringify(values, null, 2), { mode: 0o600 })
}
function readService() {
return JSON.parse(readFileSync(join(STATE_DIR, 'service.json'), 'utf8'))
}
// Runs one of the fixture's own scripts inside the fixture, with only the
// environment it is given plus PATH and the key service address.
export function runInFixture(script, env) {
const serviceUrl = 'http://127.0.0.1:' + readService().port
const fullEnv = Object.assign({ PATH: process.env.PATH, KEY_SERVICE_URL: serviceUrl }, env)
execFileSync(process.execPath, [script], { cwd: FIXTURE, env: fullEnv, stdio: 'inherit' })
}
// Calls the lab's key service. Admin calls carry the token the service
// wrote to lab-state/service.json when it started.
export async function callService(path, options) {
const settings = options || {}
const service = readService()
const headers = Object.assign({ 'content-type': 'application/json' }, settings.headers || {})
if (settings.admin) headers['x-admin-token'] = service.adminToken
const response = await fetch('http://127.0.0.1:' + service.port + path, {
method: settings.method || 'POST',
headers: headers,
body: settings.body ? JSON.stringify(settings.body) : undefined,
})
return { status: response.status, body: await response.json() }
}
================================================================
FILE 2 OF 15: key-service.mjs
================================================================
// key-service.mjs: a stand-in for an API provider's key service, for this
// lab only. It listens on 127.0.0.1 on a port the system picks and keeps
// its keys in memory, stored by SHA-256 digest. It records each event by key
// name, never by key value. It starts with the four canaries active, the
// way a provider still holds a key that was issued and never revoked.
import { createServer } from 'node:http'
import { createHash, randomBytes } from 'node:crypto'
import { mkdirSync, writeFileSync } from 'node:fs'
import { join } from 'node:path'
import { CANARIES, STATE_DIR } from './lab-common.mjs'
const keys = new Map()
const events = []
const adminToken = randomBytes(16).toString('hex')
function digest(value) {
return createHash('sha256').update(String(value)).digest('hex')
}
function record(text) {
events.push(new Date().toISOString() + ' ' + text)
}
function addKey(name, value) {
keys.set(digest(value), { name: name, status: 'active' })
}
for (const canary of CANARIES) addKey(canary.name + '-v1', canary.value)
// POST /use: what an ordinary API call does with the key it presents.
function useKey(request) {
const key = keys.get(digest(request.headers['x-lab-key']))
if (!key) {
record('refused an unknown key')
return [401, { result: 'refused', reason: 'unknown key' }]
}
if (key.status !== 'active') {
record('refused revoked key ' + key.name)
return [401, { result: 'refused', reason: 'revoked' }]
}
record('accepted ' + key.name)
return [200, { result: 'accepted', key: key.name }]
}
function revokeKey(request, body) {
const key = keys.get(digest(body.key))
if (!key) return [404, { error: 'no such key' }]
key.status = 'revoked'
record('revoked ' + key.name)
return [200, { revoked: key.name }]
}
// Issues a replacement named after the next free version, such as orders-v2.
function issueKey(request, body) {
let version = 1
for (const key of keys.values()) {
if (key.name.startsWith(body.name + '-v')) version = version + 1
}
const name = body.name + '-v' + version
const value = 'lab-key-' + name + '-' + randomBytes(16).toString('hex')
addKey(name, value)
record('issued ' + name)
return [200, { name: name, key: value }]
}
function listEvents() {
return [200, { events: events }]
}
const ROUTES = {
'POST /use': useKey,
'POST /admin/revoke': revokeKey,
'POST /admin/issue': issueKey,
'GET /admin/events': listEvents,
}
function answer(request, text) {
const route = ROUTES[request.method + ' ' + request.url]
if (!route) return [404, { error: 'not found' }]
if (request.url.startsWith('/admin/') && request.headers['x-admin-token'] !== adminToken) {
return [403, { error: 'admin token needed' }]
}
try {
return route(request, text ? JSON.parse(text) : {})
} catch (error) {
return [400, { error: 'bad request' }]
}
}
const server = createServer((request, response) => {
let text = ''
request.on('data', (chunk) => { text = text + chunk })
request.on('end', () => {
const result = answer(request, text)
response.writeHead(result[0], { 'content-type': 'application/json' })
response.end(JSON.stringify(result[1]))
})
})
server.listen(0, '127.0.0.1', () => {
mkdirSync(STATE_DIR, { recursive: true })
const state = { port: server.address().port, pid: process.pid, adminToken: adminToken }
writeFileSync(join(STATE_DIR, 'service.json'), JSON.stringify(state), { mode: 0o600 })
console.log('key service ready on 127.0.0.1, pid ' + process.pid)
})
process.on('SIGTERM', () => {
server.closeAllConnections()
server.close(() => process.exit(0))
})
================================================================
FILE 3 OF 15: service.mjs
================================================================
// service.mjs: starts the key service in the background, or stops it by the
// process id it recorded. Usage: node service.mjs start, node service.mjs stop
import { spawn } from 'node:child_process'
import { existsSync, mkdirSync, openSync, readFileSync, rmSync } from 'node:fs'
import { join } from 'node:path'
import { setTimeout as wait } from 'node:timers/promises'
import { LAB_DIR, STATE_DIR } from './lab-common.mjs'
const stateFile = join(STATE_DIR, 'service.json')
async function start() {
mkdirSync(STATE_DIR, { recursive: true })
rmSync(stateFile, { force: true })
const log = openSync(join(STATE_DIR, 'service.log'), 'a')
const child = spawn(process.execPath, [join(LAB_DIR, 'key-service.mjs')], { detached: true, stdio: ['ignore', log, log] })
child.unref()
for (let attempt = 0; attempt < 50 && !existsSync(stateFile); attempt++) await wait(100)
const state = JSON.parse(readFileSync(stateFile, 'utf8'))
console.log('key service started on 127.0.0.1 (port chosen by the system), pid ' + state.pid)
}
function stop() {
const state = JSON.parse(readFileSync(stateFile, 'utf8'))
process.kill(state.pid, 'SIGTERM')
rmSync(stateFile)
console.log('key service stopped, pid ' + state.pid)
}
if (process.argv[2] === 'start') await start()
else if (process.argv[2] === 'stop') stop()
else console.log('Usage: node service.mjs start, or node service.mjs stop')
================================================================
FILE 4 OF 15: fixture-files/app-leaky.mjs
================================================================
// src/app.mjs, leaky version: sends one orders request and one test payment
// through the lab's key service, and writes a log. Each leak is marked LEAK.
import { appendFileSync, mkdirSync, readFileSync } from 'node:fs'
import { EOL } from 'node:os'
// LEAK: the orders key is read from a file committed to the repository.
const credentials = JSON.parse(readFileSync('config/credentials.json', 'utf8'))
const serviceUrl = process.env.KEY_SERVICE_URL
function writeLog(level, message) {
mkdirSync('logs', { recursive: true })
appendFileSync('logs/app.log', new Date().toISOString() + ' ' + level + ' ' + message + EOL)
}
async function callApi(label, key, debug) {
const request = { method: 'POST', headers: { 'x-lab-key': key } }
// LEAK: the debug line copies the whole request, key included, into the log.
if (debug) writeLog('DEBUG', label + ' request ' + JSON.stringify(request))
const response = await fetch(serviceUrl + '/use', request)
writeLog('INFO', label + ' answered ' + response.status)
console.log('app: ' + label + ' HTTP ' + response.status)
}
await callApi('orders', credentials.ordersKey, false)
await callApi('payments', process.env.PAYMENTS_KEY, true)
================================================================
FILE 5 OF 15: fixture-files/app-fixed.mjs
================================================================
// src/app.mjs, fixed version: every key comes from the environment, and the
// log records which key was used by a short SHA-256 hash, never the key.
import { createHash } from 'node:crypto'
import { appendFileSync, mkdirSync } from 'node:fs'
import { EOL } from 'node:os'
const serviceUrl = process.env.KEY_SERVICE_URL
function requireKey(name) {
const value = process.env[name]
if (!value) throw new Error(name + ' is not set in the environment')
return value
}
function keyHash(key) {
return createHash('sha256').update(key).digest('hex').slice(0, 12)
}
function writeLog(level, message) {
mkdirSync('logs', { recursive: true })
appendFileSync('logs/app.log', new Date().toISOString() + ' ' + level + ' ' + message + EOL)
}
async function callApi(label, key) {
writeLog('DEBUG', label + ' request with key sha256:' + keyHash(key))
const response = await fetch(serviceUrl + '/use', { method: 'POST', headers: { 'x-lab-key': key } })
writeLog('INFO', label + ' answered ' + response.status)
console.log('app: ' + label + ' HTTP ' + response.status)
}
await callApi('orders', requireKey('ORDERS_KEY'))
await callApi('payments', requireKey('PAYMENTS_KEY'))
================================================================
FILE 6 OF 15: fixture-files/build.mjs
================================================================
// src/build.mjs: a stand-in for a front-end build. Like NEXT_PUBLIC_ in
// Next.js and VITE_ in Vite, the PUBLIC_ prefix tells it to copy a value
// into the JavaScript file that every visitor's browser downloads.
import { mkdirSync, writeFileSync } from 'node:fs'
import { EOL } from 'node:os'
const settings = {}
for (const name of Object.keys(process.env).sort()) {
if (name.startsWith('PUBLIC_')) settings[name] = process.env[name]
}
mkdirSync('dist', { recursive: true })
const bundle = 'window.SHOP_SETTINGS = ' + JSON.stringify(settings) + ';' + EOL + 'console.log("shop loaded");' + EOL
writeFileSync('dist/client.js', bundle)
console.log('build: dist/client.js written with ' + Object.keys(settings).length + ' public setting(s)')
================================================================
FILE 7 OF 15: make-fixture.mjs
================================================================
// make-fixture.mjs: builds the leaky fixture repository in ./fixture, then
// runs its build and its app once, so each canary lands in its one place.
// Start the key service first: node service.mjs start
import { mkdirSync, realpathSync, rmSync } from 'node:fs'
import { EOL } from 'node:os'
import { join } from 'node:path'
import { CANARIES, FIXTURE, commit, copyIntoFixture, git, runInFixture, writeFixtureFile, writeHostEnv } from './lab-common.mjs'
const canary = {}
for (const item of CANARIES) canary[item.name] = item.value
// A new, separate repository. Stop if git resolved to any other one.
rmSync(FIXTURE, { recursive: true, force: true })
mkdirSync(FIXTURE, { recursive: true })
git(['init', '--quiet', '--initial-branch=main'])
const topLevel = realpathSync(git(['rev-parse', '--show-toplevel']).trim())
if (topLevel !== realpathSync(FIXTURE)) throw new Error('the fixture is not its own repository')
console.log('fixture repository created, git directory: ' + git(['rev-parse', '--git-dir']).trim())
// Commit 1: the app, with the orders key written into a configuration file.
writeFixtureFile('README.md', 'Fixture shop app for the secrets exposure lab. Illustration only.' + EOL)
writeFixtureFile('.gitignore', 'dist/' + EOL + 'logs/' + EOL)
writeFixtureFile('config/credentials.json', JSON.stringify({ ordersKey: canary.orders }, null, 2) + EOL)
copyIntoFixture('app-leaky.mjs', 'src/app.mjs')
copyIntoFixture('build.mjs', 'src/build.mjs')
commit('Start the shop app', '2026-10-01T09:00:00+10:00')
// Commit 2: an environment file committed by mistake.
writeFixtureFile('.env', 'MAILER_KEY=' + canary.mailer + EOL)
commit('Add mailer settings', '2026-10-08T09:00:00+11:00')
// Commit 3: the file deleted and ignored. Its key is still in commit 2.
rmSync(join(FIXTURE, '.env'))
writeFixtureFile('.gitignore', 'dist/' + EOL + 'logs/' + EOL + '.env' + EOL)
commit('Remove .env from the repository', '2026-10-15T09:00:00+11:00')
// The build: a secret key given the public prefix goes into the bundle.
runInFixture('src/build.mjs', { PUBLIC_SEARCH_KEY: canary.search })
// The host keeps the payments key outside the code, and the app logs it.
writeHostEnv({ PAYMENTS_KEY: canary.payments })
runInFixture('src/app.mjs', { PAYMENTS_KEY: canary.payments })
================================================================
FILE 8 OF 15: search.mjs
================================================================
// search.mjs: looks for this lab's canary pattern, plus any exact values it
// is given, in five places in the fixture. It finds this lab's canary format
// only. It is not a general secret scanner.
import { existsSync, readdirSync, readFileSync, statSync } from 'node:fs'
import { join, relative } from 'node:path'
import { CANARY_PATTERN, FIXTURE, findCanaries, git, lines, mask } from './lab-common.mjs'
export const AREA_LABELS = {
tree: 'working tree',
history: 'history',
objects: 'object store',
bundle: 'client bundle',
logs: 'logs',
}
function listFiles(dir, skipNames) {
const files = []
for (const name of readdirSync(dir)) {
if (skipNames.includes(name)) continue
const path = join(dir, name)
if (statSync(path).isDirectory()) files.push(...listFiles(path, []))
else files.push(path)
}
return files
}
function matchesIn(text, exactValues) {
const found = findCanaries(text)
for (const value of exactValues) {
if (text.includes(value)) found.push(value)
}
return found
}
function searchFiles(dir, skipNames, exactValues) {
const hits = []
if (!existsSync(dir)) return hits
for (const file of listFiles(dir, skipNames)) {
for (const value of matchesIn(readFileSync(file, 'utf8'), exactValues)) {
hits.push({ where: relative(FIXTURE, file), value: value })
}
}
return hits
}
// git grep exits with status 1 when nothing matches, which is not an error here.
function gitGrep(args) {
try {
return git(['grep', '-I', '-o'].concat(args))
} catch (error) {
if (error.status === 1) return ''
throw error
}
}
// Every commit reachable from any ref (branches, tags and the rest), each
// searched as a whole tree, not only as the lines it changed.
function searchHistory(exactValues) {
const commits = lines(git(['rev-list', '--all']))
if (commits.length === 0) return []
const searches = [['-E', '-e', CANARY_PATTERN]]
for (const value of exactValues) searches.push(['-F', '-e', value])
const hits = []
for (const search of searches) {
for (const line of lines(gitGrep(search.concat(commits)))) {
const parts = line.split(':')
hits.push({ where: parts[0].slice(0, 7) + ' ' + parts[1], value: parts.slice(2).join(':') })
}
}
return hits
}
// Every object git holds, reachable or not, so leftovers of a rewrite count.
function searchObjects(exactValues) {
const hits = []
for (const line of lines(git(['cat-file', '--batch-all-objects', '--batch-check']))) {
const parts = line.split(' ')
if (parts[1] !== 'blob') continue
for (const value of matchesIn(git(['cat-file', 'blob', parts[0]]), exactValues)) {
hits.push({ where: 'blob ' + parts[0].slice(0, 7), value: value })
}
}
return hits
}
export function searchFixture(exactValues) {
const values = exactValues || []
return {
tree: searchFiles(FIXTURE, ['.git', 'dist', 'logs'], values),
history: searchHistory(values),
objects: searchObjects(values),
bundle: searchFiles(join(FIXTURE, 'dist'), [], values),
logs: searchFiles(join(FIXTURE, 'logs'), [], values),
}
}
export function printSearch(found) {
for (const area of Object.keys(AREA_LABELS)) {
console.log(AREA_LABELS[area].padEnd(14) + found[area].length + ' match(es)')
for (const hit of found[area]) console.log(' ' + hit.where.padEnd(32) + mask(hit.value))
}
}
================================================================
FILE 9 OF 15: serve-check.mjs
================================================================
// serve-check.mjs: starts a plain static file server on 127.0.0.1, with the
// web root it is given, and requests five paths from it. It shows what a
// server hands out when its web root is the whole project folder and when it
// is only the build output. It never lists a directory.
import { createServer } from 'node:http'
import { readFile } from 'node:fs/promises'
import { join, normalize, sep } from 'node:path'
import { findCanaries, lines, mask } from './lab-common.mjs'
export const CHECKED_PATHS = ['/.env', '/.git/HEAD', '/.git/config', '/config/credentials.json', '/client.js']
function startStaticServer(webRoot) {
const server = createServer(async (request, response) => {
const path = decodeURIComponent(new URL(request.url, 'http://lab.invalid').pathname)
const target = normalize(join(webRoot, path))
try {
if (!target.startsWith(webRoot + sep)) throw new Error('outside the web root')
const body = await readFile(target)
response.writeHead(200, { 'content-type': 'text/plain' })
response.end(body)
} catch (error) {
response.writeHead(404, { 'content-type': 'text/plain' })
response.end('Not found')
}
})
return new Promise((resolve) => server.listen(0, '127.0.0.1', () => resolve(server)))
}
export async function checkWebRoot(webRoot) {
const server = await startStaticServer(webRoot)
const results = []
for (const path of CHECKED_PATHS) {
const response = await fetch('http://127.0.0.1:' + server.address().port + path)
const body = await response.text()
const firstLine = lines(body)[0] || ''
results.push({ path: path, status: response.status, firstLine: firstLine, canaries: findCanaries(body) })
}
server.close()
return results
}
export function printWebRootCheck(label, results) {
console.log('web root: ' + label)
for (const result of results) {
let shown = result.firstLine
for (const value of findCanaries(shown)) shown = shown.split(value).join(mask(value))
shown = shown.slice(0, 40)
console.log(' GET ' + result.path.padEnd(26) + result.status + ' canaries in body: ' + result.canaries.length + ' first line: ' + shown)
}
}
================================================================
FILE 10 OF 15: detect.mjs
================================================================
// detect.mjs, step 1: where are the canaries? Searches the five places, then
// asks what a web server would hand out if its web root were the project
// folder. It changes nothing.
import { CANARY_PATTERN, FIXTURE } from './lab-common.mjs'
import { printSearch, searchFixture } from './search.mjs'
import { checkWebRoot, printWebRootCheck } from './serve-check.mjs'
console.log('pattern searched for: ' + CANARY_PATTERN)
const found = searchFixture()
printSearch(found)
const distinct = new Set()
for (const area of Object.keys(found)) {
for (const hit of found[area]) distinct.add(hit.value)
}
console.log('distinct canaries found: ' + distinct.size)
console.log('')
printWebRootCheck('the whole project folder (misconfigured)', await checkWebRoot(FIXTURE))
================================================================
FILE 11 OF 15: contain.mjs
================================================================
// contain.mjs, step 2: revoke every key the search found, at the key
// service, before anything is cleaned up. Then run the deployed app on its
// old keys to show the cost: it stops working until step 3 replaces them.
import { callService, mask, readHostEnv, runInFixture } from './lab-common.mjs'
import { searchFixture } from './search.mjs'
const found = searchFixture()
const values = new Set()
for (const area of Object.keys(found)) {
for (const hit of found[area]) values.add(hit.value)
}
for (const value of values) {
const result = await callService('/admin/revoke', { admin: true, body: { key: value } })
console.log('revoke ' + mask(value).padEnd(32) + 'HTTP ' + result.status + ' ' + JSON.stringify(result.body))
}
console.log('')
console.log('the deployed app, still on its old keys:')
runInFixture('src/app.mjs', readHostEnv())
================================================================
FILE 12 OF 15: rotate.mjs
================================================================
// rotate.mjs, step 3: issue a replacement for each revoked key into the host
// environment, change the app to read every key from the environment, rebuild
// the bundle without the secret, and run the app on the new keys. New key
// values are never printed.
import { rmSync } from 'node:fs'
import { join } from 'node:path'
import { FIXTURE, callService, commit, copyIntoFixture, runInFixture, writeHostEnv } from './lab-common.mjs'
const VARIABLES = { orders: 'ORDERS_KEY', mailer: 'MAILER_KEY', search: 'SEARCH_KEY', payments: 'PAYMENTS_KEY' }
const hostEnv = {}
for (const name of Object.keys(VARIABLES)) {
const result = await callService('/admin/issue', { admin: true, body: { name: name } })
hostEnv[VARIABLES[name]] = result.body.key
console.log('issued ' + result.body.name.padEnd(12) + 'stored as ' + VARIABLES[name] + ' in the host environment')
}
writeHostEnv(hostEnv)
// The code change: no key file, every key read from the environment.
rmSync(join(FIXTURE, 'config', 'credentials.json'))
copyIntoFixture('app-fixed.mjs', 'src/app.mjs')
commit('Read keys from the environment', '2026-11-14T10:00:00+11:00')
// The search key now stays on the server, so nothing carries the PUBLIC_ prefix.
runInFixture('src/build.mjs', {})
runInFixture('src/app.mjs', hostEnv)
================================================================
FILE 13 OF 15: clean.mjs
================================================================
// clean.mjs, step 4: remove the two files that held keys from every commit
// with git filter-repo, then redact the revoked keys from the log without
// dropping a line. It runs after step 2 revoked the keys, never instead.
import { spawnSync } from 'node:child_process'
import { readFileSync, writeFileSync } from 'node:fs'
import { join } from 'node:path'
import { CANARY_PATTERN, FIXTURE, GIT_ENV, git, lines } from './lab-common.mjs'
const FILTER_ARGS = ['filter-repo', '--sensitive-data-removal', '--invert-paths', '--path', '.env', '--path', 'config/credentials.json', '--force']
console.log('history before: ' + lines(git(['rev-list', '--all'])).length + ' commits')
console.log('running: git ' + FILTER_ARGS.join(' '))
const env = Object.assign({}, process.env, GIT_ENV)
const rewrite = spawnSync('git', ['-C', FIXTURE].concat(FILTER_ARGS), { env: env, stdio: 'inherit' })
if (rewrite.status !== 0) throw new Error('git filter-repo failed')
console.log('history after:')
console.log(git(['log', '--all', '--format=%h %ad %s', '--date=short', '--name-only']).trim())
// The log keeps every line, so it still tells you what happened and when.
const logFile = join(FIXTURE, 'logs', 'app.log')
let redacted = 0
const cleaned = readFileSync(logFile, 'utf8').replace(new RegExp(CANARY_PATTERN, 'g'), () => {
redacted = redacted + 1
return 'REDACTED-REVOKED-KEY'
})
writeFileSync(logFile, cleaned)
console.log('logs/app.log: ' + redacted + ' value(s) redacted, ' + lines(cleaned).length + ' lines kept')
================================================================
FILE 14 OF 15: prove.mjs
================================================================
// prove.mjs, step 5: shows that the old keys are dead, the new keys work and
// no canary or new key value is left anywhere the lab can look. Exits with
// code 1 if any check fails.
import { readFileSync } from 'node:fs'
import { join } from 'node:path'
import { CANARIES, FIXTURE, callService, git, lines, mask, readHostEnv } from './lab-common.mjs'
import { AREA_LABELS, searchFixture } from './search.mjs'
import { checkWebRoot } from './serve-check.mjs'
const failures = []
function check(passed, text) {
console.log((passed ? 'PASS ' : 'FAIL ') + text)
if (!passed) failures.push(text)
}
console.log('1. Each old key, presented to the key service')
for (const canary of CANARIES) {
const result = await callService('/use', { headers: { 'x-lab-key': canary.value } })
check(result.status === 401, mask(canary.value).padEnd(32) + 'HTTP ' + result.status + ' ' + result.body.reason)
}
console.log('2. Each new key, presented to the key service')
const hostEnv = readHostEnv()
for (const name of Object.keys(hostEnv)) {
const result = await callService('/use', { headers: { 'x-lab-key': hostEnv[name] } })
check(result.status === 200, name.padEnd(32) + 'HTTP ' + result.status + ' ' + result.body.result + ' ' + result.body.key)
}
console.log('3. What the key service recorded about the refused attempts')
const events = await callService('/admin/events', { method: 'GET', admin: true })
for (const event of events.body.events.filter((text) => text.includes('refused'))) {
console.log(' ' + event.slice(event.indexOf(' ') + 1))
}
console.log('4. The first changed commit git filter-repo reported is gone')
const firstChanged = readFileSync(join(FIXTURE, '.git', 'filter-repo', 'first-changed-commits'), 'utf8')
for (const hash of lines(firstChanged)) {
let type = 'missing'
try {
type = git(['cat-file', '-t', hash]).trim()
} catch (error) {
type = 'missing (git cat-file -t ends with a fatal error)'
}
check(type.startsWith('missing'), hash.slice(0, 7) + ': ' + type)
}
console.log('5. No canary and no new key value in any of the five places')
const found = searchFixture(Object.values(hostEnv))
for (const area of Object.keys(AREA_LABELS)) {
check(found[area].length === 0, AREA_LABELS[area].padEnd(32) + found[area].length + ' match(es)')
}
console.log('6. A web server whose web root is only the build output')
for (const result of await checkWebRoot(join(FIXTURE, 'dist'))) {
const expected = result.path === '/client.js' ? 200 : 404
check(result.status === expected && result.canaries.length === 0, ('GET ' + result.path).padEnd(32) + 'HTTP ' + result.status)
}
console.log(failures.length === 0 ? 'proof complete: every check passed' : 'proof failed: ' + failures.length + ' check(s) failed')
process.exitCode = failures.length === 0 ? 0 : 1
================================================================
FILE 15 OF 15: leak.test.mjs
================================================================
// leak.test.mjs: the regression test. One test per place the lab searches.
// It fails while any canary is left in the fixture and passes once all five
// places are clean. Run it with: node --test leak.test.mjs
import { test } from 'node:test'
import assert from 'node:assert/strict'
import { mask } from './lab-common.mjs'
import { AREA_LABELS, searchFixture } from './search.mjs'
const found = searchFixture()
for (const area of Object.keys(AREA_LABELS)) {
test('no canary in the ' + AREA_LABELS[area], () => {
const places = found[area].map((hit) => hit.where + ' ' + mask(hit.value))
assert.equal(places.length, 0, 'canary found in: ' + places.join('; '))
})
}
================================================================
PART 2: THE RECORDED RUN
================================================================
Each command was run from the secrets-lab folder, in this order. Every output is copied from that run, and each place where lines were cut is marked.
Command: node service.mjs start
key service started on 127.0.0.1 (port chosen by the system), pid 81906
Command: node make-fixture.mjs
fixture repository created, git directory: .git
commit 1819d8c 2026-10-01 Start the shop app
commit 5888342 2026-10-08 Add mailer settings
commit 0f017ba 2026-10-15 Remove .env from the repository
build: dist/client.js written with 1 public setting(s)
app: orders HTTP 200
app: payments HTTP 200
Reading it: three commits with fixed dates and the invented author. The build copied one PUBLIC_ setting, the search key, into dist/client.js, and the app's two calls succeeded, writing the payments key into logs/app.log on the way.
Command: node detect.mjs
pattern searched for: lab-canary-[a-z]+-[0-9a-f]{16}
working tree 1 match(es)
config/credentials.json lab-canary-orders-...4b85
history 4 match(es)
0f017ba config/credentials.json lab-canary-orders-...4b85
5888342 .env lab-canary-mailer-...3a95
5888342 config/credentials.json lab-canary-orders-...4b85
1819d8c config/credentials.json lab-canary-orders-...4b85
object store 2 match(es)
blob 5e1e6c5 lab-canary-mailer-...3a95
blob 7d38da6 lab-canary-orders-...4b85
client bundle 1 match(es)
dist/client.js lab-canary-search-...4b36
logs 1 match(es)
logs/app.log lab-canary-payments-...b653
distinct canaries found: 4
web root: the whole project folder (misconfigured)
GET /.env 404 canaries in body: 0 first line: Not found
GET /.git/HEAD 200 canaries in body: 0 first line: ref: refs/heads/main
GET /.git/config 200 canaries in body: 0 first line: [core]
GET /config/credentials.json 200 canaries in body: 1 first line: {
GET /client.js 404 canaries in body: 0 first line: Not found
Reading it:
working tree The orders key in config/credentials.json, where the app reads it today.
history Four hits in three commits: the orders key in all three, because config/credentials.json never changed, and the mailer key in 5888342 only. The commit that deleted .env, 0f017ba, has no .env file. The commit before it still does.
object store Two blobs: the credentials file, unchanged across the three commits, and the deleted .env file, which sits in the fixture's .git folder.
client bundle The search key, copied in because its variable carried the PUBLIC_ prefix.
logs The payments key: held in the environment, then copied into the log by one debug line.
web root With the project folder as the web root, /.git/HEAD and /.git/config return git's own files and /config/credentials.json returns the orders key. /.env returns 404 only because the file was deleted; its key is still in the .git folder this server hands out. /client.js returns 404 because, in this layout, the bundle sits under /dist.
Command: node --test leak.test.mjs
✖ no canary in the working tree (1.229792ms)
✖ no canary in the history (0.082167ms)
✖ no canary in the object store (0.061083ms)
✖ no canary in the client bundle (0.058375ms)
✖ no canary in the logs (0.05275ms)
ℹ tests 5
ℹ suites 0
ℹ pass 0
ℹ fail 5
ℹ cancelled 0
ℹ skipped 0
ℹ todo 0
ℹ duration_ms 146.757
✖ failing tests:
test at leak.test.mjs:12:3
✖ no canary in the working tree (1.229792ms)
AssertionError [ERR_ASSERTION]: canary found in: config/credentials.json lab-canary-orders-...4b85
1 !== 0
(stack trace and error details removed here: they hold local file paths)
test at leak.test.mjs:12:3
✖ no canary in the history (0.082167ms)
AssertionError [ERR_ASSERTION]: canary found in: 0f017ba config/credentials.json lab-canary-orders-...4b85; 5888342 .env lab-canary-mailer-...3a95; 5888342 config/credentials.json lab-canary-orders-...4b85; 1819d8c config/credentials.json lab-canary-orders-...4b85
4 !== 0
(stack trace and error details removed here: they hold local file paths)
test at leak.test.mjs:12:3
✖ no canary in the object store (0.061083ms)
AssertionError [ERR_ASSERTION]: canary found in: blob 5e1e6c5 lab-canary-mailer-...3a95; blob 7d38da6 lab-canary-orders-...4b85
2 !== 0
(stack trace and error details removed here: they hold local file paths)
test at leak.test.mjs:12:3
✖ no canary in the client bundle (0.058375ms)
AssertionError [ERR_ASSERTION]: canary found in: dist/client.js lab-canary-search-...4b36
1 !== 0
(stack trace and error details removed here: they hold local file paths)
test at leak.test.mjs:12:3
✖ no canary in the logs (0.05275ms)
AssertionError [ERR_ASSERTION]: canary found in: logs/app.log lab-canary-payments-...b653
1 !== 0
(stack trace and error details removed here: they hold local file paths)
The process exits with code 1. Each failure names the place and the masked canary.
Command: node contain.mjs
revoke lab-canary-orders-...4b85 HTTP 200 {"revoked":"orders-v1"}
revoke lab-canary-mailer-...3a95 HTTP 200 {"revoked":"mailer-v1"}
revoke lab-canary-search-...4b36 HTTP 200 {"revoked":"search-v1"}
revoke lab-canary-payments-...b653 HTTP 200 {"revoked":"payments-v1"}
the deployed app, still on its old keys:
app: orders HTTP 401
app: payments HTTP 401
Reading it: the four keys are revoked at the key service before anything is deleted or rewritten. The deployed app, still on its old keys, now gets 401 on both calls. That is the cost of revoking first, and it lasts until step 3. Its debug line also wrote the revoked payments key into the log a second time, which is why step 4 redacts two values, not one.
Command: node rotate.mjs
issued orders-v2 stored as ORDERS_KEY in the host environment
issued mailer-v2 stored as MAILER_KEY in the host environment
issued search-v2 stored as SEARCH_KEY in the host environment
issued payments-v2 stored as PAYMENTS_KEY in the host environment
commit 75e14d5 2026-11-14 Read keys from the environment
build: dist/client.js written with 0 public setting(s)
app: orders HTTP 200
app: payments HTTP 200
Reading it: four replacement keys, held only in lab-state/host-env.json, which stands in for your host's environment settings and is never committed, built or served. Commit 75e14d5 deletes config/credentials.json and changes the app to read every key from the environment and to log the first 12 characters of a SHA-256 hash instead of the key. The rebuilt bundle has no public settings, and both calls succeed again.
Command: node clean.mjs
history before: 4 commits
running: git filter-repo --sensitive-data-removal --invert-paths --path .env --path config/credentials.json --force
Parsed 4 commitsHEAD is now at 572ee61 Read keys from the environment
New history written in 0.06 seconds; now repacking/cleaning...
You rewrote 4 (of 4) commits.
NOTE: First Changed Commit(s) is/are:
1819d8c4a6b3633d314b89fbb130676ee5c32496
NOTE: LFS object orphaning not checked (LFS not in use)
Repacking your repo and cleaning out old unneeded objects
Completely finished after 0.15 seconds.
NEXT STEPS FOR YOUR SENSITIVE DATA REMOVAL:
* If you are doing your rewrite in multiple steps, ignore these next steps
until you have completed all your invocations of git-filter-repo.
* (bullet cut here: it refers the reader to git filter-repo's own documentation)
* Inspect this repository and verify that the sensitive data is indeed
completely removed from all commits.
* Force push the rewritten history to the server:
git push --force --mirror origin
* Contact the server admins for additional steps they need to take; the
First Changed Commit(s) may come in handy here.
* (bullet cut here: it refers the reader to git filter-repo's own documentation)
* (bullet cut here: it refers the reader to git filter-repo's own documentation)
history after:
572ee61 2026-11-14 Read keys from the environment
src/app.mjs
b530fb6 2026-10-15 Remove .env from the repository
.gitignore
ebc1b8d 2026-10-01 Start the shop app
.gitignore
README.md
src/app.mjs
src/build.mjs
logs/app.log: 2 value(s) redacted, 10 lines kept
Reading it:
- git filter-repo rewrote all four commits and gave the first changed commit as 1819d8c. Its progress counter and the "HEAD is now at" message share one line in the captured output. A carriage return written before the counter was left out, which changes nothing a terminal would show.
- "Add mailer settings" is gone. It only added .env, so it became empty and was pruned, which is what git filter-repo does by default with a commit that becomes empty.
- The three remaining commits have new ids, and neither .env nor config/credentials.json appears in any of them.
- --force was needed because the fixture is not a fresh clone: in a separate trial without it, git filter-repo refused to rewrite the fixture and asked for a fresh clone, or --force to proceed anyway. git filter-repo's documentation describes --force as ignoring its fresh clone safety checks, for an irreversible rewrite that by default ends with an immediate pruning of reflogs and old objects. That pruning is why the object store search in step 5 comes back empty. On a real repository, follow GitHub's procedure and work in a fresh clone of the remote, which is what the safety check looks for, so --force is not needed. git filter-repo's documentation calls it a really bad idea to get in the habit of always specifying --force, and says not to recommend it without first explaining that it puts repository data at risk.
- The three cut bullets refer to git filter-repo's documentation on sensitive data removal: the detail behind the steps, how colleagues clean their clones (the documentation calls deleting existing clones and cloning again the easiest way), and how to prevent repeats.
- The fixture has no remote, so there is nothing to force push and no server to clean. On a real repository, GitHub's procedure continues with git push --force --mirror origin and, where needed, a request to GitHub Support.
- The log keeps all ten lines. Only the two revoked key values were replaced.
Command: node prove.mjs
1. Each old key, presented to the key service
PASS lab-canary-orders-...4b85 HTTP 401 revoked
PASS lab-canary-mailer-...3a95 HTTP 401 revoked
PASS lab-canary-search-...4b36 HTTP 401 revoked
PASS lab-canary-payments-...b653 HTTP 401 revoked
2. Each new key, presented to the key service
PASS ORDERS_KEY HTTP 200 accepted orders-v2
PASS MAILER_KEY HTTP 200 accepted mailer-v2
PASS SEARCH_KEY HTTP 200 accepted search-v2
PASS PAYMENTS_KEY HTTP 200 accepted payments-v2
3. What the key service recorded about the refused attempts
refused revoked key orders-v1
refused revoked key payments-v1
refused revoked key orders-v1
refused revoked key mailer-v1
refused revoked key search-v1
refused revoked key payments-v1
4. The first changed commit git filter-repo reported is gone
PASS 1819d8c: missing (git cat-file -t ends with a fatal error)
5. No canary and no new key value in any of the five places
PASS working tree 0 match(es)
PASS history 0 match(es)
PASS object store 0 match(es)
PASS client bundle 0 match(es)
PASS logs 0 match(es)
6. A web server whose web root is only the build output
PASS GET /.env HTTP 404
PASS GET /.git/HEAD HTTP 404
PASS GET /.git/config HTTP 404
PASS GET /config/credentials.json HTTP 404
PASS GET /client.js HTTP 200
proof complete: every check passed
Reading it:
1 and 2 The provider side of the proof: each old key is refused as revoked, and each new key is accepted.
3 The key service recorded six refused attempts by key name: the deployed app's own two calls after revocation in step 2, then the four made in this step. An attempt on a dead key can come from a component of your own.
4 git cat-file -t on the first changed commit ends with a fatal error. git filter-repo's documentation gives this check for a clone that was cleaned up rather than cloned again; here it is run on the rewritten fixture itself.
5 No canary and no new key value in any of the five places.
6 With only dist/ as the web root, the four sensitive paths return 404 and the bundle is still served.
Command: node --test leak.test.mjs
✔ no canary in the working tree (0.667541ms)
✔ no canary in the history (0.046708ms)
✔ no canary in the object store (0.042833ms)
✔ no canary in the client bundle (0.036667ms)
✔ no canary in the logs (0.039708ms)
ℹ tests 5
ℹ suites 0
ℹ pass 5
ℹ fail 0
ℹ cancelled 0
ℹ skipped 0
ℹ todo 0
ℹ duration_ms 138.310666
The process exits with code 0, which a pipeline step that treats a non-zero exit code as a failure counts as a pass.
Command: node service.mjs stop
key service stopped, pid 81906
The key service is stopped by the process id it recorded when it started.
================================================================
PART 3: THE SAME SEQUENCE ON YOUR OWN SYSTEMS
================================================================
Run these steps only on systems you own or are authorised in writing to test.
Before you change anything
[ ] List each key: its provider, what it can reach, where it is configured, who can rotate it, and whether the provider documents it as safe to expose.
[ ] Record where you found each exposed copy and since when. The cleanup removes that evidence.
[ ] Check the provider's dashboard and logs for use of each key that you do not recognise.
Contain
[ ] Stop the source, so a replacement cannot leak the same way: remove the public prefix, the debug line or the served file.
[ ] Revoke each exposed key, or rotate it with the shortest overlap the deployment allows. Revoke at once if there is any sign someone else is using it.
[ ] If someone else has used a key, or it could reach personal information, bring in an incident response specialist and take advice about the Notifiable Data Breaches scheme.
Rotate
[ ] Create the replacement at the provider and put it only in the host's environment settings or secrets store.
[ ] Move every component to the new key, then expire or delete the old one.
[ ] Rebuild and redeploy anything that had the old value built in.
Clean up
[ ] History: if a rewrite is worth its side effects, run git filter-repo with --sensitive-data-removal in a fresh clone, force push, ask GitHub Support about cached views and pull request references if needed, and have colleagues clone again.
[ ] Logs: remove the secret wherever the logs are kept without losing the record of what happened. For a GitHub Actions run log, delete the log and rotate the secret.
[ ] Served files: deploy only the build output, so .git, .env and configuration files are not in the web root.
Prove
[ ] Call the provider's API with the old key from your own computer, using a read-only request, and confirm it is refused.
[ ] Search the working tree, every commit, the build output and the logs again, adding a distinctive part of each new key to the search.
[ ] Request /.env, /.git/HEAD and /.git/config on each of your hostnames and confirm an error with none of the file's content.
[ ] After a history rewrite, run git cat-file -t on each first changed commit in each clone and confirm a fatal error.
Prevent the next one
[ ] Ignore environment files before the first commit, and turn on push protection or a pre-commit check.
[ ] Add a regression check, like leak.test.mjs, that fails the build if a revoked key, or a canary you planted on purpose, turns up again.What counts as an exposed secret, and what does not
An exposed API key is a credential that has reached a place where more people or systems can read it than ever needed to: a public repository, the JavaScript every visitor downloads, a log file, or a file the web server hands to anyone who asks for it.
The same goes for every other secret an application holds. OWASP's Secrets Management Cheat Sheet lists the more common types of secret worth detecting, among them API keys, connection strings, credentials, passwords, private keys, session tokens and application configuration files.
Not every key in a browser is a leak. Some providers design a key to be public, put the protection somewhere else, and say so in their own documentation. Stripe says its publishable key can go in front-end code or applications you distribute, and that it cannot perform sensitive operations such as creating charges or reading account data.
Supabase lists its publishable key as safe to expose online and says it only reaches what Row Level Security allows. Firebase says that if an app's setup follows its guidelines, API keys restricted to Firebase services do not need to be treated as secrets and are safe to include in code or configuration files.
It also says the security of Realtime Database, Cloud Firestore and Cloud Storage data is enforced by Firebase Security Rules, and protection of covered APIs by Firebase App Check, not by keeping the API key secret.
The same pages draw the other line just as plainly. Stripe says only publishable keys are safe to expose outside your application's back end, and that you are responsible for protecting the others, restricted keys included. Supabase says a secret key bypasses every Row Level Security policy and must never be put in a browser, a shipped application or source control.
So the question for any key you find is not whether it looks secret. It is what the key's own provider says about exposing it, and what the key can do in someone else's hands.
A key that is meant to be public still deserves the limits its provider offers: Firebase, for one, calls it critical to apply appropriate API restrictions to every API key in a project.
Two neighbouring guides take the questions on either side of this one. Session tokens are secrets too, but how they are issued, rotated and ended belongs to the session management guide. Personal and business data that a page or API hands to the wrong person is the subject of the sensitive data exposure guide. This page stays with credentials: the keys, tokens and passwords an application uses to reach other systems.
Five places keys leak, and why each copy is hard to take back
A leaked key is a copy, and changing the original does nothing to the copy.
Four of the places below match the four canaries in the fixture on this page. The fifth, the web server, is checked by the same fixture with a small local server.
The plainest leak is a key typed into a configuration file, a settings module, a test fixture or a seed script, then committed. MITRE describes it as CWE-798, Use of Hard-coded Credentials, and names an outbound variant: a product that holds hard-coded credentials for connecting to another component, which the entry says applies to front-end systems that authenticate with a back-end service.
ASVS 5.0 requirement 13.3.1, at Level 2, asks for a secrets management solution, such as a key vault, to be used to securely create, store, control access to and destroy backend secrets, and says secrets must not be included in application source code or included in build artefacts.
Stripe puts it in one line for its own keys: don't put keys in source code or configuration files checked into version control.
In the fixture, the orders key sits in config/credentials.json. The first search finds it in the working tree and in all three commits, because the file never changed.
Deleting the line, or the whole file, adds a new commit. It does not change the commits before it, and they still hold the file as it was. The fixture's mailer key shows this.
It was committed in a .env file on 8 October and deleted a week later, and the search still finds it, because the commit that deleted the file (0f017ba) has no .env but the commit before it (5888342) does.
Adding .env to .gitignore afterwards changes nothing about that: git's documentation says a gitignore file specifies intentionally untracked files, and that files already tracked by Git are not affected.
The copies also travel beyond the repository you control. GitHub's documentation on removing sensitive data warns that even after history is rewritten and force-pushed, the old commits may still be accessible in clones and forks, directly through their SHA-1 hashes in cached views on GitHub, and through pull requests that reference them.
You cannot remove the data from other people's clones; GitHub says you send them instructions so they clean their own copies.
git's cat-file documentation describes a batch mode that works on all objects in the repository, not just reachable objects. The fixture's search uses it to read the object store as well as the commits, because a search of the commits covers only what they reach.
A browser can only run a script it can read, so a key inside a client bundle is public from the moment the bundle is deployed. Next.js and Vite both decide by a variable's name whether its value goes into the bundle.
Next.js inlines variables prefixed with NEXT_PUBLIC_ into the JavaScript sent to the browser at build time, and keeps unprefixed variables on the server. Vite exposes variables prefixed with VITE_ in client-side source code after bundling, and says they should not contain sensitive information such as API keys.
The fixture's build script copies that convention with a PUBLIC_ prefix, and a search key given the prefix ends up in dist/client.js.
Fixing the name fixes the next build, not the ones already made. Next.js notes that a built app no longer responds to changes to these variables, and that NEXT_PUBLIC_ variables are frozen with the value evaluated at build time. On this page's reading, a build made before the fix therefore still carries the old key.
The OWASP Web Security Testing Guide v4.2 test WSTG-INFO-05 takes the same look from outside: its objectives include reviewing JavaScript files for information leakage and checking whether source map files exist.
A log keeps a copy of whatever the code wrote to it, for as long as the log is kept. A debug line that prints a whole request or a whole settings object is one way a key gets there. In the fixture, the payments key was correctly held in the environment and still reached logs/app.log through a single debug line that printed the request headers.
MITRE catalogues this as CWE-532, Insertion of Sensitive Information into Log File, and its mitigations include not writing secrets into log files and removing debug log files before deploying to production. ASVS 5.0 requirement 16.2.5, at Level 2, asks the application to enforce logging based on the data's protection level when it logs sensitive data, and gives as an example that it may not be allowed to log certain data, such as credentials or payment details.
Supabase's guidance for its secret keys is not to log a key, even one that looks invalid, and to store a SHA-256 hash of it instead when you need to record which key was used. The fixed version of the fixture app logs the first 12 characters of such a hash.
Two things make a log harder to deal with than code. The copies keep accumulating until the code changes: after the fixture's keys were revoked, the deployed app wrote the dead payments key into its log a second time. And a log can travel further than the server that wrote it, into whatever collects, forwards or stores it, so the search has to follow it.
For GitHub Actions run logs, GitHub's guidance is to delete a log that received an unredacted secret and rotate the secret.
If the folder a site is served from is the project folder rather than only the build output, anything in it may be requested by path: an environment file, a configuration file, or the .git folder that holds the repository's history.
ASVS 5.0 requirement 13.4.1, at Level 1, asks for applications to be deployed without source control metadata such as the .git or .svn folders, or in a way that makes those folders inaccessible both externally and to the application itself.
MITRE's CWE-527 describes a source-control repository made accessible to unauthorised actors, and notes that its metadata may include detailed diff data about how files have changed, which could reveal source code snippets that were never intended to be made public.
The latest edition of the OWASP Web Security Testing Guide, which says of itself that it may frequently change, lists .env among credentials and secrets that should generally never be publicly served in its test for old, backup and unreferenced files (WSTG-CONF-04), and lists .git/ as a special path to check.
The fixture shows both. With the project folder as the web root, /.git/HEAD and /.git/config answer 200 with git's own files, and /config/credentials.json returns the orders key. /.env answers 404 only because the file was deleted from the folder; its key is still in the .git folder the same server is handing out.
How to confirm a leak on your own property
Run these checks only on repositories, builds, logs and domains you own or are authorised in writing to test.
Each one looks at your own property from the inside or requests your own hostnames, and none of it involves looking for anyone else's keys.
- Start with an inventory. List each key the application uses: the provider, what the key can reach, where it is configured, who can rotate it, and whether the provider documents it as safe to expose. A search only finds what it is told to look for. The fixture's search is an example: it matches the lab's own canary format and would pass straight over a real key in any other format.
- Search the working tree and every commit. git grep searches tracked files, and given a list of commits it searches the blobs in each commit's tree; git rev-list --all lists the commits reachable from every ref and from HEAD. Search for a distinctive part of each key rather than pasting the full value into a command, where it would stay in your shell history.
- If the repository is on GitHub, read its secret scanning alerts. GitHub says secret scanning covers the entire Git history on all branches for known secret types. It runs automatically for free on public repositories, and private and internal repositories owned by an organisation need GitHub Secret Protection enabled on GitHub Team or the one other plan GitHub's page names for those repositories. A private repository owned by a personal account is in neither group; GitHub's page covers user-owned repositories only under the managed-user and self-hosted server offerings it names. Custom patterns, which are regular expressions you define, cover formats the defaults do not.
- Search what you deploy, not only what you commit. The files your deploy step uploads are what visitors receive, so search the built JavaScript, and any source maps, for each key. The JavaScript bundle secret scanner in the related guides is a browser-based tool for that pass; treat what it reports as leads for a person to confirm.
- Search the logs where they end up: on the server, in whatever collects or forwards them, and in your CI run logs.
- Request the sensitive paths on your own hostnames, staging included. From a private browser window, or with a command-line HTTP client, request /.env, /.git/HEAD and /.git/config. A correctly configured server answers with an error, such as 404 or 403, and none of the file's content. Read the body as well as the status code: the latest WSTG-CONF-04 says to confirm a response actually returns the file's content, not a custom 404 or an access-denied page, and notes that custom "not found" pages returning 200 are a common false positive. In the fixture, the first line of the real HEAD file is ref: refs/heads/main.
- For each key you confirm, look it up in the provider's dashboard: whether it is still active and, where the provider shows it, when it was last used. Test whether a key works only if the key is yours.
The first hour: revoke or rotate before you clean anything
Once a key is confirmed exposed, the instinct is to delete it from wherever it was found.
That removes one copy and leaves the key working. GitHub's documentation on removing sensitive data sets the order: when the data is a secret, the first step is to revoke and/or rotate it. Once it is revoked or rotated it can no longer be used for access, and GitHub says that may be enough on its own. git filter-repo's documentation also says to revoke and rotate the credential first, and GitHub's secret scanning page adds that removing a secret from Git history is time-intensive and often unnecessary once the credential has been revoked.
OWASP's Secrets Management Cheat Sheet breaks containment into four steps, in this order: revocation, immediate for keys that were exposed; rotation, so a new secret can be created and put in place quickly; deletion of the revoked or rotated secret from the exposed system, code and logs included; and logging, so whoever handles the incident knows who had access, when the key was used and when it was last rotated.
Supabase adds two conditions in its guide to rotating a leaked key. Fix the root cause before you rotate anything, or the new key can leak the same way. And if someone is actively using the leaked key, retire it right away, even though components that still use it stop working until it is replaced.
The fixture takes that path: contain.mjs revokes all four keys first, and the deployed app's next two calls fail with 401 until rotate.mjs fixes the code and puts the replacements in place.
- Write down what you found before you change it: where each copy was, since when, and who could read that place. The cleanup removes the evidence.
- Look in the provider's dashboard and logs for any use of each key that you do not recognise.
- Stop the source, so a replacement cannot leak the same way: take the served file away, switch off the debug line, or remove the public prefix.
- Revoke each exposed key, or rotate it with the shortest overlap your deployment allows. Revoke at once if there is any sign someone else is using it.
- If someone else has used a key, or it could reach customers' personal information, bring in an incident response specialist. Clever Ops does not provide incident response or monitoring.
- If personal information may have been reached, get advice early. The OAIC explains that under the Notifiable Data Breaches scheme, an organisation or agency covered by the Privacy Act 1988 must notify affected individuals and the OAIC when a data breach is likely to result in serious harm to a person whose personal information is involved.
- If you hold cyber insurance, read what the policy says about telling the insurer before you engage anyone.
Rotating a key without an outage, and when to accept one
Rotation is two changes that have to meet: the provider starts accepting a new key, and every component that used the old key moves to it.
If the old key dies first, whatever still uses it fails, as the fixture's app did between steps 2 and 3. Some providers support an overlap, in which both keys work for a while, so the move can happen without a gap.
Stripe's documentation describes one. Rotating a key generates a replacement that is ready to use immediately, and the dashboard asks for an expiration date for the old key: choosing Now deletes the old key, and choosing a time displays the time remaining until it expires.
Stripe says that when you rotate a key in the Dashboard, both the old and new keys work for up to 7 days. To avoid downtime, it suggests, if possible, using the new key from a small subset of servers or services first and watching server logs for errors, and checking the old key's request logs so it is expired only after its request volume has been at zero for a few hours or days.
Supabase's guide follows the same shape for a leaked secret key: create a new secret key, replace the compromised key everywhere the application uses it, confirm every component uses the new one, then delete the old key, which cannot be undone. Unless someone is actively using the leaked key, it says to wait until every component has moved before retiring it.
For an exposed key, an overlap is also a window in which the exposed key still works. Keep it as short as the deployment allows, and skip it when there is any sign of use you do not recognise. Restarting the application on a new key is not the same as killing the old one.
In its section on automating secrets management, where it discusses dynamic secrets, OWASP's cheat sheet notes that restarting the application does not itself revoke stolen credentials, which remain usable until the backing service expires or revokes them.
Two habits make the next rotation smaller. Supabase recommends a separate secret key for each backend component, so a leak from one component means rotating only that key. OWASP recommends rotating secrets regularly, so a stolen credential works only for a short time, and ASVS 5.0 requirement 13.3.4, at Level 3, asks for secrets to be configured to expire and be rotated based on the application's documentation.
Cleaning up history, bundles, logs and served files
Cleanup comes after the key is dead.
It limits what the old copies reveal; it does not take the key back. Anyone who cloned the repository, forked it or copied a log before the cleanup still has the key, which is why the revocation in the first hour matters more than anything in this section.
GitHub's procedure uses git filter-repo, in a version with the --sensitive-data-removal flag (2.47 or later), run in a clone of the repository. To delete a specific file from all branches, tags and refs it uses --invert-paths with --path and the file's path in git.
If the file used to exist at any other paths, GitHub says to either add an extra --path argument for that file or run the command a second time naming the alternative path. To replace text listed in a file across the history's non-binary files it uses --replace-text. git's own documentation for its built-in tool, git filter-branch, says that tool has a plethora of pitfalls and that its use is not recommended, and asks users to use an alternative history filtering tool such as git filter-repo.
GitHub lists the side effects, among them: changed hashes for the commit that introduced the data and every commit after it; a high risk of recontamination, because a colleague with an older clone who runs git pull and then git push brings the data straight back; branch protections against force pushes that have to be lifted for a time; closed pull requests that lose their diff views; and lost signatures on commits and tags.
After the force push, GitHub Support can remove cached views and references to the data in pull requests.
GitHub says Support takes those steps once you have cleaned up all references other than pull requests and no forks reference the data, and that it only assists where it determines the risk cannot be mitigated by rotating the affected credentials. If the commit that introduced the data exists in any forks, GitHub says it will continue to be accessible there and you need to coordinate with the forks' owners. git filter-repo's documentation adds that the easiest way to clean colleagues' copies is for everyone to delete their clones and clone again.
The fixture runs git filter-repo with --force, which its documentation describes as ignoring the fresh clone safety checks for an irreversible rewrite that by default ends with an immediate pruning of reflogs and old objects. The fixture needs it because it is the only copy and not a fresh clone.
The rewrite changed all four commits, dropped "Add mailer settings" because it became empty once .env was removed, and reported 1819d8c as the first changed commit.
Rebuild without the value and redeploy, then deal with the builds that came before it. Next.js says NEXT_PUBLIC_ variables are frozen with the value evaluated at build time. On this page's reading, any earlier build your host still serves, and any cached copy of one, still carries the old key, which is one more reason the key has to be dead before the cleanup starts.
OWASP's cheat sheet says secrets in logs must have a process for removing the secret while maintaining log integrity. The fixture replaces each revoked key with REDACTED-REVOKED-KEY and keeps all ten lines, so the log still shows when each call was made and what it returned. For a GitHub Actions run log that an unredacted secret was sent to, GitHub's guidance is to delete the log and rotate the secret.
Deploy only the build output, so the web root never contains .git, .env or configuration files in the first place. For the .git folder, that is the first of the two options ASVS 13.4.1 gives: deploy without source control metadata, or make those folders inaccessible both externally and to the application itself.
For the rest, the WSTG v4.2 test for old, backup and unreferenced files advises storing data files, log files and configuration files in directories the web server cannot reach. In the fixture, a web root of dist/ answers 404 for all four sensitive paths and still serves client.js.
Proving the old key is dead
Proof has two halves: the old key no longer works, and no copy of it, or of its replacement, is left where you looked.
Only the provider can answer the first. For a real key, that means calling the provider's API from your own computer with the old credential and seeing it refused. Stripe, for example, says a request that includes an expired key gets an authentication error.
Use a read-only request, and only with keys that are yours: a key you found that belongs to someone else is reported to its owner, not tried.
Then look at what the provider recorded after the key was revoked. OWASP's cheat sheet lists being revocable, including the logging of attempts to use a revoked secret, among the properties a secret should have. The fixture's key service records each refused attempt by key name, never by value, and its record shows six: the deployed app's own two calls after revocation, then the four made by the proof step. An attempt on a dead key may come from a component of your own that nobody moved.
The second half is the searches again, with the new keys added, plus the served paths. After a history rewrite, git filter-repo's documentation gives a check for a clone that was cleaned up rather than cloned again: git cat-file -t on each first changed commit must end in a fatal error.
In the fixture every one of these passes, with the clone check run on the rewritten fixture itself. The regression test then holds the result in place. It fails while any canary is left in any of the five places and passes once they are clean.
The Node.js test runner sets the exit code to 1 when a test fails, so a pipeline step that treats a non-zero exit code as a failure fails with it.
Stopping the next leak
Each leak in the fixture had a small, specific cause, and each has a small, specific guard.
- Keep keys out of code. GitHub advises against hardcoding secrets and suggests environment variables or a secret management service that injects them at runtime. Stripe suggests your hosting platform's secrets vault and, where you cannot use one, environment variables. Fail loudly when a key is missing, as the fixed fixture app does, rather than falling back to a value in the code.
- Ignore environment files before the first commit. Next.js says its default create-next-app template adds all .env files to .gitignore and that you almost never want to commit them. Vite says .env.*.local files can contain sensitive variables and that you should add *.local to .gitignore. An ignore rule added later does not remove a file that is already tracked.
- Check before the push. GitHub's push protection blocks pushes that contain secrets before they reach the repository. Push protection for users is on by default and stops you pushing secrets to public repositories on GitHub.com; push protection for repositories needs GitHub Secret Protection and is off until someone enables it. Once push protection for repositories is enabled, GitHub says you can customise it by defining custom patterns, regular expressions you define to detect secrets specific to, for example, your repository or organisation, and you can choose to enable push protection for each pattern. That is one way a format of your own, such as a canary, could be blocked at the push. A pre-commit hook, which GitHub and OWASP both suggest, checks for a secret before it is committed at all.
- Give the public prefix only to values the provider calls safe to expose. NEXT_PUBLIC_ and VITE_ mean every visitor gets a copy.
- Log a hash, never the key, and keep request dumps out of production. CWE-532's mitigations include removing debug log files before deploying to production.
- Deploy the build output, not the repository, so nothing under .git or in an environment file sits in the web root.
- Plant canaries of your own: a unique, harmless value in the places a real key would sit, searched for on every build. A canary that turns up somewhere new shows a copy escaping before a real key does.
- Keep pipeline secrets narrow. How a pipeline hands secrets to its jobs, and what a compromised job can read, is covered in the CI/CD security guide.
- Watch for a dead key being presented. Alerting on events like that, and keeping secrets out of logs by design, belong with the security logging and monitoring guide.
Where a Clever Ops check fits
Clever Ops does not monitor repositories, logs or sites and does not respond to incidents.
If someone else has used one of your keys, start with the provider and an incident response specialist. What Clever Ops offers sits on either side of an incident: a check before one, and a check after the keys are rotated.
Each report records what was checked and on which dates. It is not monitoring, and a key committed the day after the report is outside what it can speak to. If Clever Ops built or hosts the site, the work is a security review rather than an independent test, and the report says who built it and who hosts it.
- The website security check reads the page source and JavaScript your site sends to every browser for keys and secrets, and looks for configuration files and other leftovers at paths the web server will serve. A key it finds is recognised by its format, not tried.
- A secure code review looks for keys in source files, test fixtures, seed scripts, configuration, build output and the history of the branches in scope, and reports each one with what it can reach and a note to rotate it.
- The vibe-coded app security check searches the JavaScript an AI-built app sends to browsers and, if you share it, the exported code and its history, for secret and service keys.
Frequently asked questions
No. Some keys are designed to be public, and the provider's documentation says so: Stripe's publishable key can go in front-end code, and Supabase lists its publishable key as safe to expose online. A secret, server-side or restricted key in the same place is a leak. Check what the provider says about that exact key type, then restrict even the public ones as far as the provider allows.
No. The commits before the deletion still hold it, and so does every clone and fork made from them. Revoke or rotate the key at the provider first. GitHub's documentation says that once a secret is revoked or rotated it can no longer be used for access, and that this may be all you need. Rewriting history comes after that, if at all.
Only after the key is revoked, and only if what remains exposed is worth the disruption. GitHub lists the side effects, including changed commit hashes, lost diffs on closed pull requests and the risk that an old clone pushes the data straight back. If you go ahead, GitHub's documentation uses git filter-repo with the --sensitive-data-removal flag, run in a clone of the repository.
Request /.env, /.git/HEAD and /.git/config on each of your own hostnames from a private browser window. A correctly configured server returns an error and none of the file's content. Read the body as well as the status, because a custom not-found page can answer 200. If any of them returns the real file, treat every key it holds as exposed, and deploy only your build output from then on.
Where the provider supports it, run old and new keys side by side: create the new key, move every component to it, watch the old key's requests fall to zero, then expire it. Stripe lets you choose when the old key expires when you rotate, and Supabase's guide has you replace the key everywhere before deleting the old one. If there is any sign someone else is using the exposed key, revoke it at once and accept the short outage.
Call the provider's API with the old key from your own computer, using a read-only request, and confirm it is refused. Check the provider's logs for any use after you revoked it. Then search the repository history, the build output and the logs again, adding the new key to the search so you know it has not leaked the same way.
No. Clever Ops does not provide incident response or monitoring, so start with the provider and an incident response specialist. Once the keys are rotated, a website security check or a secure code review can look for keys the browser can read, files the server hands out and secrets in your code, and the report records what was checked.
Scope the right test for your application
Tell us what you have built, who uses it and when it needs to launch. You will get a plain-English recommendation: a website audit, a web application test or a referral to a specialist.