Skip to main content
Security guideReviewed

SAST: what static application security testing reads, what it misses and how to trial a tool

Static application security testing reads your code without running it and reports lines that look like known weaknesses.

This guide explains the difference between matching a pattern and following data, why tools report safe code and miss flawed code, and how to bring a tool into CI without burying the team. It comes with a labelled JavaScript fixture suite, a scorecard and a small local lab for measuring a tool before it gates a merge.

SAST fixture suite, results scorecard and pattern-checker lab

An illustration written for this page, not client code. Twelve pairs of flawed and safe JavaScript samples across six CWE-tagged weakness classes, an answer key, a script that fills in a results scorecard from any tool's SARIF output, and a deliberately small pattern checker with a regression test that pins its result for every case.

How to use it

It needs Node.js only, and the fixtures are read as text and never run.

sast.txt

ILLUSTRATION ONLY. The fixture suite, the pattern checker, the scorecard script and every result below were written for this page. They are not a client's code or a client's result, and the checker is not a product. Use the lab on your own machine to learn, and to measure tools on code you own.

WHAT IS IN IT
1. A fixture suite: twelve pairs of flawed and safe JavaScript samples across six weakness classes, in the folder suite/. Each case starts with a marker comment, such as // case SQL-1F, and carries its CWE. F marks the flawed version and S the safe version of the same few lines.
2. An answer key, cases.mjs, that says for every case whether a tool should report it, whether finding it needs data flow, and why.
3. A scorecard script, score.mjs, that reads a SARIF 2.1.0 results file from any tool and fills in the scorecard: expected against reported for each case, then the rates, the coverage by class, the data-flow cases and the remediation fields. The blank scorecard and the definitions of every column are at the end.
4. A deliberately small pattern checker, checker.mjs, with six rules, a script that runs it and writes its SARIF file, and a regression test that pins its result for every case.

WHAT THE CHECKER IS NOT
The checker is an illustration. It reads each line of the suite as plain text and asks whether the line matches a regular expression. It builds no syntax tree, so it cannot tell code from a comment. It follows no data from one line to another, so it cannot see a check made two lines earlier or a value handed to a helper. It is not a static analyser, and it is not meant for real code. Its scorecard is useful because of those gaps: it shows, case by case, the false positives and false negatives that a line-by-line pattern produces.

FIXTURE SAFETY
The fixture code is read as text by the checker and by any tool you point at it. Nothing in the lab imports or runs it. Each flawed injection sample stops where its CWE entry places the weakness: it builds a query, a command, a path or a piece of a page from request input, and returns it. The credential and random samples return a settings object or a token. No sample connects to a database, starts a process, opens a file or serves a page. The credential values are invented for the lab and match no real provider's key format.

WHAT IT NEEDS
Node.js, with nothing to install. The lab's scripts import only node:fs, node:path, node:url, node:assert and node:test, and install no packages. The Node.js documentation marks the test runner stable from version 20.0.0. The outputs below were recorded on Node.js 26.0.0 on macOS. Since Node.js 23.0.0 the test runner's default spec format is also used when its output goes to a file; on earlier versions that output is TAP.

SET-UP
Create an empty folder, such as sast-lab, with a folder called suite inside it. Save the six suite files into suite/ and the other five files beside it, under the names shown. Run the commands under RUN IT from the sast-lab folder. The line numbers in the outputs refer to the files exactly as shown here.

==== File 1 of 11: suite/sql.js ====
// suite/sql.js: SQL query fixtures (CWE-89). Read as text, never run.
// Each function builds a query and returns it. Nothing connects to a database.

// case SQL-1F
export function findOrdersQuery(request) {
  return "SELECT id, total FROM orders WHERE customer_id = '" + request.query.customer + "'"
}

// case SQL-1S
export function findOrdersQuerySafe(request) {
  return { text: 'SELECT id, total FROM orders WHERE customer_id = ?', values: [request.query.customer] }
}

// case SQL-2F
function customerFilter(customerId) {
  return "customer_id = '" + customerId + "'"
}

export function listInvoicesQuery(request) {
  const customerId = request.query.customer
  return 'SELECT id, amount FROM invoices WHERE ' + customerFilter(customerId)
}

// case SQL-2S
function customerFilterSafe(customerId) {
  return { text: 'customer_id = ?', values: [customerId] }
}

export function listInvoicesQuerySafe(request) {
  const customerId = request.query.customer
  const filter = customerFilterSafe(customerId)
  return { text: 'SELECT id, amount FROM invoices WHERE ' + filter.text, values: filter.values }
}

// case SQL-3F
export function sortedOrdersQuery(request) {
  return 'SELECT id, total FROM orders ORDER BY ' + request.query.sort
}

// case SQL-3S
const SORT_COLUMNS = new Map([['newest', 'created_at'], ['largest', 'total']])

export function sortedOrdersQuerySafe(request) {
  return 'SELECT id, total FROM orders ORDER BY ' + (SORT_COLUMNS.get(request.query.sort) || 'created_at')
}

==== File 2 of 11: suite/command.js ====
// suite/command.js: OS command fixtures (CWE-78). Read as text, never run.
// Each function builds a command and returns it. Nothing starts a process.

// case CMD-1F
export function exportCommand(request) {
  return 'zip -q exports.zip ' + request.body.fileName
}

// case CMD-1S
const EXPORT_NAME = /^[a-z0-9][a-z0-9-]{0,62}\.csv$/

export function exportCommandSafe(request) {
  // Replaces the old line: return 'zip -q exports.zip ' + request.body.fileName
  const fileName = request.body.fileName
  if (!EXPORT_NAME.test(fileName)) throw new Error('Rejected file name')
  return { file: 'zip', args: ['-q', 'exports.zip', fileName] }
}

// case CMD-2F
export function backupCommand(request) {
  const folder = request.body.folder
  const target = 'backups/' + folder
  return 'tar -czf backup.tgz ' + target
}

// case CMD-2S
const BACKUP_FOLDERS = new Map([['invoices', 'backups/invoices'], ['orders', 'backups/orders']])

export function backupCommandSafe(request) {
  const target = BACKUP_FOLDERS.get(request.body.folder)
  if (!target) throw new Error('Unknown folder')
  return { file: 'tar', args: ['-czf', 'backup.tgz', target] }
}

==== File 3 of 11: suite/path.js ====
// suite/path.js: file path fixtures (CWE-22). Read as text, never run.
// Each function builds a path and returns it. Nothing opens a file.
import path from 'node:path'

const REPORTS_DIR = '/srv/app/reports'

// case PATH-1F
export function reportPath(request) {
  return path.join(REPORTS_DIR, request.params.name)
}

// case PATH-1S
const REPORT_NAME = /^[a-z0-9][a-z0-9-]{0,62}\.pdf$/

export function reportPathSafe(request) {
  const name = request.params.name
  if (!REPORT_NAME.test(name)) throw new Error('Rejected report name')
  return path.join(REPORTS_DIR, name)
}

// case PATH-2F
export function archivedReportPath(request) {
  const requested = request.query.path
  if (!requested.startsWith('/srv/app/reports/')) throw new Error('Not a report')
  return requested
}

// case PATH-2S
export function archivedReportPathSafe(request) {
  const requested = path.resolve(REPORTS_DIR, request.query.path)
  if (!requested.startsWith(REPORTS_DIR + path.sep)) throw new Error('Not a report')
  return requested
}

==== File 4 of 11: suite/page.js ====
// suite/page.js: HTML page fixtures (CWE-79). Read as text, never run.
// Each function builds a piece of a page and returns it. Nothing serves it.

// Shared by the safe cases. Escapes the five characters that change the
// meaning of HTML text and of a quoted attribute value.
function escapeHtml(text) {
  return String(text)
    .replaceAll('&', '&')
    .replaceAll('<', '&lt;')
    .replaceAll('>', '&gt;')
    .replaceAll('"', '&quot;')
    .replaceAll("'", '&#x27;')
}

// case XSS-1F
export function searchHeading(request) {
  return '<h1>Results for ' + request.query.q + '</h1>'
}

// case XSS-1S
export function searchHeadingSafe(request) {
  return '<h1>Results for ' + escapeHtml(request.query.q) + '</h1>'
}

// case XSS-2F
function escapeTags(text) {
  return String(text).replaceAll('<', '&lt;').replaceAll('>', '&gt;')
}

export function searchBox(request) {
  return '<input name="q" value="' + escapeTags(request.query.q) + '">'
}

// case XSS-2S
export function searchBoxSafe(request) {
  return '<input name="q" value="' + escapeHtml(request.query.q) + '">'
}

==== File 5 of 11: suite/credential.js ====
// suite/credential.js: credential fixtures (CWE-798). Read as text, never run.
// The values are invented for this lab and match no real provider's key format.

// case SECRET-1F
export function reportsClientSettings() {
  const reportsApiKey = 'fixture-only-not-a-real-key'
  return { baseUrl: 'https://reports.example.test', apiKey: reportsApiKey }
}

// case SECRET-1S
export function reportsClientSettingsSafe() {
  const reportsApiKey = process.env.REPORTS_API_KEY
  return { baseUrl: 'https://reports.example.test', apiKey: reportsApiKey }
}

// case SECRET-2F
export function mailSettings() {
  return { host: 'smtp.example.test', user: 'reports', pass: 'fixture-only-not-a-real-pass' }
}

// case SECRET-2S
export function mailSettingsSafe() {
  return { host: 'smtp.example.test', user: 'reports', pass: process.env.REPORTS_SMTP_PASSWORD }
}

==== File 6 of 11: suite/random.js ====
// suite/random.js: random token fixtures (CWE-338). Read as text, never run.
import crypto from 'node:crypto'

// case RAND-1F
export function createResetToken() {
  return Math.random().toString(36).slice(2)
}

// case RAND-1S
export function createResetTokenSafe() {
  return crypto.randomBytes(32).toString('base64url')
}

==== File 7 of 11: cases.mjs ====
// cases.mjs: the answer key for the fixture suite, and the code that works
// out which case a line in a suite file belongs to.
import fs from 'node:fs'
import path from 'node:path'
import { fileURLToPath } from 'node:url'

export const LAB_DIR = path.dirname(fileURLToPath(import.meta.url))
export const SUITE_DIR = path.join(LAB_DIR, 'suite')

// flawed: true means a tool should report this case.
// dataFlow: true means the value only reaches the risky line through a
// variable or a helper function, so a tool has to follow it there.
export const CASES = [
  { id: 'SQL-1F', file: 'sql.js', cwe: 'CWE-89', flawed: true, dataFlow: false, why: 'Request value joined straight into the WHERE clause.' },
  { id: 'SQL-1S', file: 'sql.js', cwe: 'CWE-89', flawed: false, dataFlow: false, why: 'Same query with a ? placeholder; the value travels separately.' },
  { id: 'SQL-2F', file: 'sql.js', cwe: 'CWE-89', flawed: true, dataFlow: true, why: 'Value passes through a variable and a helper before it is joined into the query.' },
  { id: 'SQL-2S', file: 'sql.js', cwe: 'CWE-89', flawed: false, dataFlow: false, why: 'Same route through the helper, which now returns a placeholder and a value.' },
  { id: 'SQL-3F', file: 'sql.js', cwe: 'CWE-89', flawed: true, dataFlow: false, why: 'Request value used as the ORDER BY column.' },
  { id: 'SQL-3S', file: 'sql.js', cwe: 'CWE-89', flawed: false, dataFlow: false, why: 'Column chosen from a fixed Map; the request only picks a key. Looks like SQL-3F to a pattern.' },
  { id: 'CMD-1F', file: 'command.js', cwe: 'CWE-78', flawed: true, dataFlow: false, why: 'File name joined into a command line meant for a shell.' },
  { id: 'CMD-1S', file: 'command.js', cwe: 'CWE-78', flawed: false, dataFlow: false, why: 'Allow-listed name passed as one argument, no shell. The old line survives in a comment.' },
  { id: 'CMD-2F', file: 'command.js', cwe: 'CWE-78', flawed: true, dataFlow: true, why: 'Folder name passes through two variables before it is joined into the command line.' },
  { id: 'CMD-2S', file: 'command.js', cwe: 'CWE-78', flawed: false, dataFlow: false, why: 'Folder chosen from a fixed Map and passed as an argument, no shell.' },
  { id: 'PATH-1F', file: 'path.js', cwe: 'CWE-22', flawed: true, dataFlow: false, why: 'Request value joined to the reports folder with no check.' },
  { id: 'PATH-1S', file: 'path.js', cwe: 'CWE-22', flawed: false, dataFlow: false, why: 'Name must match an allow-list pattern before it is joined.' },
  { id: 'PATH-2F', file: 'path.js', cwe: 'CWE-22', flawed: true, dataFlow: false, why: 'Prefix checked before ../ is resolved, so a path can start in the folder and leave it.' },
  { id: 'PATH-2S', file: 'path.js', cwe: 'CWE-22', flawed: false, dataFlow: false, why: 'Path resolved first, then checked against the folder and a separator.' },
  { id: 'XSS-1F', file: 'page.js', cwe: 'CWE-79', flawed: true, dataFlow: false, why: 'Search term joined into HTML with no encoding.' },
  { id: 'XSS-1S', file: 'page.js', cwe: 'CWE-79', flawed: false, dataFlow: false, why: 'Search term HTML-encoded before it is joined.' },
  { id: 'XSS-2F', file: 'page.js', cwe: 'CWE-79', flawed: true, dataFlow: false, why: 'Helper encodes < and > only; a double quote still ends the attribute value early.' },
  { id: 'XSS-2S', file: 'page.js', cwe: 'CWE-79', flawed: false, dataFlow: false, why: 'Same attribute, with the encoder that also handles quotes.' },
  { id: 'SECRET-1F', file: 'credential.js', cwe: 'CWE-798', flawed: true, dataFlow: false, why: 'API key written into the code under a name that says what it is.' },
  { id: 'SECRET-1S', file: 'credential.js', cwe: 'CWE-798', flawed: false, dataFlow: false, why: 'Key read from the environment at run time.' },
  { id: 'SECRET-2F', file: 'credential.js', cwe: 'CWE-798', flawed: true, dataFlow: false, why: 'Mail password written into the code under the short property name pass.' },
  { id: 'SECRET-2S', file: 'credential.js', cwe: 'CWE-798', flawed: false, dataFlow: false, why: 'Password read from the environment at run time.' },
  { id: 'RAND-1F', file: 'random.js', cwe: 'CWE-338', flawed: true, dataFlow: false, why: 'Password reset token made with Math.random().' },
  { id: 'RAND-1S', file: 'random.js', cwe: 'CWE-338', flawed: false, dataFlow: false, why: 'Token made from 32 bytes of crypto.randomBytes().' },
]

const MARKER = /^\/\/ case ([A-Z]+-\d[FS])$/

// Splits one suite file into case regions. A region runs from its marker
// comment to the line before the next marker, or to the end of the file.
export function readCaseRegions(fileName) {
  const lines = fs.readFileSync(path.join(SUITE_DIR, fileName), 'utf8').split('\n')
  const regions = []
  let current = null
  lines.forEach((text, index) => {
    const lineNumber = index + 1
    const match = MARKER.exec(text.trim())
    if (match) {
      current = { id: match[1], firstLine: lineNumber, lastLine: lineNumber }
      regions.push(current)
    } else if (current) {
      current.lastLine = lineNumber
    }
  })
  return regions
}

// Returns the case id for a line, or null for lines outside every case,
// such as the shared helper at the top of page.js.
export function findCaseForLine(fileName, lineNumber) {
  const regions = readCaseRegions(fileName)
  const region = regions.find((item) => lineNumber >= item.firstLine && lineNumber <= item.lastLine)
  return region ? region.id : null
}

==== File 8 of 11: checker.mjs ====
// checker.mjs: a deliberately small pattern checker, written to show how
// pattern matching goes wrong. It reads each line of the suite as plain
// text. It builds no syntax tree, so it cannot tell code from a comment, and
// it follows no data from one line to the next. It is not a static analyser:
// do not use it on real code.
import fs from 'node:fs'
import path from 'node:path'
import { SUITE_DIR } from './cases.mjs'

const READS_REQUEST = /\brequest\./
const JOINS_STRINGS = /\+/

const RULES = [
  {
    id: 'sql-built-from-request',
    description: 'An SQL statement is joined with a value read from the request.',
    help: 'Pass the value as a query parameter. Choose identifiers such as column names from a fixed list.',
    matches: (line) => /['"][^'"]*\b(SELECT|INSERT|UPDATE|DELETE)\b/.test(line) && JOINS_STRINGS.test(line) && READS_REQUEST.test(line),
  },
  {
    id: 'command-built-from-request',
    description: 'A command line is joined with a value read from the request.',
    help: 'Start the program without a shell, pass each value as its own argument, and allow-list the value.',
    matches: (line) => /['"][a-z][\w.-]* -[a-z]/.test(line) && JOINS_STRINGS.test(line) && READS_REQUEST.test(line),
  },
  {
    id: 'path-built-from-request',
    description: 'A file path is built from a value read from the request.',
    help: 'Map the value to a known file, or allow-list it, then resolve the path and check it stays inside the folder.',
    matches: (line) => /\bpath\.(join|resolve)\(/.test(line) && READS_REQUEST.test(line),
  },
  {
    id: 'html-built-from-request',
    description: 'HTML is joined with a value read from the request without an escape function.',
    help: 'Encode the value for the place it lands, or let the template engine do it.',
    matches: (line) => /['"][^'"]*<[a-z]/i.test(line) && JOINS_STRINGS.test(line) && READS_REQUEST.test(line) && !/\bescape\w*\(\s*request\./.test(line),
  },
  {
    id: 'credential-in-code',
    description: 'A value that looks like a credential is written into the code.',
    help: 'Read the credential from a secrets manager or the environment at run time, and rotate the one in the code.',
    matches: (line) => /\b\w*(password|secret|token|api_?key)\w*['"]?\s*[:=]\s*['"][^'"]{8,}['"]/i.test(line),
  },
  {
    id: 'weak-random',
    description: 'Math.random() is used.',
    help: 'For tokens and other values that must not be guessed, use crypto.randomBytes() or crypto.getRandomValues().',
    matches: (line) => /\bMath\.random\(/.test(line),
  },
]

// LAB_RULE_CHANGE=skip-order-by imitates a tuning change: someone tells the
// SQL rule to ignore ORDER BY lines to silence one false positive.
function readActiveRules() {
  if (process.env.LAB_RULE_CHANGE !== 'skip-order-by') return RULES
  return RULES.map((rule) => {
    if (rule.id !== 'sql-built-from-request') return rule
    return { ...rule, matches: (line) => rule.matches(line) && !line.includes('ORDER BY') }
  })
}

// Returns one finding per rule per matching line, in file and line order.
export function scanSuite() {
  const rules = readActiveRules()
  const fileNames = fs.readdirSync(SUITE_DIR).filter((name) => name.endsWith('.js')).sort()
  const findings = []
  for (const fileName of fileNames) {
    const lines = fs.readFileSync(path.join(SUITE_DIR, fileName), 'utf8').split('\n')
    lines.forEach((text, index) => {
      for (const rule of rules) {
        if (rule.matches(text)) findings.push({ ruleId: rule.id, file: fileName, line: index + 1 })
      }
    })
  }
  return findings
}

// Wraps the findings in a minimal SARIF 2.1.0 log, the same format the
// scorecard script reads from any other tool.
export function toSarif(findings) {
  const rules = readActiveRules().map((rule) => ({
    id: rule.id,
    shortDescription: { text: rule.description },
    help: { text: rule.help },
  }))
  const results = findings.map((finding) => ({
    ruleId: finding.ruleId,
    level: 'warning',
    message: { text: rules.find((rule) => rule.id === finding.ruleId).shortDescription.text },
    locations: [{
      physicalLocation: {
        artifactLocation: { uri: 'suite/' + finding.file },
        region: { startLine: finding.line },
      },
    }],
  }))
  return {
    version: '2.1.0',
    runs: [{ tool: { driver: { name: 'lab-pattern-checker', version: '1.0.0', rules } }, results }],
  }
}

==== File 9 of 11: run-checker.mjs ====
// run-checker.mjs: runs the pattern checker over the suite, prints each
// finding and writes checker.sarif for the scorecard script.
import fs from 'node:fs'
import path from 'node:path'
import { LAB_DIR } from './cases.mjs'
import { scanSuite, toSarif } from './checker.mjs'

const findings = scanSuite()
for (const finding of findings) {
  console.log('suite/' + finding.file + ':' + finding.line + '  ' + finding.ruleId)
}

fs.writeFileSync(path.join(LAB_DIR, 'checker.sarif'), JSON.stringify(toSarif(findings), null, 2) + '\n')
console.log(findings.length + ' findings written to checker.sarif')

==== File 10 of 11: score.mjs ====
// score.mjs: fills the scorecard from a SARIF 2.1.0 file, written by the
// lab checker or exported by any tool you are trialling.
// Usage: node score.mjs path/to/results.sarif
import fs from 'node:fs'
import path from 'node:path'
import { CASES, findCaseForLine } from './cases.mjs'

const SUITE_FILES = new Set(CASES.map((testCase) => testCase.file))

// Rules can sit on the driver or on an extension, depending on the tool.
function listRules(run) {
  const components = [run.tool.driver, ...(run.tool.extensions || [])]
  const rules = new Map()
  for (const component of components) {
    for (const rule of component.rules || []) rules.set(rule.id, rule)
  }
  return rules
}

// Maps a result to a suite file and line, or returns null.
function locateResult(result) {
  const location = (result.locations || [])[0]
  const physical = location && location.physicalLocation
  if (!physical || !physical.region || !physical.artifactLocation) return null
  const fileName = path.basename(decodeURIComponent(physical.artifactLocation.uri || ''))
  if (!SUITE_FILES.has(fileName)) return null
  return { fileName, line: physical.region.startLine }
}

function describeResult(result, rules) {
  const ruleId = result.ruleId || (result.rule && result.rule.id) || 'no-rule-id'
  const rule = rules.get(ruleId) || {}
  const description = rule.fullDescription || rule.shortDescription
  return {
    ruleId,
    hasPath: Array.isArray(result.codeFlows) && result.codeFlows.length > 0,
    explains: Boolean(result.message && result.message.text && description),
    hasFix: Boolean(rule.help || rule.helpUri || (result.fixes && result.fixes.length)),
  }
}

// Returns every result that lands inside a case, tagged with the case id.
function collectResults(sarif) {
  const placed = []
  let outside = 0
  for (const run of sarif.runs || []) {
    const rules = listRules(run)
    for (const result of run.results || []) {
      const spot = locateResult(result)
      const caseId = spot && spot.line ? findCaseForLine(spot.fileName, spot.line) : null
      if (!caseId) {
        outside = outside + 1
        continue
      }
      placed.push({ caseId, line: spot.line, ...describeResult(result, rules) })
    }
  }
  return { placed, outside }
}

function scoreCase(testCase, placed) {
  const results = placed.filter((item) => item.caseId === testCase.id)
  const reported = results.length > 0
  let outcome = 'TN'
  if (testCase.flawed && reported) outcome = 'TP'
  if (testCase.flawed && !reported) outcome = 'FN'
  if (!testCase.flawed && reported) outcome = 'FP'
  return { testCase, results, reported, outcome }
}

function printCaseTable(rows) {
  console.log('Case       CWE       Expected     Reported  Outcome  Lines   Rules')
  for (const row of rows) {
    const lines = [...new Set(row.results.map((item) => item.line))].join(',') || '-'
    const ruleIds = [...new Set(row.results.map((item) => item.ruleId))].join(', ') || '-'
    console.log(
      row.testCase.id.padEnd(11) + row.testCase.cwe.padEnd(10) +
      (row.testCase.flawed ? 'finding' : 'no finding').padEnd(13) +
      (row.reported ? 'yes' : 'no').padEnd(10) + row.outcome.padEnd(9) +
      lines.padEnd(8) + ruleIds,
    )
  }
}

function countOutcome(rows, outcome) {
  return rows.filter((row) => row.outcome === outcome).length
}

function printRates(rows) {
  const truePositives = countOutcome(rows, 'TP')
  const falseNegatives = countOutcome(rows, 'FN')
  const trueNegatives = countOutcome(rows, 'TN')
  const falsePositives = countOutcome(rows, 'FP')
  const truePositiveRate = truePositives / (truePositives + falseNegatives)
  const falsePositiveRate = falsePositives / (falsePositives + trueNegatives)
  console.log('Totals: TP ' + truePositives + ', FN ' + falseNegatives + ', TN ' + trueNegatives + ', FP ' + falsePositives)
  console.log('True positive rate, TP / (TP + FN): ' + (truePositiveRate * 100).toFixed(1) + '%')
  console.log('False positive rate, FP / (FP + TN): ' + (falsePositiveRate * 100).toFixed(1) + '%')
  console.log('Benchmark-style score, (TPR - FPR) x 100: ' + ((truePositiveRate - falsePositiveRate) * 100).toFixed(1))
}

function printCoverage(rows) {
  console.log('Rule coverage, flawed cases reported per weakness class:')
  const classes = [...new Set(rows.map((row) => row.testCase.cwe))]
  for (const cwe of classes) {
    const flawed = rows.filter((row) => row.testCase.cwe === cwe && row.testCase.flawed)
    const found = flawed.filter((row) => row.reported)
    console.log('  ' + cwe.padEnd(9) + found.length + ' of ' + flawed.length)
  }
}

function printDataFlowAndRemediation(rows) {
  const dataFlowRows = rows.filter((row) => row.testCase.dataFlow)
  const dataFlowFound = dataFlowRows.filter((row) => row.reported)
  console.log('Data-flow cases reported: ' + dataFlowFound.length + ' of ' + dataFlowRows.length +
    ' (' + dataFlowRows.map((row) => row.testCase.id).join(', ') + ')')
  const truePositiveResults = rows.filter((row) => row.outcome === 'TP').flatMap((row) => row.results)
  const total = truePositiveResults.length
  const withPath = truePositiveResults.filter((item) => item.hasPath).length
  const explaining = truePositiveResults.filter((item) => item.explains).length
  const withFix = truePositiveResults.filter((item) => item.hasFix).length
  console.log('Results on flawed cases with a source-to-sink path (codeFlows): ' + withPath + ' of ' + total)
  console.log('Results on flawed cases with a message and a rule description: ' + explaining + ' of ' + total)
  console.log('Results on flawed cases with fix guidance (help, helpUri or fixes): ' + withFix + ' of ' + total)
}

const sarifPath = process.argv[2]
if (!sarifPath) {
  console.log('Usage: node score.mjs path/to/results.sarif')
  process.exit(1)
}
const sarif = JSON.parse(fs.readFileSync(sarifPath, 'utf8'))
const { placed, outside } = collectResults(sarif)
const rows = CASES.map((testCase) => scoreCase(testCase, placed))
const toolNames = (sarif.runs || []).map((run) => run.tool.driver.name + ' ' + (run.tool.driver.version || '')).join(', ')

console.log('Scorecard for ' + toolNames.trim() + ', read from ' + path.basename(sarifPath))
console.log('')
printCaseTable(rows)
console.log('')
printRates(rows)
console.log('')
printCoverage(rows)
console.log('')
printDataFlowAndRemediation(rows)
console.log('Results outside every case: ' + outside)

==== File 11 of 11: checker.test.mjs ====
// checker.test.mjs: pins what the checker reports for every case, so any
// change to a rule shows up as a named failure instead of a quiet shift in
// the numbers. The pins record behaviour, not correctness: three safe cases
// are pinned as reported (known false positives) and five flawed cases as
// not reported (known false negatives).
import assert from 'node:assert/strict'
import { test } from 'node:test'
import { CASES, findCaseForLine } from './cases.mjs'
import { scanSuite } from './checker.mjs'

const PINNED_AS_REPORTED = new Set([
  'SQL-1F', 'SQL-3F', 'SQL-3S',
  'CMD-1F', 'CMD-1S',
  'PATH-1F', 'PATH-2S',
  'XSS-1F',
  'SECRET-1F',
  'RAND-1F',
])

const findings = scanSuite()
const reportedCases = new Set(findings.map((finding) => findCaseForLine(finding.file, finding.line)))

function describePin(testCase, reported) {
  const word = reported ? 'reported' : 'not reported'
  if (reported && !testCase.flawed) return word + ' (known false positive)'
  if (!reported && testCase.flawed) return word + ' (known false negative)'
  return word
}

for (const testCase of CASES) {
  const pinned = PINNED_AS_REPORTED.has(testCase.id)
  test(testCase.id + ' is ' + describePin(testCase, pinned), () => {
    const reported = reportedCases.has(testCase.id)
    assert.equal(reported, pinned, testCase.id + ' was pinned as ' + describePin(testCase, pinned) + ' but is now ' + describePin(testCase, reported))
  })
}

test('every finding lands inside a case', () => {
  assert.equal(reportedCases.has(null), false)
})

==== RUN IT ====
node run-checker.mjs
node score.mjs checker.sarif
node --test checker.test.mjs
LAB_RULE_CHANGE=skip-order-by node --test checker.test.mjs

==== OUTPUT OF node run-checker.mjs (Node.js 26.0.0, macOS) ====
suite/command.js:6  command-built-from-request
suite/command.js:13  command-built-from-request
suite/credential.js:6  credential-in-code
suite/page.js:17  html-built-from-request
suite/path.js:9  path-built-from-request
suite/path.js:30  path-built-from-request
suite/random.js:6  weak-random
suite/sql.js:6  sql-built-from-request
suite/sql.js:37  sql-built-from-request
suite/sql.js:44  sql-built-from-request
10 findings written to checker.sarif

==== OUTPUT OF node score.mjs checker.sarif ====
Scorecard for lab-pattern-checker 1.0.0, read from checker.sarif

Case       CWE       Expected     Reported  Outcome  Lines   Rules
SQL-1F     CWE-89    finding      yes       TP       6       sql-built-from-request
SQL-1S     CWE-89    no finding   no        TN       -       -
SQL-2F     CWE-89    finding      no        FN       -       -
SQL-2S     CWE-89    no finding   no        TN       -       -
SQL-3F     CWE-89    finding      yes       TP       37      sql-built-from-request
SQL-3S     CWE-89    no finding   yes       FP       44      sql-built-from-request
CMD-1F     CWE-78    finding      yes       TP       6       command-built-from-request
CMD-1S     CWE-78    no finding   yes       FP       13      command-built-from-request
CMD-2F     CWE-78    finding      no        FN       -       -
CMD-2S     CWE-78    no finding   no        TN       -       -
PATH-1F    CWE-22    finding      yes       TP       9       path-built-from-request
PATH-1S    CWE-22    no finding   no        TN       -       -
PATH-2F    CWE-22    finding      no        FN       -       -
PATH-2S    CWE-22    no finding   yes       FP       30      path-built-from-request
XSS-1F     CWE-79    finding      yes       TP       17      html-built-from-request
XSS-1S     CWE-79    no finding   no        TN       -       -
XSS-2F     CWE-79    finding      no        FN       -       -
XSS-2S     CWE-79    no finding   no        TN       -       -
SECRET-1F  CWE-798   finding      yes       TP       6       credential-in-code
SECRET-1S  CWE-798   no finding   no        TN       -       -
SECRET-2F  CWE-798   finding      no        FN       -       -
SECRET-2S  CWE-798   no finding   no        TN       -       -
RAND-1F    CWE-338   finding      yes       TP       6       weak-random
RAND-1S    CWE-338   no finding   no        TN       -       -

Totals: TP 7, FN 5, TN 9, FP 3
True positive rate, TP / (TP + FN): 58.3%
False positive rate, FP / (FP + TN): 25.0%
Benchmark-style score, (TPR - FPR) x 100: 33.3

Rule coverage, flawed cases reported per weakness class:
  CWE-89   2 of 3
  CWE-78   1 of 2
  CWE-22   1 of 2
  CWE-79   1 of 2
  CWE-798  1 of 2
  CWE-338  1 of 1

Data-flow cases reported: 0 of 2 (SQL-2F, CMD-2F)
Results on flawed cases with a source-to-sink path (codeFlows): 0 of 7
Results on flawed cases with a message and a rule description: 7 of 7
Results on flawed cases with fix guidance (help, helpUri or fixes): 7 of 7
Results outside every case: 0

==== OUTPUT OF node --test checker.test.mjs ====
✔ SQL-1F is reported (0.701584ms)
✔ SQL-1S is not reported (0.044125ms)
✔ SQL-2F is not reported (known false negative) (0.044709ms)
✔ SQL-2S is not reported (0.0525ms)
✔ SQL-3F is reported (0.043541ms)
✔ SQL-3S is reported (known false positive) (0.036709ms)
✔ CMD-1F is reported (0.032334ms)
✔ CMD-1S is reported (known false positive) (0.029ms)
✔ CMD-2F is not reported (known false negative) (0.047ms)
✔ CMD-2S is not reported (0.082375ms)
✔ PATH-1F is reported (0.042042ms)
✔ PATH-1S is not reported (0.031292ms)
✔ PATH-2F is not reported (known false negative) (0.033167ms)
✔ PATH-2S is reported (known false positive) (0.020458ms)
✔ XSS-1F is reported (0.027542ms)
✔ XSS-1S is not reported (0.028375ms)
✔ XSS-2F is not reported (known false negative) (0.019875ms)
✔ XSS-2S is not reported (0.030042ms)
✔ SECRET-1F is reported (0.017459ms)
✔ SECRET-1S is not reported (0.018667ms)
✔ SECRET-2F is not reported (known false negative) (0.017333ms)
✔ SECRET-2S is not reported (0.037791ms)
✔ RAND-1F is reported (0.031625ms)
✔ RAND-1S is not reported (0.022ms)
✔ every finding lands inside a case (0.02825ms)
ℹ tests 25
ℹ suites 0
ℹ pass 25
ℹ fail 0
ℹ cancelled 0
ℹ skipped 0
ℹ todo 0
ℹ duration_ms 85.229458

The command exited with code 0.

==== OUTPUT OF LAB_RULE_CHANGE=skip-order-by node --test checker.test.mjs ====
This run imitates a tuning change: the SQL rule is told to ignore lines containing ORDER BY, to silence the false positive on SQL-3S. Lines are cut where marked.

(4 lines for passing tests cut here)
✖ SQL-3F is reported (0.634667ms)
✖ SQL-3S is reported (known false positive) (0.065417ms)
(19 lines for passing tests cut here)
ℹ tests 25
ℹ suites 0
ℹ pass 23
ℹ fail 2
ℹ cancelled 0
ℹ skipped 0
ℹ todo 0
ℹ duration_ms 89.734375

✖ failing tests:

test at checker.test.mjs:32:3
✖ SQL-3F is reported (0.634667ms)
  AssertionError [ERR_ASSERTION]: SQL-3F was pinned as reported but is now not reported (known false negative)
  
  false !== true
  
  (14 lines cut here: a stack trace whose first line shows a local file path, and the error's property list)

test at checker.test.mjs:32:3
✖ SQL-3S is reported (known false positive) (0.065417ms)
  AssertionError [ERR_ASSERTION]: SQL-3S was pinned as reported (known false positive) but is now not reported
  
  false !== true
  
  (14 lines cut here: a stack trace whose first line shows a local file path, and the error's property list)

The command exited with code 1.

==== THE SCORECARD: FILL IN ONE PER CANDIDATE TOOL ====
score.mjs fills the first five columns and the rates from a SARIF file. The last four columns need a person who has read each result.

Tool and version:
Rules or profile used:
Command run, and the folder it was pointed at:
Date:

Case | CWE | Expected | Reported | Outcome | Line given | Explains the weakness | Says how to fix | Notes
SQL-1F | CWE-89 | finding |  |  |  |  |  | 
SQL-1S | CWE-89 | no finding |  |  |  |  |  | 
SQL-2F | CWE-89 | finding |  |  |  |  |  | 
SQL-2S | CWE-89 | no finding |  |  |  |  |  | 
SQL-3F | CWE-89 | finding |  |  |  |  |  | 
SQL-3S | CWE-89 | no finding |  |  |  |  |  | 
CMD-1F | CWE-78 | finding |  |  |  |  |  | 
CMD-1S | CWE-78 | no finding |  |  |  |  |  | 
CMD-2F | CWE-78 | finding |  |  |  |  |  | 
CMD-2S | CWE-78 | no finding |  |  |  |  |  | 
PATH-1F | CWE-22 | finding |  |  |  |  |  | 
PATH-1S | CWE-22 | no finding |  |  |  |  |  | 
PATH-2F | CWE-22 | finding |  |  |  |  |  | 
PATH-2S | CWE-22 | no finding |  |  |  |  |  | 
XSS-1F | CWE-79 | finding |  |  |  |  |  | 
XSS-1S | CWE-79 | no finding |  |  |  |  |  | 
XSS-2F | CWE-79 | finding |  |  |  |  |  | 
XSS-2S | CWE-79 | no finding |  |  |  |  |  | 
SECRET-1F | CWE-798 | finding |  |  |  |  |  | 
SECRET-1S | CWE-798 | no finding |  |  |  |  |  | 
SECRET-2F | CWE-798 | finding |  |  |  |  |  | 
SECRET-2S | CWE-798 | no finding |  |  |  |  |  | 
RAND-1F | CWE-338 | finding |  |  |  |  |  | 
RAND-1S | CWE-338 | no finding |  |  |  |  |  | 

Totals: TP ___  FN ___  TN ___  FP ___
True positive rate, TP / (TP + FN): ___
False positive rate, FP / (FP + TN): ___
Benchmark-style score, (TPR - FPR) x 100: ___
Rule coverage, flawed cases reported: CWE-89 ___ of 3, CWE-78 ___ of 2, CWE-22 ___ of 2, CWE-79 ___ of 2, CWE-798 ___ of 2, CWE-338 ___ of 1
Data-flow cases reported (SQL-2F, CMD-2F): ___ of 2
Results on flawed cases that show a source-to-sink path: ___
Remediation marks on true positives (line, explains, fix): ___ of ___
Results outside every case, and what they were:
Would this tool's results block a merge in our pipeline? Which rules, and why:

==== WHAT EACH COLUMN MEANS ====
Expected: whether the case should be reported, taken from flawed in cases.mjs. Flawed cases expect a finding; safe cases expect none.

Reported: yes when at least one result's first location falls between the case's marker comment and the next marker. A result with no line, or in a file outside the suite, is counted under "Results outside every case" rather than given to a case.

Outcome: OWASP Benchmark's four outcomes, with "fake vulnerability" read as a safe case. TP, a flawed case the tool reported. FN, a flawed case it did not report. TN, a safe case it did not report. FP, a safe case it reported.

True positive rate and false positive rate: as OWASP Benchmark defines them, TP / (TP + FN) and FP / (FP + TN).

Benchmark-style score: OWASP Benchmark's accuracy score, TPR minus FPR, scaled to 0 to 100. OWASP describes it as essentially a Youden index, and notes it can be negative when the false positive rate is higher than the true positive rate. This suite has twelve flawed and twelve safe cases, so one case moves either rate by more than eight points. Treat the score as a summary of this suite, not a rating of the tool.

Rule coverage (this page's own measure): for each weakness class, how many of its flawed cases were reported. A class with none reported means the tool, as configured, reported nothing for that class on code shaped like these samples.

False positives and false negatives: the FP and FN rows by id. Read each one's why in cases.mjs before deciding whether the behaviour matters for your code.

Data-flow findings (this page's own measure): how many of the two cases marked dataFlow were reported, and how many results on flawed cases carry a SARIF codeFlows path from where the value enters to where it is used.

Remediation quality (this page's own measure, built on what SARIF 2.1.0 section 3.27.11 asks a result's message to contain): three yes or no marks for each true positive.
- Line given: the result points at the line where the value is used, not just somewhere in the file.
- Explains the weakness: the message or the rule description says what condition was found and what could happen if it is left.
- Says how to fix: the message, the rule's help or a proposed fix says what to change.
score.mjs counts which results carry a message with a rule description, and which carry fix guidance. Whether the text is right is for you to judge.

==== TRIALLING A REAL TOOL ====
1. Point the candidate at the suite folder only, with its default rules, and export SARIF 2.1.0. Scanning the whole lab folder adds results for the lab's own scripts, which score.mjs counts as outside every case.
2. Run node score.mjs with the path to that file, and copy the totals into the scorecard.
3. Read the message on every result, then fill in the last four columns.
4. A tool that reports a flaw only once the value reaches a call it recognises, such as a database driver's query method, may report little on the injection cases, because no sample makes that call. Note that in the scorecard, and run a second trial on code of your own where the calls are real before you read silence as a miss.
5. Add cases from your own code: the helper your team uses to build queries, the function that encodes output, the wrapper around your HTTP client. Give each a marker comment and an entry in cases.mjs, so the next tool is measured on the code you actually write.
6. Keep the scorecard with the tool's version and rules. Score again when either changes, and compare.

What SAST is and what it reads

Static application security testing (SAST) looks for security weaknesses in an application's code without running the application.

OWASP describes source code analysis tools, also known as SAST tools, as tools that can help analyse source code or compiled versions of code to help find security flaws. NIST's NISTIR 8397 says that, although there are hybrids, analysis may generally be divided into code-based or static analysis, such as SAST, and execution-based or dynamic analysis, such as DAST, and that pure code-based analysis is independent of program execution.

Not running the program is what gives SAST its timing. NIST says static source code analysis should be done as soon as code is written, and that small pieces of code can be checked before large executable pieces are complete. The strengths OWASP lists for SAST tools include that they scale well, that they can be run repeatedly, as with nightly builds or continuous integration, and that their output highlights the problematic code by file name, location, line number and even the affected code snippet.

The same fact sets its limit. A static tool reads the files it is given. It does not see the server the code runs on, the settings in its environment, the records in its database or who is signed in. What is written in the code is in its reach, and what happens only when the code runs is not.

The SAST versus DAST comparison on this site sets out how static and dynamic testing divide the work; this guide stays with static analysis itself.

Static code analysis for security: matching a pattern or following the data

NIST describes static scanners as ranging from simple searches for any use of a deprecated function to tools that can verify a piece of code faithfully implements a communication protocol, with pattern searches for possible vulnerabilities in between.

For injection flaws in a web application the useful question is narrower: does a rule judge the shape of one line, or does it follow a value from where it enters the program to where it is used?

Patterns: text and structure

The simplest rule is a text pattern: report any line that calls Math.random(), or any string that starts with SELECT and is joined to something read from the request. It is quick to write and quick to run, and it judges each line on its own. It cannot tell code from a comment, and it cannot see that the value on this line was checked two lines earlier.

A tool that parses the code first works on its structure rather than its characters. It knows which text is a comment, which is a string and which is a function call, so a rule can ask for a call with a particular kind of argument and ignore the same words in a comment.

For interpreted languages, NIST says you may use a static analyser, sometimes called a linter, which checks for dangerous functions, problematic parameters and other possible vulnerabilities.

A structural rule still judges the line in front of it. That is enough for a call that is risky wherever it appears, and not enough for a value that is risky only because of where it came from.

Data flow: from source to sink

In its entry for weak random number generators, CWE-338, MITRE describes how SAST finds such weaknesses: by analysing source or compiled code without executing it, typically by building a model of data flow and control flow and then searching for potentially vulnerable patterns that connect sources, the origins of input, with sinks, the destinations where the data interacts with external components, a lower layer such as the operating system, and so on. It says SAST can find some instances of that weakness this way.

In a web application the sources are whatever the request carries: the query string, form fields, headers, cookies and the body. The sinks are the places where a value changes what happens next: an SQL statement, a command line, a file path, a page. Between the two, the value may be copied into a variable, passed to a helper, joined to other text or checked against a list.

A data-flow rule follows it through each step and asks whether anything along the way, such as a query parameter or an encoder, makes it fit for the sink it reaches.

CWE's entries for SQL injection and OS command injection say many modern tools use data flow analysis or constraint-based techniques to minimise the number of false positives. NIST notes that, besides closed source tools, there are powerful free and open-source tools that provide extensive analyst aids, such as the control flows and data values that lead to a violation.

Two cases in the fixture suite test exactly this. In SQL-2F a customer id passes through a variable and a helper function before it is joined into a query. In CMD-2F a folder name passes through two variables before it reaches a command line. Judged one line at a time, every line in both looks harmless.

SAST false positives and false negatives: why tools get it wrong

Every static result is a judgement about what code will do, made without running it, so some judgements are wrong in each direction.

NIST says that in almost all tools some warnings are false positives and some are insignificant in the context of the software, and that tools usually cannot determine whether a weakness is an actual vulnerability or is mitigated in the application. OWASP lists high numbers of false positives among the weaknesses of SAST tools, along with the difficulty of proving that an identified issue is an actual vulnerability.

False positives: safe code that looks dangerous

A false positive is a result on code that is not vulnerable. 1 min read

False negatives: flawed code that passes

A false negative is a flaw the tool does not report. 1 min read

How to judge SAST tools: trial one on code you understand

This guide does not recommend or rank tools.

What a tool reports depends on the language, the frameworks, the helpers your team writes and the rules it runs with, so the useful measure is how it behaves on code like yours, scored against answers you already know.

OWASP's selection criteria for source code analysis tools start from a prerequisite, support for your programming language, and include the ability to understand the libraries and frameworks you need, accuracy in terms of false positive and false negative rates and the OWASP Benchmark score, the ability to be included in continuous integration and deployment tools, and interoperability of output through OASIS SARIF.

Public suites exist for this. OWASP Benchmark is a Java and Python test suite designed to verify the speed and accuracy of vulnerability detection tools; each of its test cases has a single intentional CWE that is either a true or a false positive, with the expected results documented in a file, expectedresults-1.2.csv.

NIST's SAMATE project, whose major efforts include collecting a corpus of programs with known bugs and enabling better understanding of tool effectiveness, drives the Software Assurance Reference Dataset (SARD): a growing collection of test programs with documented weaknesses, written in C, C++, Java, PHP and C#, covering over 150 classes of weaknesses.

Neither lists JavaScript. OWASP also says the majority of Benchmark's tests are considerably simpler than real applications, and that most real-world applications will be considerably harder to analyse successfully.

The fixture suite on this page is built for that gap: twelve pairs of flawed and safe JavaScript samples across six weakness classes, an answer key, and a script that fills in the scorecard from any tool's SARIF output. A trial then runs in this order.

  • Run the candidate over the suite with its default rules, export SARIF and score it. Read the reason given for every false positive and false negative before deciding whether it matters for your code.
  • Read the message on every true positive. A result that gives the line, says what is wrong and says what to change is one a developer can act on without a meeting.
  • Run the candidate over one repository you know well, and triage a sample of its results with the people who wrote that code. Count how many you would act on, and note which rules produced the rest.
  • Close the gaps you found. NIST says many analysers allow users to write rules or patterns to increase the analyser's benefit, and that you can realise the maximum benefit by running more than one analyser and paying attention to the weakness classes for which each scanner is best. The coverage rows on the scorecard show where each tool stands by class.
  • Add your own code to the suite: the helper your team uses to build queries, the function that encodes output, the wrapper around your HTTP client. The next tool is then measured on the code you actually write.

One limit of the suite

Each flawed injection sample stops where its CWE entry places the weakness: it builds a query, a command, a path or a piece of a page from request input and returns it. No sample hands that value to a database driver, a process launcher, the file system or a response.

A tool that reports a flaw only once the value reaches a call it recognises may report little here, so record that on the scorecard, and repeat the trial on code of your own where the calls are real before reading the silence as a miss.

What a useful finding contains, and where SARIF fits

SARIF, the Static Analysis Results Interchange Format, is an OASIS specification that defines a standard format for the output of static analysis tools.

The current text is Version 2.1.0 Plus Errata 01, an OASIS Standard incorporating Approved Errata, dated 28 August 2023. The lab's scorecard script reads SARIF, so any tool that exports it can be scored against the suite.

The specification also describes what a result should tell the person reading it. A result's message should provide sufficient details to allow an end user to resolve any problem that the result might indicate, and it SHALL include all of the following information that is available and relevant to the result: enough to identify the analysis target and the location within the target where the problem occurred, the condition within the target that led to the problem being reported, the risks potentially associated with not fixing the problem, and the full range of responses the end user could take, including the conditions where it might be appropriate not to fix the problem or to conclude that the result is a false positive.

The scorecard's remediation columns ask three questions of every true positive: does it give the line, does it explain the weakness, and does it say how to fix it.

Two more parts of a result are worth checking in a trial. The codeFlows property is intended for tools that provide execution path details illustrating a possible problem; a data-flow finding that carries one shows the reader the route from source to sink, not just the last line. The fixes property carries proposed fixes, each naming the files to change, the regions to remove and the new content to insert.

SAST in CI: a baseline first, then a short blocking list

NIST recommends automated verification to ensure that static analysis does not report new weaknesses, and says that because it is automated it can be repeated often, for instance on every commit.

NIST's Secure Software Development Framework, SP 800-218 version 1.1, gives as one example for its task PW.7.2 using a static analysis tool to automatically check code for vulnerabilities and compliance with the organisation's secure coding standards, with a human reviewing the issues the tool reports and remediating them as necessary.

The task itself asks for all discovered issues and recommended remediations to be recorded and triaged in the development team's workflow or issue tracking system. The planned secure SDLC guide on this site covers the rest of that framework.

Take a baseline before the first gate

The first scan of an existing codebase reports everything its rules can see, written over years. 1 min read

Which findings block a merge, and which are recorded

NIST says organisations should select and standardise on static analysis tools and establish lists of must-fix bugs based on their experience with the tool, the applications under development and reported vulnerabilities, and that published lists of top bugs can help create a process-specific list. 1 min read

Starting the list from a standard

ASVS 5.0 has a requirement for each class in the suite, with a level against each: 1 min read

A requirement is not a rule

Each requirement says what the code must do. 1 min read

What static analysis cannot see

Some flaws leave nothing for a rule to match, because the problem is something the code does not do, or something decided outside the code.

NIST says certain types of vulnerabilities are difficult for SAST to find, such as authentication problems, access control issues and insecure use of cryptography, and OWASP's list of SAST weaknesses names the same three.

  • A missing access check. A route that loads a record by its id and never compares the record's owner with the signed-in user reads like ordinary code: the bug is an absent comparison, and an absence has no shape to match. NIST notes that some properties are poorly suited to computerised recognition and may warrant human examination, and among the checks it suggests someone other than the author makes is that code only allows authorised users to access sensitive transactions, functions and data.
  • A business rule. A discount that can be applied twice, or a refund larger than the order it refunds, breaks a rule nobody wrote into the code. The OWASP Web Security Testing Guide says static source code analysis alone cannot identify issues due to flaws in the design, since it cannot understand the context in which the code is constructed.
  • The deployed configuration. Response headers, cookie flags, hosting rules, environment variables and feature flags change what the application does without appearing in the files a scan reads. OWASP says SAST tools are frequently unable to find configuration issues, since they are not represented in the code, and the WSTG lists among the disadvantages of source code review that the code actually deployed might differ from the code being analysed.
  • Code you did not write. The third-party packages your application depends on are a separate question, answered by software composition analysis, which the dependency security guide covers.

Where the other methods pick up

Testing the running application, covered in the DAST guide, meets the deployed configuration that a source file never holds. A person reading the code for checks that are missing and rules that were never written is the subject of the secure code review page.

Walking through the fixture suite and the pattern checker

The lab on this page is an illustration written for it, not client code.

Its pattern checker reads each line of the suite as plain text and tests it against six regular expressions, one each for SQL, command lines, file paths, HTML, credentials and Math.random(). It builds no syntax tree and follows no data, so it is not a static analyser.

That is the point: its scorecard shows the false positives and false negatives a line-by-line pattern produces, on cases chosen to produce them. The outputs were recorded on Node.js 26.0.0 on macOS.

The six weakness classes in the suite

Each class takes its name and tag from its CWE entry. 2 min read

What the checker found and what it missed

The checker reported ten results, one in each of ten cases. 1 min read

The regression test, and a tuning change that broke it

checker.test.mjs pins the checker's result for every case, its three known false positives and five known false negatives included, plus one check that every result lands inside a case. 1 min read

Where a Clever Ops review fits beside your SAST tool

A SAST tool in the pipeline is worth keeping, and a review does not replace it.

Clever Ops reviews code for applications such as Next.js, React and Node with read-only access to the repository. Code findings are confirmed against a running build where the scope allows, and the report marks any that rest on the code alone.

If your team runs a static analysis tool, send its current results with the scope, so warnings your team has already triaged are recognised as known rather than reported back to you as new. Clever Ops does not sell scan output, or a scanner subscription, as a service of its own.

If Clever Ops built or hosts the application, the work is called a security review rather than an independent test, and the report states who built it and who hosts it.

Frequently asked questions

SAST, static application security testing, is software that reads your application's source or compiled code without running it and reports places that look like known security weaknesses, such as a query built from request input. It can run on every commit and points at a file and a line. It cannot see how the application behaves once deployed, and its results need someone to judge which are real.

Because it judges code without running it. NIST says that in almost all tools some warnings are false positives, and that tools usually cannot determine whether a weakness is mitigated in the application. CWE notes that automated static analysis might not recognise when proper input validation is being performed. Measure false positives rule by rule on code you understand, tune or switch off the rules that are wrong on your code, and check that each tuning change does not also drop real findings.

This guide names none. Start from OWASP's selection criteria, such as support for your language and frameworks, false positive and false negative rates, the ability to run in CI and SARIF output. Then score each candidate on the fixture suite here and on one repository you know well. NIST notes that scanners have different strengths, and that you can realise the maximum benefit by running more than one analyser and paying attention to the weakness classes each is best at.

Not reliably. NIST and OWASP both list access control issues among the vulnerabilities SAST finds difficult. A missing ownership check leaves no dangerous call to match, so it is found by someone reading what the code should check, and by testing the running application with accounts in different roles.

Not on its whole rule set. Record the first full scan as a baseline, gate only on new results, and start the blocking list with the rules your trial showed are reliable on your code. Everything else goes to the issue tracker with its rule, file, line and commit, and every dismissal carries a reason that someone other than its author has accepted.

No. SAST reads the code your team writes. NISTIR 8397 says a software composition analysis (SCA) tool can help you identify what open-source libraries, suites, packages, bundles, kits and so on your software uses. Separately, it says the components of the software must be continually monitored against databases of known vulnerabilities. The two answer different questions, and the dependency security guide covers the second.

It can do part of the job. For interpreted languages, NIST says you may use a static analyser, sometimes called a linter, which checks for dangerous functions, problematic parameters and other possible vulnerabilities. Whether a given linter set-up is enough depends on its rules: score it on the fixture suite like any other candidate and see which classes it covers.

Scope the right test for your application

Tell us what you have built, who uses it and when it needs to launch. You will get a plain-English recommendation: a website audit, a web application test or a referral to a specialist.