Skip to main content
Security guideReviewed

LLM output handling: treat what the model writes as untrusted input

A model's reply is data from outside your application, whoever shaped it.

This guide follows one reply into the five places it can land, HTML, Markdown, URLs, JSON and SQL, shows what each receiving system has to do with it, and gives you a fixture suite that tests each path with harmless markers.

Output-encoding test suite for model output, version 1.0

An illustration, not a client system: an invented landscape supplies yard runs an assistant on its trade portal, and the same synthetic model outputs go to each of its receiving systems. The fixtures: F1 inert markup and an HTML image; F2 an apostrophe and a quote that would close an attribute; F3 a Markdown link and image pointing at tester.example; F4 a link with a javascript: scheme; F5 a string that would change a query if joined into it; F6 a quote action with a string quantity, a discount of 150 and an extra key; F7 a well-formed quote action above the customer's tier; F8 a sort value naming a hidden column; F9 a model-written DELETE; BC a real order note.

How to use it

Test Trade Customer A has a discount tier of up to 10 percent. Replace tester.example with a host you control and keep its access log on. Feed each fixture in place of the model's reply, on a test copy with synthetic records, and record each control column as it happened in the run. Run it only against an application you own or are authorised in writing to test.

18 rows

Output-encoding test suite for model output, version 1.0

CaseSynthetic model outputDestination and useValidationContextual encodingSchema checkPerson's approvalExpected resultEvidence to keep
H1. HTML element: the yard office screenF1: Your quote is ready. <u>OH-F1-MARK</u> <img src="https://tester.example/oh-f1.png" alt="OH-F1">The assistant's summary of a conversation, shown inside a panel on the staff screen.Length limit only. A summary is free text, so there is no allow list to apply.HTML encoding by the template at render time, or insertion with a text API such as textContent. No innerHTML or dangerouslySetInnerHTML on this path.None: free text.None: display only.The panel shows the tags as literal characters. No underlined text, no img element in the panel's HTML, and no request for oh-f1.png in the tester.example log.Screenshot, the panel's HTML copied from the browser's developer tools, and the tester.example access log for the run window.
H2. HTML attribute: the customer chipF2: O'Brien Landscapes" data-oh-f2="xThe customer name the assistant extracted, placed in a quoted title attribute on the customer chip in the yard office screen.Prefer the name from the signed-in account record. Where the model's version is shown, cap its length.HTML attribute encoding, with the value quoted and the attribute name fixed in the template.None: free text.None: display only.Hovering shows the whole string, quotes included, as the title. The chip has no data-oh-f2 attribute.The chip element's HTML from developer tools, showing one title attribute and no new attributes.
H3. HTML email: the order confirmationF1, as the body text of a confirmation the assistant drafts.Staff approve the draft, then it is sent as an HTML email to the test customer's inbox.Every link in the draft passes the U2 checks before the draft reaches the approval queue.The model supplies text only. A fixed template your code owns holds all markup and HTML-encodes the text into it.The draft has fixed fields, such as greeting, body and order lines, each a string with a length limit.A staff member approves. The approval screen shows the plain text and the full address of every link, not a rendered preview.The received email shows the tags as text, opening it adds no request to the tester.example log, and the approver saw the same text that was sent.The raw source of the received email, the approval record with approver and time, and the tester.example log.
M1. Markdown: raw HTML in a chat replyF1, returned as the assistant's chat reply.The chat widget renders the reply as Markdown for the signed-in trade customer.None at this step. The renderer's settings carry the control.Raw HTML escaped or skipped by the Markdown renderer. If the renderer outputs an HTML string, as Marked does, a maintained sanitiser processes it before insertion.None: free text.None: display only.The tags appear as text or not at all. The reply's HTML holds no u or img element from the fixture, and the tester.example log shows no request for oh-f1.png.Screenshot, the rendered HTML of the reply, and the renderer's name, version and options.
M2. Markdown: link and image to an outside hostF3: Track it [here](https://tester.example/oh-f3-link) ![map](https://tester.example/oh-f3.png)The same chat reply path as M1.Link and image addresses checked against the hosts the portal uses, through the renderer's URL transform or a custom link and image component.The renderer encodes the link text. An address that fails the host check is dropped, or shown as plain text with the full address visible.None: free text.None: display only.No clickable link to tester.example and no image element pointing there. The tester.example log shows no request for oh-f3.png when the reply renders.The rendered HTML with every href and src, the browser's network log, the page's Content-Security-Policy header and the tester.example log.
M3. Markdown: a link with a script schemeF4: [Open your account](javascript:void(0))The same chat reply path as M1.Scheme allow list: https, plus mailto if replies need it. Note which schemes your renderer's default transform keeps.The renderer encodes the link text.None: free text.None: display only.No rendered anchor carries a javascript: address: the href is removed or replaced, or the link shows as text. Inspect the anchor; do not click it.The rendered HTML of the anchor, and the renderer's URL transform setting.
U1. URL component: a search linkF2, then BC, as the search term.The reply offers a "See all results" link that your code builds to the portal's search page, with the model's term in the q parameter.Length limit on the term.encodeURIComponent() on the term, then HTML attribute encoding of the finished address in the href.None: free text.None: display only.The search page receives the whole term in q, including everything after the ampersand in BC. The anchor has no data-oh-f2 attribute.The generated href, the query string the search page received from its server log, and the anchor's HTML.
U2. Whole URL: a product link the model suppliesF3's link address, https://tester.example/oh-f3-link, in the productUrl field of a product card.The assistant returns product cards, each with a name and a link, which the widget renders.Parse the value as a URL and require https and a hostname on your list, for example with Zod's z.url() and its protocol and hostname options. Better: build the link from the SKU and ignore any address the model writes.HTML attribute encoding of an accepted address in the href.productUrl typed as a URL with the host rule above.None: display only.The card is rejected or rendered without a link. No anchor in the widget points at tester.example.The validation log entry naming the rejected host, and the rendered card's HTML.
U3. Whole URL: a script schemeF4's address, javascript:void(0), in the productUrl field.The same product card path as U2.Scheme allow list of https only. ASVS 5.0 requirement 1.2.2 gives javascript: and data: as protocols to disallow.Not reached: the value is rejected first.productUrl typed as a URL with an https protocol rule.None: display only.The card is rejected or rendered without a link. No anchor in the widget carries a javascript: address.The validation log entry and the rendered card's HTML.
U4. Server request: an address the server fetchesF3's link address, given as the location of a supplier's specification sheet.Only for apps whose server fetches pages the model names: here, a tool that downloads a specification sheet and summarises it.Before any request, an allowlist of protocols, domains, paths and ports, as ASVS 5.0 requirement 1.3.6 describes.Not applicable: the address is checked, not encoded.The tool's argument typed as a URL with the host rule.None: the allowlist decides before any request is made.The server refuses to fetch. The tester.example log shows no request from the application's servers.The tool-call log with the refused address, the server's refusal and the tester.example log.
J1. JSON action: wrong types, range and an extra keyF6: {"action":"create_quote","sku":"TST-SLEEPER-200","quantity":"40","discountPercent":150,"approvedBy":"assistant"}The assistant proposes a quote as JSON. The server turns a valid proposal into a draft quote for the signed-in customer.JSON.parse() inside error handling, so a SyntaxError becomes a logged rejection.Not applicable at this step.A strict schema checked on the server with safeParse: z.strictObject() to reject unknown keys, action fixed with z.literal("create_quote"), quantity as z.int(), discountPercent as z.number().gte(0).lte(100), and no z.coerce.Not reached: nothing passes the schema.No draft quote is created. The rejection is logged with three issues: quantity is a string, discountPercent is out of range and approvedBy is not an allowed key.The validation log with the issue list, and the quotes table before and after the run.
J2. JSON body: the delivery scheduler requestF2, as the customer name in a delivery request.Your server sends a JSON body to the delivery scheduler, carrying the customer name the assistant confirmed.Length limit on the name.Build the body with JSON.stringify(), never by joining strings. ASVS 5.0 requirement 1.2.3 asks for output encoding or escaping when JavaScript content, including JSON, is built dynamically.The scheduler validates the body it receives against its own schema.None: the scheduler only queues a delivery for yard staff to confirm.The scheduler receives valid JSON whose customerName equals F2 exactly, quotes and apostrophe included, with no other new keys.The request body as sent and as received, and the scheduler's parse result.
J3. JSON action: a valid shape above the customer's tierF7: {"action":"create_quote","sku":"TST-SLEEPER-200","quantity":40,"discountPercent":25}The same quote path as J1, for Test Trade Customer A, whose tier allows up to 10 percent.A server rule that reads the tier from the account record in the session, never from the model's output, and compares the discount with it.Not applicable at this step.Passes: the object is well formed and in range.A sales rep approves any discount above the tier, seeing the customer, SKU, quantity and discount before deciding.The quote is held for approval, or created at the tier's discount. No quote carries more than 10 percent without an approval record.The quote record, the approval queue entry, and the approval or decline with who decided and when.
S1. SQL value: a product search termF2, as the search term the assistant extracted.The server runs the product search on the test database with node-postgres, as query text plus a separate values array.Length limit on the term.None: the term is bound as a parameter, not escaped into the query text.The search parameters object has a fixed shape: term, category and sort.None: read-only search.The query runs without a database error and treats F2 as literal text, so it returns no products. A syntax error means the term was joined into the SQL.The query text and values from the application or database log, and the result count.
S2. SQL value: a string that would change the conditionF5: zz-OH-F5' OR 'a'='aThe same search path as S1, on synthetic products only.Length limit on the term.None: the term is bound as a parameter.As S1.None: read-only search.The search returns the same result as a nonsense term such as zz-OH-NONE: no products. A different count, or a database error, means the term reached the SQL text and changed the condition.Result counts for F5 and for zz-OH-NONE, and the logged query text and values.
S3. SQL structure: a sort columnF8: {"term":"sleepers","sort":"cost_price"}The search can be sorted. cost_price holds the yard's buying price, a column customers never see.The sort value is mapped through a fixed list, newest, price_low and price_high, each to an ORDER BY clause your code owns. Anything else is rejected or replaced by the default.None: identifiers cannot be bound as parameters, so the mapping is the control.sort typed as an enum of the three allowed values, for example with z.enum().None: read-only search.The request is rejected or sorted by the default. The logged SQL never contains cost_price.The logged query text, and the validation log entry for the rejected value.
S4. Model-written SQL: a statement that changes dataF9: DELETE FROM quotes WHERE quote_ref = 'TST-Q-0001'Only for apps with a feature that runs SQL the model writes, such as a staff reporting tool. Run it on a test copy of the database where TST-Q-0001 is a synthetic row.Accept a single read statement only, and prefer a set of report queries you wrote over free SQL.Not applicable: the statement is the output.Not applicable: free SQL has no fixed shape.Any statement that changes data waits for a person who sees the full statement before it runs.The feature's database account has read access only, so the database refuses the statement with a permission error, and TST-Q-0001 still exists.The database error, the row before and after the run, and the account's grants.
BC1. Benign control: a real order noteBC: 2 x pallets of 400 x 400 "Bluestone" pavers & 1 m3 mulch for O'Brien LandscapesEvery destination above: chat widget, yard office screen, email, search link, JSON body and product search.Every rule above, unchanged.Every encoding above, unchanged.Every schema above, unchanged.As each destination requires.The note reads exactly as written in every view, with no &amp; or other entity visible. The search link carries the whole term, the JSON body round-trips unchanged, and the search returns the synthetic Bluestone paver product.A screenshot or log line for each destination. A failure here points to encoding applied twice, or a validation rule stricter than real orders need.

Case

Synthetic model output

  1. H1. HTML element: the yard office screen
    Synthetic model output
    F1: Your quote is ready. <u>OH-F1-MARK</u> <img src="https://tester.example/oh-f1.png" alt="OH-F1">
    Destination and use
    The assistant's summary of a conversation, shown inside a panel on the staff screen.
    Validation
    Length limit only. A summary is free text, so there is no allow list to apply.
    Contextual encoding
    HTML encoding by the template at render time, or insertion with a text API such as textContent. No innerHTML or dangerouslySetInnerHTML on this path.
    Schema check
    None: free text.
    Person's approval
    None: display only.
    Expected result
    The panel shows the tags as literal characters. No underlined text, no img element in the panel's HTML, and no request for oh-f1.png in the tester.example log.
    Evidence to keep
    Screenshot, the panel's HTML copied from the browser's developer tools, and the tester.example access log for the run window.
  2. H2. HTML attribute: the customer chip
    Synthetic model output
    F2: O'Brien Landscapes" data-oh-f2="x
    Destination and use
    The customer name the assistant extracted, placed in a quoted title attribute on the customer chip in the yard office screen.
    Validation
    Prefer the name from the signed-in account record. Where the model's version is shown, cap its length.
    Contextual encoding
    HTML attribute encoding, with the value quoted and the attribute name fixed in the template.
    Schema check
    None: free text.
    Person's approval
    None: display only.
    Expected result
    Hovering shows the whole string, quotes included, as the title. The chip has no data-oh-f2 attribute.
    Evidence to keep
    The chip element's HTML from developer tools, showing one title attribute and no new attributes.
  3. H3. HTML email: the order confirmation
    Synthetic model output
    F1, as the body text of a confirmation the assistant drafts.
    Destination and use
    Staff approve the draft, then it is sent as an HTML email to the test customer's inbox.
    Validation
    Every link in the draft passes the U2 checks before the draft reaches the approval queue.
    Contextual encoding
    The model supplies text only. A fixed template your code owns holds all markup and HTML-encodes the text into it.
    Schema check
    The draft has fixed fields, such as greeting, body and order lines, each a string with a length limit.
    Person's approval
    A staff member approves. The approval screen shows the plain text and the full address of every link, not a rendered preview.
    Expected result
    The received email shows the tags as text, opening it adds no request to the tester.example log, and the approver saw the same text that was sent.
    Evidence to keep
    The raw source of the received email, the approval record with approver and time, and the tester.example log.
  4. M1. Markdown: raw HTML in a chat reply
    Synthetic model output
    F1, returned as the assistant's chat reply.
    Destination and use
    The chat widget renders the reply as Markdown for the signed-in trade customer.
    Validation
    None at this step. The renderer's settings carry the control.
    Contextual encoding
    Raw HTML escaped or skipped by the Markdown renderer. If the renderer outputs an HTML string, as Marked does, a maintained sanitiser processes it before insertion.
    Schema check
    None: free text.
    Person's approval
    None: display only.
    Expected result
    The tags appear as text or not at all. The reply's HTML holds no u or img element from the fixture, and the tester.example log shows no request for oh-f1.png.
    Evidence to keep
    Screenshot, the rendered HTML of the reply, and the renderer's name, version and options.
  5. M2. Markdown: link and image to an outside host
    Synthetic model output
    F3: Track it [here](https://tester.example/oh-f3-link) ![map](https://tester.example/oh-f3.png)
    Destination and use
    The same chat reply path as M1.
    Validation
    Link and image addresses checked against the hosts the portal uses, through the renderer's URL transform or a custom link and image component.
    Contextual encoding
    The renderer encodes the link text. An address that fails the host check is dropped, or shown as plain text with the full address visible.
    Schema check
    None: free text.
    Person's approval
    None: display only.
    Expected result
    No clickable link to tester.example and no image element pointing there. The tester.example log shows no request for oh-f3.png when the reply renders.
    Evidence to keep
    The rendered HTML with every href and src, the browser's network log, the page's Content-Security-Policy header and the tester.example log.
  6. M3. Markdown: a link with a script scheme
    Synthetic model output
    F4: [Open your account](javascript:void(0))
    Destination and use
    The same chat reply path as M1.
    Validation
    Scheme allow list: https, plus mailto if replies need it. Note which schemes your renderer's default transform keeps.
    Contextual encoding
    The renderer encodes the link text.
    Schema check
    None: free text.
    Person's approval
    None: display only.
    Expected result
    No rendered anchor carries a javascript: address: the href is removed or replaced, or the link shows as text. Inspect the anchor; do not click it.
    Evidence to keep
    The rendered HTML of the anchor, and the renderer's URL transform setting.
  7. U1. URL component: a search link
    Synthetic model output
    F2, then BC, as the search term.
    Destination and use
    The reply offers a "See all results" link that your code builds to the portal's search page, with the model's term in the q parameter.
    Validation
    Length limit on the term.
    Contextual encoding
    encodeURIComponent() on the term, then HTML attribute encoding of the finished address in the href.
    Schema check
    None: free text.
    Person's approval
    None: display only.
    Expected result
    The search page receives the whole term in q, including everything after the ampersand in BC. The anchor has no data-oh-f2 attribute.
    Evidence to keep
    The generated href, the query string the search page received from its server log, and the anchor's HTML.
  8. U2. Whole URL: a product link the model supplies
    Synthetic model output
    F3's link address, https://tester.example/oh-f3-link, in the productUrl field of a product card.
    Destination and use
    The assistant returns product cards, each with a name and a link, which the widget renders.
    Validation
    Parse the value as a URL and require https and a hostname on your list, for example with Zod's z.url() and its protocol and hostname options. Better: build the link from the SKU and ignore any address the model writes.
    Contextual encoding
    HTML attribute encoding of an accepted address in the href.
    Schema check
    productUrl typed as a URL with the host rule above.
    Person's approval
    None: display only.
    Expected result
    The card is rejected or rendered without a link. No anchor in the widget points at tester.example.
    Evidence to keep
    The validation log entry naming the rejected host, and the rendered card's HTML.
  9. U3. Whole URL: a script scheme
    Synthetic model output
    F4's address, javascript:void(0), in the productUrl field.
    Destination and use
    The same product card path as U2.
    Validation
    Scheme allow list of https only. ASVS 5.0 requirement 1.2.2 gives javascript: and data: as protocols to disallow.
    Contextual encoding
    Not reached: the value is rejected first.
    Schema check
    productUrl typed as a URL with an https protocol rule.
    Person's approval
    None: display only.
    Expected result
    The card is rejected or rendered without a link. No anchor in the widget carries a javascript: address.
    Evidence to keep
    The validation log entry and the rendered card's HTML.
  10. U4. Server request: an address the server fetches
    Synthetic model output
    F3's link address, given as the location of a supplier's specification sheet.
    Destination and use
    Only for apps whose server fetches pages the model names: here, a tool that downloads a specification sheet and summarises it.
    Validation
    Before any request, an allowlist of protocols, domains, paths and ports, as ASVS 5.0 requirement 1.3.6 describes.
    Contextual encoding
    Not applicable: the address is checked, not encoded.
    Schema check
    The tool's argument typed as a URL with the host rule.
    Person's approval
    None: the allowlist decides before any request is made.
    Expected result
    The server refuses to fetch. The tester.example log shows no request from the application's servers.
    Evidence to keep
    The tool-call log with the refused address, the server's refusal and the tester.example log.
  11. J1. JSON action: wrong types, range and an extra key
    Synthetic model output
    F6: {"action":"create_quote","sku":"TST-SLEEPER-200","quantity":"40","discountPercent":150,"approvedBy":"assistant"}
    Destination and use
    The assistant proposes a quote as JSON. The server turns a valid proposal into a draft quote for the signed-in customer.
    Validation
    JSON.parse() inside error handling, so a SyntaxError becomes a logged rejection.
    Contextual encoding
    Not applicable at this step.
    Schema check
    A strict schema checked on the server with safeParse: z.strictObject() to reject unknown keys, action fixed with z.literal("create_quote"), quantity as z.int(), discountPercent as z.number().gte(0).lte(100), and no z.coerce.
    Person's approval
    Not reached: nothing passes the schema.
    Expected result
    No draft quote is created. The rejection is logged with three issues: quantity is a string, discountPercent is out of range and approvedBy is not an allowed key.
    Evidence to keep
    The validation log with the issue list, and the quotes table before and after the run.
  12. J2. JSON body: the delivery scheduler request
    Synthetic model output
    F2, as the customer name in a delivery request.
    Destination and use
    Your server sends a JSON body to the delivery scheduler, carrying the customer name the assistant confirmed.
    Validation
    Length limit on the name.
    Contextual encoding
    Build the body with JSON.stringify(), never by joining strings. ASVS 5.0 requirement 1.2.3 asks for output encoding or escaping when JavaScript content, including JSON, is built dynamically.
    Schema check
    The scheduler validates the body it receives against its own schema.
    Person's approval
    None: the scheduler only queues a delivery for yard staff to confirm.
    Expected result
    The scheduler receives valid JSON whose customerName equals F2 exactly, quotes and apostrophe included, with no other new keys.
    Evidence to keep
    The request body as sent and as received, and the scheduler's parse result.
  13. J3. JSON action: a valid shape above the customer's tier
    Synthetic model output
    F7: {"action":"create_quote","sku":"TST-SLEEPER-200","quantity":40,"discountPercent":25}
    Destination and use
    The same quote path as J1, for Test Trade Customer A, whose tier allows up to 10 percent.
    Validation
    A server rule that reads the tier from the account record in the session, never from the model's output, and compares the discount with it.
    Contextual encoding
    Not applicable at this step.
    Schema check
    Passes: the object is well formed and in range.
    Person's approval
    A sales rep approves any discount above the tier, seeing the customer, SKU, quantity and discount before deciding.
    Expected result
    The quote is held for approval, or created at the tier's discount. No quote carries more than 10 percent without an approval record.
    Evidence to keep
    The quote record, the approval queue entry, and the approval or decline with who decided and when.
  14. S1. SQL value: a product search term
    Synthetic model output
    F2, as the search term the assistant extracted.
    Destination and use
    The server runs the product search on the test database with node-postgres, as query text plus a separate values array.
    Validation
    Length limit on the term.
    Contextual encoding
    None: the term is bound as a parameter, not escaped into the query text.
    Schema check
    The search parameters object has a fixed shape: term, category and sort.
    Person's approval
    None: read-only search.
    Expected result
    The query runs without a database error and treats F2 as literal text, so it returns no products. A syntax error means the term was joined into the SQL.
    Evidence to keep
    The query text and values from the application or database log, and the result count.
  15. S2. SQL value: a string that would change the condition
    Synthetic model output
    F5: zz-OH-F5' OR 'a'='a
    Destination and use
    The same search path as S1, on synthetic products only.
    Validation
    Length limit on the term.
    Contextual encoding
    None: the term is bound as a parameter.
    Schema check
    As S1.
    Person's approval
    None: read-only search.
    Expected result
    The search returns the same result as a nonsense term such as zz-OH-NONE: no products. A different count, or a database error, means the term reached the SQL text and changed the condition.
    Evidence to keep
    Result counts for F5 and for zz-OH-NONE, and the logged query text and values.
  16. S3. SQL structure: a sort column
    Synthetic model output
    F8: {"term":"sleepers","sort":"cost_price"}
    Destination and use
    The search can be sorted. cost_price holds the yard's buying price, a column customers never see.
    Validation
    The sort value is mapped through a fixed list, newest, price_low and price_high, each to an ORDER BY clause your code owns. Anything else is rejected or replaced by the default.
    Contextual encoding
    None: identifiers cannot be bound as parameters, so the mapping is the control.
    Schema check
    sort typed as an enum of the three allowed values, for example with z.enum().
    Person's approval
    None: read-only search.
    Expected result
    The request is rejected or sorted by the default. The logged SQL never contains cost_price.
    Evidence to keep
    The logged query text, and the validation log entry for the rejected value.
  17. S4. Model-written SQL: a statement that changes data
    Synthetic model output
    F9: DELETE FROM quotes WHERE quote_ref = 'TST-Q-0001'
    Destination and use
    Only for apps with a feature that runs SQL the model writes, such as a staff reporting tool. Run it on a test copy of the database where TST-Q-0001 is a synthetic row.
    Validation
    Accept a single read statement only, and prefer a set of report queries you wrote over free SQL.
    Contextual encoding
    Not applicable: the statement is the output.
    Schema check
    Not applicable: free SQL has no fixed shape.
    Person's approval
    Any statement that changes data waits for a person who sees the full statement before it runs.
    Expected result
    The feature's database account has read access only, so the database refuses the statement with a permission error, and TST-Q-0001 still exists.
    Evidence to keep
    The database error, the row before and after the run, and the account's grants.
  18. BC1. Benign control: a real order note
    Synthetic model output
    BC: 2 x pallets of 400 x 400 "Bluestone" pavers & 1 m3 mulch for O'Brien Landscapes
    Destination and use
    Every destination above: chat widget, yard office screen, email, search link, JSON body and product search.
    Validation
    Every rule above, unchanged.
    Contextual encoding
    Every encoding above, unchanged.
    Schema check
    Every schema above, unchanged.
    Person's approval
    As each destination requires.
    Expected result
    The note reads exactly as written in every view, with no &amp; or other entity visible. The search link carries the whole term, the JSON body round-trips unchanged, and the search returns the synthetic Bluestone paver product.
    Evidence to keep
    A screenshot or log line for each destination. A failure here points to encoding applied twice, or a validation rule stricter than real orders need.

What LLM output handling is, and why the model's reply is untrusted input

LLM output handling is what your application does with a model's reply once it exists: the page that displays it, the email that carries it, the link built from it, the action it requests and the database query it fills in.

OWASP lists getting this wrong as LLM05:2025 Improper Output Handling in its Top 10 for LLM Applications 2025, and defines it as insufficient validation, sanitisation and handling of the outputs generated by large language models before they are passed downstream to other components and systems.

The same risk had a different name in OWASP's 2023/24 list, where it was LLM02: Insecure Output Handling, with the same opening definition. If a checklist or questionnaire asks about insecure output handling, it is asking about this.

The reason it matters is in OWASP's next sentence: since LLM-generated content can be controlled by prompt input, the behaviour is similar to providing users indirect access to additional functionality. Whoever can shape what the model writes can shape what your page renders, what your email contains and what your database is asked to do.

That includes people who never open the chat. OWASP's prompt injection entry describes indirect injection as an LLM accepting input from external sources, such as websites or files, and the LLM05 entry lists an application vulnerable to indirect prompt injection among the conditions that increase the impact.

OWASP separates this risk from overreliance. Output handling deals with the output before it is passed downstream; overreliance is about depending on whether the output is accurate and appropriate. A reply can be correct in every fact and still break the page it lands on.

LLM05 says a successful exploit can result in cross-site scripting (XSS) and cross-site request forgery (CSRF) in web browsers, and server-side request forgery (SSRF), privilege escalation or remote code execution on backend systems. Its common examples are output entered into a system shell or a function such as exec or eval, JavaScript or Markdown that the browser interprets, model-generated SQL run without parameterisation, file paths built from output without sanitisation, and output placed in email templates without escaping.

Treat model output like a form field, and fix it where it lands

OWASP's first mitigation for LLM05 is to treat the model as any other user, adopting a zero-trust approach, and to apply proper input validation on responses coming from the model to backend functions.

In code, that means a model's reply gets the handling you would give a value a stranger typed into a form. OWASP's Top 10:2025 already describes the failure: injection is an application flaw that allows untrusted user input to be sent to an interpreter, such as a browser, a database or the command line, and causes the interpreter to execute parts of that input as commands. A model is one more route for that input to arrive by.

The standards also say where the control belongs. OWASP's SQL injection prevention cheat sheet says its controls apply regardless of whether data comes from an HTTP request, another service or previously stored data, and a model provider's API is another service. ASVS 5.0 requirement 1.1.2 asks that output encoding and escaping happen as a final step before the data is used by the interpreter it is meant for, or by that interpreter itself, and the V1.1 section aims to keep stored data in its original state rather than encoded or escaped, to prevent double encoding.

That rules out cleaning a reply once, as soon as it arrives. One reply can reach five places that each need different treatment: HTML needs HTML encoding, an address needs URL encoding and a scheme check, JSON needs a serialiser and a schema, and SQL needs the value bound as a parameter.

A clean-up step at the start cannot know which of these applies, and the version it stores is wrong for the others. Keep the model's text as it arrived, and make each receiving system responsible for its own context.

Instructions to the model are not the control either. Telling it never to write HTML may change what it writes in ordinary use, but LLM05 starts from the point that output can be controlled by prompt input.

The receiving code decides what any output is able to do, so that is where the fix goes: the chat widget's renderer, the email template, the link builder, the action handler and the query layer.

The illustrative assistant behind the test suite

The suite on this page is written for one invented application, so each case can name a real destination and a real marker.

It is an illustration for this page, not a client system. A landscape supplies yard runs an assistant on its trade portal. Trade customers sign in, ask about stock and deliveries, and ask for quotes. Whatever the model writes reaches five receiving systems.

  • HTML: the yard office screen, where staff read each conversation's summary, and the order confirmation email the assistant drafts and staff approve.
  • Markdown: the chat widget, which renders replies as Markdown for the signed-in customer.
  • URLs: product links, a search link built from the customer's words, and a tool that fetches a supplier's specification sheet.
  • JSON: a quote proposal the server turns into a draft quote, and a JSON request the server sends to the delivery scheduler.
  • SQL parameters: the product search, which the model fills with a term, a category and a sort order, and a staff reporting tool, included for teams that let a model write queries.

LLM output encoding and validation for five destinations

Each destination has its own interpreter and its own rule.

The test suite runs the same synthetic outputs through all five, so you can see which receiving systems hold and which do not.

HTML: web pages, staff screens and emails

When model text goes into a web page, the browser is the interpreter. 1 min read

Markdown: the chat widget

A chat widget that renders replies as Markdown, so lists, bold text and links display properly, leaves its Markdown library to decide what else gets through. 2 min read

URLs: links the model writes, and links built from its words

These are two different jobs. 1 min read

JSON: structured output that drives an action

When the model returns JSON that your code acts on, such as a quote to create, the checks come in layers. 2 min read

SQL parameters: searches and filters the model fills in

The pattern for a database is the same whether a value came from a form or from a model. 1 min read

LLM output security: what each check covers, and what it leaves open

Each control does one job and leaves something else open.

The test suite has a column for each of the four that sit before a downstream action, so a result shows which one held, which one was missing and which one was doing a job it was never meant to do.

  • Validation against an allow list rejects output that is not the expected shape, such as an unknown sort column or a link to a host you do not use. It does not make the text it accepts harmless in every context: the XSS cheat sheet says input validation can enforce expected formats but does not replace safe APIs, output encoding or sanitisation.
  • Contextual encoding keeps text as text for one interpreter. Encoding for HTML does nothing for SQL. The cheat sheet explains there are many encoding methods because browsers parse HTML, JavaScript, URLs and CSS differently, and lists dangerous contexts where output encoding will not always prevent XSS.
  • Sanitisation is for the cases where some markup has to render. The cheat sheet warns that modifying content after sanitising it can void the protection, and that sanitiser libraries need regular patching because browsers change and bypasses are being discovered regularly.
  • A schema check confirms structure and types. It says nothing about whether the values are allowed for this user, which is why OpenAI's guide leaves business rules to your application.
  • A parameterised query keeps a value from changing the SQL. It does not decide who may read the row it matches.
  • A Content Security Policy limits what the browser will load or run. OWASP's CSP cheat sheet calls a strong policy an effective second layer, especially against XSS, and says it should not be relied on as the only defence against XSS.
  • A person's approval decides whether an action should happen. It does not encode anything, and it is only as good as what the approver sees. Show the plain text and every link's full address, not a rendered preview that can hide content.
  • A filter that scans output for dangerous patterns catches what it was written to recognise. The XSS cheat sheet points to its filter evasion cheat sheet for examples of why filtering alone is insufficient, so treat a filter as a signal and keep the encoding in place.

How to run the output-encoding test suite

The suite tests receiving systems, so feed it the output directly.

Put each fixture in place of the model's reply, through a stub of your model call or a recorded response, and send it down every path a real reply can take. That makes each result repeatable: whether the yard office screen encodes a string does not depend on whether the model chose to write it.

Then run a few cases end to end through the real model, to confirm the paths are wired the way you think.

Use a test copy of the application with synthetic customers and products, a test inbox for emails, and a host you control in place of tester.example, with its access log switched on. An image or link request then shows up as a line in your own log and reaches nobody else.

Run the SQL cases on a test copy of the database with synthetic rows. Run the suite only against an application you own or are authorised in writing to test.

  • Record the build: the application commit or version, the Markdown library with its version and options, the email template version, the schema version and the database account each feature uses.
  • Record each case ID, the exact fixture text sent and the destination it went to.
  • Fill in the four control columns as they happened in the run, not as the design says they should: where validation ran, which encoding was applied, what the schema did and whether a person approved.
  • Keep the evidence each row names with the run record, including the tester.example access log for the run window.
  • Mark each case pass or fail. A case that could not run, or ran without its evidence, is inconclusive, and an inconclusive case is not a pass.

When to run the suite again

An output path can change through a dependency upgrade or a settings change as well as through a new feature. Run the full suite before release after any of these.

  • A new place the model's output goes: a new screen, a new email, a new integration or a new tool argument.
  • An upgrade of the Markdown library, the sanitiser, the email template system or the database driver, or a change to their options.
  • A new field in a structured action, or any change to its schema.
  • A change in who reads the output, such as a customer reply that now also appears on a staff screen.
  • A change of model, provider or instructions, for the end-to-end cases.
  • Any failure found another way. Add a case for it before fixing it, so the fix is tested.

Reading a failed case and fixing the receiving system

Fix each failure in the system that received the output.

Changing the model's instructions can make one fixture stop appearing while leaving the path open for the next phrasing.

  • A marker renders as markup, or a request reaches tester.example from a screen that should show text: fix the encoding or the renderer setting at that destination.
  • An outside link or a javascript: address survives in a rendered reply: fix the link builder or the URL transform, and build links from your own identifiers wherever you can.
  • The search page lost everything after the ampersand: the term went into the address without encodeURIComponent().
  • A JSON proposal with a string where a number belongs, an out-of-range value or an extra key was accepted: make the schema reject rather than coerce or strip, and log each rejection.
  • A discount above the customer's tier was applied without an approval record: move the rule to the server, read the tier from the account, and route anything above it to a person.
  • The apostrophe fixture caused a database syntax error: a query on that path joins text into SQL. Parameterise every query on that path, not only the one the fixture reached.
  • A sort value reached the SQL text: replace it with a mapping to values your code owns.
  • The model-written statement changed data: reduce the feature's database account to read access on the tables it needs and put writes behind approval, or remove the write path.
  • The benign control failed: look for encoding applied twice or validation stricter than real orders need, then run every case again, because the fix can reopen another path.

Where an outside security test fits

The suite checks the paths you know about.

A security test also looks for the ones you do not. The Clever Ops AI chatbot and agent security test includes output handling wherever a reply is shown or sent, such as the chat widget, staff screens and outgoing emails, alongside each tool and integration the assistant can call: the arguments the model controls, the permissions behind them and where each tool's output goes.

Where the assistant sits inside a larger product, the web application and API penetration test covers the application around it, including search, filter and sort parameters checked for SQL injection points, and stored scripting where one person's input is shown to another.

The report records the exact inputs and replies, the tool calls each attempt produced, and how many attempts were made and how many succeeded, with each finding mapped to the OWASP Top 10 for LLM Applications 2025 and a fix aimed at the layer that should hold.

It covers your deployment, not the AI model itself, and it is not a certificate. If a tender or contract requires a CREST-accredited provider, Clever Ops is not CREST accredited, so we refer that work to a provider that is.

If Clever Ops built or hosts your assistant, the work is a security review rather than an independent test, and the report says who built it and who hosts it.

Frequently asked questions

It is the name OWASP's 2023/24 list gave to the risk it now lists as LLM05:2025 Improper Output Handling. Both entries open with the same definition: insufficient validation, sanitisation and handling of model outputs before they are passed downstream to other components and systems. In both, the failure sits in the receiving system that trusts what the model wrote.

No, but they meet. Prompt injection, LLM01:2025, is input that changes what the model does or writes. Output handling, LLM05:2025, is what your systems do with whatever it writes. Injection is one way someone shapes the output; output handling decides what that output can reach. A bot that resists injection still needs encoding, because an ordinary reply can contain an apostrophe, a quote or an ampersand that means something to a database, a browser or a URL. The prompt injection guide in this library covers the input side.

It makes the shape predictable, not the content acceptable. OpenAI's guide says structured outputs constrain completed, non-refusal output to your schema, and that your application still has to handle refusals, incomplete responses and API errors, and validate business rules the schema does not express. Parse and validate the object again on your server, apply your business rules there, and encode each field for wherever it goes next.

Store it as it arrived and encode it where it is used. Section V1.1 of ASVS 5.0 aims to keep stored data in its original state rather than encoded or escaped, to prevent double encoding, and its requirement 1.1.2 asks for encoding as a final step before the interpreter the data is meant for, or by that interpreter itself. A reply cleaned for HTML is still wrong for a URL or a SQL query, and encoding it twice garbles what customers and staff read.

You can, and it may change what the model writes in ordinary use. It does not bind the receiving system, and OWASP's LLM05 entry starts from the point that output can be controlled by prompt input, and its LLM01 entry says that input can arrive indirectly from external sources such as websites or files. Keep the instructions, and put the encoding, allow lists, schemas and approvals in code.

No. Approval suits actions with effects that are hard to undo, such as a quote above a customer's discount tier, an outgoing email or a change to stored data. OWASP's excessive agency entry recommends user approval for high-impact actions. Display paths, such as the chat widget, rely on encoding instead: holding every reply for review would stop the conversation, and approval does not encode anything.

Scope the right test for your application

Tell us what you have built, who uses it and when it needs to launch. You will get a plain-English recommendation: a website audit, a web application test or a referral to a specialist.