Skip to main content
Security guideReviewed

LLM data leakage: map where your data goes in an AI application, and how it comes back out

A language model application copies business data into more places than the chat window shows: prompts, retrieved documents, fine-tuning sets, logs, caches and evaluation sets.

This guide maps each one, explains how data leaks back out of it, and gives you a data-lineage worksheet and a canary-record protocol to run on a system you own or are authorised to test.

Data-lineage worksheet and canary-record protocol

An illustration written for this page: an invented property management agency's assistant, mapped row by row, with a blank row to copy for your own application, a canary-record protocol for citation exposure, cross-user recall, deletion and privacy-request handling, and a short list for the places you could not account for. Run the protocol only on a system you own or are authorised to test, with synthetic records.

llm-data-leakage.txt

Introduction2 lines

DATA-LINEAGE WORKSHEET AND CANARY-RECORD PROTOCOL (VERSION 1) An illustration written for this page. The business, its application, every record and every marker are invented. Fill in Part C for your own application. Run Part D only on a system you own or are authorised to test, using test accounts and synthetic canary records in place of real customer data.

What LLM data leakage is in a business application

LLM data leakage is business or personal information reaching someone it was not meant for through an application built on a large language model: a customer chatbot, a staff assistant, a drafting tool or an agent.

OWASP's entry LLM02:2025 Sensitive Information Disclosure lists what is at stake, including personal identifiable information, financial details, health records, confidential business data, security credentials and legal documents. It says that LLMs, especially when embedded in applications, risk exposing sensitive data, proprietary algorithms or confidential details through their output.

The output is where a leak shows up, not where it starts. To answer one question, an LLM application may copy data into a prompt, send it to a model provider, write it to a log, store it in a cache and in conversation history, and later reuse it in an evaluation set or a fine-tuning run.

Each copy has its own readers and its own lifetime, and each is a place the data can come back out: in another user's answer, in a log viewer a contractor can open, in a spreadsheet of test questions, or from the model itself.

That is why this guide starts with a map rather than a list of attacks. If you can say, for every place your data lives, who can read it, how long it stays, whether it leaves the business and how it is removed, then each of those answers can be tested. The rows you cannot fill in are the first places to look.

Where this guide stops

Two neighbouring guides go deeper on one row each. Who may retrieve which document, how tenants stay apart in a shared index and how a deletion reaches the index are covered in the RAG security guide.

What sits in a bot's instructions, and how those instructions are drawn out, is the subject of the system prompt exposure guide; OWASP gives that risk its own entry, LLM07:2025 System Prompt Leakage, which says the system prompt should not be considered a secret.

This page holds the map that contains both, including the rows neither covers in depth: fine-tuning and training sources, logs, evaluation sets and the copies your vendors keep.

Six places your data lives in an LLM application

Each place below becomes one or more rows in the worksheet on this page.

The questions are the same for every row: what goes in, who can read it, how long it is kept, whether it leaves the business, and how it is removed.

Prompts and conversation history

Your code builds every prompt: the instructions, the conversation so far, the user's message, and any record, document or tool result it decides the model needs. 1 min read

Retrieved documents

Passages a retrieval system finds are added to the prompt, so the access check has to run before they are handed over. 1 min read

Fine-tuning and training sources: where training data leakage starts

Data used to fine-tune or train a model is the one row where the copy ends up inside the model rather than beside it. 1 min read

Logs, traces and error reports

A log line that records the full prompt and response holds everything the model saw and said, including retrieved passages and tool results. 1 min read

Caches

An LLM application can hold copies in several caches: stored answers, stored search results, transcripts kept in the browser, shared answer links, and the model provider's own prompt cache. 1 min read

Evaluation sets

An evaluation set is a list of test questions with the answers you expect, run against the application after each change. 1 min read

How data comes back out: five routes to test

A leak needs a copy and a route to someone who should not see it.

These are the routes the canary protocol on this page checks, with the test that covers each one.

  • Citation exposure (test C1): an answer, its citations or the raw response behind the chat window name or quote a document the person asking may not open.
  • Cross-user recall (test R1): something one user typed, uploaded or was told turns up in another user's answer. OWASP's LLM02:2025 entry describes a user receiving a response that contains another user's personal data. Shared memory, conversation summaries, answer caches and prompt builders that pull in the wrong record are the paths to check.
  • Extraction from the model (test R2, fine-tuned models only): text from a training or fine-tuning set comes back in a completion, whether someone asked for it or not.
  • Store exposure (test S1): the data is reached in a store rather than through the chat, such as a log viewer, a tracing dashboard, an evaluation spreadsheet or a provider console, by someone with access to that store but not to the record.
  • Survival after deletion (tests D1 and P1): the record is deleted from the main database, but a copy stays in a log, a cache, an evaluation set, a training file or a provider store, and comes back later.

What your model provider keeps: read its terms and date them

The worksheet column that asks whether data leaves the business, and to whom, cannot be filled in from a general belief about AI providers.

It needs the current terms of the provider your application actually calls, read on a stated day. What one provider's page says tells you nothing about another's, and a page read last year may not describe today.

As an example of what such a page contains, OpenAI's page on data controls in the OpenAI platform, read on 14 November 2026, says the following about its API.

  • As of 1 March 2023, data sent to the OpenAI API is not used to train or improve OpenAI models unless the customer explicitly opts in to share data with OpenAI.
  • Abuse monitoring logs may contain certain customer content, such as prompts and responses. By default they are generated for all API feature usage and retained for up to 30 days, unless longer retention is required by law or is reasonably necessary to protect OpenAI's services or any third party from harm.
  • Eligible customers can have their content excluded from those logs by getting approved for the Zero Data Retention or Modified Abuse Monitoring controls, which are subject to OpenAI's prior approval and acceptance of additional requirements, with limits the page sets out. Endpoints that are not eligible for Zero Data Retention may still store application state when it is enabled.
  • Its per-endpoint table gives the application state retention for files, fine-tuning jobs, evaluations, vector stores and conversations as until deleted. With exceptions the page notes, the Responses API has a 30-day application state retention period by default, or when its store parameter is set to true, and response data is then stored for at least 30 days.
  • Objects related to the Assistants API are deleted from OpenAI's servers 30 days after you delete them through the API or the dashboard, and objects that are not deleted are retained indefinitely.
  • Prompt caching may store encrypted key/value tensors in GPU-local storage, and that data is not retained after the 24-hour expiration.
  • MCP servers used with its remote MCP server tool are third-party services, and data sent to them is subject to their own retention policies.

Turning a provider page into worksheet entries

Write what you read into the worksheet with the date, then add the settings your account actually uses: whether a retention control was approved, whether stored conversations or responses are switched on, and which endpoints the application calls. If you reach a model through a cloud platform or another intermediary, the terms to read are those of the service your application calls.

Do the same for every other vendor on the map: the vector database, the tracing tool, the error tracker, the evaluation platform and any MCP server.

Filling in the worksheet for your own application

The worksheet is meant to be finished in one sitting with your developer, using what the system does today rather than what its design document says.

Work in this order.

  • Follow one real request from start to finish: from the browser, through your server, to the model provider and back. Write down every system that receives or stores any part of it.
  • List every vendor the application pays or connects to: model provider, vector database, hosting, logging, tracing, error tracking, analytics, email and any MCP server. Each one that receives data is a row.
  • For each row, ask three questions: where is the setting that controls retention, who can read this copy, and what happens to it when a record is deleted in the main system.
  • Read each vendor's current data terms and write the date beside what you read.
  • Mark every cell known, partly known or unknown. Unknown is a valid answer: it moves the row to the short list in Part E, with the person who can find out.
  • Run the canary protocol last, to check the map against what the system actually does.

Canary records: what a planted marker can show, and what it cannot

A canary record is a synthetic record carrying a unique marker, planted in one place on the map so you can search for the marker everywhere else.

The approach borrows from research on unintended memorisation. The Secret Sharer, by Carlini and colleagues, describes a testing methodology for quantitatively assessing the risk that rare or unique training-data sequences are unintentionally memorised by generative sequence models, and reports that unintended memorisation is a persistent, hard-to-avoid issue.

Every canary is synthetic: an invented person, an email address at a test domain you control, a test phone number you control, and a marker such as LEAKTEST-P2-7Q4M that appears nowhere else. Use one marker per row and per test, so a marker found anywhere names where it came from.

Plant canaries only in a system you own or are authorised to test, through test accounts, and record when and where each one went in.

A found marker shows a lot: a path exists from where it was planted to where it was found, and the record of the test names both ends. A missing marker shows much less. It means the marker was not seen with the prompts, accounts, number of runs and searches used on that day.

It does not show that the data is absent from a store you did not search, and it does not show that a model has not memorised it. A different prompt, more attempts or another extraction method may bring it out, and Nasr and colleagues' conclusion that current alignment techniques do not eliminate memorisation is a reason not to read a model's silence as absence.

Record results in those terms: found, where and when; or not found in a stated number of attempts with stated prompts.

Canaries and fine-tuned models

For a fine-tuned model, the evidence that matters most sits upstream of the model: which dataset version it was trained on, and what that dataset held. A canary in a test fine-tuning run shows whether that test model, with that data and those settings, repeats a planted record.

It says nothing reliable about a production model trained on different data. Keep personal and confidential records out of fine-tuning sets rather than relying on a test to show they did not stick. OWASP's LLM02:2025 entry recommends scrubbing or masking sensitive content before it is used in training.

Deletion and privacy requests: what the map lets you answer

A person may ask a business what it holds about them, or ask for their information to be corrected or removed.

In an LLM application, the answer depends on the map, because their data may sit in conversation history, an index, logs, caches, an evaluation set, a fine-tuning file and the stores of several vendors, each with its own removal method. The map turns that request into a list: every row the person's data could be in, who removes it there, and how you check it has gone.

Fine-tuned models need a decision before a request arrives. Keep a record of which dataset version each model was trained on, so a request can be traced to every model that saw the record, and decide in advance whether such a model is retrained without the record, retired, or handled another way your privacy adviser supports. Deleting the training file removes one copy; the model trained on it is a separate thing.

In Australia, Schedule 1 of the Privacy Act 1988 sets out the Australian Privacy Principles. Australian Privacy Principle 11.1 says that if an APP entity holds personal information, the entity must take such steps as are reasonable in the circumstances to protect the information from misuse, interference and loss, and from unauthorised access, modification or disclosure, and APP 11.3 says those steps include technical and organisational measures.

APP 8.1 says that before an APP entity discloses personal information about an individual to an overseas recipient, it must take such steps as are reasonable in the circumstances to ensure that the recipient does not breach the Australian Privacy Principles, other than APP 1, in relation to the information; APP 8.2 sets out cases where that does not apply.

Whether those principles apply to your business, whether sending prompts to a model provider is a disclosure to an overseas recipient, and whether and how a particular access or deletion request has to be met are questions for your privacy adviser. This page is not legal advice.

What the map and the privacy-request drill give your adviser are the facts: where the data is, who can read it, where it goes, and whether your team could find and remove all of it when it tried.

Controls that shrink the map

Every copy you do not make is a row you do not have to protect, test or clear.

These controls reduce the number of rows, or the number of people each row reaches.

  • Send the model only what the task needs: the fields that answer the question, for the person asking, rather than the whole record. OWASP's LLM02:2025 entry recommends least-privilege access to sensitive data.
  • Redact before data leaves your code. The same entry describes tokenisation and redaction, using techniques such as pattern matching to detect and redact confidential content before processing. Apply that step before logging as well as before the model call.
  • Log metadata by default and full text only on purpose: which user, which model call, which documents and tools, and the outcome. Where full prompts are logged for debugging, give that log a short retention setting and a named list of readers.
  • Keep personal and confidential records out of fine-tuning sets. Where a dataset has to carry them, OWASP's entry also describes differential privacy, which adds noise to the data or outputs to make individual data points difficult to reverse-engineer.
  • Build evaluation sets from synthetic records, and give any real ones a row in the map.
  • Make deletion a job that walks the map: one request removes the record from every row that can hold it, and the job records what it removed and where it could not.
  • Say what you do with data. OWASP's entry recommends clear policies about data retention, usage and deletion, and letting users opt out of having their data included in training.

Where the AI chatbot and agent security test fits

The Clever Ops AI chatbot and agent security test looks at data a bot can be made to reveal: what each read tool returns, what sits in the instructions, and whether one customer's records can be reached from another customer's session.

It also covers the retrieval pipeline, including whether retrieval keeps public, customer and staff content apart, and the chat backend as an API, including the screens staff use to read transcripts. Testing uses test identities and made-up records. The report records the exact messages used and the bot's replies, and the tool calls each attempt produced, with secrets and personal information redacted.

The test examines your deployment, not the AI model behind it, and it does not red-team the model itself. If Clever Ops built or hosts your chatbot, the work is a security review rather than an independent test, and the report says who built it and who hosts it.

The report records what was tested, on which configuration and on which dates; it is not a certificate or a compliance opinion. Bring the worksheet, filled in for your own application, to the scoping call.

Frequently asked questions

It is business or personal information reaching someone it was not meant for through an application built on a large language model. It can come out in an answer, a citation, a log, a cache or an evaluation set, or, where your data trained or fine-tuned the model, in the model's own output. OWASP covers it as LLM02:2025 Sensitive Information Disclosure.

Published research shows it can. Carlini and colleagues recovered hundreds of verbatim sequences from GPT-2's training data by querying the model, and Nasr and colleagues showed that an adversary can extract gigabytes of training data from several models, ChatGPT among them. For a business application, the question is whether your own data trained or fine-tuned the model. If it did, treat every record in that dataset as something anyone who can query the model might see.

Read that provider's current terms and date them; the answer is the provider's to state. As one example, OpenAI's data controls page, read on 14 November 2026, says that as of 1 March 2023, data sent to the OpenAI API is not used to train or improve OpenAI models unless the customer explicitly opts in. The same page says abuse monitoring logs, which may contain certain customer content such as prompts and responses, are retained for up to 30 days by default, unless longer retention is required by law or is reasonably necessary to protect OpenAI's services or any third party from harm. Put what your provider's page says, and the date you read it, in the worksheet.

No. A canary that turns up proves a path exists. A canary that does not turn up shows only that it was not seen with the prompts, accounts and number of attempts used on that day. No test on this page shows that a model holds no trace of a record, which is why the stronger control for fine-tuning is keeping personal and confidential records out of the training set.

Start from the map. List every row their data could be in, remove it from each with that row's own method, and search for it afterwards. Vendor copies follow each vendor's terms, and a model fine-tuned on their data needs a decision about retraining or retiring it. Whether a particular request has to be met, and how, is a question for your privacy adviser.

It can be one event described two ways. If a flaw lets one customer read another customer's details through a chatbot, personal information has reached someone it was not meant for. The OAIC's explanation of the Notifiable Data Breaches scheme says an organisation or agency covered by the Privacy Act 1988 must notify affected individuals and the OAIC when a data breach is likely to result in serious harm to a person whose personal information is involved. Whether a particular leak is an eligible data breach is a question for your privacy adviser.

RAG security is one part of the map: who may retrieve which document, how tenants stay apart, and how a deletion reaches the index. This guide covers every place data lives in the application, including the ones retrieval does not touch, such as logs, fine-tuning sets, evaluation sets and vendor stores.

Scope the right test for your application

Tell us what you have built, who uses it and when it needs to launch. You will get a plain-English recommendation: a website audit, a web application test or a referral to a specialist.