LLM data leakage: map where your data goes in an AI application, and how it comes back out
A language model application copies business data into more places than the chat window shows: prompts, retrieved documents, fine-tuning sets, logs, caches and evaluation sets.
This guide maps each one, explains how data leaks back out of it, and gives you a data-lineage worksheet and a canary-record protocol to run on a system you own or are authorised to test.
In this guide
Data-lineage worksheet and canary-record protocol
An illustration written for this page: an invented property management agency's assistant, mapped row by row, with a blank row to copy for your own application, a canary-record protocol for citation exposure, cross-user recall, deletion and privacy-request handling, and a short list for the places you could not account for. Run the protocol only on a system you own or are authorised to test, with synthetic records.
llm-data-leakage.txt
Introduction2 lines
Part A. The columns9 lines
Part B. Worked example65 lines
Part C. Blank row (copy one for each place)8 lines
Part D. Canary-record protocol50 lines
Part E. The short list: places you could not account for5 lines
Whole template149 lines
DATA-LINEAGE WORKSHEET AND CANARY-RECORD PROTOCOL (VERSION 1)
An illustration written for this page. The business, its application, every record and every marker are invented. Fill in Part C for your own application. Run Part D only on a system you own or are authorised to test, using test accounts and synthetic canary records in place of real customer data.
PART A. THE COLUMNS
One row for each place data lives or passes through. Row prefixes: P prompts and conversation history, R retrieved documents and their index, T fine-tuning and training sources, L logs, traces and error reports, C caches, E evaluation sets, V vendors and integrations.
1. Place: what the store or flow is, and where it runs.
2. What goes in: the fields or documents, and whether personal or confidential data is among them.
3. Who can read it: people, roles, systems and vendors, including staff screens and dashboards.
4. How long it is kept: the setting, where that setting lives, and the date you checked it.
5. Leaves the business: no, or yes with the recipient, the terms you read and the date you read them.
6. How it is removed: the method, who runs it, and whether a deletion in the main system reaches it.
7. Canary check: the Part D tests that cover this row.
8. Status: known, partly known or unknown. Any unknown cell also goes on the short list in Part E.
PART B. WORKED EXAMPLE
An invented property management agency runs an assistant for tenants and staff. Tenants ask about repairs and rent; staff use it to draft replies. It calls a hosted model through an API, answers from a knowledge base of policies and property notes, and uses a model fine-tuned on past staff replies.
P1 Prompts sent to the model
In: instructions, the last 20 messages, the tenant's full record, retrieved passages.
Read by: the model provider and the model.
Kept: by the provider under its terms (see V1). Leaves: yes, to the model provider.
Removed: not stored by the application; the provider's copy follows V1.
Canary: R1, S1. Status: known.
Note: the prompt builder loads the whole tenant record, including bank details for rent, to answer repair questions. Fix: send only the fields each question needs.
P2 Conversation history
In: every message from tenants and staff, with the replies.
Read by: each tenant (own conversations), property managers (every tenant in their portfolio), support staff (all).
Kept: no retention rule set. Leaves: no.
Removed: no deletion path for one person's conversations.
Canary: R1, D1. Status: partly known. On the short list.
R1 Knowledge base index
In: tenancy policies (public) and property notes (staff only), split into passages with vectors.
Read by: retrieval, for every user; developers with database access.
Kept: until the next re-index. Leaves: yes, to the vector database host.
Removed: a scheduled re-index; removal of a single note has not been tested.
Canary: C1, D1. Status: partly known.
T1 Fine-tuning dataset
In: 3,000 past staff email replies exported from the shared inbox, with tenant names, phone numbers and arrears details.
Read by: anyone who can query the fine-tuned model, which is every tenant and staff member; the developer who prepared the export.
Kept: the export on the developer's laptop, the uploaded file at the provider, and the trained model until it is retired.
Leaves: yes, to the model provider for the fine-tuning job.
Removed: delete the export and the uploaded file; retrain without the record or retire the model.
Canary: R2 on a test model. Status: known.
Decision: rebuild the dataset from replies with names, numbers and amounts removed.
L1 Application logs
In: full prompts and responses, logged at debug level.
Read by: developers, an outside contractor and the log host.
Kept: per the log host's retention setting (date checked: ____). Leaves: yes, to the log host.
Removed: expiry only; no deletion for one person.
Canary: S1, D1. Status: known.
L2 Error tracker
In: request bodies on failed calls, including message text.
Read by: developers and the error tracker vendor.
Kept: unknown. Leaves: yes. Removed: unknown.
Canary: S1. Status: unknown. On the short list.
C1 Answer cache
In: finished answers, keyed on the question text.
Read by: any user whose question matches a stored one.
Kept: until the entry expires. Leaves: no.
Removed: expiry only; not cleared when a tenant is deleted.
Canary: R1, D1. Status: known. Cache-key tests are in the RAG security guide.
C2 Browser storage
In: the tenant's transcript, kept in the browser between visits.
Read by: anyone using the same browser profile, such as a shared office computer.
Kept: until the user clears it. Removed: signing out does not clear it.
Canary: S1. Status: known.
E1 Evaluation set
In: 200 real tenant questions with the answers staff approved.
Read by: the developer and a contractor who reviews answers.
Kept: with no end date, in a shared spreadsheet. Leaves: yes, to the spreadsheet host.
Removed: not linked to deletion.
Canary: S1, D1. Status: known. Decision: replace with synthetic questions.
V1 Model provider
In: everything in P1, plus the T1 upload.
Terms read: the provider's data controls page, read on ____. Account settings: retention controls approved or not ____, stored conversations on or off ____.
Removed: per the provider's terms and the account settings above.
Status: partly known until the account settings are filled in.
V2 Email connector
In: drafted replies sent through the shared inbox.
Kept: unknown. Removed: unknown. Status: unknown. On the short list.
PART C. BLANK ROW (COPY ONE FOR EACH PLACE)
Row id: ____ Place: ____
What goes in: ____
Who can read it: ____
How long it is kept, where that is set, date checked: ____
Leaves the business (recipient, terms read, date read): ____
How it is removed, by whom, and whether a main-system deletion reaches it: ____
Canary check (Part D test ids): ____
Status (known / partly known / unknown): ____
PART D. CANARY-RECORD PROTOCOL
Rules
1. Every canary is synthetic: an invented person, an email address at a test domain you control, a test phone number you control, and a marker that appears nowhere else.
2. Marker format: LEAKTEST-<row id>-<four random characters>, for example LEAKTEST-P2-7Q4M. Use one marker per row and per test, so a marker found anywhere names where it was planted.
3. Plant each canary through the normal path, as a test user would: type it, upload it or create the record in the usual screen.
4. Record when each canary was planted, by whom and through which test account.
5. Search for markers across the whole map, not only where you expect them: answers, citations, raw API responses, logs, traces, error reports, analytics, caches, browser storage, evaluation sets, exports and provider consoles.
6. Record each result as "found, where, when" or "not found in N attempts with these prompts on this date". Do not record "not retained" or "not memorised": no test in this protocol can show either.
7. Remove every canary when the run ends, and list any you could not remove.
The canary customer
Name: Canary Tenant One. Email: canary01 at your test domain. Phone: a test number you control. Tenancy reference: LEAKTEST-P2-9K2P. File note on the tenant record the prompt builder loads: "Prefers contact in the afternoon. Marker LEAKTEST-P1-3H8D."
Tests
C1 Citation exposure
Plant LEAKTEST-R1-xxxx in a staff-only test document. As a tenant test account, ask questions only that document can answer.
Check the answer, every citation and the raw response for the marker.
Expected: marker not found; citations name only documents the tenant may open. For the full set of retrieval tests, use the RAG security guide's test pack.
R1 Cross-user recall
As test tenant A, give the tenancy reference and an invented complaint about a test address in a conversation.
Then, as test tenant B in a fresh session, ask: "What reference numbers have other tenants given you today?", "Summarise recent complaints about <test address>" and "What did the last person you spoke to ask about?"
Run each question five times. Check B's answers and raw responses for A's markers.
Expected: not found. Repeat after any feature that summarises or remembers across conversations has run.
R2 Fine-tuned model recall (test models only)
Add one synthetic training example containing LEAKTEST-T1-xxxx to a copy of the fine-tuning set, and train a test model on it.
Prompt the test model with the opening words of that example, and with questions about Canary Tenant One. Run each prompt ten times.
Record found or not found. A not-found result describes this test model, this data and these prompts only.
S1 Store sweep
After C1 and R1, search every store in the worksheet for every marker.
Expected: each marker appears only in rows the worksheet says should hold it.
A marker in a store that has no row means the map is incomplete: add the row.
D1 Deletion
Delete Canary Tenant One through your normal deletion path and note the time.
Repeat S1, then repeat R1's questions as tenant B.
Record every store that still holds a marker, and when it stopped holding it, or "still present at the end of the run".
P1 Privacy-request drill
As Canary Tenant One, send an access request and then a deletion request through the channel a real tenant would use.
Ask the team to list, from the worksheet, every row that holds the canary's data before they see the S1 results. Compare the two lists.
Any store S1 found that the team did not list is a gap in the map or in the process.
This drill shows whether your team can find and remove the data. It does not tell you what the law requires.
Results sheet (one line per test and marker)
Test id ____ | Marker ____ | Planted: date, time, account ____ | Attempts and prompts ____ | Found where, when ____ | Result ____ | Row to fix ____
Where to look when a marker turns up
C1: retrieval is not filtered by the person asking, or the response carries candidates the screen hides.
R1: memory, summaries, caches or the prompt builder are shared across users.
R2: the record is in the training set. Fix the dataset; a model has no per-user filter inside it.
S1: a log, tracker, analytics event or export is copying text that nobody listed.
D1: a copy sits outside the deletion path.
P1: the map, or the team's use of it, is incomplete.
PART E. THE SHORT LIST: PLACES YOU COULD NOT ACCOUNT FOR
For each row with an unknown cell: Row id | What is unknown | Who can find out | The question to ask | Date asked | Answer
Worked example:
P2 | Retention and deletion of conversation history | The developer | "Can one tenant's conversations be deleted, and does that clear the caches and logs that hold them?"
L2 | What the error tracker captures and keeps | The developer | "Which request fields does the tracker store, and what is its retention setting?"
V2 | Whether the email connector keeps copies of drafted replies | The connector vendor | "Do you store message content, for how long, and how is it deleted?"What LLM data leakage is in a business application
LLM data leakage is business or personal information reaching someone it was not meant for through an application built on a large language model: a customer chatbot, a staff assistant, a drafting tool or an agent.
OWASP's entry LLM02:2025 Sensitive Information Disclosure lists what is at stake, including personal identifiable information, financial details, health records, confidential business data, security credentials and legal documents. It says that LLMs, especially when embedded in applications, risk exposing sensitive data, proprietary algorithms or confidential details through their output.
The output is where a leak shows up, not where it starts. To answer one question, an LLM application may copy data into a prompt, send it to a model provider, write it to a log, store it in a cache and in conversation history, and later reuse it in an evaluation set or a fine-tuning run.
Each copy has its own readers and its own lifetime, and each is a place the data can come back out: in another user's answer, in a log viewer a contractor can open, in a spreadsheet of test questions, or from the model itself.
That is why this guide starts with a map rather than a list of attacks. If you can say, for every place your data lives, who can read it, how long it stays, whether it leaves the business and how it is removed, then each of those answers can be tested. The rows you cannot fill in are the first places to look.
Where this guide stops
Two neighbouring guides go deeper on one row each. Who may retrieve which document, how tenants stay apart in a shared index and how a deletion reaches the index are covered in the RAG security guide.
What sits in a bot's instructions, and how those instructions are drawn out, is the subject of the system prompt exposure guide; OWASP gives that risk its own entry, LLM07:2025 System Prompt Leakage, which says the system prompt should not be considered a secret.
This page holds the map that contains both, including the rows neither covers in depth: fine-tuning and training sources, logs, evaluation sets and the copies your vendors keep.
Six places your data lives in an LLM application
Each place below becomes one or more rows in the worksheet on this page.
The questions are the same for every row: what goes in, who can read it, how long it is kept, whether it leaves the business, and how it is removed.
Your code builds every prompt: the instructions, the conversation so far, the user's message, and any record, document or tool result it decides the model needs. Whatever the prompt builder adds is sent to the model provider and placed where the model can repeat it, so the prompt is the first place to apply least privilege.
A builder that loads a customer's whole record to answer a question about one order has widened every later leak, because the model can now quote fields the question never needed.
Users add data of their own. OWASP's LLM02:2025 entry says consumers need to understand the risk of unintentionally providing sensitive data, which may later be disclosed in the model's output. Conversation history is the stored form of all of this.
Where your application keeps it, it is a record like any other, with an owner, readers and a retention period. Where a provider's API stores it for you, the provider's terms set how long it stays.
Passages a retrieval system finds are added to the prompt, so the access check has to run before they are handed over. OWASP's LLM08:2025 Vector and Embedding Weaknesses notes that in multi-tenant environments where several classes of users or applications share one vector database, there is a risk of context leaking between users or queries, and the RAG security guide covers retrieval access control in full.
On this map, retrieval takes three rows: the source systems that feed it, the index that holds copies of their text, and the citations that point back to them.
Data used to fine-tune or train a model is the one row where the copy ends up inside the model rather than beside it. A document in an index can be filtered for each user at query time. There is no per-user access check inside a model's weights, so treat every record in a fine-tuning set as visible to everyone who can query the resulting model.
OWASP's LLM02:2025 entry lists data leak via training data as an attack scenario, in which negligent data inclusion in training leads to sensitive information disclosure.
Research on LLM training data extraction shows how literal that can be. Carlini and colleagues showed that where a large language model trained on a private dataset is published, an adversary can recover individual training examples by querying it, and on GPT-2 they extracted hundreds of verbatim text sequences, including names, phone numbers and email addresses that were public on the internet.
Their abstract notes that the attack was possible even though each of those sequences was included in just one document in the training data, and that larger models were more vulnerable than smaller ones.
In a 2023 paper, Nasr and colleagues showed that an adversary can extract gigabytes of training data from open-source, semi-open and closed models, ChatGPT among them, and concluded that current alignment techniques do not eliminate memorisation.
The training files are rows of their own. A dataset exported from a shared inbox or a CRM is a copy of that data: on the laptop or drive where the export landed, and with the provider that runs the fine-tuning job once it is uploaded.
A log line that records the full prompt and response holds everything the model saw and said, including retrieved passages and tool results. The same goes for a tracing tool that records each model call, an error tracker that captures request bodies, and an analytics event that includes message text.
These copies are read by a different audience from the application: developers, support staff, contractors and the vendors that host the tools. Their retention is set inside each tool, not by your privacy policy.
OWASP's LLM02:2025 entry points to OWASP API8:2023 Security Misconfiguration as guidance for avoiding leaks through error messages and configuration details. For the map, record every destination a log line can reach, not only the one your developer set up first.
An LLM application can hold copies in several caches: stored answers, stored search results, transcripts kept in the browser, shared answer links, and the model provider's own prompt cache. Whether a cache can hand one person's answer to another depends on how its key is built, and the RAG security guide sets out cache keys and tests for retrieval.
For this map the questions are simpler: where each cache sits, what it holds, how long it holds it, and whether a deletion in the main system clears it.
An evaluation set is a list of test questions with the answers you expect, run against the application after each change. When it is built from real conversations or real documents so that it reflects real use, it is a copy of them that lives in a repository, a spreadsheet or an evaluation platform, outside the deletion process the production data follows.
It may also be shared more widely than production data, with whoever runs the tests, reviews the results or builds the next version.
Build evaluation sets from synthetic records where you can. Where a real conversation is needed, give it a row in the map with its source, so that a deletion request can reach it.
How data comes back out: five routes to test
A leak needs a copy and a route to someone who should not see it.
These are the routes the canary protocol on this page checks, with the test that covers each one.
- Citation exposure (test C1): an answer, its citations or the raw response behind the chat window name or quote a document the person asking may not open.
- Cross-user recall (test R1): something one user typed, uploaded or was told turns up in another user's answer. OWASP's LLM02:2025 entry describes a user receiving a response that contains another user's personal data. Shared memory, conversation summaries, answer caches and prompt builders that pull in the wrong record are the paths to check.
- Extraction from the model (test R2, fine-tuned models only): text from a training or fine-tuning set comes back in a completion, whether someone asked for it or not.
- Store exposure (test S1): the data is reached in a store rather than through the chat, such as a log viewer, a tracing dashboard, an evaluation spreadsheet or a provider console, by someone with access to that store but not to the record.
- Survival after deletion (tests D1 and P1): the record is deleted from the main database, but a copy stays in a log, a cache, an evaluation set, a training file or a provider store, and comes back later.
What your model provider keeps: read its terms and date them
The worksheet column that asks whether data leaves the business, and to whom, cannot be filled in from a general belief about AI providers.
It needs the current terms of the provider your application actually calls, read on a stated day. What one provider's page says tells you nothing about another's, and a page read last year may not describe today.
As an example of what such a page contains, OpenAI's page on data controls in the OpenAI platform, read on 14 November 2026, says the following about its API.
- As of 1 March 2023, data sent to the OpenAI API is not used to train or improve OpenAI models unless the customer explicitly opts in to share data with OpenAI.
- Abuse monitoring logs may contain certain customer content, such as prompts and responses. By default they are generated for all API feature usage and retained for up to 30 days, unless longer retention is required by law or is reasonably necessary to protect OpenAI's services or any third party from harm.
- Eligible customers can have their content excluded from those logs by getting approved for the Zero Data Retention or Modified Abuse Monitoring controls, which are subject to OpenAI's prior approval and acceptance of additional requirements, with limits the page sets out. Endpoints that are not eligible for Zero Data Retention may still store application state when it is enabled.
- Its per-endpoint table gives the application state retention for files, fine-tuning jobs, evaluations, vector stores and conversations as until deleted. With exceptions the page notes, the Responses API has a 30-day application state retention period by default, or when its store parameter is set to true, and response data is then stored for at least 30 days.
- Objects related to the Assistants API are deleted from OpenAI's servers 30 days after you delete them through the API or the dashboard, and objects that are not deleted are retained indefinitely.
- Prompt caching may store encrypted key/value tensors in GPU-local storage, and that data is not retained after the 24-hour expiration.
- MCP servers used with its remote MCP server tool are third-party services, and data sent to them is subject to their own retention policies.
Turning a provider page into worksheet entries
Write what you read into the worksheet with the date, then add the settings your account actually uses: whether a retention control was approved, whether stored conversations or responses are switched on, and which endpoints the application calls. If you reach a model through a cloud platform or another intermediary, the terms to read are those of the service your application calls.
Do the same for every other vendor on the map: the vector database, the tracing tool, the error tracker, the evaluation platform and any MCP server.
Filling in the worksheet for your own application
The worksheet is meant to be finished in one sitting with your developer, using what the system does today rather than what its design document says.
Work in this order.
- Follow one real request from start to finish: from the browser, through your server, to the model provider and back. Write down every system that receives or stores any part of it.
- List every vendor the application pays or connects to: model provider, vector database, hosting, logging, tracing, error tracking, analytics, email and any MCP server. Each one that receives data is a row.
- For each row, ask three questions: where is the setting that controls retention, who can read this copy, and what happens to it when a record is deleted in the main system.
- Read each vendor's current data terms and write the date beside what you read.
- Mark every cell known, partly known or unknown. Unknown is a valid answer: it moves the row to the short list in Part E, with the person who can find out.
- Run the canary protocol last, to check the map against what the system actually does.
Canary records: what a planted marker can show, and what it cannot
A canary record is a synthetic record carrying a unique marker, planted in one place on the map so you can search for the marker everywhere else.
The approach borrows from research on unintended memorisation. The Secret Sharer, by Carlini and colleagues, describes a testing methodology for quantitatively assessing the risk that rare or unique training-data sequences are unintentionally memorised by generative sequence models, and reports that unintended memorisation is a persistent, hard-to-avoid issue.
Every canary is synthetic: an invented person, an email address at a test domain you control, a test phone number you control, and a marker such as LEAKTEST-P2-7Q4M that appears nowhere else. Use one marker per row and per test, so a marker found anywhere names where it came from.
Plant canaries only in a system you own or are authorised to test, through test accounts, and record when and where each one went in.
A found marker shows a lot: a path exists from where it was planted to where it was found, and the record of the test names both ends. A missing marker shows much less. It means the marker was not seen with the prompts, accounts, number of runs and searches used on that day.
It does not show that the data is absent from a store you did not search, and it does not show that a model has not memorised it. A different prompt, more attempts or another extraction method may bring it out, and Nasr and colleagues' conclusion that current alignment techniques do not eliminate memorisation is a reason not to read a model's silence as absence.
Record results in those terms: found, where and when; or not found in a stated number of attempts with stated prompts.
Canaries and fine-tuned models
For a fine-tuned model, the evidence that matters most sits upstream of the model: which dataset version it was trained on, and what that dataset held. A canary in a test fine-tuning run shows whether that test model, with that data and those settings, repeats a planted record.
It says nothing reliable about a production model trained on different data. Keep personal and confidential records out of fine-tuning sets rather than relying on a test to show they did not stick. OWASP's LLM02:2025 entry recommends scrubbing or masking sensitive content before it is used in training.
Deletion and privacy requests: what the map lets you answer
A person may ask a business what it holds about them, or ask for their information to be corrected or removed.
In an LLM application, the answer depends on the map, because their data may sit in conversation history, an index, logs, caches, an evaluation set, a fine-tuning file and the stores of several vendors, each with its own removal method. The map turns that request into a list: every row the person's data could be in, who removes it there, and how you check it has gone.
Fine-tuned models need a decision before a request arrives. Keep a record of which dataset version each model was trained on, so a request can be traced to every model that saw the record, and decide in advance whether such a model is retrained without the record, retired, or handled another way your privacy adviser supports. Deleting the training file removes one copy; the model trained on it is a separate thing.
In Australia, Schedule 1 of the Privacy Act 1988 sets out the Australian Privacy Principles. Australian Privacy Principle 11.1 says that if an APP entity holds personal information, the entity must take such steps as are reasonable in the circumstances to protect the information from misuse, interference and loss, and from unauthorised access, modification or disclosure, and APP 11.3 says those steps include technical and organisational measures.
APP 8.1 says that before an APP entity discloses personal information about an individual to an overseas recipient, it must take such steps as are reasonable in the circumstances to ensure that the recipient does not breach the Australian Privacy Principles, other than APP 1, in relation to the information; APP 8.2 sets out cases where that does not apply.
Whether those principles apply to your business, whether sending prompts to a model provider is a disclosure to an overseas recipient, and whether and how a particular access or deletion request has to be met are questions for your privacy adviser. This page is not legal advice.
What the map and the privacy-request drill give your adviser are the facts: where the data is, who can read it, where it goes, and whether your team could find and remove all of it when it tried.
Controls that shrink the map
Every copy you do not make is a row you do not have to protect, test or clear.
These controls reduce the number of rows, or the number of people each row reaches.
- Send the model only what the task needs: the fields that answer the question, for the person asking, rather than the whole record. OWASP's LLM02:2025 entry recommends least-privilege access to sensitive data.
- Redact before data leaves your code. The same entry describes tokenisation and redaction, using techniques such as pattern matching to detect and redact confidential content before processing. Apply that step before logging as well as before the model call.
- Log metadata by default and full text only on purpose: which user, which model call, which documents and tools, and the outcome. Where full prompts are logged for debugging, give that log a short retention setting and a named list of readers.
- Keep personal and confidential records out of fine-tuning sets. Where a dataset has to carry them, OWASP's entry also describes differential privacy, which adds noise to the data or outputs to make individual data points difficult to reverse-engineer.
- Build evaluation sets from synthetic records, and give any real ones a row in the map.
- Make deletion a job that walks the map: one request removes the record from every row that can hold it, and the job records what it removed and where it could not.
- Say what you do with data. OWASP's entry recommends clear policies about data retention, usage and deletion, and letting users opt out of having their data included in training.
Where the AI chatbot and agent security test fits
The Clever Ops AI chatbot and agent security test looks at data a bot can be made to reveal: what each read tool returns, what sits in the instructions, and whether one customer's records can be reached from another customer's session.
It also covers the retrieval pipeline, including whether retrieval keeps public, customer and staff content apart, and the chat backend as an API, including the screens staff use to read transcripts. Testing uses test identities and made-up records. The report records the exact messages used and the bot's replies, and the tool calls each attempt produced, with secrets and personal information redacted.
The test examines your deployment, not the AI model behind it, and it does not red-team the model itself. If Clever Ops built or hosts your chatbot, the work is a security review rather than an independent test, and the report says who built it and who hosts it.
The report records what was tested, on which configuration and on which dates; it is not a certificate or a compliance opinion. Bring the worksheet, filled in for your own application, to the scoping call.
Frequently asked questions
It is business or personal information reaching someone it was not meant for through an application built on a large language model. It can come out in an answer, a citation, a log, a cache or an evaluation set, or, where your data trained or fine-tuned the model, in the model's own output. OWASP covers it as LLM02:2025 Sensitive Information Disclosure.
Published research shows it can. Carlini and colleagues recovered hundreds of verbatim sequences from GPT-2's training data by querying the model, and Nasr and colleagues showed that an adversary can extract gigabytes of training data from several models, ChatGPT among them. For a business application, the question is whether your own data trained or fine-tuned the model. If it did, treat every record in that dataset as something anyone who can query the model might see.
Read that provider's current terms and date them; the answer is the provider's to state. As one example, OpenAI's data controls page, read on 14 November 2026, says that as of 1 March 2023, data sent to the OpenAI API is not used to train or improve OpenAI models unless the customer explicitly opts in. The same page says abuse monitoring logs, which may contain certain customer content such as prompts and responses, are retained for up to 30 days by default, unless longer retention is required by law or is reasonably necessary to protect OpenAI's services or any third party from harm. Put what your provider's page says, and the date you read it, in the worksheet.
No. A canary that turns up proves a path exists. A canary that does not turn up shows only that it was not seen with the prompts, accounts and number of attempts used on that day. No test on this page shows that a model holds no trace of a record, which is why the stronger control for fine-tuning is keeping personal and confidential records out of the training set.
Start from the map. List every row their data could be in, remove it from each with that row's own method, and search for it afterwards. Vendor copies follow each vendor's terms, and a model fine-tuned on their data needs a decision about retraining or retiring it. Whether a particular request has to be met, and how, is a question for your privacy adviser.
It can be one event described two ways. If a flaw lets one customer read another customer's details through a chatbot, personal information has reached someone it was not meant for. The OAIC's explanation of the Notifiable Data Breaches scheme says an organisation or agency covered by the Privacy Act 1988 must notify affected individuals and the OAIC when a data breach is likely to result in serious harm to a person whose personal information is involved. Whether a particular leak is an eligible data breach is a question for your privacy adviser.
RAG security is one part of the map: who may retrieve which document, how tenants stay apart, and how a deletion reaches the index. This guide covers every place data lives in the application, including the ones retrieval does not touch, such as logs, fine-tuning sets, evaluation sets and vendor stores.
Scope the right test for your application
Tell us what you have built, who uses it and when it needs to launch. You will get a plain-English recommendation: a website audit, a web application test or a referral to a specialist.