AI data leakage in healthcare: what leaves when patient records go into ChatGPT or Claude.

A medical practice holds the most sensitive records most people will ever have written about them: diagnoses, medications, mental health, family history and what was said in the room. Under the Privacy Act health information is sensitive information, and a private business that provides a health service and holds health information is covered whatever its turnover. In NSW the Health Records and Information Privacy Act 2002 adds its own principles, and Good medical practice asks doctors to keep records accurate and secure and information confidential. This series takes ten documents a general practice with allied health produces every week, marks every item those obligations reach, and shows where each one goes when the document is pasted into ChatGPT or Claude. Each page ends with the same document on Ermos, inside the practice.

The documents, one at a time

LetterThe GP referral letter

Identity, symptoms, history, medications, results and a bereavement. A patient’s whole clinical picture, in one page.

See what leaves → NotesThe GP consultation note

Symptoms, alcohol, a family cancer history, an abnormal result and an apology. What the patient said before it became a plan.

See what leaves → FormThe new patient intake form

Identity, Medicare, mental health history, drug use, an emergency contact and a choice about AI. The first page of the record.

See what leaves → EmailThe internal abnormal pathology result email

A patient, the values, a suspected cancer and how to call without telling the household. Written in a hurry before leave.

See what leaves → RecordThe GP mental health treatment plan

Scores, a diagnosis, a risk assessment, substance use and who must not be told. The most intimate page in the record.

See what leaves → SpreadsheetThe billing and recall spreadsheet

Names against birth dates, mobiles, item numbers, condition codes and notes. A whole patient list, one row each.

See what leaves → TranscriptThe AI scribe consultation transcript

The consent line, the symptoms, a bereavement, drinking, a family dispute and another patient’s name. Every word as spoken.

See what leaves → Report or memoThe medical report to an insurer

A claim number, a diagnosis, the treatment, prior depression, drinking, an opinion on work capacity and the patient’s finances.

See what leaves → Text or chat messageThe appointment and recall SMS thread

A name and number, a doctor, a blood test, a family cancer history and a request to stop texting. A recall, in eight lines.

See what leaves → LetterThe patient complaint response

The patient, a missed follow-up, the findings, the fixes, the indemnity notice and where to complain next. A bad week, in writing.

See what leaves →

How the leakage happens

The mechanism is the same for every document. The whole text is transmitted to the provider’s servers. On personal plans the user cannot choose where it is stored or processed, and for the major providers that has generally meant outside Australia. On consumer tiers it can be retained and used to train future models. The provider’s staff and systems can reach it under the provider’s own policies. Once sent, the practice no longer controls it. Our on-premise vs cloud comparison sets out the four ways to run AI and where each one puts your documents.

The obligations in play

Each marked item on these pages links to the guide that covers its obligation:

The same documents on Ermos

Ermos Dominion drafts and searches on a unit in your office, so the documents never leave it. Ermos Edge keeps your documents on your own instance in an Australian data centre and sends only permitted excerpts, per question, to a named Australian provider. Every page in this series ends with both, item by item.

Common questions

Can healthcare practices use ChatGPT or Claude with patient information?
Not without meeting the practice's privacy duties, which still apply to every paste. The Office of the Australian Information Commissioner (OAIC) recommends in its guidance on commercially available AI products that organisations do not enter personal information, and particularly sensitive information, into publicly available generative AI tools, because of the significant and complex privacy risks. Health information is sensitive information under the Privacy Act, and a business that provides a health service and holds health information is generally covered even if it is small. APP 6 limits using or disclosing it for another purpose unless the patient consents or an exception applies, and Ahpra says the practitioner remains responsible and must apply human judgment to any AI output.
Do ChatGPT and Claude train on what staff paste in?
It depends on the account each person is using. As at October 2026, whether a conversation on a personal plan (ChatGPT Free, Plus or Pro; Claude Free, Pro or Max) can be used to train the models depends on a setting each user controls. Business plans (ChatGPT Business and Enterprise, Claude Team and Enterprise) and the developer APIs sit under commercial terms, and OpenAI and Anthropic each say they do not train on that content by default (OpenAI, Anthropic). On either product, the thumbs up and thumbs down buttons can send a whole conversation back for training. For a medical practice the question matters because what staff paste is a patient's diagnosis, medications and what was said in the consulting room.
Do business plans such as ChatGPT Business or Claude Team solve the problem for healthcare practices?
Only in part. A business plan changes the default on training, but it does not change who receives the content. On a personal plan the user cannot choose where conversations are stored or processed. Some business plans let an organisation choose to store data at rest in Australia (OpenAI), which still leaves the content with the AI company. The OAIC's guidance also tells organisations to review carefully the terms and settings that will apply to their use of any AI product. For a practice, a referral or a mental health treatment plan would still arrive at that AI company in full, and Ahpra's guidance asks practitioners to know where patient data is located. Our comparisons with ChatGPT Enterprise and Claude weigh the trade-offs.
What is the safest way for a medical practice to use AI with patient documents?
Ermos is built so the document stays under your control. On the Dominion AI unit it is drafted inside your office and none of it is sent to an AI provider. On Ermos Edge your documents and index stay on your own instance in an Australian data centre, and only the question and the excerpts the user is permitted to see go to a named Australian provider that is contractually barred from keeping them or training any model on them. For a healthcare practice, staying in control means patient records stay where the practice can protect them, and a doctor checks every draft that reaches the record.
What is AI data leakage in a healthcare practice?
Patient information leaving the practice's control through an AI tool. When a referral, a note or a transcript is pasted into ChatGPT or Claude it is transmitted to the provider's servers, can be retained on the provider's terms, can be used to train future models on consumer tiers, and can be reached by the provider's staff and systems. What leaks is a patient's health, often with their family and employer.
Does the Privacy Act apply to a small medical practice?
Generally yes. Section 6D(4)(b) of the Privacy Act takes a business out of the small business exemption if it provides a health service to another individual and holds health information other than in an employee record. The OAIC says the same: an organisation that provides a health service and holds health information is covered, even if it is a small business.
What do Ahpra and the RACGP say about AI in practice?
Ahpra says the practitioner remains responsible, must apply human judgment to AI output, should know where patient data is stored and whether it trains the model, and should obtain informed consent for AI scribes. The RACGP asks practices to get patient consent at the start of each consultation using a scribe and ideally to have the data stored in Australia.

See Dominion answering from your own documents.

A demonstration takes 30 minutes, uses no client data, and comes with no obligation.

Book a demo
Keep reading
AI for healthcare The Privacy Act, the APPs and AI All professions