Guardrails for AI in a medico-legal file: what the software must never do, and where each rule has to live.

Australia decided in December 2025 not to legislate mandatory guardrails for AI. The National AI Plan relies on existing laws and the regulators who already enforce them, supported by voluntary guidance, which means the guardrails around a medico-legal file are whatever the practice and its software actually build.¹ The word gets used loosely in vendor material, so the useful question about any guardrail is a practical one: when the model does not cooperate, what stops the bad outcome?

Where a rule is enforced decides whether it holds

Every rule an AI product claims to follow lives somewhere. It can live in the prompt, as an instruction the model is asked to obey. It can live in code, as a check that runs on the model's output whatever the model wrote. It can live in the absence of a feature, where the risky capability simply cannot be reached. Or it can live in the workflow, as a step a person takes before the output is relied on. These places carry very different weight, because a language model follows instructions most of the time and not all of the time.

Prompt-level rules are still worth having as a first layer that keeps most output clean, but they cannot be the only layer for anything that matters. A buyer should ask which row of that table each item on a vendor's guardrail list sits in. The National AI Centre's guidance on AI adoption points the same way, asking organisations to test and monitor their AI systems and to keep a person able to intervene, rather than trusting a system to behave as designed.²

The opinion is the one output the software must not produce

The firmest line in medico-legal AI runs around the opinion itself. The Personal Injury Commission's procedural direction allows generative AI for work such as chronologies, indexes and summaries of documents, and prohibits using it to generate the content of material meant to reflect a person's evidence or opinion.³ An expert's opinion is the expert's evidence, so a tool that drafts it, or proposes a diagnosis, a causation finding or an impairment percentage, produces exactly the content the rules reserve to the person.

The human factors evidence explains why "the doctor reviews the draft" is weaker protection than it sounds. A systematic review of automation bias by Lyell and Coiera found it in single tasks with high verification complexity, and concluded that it tracks the cognitive load of the decision task.⁴ A drafted opinion is the most expensive thing in the file to verify, because checking it properly means doing the reasoning again, so it invites anchoring exactly where the stakes are highest.

The strongest guardrail here is therefore absence. An instruction telling the model not to give opinions is a sensible first layer during extraction, and the reliable barrier sits behind it: a product with no drafting path at all, with no button, no route on the server and nothing for a tired examiner to accept at eleven at night.

Every fact carries its page, and the page is checked

The second guardrail concerns the facts the opinion will rest on. A chronology entry reading "MRI 12 March 2024: L4/5 disc protrusion abutting the L5 nerve root" is useful only if it also says where that sentence came from, and only if the citation holds when the examiner repeats it under cross-examination. A language model can produce a fluent entry with a plausible page number that is twenty pages out, or a date that appears nowhere in the record.

Grounding means the software checks the model's work against the source instead of accepting it as written. In practice that check should:

• confirm that each quoted passage actually appears on the page it cites, and correct the page when the passage turns up nearby;
• mark an entry as unverified, visibly, when the passage cannot be found on its cited page, so it never carries the same confidence as a checked one;
• drop a quotation that appears on no page the software read;
• keep a date off the timeline when the date itself cannot be found in the record.

Even a good check leaves the examiner with work to do, and a vendor that calls its facts "verified" should be asked what the word means in code. Confirming that a quotation sits on a page says nothing about whether the summary beside it states the quotation fairly. An honest product presents its checks as aids to the examiner's own reading and keeps the source page one click away.

Conflicts and gaps are shown, never resolved

Medical records disagree with each other routinely. The GP's letter gives one date of injury and the employer's incident report another, while two range-of-motion measurements taken a month apart differ by thirty degrees. Each disagreement may matter to the opinion, and deciding which version to prefer is part of the expert's job.

Software that silently picks one reading, merges two entries into a tidy single line, or orders the file by what a model judged relevant has made that decision without telling anyone. The guardrail is to show both readings side by side with their sources, to disclose plainly which pages were never read, and to keep any ordering of the file rule-based and explainable, so that the examiner can always reach the document a rule placed lower down.

The brief is untrusted input

A 900-page brief is assembled from dozens of authors the practice has never met, and a language model reads every word of it. OWASP ranks prompt injection first among the risks for LLM applications and describes an indirect form, where content in a file or website the model processes alters its behaviour in unintended ways.⁵ A planted instruction is unlikely in a typical brief, but the brief is exactly the kind of external content that risk describes, and it arrives in volume.

The defences OWASP recommends fit a chronology tool well: separate and clearly mark the untrusted content, constrain what the model is asked to do, define the output format and validate against it, and give the model the least access it needs.⁵ A model that can only return structured entries for code to check, and cannot send email, change a matter or reach the internet, leaves an injected instruction very little to work with. The National AI Centre's guidance names testing for prompt manipulation among its risk-proportionate evaluations for the same reason.²

Where the file goes, and what it trains

The last group of guardrails concerns the data itself, and here the OAIC has been specific. Its guidance on commercially available AI products asks organisations to check whether a product's terms give the developer access to the data entered, to consider where the servers are located and whether personal information could leave Australia, and to keep a person responsible for verifying the accuracy of personal information the AI produces.⁶ For material produced under compulsion in proceedings, PIC13 adds that it may go into a generative AI program only if it stays in a controlled environment, is used only for the proceeding and is not used to train a model.³

Turned into questions a practice can put to a vendor in writing, those obligations become a short list:

• Where are text recognition and AI inference run, and does any step use an outside AI service?
• Is any of our content, including the corrections our staff make, used to train or tune a model?
• Who at the vendor can open a brief, and is multi-factor sign-in mandatory for everyone who can?
• Is every read and change recorded, and how would the practice obtain that record?
• When is each copy deleted, backups included, and is that schedule written into the terms?

Sort the guardrail list by where each rule lives

A vendor's guardrail list is a set of claims, and the practical response is to sort it. Rules enforced by absence or by code hold when the model misbehaves, rules held only in a prompt hold most of the time, and rules that depend on review hold as well as the reviewer's time allows. A practice that knows which is which can put its own effort where the software is thinnest, and can explain its arrangements when a court or tribunal asks how AI touched the file.

Trenthos IME is built to those lines. Report and opinion drafting is switched off at the server, so the product produces a page-referenced chronology, a document index and a topic view. A quotation that cannot be found on its cited page is marked unverified, conflicting readings sit side by side, and AI inference runs on hardware Trenthos controls in Australia. No case content trains a model. The documentation sets out each control.