Privacy-preserving medical AI: federated learning, synthetic data, and the limits of de-identification.

Most of the medical AI you read about was trained somewhere else, on someone else's patients, and that is not a footnote but the central problem.

A model learns the population it was trained on, which means a retinal model built largely on overseas cohorts encodes the disease prevalence, the imaging hardware, and the demographics of those cohorts, and then quietly underperforms on the population sitting in front of an Australian clinician. This is not a hunch, because external-validation studies repeatedly show models that look strong on their home data losing accuracy when they meet a new population, a different camera, or an ethnic mix they did not see in training.¹² The reflexive fix is to use more data, but the harder and more honest problem is using it without moving patients' records out of the institutions that hold them, or out of the country.

There is a small toolkit for that problem, and although none of the tools is magic and each is routinely oversold, taken together and understood for what they cannot do they describe a credible way to advance medical AI without trading away privacy to get there.

De-identification is a discipline, not a guarantee

Stripping names and dates feels like it should make a dataset safe, but on its own it does not, because re-identification works by linkage. A de-identified record, combined with an external source such as an electoral roll, a hospital admission date, or a rare diagnosis in a small town, can narrow a "de-identified" row down to one person.

Australia has already run this experiment in public. In 2016 a government department released a de-identified dataset of Medicare and pharmaceutical claims covering roughly a tenth of the population, and researchers soon showed it was not as protected as it looked, because supposedly protected service-provider identifiers could be unpicked and individual patients could in principle be re-identified by linking the records against other known facts³, so the dataset was pulled within a day. The privacy regulator later found the release had breached the Privacy Act.⁴ Nothing was wrong with the de-identification in the naive sense, since the names were gone, but it simply was not robust against a determined attacker holding outside information, which is the threat that actually matters.

This is exactly why the careful word is de-identified and never "anonymous", because anonymous asserts that re-identification is impossible whereas de-identified asserts only that it has been made suitably difficult for a defined context and a defined threat model. The first is a promise no one can keep, while the second is an engineering statement you can defend. Techniques such as k-anonymity⁵ formalise that idea by ensuring any record is indistinguishable from at least k-1 others across the quasi-identifiers an attacker might know, and refinements like l-diversity⁶ add that the sensitive values in each group must vary. These techniques help, but they also collapse on free-text clinical notes and raw images, where the identifying detail hides in the content itself and no amount of column-generalisation reaches it.

Differential privacy: a budget for what can be revealed

Differential privacy reframes the question, because instead of asking whether a dataset is safe it asks how much any single person's presence changes what you publish, and it bounds that change with calibrated noise.⁷ The parameter usually written as epsilon is a privacy budget, so an epsilon near 1 is generally considered strong protection while values up in the double digits offer little. That budget is cumulative, which means every query or training run spends some of it and you cannot earn it back, and it is this accounting, rather than the noise itself, that is the real discipline, because it forces you to treat each look at the data as a withdrawal from a finite account.

In training, differential privacy shows up as DP-SGD⁸, where you clip the gradient contributed by each example and add calibrated noise so that no single patient can move the model by more than a bounded amount, which lets the model learn the population without memorising the individual.

This protection is not costless, and the cost is not evenly spread, because the noise lands hardest on the tail. The rare diagnosis carried by a handful of patients is precisely the signal differential privacy is designed to blur, and it is precisely the one medicine often cares about most, so differential privacy is a dial with a real exchange rate between privacy and accuracy rather than a switch you flip to "private" and forget.

Federated learning: move the model, not the data

Federated learning is simple and genuinely powerful, because instead of pooling everyone's records into one central dataset, you send the model out to each site, train it locally on data that never moves, and combine only the learned updates,⁹ so the raw records stay where they are. This dovetails with data residency, because if the records never leave the premises then a large part of the offshore-transfer problem simply dissolves, since there is no transfer to govern.

But the shared updates are not safe by default, and federation is too often sold as if they were. A line of research beginning with "Deep Leakage from Gradients"¹⁰ showed that the gradients exchanged during training can be inverted to reconstruct the original training examples, in some cases recovering input images close to pixel-perfect. Because of that, serious systems add secure aggregation, a cryptographic scheme in which the server only ever sees the sum of many sites' updates and never any single site's,¹¹ and frequently differential privacy on top of that. Federation answers where the data is, but it does not by itself answer what the shared gradients reveal, which takes the other tools layered on.

Synthetic data: useful, bounded, not magic

Synthetic clinical data, generated to mirror the statistical shape of real records while aiming not to reproduce any individual patient, is genuinely useful, because it is good for software testing, for sharing the structure of a dataset, and for augmenting cases that are too rare to learn from directly.

The bound is that a generative model trained on real patients can memorise and regurgitate them, and membership-inference attacks can sometimes establish whether a specific person was in the training set at all.¹² Synthetic data is therefore only as private as the process that produced it, so that process deserves the same scrutiny as any other release, often including differential privacy in the generator itself, so that no single real record can leave too strong a fingerprint on the synthetic output.

A model trained purely on synthetic data also inherits the synthesiser's blind spots, because it cannot contain signal the generator never captured, and it can quietly amplify the generator's biases. Synthetic data is therefore a strong complement to real-world evidence, but it is not a replacement for it.

Why this matters more here

Australia is a smaller, distinct population with its own disease patterns, its own fleet of imaging hardware, and a clear legal and cultural preference for keeping health data on-shore, which sharpens the dilemma. Importing a model trained elsewhere imports its blind spots, while exporting our data to train one imports a privacy and sovereignty problem, and as the 2016 episode showed, "de-identified and released" is not a safe default. For a long time those have looked like the only two options on the table.

The techniques above are how you refuse that choice, because the lesson of the withdrawn dataset is not to de-identify harder but to bring the computation to the data rather than the data to the computation. That means federated learning so the records stay put, differential privacy so what leaves is accounted for, careful de-identification with its limits stated plainly, and privacy-aware synthetic data where it fits. No single one of these is sufficient, but combined and honestly bounded they are a path.

What we're building toward

Trenthos Research treats these as the constraints we are designing toward, which means keeping health data in Australia, working with de-identified data where the research allows, and only ever using patient information in a way consistent with our Privacy Policy and the agreements that govern each clinic. We describe this as a direction we are building toward, so it names research themes we are actively working in, not delivered systems or published results.

The claim worth making is modest and real, because you do not have to choose between better tools and strong privacy, but you do have to engineer for both from the start. The work is therefore in the combination: federated training under a tracked privacy budget, de-identification whose limits are stated rather than assumed, and being explicit about the point where each technique stops working.