Skip to content
Markdown

Membership inference against fine-tuned LLMs

Scope: what an adversary with query access learns about the data used to fine-tune a served model. This page covers why membership inference barely works against pre-trained models but works well against fine-tuned ones, which serving and training knobs move exposure, why the headline metric in this literature understates operational risk, what the selective-obfuscation defence costs against differential privacy, and where the released implementation departs from the published algorithm. The training-time counterpart, where the leak is the gradient rather than the served model, is gradient leakage in distributed training; the inference-time counterparts are privacy-aware split inference and covariant obfuscation.

Primary source: Kaiyuan Zhang, Siyuan Cheng, Hanxi Guo, Yuetian Chen, Zian Su, Shengwei An, Yuntao Du, Charles Fleming, Ashish Kundu, Xiangyu Zhang, Ninghui Li, "SOFT: Selective Data Obfuscation for Protecting LLM Fine-tuning against Membership Inference Attacks", arXiv:2506.10424, USENIX Security 2025 (Purdue University and Cisco Research). Code at https://github.com/KaiyuanZh/SOFT, MIT licensed, read at commit 7ca2b7b3b44e352e8ade6bea62889156fe1bff94.

What this page adds. The Python block is executed and asserted (Python 3.13, numpy 2.4.6). It implements both metrics from scratch, reproduces every published aggregate in the paper's Tables 1 through 4, and then constructs the case that motivates using the second metric at all. The section comparing paper against released code is this page's reading of the repository, not the paper's description of itself.

Scale matters when reading these results. The defence experiments use 1,000 fine-tuning samples against 1,000 held-out samples, on Llama-3.2-1B and 3B, with the leakage analysis on Pythia models up to 6.9B. Nothing here has been measured at production fine-tuning-set sizes.

Overview

Membership inference asks whether a specific record was in a model's training set. Against pre-trained LLMs the attack mostly does not work, and the literature has converged on why: each document is seen roughly once during pre-training, so there is little to overfit to, and several early positive results turned out to be measuring a distribution shift between the member and non-member splits rather than memorisation.

Fine-tuning breaks that. A fine-tuning run passes over a small dataset several times with a learning rate chosen to move the model, which is exactly the regime that produces a measurable loss gap between seen and unseen text. On the same models and the same datasets, attacks that score near chance against the pre-trained checkpoint reach a ten-attack average of 0.819 after three epochs of full fine-tuning, with the strongest single attack between 0.943 and 0.955 across six datasets. The paper reports a significant jump after a single epoch.

For anyone operating a fine-tuning product this is the relevant threat, and it is cheap to mount. The attacker needs query access that returns token probabilities, and the strongest variants additionally need a reference model trained on similar data, which for open-weight backbones is a download.

flowchart LR
  subgraph TRAIN["Fine-tuning: exposure is created here"]
    DATA["Private corpus"] --> SEL["Control 1<br/>paraphrase low-loss samples"]
    SEL --> FT["Full FT, mean AUC 0.82<br/>or LoRA, mean AUC 0.64"]
  end
  subgraph SERVE["Serving: exposure is exercised here"]
    MODEL["Fine-tuned model"] --> API["Control 2<br/>does the API return log-probs?"]
  end
  subgraph ADV["Adversary"]
    REF["Reference model<br/>same open-weight backbone"] --> SCORE["Ratio / Ensemble score"]
    SCORE --> OUT["member or non-member<br/>read as TPR at 1% FPR"]
  end
  FT --> MODEL
  CAND["Candidate record"] --> API
  CAND --> REF
  API --> SCORE

Core knowledge

What moves exposure

Four factors, all of them knobs an infrastructure team already owns.

Model size. Attack AUC rises monotonically across the Pythia family from 70M to 6.9B on the same data and schedule. Bigger backbones memorise their fine-tuning sets more.

Epochs. Exposure grows with passes over the data, and one epoch is already well above chance. There is no safe number of epochs, only a smaller one.

Full fine-tuning against LoRA. Averaged over six datasets and ten attacks, full fine-tuning sits at 0.819 AUC and LoRA at 0.641. The paper attributes the gap to intruder dimensions, the high-magnitude singular directions that LoRA updates introduce and that full fine-tuning does not, which limit how closely LoRA fits the target data.

LoRA rank. The protection is a side effect of restricted capacity, so it erodes as rank rises. Raising rank to recover quality also recovers leakage.

Dataset. Code leaks differently from prose. On the GitHub subset a bag-of-words classifier that never sees the model reaches 0.649 to 0.747 AUC, which means member and non-member splits are separable from surface statistics alone. That classifier is the control every evaluation needs: an AUC above 0.5 for a model-free baseline says the split is contaminated, not that the model leaked.

Why average AUC is the wrong headline

This literature reports AUC averaged over attacks, and an operator is not exposed to the average attack. Two corrections matter.

The first is to read the maximum. Across the six datasets the reference-based Ratio attack scores 0.943 to 0.955 against full fine-tuning while the ten-attack mean is 0.819. The mean is diluted by attacks that do not work.

The second is that AUC is a whole-distribution statistic and privacy failures are concentrated in a tail. A defence can push the bulk of member losses down until aggregate AUC looks like chance while leaving a small subgroup perfectly memorised, and it is that subgroup, the outlier record, the rare diagnosis, the unique identifier, whose disclosure actually harms someone. True positive rate at a low false positive rate is the metric that sees it. The executed block below constructs the case.

# Runnable on system python3 (numpy only). Membership inference scoring, the two
# metrics that matter, and why reporting only the first one hides the risk.
# Published numbers are from SOFT (arXiv:2506.10424, USENIX Security 2025) Tables 1-4.

import numpy as np


def roc_auc(member, nonmember):
    """AUC via the rank identity: P(score_member > score_nonmember), ties at 0.5."""
    s = np.concatenate([member, nonmember])
    order = np.argsort(s, kind="mergesort")
    ranks = np.empty(len(s), float)
    ranks[order] = np.arange(1, len(s) + 1)
    _, inv, cnt = np.unique(s, return_inverse=True, return_counts=True)
    tie_mean = np.zeros(len(cnt))
    np.add.at(tie_mean, inv, ranks)
    ranks = (tie_mean / cnt)[inv]  # average rank within each tie group
    n_m, n_n = len(member), len(nonmember)
    return (ranks[:n_m].sum() - n_m * (n_m + 1) / 2) / (n_m * n_n)


def tpr_at_fpr(member, nonmember, fpr=0.01):
    """Fraction of members caught at a threshold that admits `fpr` of non-members.
    The threshold is set on the non-member distribution, as an attacker would."""
    thr = np.quantile(nonmember, 1.0 - fpr)
    return float((member > thr).mean())


rng = np.random.default_rng(11)

# 1. Sanity: a perfect separator scores 1.0, an uninformative one scores 0.5.
assert abs(roc_auc(np.array([3., 4., 5.]), np.array([0., 1., 2.])) - 1.0) < 1e-12
assert abs(roc_auc(np.array([1., 2.]), np.array([1., 2.])) - 0.5) < 1e-12
big = rng.standard_normal(200_000)
assert abs(roc_auc(big[:100_000], big[100_000:]) - 0.5) < 0.01
print(f"1. AUC implementation checks out (perfect 1.0, tied 0.5, random "
      f"{roc_auc(big[:100_000], big[100_000:]):.3f})")

# 2. Audit the published aggregates. Each column is one dataset; each row one attack.
FT_AUC = {  # Table 1, full fine-tuning column, Llama-3.2-3B
    "ArXiv":  [.822, .811, .785, .615, .757, .952, .508, .840, .764, .807],
    "HNews":  [.900, .910, .845, .627, .800, .943, .521, .907, .740, .886],
    "PubMed": [.895, .893, .850, .645, .856, .947, .528, .908, .868, .884],
    "PileCC": [.887, .902, .858, .668, .842, .949, .504, .895, .844, .942],
    "Wiki":   [.936, .939, .887, .669, .912, .944, .507, .938, .925, .925],
    "GitHub": [.846, .871, .820, .613, .869, .955, .649, .851, .847, .944],
}
SOFT_AUC = {
    "ArXiv":  [.525, .521, .517, .510, .519, .558, .505, .533, .518, .568],
    "HNews":  [.515, .517, .515, .489, .511, .533, .523, .515, .500, .567],
    "PubMed": [.496, .509, .541, .499, .503, .541, .518, .511, .516, .546],
    "PileCC": [.519, .533, .522, .518, .518, .552, .511, .532, .513, .604],
    "Wiki":   [.530, .532, .536, .512, .533, .576, .507, .529, .530, .587],
    "GitHub": [.625, .647, .591, .515, .598, .516, .660, .627, .620, .669],
}
PUBLISHED_COL = {"ArXiv": .766, "HNews": .808, "PubMed": .827,
                 "PileCC": .829, "Wiki": .858, "GitHub": .827}
for k, v in FT_AUC.items():
    got = np.mean(v)
    assert abs(got - PUBLISHED_COL[k]) < 5e-4, f"{k}: {got:.4f} vs {PUBLISHED_COL[k]}"
ft_mean, soft_mean = np.mean([np.mean(v) for v in FT_AUC.values()]), \
                     np.mean([np.mean(v) for v in SOFT_AUC.values()])
assert abs(ft_mean - 0.819) < 5e-4 and abs(soft_mean - 0.540) < 5e-4
print(f"2. published averages reproduce: full fine-tune {ft_mean:.3f} (paper 0.819), "
      f"SOFT {soft_mean:.3f} (paper 0.540)")

# 3. The Ratio attack is the one that matters, and averaging buries it. The mean over
# ten attacks is what the paper headlines, but an operator is exposed to the best
# attack, not the average one.
ratio_ft = [v[5] for v in FT_AUC.values()]
ratio_soft = [v[5] for v in SOFT_AUC.values()]
ens_ft = [v[9] for v in FT_AUC.values()]
print(f"3. worst single attack, full fine-tune: Ratio {min(ratio_ft):.3f}-"
      f"{max(ratio_ft):.3f}, Ensemble up to {max(ens_ft):.3f}   "
      f"(10-attack mean {ft_mean:.3f})")
assert max(ratio_ft) > ft_mean + 0.12, "the best attack far exceeds the reported mean"

# 4. The load-bearing point: AUC near 0.5 does not mean safe. Construct a fine-tune
# where the bulk of members is pushed slightly below the non-members while a small
# subgroup stays perfectly memorised. Aggregate AUC lands at chance; the attacker
# still identifies the memorised subgroup almost perfectly at a 1% false positive rate.
n = 400_000
frac = 0.08
nonmem = rng.standard_normal(n)
bulk = rng.standard_normal(int(n * (1 - frac))) - 0.1546  # shifted just below
memorised = rng.standard_normal(int(n * frac)) + 8.0  # far above any non-member
mem = np.concatenate([bulk, memorised])
auc, tpr = roc_auc(mem, nonmem), tpr_at_fpr(mem, nonmem, 0.01)
print(f"4. AUC {auc:.3f} (looks like chance) but TPR@1%FPR {tpr:.3f} "
      f"= {tpr/0.01:.0f}x the false positive rate")
assert abs(auc - 0.5) < 0.01, "aggregate AUC must look like chance"
assert tpr > 0.07, "the memorised subgroup is still fully exposed"
assert tpr / 0.01 > 7, "exposure is an order of magnitude above chance"

# 5. Same check against a defence that genuinely flattens the distribution: no
# memorised subgroup, so both metrics agree that there is nothing to find.
flat = rng.standard_normal(n) - 0.1546
auc_f, tpr_f = roc_auc(flat, nonmem), tpr_at_fpr(flat, nonmem, 0.01)
print(f"5. control, no memorised tail: AUC {auc_f:.3f}, TPR@1%FPR {tpr_f:.3f}")
assert tpr_f < 0.012, "a genuinely flat defence keeps TPR at the false positive rate"
assert tpr > 6 * tpr_f, "the two cases are indistinguishable by AUC but not by TPR"
print("all assertions passed")

Executed output:

1. AUC implementation checks out (perfect 1.0, tied 0.5, random 0.502)
2. published averages reproduce: full fine-tune 0.819 (paper 0.819), SOFT 0.540 (paper 0.540)
3. worst single attack, full fine-tune: Ratio 0.943-0.955, Ensemble up to 0.944   (10-attack mean 0.819)
4. AUC 0.499 (looks like chance) but TPR@1%FPR 0.086 = 9x the false positive rate
5. control, no memorised tail: AUC 0.457, TPR@1%FPR 0.007
all assertions passed

Every published aggregate in Tables 1 through 4 reproduces from the per-cell values, so the paper's arithmetic is sound and its comparisons are internally consistent. Steps 4 and 5 are the reason to insist on both metrics: two situations that are indistinguishable by AUC differ by more than tenfold in the rate at which an attacker can confidently finger a training record. The paper reports both, which is the right practice; much of the surrounding literature reports only AUC.

The defence: obfuscate what the model has already learned

The insight behind SOFT is that vulnerability is not uniform across the dataset. Samples the model fits best are the ones a loss-based attack identifies, so those are the ones worth replacing. The published algorithm computes a threshold from the mean loss on a held-out validation set, selects fine-tuning samples whose loss falls below it, replaces those with paraphrases, and repeats the selection every epoch. A strength parameter controls how much of each sample is rewritten; at the default only the second half is paraphrased and the first half is kept intact.

The measured result is a good trade. Average AUC falls from 0.819 to 0.540 and average true positive rate at 1% false positive rate falls from 0.360 to 0.029, against 0.014 for the pre-trained model that has never seen the data. Perplexity on the ArXiv task moves from 9.78 under undefended fine-tuning to 10.49, a 7% rise, against 12.26 for the pre-trained model, so most of the benefit of fine-tuning survives. An LLM-as-judge utility score puts the defended model at about 0.66 against 0.68 undefended. Training overhead is 15.73%.

Paper against released code

The repository implements a different selection rule from the one the paper describes, in two ways. Both are visible in select_epoch_data in finetune.py, which a per-epoch callback invokes at the start of every epoch.1

The threshold is not the validation loss. The published algorithm sets the cutoff to the mean loss over a validation set and selects every sample below it, so the selected fraction floats with how well the model currently fits. The code sorts the training samples by loss and takes a fixed fraction of the lowest, governed by a --select_ratio flag that defaults to 0.5. No validation set is consulted in the selection path at all.

The selection also happens once. A counter guards the branch, so on the first epoch the bottom fraction is chosen, and on every later epoch the guard sends the code down a path that marks all samples for replacement. Under the default three-epoch schedule that means one epoch of selective obfuscation followed by two epochs in which every training sample is served in paraphrased form. The paper's framing, that selectivity is what preserves utility, describes a schedule the released code runs for one epoch out of three.

Neither divergence makes the reported numbers wrong, and the second plausibly explains why the defended model lands so close to the pre-trained baseline on membership metrics. It does mean that anyone reimplementing from the paper will build something different from what was measured, and that --select_ratio is the knob to tune rather than a validation threshold.

The second thing to know before adopting the released pipeline is where the paraphrases come from. The paper says "state-of-the-art production LLMs, such as GPT-4 and Claude-3.5". The code's synthesis path constructs an OpenAI client, hard-codes gpt-4o-mini-2024-07-18, and sends the second half of each member document to the API. That is the private fine-tuning data, transmitted to a third party, in service of protecting it. For PII, clinical, or contractually restricted corpora this is disqualifying as shipped. The repository does provide an alternative path that draws replacements from precomputed neighbour text without any API call, and that is the one to build on; it is not the path the module's own entry point uses.

What it costs against differential privacy

DP-SGD and its LoRA variant are the principled comparison, and the paper's Table 5 is the clearest statement of why they disappoint here. That table runs on Llama-3.2-1B rather than the 3B used elsewhere, so its perplexities are on a different scale from the ones quoted above and should only be read against each other. At the strong end, a privacy budget of 0.01 drives attack AUC to roughly 0.50, but perplexity lands at 13.21 against 13.19 for the pre-trained model: at the noise level where the defence works, fine-tuning has bought nothing at all. At the weak end, a budget of 100 recovers perplexity to 11.66 while letting the ensemble attack back up to 0.735. The selective-paraphrase defence reaches 11.58 perplexity, better than every DP-LoRA setting, at 0.573 ensemble AUC. Reported overhead is 15.73% against 67.03% for DP-LoRA.

The caveat is that these are not the same kind of guarantee. Differential privacy makes a worst-case statement about any record; selective paraphrasing is a heuristic evaluated against a fixed attack suite. If a compliance argument needs a bound, this defence does not supply one.

Adaptive attackers

The published headline figures assume the attacker does not know the defence. When it does, the defence weakens measurably. Against an attacker who knows both the paraphrasing model and the selection strategy but not their hyperparameters, AUC rises from 0.568 to 0.595 and true positive rate at 1% false positive rate rises from 0.033 to 0.149. An attacker who knows only the selection strategy reaches 0.651 AUC. So the metric that matters most degrades by roughly a factor of 4.5 under the strongest adaptive attack tested, and the abstract's claim that the method "effectively reduces privacy risks" is measured in the non-adaptive setting. The defence still cuts exposure substantially from the undefended 0.258, but 0.149 is not 0.033.

Don't-miss checklist

  • Run a bag-of-words baseline on the member and non-member split before running any attack. Above 0.5 means the split is contaminated and every downstream number is measuring distribution shift.
  • Report true positive rate at 1% and 0.1% false positive rate alongside AUC. Never publish or accept AUC alone.
  • Evaluate the strongest attack, not the mean. Reference-based attacks win consistently, and a reference model is a download for any open-weight backbone.
  • Decide whether the serving API returns token log-probabilities. Most of these attacks score a candidate using per-token likelihoods, so withholding them raises the cost of the attack considerably, though attacks that work from generated text alone exist and this is not a complete defence.
  • Treat LoRA rank as a privacy parameter as well as a quality one, and record the rank alongside the exposure measurement.
  • If you adopt the released defence, switch the paraphrase source to a local model before it touches production data.
  • Re-measure after every change to epochs, rank, or dataset size. None of the published numbers transfer across those.

Failure modes

  • Reporting AUC alone. A defence that flattens the bulk while leaving a memorised tail scores at chance and still exposes the records that matter.
  • A contaminated evaluation split. Overlapping member and non-member distributions produce high AUC with no memorisation, which reads as a leak that a defence then appears to fix.
  • Raising LoRA rank for quality. Exposure rises with it, silently.
  • Sending private data to a paraphrase API. The defence's own preprocessing becomes the disclosure.
  • Reimplementing from the paper's Algorithm 1. The result differs from the evaluated code in both threshold and schedule.
  • Assuming pre-training results transfer. Attacks that score near chance against a base model are not evidence about the model you fine-tuned from it.
  • Treating the defence as a compliance control. It is evaluated against a fixed attack suite and carries no bound; adaptive attackers recover a substantial part of the exposure.

Open questions and validation

  • The evaluation uses 1,000-sample fine-tuning sets on 1B and 3B models. Production fine-tuning sets are larger and the interaction between dataset size, epochs, and the selected fraction is unmeasured.
  • The paper's paraphrasing quality depends on a production LLM, so results are tied to a model that changes underneath the experiment. The code pins gpt-4o-mini-2024-07-18; the paper names different models.
  • Whether the near-baseline defended metrics come from selective obfuscation or from the code's whole-dataset replacement on epochs two and three cannot be separated from the published results.
  • No result addresses fine-tuning on data that is also in the pre-training corpus, which the paper explicitly allows and which describes most public-corpus fine-tuning.
  • The utility measurement is perplexity plus an LLM judge on 100 generated questions. Neither shows whether the specific knowledge the fine-tune was meant to install survived paraphrasing.

References

Related: Gradient leakage in distributed training · Gradient inversion under federated averaging · Covariant obfuscation for private inference · Activation obfuscation on untrusted accelerators · Privacy-aware split inference · Security, isolation and multi-tenancy · GPU confidential computing · Fine-tuning and post-training · Glossary

  1. finetune.py at commit 7ca2b7b3b44e352e8ade6bea62889156fe1bff94. CustomCallback.on_epoch_begin calls select_epoch_data at the start of every epoch. That method reads if self.count_change > 0: select_ids = list(range(len(self.raw_train_dataset))), and otherwise sorts all training losses ascending and takes int(len(sorted_scores) * self.select_ratio) of them, where select_ratio comes from --select_ratio with a default of 0.5. count_change is incremented on every call, so the quantile branch runs only on the first epoch. Compare Algorithm 1 in the paper, whose DATA_SELECTION computes tau as the mean validation loss and selects l_f < tau on each of T epochs. The paraphrase source is data/obfuscation.py: build_editing_dataset_w_synthesis instantiates OpenAIPrompter(api_key=os.getenv('OPENAI_API_KEY'), model_name='gpt-4o-mini-2024-07-18') and calls it on each member document's second half; the module's __main__ block uses that function. build_editing_dataset is the local alternative and draws replacements from the dataset's precomputed neighbour text. Section 5.4 of the paper also contains a direction error worth flagging when reading it: "DP-LoRA with more noise added (e.g. ε = 0.01) increases the attack efficacy, as indicated by low AUC-ROC scores and TPR", where a low AUC indicates the opposite. ↩