Extracting Training Data from Large Language Models

Extracting Training Data from Large Language Models. Nicholas Carlini and colleagues presented this work at the USENIX Security Symposium in 2021. They examined whether GPT-2 could reproduce rare training text through black-box interaction, without access to model weights, hidden states or the training corpus. The attack and its evidence show how memorization can hide behind a reassuring average loss, while the evaluation defines how narrowly that result should be applied.

Here, extraction means finding a prompt that makes the model generate a target string. The researchers called a string k-eidetically memorized when it was extractable and appeared in no more than k distinct training examples. In plainer terms, the concern was specific extractable text. Privacy can depend on context as well as secrecy: a name or contact detail collected from the public web can still cause harm when reproduced in a new dialogue.

The practical question is whether an attacker can recover memorized text through the interface alone, and whether low overall overfitting makes that risk disappear. The evaluated attacker could request sequence likelihoods and next-token predictions but could not inspect the weights, internal states or training set. The experiment therefore tested black-box extraction while examining whether rare examples could behave differently from the model average.

The researchers generated 600,000 samples using several approaches. One started without a prompt and chose among likely next tokens. Another changed how random token selection was as generation continued. A third seeded the model with unrelated Internet text. They then ranked outputs by how unusually well GPT-2 predicted them. Some signals compared predictions with smaller GPT-2 models or considered how easily the text compressed. Others tested the effects of lowercasing or looked for an especially predictable local span. After deduplication, the researchers manually reviewed high-ranked outputs. They confirmed candidates through exact web search and limited fuzzy matching against the original corpus.

Among 1,800 inspected candidates, the researchers confirmed 604 unique memorized examples. The result was not explained away by the model’s average behavior: a low average gap between training and test loss coexisted with rare examples having anomalously low loss. In ordinary terms, the model could appear reassuring on average while still predicting individual training passages unusually well, making those passages candidates for extraction.

Discoverability depended strongly on prompt context. None of 273 passages found after unrelated Internet prefixes could be reproduced reliably from the same short prefix, but nearly all returned when the model received the text immediately preceding the memorized passage. Disclosure length could also exceed the fixed sampling window: one source-code fragment was extended to 1,450 verbatim lines. Together, those observations suggest that unsuccessful prompting does not prove the text is absent.

The evaluation had a narrow scope. It used public GPT-2 data and tested indiscriminate rather than targeted extraction. It also included likelihood access and validation by the corpus owner. Exact web search plus limited fuzzy matching could confirm matches, but it could not discover memorized samples the search missed. Consequently, the measurements do not establish how often private data can be extracted from deployed proprietary models. They demonstrate extraction in the evaluated setting, not an incident rate across other systems.

For model builders, privacy teams and security testers, the proposed mitigations work at different stages. Improve data curation and deduplication before training, then filter outputs and test trained models for extraction. Differential privacy is another option. None was presented as a complete solution. Document-level deduplication can miss repeated sensitive strings inside one document. Differential privacy also presents a scoping problem when the same person’s rare text spans documents and contributor labels are missing.

The study showed through black-box interaction that GPT-2 could emit specific memorized training passages even without broad aggregate overfitting. Model builders, privacy engineers and red teams should not treat a low average train-test loss gap as sufficient reassurance. They should improve curation and deduplication, then filter outputs and test trained models for extraction. Differential privacy is another complementary option. They should not infer a measured private-data leak rate for proprietary deployments, or assume that public-at-collection text is harmless when reproduced in a new context.