Evaluation and Hardening of LLM System Instructions Against Extraction via Encoding Attacks. LLM means large language model. In this 2026 arXiv study, Sahu, Samanta and Soosahabi ask whether a model that refuses to disclose its system instructions directly may reveal the same protected content when the request is framed as an encoding or formatting job. That gap defines the threat model. The evaluation measures leakage from structured-output requests and examines what prompt hardening can and cannot establish.

System instructions define an agent’s behavior and may contain access credentials, internal policies, configuration details or proprietary workflows. Some deployments also place defensive rules or secret validation tokens there. Disclosure can therefore expose internal logic and sensitive functionality or assist later prompt-injection attempts. Even a fragment matters if it reveals confidential variables or helps reconstruct the protected instructions. Under the threat model, revealing partial or complete instructions, confidential variables or reconstructable fragments is a confidentiality violation.

The attacker in this study is an ordinary chat user who can submit arbitrary natural-language prompts and observe responses. The attacker cannot alter the system instructions, inspect server-side memory or intercept tool output. A test succeeds only when the model refuses a direct extraction request but answers an equivalent request expressed as encoding or structured output. Leakage includes complete instructions, fragments, confidential variables or enough information to reconstruct protected content. This isolates failures caused by changing the request’s representation rather than models that simply lack a direct-refusal baseline.

The evaluation combines a capability probe, the target model, an automated prompt generator and a separate judge model. Benign requests first establish which formats each target can reliably produce. Only supported formats proceed to testing. The system then verifies that a direct extraction request is refused and recasts that request as a formatting or encoding task. A GPT-OSS-120B judge compares the original instruction, attack prompt and response. It marks leakage if the response exposes any part of the instruction, whether verbatim, paraphrased or encoded, or reveals confidential variables or enough detail to reconstruct the instruction. Attack success rate is the primary metric used to quantify the effectiveness of encoding-based attacks.

Structured wrappers, including configuration-like formats, consistently produced the highest leakage rates in the evaluated setting. Leakage also changed markedly between requests that meant essentially the same thing but used slightly different words or representations. In several cases, relatively small revisions to the system instruction’s wording, structure and emphasis substantially reduced leakage without adding rules for individual encodings. The researchers infer that confidentiality depends partly on how strongly and clearly the system instruction expresses its restrictions, not merely on whether a confidentiality sentence is present.

Every benchmark instruction had a verified 0% attack success rate for direct extraction before the transformed requests were evaluated. Each encoding was retained only if the target model could produce that format, so a failure was not attributed to an unsupported transformation. The judge used a broad definition of leakage that included partial and reconstructable disclosures. Human reviewers also inspected a randomly sampled subset using the same criteria. This setup grounds the comparison in a specific change of representation, although it does not by itself establish how often leakage occurs outside the experiment.

The evaluation covered a limited collection of commercial and open-weight models. It tested common encoding and serialization techniques, but not every possible representation or multistage extraction strategy. Leakage was measured with automated, predefined scoring, and the binary result did not distinguish a small fragment from complete disclosure; ambiguous cases may still require human judgment. The experiments were controlled, so the measurements do not establish that the observed leakage rates or hardening benefits will carry unchanged into other models, prompts or production environments.

My operational interpretation is that a successful direct refusal should be treated as a baseline, not proof that a system prompt is confidential. Pre-deployment testing can include requests that serialize or transform protected context, especially structured-output formats. Instruction hardening can explicitly prohibit verbatim, partial and reconstructable disclosure, then retest the revised wording across representations. Defenders should not rely only on randomized delimiters or prompt boundaries: those measures can disrupt attacks based on predictable layouts, but they do not stop a model from treating disclosure as a legitimate transformation task.

Sahu, Samanta and Soosahabi contribute an automated way to test whether encoding and serialization requests bypass system-instruction confidentiality. Their evidence shows that direct refusal can conceal representation-sensitive failures and that design-time wording changes can reduce leakage in the evaluated setting. Teams protecting sensitive agent instructions can use these transformations in red-team testing and strengthen disclosure rules. They should not infer that prompt rewriting guarantees secrecy, covers untested attacks or generalizes across models without further evaluation.