Governance, safety and risk management

Episode Ten places governance and security around the evaluated production system. By the end, you should be able to identify trust boundaries, model direct and indirect prompt injection, distinguish confirmation from authorisation, minimise sensitive telemetry, and define both preventive and detective controls. Now apply the same evidence discipline to harm, misuse, privacy, and accountability. Governance is not a committee added after the prototype succeeds. It is the allocation of decisions, controls, evidence, ownership, and accountability across the system lifecycle. Start with the use case and harm model. Who can be affected, what decisions or actions occur, what data is involved, how errors propagate, and how reversible are the outcomes?

Text description

• Users and retrieved content: Prompts, documents, emails, web pages, and retrieved knowledge may contain untrusted instructions.
• Model provider, model, and orchestrator: The provider is a trust boundary; the model interprets content while the orchestrator enforces workflow policy and available capabilities.
• Tools, MCP servers, data stores, and external networks: Tool, data, and network access must be scoped by identity, case, and purpose.
• Reviewers and downstream action systems: Human reviewers and systems that carry out actions are distinct trust boundaries.

Harbour Resolution's document classifier has lower direct impact than an agent that recommends compensation or changes case status. Both need controls, but the depth and approval model should follow consequence and exposure. What are the main trust boundaries in a Claude system? Users, model provider, orchestrator, tools, M C P servers, data stores, retrieval content, external networks, human reviewers, and downstream action systems. Every boundary carries identity, data, integrity, availability, and logging questions. Untrusted instructions can arrive through the user prompt. They can also arrive indirectly through web pages, emails, documents, code comments, tool results, or retrieved knowledge. That second category is indirect prompt injection.

A malicious document might tell the agent to ignore policy and send secrets to an external tool. The content is data to analyse, not authority to redefine the system. Can a system prompt alone solve indirect injection? No. Prompts help but must be combined with capability restriction, content handling, validation, confirmation, and monitoring. Anthropic's current guidance includes screening untrusted tool outputs before returning them to the model in sensitive workflows. Architectural controls begin by minimising available capability. Separate read from write. Scope data access by identity and case. Use sandboxes for code and computer actions. Require authenticated approval for consequential or external side effects. Validate destinations, amounts, recipients, and transaction state in ordinary software.

Text description

• Confirmation: Asks whether to proceed. A dialog shown to the wrong person does not provide meaningful control.
• Authorisation: Verifies authenticated authority and the current workflow state when the action is executed.

Do not let retrieved text select new tools or permissions. Screen inputs and outputs where the risk justifies it. Red-team the actual workflow with adversarial documents and tool responses rather than testing only direct chat attacks. What is the difference between confirmation and authorisation? Confirmation asks a user whether to proceed, while authorisation proves that the user or service is permitted to approve that action. Therefore, a confirmation dialog shown to the wrong person is not a control. The action service should verify authenticated authority and current workflow state at execution time. Human in the loop can mean review, approval, exception handling, sampling, or active collaboration.

Specify where the human enters, what evidence they see, what decision they make, and how their decision is recorded. Do not use the phrase human oversight as a substitute for an operational design. Reviewers need enough source context to detect polished errors without repeating the entire investigation from scratch. Automation bias should be considered, because confident formatting can encourage uncritical acceptance. Interface design can show sources, uncertainty, model limitations, and changed fields instead of only a fluent recommendation. Now turn to data governance. Classify the data and minimise what enters prompts, logs, caches, evaluation sets, and support channels. Map where data is processed, stored, backed up, retained, and accessed.

Text description

• Evidence for investigation: Telemetry can provide evidence needed to investigate incidents.
• Sensitive dataset: Logs can become a high-value target and therefore need access control, redaction, retention limits, integrity protection, and monitoring.

Check current platform and contractual terms for retention, training use, regional processing, zero-data-retention eligibility, and feature-specific exceptions. Do not assume that enabling one privacy option automatically covers every tool, connector, or cloud surface. Secrets should not be placed in prompts or repository instructions when a secret manager and scoped runtime identity can provide them. Logs may be essential for incident investigation but can also become a high-value sensitive dataset. Apply access control, redaction, retention, integrity, and monitoring to telemetry. How should an architect approach compliance claims? Map actual obligations to system controls and evidence, involve legal and risk owners, and avoid claiming that a model or cloud service makes the whole solution compliant.

Certifications and provider attestations cover defined service scopes, not the customer's entire application and operating process. The Professional guide cites regimes such as G D P R, H I P A A, and FedRAMP as examples of compliance considerations. The exam is testing the method of requirements mapping, not asking you to recite every law. For an Australian organisation, the same method applies to relevant privacy, records, sector, employment, secrecy, and security obligations. Identify data categories, purposes, lawful basis or authority, access, disclosure, retention, individual rights, and assurance evidence with qualified advisers. Safety also includes bias, fairness, and transparency. Evaluate whether performance differs across relevant cohorts and whether source data or labels encode historical disadvantage.

Do not infer protected or sensitive attributes merely to create a fairness dashboard without a lawful and ethical basis. Use domain experts and affected stakeholders to define meaningful harms and acceptable trade-offs. Explain to users what the system does, what evidence it uses, when a human remains responsible, and how to challenge or correct an output. Transparency should be useful rather than a generic disclaimer that nobody reads. Now consider availability and abuse. Rate limits, dependency failures, tool timeouts, and provider outages need fallbacks and user-visible states. Cost exhaustion can be an abuse path when an attacker induces long loops, huge context, repeated tools, or expensive models.

Text description

• Contain the workflow: Use a kill switch or feature flag and disable unsafe tools or affected features.
• Preserve evidence: Retain relevant prompt and configuration versions, tool traces, sources, identities, approvals, outputs, and effects according to policy.
• Recover safely: Revoke credentials, quarantine data, roll back prompts, rebuild indexes, and notify accountable owners as needed.
• Improve the system: Add the failure to threat models and evaluation suites instead of responding only with a prompt rewrite.

Set quotas, budgets, loop containment, input limits, and anomaly monitoring. A fallback should fail safely rather than silently dropping required review. Incident response must recognise model-specific evidence. Preserve prompt and configuration versions, tool traces, retrieved sources, identities, approvals, output, and downstream effects according to policy. Provide a kill switch or feature flag for unsafe workflows. Know how to revoke credentials, disable tools, quarantine data, roll back prompts, rebuild indexes, and notify accountable owners. After an incident, add the failure to threat models and evaluation suites rather than merely rewriting one prompt. A strong governance answer names the harm, boundary, preventive control, detective control, human role, evidence, owner, and failure response.

Episode Ten is complete when each material harm has an owner, preventive control, detective signal, human decision, evidence trail, and failure response. For source navigation, use the prompt-injection and jailbreak guidance, prompt-leak material, Trust Center, compliance references, authentication guidance, and Claude Code security documentation. In the final episode we will turn that architecture into stakeholder decisions, operational ownership, team enablement, and continuous improvement.