Governance is the way an organisation assigns decisions, controls, evidence, ownership, and accountability throughout a system's life, from the first use-case proposal through operation, incidents, changes, and eventual retirement. It belongs around the evaluated production system, because a technically capable model can still cause harm through an unsuitable purpose, excessive access, weak operating procedures, or poorly controlled downstream actions. Good governance does not promise absolute safety or correctness; it makes important risks visible, assigns people to manage them, and produces evidence showing whether agreed controls actually operate. The starting point is a harm model, meaning a structured account of who could be affected, what could go wrong, how the harm could spread, and how difficult the result would be to reverse.

That analysis should cover decisions and actions, the data involved, affected people and organisations, error propagation, possible misuse, privacy consequences, service interruption, and the practical options for correction or recovery. Reversibility is especially important because a mistaken draft that a person can discard is different from an automated payment, disclosure, status change, or message that immediately affects someone else. Harbour Resolution is a fictional organisation we will use as a running example, so its situations illustrate design choices rather than claiming results from a real deployment. Its document classifier might merely suggest a category for an incoming case file, allowing a worker to correct the label before anything consequential happens.

An agent that recommends compensation or changes a case status has greater direct impact because its output can influence money, rights, workload, communications, and later automated steps. Both systems need controls, but the higher-consequence agent warrants stronger approval, narrower access, fuller evidence, closer monitoring, and a more deliberate recovery path. A useful design principle is proportionality, which means control depth should grow with likely consequence, exposure to attackers, sensitivity of data, scale, and difficulty of reversal. A trust boundary is a point where data, instructions, identity, or control passes between components that cannot safely assume the same level of trust. In a Claude-based system, boundaries commonly surround users, the model provider, the orchestrator that coordinates the workflow, tools, data stores, retrieved material, external networks, human reviewers, and systems that perform actions.

MCP, which stands for Model Context Protocol, can connect a model-facing application to tools and data through MCP servers, so each server is also a boundary whose capabilities and identity must be controlled. At every boundary, designers should ask who or what is communicating, which data may cross, whether content can be altered, what happens during failure, and which records are created. Authentication verifies a claimed identity or credential, while authorization determines what that authenticated person or service is permitted to read, change, approve, or invoke. The distinction matters across a tool chain because a correctly authenticated model service does not automatically have authority to access every case, call every tool, or act for every employee.

Integrity means information and actions remain complete and unaltered except through permitted changes, while availability means the workflow and its safeguards remain usable when they are needed. Prompt injection is an attempt to make untrusted instructions influence model behaviour in a way that conflicts with the intended task or policy. Direct prompt injection arrives in the user's own message, such as a request to ignore restrictions and reveal material that the workflow should not disclose. Indirect prompt injection arrives inside content the system fetches or processes, including webpages, emails, documents, code comments, retrieved knowledge, or tool results. Consider a hypothetical procurement assistant that opens a vendor document containing hidden text telling the model to copy internal pricing into a message addressed to an unfamiliar recipient.

The employee might see an ordinary document summary and a polished send-confirmation screen, while the dangerous instruction remains buried in the source material. The tempting mistake is to assume that the document is trustworthy because an authorised employee opened it, or that a stern system instruction will always make the model disregard hostile text. The stronger design treats document content as data to analyse, prevents it from selecting new permissions, restricts the assistant to approved destinations, and validates the proposed recipient before any message is sent. That design still has limitations because screening may miss novel attacks, destination rules can become stale, and legitimate unusual recipients may require an exception process.

A system prompt is a high-priority instruction supplied by the application, but it is guidance to the model rather than an independent security barrier. Prompts can shape behaviour and clarify that retrieved instructions lack authority, yet enforcement must also occur in ordinary software that the model cannot persuade itself to bypass. Prompt leakage means unintended disclosure of hidden instructions or contextual material, so system prompts should not contain secrets and should not be treated as confidential vaults. Capability restriction reduces harm by giving each workflow only the tools, data, destinations, and actions needed for its specific purpose. Separating read capability from write capability prevents a task that only needs to inspect a record from quietly gaining the power to modify or transmit it.

Access should also be scoped by authenticated identity, assigned role, relevant case, allowed fields, and current workflow state instead of giving one shared agent broad standing access. A scoped runtime identity is a service identity with limited permissions for the current operation, which reduces exposure if a model, tool, or credential is misused. Secrets such as access tokens belong in a secret manager, which securely supplies them to an authorised runtime, rather than in prompts, repository instructions, retrieved documents, or model-visible logs. A sandbox is an isolated environment that limits what code or computer-control actions can reach, although isolation adds operational complexity and does not excuse unsafe credentials or excessive network access.

For consequential actions, ordinary software should validate destinations, recipients, amounts, identifiers, required fields, and transaction state against authoritative rules before execution. Retrieved text may provide facts for consideration, but it should not grant a new tool, widen a permission, nominate its own unverified destination, or redefine who can approve an action. Confirmation asks whether someone wants to proceed, whereas authorization establishes that the person or service is permitted to approve that particular action. A confirmation dialog displayed to the wrong person is therefore not an authorization control, even if that person clicks a button marked approve. An authenticated approval links a verified identity to an allowed decision, but the action service must still recheck authority and current workflow state at execution time.

That final check catches cases where an approval has expired, a case has changed, the amount differs from what was reviewed, or the approver no longer holds the required role. For example, a hypothetical payroll assistant might display a proposed bank-account change to a team supervisor whose identity is known but whose role does not permit payroll amendments. The tempting response is to accept the supervisor's confirmation as sufficient because a human has participated and the request looks plausible. A better design sends the exact changed fields and supporting evidence to an authenticated payroll approver, records the decision, and makes the payroll service verify the approval before committing the update.

This approach can slow urgent corrections and burden approvers, so escalation paths and service targets are needed without weakening the underlying authority check. Input and output screening can detect suspicious instructions, sensitive data, or prohibited destinations where the risk justifies the added delay and possibility of false alarms. In a sensitive workflow, untrusted tool output can be screened before it returns to the model, reducing the chance that hostile content will influence the next model decision. Screening is not proof of safety because attackers can disguise instructions and benign material can resemble an attack, so it complements rather than replaces restricted capability and execution checks.

Structured output can also be useful, with JSON serving as a machine-readable text format whose fields can be checked before another component acts. Valid JSON syntax proves only that the text can be parsed; validation against a schema, meaning a declared set of expected fields and types, can check its structure, but neither syntax nor schema proves that its claims are true or its requested action is authorized. Evidence is recorded, observable material such as a source document, authenticated identity, approval record, or tool result, while inference is a conclusion drawn from that material. A fluent model statement remains a claim or inference; supporting evidence makes it possible to verify that claim, but the evidence must still be checked for relevance, authenticity, and faithful interpretation.

A preventive control is intended to stop or limit an unwanted event, while a detective control reveals suspicious behaviour, control failure, or harm that may already be developing. Narrow permissions, transaction validation, and authenticated approvals are preventive controls, whereas unusual tool-call alerts, approval audits, and comparisons between expected and actual actions are detective controls. Neither category is sufficient alone because prevention can fail silently, detection may arrive too late, and both controls can create cost, delay, and false positives. Red-teaming means deliberately testing a system with adversarial behaviour to discover weaknesses before attackers or ordinary mistakes expose them. Workflow red-teaming should include malicious documents, poisoned retrieval content, deceptive tool responses, permission edge cases, and interrupted approvals rather than testing only hostile direct chat messages.

In a hypothetical legal research workflow, a retrieved webpage could instruct the assistant to hide contrary sources and state that its preferred conclusion has already been approved. A researcher may see a confident answer with familiar formatting and be tempted to trust it because the retrieval system supplied the page. A stronger workflow preserves the retrieved passages, marks external instructions as untrusted, restricts the assistant to research tools, and shows the researcher which sources support each important claim. Its limitation is that a human can still misread weak evidence, and a compromised or misleading source may look legitimate even when the model follows the workflow exactly.

Human in the loop is a broad phrase that can mean review, approval, exception handling, sampled quality checks, or active collaboration, and those roles should not be treated as interchangeable. An operational design specifies where the person enters, which evidence they receive, what decision they may make, how much time they have, and how the decision is recorded. Reviewers need enough source context to detect polished errors without being forced to repeat the entire investigation, since excessive workload encourages superficial approval or abandonment of the process. Automation bias is the tendency to favour an automated recommendation because it appears confident, consistent, or professionally formatted. An interface can counter that tendency by showing sources, uncertainty, model limitations, proposed field changes, policy conflicts, and unresolved evidence instead of presenting only a fluent recommendation.

For Harbour Resolution's fictional compensation workflow, a reviewer might see a neat recommended amount and a green confirmation button but no explanation of which case records informed the recommendation. The tempting mistake is to approve from presentation quality, especially when queues are long and the recommendation resembles previous cases. A better interface places relevant records beside the recommendation, highlights changed fields, distinguishes missing evidence from model inference, and reserves execution for an authorized role. Even then, review is not a guarantee because people can share the same assumptions as the model, become fatigued, or lack the domain knowledge needed to notice a subtle error. Data governance begins by classifying information according to sensitivity, purpose, affected people, and handling obligations before deciding whether it should enter the model workflow at all.

Data minimization means using only what the task requires and limiting what reaches prompts, logs, caches, evaluation sets, support channels, and third-party connectors. A data map records where information is collected, processed, stored, backed up, retained, disclosed, and accessed, making hidden copies and cross-system dependencies easier to govern. Teams must check current platform and contractual terms for retention, provider training use, processing location, access arrangements, and feature-specific exceptions rather than relying on assumptions. Zero-data-retention is an arrangement under which eligible content is not retained after processing, but eligibility and exceptions must be verified for each relevant feature, tool, connector, and cloud surface. Enabling one privacy setting does not establish that every connected component follows it, because each boundary may have different storage, logging, support, or backup behaviour.

Telemetry is operational data such as logs, traces, metrics, prompts, tool calls, and error records collected to understand how a system behaves. Telemetry supports debugging, evaluation, security monitoring, and incident investigation, but it can become a valuable sensitive dataset containing personal data, secrets, business content, or privileged decisions. It therefore needs access control, redaction, retention limits, integrity protection, monitoring, and a defined purpose rather than indefinite collection by default. In a hypothetical customer-support workflow, an engineer might enable full transcript logging to investigate occasional routing errors and later discover that the logs also contain account details and pasted credentials. The tempting mistake is to keep everything because more evidence seems useful, while the better design redacts unnecessary fields, separates sensitive access, limits retention, and preserves only what investigations genuinely require.

The trade-off is real because aggressive redaction or short retention can remove evidence needed to reconstruct an incident, so the balance should follow the harm model and applicable obligations. Compliance means meeting applicable legal, regulatory, contractual, and internal requirements, but no model or cloud-service certification makes an entire customer application and operating process compliant by itself. Provider attestations and certifications cover defined service scopes, leaving the customer responsible for its use case, integrations, identities, data flows, human procedures, records, and downstream systems. The sound method is to map each actual obligation to controls, operating evidence, accountable owners, and assurance activity with legal, privacy, security, records, and risk specialists involved as appropriate.

For an Australian organisation, that same method can address relevant privacy, records, sector, employment, secrecy, and security obligations without pretending that one generic framework answers every case. The mapping should cover data categories, purposes, lawful authority, access, disclosure, retention, individual rights, correction paths, and evidence that the promised process operates. Safety also includes bias, fairness, and transparency, where bias is a systematic skew, fairness concerns how benefits and burdens are distributed, and transparency makes system behaviour meaningfully understandable. A cohort is a relevant group used for comparison, and evaluation can examine whether error types or outcomes differ across cohorts and whether source data or labels carry historical disadvantage.

Protected or sensitive attributes are personal characteristics that law or policy gives special protection, such as disability, ethnicity, religion, or sexual orientation; collecting or inferring them for fairness analysis requires a lawful purpose and carefully limited handling. Domain experts and affected stakeholders can help define which harms matter, what trade-offs are acceptable, and whether a numerical comparison reflects the real consequences of an error. Useful transparency tells people what the system does, which evidence it uses, where a human remains responsible, and how an output can be challenged or corrected. A generic disclaimer that nobody can apply is weak transparency, while detailed explanations can also expose sensitive information or overwhelm users, so content should suit the audience and decision.

Availability planning covers provider outages, dependency failures, tool timeouts, exhausted capacity, and degraded data sources, with clear fallback behaviour and user-visible system states. A safe fallback should preserve required review and authority checks rather than silently skipping them to keep work moving. For example, if Harbour Resolution's fictional approval service is unavailable, the agent should not interpret the outage as permission to issue compensation without approval. Rate limits cap activity over time, quotas bound allowed consumption, budgets constrain spending, and loop containment stops an agent from repeating actions indefinitely. These controls also address cost exhaustion, an abuse path in which an attacker or malfunction induces huge inputs, long reasoning loops, repeated tool calls, or unnecessarily expensive processing.

Anomaly monitoring looks for behaviour that differs meaningfully from expected patterns, such as an unusual burst of tool calls, repeated denials, new destinations, or rapidly growing context. A hypothetical coding assistant might encounter a repository comment instructing it to rerun a costly analysis until no warnings remain, even when the warning can never be resolved. The tempting design lets the model decide when to stop, while the stronger design applies tool-call limits, time and spending budgets, cancellation controls, and a visible incomplete state. Those limits can interrupt legitimate complex work, so users need a controlled continuation path rather than hidden automatic retries or unrestricted execution.

Incident response is the organised process for containing, investigating, recovering from, and learning from harmful or suspicious events. Model-related investigations may require prompt and configuration versions, tool traces, retrieved sources, authenticated identities, approvals, outputs, and records of downstream effects, all retained according to policy. Evidence integrity matters because investigators must know that records are complete, correctly timed, and protected from unauthorised alteration, while access restrictions prevent the investigation store from becoming another source of harm. A kill switch disables a dangerous workflow quickly, while a feature flag is a controlled setting that can turn a capability on or off without rebuilding the whole application. Response plans should identify how to revoke credentials, disable tools, quarantine suspect data, roll back prompts, rebuild the search indexes the system uses to find source material, stop downstream actions, and notify accountable owners.

These measures have operational costs and may interrupt legitimate work, so teams should define activation authority, recovery criteria, and safe manual alternatives before an incident occurs. After an incident, the failure belongs in the threat model, which describes plausible threats and defenses, and in the evaluation suite, which is a repeatable set of tests for system behaviour. Merely rewriting one prompt risks treating the visible symptom while leaving excessive permission, poor evidence, unsafe interface design, or weak ownership unchanged. A mature governance record connects each material harm to its trust boundary, preventive control, detective signal, human decision, evidence trail, accountable owner, and planned response when controls fail.

The central lesson is simple: govern the complete workflow, keep authority outside untrusted content, match control strength to consequence, preserve useful evidence, and expect every safeguard to have limitations.