Integration, retrieval and protocol selection

Text description

• Ingest sources: Bring content into the controlled system and record its origin and transfer status.
• Parse and structure: Convert different formats into consistent structure while preserving meaningful features.
• Chunk and describe: Create retrievable units and attach metadata such as version, authority, ownership, and effective period.
• Index for search: Make chunks searchable through lexical, semantic, structured, or combined methods.
• Retrieve and order: Find candidates, apply required filters, and rank the permitted, applicable evidence.
• Assemble context and answer: Give Claude selected evidence, then connect important claims to their supporting sources.

This episode connects Claude to changing documents, enterprise services, and permission systems while keeping the resulting architecture understandable enough to diagnose when something goes wrong. Integration is where a convincing answer meets less glamorous realities such as stale files, inconsistent data shapes, network boundaries, hidden permissions, service failures, and operational scale. RAG, short for retrieval-augmented generation, is a design in which a system finds relevant source material and places selected evidence into Claude's working context before asking for an answer. A typical RAG workflow brings content into the system, turns each source into consistent structure, splits it into searchable pieces, attaches descriptive details, makes those pieces searchable, finds possible evidence, orders the best matches, and gives Claude the selected material for its answer.

Ingestion means bringing source material into the controlled pipeline while recording what arrived, where it came from, and whether the transfer completed successfully. Parsing turns formats such as documents, transcripts, and source files into consistent structure, but a parser can silently lose headings, tables, speaker labels, or access markings. A chunk is a retrievable unit of content, and chunking is an architectural choice because its boundaries determine which ideas can reach the model together. There is no universal chunk size because useful boundaries follow meaning and structure rather than an arbitrary character count. A contract might be divided by clauses and schedules so that a liability clause remains connected to its definitions and exceptions.

A policy might be divided by headings while retaining its version and effective period, meaning the interval during which that policy applies. A call transcript might use speaker turns and time windows, because separating a promise from the customer's preceding question could change its apparent meaning. Source code might be divided by functions, classes, or modules while preserving dependency relationships that show which components call or rely on one another. Parent-child relationships let a retrieved paragraph recover its surrounding section and document, rather than forcing one small fragment to carry the whole meaning alone. Chunks that are too small can omit qualifications, while chunks that are too large can bury the useful passage in irrelevant text and consume limited context space.

Text description

• Topic similarity alone: A highly similar obsolete policy may rank first and produce a polished but incorrect answer.
• Applicability before ranking: Filter for the case date and jurisdiction, then rank the remaining sources by relevance and authority.

Metadata is data describing a source, such as its jurisdiction, product, confidentiality, policy version, effective period, level of authority, or the isolated customer or organisational domain that owns it. Harbour Resolution is a fictional organisation used here as a running example, and its metadata might also identify case type, document owner, and the customers or teams allowed to use a record. In a hypothetical Harbour Resolution case, an analyst might see a polished answer based on an obsolete policy because the tempting design ranked by topic similarity without first checking the effective period. A better design filters for the case's date and jurisdiction before ranking, although it still depends on those metadata fields being complete and correct.

An embedding is a numerical representation intended to place conceptually similar text near each other, even when the text uses different words. A vector index is a searchable structure for those numerical representations, while a lexical index records words and terms for exact or near-exact matching. When a request arrives, retrieval produces a candidate set, meaning a limited collection of possibly useful chunks that later stages can filter and order. Source attribution connects an answer or claim to the material supporting it, which is especially important when reviewers need to audit why the system said something. Correct citation syntax only means the citation follows the required structure, while citation truth requires the cited source to exist and genuinely support the associated claim.

Text description

• Authenticated request: Begin with trusted identity and the current request context.
• Permission filter: Enforce tenant, role, case, and document access before exposing content.
• Applicability filter: Remove material for the wrong date, jurisdiction, product, tenant, or case type.
• Relevance and authority ranking: Order the permitted, applicable candidates by usefulness and controlling authority.
• Selected model context: Only the surviving evidence is supplied to Claude; a later prompt cannot reliably undo an earlier disclosure.

That is the distinction between syntax and truth: well-formed output can still contain a wrong identifier, an irrelevant source, or a claim the source never made. Applicability asks whether a source governs the present situation, while authority asks whether that source is sufficiently official or controlling for the decision. Semantically similar material can therefore be obsolete, unauthorised, superseded, or written for a different jurisdiction, product, tenant, or case type. Permission and applicability filters should run before restricted content enters Claude's context, followed by relevance and authority ranking among the permitted candidates. That order is crucial. A model cannot reliably erase knowledge from its current context merely because a prompt later tells it to ignore a record that the user was never allowed to see.

Authentication establishes who or what is calling, typically by validating a user, service, or workload identity. Authorization decides which data and operations that authenticated identity may access in the current situation. A tenant is an isolated customer or organisational domain within a shared system, so tenant filtering helps prevent one domain's records from crossing into another. The retrieval service should enforce tenant, role, case, and document-level rules using trusted identity supplied through the authenticated connection. An agent must not gain access by putting a claimed administrator role into a tool argument, because self-authored text is not trusted proof of authority. Service-to-service requests should use scoped credentials that permit only necessary actions, short-lived authentication credentials where supported, and a traceable workload identity identifying the calling software.

In a hypothetical payroll office, an employee might receive an accurate answer drawn from an executive-only compensation file if retrieval uses the employee's query but forgets the employee's authorization scope. The better design sends trusted identity to the retrieval service and filters there, although incorrect directory groups or document labels can still produce overexposure or mistaken denial. Lexical search matches words, phrases, and identifiers, making it strong for rare terms, quoted language, account codes, and exact policy numbers. For a request about policy P R one four seven, lexical lookup with policy-number and applicability filters should dominate because the identifier already expresses precise intent. Lexical search alone can miss a relevant passage when the user paraphrases its meaning or the source uses different terminology.

Text description

• Lexical search: Strong for exact words, quoted language, rare terms, account codes, and policy identifiers, but can miss paraphrases.
• Semantic search: Finds conceptually similar wording, but similarity alone does not prove factual relevance, authority, permission, or applicability.
• Hybrid retrieval: Combines exact and conceptual matching for broader coverage, with additional tuning, computation, and failure analysis.
• Reranking: Spends more computation to reorder a smaller candidate set, adding latency, cost, and another observable failure point.

Semantic search compares embeddings to find conceptual similarity, which helps when a request and its answer discuss the same idea in different words. Semantic similarity is not proof of factual relevance, authority, permission, or current applicability, so vector ranking cannot replace those controls. For a hypothetical question about contacting a vulnerable customer through an authorised representative, semantic retrieval can connect everyday wording with policy concepts that do not repeat the question verbatim. A hybrid approach combines lexical and semantic retrieval, which can cover both exact identifiers and paraphrased concepts but requires more tuning, computation, and failure analysis. Reranking means spending additional computation to reorder a smaller candidate set with a more precise relevance assessment before selecting the final context.

Authority and effective-date rules should remain explicit during reranking, because a highly relevant superseded policy must not outrank the current controlling one merely by sounding closer. Reranking may improve ordering on difficult queries, but it adds latency, meaning response delay, as well as cost and another component whose errors need to be observed separately. Query expansion adds related search terms, while query decomposition breaks a broad request into smaller searches whose results can later be combined. A broad workplace-conduct request might be decomposed into reporting, confidentiality, retaliation, and escalation topics, provided those searches remain anchored to the person's actual question. Expansion and decomposition can drift by introducing assumptions the user never made, retrieving excessive material, or combining rules that apply to different situations.

Each retrieval stage should be evaluated on representative queries with known permissions, applicable sources, expected evidence, and realistic wording rather than a few convenient demonstrations. Useful evaluation separates whether the right material was available, whether it was retrieved, whether it was permitted, whether it reached the context, and whether the answer used it faithfully. Progressive discovery starts with a compact catalogue, index, or tool surface and fetches deeper detail only when the task shows that it is needed. In a hypothetical procurement workflow, Claude might first see a short catalogue of contract repositories, then inspect the relevant region and finally fetch one applicable clause. That design reduces up-front context but can miss information when catalogue descriptions are poor, a discovery step fails, or the model chooses the wrong branch.

Text description

• Progressive discovery: Start with a compact catalogue and fetch detail when needed; this reduces up-front context but can fail through poor descriptions or a wrong branch.
• Monolithic context: Provide a large body of information at once; everything is visible, but size, delay, distraction, and staleness increase.
• Small cached packet: A small, stable packet reused across similar requests may suit caching, but invalidation is still required when content or permissions change.

Monolithic context puts a large body of information or capability definitions into the request at once, making everything immediately visible but increasing size, delay, distraction, and staleness. A library catalogue is a useful analogy for progressive discovery because it guides deeper lookup, although unlike a library, an automated catalogue can also hide material through permission rules or faulty metadata. The choice between progressive discovery and monolithic context depends on corpus size, task predictability, repeated reuse, latency, and the consequences of missing a discovery step. Large, changing enterprise collections generally favour controlled retrieval over indiscriminate inclusion because rebuilding each request with everything quickly becomes unwieldy. A small, stable policy packet reused across many similar requests may instead suit prompt caching, which reuses previously processed input to avoid repeatedly handling identical material.

Caching is not a freshness strategy, so the design still needs a clear way to invalidate cached material when policies or permissions change. Documents alone do not justify vector search, because exact lookup, structured database queries, a small cached packet, or ordinary full-text search may fit the data and questions better. After retrieval is designed, the next decision is how Claude or its surrounding application will reach the required service or capability. An API, which stands for application programming interface, is a defined software contract for sending requests and receiving responses from another service. A direct API is a good fit when one application owns a stable, purpose-built integration and does not need broad capability discovery across many clients.

In a hypothetical internal claims application, a narrow API adapter might expose case lookup and draft saving while deliberately omitting account deletion available in the larger backend. A CLI, which stands for command-line interface, lets software invoke command-oriented tools that normally run in a terminal or automated job. In a hypothetical build team, a familiar CLI might work on a developer laptop but fail in an isolated build environment because its configuration file, credentials, or executable version differs. Passing model-produced text directly into a shell is tempting but risky, so a safer wrapper should use fixed commands, validated arguments, restricted credentials, and an isolated environment.

Even with those protections, a CLI integration retains environment, dependency, output-parsing, and process-control risks that a stable service API may avoid. MCP, which stands for Model Context Protocol, is a standard way for an AI application, called a client, to discover what a connected service, called a server, can provide and then request information or actions from it. A hypothetical enterprise research capability shared by several Claude applications might justify an MCP server because a common interface reduces the need for a separate adapter in every client. MCP does not make the underlying capability trustworthy, authorised, fast, or well designed, so the server must still validate inputs, enforce identity, limit output, and expose useful errors.

Text description

• Direct API: Fits a stable, purpose-built integration owned by one application, especially when broad capability discovery is unnecessary.
• CLI wrapper: Can reuse command-oriented tools, but needs fixed commands, validated arguments, restricted credentials, and an isolated environment.
• MCP client and server: Supports shared capability discovery across AI applications, while the server still must validate, authorize, limit output, and expose errors.
• Agent-to-agent protocol: Supports independently governed agents exchanging tasks or results, with additional identity, state, ownership, and trust complexity.

A protocol is an agreed set of message formats and interaction rules, and agent-to-agent communication uses such rules to let independently operating agents exchange tasks or results while adding identity, state, ownership, and trust complexity. In a hypothetical supplier review, one agent might request a financial check from an independently governed agent and receive evidence plus a signed record of the requested task and its status. A signed task state can help show what was requested and which service produced the response, but it does not prove that the evidence was complete or the conclusion was correct. The simplest suitable mechanism should meet the actual requirements for interoperability, meaning systems can work together, as well as discovery, security, latency, lifecycle, and ownership.

Protocol novelty is not business value, so a private service used by one application may need only a careful API adapter rather than a general protocol layer. Capability bloat occurs when an agent is given many irrelevant tools, making selection harder, evaluation broader, and mistakes or compromise more consequential. In a hypothetical support workspace, a drafting assistant might display a user-deletion tool simply because the backend offers one, creating a dangerous option unrelated to drafting. The better design groups capabilities by role and workflow stage so each agent or service sees only what its current responsibility requires. An allowlist is an explicit set of permitted tools or operations, and per-tool configuration can further limit resources, argument ranges, and credentials.

Text description

• Model prompt: Guides model behaviour but cannot itself deny an operation, withhold restricted data, or grant authority.
• Application controls: Constrain exposed tools and validate arguments, resources, ranges, and workflow state.
• Trusted service enforcement: Checks authenticated identity, authorization, transaction limits, and policy before allowing data access or action.
• Authorized approval: An identified person or service with the required permission approves higher-risk actions; simple confirmation is not equivalent.

Materially different privileges should often be separated into different agents or services, although additional boundaries create more handoffs, operational components, and possible failure points. A prompt is an instruction expressed to the model, while enforcement is a control implemented by a trusted service that can actually deny an operation or withhold data. Prompts can guide behaviour, but they cannot replace server-side authorization, argument validation, transaction limits, or policy checks. Confirmation means a person acknowledges the proposed action, while authorized approval means an identified person or service with the required permission grants approval under policy. In a hypothetical refund workflow, a support worker might see a clear confirmation screen and assume clicking continue authorizes a large payment, even though the worker lacks approval authority.

A sounder design checks the worker's identity and approval limit in the payment service and routes higher-risk actions to an authorised approver, accepting that this may add delay. Observability means collecting enough structured operational evidence to understand what happened across retrieval, generation, tools, and human decision points. A trace identifier is a shared correlation value that links events from one request as it crosses services without pretending that the identifier itself proves correctness. Where policy permits, a trace can record the authenticated principal, meaning the verified caller, along with prompt and configuration versions, model selection, and retrieval query. It can also record selected source identifiers, tool calls, stage timings, token use, errors, validation results, and the eventual human outcome when that outcome is appropriate to retain.

Model tokens are the small units of text processed by the model, so token use helps explain context size and resource consumption without measuring answer quality by itself. Logs can contain personal, confidential, or retrieved content, so fields should be minimised or redacted and access, encryption, and retention should be deliberately governed. One total latency number, meaning the end-to-end delay, cannot reveal whether time was spent in retrieval, reranking, model generation, tool calls, retries, or an approval queue. Distributed tracing or correlated event logs attach related stage events to the same trace, allowing a slow or failed request to be examined component by component. Suppose, hypothetically, that Harbour Resolution refreshes a policy collection and analysts then begin seeing wrong answers, with polished wording making it tempting to blame Claude immediately.

Text description

• Ingestion and parsing: Compare expected document counts, transfer outcomes, parse failures, and whether important structure survived.
• Chunks and relationships: Inspect boundaries and parent-child links for rules separated from definitions, exceptions, or surrounding sections.
• Metadata and indexes: Check effective dates, permissions, index or embedding versions, and whether current sources were wrongly excluded.
• Candidates and reranking: Review retrieved candidates, ordering, authority, freshness, and applicability.
• Final context and tools: Confirm exactly what Claude received and inspect any tool operation, identity, permissions, response, validation, errors, and retries.
• Generation and verification: Only after context is confirmed, examine instructions, model behaviour, answer formatting, and claim-level support.

The investigation should first compare ingestion counts and parse failures to determine whether all expected documents arrived and whether their structure survived processing. Next, compare chunk boundaries and parent-child links, because a refreshed parser may have separated an exception from the rule it qualifies. Then inspect metadata, embedding or index version, permission-filter behaviour, and effective dates to find stale representations or wrongly excluded current sources. After that, examine the retrieved candidates, reranking order, source authority, freshness, and final context assembly to confirm exactly what evidence Claude received. If a tool contributed data or performed an action, inspect the requested operation and arguments, the identity and permissions used, the tool's response, validation results, errors, and retries before deciding whether the failure occurred in the tool or in generation.

Only after the supplied context is confirmed should the investigation focus on generation instructions, model behaviour, or answer formatting. If the correct chunks arrived but the answer ignored a buried exception, likely areas include context ordering, chunk coherence, conflicting evidence, unclear instructions, and excessive irrelevant material. A verification pass can compare important generated claims with the selected sources, but it adds computation and can repeat the same mistake if it receives the same incomplete evidence. Retrieval success and answer faithfulness are separate measures because finding the right passage does not guarantee the generated answer will use it accurately. Evidence is the retrieved source material, while inference is the conclusion drawn from that material, and a fluent inference may still overreach, omit a condition, or combine incompatible rules.

Validation should therefore check support at the claim level where the use case warrants it, while acknowledging that no automated check guarantees complete correctness or compliance. The practical goal is not an architecture that can never fail, but one that limits access, records where sources and results came from, exposes trade-offs, and leaves enough evidence to locate failures. The synthesis is simple: retrieve only applicable authorised evidence, choose the least complex integration that meets the need, enforce privileges outside the prompt, and trace each stage without confusing confident language with verified truth.