Agents need rich context to perform complex reasoning. But if you hand over unredacted PII to a third-party model provider, you've already lost. From a regulatory perspective, egress equals processing.

A lot of folks try to slap post-LLM guardrails on their systems. They intercept the payload after the model generates a response and try to scrub it. This is fundamentally inadequate for privacy.

Sending the data to external hardware is already a data breach. Furthermore, post-generation interception is highly vulnerable to relay tampering. An adversarial relay can silently rewrite a safety-aligned response before a tool executes it.

So, what's the pragmatic fix? You have to shift to deterministic, infrastructure-level enforcement before the prompt ever hits the model.

The Problem With Simple Masking

I put this data mitigation layer directly in the agent's hot path as a separate middleware layer. It operates completely independently of the model weights or system prompts. I don't use a secondary LLM for this; I rely on rigid, rule-based logic and high-performance regex.

You need deterministic speed at this interception layer. It ensures auditability and keeps latency exceptionally low.

This matters because the attack surface for agentic systems is expanding fast. Attackers easily bypass naive blocklists using scrambled text or synonym substitution.

The scariest threat right now is indirect prompt injection. An attacker can hide malicious instructions in a PDF that your RAG system blindly retrieves. Your agent then acts on behalf of the attacker, using your user's permissions.

We also have to defend against multi-turn Crescendo attacks. These use benign queries that gradually escalate over several conversational turns to trick the model's safety filters.

When scrubbing data before it hits the API, simple masking is a trap. Replacing everything with a generic [REDACTED] tag destroys the task semantics. It completely paralyzes the model's ability to do mathematical reasoning or maintain vector embeddings.

Instead, I've been implementing typed placeholders. I replace sensitive spans with semantically meaningful tags, like <Health_Info_1>, right on the edge device.

The cloud model maintains its contextual coherence and logical reasoning using these typed placeholders. The raw secrets simply never leave the local device.

Securing Legacy APIs and Continuous Evaluation

Sometimes, you have to hit a legacy API or database that strictly validates data formats. In those cases, typed placeholders fail. That's when I reach for Format-Preserving Encryption (FPE).

FPE scrambles the data into mathematically secure ciphertext but keeps the original length and format. A 16-digit credit card number stays a valid 16-digit number, keeping your schemas happy.

To test all this, static point-in-time audits are basically useless. LLMs are non-deterministic, so you have to build continuous privacy evaluations directly into your CI/CD pipeline.

I treat red-teaming as an automated engineering control. My deployment pipelines simulate multi-turn injections and check if the agent leaks synthetic data. If it does, the release is automatically blocked.

Ultimately, this is how you handle the realities of GDPR and strict privacy guidelines. By confining identifiable data to internal environments using deterministic pre-model redaction, you massively reduce your legal exposure. It's the most practical way I've found to build secure agents right now.

Hitting that hard token limit means the system either crashes or starts evicting vital instructions. The most practical fix for this is memory compaction.

What is memory compaction?

Memory compaction isn't just asking the model to "summarize this chat." It is the process of condensing past conversational history into a dense, meaningful representation of state.

Why bother? Because it keeps your agent running autonomously without burning through massive amounts of API credits. It preserves the exact intent and state needed for an AI to actually execute over long periods.

The anatomy of the problem

When you compress context, you have to decide what survives the cut. You absolutely must preserve the overall intent, the current execution state, and historical progress.

We are fighting strict constraints here. We have hard token limits, we need low retrieval latency, and we have to balance accuracy against abstraction. You usually achieve this through a mix of entity extraction and structured state formatting.

Three timeframes of a complete system

If you look closely at how production systems handle this, you realize they rarely pick just one compaction method. Instead, they use a stacked memory architecture that operates across three distinct timeframes.

The Checkpoint Layer (Long-Term)

This is the macro layer. It holds the overall project map, the ultimate goal, and major milestones.

You don't feed this massive block of text into every prompt—it’s too expensive and distracting for the model. Instead, you only pull the checkpoint when the agent switches major tasks, starts a brand-new session, or needs to recover from a crash. It acts as an anchor to re-orient the system when it needs to see the whole board.

The Status Board Layer (Mid-Term)

Think of this as the agent's active work session, functioning essentially as a strict Kanban board. It tracks exactly what got done today and what’s currently blocking progress.

This layer is referenced constantly while the agent is working. It forms a bridge between the giant checkpoint map and the immediate task. By clearly separating "Done" from "Blocked," it forces clarity and stops the model from hallucinating progress it hasn't actually made yet.

The Active Context Layer (Short-Term)

This is the micro layer, and it is pure execution. It throws out all the historical baggage and focuses on one thing: what the agent needs to know right this second to write the next line of code.

This layer only tracks immediate constraints, active variables, and the specific file that is currently open. Because it strips away the past and the future, it’s incredibly token-efficient. It keeps the agent locked onto the immediate micro-task without getting distracted by previous mistakes or overarching project goals.

Routing is how we fix this. The concept is simple: dynamically direct incoming prompts to the most appropriate model based on the query itself.

This solves the classic engineering trilemma by saving expensive tokens for hard problems while reducing latency for simple tasks. I've been tinkering with different routing architectures lately, and I've noticed they naturally build on top of each other across four distinct levels of complexity.

Level 1: Just use an if-statement

The absolute simplest approach is using hardcoded logic to route the prompt before it ever touches a neural network. My favorite technique here is context length routing.

You just count the tokens before generation. If it's under 8k, I route it to a local Llama 3 8B instance running on my Mac. If it's over 100k, I hand it off to a model with a massive context window like Gemini 3.5 Pro.

You can also use regex to scan for keywords like "SQL" or "Python" to instantly trigger a specialized coding model. It costs nothing and has zero latency.

The catch? It’s incredibly fragile. A prompt like "Tell me a joke about Python" will falsely trigger your coding route, which means we need a slightly smarter approach to understand intent.

Level 2: Embedding-based semantic routing

To fix the fragility of regex, we have to move beyond exact keyword matches and actually evaluate the meaning of the prompt. This is where semantic routing steps in.

You define routing paths using a handful of exemplar sentences and convert them into vector embeddings. When a new query hits your API, you embed it and calculate the cosine similarity against your predefined paths.

I highly recommend checking out libraries like semantic-router for this. It's surprisingly fast if you run a tiny embedding model like all-MiniLM-L6-v2 locally on your CPU.

But as smart as semantic routing is, it still has a ceiling. Your accuracy depends entirely on maintaining a vector space and curating great exemplar data, which eventually becomes a maintenance bottleneck.

Level 3: Using a model to pick the model

When embeddings aren't enough, you can use an actual machine learning model to act as a traffic cop. The classic approach is fine-tuning a lightweight transformer like BERT to classify queries into specific intent buckets.

If you don't have a dataset to train BERT yet, you can use the "LLM-as-a-router" fallback. Just ask a fast, cheap model like Claude Haiku or Gemini Flash to read the query and output a JSON route.

A prompt like "Categorize this query as MATH, CREATIVE, or CHAT and output only the category name" works wonders. Constrained generation keeps the router from hallucinating.

Here is my favorite trick for this level. Use that LLM router to log 10,000 synthetic routing decisions, then use those logs to fine-tune a tiny BERT model so you can drop your routing latency back to practically zero.

Level 4: Cascading and escalation

Even with a perfect classifier, picking a single model upfront isn't always the right move. Cascading routing fixes this by dynamically escalating to a more capable model mid-flight.

In a single-turn setup, you send the query to the cheapest model first and have it score its own output. If the confidence score is low, you throw away the bad generation and escalate the prompt to a heavier frontier model.

You can also do this statefully across a conversation. Start a chat session with a small, local model and monitor the state to see if it gets stuck.

If the conversation hits five continuous turns without resolving, or the user repeatedly types "No, that's not what I meant," you pull the ripcord. You pass the entire context history to an advanced model like GPT-5 to step in, figure out the mess, and finish the job.

If your agent decides to ask a clarifying question instead of giving an immediate answer, a static dataset has no way to respond, and the test just breaks. Because of this, developers often default to manual human QA, which is painfully slow, or they rely almost entirely on online testing.

Testing in production by shipping it and monitoring live interactions is tempting, but it's incredibly risky. You don't want to discover that a minor prompt tweak broke your fallback routing just because it frustrated hundreds of real users first. The feedback loop is too slow, and burning real user goodwill is expensive.

My solution to this lately is building a "Simulation User"—an LLM specifically prompted to act as a human talking to my AI agent. This accelerates the evaluation loop dramatically, lets me test specific personas, and solves the headache of managing mock data for tool integrations.

Bootstrapping personas from real data

You can dictate a simulator's behavior, goals, and communication style just by tweaking its system prompt. But instead of guessing what your users will say, I always prefer to bootstrap these personas directly from actual historical chat logs. You can mine anonymized data to extract common intents, weird phrasings, and actual edge cases, then use those insights to automatically generate scenario prompts for the simulator.

I like setting up a few distinct personas to really stress-test the agent. For example, there's the "Happy Path" user who is clear and concise, contrasted with the "Chaotic" user who uses slang, gives partial info, and constantly changes the subject. I also throw in a "Frustrated" customer to specifically test the agent's empathy, de-escalation, and fallback routing. By combining historical data with defined personas, I can deterministically test the agent across thousands of highly realistic scenarios.

Grading the results with LLM-as-a-judge

Once your Simulator and Agent are chatting, you can automate the process to spin up 100 concurrent conversations in minutes. But this introduces a new bottleneck: manual QA. Every time I tweak an agent's system prompt, I risk breaking something else in a regression, and I absolutely do not want to read 100 simulated transcripts manually to see if they worked.

Instead, I take the completed conversation logs between the agent and the simulator and pipe them straight into an LLM-as-a-judge workflow for evaluation. I do this because it's the only practical way to scale complex, qualitative grading across hundreds of test runs without blocking the release cycle. I just hand the judge model the transcript and a strict rubric to evaluate task completion (did they reach the goal?), turn count (was it too slow or redundant?), and tone (did the agent stay polite and within guardrails?). Hooking this up to a CI pipeline means developers get instant, quantitative metrics every time they push code.

Solving the tool integration headache

Evaluating the back-and-forth chat is only half the battle. Real AI agents actually take action by executing database lookups and hitting APIs. Testing this without spamming production databases or writing brittle mock servers is a huge pain.

My simulation environment solves this by intercepting the agent's tool calls at runtime—like lookup_order_status(123). Instead of hitting a real database, I have the framework use a fast, cheap LLM to generate a plausible mock JSON response on the fly.

I am a massive fan of boring, foundational technology. But while this paper makes one excellent point, its overarching conclusion is fundamentally flawed.

Stacking the deck for grep

The paper's central claim about the superiority of grep is built on a heavily stacked deck. The authors evaluated their claims using a tiny 116-question subset of a benchmark focused on long-horizon conversational memory.

While grep is great at finding specific dates in a chat log, this dataset is wildly unrepresentative of the challenges we face building real RAG systems. Navigating dense technical documentation or parsing legal contracts requires semantic understanding, which is exactly where grep fails.

Worse, the authors pre-processed their data into a highly structured format optimized for regex. They essentially solved the hardest part of the problem beforehand. It is entirely unsurprising that a tool designed for exact pattern matching wins when the data has been explicitly formatted into exact patterns.

The semantic and scalability walls

If you try to use this approach in a real-world application right now, you will immediately hit two walls. The first is semantic flexibility. If a user asks your agent about "termination clauses," but the document uses the phrase "cancellation conditions," a simple grep returns nothing.

The second wall is scalability. grep requires a full linear scan of your text. While that works for small local datasets, it falls apart against enterprise corpora. Running broad regex scans over hundreds of thousands of documents is computationally expensive and slow. Your agents will quickly exhaust their token budgets and spike latency trying to triage massive, unstructured shell outputs.

Breaking down the middle ground

The most frustrating aspect of this paper is the false dichotomy it presents. It assumes you must choose between computationally expensive dense vector search or unscalable, linear shell commands.

We shouldn't regress to linear shell commands when there is an established, robust middle ground. If we actually want to build practical tools, here are three better approaches you can use right now.

Pure BM25

BM25 uses an inverted index, meaning a search across millions of documents is nearly instantaneous. Better yet, it actually ranks the results based on term frequency.

This gives you the speed and precision of keyword matching without forcing your agent to do the heavy lifting of sorting through endless shell output.

Agent-Weighted Hybrid Search

You don't actually have to choose between lexical and semantic search. You can easily run both BM25 and a standard vector search in parallel.

The fun part is using a lightweight AI agent to dynamically choose the weights of each based on the query. If a user asks for a specific "error code 502", the agent cranks up the BM25 weight; if they ask a conceptual question, it leans heavily on the vector search.

Vectorless RAG

I won't rehash my full deep-dive on Vectorless RAG here, but the TL;DR is to throw out the vector database entirely and replace it with an LLM-driven reasoning loop.

Instead of arbitrary chunking, you parse your document into a semantic JSON tree. The agent reads this "Table of Contents," reasons about which section holds the answer, and fetches the exact, unfragmented text.

By now, the new reality is obvious: generating natural language is no longer the bottleneck. Models can spit out incredibly fluent dialogue for practically zero cost.

The real scarce resources in 2026 are token budgets, API rate limits, and the model's context window.

Managing those exact constraints is why harness engineering has become the defining trend for conversational AI this year.

Why harnesses matter right now

An agent harness is everything surrounding an AI model that grounds it in reality and ties it to a specific conversational flow.

Think of it like a dog's walking harness. It anchors the conversational agent to prevent it from drifting wildly off-topic, hallucinating fake policies, or bankrupting your token budget with endless loops.

As an industry, we are aggressively shifting from trying to write the perfect "system prompt" to building robust execution environments.

I’m finding that building the right harness is the only way to deploy reliable chat interfaces, especially if you want to run smaller models locally on constrained hardware.

Anatomy of a conversational harness

Unlike traditional machine learning harnesses, which are basically glorified test suites, a conversational harness manages the live interaction loop.

It wraps the core chat loop and provides concrete tool registries. This is how you give your model safe access to execute external APIs—like checking a user's order status or fetching live weather data—without letting it make unauthorized state changes.

A massive part of this is context management. You need primitives that automatically compact older conversation history to protect the model's limited context window during long chat sessions.

You also need strict guardrails. I always implement hard limits to kill responses if a bot starts repeating itself or triggers too many internal tool calls before answering the user.

Another trick I love is deterministic execution. Don't let a black-box AI handle sensitive data collection.

When a user needs to authenticate or enter a credit card, the harness should intercept that intent. Offload that predictable task to traditional UI components, securely process it, and hand control back to the AI.

Adapting systems for chat agents

This requires a total mindset shift because we actually have to design our backend APIs to be agent-friendly.

Standardization is a massive multiplier here. If your internal APIs return predictable, constrained data, it requires far less attention for a local model to parse the results and reply to the user.

My favorite technique right now is prompt-injecting through API error messages.

Instead of an API just returning a 400 error to the bot, I engineer my endpoints to return specific remediation steps. If an order lookup fails, the error tells the bot, "Ask the user to confirm their 5-digit zip code." It acts as a hidden prompt injection that guides the model to self-heal the conversation.

You also need just-in-time context. Don't front-load your entire company FAQ and overwhelm the system prompt.

A smart harness waits to inject the return policy into the context window until the user actually asks about refunds.

It works... until it doesn’t.

I’ve been banging my head against the wall with traditional RAG lately, especially on dense technical documentation. Blindly slicing a document into "chunks" obliterates the author's narrative flow. Worse, semantic similarity is a terrible proxy for factual relevance. Just because a chunk sounds like the query doesn't mean it holds the answer.

Lately, I’ve been experimenting with a totally different approach: Vectorless RAG (sometimes called Reasoning-based RAG). It throws out the vector database entirely. Instead of static math, it uses an LLM to perform agentic, context-aware retrieval.

Here’s a breakdown of how it works, what the trade-offs are, and how this exact same pattern is quietly taking over codebase search tools like Claude Code.

How Vectorless RAG Works

Vectorless RAG treats retrieval as an iterative reasoning task. It’s basically teaching an LLM how to read a book: look at the table of contents, find the right chapter, read it, and see if you have the answer.

Phase 1: The "In-Context" Tree Index

Instead of artificial chunking, we parse the document into a semantic, JSON-based hierarchy—essentially a highly detailed Table of Contents. This tree structure lives right in the LLM's context window.

• Nodes: Chapters or sections become nodes.

• Metadata: Every node gets a node_id, a title, a brief summary, and pointers to the raw data (like page or line numbers).

• Hierarchy: Nodes contain sub-nodes, mapping out the whole document recursively.

Because we chunk by meaning (sections/chapters) rather than arbitrary token counts, we avoid context fragmentation entirely.

Phase 2: The Agentic Retrieval Loop

When a query comes in, the agent doesn't embed it. It reads the tree and executes a loop:

1. Read the ToC: Fetch the tree (just the structure and summaries, not the full text).

2. Reasoning: Evaluate the user's intent. Which node logically contains the answer?

3. Extract: Fetch the exact, unfragmented text for that specific node_id.

4. Evaluate: Ask: "Is this enough to answer the question?" If yes, generate the response. If no, go back to step 1 and pick a different node based on what was just learned.

The Claude Code Parallel: Vectorless Codebase Search

The shift away from vector DBs isn't just for PDFs. I've noticed the exact same architectural shift happening in developer tools. Look at how Anthropic's Claude Code navigates massive local repositories. It doesn't rely on embedded code snippets; it operates as an agent (Understand → Plan → Act → Verify).

Here is how Claude Code mirrors the Vectorless RAG pattern:

The Trade-Offs: Is it worth it?

Vectorless RAG solves the semantic mismatch problem, but it introduces new constraints. Here is the pragmatic breakdown.

The Good

• True Relevance: Queries are about intent. An agent can deduce that "how to handle errors" maps to a specific chapter, even if the semantic overlap is low.

• Zero Fragmentation: You get whole, coherent sections of text. Hallucinations drop significantly.

• Handles Cross-References: Traditional RAG chokes on "see Appendix G" because the text lacks similarity to the target data. An agent just looks up Appendix G in its ToC.

• Infrastructure: You can rip out your vector database entirely.

The Bad

• Latency is high: A vector lookup takes milliseconds. An LLM reading a JSON tree and executing a multi-step reasoning loop takes seconds. You have to design your UI around this delay.

• It gets expensive: Pumping a massive ToC into the prompt for every query, plus the tokens for the reasoning loop, burns through API credits much faster than a static vector search.

• Scale Limits: You can't put the ToC of a million documents into a prompt. For massive corpora, you still need a traditional search pass to pre-filter down to a handful of relevant documents before the agent takes over.

Final Thoughts

Vectorless RAG is a fascinating shift. By treating documents like structured narratives instead of bags of embeddings, we unlock a level of precision that traditional RAG struggles to match.

While I wouldn't use it to filter Wikipedia, for deep, accurate Q&A on complex specs or codebase engineering, agentic retrieval is rapidly becoming the new standard. If you're building local coding agents or high-stakes document tools, it's time to start experimenting with reasoning loops over static vector math.

In the first piece, All You Need is a Good Chunking, I described the "meat cleaver" approach: slicing documents arbitrarily by character or token count. It’s the easiest way to get a prototype running, but it’s fundamentally broken. It destroys context. When you feed these broken, out-of-context chunks to an LLM, the result is hallucination and confusion.

My follow-up piece dives into how we move past the meat cleaver and pick up the scalpel. Here is how I'm thinking about the three state-of-the-art approaches I outlined:

Semantic Chunking

This is my current go-to for most of my personal projects. The math here—calculating cosine similarity between sentence embeddings to find "valleys" in the topic—is incredibly cheap to run.

If you're using an embedding model like all-MiniLM-L6-v2 locally, you don't even need a GPU; it runs blazingly fast on a Mac M-series chip. I’ve built prototypes using sqlite-vec (the successor to sqlite-vss) where I just store the individual sentences and their embeddings in a SQLite database, and then run a quick Python script to group them based on similarity drops. It's a massive upgrade over fixed-size chunking and costs fractions of a cent.

Neural Chunking

This is a fascinating approach that I haven't experimented with as much yet. Using a custom BERT model specifically trained for sequence classification and boundary detection makes a ton of sense for highly structured documents.

I really appreciate that tools like Chonkie are bundling this. The barrier to entry for running specialized NLP models used to be configuring a massive PyTorch pipeline; now it's just a pip install away. The latency overhead is a real trade-off, though, especially if you are trying to ingest documents on the fly.

Agentic Chunking

This is the absolute gold standard for quality, and it's where things get really interesting from a prompt engineering perspective.

I noted in the original post that this is "exorbitantly expensive," but the math on that is actually changing rapidly. A year ago, using GPT-4 for this would have completely blown your API budget. Today? Running an agentic "proposition extraction" pipeline using Gemini 1.5 Flash or Claude 3.5 Haiku is shockingly affordable.

I've been running experiments piping messy documents through these fast, cheap models, prompting them to rewrite the text into self-contained propositions before embedding them. It solves the "pronoun problem" (where a chunk starts with "He did it" and the embedding model has no idea who "He" is) beautifully.

The Takeaway

I strongly stand by my golden rule: Don't use Agentic chunking if Semantic gets the job done.

Start with the dumbest thing that could possibly work. If token limits fail, upgrade to semantic chunking. Only pull out the heavy LLM-based agentic chunking if you are working with incredibly messy data and your retrieval metrics prove you actually need it.

The culprit is almost always the chunking strategy. The core tension of RAG is chunk size: small chunks give you precise search hits, while large chunks give the LLM the surrounding context it needs to actually understand the text.

If you just naively slice a document up by character count, you end up chopping crucial sentences in half. You feed the LLM broken context, and it confidently hallucinates a response.

I’ve been looking at three distinct ways to fix this—Sentence, Recursive, and Hierarchical chunking—and the tooling ecosystem around them is finally getting genuinely interesting.

Sentence chunking: respecting grammatical boundaries

A sentence chunker tries to break text strictly at natural boundaries like periods or exclamation points. It splits the document into individual sentences, and then greedily batches them together until it hits your token limit.

Historically, the standard way to do this in LangChain was by dragging in massive Natural Language Processing libraries like spaCy or NLTK. I always hated this approach. Pulling in hundreds of megabytes of heavy NLP dependencies just to find a period feels absurdly wasteful.

This is why I’ve been really enjoying Chonkie. It’s a newer, ultra-lightweight library that handles sentence chunking perfectly while only requiring an 11MB base install. It runs exponentially faster than the older NLP approaches and respects your token limits beautifully.

Recursive chunking: the pragmatic default

Recursive chunking is the absolute gold standard for general-purpose text. Instead of a single rule, it uses a prioritized waterfall of separators.

It first tries to split by double newlines (\n\n) to get natural paragraphs. If a paragraph exceeds your token limit, it falls back to single newlines (\n), then spaces, and finally individual characters.

LangChain’s RecursiveCharacterTextSplitter does exactly this using standard string operations, which means it executes incredibly fast on virtually any hardware. Chonkie also ships with a highly optimized recursive chunker that pairs neatly with your specific LLM's tokenizer.

Because it prioritizes newlines, it perfectly preserves Markdown headers, bulleted lists, and code blocks. Unless you have a specific reason not to, this should be your default strategy.

The catch is that grammatical chunkers fail spectacularly on structured text. If you feed them Markdown lists or code blocks that lack periods, they completely mangle the formatting. I only use this approach for massive walls of unstructured prose, like podcast transcripts or raw audio logs.

Hierarchical chunking: throwing vision models at PDFs

What happens when your source material is a messy corporate PDF full of tables and multi-column layouts? Recursive chunkers completely choke on these because the line breaks are essentially arbitrary.

Hierarchical chunking treats the document as a visual structure rather than a flat string. It identifies where a table or a section lives, and injects a "breadcrumb trail" of context into every single chunk. A tiny bullet point gets permanently tagged with its parent path, like [Annual Report → Financials → Q3].

IBM’s Docling is currently the most compelling tool I’ve seen for this. It uses vision-language AI models (like Granite-Docling) to perform layout analysis on raw PDFs before doing any text splitting. Its HybridChunker ensures a table is never split across chunks and that context is perfectly preserved.

The trade-off here is hardware. Running vision models over a 100-page PDF will absolutely punish your unified memory and spike your ingestion costs. It is computationally heavy, but if you need to reliably query financial reports or complex contracts, it is practically mandatory.

The verdict

Choosing the right chunking method dictates the ceiling of your entire RAG pipeline. Here is my current mental model for picking one:

Start with a recursive chunker around 500 tokens. It's fast, cheap, and handles 90% of what most people are trying to build.

Only upgrade to heavy, structure-aware tools like Docling when you actually observe your pipeline failing to read tables or losing the plot on complex documents.

Here is a summary of our main points:

We defined this new paradigm as "Agent Experience" (AX), where the user’s role evolves from an active operator to a supervisor. Unlike simple assistants like Siri, true AI agents can independently manage complex, multi-step goals. This means users will no longer navigate intricate workflows but will instead state their objectives and oversee the AI's execution.

Our article posits that designers must now build "cockpits" or dashboards for monitoring and intervention, rather than step-by-step task flows. In this new world, natural language becomes the primary interface, and the core design challenge is to establish trust by ensuring the user remains the ultimate authority with clear pathways to step in and manage their AI counterparts. We concluded that the companies that succeed will be those that best empower users to supervise these increasingly autonomous systems confidently.

Read the Full Article Here

Then came the Generative AI explosion. Suddenly, the old playbook feels inadequate. We're no longer just predicting a "churn" vs. "no churn" label. We’re generating nuanced text for marketing, complex code for features, and intricate product designs. The very definition of a "good" output has become subjective and context-dependent.

This paradigm shift is forcing us to rethink evaluation from the ground up. For those on the front lines building these products, this isn't just an academic exercise; it's a critical bottleneck to shipping reliable software.

The Two Worlds of GenAI Evaluation: Closed and Open-Ended

The first step to building a coherent strategy around GenAI is to distinguish between the two fundamental types of tasks your system might be performing.

Closed-Ended Predictions:

This is familiar territory. The model's job is to produce a specific, constrained output. Because there's a definite "right" answer, we can lean on our traditional toolkit.

Example: You're building a feature to automatically categorize incoming support tickets ("Billing Issue", "Technical Glitch", "Feature Request").

Why it's closed-ended: There's a predefined, finite set of correct labels.

How you measure it: You can use precision (of all the tickets we labeled "Billing Issue," how many actually were?) and recall (of all the actual "Billing Issue" tickets, how many did we find?). These are clear, quantifiable KPIs you can build dashboards around and track release over release.

Open-Ended Predictions:

This is where the real challenge begins. Think of tasks where there are many possible "good" answers.

Example: You're launching an AI-powered feature that summarizes long customer feedback emails for internal teams.

Why it's open-ended: A 500-word email can be summarized effectively in dozens of different ways. Which one is best? It depends on what the reader needs to know.

The problem: Simple one-to-one comparisons with a "golden" summary in a test set are no longer sufficient. This is the core challenge for teams building the next generation of AI-powered features.

The Evaluation Toolkit for Open-Ended Generation

While the problem is complex, we're not flying completely blind. The field has developed several methods to bring structure to this ambiguity.

The Classics: BLEU and ROUGE

These metrics are your first-line, automated tools. They compare the words and phrases in the model's output to one or more human-written reference examples.

Let's take our feedback summarizer feature. Suppose the original feedback is: "The new dashboard is visually appealing, but the process to export reports is now much slower and requires three extra clicks. I also can't find the date filter easily."

Reference Summary: "User likes the new dashboard's look but finds report exporting slower and the date

The Modern Approaches: Semantic Similarity and LLM-as-a-Judge

To get closer to measuring actual quality, we need more sophisticated tools.

1. Semantic Evaluation: This measures if the meaning is the same, even if the words are different. In the example above, a good semantic similarity metric would score "Generated Summary B" highly against the reference because the core concepts are identical. This is a much better signal of true understanding.

2. LLM-as-a-Judge: This is a game-changer for iterating quickly. You use a powerful LLM as a tireless, automated evaluator. As a team, you define the rubric.

Example: To evaluate the summaries from your AI feature, you can use an API call to a "Judge" LLM with a prompt like this:

You are an expert editor. Please evaluate the following AI-generated summary based on the original customer feedback.

Original Feedback: [Insert original email here]

AI-Generated Summary: [Insert summary to be evaluated here]

Please score the summary on a scale of 1-5 for the following criteria:

1. Conciseness: Is the summary brief and to the point?

2. Factual Accuracy: Does the summary correctly represent all key points from the original feedback?

3. Clarity: Is the summary unambiguous and easy to understand?

Provide a score and a brief justification for each.

This approach is faster and cheaper than constant human review and allows you to scale your evaluation based on criteria that your team defines as important for the product.

Special Case: Evaluating RAG Systems

Retrieval-Augmented Generation (RAG) is the architecture powering most modern enterprise chatbots. It has two parts: finding the right info (Retrieval) and then using it to talk (Generation). To debug effectively, you must evaluate them separately.

• Example: A customer support chatbot that answers questions based on a company's knowledge base. A user asks, "How do I get a refund for my subscription?"

• 1. Evaluate the Retrieval: Did the system pull the correct document?

  • Success: The RAG system retrieves the "Refund Policy" article.

  • Failure: The system retrieves the "Subscription Upgrade" article.

  • The Metric: Use classic search metrics like NDCG or simple top-k hit rate (e.g., "was the correct document in the top 3 results 95% of the time?"). This tells you if your retrieval component—be it a vector database or a search API—is effective.

• 2. Evaluate the Generation: Assuming it found the right document, did it generate a good answer?

  • Success: "To get a refund, go to your Account Settings, click 'Subscription,' and follow the 'Cancel and Refund' link. You are eligible for a refund if you cancel within 14 days of purchase."

  • Failure: "The document mentions refunds." (Accurate but useless).

  • The Metric: Here you use the open-ended techniques: LLM-as-a-Judge to check for helpfulness, or human spot-checks.

By separating these, you can pinpoint the real problem. Is the chatbot failing because the knowledge base is bad (a content/retrieval problem) or because it's bad at explaining things (a generation problem)? This distinction is critical for assigning bugs and planning sprints.

It All Comes Down to Data and a Hybrid Approach

The uncomfortable truth for any team building with AI is that the biggest challenge isn't the metrics; it's the test data. Creating a high-quality, diverse evaluation set that reflects real-world usage is an expensive, ongoing engineering and product effort. I’ve seen many teams deprioritize this, only to build evaluation systems that show green lights while the actual user experience stagnates.

The most successful GenAI companies, like OpenAI and Anthropic, have a core competency in data curation for evaluation. This is not a coincidence.

An Action Plan for Teams:

1. Treat the Test Set as a Product: It needs to be versioned, maintained, and expanded alongside your main product. This is a shared responsibility: product managers define the key use cases to cover, and engineers build the infrastructure to test against them reliably.

2. Build a Hybrid Evaluation Dashboard: Don't rely on one number. Your team's dashboard should provide a complete picture:

   • Automated Metrics (ROUGE/Semantic Similarity): For a quick, directional pulse in your CI/CD pipeline.

   • LLM-as-a-Judge Scores: For scalable, criteria-based quality checks on nuanced attributes like "helpfulness" or "brand tone."

   • Human Feedback: A direct pipeline from a "thumbs up/down" button in the UI or periodic human reviews on critical flows.

3. Define "Quality" Holistically: Your team's definition of done must go beyond factual correctness. Your evaluation rubric should include:

   • Safety: Does it avoid harmful or inappropriate language?

   • Tone of Voice: Does it sound like your brand?

   • User Satisfaction: Is the user actually achieving their goal?

4. Deconstruct Your Systems: For multi-step systems like RAG, insist on component-level metrics. When a metric drops, you need to know which part of the system broke to debug efficiently.

Building in the GenAI era requires a new level of rigor in how we define and measure quality. Moving past simplistic scores to a holistic evaluation framework is no longer optional—it's the core work of building AI products that are not just impressive demos, but are also reliable, safe, and genuinely useful.

AI Is Becoming More Intuitive

The early days of LLMs often required users to experiment with different phrasings to get the best results. However, modern AI models are trained on vast amounts of data and improved architectures that enable them to interpret instructions more naturally. Instead of needing a carefully structured prompt, today’s models can process vague, casual, or even slightly ambiguous commands with ease.

For example, early models required precise formatting, explicit step-by-step breakdowns, and structured wording. Now, newer models can infer context, understand implied meaning, and generate useful outputs without the need for complex prompt tuning. This means that instead of focusing on how to "trick" the AI into giving the best answer, users can simply ask questions as they would to a knowledgeable human.

The Overhyped Industry of "Prompt Engineering"

As with any emerging technology, a subset of self-proclaimed experts have positioned themselves as gatekeepers, offering courses, guides, and consulting services on how to craft the perfect prompt. While some strategies may have been helpful in the past, the need for such expertise is rapidly fading.

Most prompt engineering advice boils down to common-sense practices like being clear, specifying output format, or providing context—all things that even casual users can figure out intuitively. The AI itself is improving at handling ambiguity, reducing the necessity for highly refined prompts. As a result, the idea that businesses need dedicated "prompt specialists" is becoming increasingly outdated.

The Future: Conversational AI, Not Manual Tweaking

Instead of relying on highly specific prompts, the future of LLMs is in their ability to engage in dynamic, natural conversations. AI systems are evolving to ask clarifying questions, refine their own outputs, and adapt based on user feedback. This means that rather than needing a human to master a rigid prompting technique, AI itself will adjust based on user intent.

Think about how we interact with human assistants: we don’t script perfect instructions in advance; we communicate, clarify, and refine our requests in real time. That’s exactly where AI is heading. The need to manually craft prompts will soon be seen as an unnecessary relic of early AI experimentation.

1. Define SLIs and SLOs Beyond Model Accuracy

Traditional ML metrics like accuracy or F1 scores are insufficient for production systems. SRE emphasizes Service Level Indicators (SLIs) and Service Level Objectives (SLOs) to quantify reliability. For ML systems, this includes:

• Latency: Response time for model inference.

• Availability: Uptime of ML APIs or services .

• Data Drift: Monitoring input distribution shifts that degrade model performance.
  By setting SLOs for these metrics, teams can prioritize reliability alongside accuracy.

2. Automate Deployment and Monitoring

SRE reduces manual toil through automation, a practice critical for ML workflows:

• CI/CD Pipelines: Automate model deployment with rollback capabilities to handle faulty updates.

• Self-Healing Systems: Use ML to detect anomalies (e.g., data pipeline failures) and trigger remediation.

• Testing: Integrate automated canary testing to validate model performance in staging before full rollout.
  Automation minimizes human error and accelerates iteration.

3. Prioritize Observability for Silent Failures

ML systems often fail silently (e.g., gradual accuracy decay). SRE-inspired observability includes:

• Model Metrics: Track precision/recall over time and correlate with infrastructure health.

• Data Lineage: Monitor data pipelines to catch preprocessing errors or missing features.

• Root Cause Analysis: Use tools like tracing to link model failures to specific code or data changes.
  Comprehensive observability helps detect issues before users are impacted.

4. Formalize Incident Response for Model Failures

Treat model failures like system outages using SRE incident management practices:

• Runbooks: Document steps to diagnose and resolve common issues (e.g., data drift).

• Blameless Postmortems: Analyze failures to improve processes rather than assign blame.

• Escalation Paths: Define roles for triaging severe incidents (e.g., automated rollbacks vs. human intervention).
  Proactive incident management reduces downtime and builds trust.

5. Design for Resilience

SRE emphasizes building systems that withstand failures. ML engineers should:

• Implement Fallbacks: Deploy simpler models (e.g., rule-based systems) as backups during outages.

• Redundancy: Replicate data pipelines and model servers to avoid single points of failure.

• Chaos Engineering: Test system resilience by intentionally injecting failures (e.g., synthetic data corruption).
  Resilient design ensures graceful degradation under stress.

Conclusion

Adopting SRE principles bridges the gap between experimental ML and production-grade systems. By focusing on reliability metrics, automation, observability, and resilience, ML engineers can create solutions that are not just innovative but also dependable at scale. As ML systems grow in complexity, the SRE mindset—proactive, data-driven, and iterative—will be indispensable for maintaining performance and user trust .

Key Provisions and Banned AI Applications

As of today, certain AI applications are outright banned to protect fundamental rights, privacy, and societal well-being. These include:

❌ Social Scoring Based on Personal Behavior: AI systems that rank individuals based on their social conduct, similar to systems used in some authoritarian regimes.

❌ Manipulative or Deceptive AI: Technologies designed to influence user decisions through covert manipulation or exploitation of psychological vulnerabilities.

❌ AI Exploiting Vulnerabilities: Systems targeting individuals based on specific vulnerabilities related to age, disability, or socioeconomic status.

❌ Crime Prediction Based on Appearance: AI models making predictions about criminal behavior based solely on physical traits.

❌ Biometric AI Inferring Personal Characteristics: Technologies that deduce sensitive personal information, such as sexual orientation, from biometric data.

❌ Real-Time Biometric Surveillance in Public Spaces: The use of AI for continuous biometric monitoring in public areas without stringent legal oversight.

❌ Emotion Recognition at Work or School: AI tools aimed at analyzing emotions in professional or educational environments, which can lead to invasive surveillance.

❌ Facial Recognition Databases from Online Scraping: Databases created by harvesting facial images from the internet without explicit consent.

What This Means for Businesses

Organizations operating within the EU or offering AI products and services in the region must ensure compliance with these regulations. Non-compliance can result in hefty fines and reputational damage. Businesses should:

• Conduct thorough audits of their AI systems.

• Eliminate or modify prohibited functionalities.

• Implement robust governance frameworks to oversee AI ethics and compliance.

Looking Ahead

The EU AI Act represents a significant shift towards ethical AI development and responsible deployment. As additional compliance deadlines approach, businesses must stay proactive, continuously adapting to meet evolving regulatory requirements. This new era of AI governance is not just about legal compliance—it’s about fostering trust, accountability, and fairness in technology.

Enter Deep Reinforcement Learning (DRL)

Unlike traditional models that optimize for short-term gains, such as achieving an immediate click, Deep Reinforcement Learning (DRL) takes a broader perspective. It focuses on maximizing user lifetime value (LTV), a metric that captures the long-term financial contribution of a user.

🔍 What Does This Mean in Practice?

🔥 1. Beyond the First Click

DRL models evaluate how each ad impression influences not just initial clicks but also downstream actions like sign-ups, purchases, repeat visits, and brand loyalty. It shifts the focus from “Did they click?” to “Did that click lead to meaningful engagement?” By analyzing long-term user behavior, DRL ensures that ads are optimized to attract users who are more likely to convert into loyal customers.

♻️ 2. Handling Delayed Rewards

In traditional models, the value of an ad is often judged immediately after the click. DRL, however, excels in environments where rewards are delayed. Even if a user doesn’t convert right away, DRL algorithms, such as Q-learning with discount factors, track the impact of that interaction over time. This approach allows advertisers to understand the cumulative value of each ad impression, accounting for both immediate responses and future actions.

🎯 3. Smarter Budget Allocation

When advertisers chase clicks, they risk overspending on ads that generate high CTR but low conversion rates. DRL changes the game by enabling smarter budget allocation. It helps advertisers identify and invest in strategies that nurture long-term customer relationships. This leads to maximized ROI, as funds are directed toward campaigns that drive sustainable growth rather than fleeting engagement.

The Bottom Line

While CTR can provide quick insights, it doesn’t capture the full story. Optimizing for long-term value through DRL allows advertisers to build deeper connections with their audience, enhance brand loyalty, and achieve greater financial returns. In the evolving landscape of AdTech, focusing on LTV isn’t just a smarter strategy—it’s the future of digital advertising.