Fix Agent Inconsistency by Curating Data, Not Rewriting Prompts

The central insight from Databricks' new Genie Agents guidance is a diagnostic reframe: when an agent gives inconsistent or wrong answers, the usual instinct is to fix the prompt. Databricks says that instinct is wrong. The real culprit is missing business context—and Genie Agents fix that at the source, not the surface.

Teams can now spin up domain-specific Genie Agents using Genie One or Genie Code from a single descriptive prompt, without manual configuration. A valid bootstrap: "Use our incident runbooks and service-health data to create an agent that helps support engineers investigate production incidents." That's enough for Genie to identify the problem domain and locate the relevant Unity Catalog assets.

The runtime connects to two data types. Structured: governed tables, metric definitions, dashboards, and data-quality rules in Unity Catalog. Unstructured: PDFs, Word documents, presentations, and images in Unity Catalog volumes. When a user asks a question, the agent retrieves the most relevant content across both modalities and reasons over them together. Permissions are enforced per user at query time via Unity Catalog RBAC—the agent never surfaces data the user couldn't already access.

The "first revenue table it finds" problem is what happens when context is thin. A generic agent hits a keyword match rather than the metric view finance actually maintains, returning a technically sourced but operationally wrong number. Genie Agents short-circuit this by grounding responses in curated Unity Catalog semantics: metric views, domains, and certification flags that distinguish the authoritative asset from the noise. The prompt provides direction; the Unity Catalog layer provides correctness.

Databricks prescribes a narrow-first rollout. Pick a single repetitive analysis job—incident investigation, product FAQ, cloud-cost explanation, data pipeline monitoring. Run the agent against past cases where the right answer is already known. For the incident agent, that means checking whether it cites the correct runbook and service-health table, not just whether it arrives at the right root cause. Genie Agents include built-in benchmark tooling: define a question set with expected answers, run it, get a numeric accuracy score. Re-run the same benchmark after each context update so "seems better" becomes a trackable number. A monitor tab surfaces the actual questions users submit in production, feeding gaps directly back into the benchmark as the agent expands.

Expansion is additive, not architectural. A Sales Opportunity Data agent starts by surfacing pipeline risk from CRM tables; a later version adds deal-summary drafting and win-rate Q&A by segment. A Logistics Management agent graduates from flagging shipment delays to recommending reroutes and tracking carrier performance. Each expansion layer builds on a version that has already passed benchmarks, not one that "felt good" in testing.

The hard part—which Databricks is explicit about—is curation, not prompting. Single-prompt creation doesn't compress the work of maintaining authoritative data. It just makes that work self-serve. If Unity Catalog semantics are stale or runbooks are outdated, the agent surfaces that staleness at scale. The canonical failure mode isn't a bad prompt; it's an outdated runbook the agent cites with confidence.

Architect's takeaway: if your Genie Agent returns inconsistent answers, audit your Unity Catalog assets before touching the prompt.