Bender: Vectara’s AI agent for engineering operations

Here’s something nobody wants to deal with.

A few weeks ago, one of our indexing services started logging failures. The service at fault was Titan, which turns uploaded content into searchable data for Vectara’s retrieval layer. Karpenter, an automated cloud server manager that sizes and scales infrastructure to save costs, evicted a Titan pod during a rollout. This is serious because it could prevent our customers from uploading data to the Vectara Platform.

We traced the problem to a bug in Titan’s code, involving a race condition between its acceptance of new requests and its shutdown process. But we decided not to fix this bug ourselves. Instead, we handed it off to Bender, our engineering operations agent.

We built Bender on the Vectara Platform. It can read, investigate, update incident records, and open pull requests. The only thing it can’t do is deploy code or directly change production workloads or configurations. Those are mission-impacting events, so we gate them on human review.

Bender autonomously wrote a fix for the Titan bug, which we reviewed and deployed. Once the code was live, Titan gracefully balanced its workload between handling new requests and completing in-flight work. From that point onward, we observed none of the failures that we originally saw.

What Bender does

In most production problems, evidence is spread across tools. This adds complexity. The alert, for example, could be in Datadog, the deploy in Argo CD, and the pod state in Kubernetes. Our Bender agent delivers context by providing clear information that on-call engineers need and can work with. The agent’s job is to:

• Find the likely cause: It reads logs, Kubernetes state, recent deployments, the application code, and correlates requests with trace IDs across logs. Each diagnosis is backed by evidence. It also says where it stops.
• Sort alerts by importance: Test traffic, internal failures, and monitoring checks all cases alerts to fire. So do customer-facing outages. The agent classifies each one so the team knows which is which.
• Prepare the fix: When the fix is clear, Bender writes the code or config change and opens a pull request. It does not deploy. A pull request must pass through the team's normal review, but the process moves quickly because the evidence gathered by Bender helps us understand the change.
• Keep incidents moving between shifts: On a schedule, Bender refreshes evidence on open incidents, checks whether an old problem has resurfaced, and flags the ones that need a new human decision. It writes the shift handoff and keeps the incident timeline. Before it opens anything, it checks the history to avoid creating duplicates.

Performance review

Let’s take a look at what this means in practice. Here’s a one-month performance review for Bender. Over the period of July 21 through August 21, 2026:

• Bender reviewed 84 incident records using Rootly, a popular AI-native on-call and incident management platform.
• It opened 187 pull requests across Vectara repositories. 130 of those were merged in the same month.
• As part of its work, Bender linked Slack triage threads to Rootly, checked for duplicates, and pulled evidence from relevant sources, including Datadog, Kubernetes, Prometheus, and Argo CD.
• Its workload covered test traffic, internal failures, monitoring checks, and customer-affecting incidents while retaining evidence trails, CI, and human review gates.

Not bad for a team member who never comes around the office.

Building Bender

We built Bender using our own agentic system, which we think of as four layers:

1. The Vectara Platform forms the core. It provides the main architecture, agent runtime and harness, tools, sub-agents, and access controls.
2. Configuration defines agentic behavior through specific instructions, tool configurations, and context management directives. It can be used for routing and approval.
3. Custom integrations connect agents to specific tools such as GitHub, Slack, Notion, and our operational systems. While we offer prebuilt integrations, they’re typically customized for each use case’s environment and security requirements.
4. Operating processes are the internal rules: the escalation paths, impact classification, and verification checks encoded directly into the agent’s workflow.

Here are the key design decisions that drive Bender’s behavior:

• Production Kubernetes access is technically read-only at two independent layers. The MCP server starts in read-only mode, and its service account is restricted by Kubernetes RBAC to get, list, and watch operations on an allowlisted set of resources. It has no create, update, patch, or delete permissions and no pods/exec access, so it cannot mutate workloads or run commands inside containers. This is a deployment-enforced boundary rather than a prompt instruction.
• Bender escalates when relevant evidence is unavailable, a human decision is required, an enterprise customer signal is detected, or work falls outside its scope. It routes the request to a verified owner when one exists, otherwise to the responsible team.
• Code changes go through GitHub CI and repo checks, AI-assisted review where configured, and human review before being merged.

What we learned along the way

Building Bender was fun and resulted in a positive business impact, but we also gained many insights from the process:

1. Prevention requires better guardrails: One of Bender’s audit runs opened a high volume of pull requests, about 40% of which were duplicates, and suggested a change that weakened a current policy. The change was stopped by the human reviewer, who then corrected the agent's guardrails to prevent recurrent behavior. The audit failure taught us to implement duplicate detection, policy-aware checks, and stronger evaluation criteria while designing agents.
2. Narrow beats general: Having a role-specific agent with a clear scope is more useful than one assistant attempting everything. Work outside an agent’s scope is routed to the owner, so each agent’s boundary is predictable to its users.
3. System safety relies on architectural controls: Instructing an agent to seek confirmation before executing an operational action technically does not prevent it from doing so. True operational guarantees come from strong API controls and permission boundaries verified at each deployment. We learned to strictly adhere to this standard and advise anyone evaluating agent vendors to demand the same level of verification.

In future blog posts, we’ll share a few other agents we’ve built and discuss how we turned these insights into action.

Building your own agents

An engineer can apply these agentic patterns in multiple domains, even outside incident response. For example, one of our customers runs a code-review agent on code that describes how their chips move the data between hardware registers. The agent calls the company’s Vectara-powered assistant to pull the specs, requirements, and design docs that apply to the code under review. Since the agent is accessing multiple scenarios during code review, it can flag undetected errors that could be missed otherwise.

Want to put an agent to work? Start building on the Vectara Platform, for free.