Agentic Next-Best-Action with LangSmith Observability

The demo has two sides. Customers see a simple chat. Product, risk, and engineering teams see everything happening behind it. To start, pick a customer from a slate that spans low, medium, and high risk tiers. A low- or medium-risk customer gets a few clarifying questions and then a tailored card offer. A high-risk customer is screened out at the first turn and politely directed to a human agent. A live panel beside the chat tracks the agent as it works: the current workflow stage, the customer’s risk score against the guardrail threshold, how many clarifying questions have been used, and which offer was selected.

In LangSmith, every turn appears as a traced run with its latency and token count. Turns from the same chat are grouped into one conversation thread, and the thumbs-up or thumbs-down from the chat is attached to the specific reply it rated. Opening any turn shows the full chain of agent steps behind it, so an unexpected recommendation can be traced to the exact step that produced it. The same view holds offline experiments on a golden dataset of scripted conversations, so teams can compare a new prompt or model against the current one before it goes live.  Watch the demo

• Multi-turn LangGraph Multi-Agent System. The application processes incoming customer inquiries and proactively upsells credit card offers based on customer preferences.

• Customers are filtered by risk and rerouted to a different marketing channel if needed. Agents interact with the customer and consult an internal rule engine to come up with the best credit card offer for upsell.

• Agent decisions are tracked in real time via LangSmith Observability where stakeholders can monitor interactions and agent performance. Conversations tracked as root runs featuring nested runs to evaluate every piece of the interaction into a coherent logical thread.

Banks and card issuers use chat to offer upgrades and cross-sells, but three requirements keep a general-purpose LLM chatbot from handling that traffic:

1. Regulated recommendations. Every offer must be tied to a rule-based eligibility check on risk tier, age, and income, not left to the LLM’s judgment alone.

2. Fail-safe decline path. High-risk customers must be routed to a polite decline that quotes no specific product figures and redirects them to a human agent.

3. Auditability. Product, risk, and compliance teams need to inspect every conversation turn by turn, replay it against new prompts, and grade it offline before promoting a model change.

This blueprint meets all three by separating the conversation from the decision: the LLM talks, a rule engine and risk guardrail decide, and LangSmith records and grades every turn. The result is a personalized offer channel that runs at chat scale while staying explainable to the teams accountable for it.

The blueprint is built from five components. Four run on Cloudera AI, and LangSmith provides the observability layer:

• Multi-agent system: the NBA engine, built as a LangGraph workflow. Each customer message passes through intake, risk screening, and intent routing, then on to a clarifying question, an offer, a follow-up answer, or a decline. Conversation state is checkpointed per thread.

• Agent observability: LangSmith traces every turn as a root run, with each agent step nested beneath it. Turns are grouped into conversation threads, customer feedback is stored against each reply, and offline evaluation experiments run here too.

• Model serving: Cloudera AI Inference Service hosts the Nemotron LLM behind every language step: extracting customer details, routing, asking questions, pitching offers, and judging evaluations. It can also serve the optional XGBoost or PyTorch risk classifier, exported to ONNX.

• Data & metadata: the customer profile table that drives the risk guardrail, and the offer catalog with its eligibility rules, which feeds the rule engine. Both use SQLite in the demo.

• Web layer: a FastAPI service that exposes the agent as a JSON API and serves a lightweight TypeScript chat UI from a single port. It’s deployed as a Cloudera AI Application with no build step.