We’ve noticed that several of our retail clients are building new applications, or modernizing existing ones, that center on customer experience. The returns process is a key part of those applications: it supports retail operations and shapes customer satisfaction. We’ve built a demo application, Returns Desk, that addresses a core need in that process: deciding each return request quickly, consistently and at a known cost, using a language model only for the one step that needs it, reading what the customer wrote or photographed. The code, tests, sample data and Azure setup are public at github.com/keyva/returns-desk. It’s relevant to anyone planning to put generative AI into a customer-facing process, because where the model sits decides what the process costs, how far it can be trusted, and who answers for each decision.
Most of a business process should never reach a language model. The useful work is finding the one step that genuinely needs one, and building everything around it as ordinary, testable code.
When teams first put generative AI into a workflow, the temptation is to hand the model the whole job: read the request, apply the policy, work out the refund, write the reply. It demos well and fails quietly. The model is slow and costs money on every call, and it can’t promise to get the arithmetic right twice in a row. Worse, nobody can say which rule it applied.
We built Returns Desk to show the alternative on a process every retailer runs. It is the returns desk of Larkfield Home & Apparel, a fictitious retailer of kitchenware and clothing. Customers never see the desk itself: it is the back-office layer behind a self-service returns portal, where each request is approved, priced or sent to a person. The question it answers is simple: where exactly does the model belong?
The demo: a returns desk with one hard question
A customer asks to send an item back. They name the order and the item, and either pick a reason from a list or explain in their own words, often with a photo. Larkfield has to decide whether to refund, replace, deny or escalate, and who pays for the return.
Almost all of that is policy, and policy is a table. The return window, which categories can come back, when a photo is required, the per-quarter limit, the fee: each fits in a cell. Only one question has no cell, because nobody typed the answer into a field: who is responsible for the problem? Was it the retailer, the carrier or the customer, or is it unclear? Answering it means reading what a person wrote and looking at what they photographed.
We used one test to draw the line. If an intern could write the rule in an afternoon from the policy document, it is code. If the answer depends on reading what a person wrote or looking at a picture, it is the model.
How a return request moves through the desk
Policy turns away or settles most requests before a model is involved; only a customer’s own words reach the engines, and nothing takes effect until the validator has checked it.
What the rules handle: most of it, for free
Every rule Larkfield has lives in one small JSON file that fits on a slide. It sets a 30-day window counted from the carrier’s delivery date, three returns per customer per quarter, and categories such as perishables and final-sale items that can’t come back at all. A table says, for each category and responsible party, which outcomes are allowed, what fee applies and who pays return postage.
Policy checks every request before any model sees it, and many never go further:
- A clearance throw blanket marked final sale is denied because final-sale items can’t come back. No model is called and nothing is billed.
- A jacket returned 47 days after delivery is denied on the window, with a reply that cites the rule.
- A T-shirt returned with the reason “changed my mind” maps straight to the customer, who gets a refund. There is nothing to read, so no model is needed.
Because the rules are data, the business can change them without touching the model or its prompt. During the demo we raise the auto-approval limit in the file, reload, and watch a request that went to a person get approved automatically instead.
What the models handle: one question, asked two ways
When a customer explains in their own words, the request goes to two engines on Microsoft Foundry, both running the same model. They are asked exactly one question: who is responsible?
- The generative engine reads the statement and the photo. It answers with a responsible party, a one-sentence reason for the clerk, a draft reply to the customer, and any flags it noticed, such as damage in transit or possible abuse. Its output follows a fixed schema, so the code never parses free text.
- The probabilistic engine uses Foundry’s Decisions API to return a probability for each of the four parties and a confidence, with no prose. The probability for the named party is what lets policy set a floor for trusting the answer.
Take the crushed kettle. A $64 stovetop kettle arrives in a crushed box with its handle snapped off, and the customer attaches a photo. The generative engine named the carrier and flagged damage in transit. The probabilistic engine put the carrier at 84%. The board shows the agreement as a green chip.
Neither engine decides anything. Each answer is an input to policy and to the clerk, and neither engine ever picks the outcome or touches the amount.
Guardrails: the engines read, the code decides
A request is approved without a person only when every condition in policy holds:
- Both engines name the same responsible party.
- The probabilistic engine gives that party at least 80%.
- The refund is $75 or less.
- No blocking flag is raised, such as possible abuse or signs of use.
- The customer is under their per-quarter limit.
The kettle passes all five and is refunded $64 with no one involved. Everything else goes to the clerk, including any parcel the carrier never reported delivered, since policy can’t count a return window without a delivery date. A $140 stand mixer with the same crushed-box story goes to the clerk because it is over the limit. A blender with a grinding noise after two months is a real judgment call, where a defect and normal wear are both plausible. A customer who says headphones never arrived, against a delivery record and on their third request this quarter, gets flagged and escalated.
Every resolution then passes through a validator, whoever made it. The outcome must be one the table allows, and the refund must equal the price paid minus the fee. A replacement must be in stock, a denial must carry a note, and no refund can ever exceed what was paid. The arithmetic is never the model’s job.
There is a security point here too. Both engines read the same customer text, so a statement written to mislead one can mislead both, and their agreement proves nothing. The same goes for honest mistakes: both engines run the same model, so they tend to be wrong together, and agreement is a weak check on accuracy. What limits the damage is code: the cap, the table and the validator.
Measuring it: labeled requests, not impressions
A model in a business process needs a number behind it, not a good feeling from the demo. Returns Desk ships with 46 hand-written requests whose answer is already known. Forty include a customer statement, ten for each responsible party, written with the variety real customers bring. The other six carry only a reason code.
An evaluation runs both engines over every labeled request. It reports how often each engine named the known party, a grid of known party against named party, and the cost per request. In production, each resolution a clerk confirms would become a new labeled request, so the measurement could grow with use. The demo only re-scores its own sample set.
Two choices kept it honest. We wrote the statements by hand rather than generating them, so the test doesn’t measure how well one model reads another model’s writing. And we left out claims about how well-calibrated the probabilities are, because 40 requests are too few to support them. That makes the 80% floor a starting setting, not a proven one. Tune it against labeled requests before trusting it with money.
What changes at enterprise scale
The split between rules and model holds at any size. What a production rollout adds is everything around it:
- Requests arrive from the returns portal and the contact center. Approved refunds go to the payment platform, and carrier cases go to a claims process instead of stopping at a label.
- Identity and audit. Every resolution records which clerk approved it, from the same sign-in that guards the board. Policy changes go through change control like any other release.
- Many clerks work the queue at once, and the resolution log lives in a database rather than a file. Engine calls are made only for requests with a statement, so cost grows with the share of returns that need reading, not with total volume.
- Measurement on real returns. The engines are evaluated on the retailer’s own labeled returns. Auto-approval runs in shadow mode first, logging what it would have done next to the clerk’s decision, before any limit is raised.
Takeaways for your own processes
The pattern carries well beyond returns: claims, support tickets, invoice exceptions, access requests. Wherever a person’s words meet a set of rules, the same split applies.
- Find the one question. List every decision in the process and apply the intern test. Usually only one or two need a model.
- Put the rules in data. A policy file the business can read and edit beats rules buried in a prompt, and it costs nothing to run.
- Ask the model for an input, not a decision. A responsible party with a reason is something code and people can check. A finished outcome isn’t.
- Let code bound the risk. Caps, allowed outcomes and arithmetic checks hold no matter what the model says or what a customer writes to it.
- Keep a person on the hard cases. Automation should clear the easy, agreed, low-value requests, so the clerk only sees what needs judgment.
- Measure before you trust. A labeled set and a repeatable evaluation turn “it seems to work” into accuracy and cost per request.
Try it yourself
Returns Desk is open source under the MIT license at github.com/keyva/returns-desk.
- No cloud account: clone it and run the tests. They use canned engine answers and never call Azure.
- The full demo: you need an Azure subscription, uv and OpenTofu. The included OpenTofu sets up the Foundry resources with Microsoft Entra ID sign-in and API keys turned off. The README walks through running the board locally and deploying it to Azure Container Apps. The hosting steps haven’t been run end to end yet, so treat them as the intended order rather than a tested script.
The sample photos are AI-generated, and the prompts are in the repo. The design document includes a five-minute demo script that walks seven requests across all four paths, from a policy denial to an escalation.
If you have a process where people’s words meet a set of rules, Keyva can help you find the one question that needs a model and build the rest as ordinary code.
 | Eddie Dunn - DevOps Engineer Eddie brings more than 25 years of experience in the IT field, with a background spanning Linux and infrastructure engineering, platform engineering, and software development. He has extensive experience with Ansible Automation Platform, CI/CD pipeline architecture, and Infrastructure as Code. At Keyva, Eddie works on projects involving Ansible, Terraform, Kubernetes, and CI/CD. Eddie is passionate about DevOps, automation, solving client problems, and learning new technologies. |