Momos · May 2022 – Sep 2025 · Abu Dhabi
Automating customer replies for global brands
Retrieval-augmented drafting of on-brand replies to public reviews across many brand clients, with confidence-based routing to humans. Routine cases fully automated; response time went from days to minutes.
Role
Lead, team of 3–4
Risk
Public brand voice
Outcome
Days → minutes
The stakes
Every reply this system writes is published under a brand's name, in public, permanently. That single fact determined the whole architecture. The failure mode is not a bad F1 score. It is a screenshot on social media of a restaurant chain cheerfully thanking someone for a review about food poisoning.
Meanwhile the manual process did not scale. Ops teams replied by hand, so reviews sat for days and the ones that got answered were the ones that happened to be on top. Each brand had its own voice, its own escalation rules, and its own idea of what “on-brand” meant.
So the goal was never “automate replies.” It was: automate the boring majority with high confidence, and make the system reliably recognise the minority it should not touch.
The system, in three stages
1 · Classify & retrieve
intent, sentiment and severity, then brand voice, policy and past approved replies
2 · Draft
grounded generation against the retrieved passages
3 · Guard & route
policy checks and confidence scoring decide: auto-publish, or human queue
high confidence → auto-publish
the routine majority
low confidence → human queue
pre-drafted; every edit captured as training signal
Three decisions, and what I turned down
01
Compute confidence outside the model
I never trusted a model's self-reported certainty. Routing combined signals the model does not control: how well the draft was supported by retrieved passages, review severity and intent class, whether any policy trigger fired, and how similar the case was to previously approved replies. Any one of them could veto automation on its own.
Rejected: asking the LLM to rate its own confidence 1–10. It is confidently wrong in exactly the cases you need it to hesitate.
02
Ship with the threshold set far too cautiously
We launched routing almost everything to humans and moved the threshold down only as the approval-rate data justified it, brand by brand. Slower to value, but the first thing every client experienced was a system that was right, and it gave me a clean labelled stream of human edits to calibrate against.
Rejected: launching at an aggressive automation rate and dialling back after incidents. With public brand text you do not get to un-ship trust.
03
Keep brand voice in the retrieval store
Each brand's voice, policies and past approved replies live in a retrievable store. Onboarding a new brand is a data task. A policy change takes effect on the next reply. And every draft traces back to the passages that justified it, which is what made legal and brand teams comfortable.
Rejected: a fine-tuned model per brand. Better stylistic mimicry, but per-client training cost, no traceability, and a retraining cycle every time a brand changed its mind.
What came of it
Days → minutes
Response latency for brand clients. Routine cases fully automated; ops teams spend their time on the cases that need judgement.
Multi-brand
Shipped to production across multiple brand clients on shared infrastructure. Onboarding a new brand needs no model work.
The human queue turned out to be the most valuable part of the product. Every edit an ops person made was a labelled correction, and that stream is what let us raise the automation rate safely instead of hopefully.