Momos · May 2022 – Sep 2025 · Abu Dhabi
Turning thousands of reviews into incidents a brand can act on
A multi-agent system that reads every review and separates the incidents a brand must address from the compliments it should celebrate - deduplicated, evidence-linked, and aggregated down to the location, the dish and the staff member. The LLM migration behind it: 30% better accuracy at 60% lower cost.
Role
Lead DS, built solo
Users
Brand & ops leaders
Outcome
−90% manual hours
The stakes
A negative incident in a public review is not a data point - it is an open wound. The brand has to address it, win the customer back, and ideally see the review itself turn positive before it shapes what the next hundred people think. A compliment is the opposite kind of signal: something working that deserves recognition. The two demand different actions, which is why this system starts by telling them apart instead of averaging them into a sentiment score.
The bar for the output stayed strict: every incident had to name a specific, countable thing, carry the exact phrases from the reviews that evidence it, and roll up to where the fix lives - the brand, the location, the dish, or the person. If an area manager could not click through to the evidence in five seconds, it was not an insight. It was a vibe.
The system - a pipeline of specialised agents
Reviews - every source, every language
Agent 01 · Screener - small, fast, non-reasoning
Reads every review and answers two cheap questions: is the sentiment negative, and is an actual issue mentioned? Splits the stream into incident candidates and compliments.
Agent 02 · Reasoner - match before create
Checks each batch against the incidents already on record, via retrieval. A match joins the existing incident; a genuinely new problem becomes a new incident under defined rules - never on a whim.
Agent 03 · Evaluator - dedupe and enrich
Catches the same incident created twice across batches, merges overlaps with the record, then attaches the evidence: the exact phrases that tie each review to the incident, plus any dish or staff member named.
The aggregated view - where the fix lives
The same pipeline runs a second time with positive intent: compliments get identified, deduplicated and aggregated the same way - what to celebrate, which locations and people to recognise.
Three decisions, and what I turned down
01
Screen with a small model, reason with a big one
Most reviews are not incidents, and a reasoning model reading all of them is money spent on nothing. A small non-reasoning model does the cheap triage - negative sentiment, issue present - and only the flagged minority reaches the reasoning agent, in batches. The expensive intelligence is spent exactly where the stakes are.
Rejected: one large model reading everything. Simpler to build, and the unit economics stop working the day a client with real volume signs.
02
Match before create - and let an evaluator audit the matcher
An incident tracked twice is worse than not tracked: it splits the evidence and halves the apparent severity. Every candidate is retrieved against the incidents already on record and joins one if it matches; only rule-qualified novelty creates a new incident. Because batches run independently, a third agent audits the output - catching the same incident born twice in parallel and merging overlaps before anyone sees them.
Rejected: free-form theme generation per run. Beautiful demos, uncomparable outputs - two runs on the same data produced two different worlds, and no trend line survives that.
03
Aggregate to where the fix lives, with the receipts attached
An operator cannot act on "sentiment is down." They can act on a brand-level issue, a location's problem, a staff member's pattern, or a dish that needs fixing - so that is exactly how incidents aggregate, each carrying the verbatim phrases that evidence it. Pin an incident and the view tracks whether it is actually improving. The receipts are the single feature that turned scepticism into adoption in client meetings.
Rejected: a narrative summary paragraph. It reads well, it is impossible to verify, and nobody can act on a paragraph.
What came of it
−90%
of manual review-analysis time removed.
+30%
extraction accuracy after the LLM migration, on a held-out labelled set.
−60%
cost per document, which made per-brand daily runs affordable.
Underneath all of it sat a labelled evaluation set and a harness scoring extraction accuracy and cost per document side by side - built before the LLM stack migration, which is what made the migration a measurement instead of a leap of faith, and what lets 30% accuracy up and 60% cost down be stated as facts. The harness outlived the decision it was built for.