Case study · Harmful-information detection & triage
Keeping an LLM pipeline honest during a live public-health emergency.
Three automated detection-and-triage pipelines I designed and run daily on the Claude API: the layer enterprise social-listening tools don’t provide.
Health disinformation during an outbreak, donation-scam impersonations, narratives that put field staff at risk. Enterprise social-listening tools surface volume, reach and sentiment, not whether a narrative could affect trust, access or staff safety. At the IFRC, I built the missing layer: pipelines that detect, score and triage harmful-information signal every day, and ship it as an opt-in expert digest.
Deterministic first. Every item enters with a harm score from the 12+-category lexicon, computed before the model ever sees it.
Persistent numbering. N1, N2… persist across editions, so an expert can track how a narrative evolves day over day.
Provenance by prefix. IDs carry their source: s = Sprinklr, g = GNews, r = RSS, d = GDELT. Every claim traceable to its record.
Threats caught outside the model. A proximity-aware detector flags direct threats to staff, too important to leave to a probabilistic step.
Conditional sections. They render only when there is signal. An empty section is itself information the reader can trust.
The register closes the loop. Assessment → item → source. This is what makes the digest auditable rather than just plausible.
“The model summarises what passes the gate. It never decides what passes.”
Design rule no. 1, and the reason the audience can trust it.
Mid-crisis, an OAuth token expired and the social feed went dark with 401s. The pipeline could keep shipping confident assessments on a silently narrowed base. The fix: an empty-feed guard, so a dark source now degrades the edition visibly. LLM systems fail open by default; honesty has to be engineered in.
Redundant retry crons meet a hard rule: at most one send per day. When credit exhaustion killed a run mid-execution, a naive re-run would double-send. What held: state committed only on confirmed send, pushed with rebase/autostash to survive same-day races.
What surfaces is decided by code, not the model: lexicon scores, source-risk tiers, relevance gates, plus a proximity-aware detector for threats against staff. Fail closed: a skipped edition beats a malformed one in an expert’s inbox.
A complete review-and-triage workflow, from raw signal to a decision a person can act on. Three principles hold it together: the decisions that matter stay in code, not the model; every assessment can be traced back to its source; and human attention is spent on the few signals most likely to matter.
Developed independently as an experimental monitoring workflow; all operational content here is redacted. This write-up focuses on system design and lessons learned, not a service.
Interested in the thinking behind this system? A code-and-prompt walkthrough is available on request.
Get in touch →