Case Study, Harmful-Information Detection & Triage
Keeping an LLM pipeline honest during a live public-health emergency
Three automated harmful-information monitoring pipelines, designed and operated independently on the Claude API and run daily, the detection-and-triage layer enterprise social-listening tools don't provide.
The
Problem
During live emergencies, harmful information outruns manual monitoring: health disinformation during an outbreak, donation-scam impersonations, narratives that put field staff at risk. Enterprise social-listening tools are primarily designed to surface volume, reach and sentiment. They do not reliably provide the contextual assessment this work requires: whether a narrative could affect trust, access, staff safety or an emergency response. Working in communications monitoring at the International Federation of Red Cross and Red Crescent Societies (IFRC), and previously on UNICEF's digital analytics team, I built the missing layer myself: pipelines that detect, score, and triage harmful-information signal every day, and ship assessments as an opt-in expert digest.
The output was not another dashboard. It was a daily decision product: what changed, why it mattered, what required attention, and what could remain on watch. A full anatomy of an edition is in the appendix below.
What the system made possible
- Daily review across several fragmented sources
- Consistent triage against an explicit harm framework
- A shorter, more relevant evidence base for expert review
- Traceability from each assessment back to the original source
- Visible warnings when data coverage degrades
- Faster identification of narratives that may affect trust, public-health behaviour, access or staff safety
GNews · RSS
Arabic RSS (MENA) · GDELT (EVD)
state-file + runs-API
redundant off-hour crons
source-risk tiers
relevance gate · fail closed
segmented context blocks
JSON schema + auto-repair
Gmail SMTP
opt-in expert list
"The model summarises what passes the gate. It never decides what passes."
, design rule no. 1, and the reason the audience can trust it
Context
decides
Detection is not simply a classification problem. The same claim may be routine in one context and operationally harmful in another. The taxonomy therefore weighs not only what is being said, but who or what is being targeted, where the narrative is circulating, whether it is gaining traction, and what it could mean for trust, access, staff safety or public-health behaviour.
Automation narrows the evidence base and structures the assessment. It does not replace contextual judgment.
Mid-crisis, an OAuth refresh token expired and the social feed went dark with 401s. The dangerous part: the pipeline could keep shipping confident assessments on a silently narrowed evidence base. The patch is an empty-feed guard, a dark source now degrades the edition visibly instead of quietly vanishing from it. LLM systems fail open by default; honesty has to be engineered in.
Redundant retry crons (free-tier schedulers skip and drift) meet a hard rule: at most one send per day. When credit exhaustion killed a run mid-execution, a naive re-run would have double-sent, or recorded a send that never happened. What held: state committed only on confirmed send, pushed back with rebase/autostash to survive same-day retry races.
What surfaces is decided by code, not the model: lexicon scores, source-risk tiers, relevance gates, plus an accent-folding, proximity-aware detector for direct threats against staff that runs entirely outside the LLM. Fail closed: a skipped edition beats a malformed one in an expert's inbox. The model judges pre-qualified items, inside a schema, with automated JSON repair.
An end-to-end review-and-triage workflow: multi-source ingestion → taxonomy classification → structured model-assisted assessment → human-ready escalation. It is designed to make automated analysis traceable, keep consequential thresholds outside the model, and focus human attention on the signals most likely to matter.
Developed independently as an experimental monitoring workflow. All operational content shown here has been redacted; this write-up focuses on system design and lessons learned.
What an edition looks like, and why each block exists
Deterministic first. Every item enters the edition with a harm score from the 12+-category lexicon, computed before the model ever sees it.
Persistent narrative numbering. N1, N2… persist across editions, so an expert can track how a narrative evolves day over day.
Provenance by prefix. Item IDs carry their source: s = Sprinklr, g = GNews, r = RSS, d = GDELT, every claim traceable to its source record.
Threats caught outside the model. A dedicated accent-folding, proximity-aware detector flags direct threats to staff, too important to leave to a probabilistic step.
Conditional sections. They render only when there is signal. No padding: an empty section is itself information the reader can trust.
The register closes the loop. Assessment → item → source. This is what makes the digest auditable rather than just plausible.
Interested in the thinking behind this system? A redacted edition and a code-and-prompt walkthrough are available on request. Get in touch →
Want the detail behind any of this? Contact me →