New Zealand-based startup ThroughLine, which already works with OpenAI, Anthropic, and Google, is exploring ways to redirect at-risk users toward human support and intervention services, News.Az reports, citing Reuters.
ThroughLine currently provides crisis response systems for issues such as self-harm, domestic violence, and eating disorders. When AI detects signs of distress, users can be connected to a global network of helplines and real-world support.
Now, the company is looking to expand that model to address violent extremism — a growing concern as more people turn to AI chatbots for sensitive conversations.
The initiative is being developed in collaboration with Christchurch Call, created after the 2019 Christchurch attack in New Zealand.
The goal is to build a hybrid system that combines AI chatbots trained to respond to early signs of radicalization with referrals to human-led services.
The move comes amid increasing scrutiny of AI platforms over safety concerns, including allegations that chatbots may fail to prevent — or even indirectly enable — harmful behavior.
Experts say the challenge goes beyond simply removing content.
Instead, the focus is shifting toward understanding user behavior and offering meaningful intervention before situations escalate.
While the technology is already being tested, there is no confirmed launch date.
Key questions remain about how follow-up systems will work, including whether authorities should be alerted in high-risk cases — and how to avoid pushing users toward less regulated platforms.
Researchers also warn that abruptly cutting off conversations could leave vulnerable individuals without support, potentially increasing risk rather than reducing it.
As AI tools become more deeply integrated into everyday life, companies are under pressure to ensure they not only provide information — but also act responsibly when users show signs of distress or dangerous behavior.
This new approach suggests a shift toward combining technology with human intervention, aiming to make AI platforms safer without losing the trust of users.
