At Tumeryk, as you can imagine, we have conversations with AI customers every day. However, it’s truly amazing how many of these conversations start with “What do you mean, I cannot trust my AI?” Despite having countless examples to turn to, most AI customers either do not believe it can happen to them or they’re confident that they have guardrails in place to mitigate any AI issues.
While the examples are increasingly everywhere – customers, reports, research – that ungoverned AI can at best provide incremental value while at worst can undermine multi-billion-dollar investments, the risks have so far not forestalled all unmanaged AI implementations. Word came out just two weeks ago of another prime example, this time from an AI startup, Anysphere. Most shockingly about this example is not the value of the lost equity, which was by some estimates in the multiple billions in valuation lost by Anysphere, but rather that an AI-focused organization would not invest to mitigate what they all knew was possible in the first place.
According to reports, Anysphere launched an AI-powered code generation product called Cursor. For a while everything was good and Cursor generated buzz among the targeted developer communities. However, a few weeks ago a Cursor user reported on Hacker News and Reddit that customers were being logged out unexpectedly when switching between devices. Upon contacting customer support, they received an emailed response from “Sam,” stating that the logouts were “expected behavior” under a new login policy.
However, there was no new policy, and the response came from an AI-powered bot. The explanation provided was inaccurate and fabricated by the AI. This lapse has led to a number of cancellations and highlights potential challenges for other startups using automation.
Just as meaningful to Anysphere, who was actively in the venture market looking to raise their next funding round, the timing couldn’t have been worse. Their pre-Cursor issue valuation was set at $10B. There has been no indication what their current valuation is but undoubtedly it’s less than that.
As we have reiterated so many times, all of this could have been mitigated with the proper guardrails and an AI Trust Score™ in place. Anysphere could have been alerted immediately to an improper response from their AI agent and have worked to override the system to address customer issues correctly to their satisfaction.
The bigger issue is that this won’t be the last time faulty or untrustworthy AI creates issues that impact companies financial or reputation. At this point we cannot even suggest that Anysphere is a canary in the coal mine. There are simply too many examples.
At Tumeryk we’re focused on issues just like this. And it may not even be hallucination. Issues arise from data leakage or unauthorized access to AI systems. It could be privacy or compliance issues or bias or discrimination. We’ve developed an AI Trust Score™ that tracks all of these issues and provides the guardrails and management tools to identify and alert you to these before they become customer-facing.
