Jacob Coxon and the Stupidity of AI

This is not going to be the usual news piece, but more a commentary on a very common story that went viral today.

Jacob Coxon, an AI researcher who worked both at OpenAI and Anthropic, announced his departure from the company behind Claude over a concern that sounds like sci-fi: AI is going to kill us all.

The X post

Coxon shared his thoughts and the reasons for his resignation on X.

"Do not underestimate the power of this technology," he continued on the popular social media platform.

For a normal person, reading these words sounds alarming and might even make us smile: it's hard to believe that a technology that isn't even able to follow a sequence of simple instructions without making you wonder why you're still paying for a subscription would be capable of doing all this.

But it's also true that maybe insiders know things we don't.

The different perception: Anthropic vs. OpenAI

The very first question that came to my mind was: if it's so dangerous, why continue building it?

And I discovered that it is actually a very common one, so common that even Coxon addressed it in his post. Maybe the reply is even more alarming.

"At OpenAI, many have not deeply internalized the civilization stakes. At Anthropic, the stakes are well-understood, but they are locked in a race to get there first."

Why does Anthropic want to make this move? We might think it's for prestige and money, but it looks like it's because they believe they would be the only ones who would behave responsibly. So, they want to be the first ones to get to such a level of sophistication to control the outcome.

Anthropic and its philosophy

Anthropic's positioning has, from the very beginning, been that of the responsible pioneers.

Its founder, Daniela Amodei, decided to create the company after leaving OpenAI due to concerns over the ChatGPT creator's stance on safety.

We all know the way the company gets close to people: parties dedicated to creators, attracting the brightest minds in research, and even half-conflicting with national AI interests.

Of course, we can't know where the line that delimits the truth from the mundane desire for primacy and profit is, but the most recent history involving the company poses a much harder question: where should the limits for private companies be—the line beyond which the state enters the game?

The U.S. vs. Anthropic

One of the most telling stories involving Anthropic in recent months was the block that sounded a lot like a conflict between a company strictly connected to the U.S. administration—which happens to also be one of the investors in Anthropic, Amazon—and Amodei's company.

Just a couple of days after the launch of Fable 5—whose underlying architecture is Mythos—the U.S. forced Anthropic to block public access. The restriction stemmed from national security concerns over the model's capabilities and the potential for it to be misused abroad.

It was Amazon that reportedly initiated the call, forcing Anthropic to publicly acknowledge a restriction that caught the company completely out of nowhere.

People went crazy, especially because many had upgraded to the highest-tier subscriptions just to try the model, only to be suddenly cut off—along with their subscription money. But that is not the main point.

The point is that the U.S. decided on the block after a call from a private company, which might have had a conflict of interest since Amazon is a player in the AI game too.

Anthropic had to review the model, and we all know how hard it was to find the same quality outputs after the review.

Fast forward, and another story hit the headlines. This time, however, it involved OpenAI.

A company acting as a state intelligence department

Recently, OpenAI announced it had found and blocked a social attack built by a Russian group that used public indexes to steal research and publish the work under the name of a think tank allegedly based in Israel.

The point was spreading authoritative work that reinforced the idea of Russian power against Western decline.

OpenAI did it all by itself, from the mission to the announcement, opening once again the question: where's the line between the competencies of a government and those of a private company, despite how powerful it can be?

What's the point of all this

The Coxon story might or might not be a marketing move.

We got unfortunately too used to this kind of dramatic announcement: the latest launch is always the best one, the latest chip is always the most sustainable and efficient, the latest model is always the one that brings us closer to the scary AGI.

But of course, we give the benefit of the doubt. And we also respect the concerns of a person who dedicates his life to research and is deeply involved in the AI giants' processes and methods. And it's well known that he resigned right before the vesting of his equity.

The only doubt that arises is because we need to sign NDAs even to write a blog post. I can't imagine what kind of documentation people working at these companies have to sign before being able to say even a single word about those companies.

It's true that even Dario Amodei speaks about similar fears publicly, as well as Sam Altman, but it is also true that people should have concrete proof of these statements.

It is also curious that all this is happening right before a possible Anthropic IPO in October, which, as I said, has always positioned itself as the most responsible among tech giants.

Moreover, another detail that caught my attention is that OpenAI's stance on the topic was dismissed in a single line, while for Anthropic all details were provided. Coxon used the specific words "act responsibly".

True or not, once again the question arises. Where's the line? The actions of these companies leave us with the doubt that we're leaving very large parts of our lives and security in the hands of a few, and there's currently no kind of protection, no kind of common ground, neither among companies nor among states, to find a secure way to manage AI tech.

In this sense, a technology that moves trillions and enters every aspect of our lives is governed by a special kind of stupidity. After all, if the endgame of this breakneck pace is a technology that destroys the world and kills us all, these tech giants will achieve the ultimate corporate failure—eroding their own power simply by running out of humans to sell it to.

Editor's note

Every piece published on The Bright Minded goes through careful verification, but mistakes can happen. If you spot an error, have additional information, or want to flag anything, write to rosalia@thebrightminded.com.