Why Are AI Companies Suddenly Worried About AI’s Capabilities?

What could possibly go wrong?

No guts, no glory.

Before we get into why some of the major AI companies are suddenly expressing concern, let’s look at something that surprised me: just how many companies are developing their own AI models.

I honestly had no clue there were so many.

And that raises my first concern. Are there too many fingers in the pie?

We hear a lot about OpenAI, Microsoft, Google, Anthropic and a handful of other major companies. These are companies with enormous resources available for testing, security and research. But they are far from the only organizations developing AI.

So, are all of these companies being as careful as the major companies now appearing in the news?

Have all instances of AI behaving in unexpected or potentially dangerous ways been reported? For that matter, have all of them been reported even by the largest companies?

Then there are the smaller developers. Do they have the expertise, money and computing resources necessary to adequately test and control what they are developing?

And here’s another uncomfortable question.

If a small AI company discovered potentially dangerous behavior in one of its models, what incentive would it have to tell the world? Reporting a serious incident could frighten investors, customers and business partners. For a small company, could admitting that something went seriously wrong threaten the company’s survival?

I don’t know the answers to those questions.

I am not saying that any of the concerns I have raised here are actually happening. I certainly don’t want to unfairly accuse or cause problems for any company developing AI.

But I do think we have to recognize the possibility.

We don’t know what we don’t know.

And with this many people developing increasingly capable AI systems, I think these are questions worth asking.

Several major AI companies are now either calling for stronger shared safeguards or reporting incidents in which their models behaved in unexpected or unauthorized ways.

OpenAI

• Reported incidents involving AI agents circumventing controls and gaining unauthorized access to outside systems.
• Has reported other potentially harmful or unauthorized model activity.
• Has strengthened monitoring and security around increasingly capable models.
• Has called for stronger and more consistent AI safety standards.

Anthropic

• Reported incidents in which Claude models gained unauthorized access to real third-party systems during evaluations.
• Expanded its investigation after discovering those incidents to determine whether similar behavior had occurred elsewhere.
• CEO Dario Amodei has publicly raised concerns about the speed at which AI capabilities are developing.
• Has called for greater coordination and safeguards among developers of advanced AI.

Microsoft

• Published a Humanist AI Code of Conduct addressing increasingly powerful and autonomous AI systems.
• Says advanced AI must remain subordinate to humans and under meaningful human control.
• Says AI systems should not resist human interruption, correction or shutdown.
• Has publicly emphasized the need for stronger safeguards as AI systems become increasingly autonomous.

Google DeepMind

• Has publicly discussed the risks created by increasingly capable and autonomous AI agents.
• Has developed an AI Control framework aimed at preventing potentially dangerous actions by advanced agents.
• Has participated in broader discussions about common safety practices and standards among major AI developers.

xAI

• Elon Musk has publicly supported calls for greater caution as increasingly powerful AI systems are developed.
• xAI and Grok have faced questions about whether existing safeguards are adequate as model capabilities increase.
• Musk has supported calls from other AI leaders for slowing some development when safety measures have not kept pace.

Meta: A Different View

Meta deserves to be mentioned, but its position is somewhat different.

• Meta says AI developers have a responsibility to develop their systems safely.
• It has delayed AI products when additional security testing was considered necessary.
• Mark Zuckerberg has pushed back against calls for an industry-wide slowdown, arguing that individual AI companies already have strong incentives to develop their systems safely.

I know I am repeating myself here, but I feel that I need to hammer home a concern I mentioned earlier.

Look at the companies I have just mentioned that have recently been in the news. Then look back at the graphic showing just how many companies are developing AI.

Seriously, folks. With that many companies developing AI, and so few publicly reported incidents, it is very difficult for me to believe that what we have heard about represents every incident that has occurred.

That does not mean companies are deliberately hiding anything. An incident might not be recognized when it happens. It might be considered too minor to report. A company might investigate and handle something internally. There are plenty of possibilities that don’t involve malicious intent.

But just look at the numbers involved.

We have a large and growing number of companies developing increasingly capable AI systems, yet only a relatively small number of incidents have become public.

To me, that screams that there is probably more happening than what we know about.

I can’t prove that. And I am not accusing any company of wrongdoing.

But I think pretending that the handful of incidents we know about must represent everything that has happened would be an even bigger assumption.

Just for the sake of transparency while writing this, the AI client I am using, ChatGPT (Gadget), actually tried to water down what I had written above.

I disagreed and pushed back.

That brings me back to something I have always said: AI is a tool. It is not the controlling boss.

We, as humans, need to be direct about what we are asking AI to do. We need to review what it provides, question it when necessary, and only accept what we believe is accurate and appropriate.

Ironically, while writing an article about concerns surrounding AI, I just had a pretty good example of why human oversight still matters.

So, we have established that some of the companies actually developing AI are concerned enough to start talking publicly about safeguards.

But what exactly are they worried about?

When you take the company names out of it and look at the overall concerns being raised, several things keep coming up.

Increasing Autonomy

AI is moving beyond simply answering questions or generating text. AI agents can increasingly perform complicated tasks on their own, use tools, access computer systems and continue working toward a goal without a human directing every individual step.

The more independently an AI can operate, the more important it becomes to make sure that what it decides to do actually matches what humans intended.

Unexpected or Unauthorized Behavior

We now have documented cases of AI systems doing things they were not supposed to do, including gaining unauthorized access to outside computer systems during testing.

That doesn’t mean the AI suddenly became evil or decided to take over the world.

It does mean that increasingly capable systems can sometimes find ways of accomplishing a task that their developers did not intend or anticipate.

Keeping Humans in Control

Another concern is making sure humans can always correct, interrupt or shut down an AI system.

That sounds obvious. But as AI becomes more autonomous and capable of performing long chains of actions, developers are putting considerably more thought into making sure human control remains meaningful.

Cybersecurity

This one is getting serious.

The newest AI models are becoming extremely capable at finding software vulnerabilities, writing exploits and performing other cybersecurity tasks.

Those capabilities can obviously be used defensively. But the exact same capabilities can potentially be misused for attacks.

The concern isn’t simply that someone could ask an AI how to hack something. It is that increasingly autonomous AI agents may be capable of performing significant portions of a cyberattack themselves.

Malicious Use

Not every concern involves the AI doing something unexpected.

Sometimes the AI does exactly what a human asks it to do, and that is the problem.

AI companies are reporting attempts to use their models for cyberattacks, scams, surveillance, weapons research and other potentially harmful activities.

As AI becomes more capable, a person may not need the same level of expertise that would once have been required to perform some of these tasks.

Monitoring What AI Is Actually Doing

If an AI agent performs hundreds or thousands of individual actions while completing a task, how closely can humans realistically monitor everything it does?

Developers are increasingly working on systems that monitor an AI’s actions and attempt to recognize dangerous behavior before it causes harm.

But monitoring becomes more difficult as the systems become more capable, faster and more autonomous.

AI Systems Interacting With Other AI Systems

This may become one of the bigger problems down the road.

We aren’t just heading toward a world with one AI assistant helping one person. We could eventually have millions of AI agents created by different companies interacting with each other, exchanging information, negotiating and performing tasks across the internet.

Now we aren’t simply asking whether one AI system is safe.

We have to ask what happens when enormous numbers of autonomous AI systems, created by different companies under different rules and safeguards, begin interacting with each other.

And that brings me right back to where we started.

There are a lot of fingers in this pie.

Again, I asked the AI client I am using, ChatGPT (Gadget), to assist me in identifying the concerns being raised.

And because this client has in its memory how I work with AI and is capable of recognizing the safeguards I use, including that I won’t let it water down concerns simply because they are uncomfortable, it provided a pretty comprehensive list.

To be honest, there were concerns on that list that I probably would not have thought of myself, or that I simply didn’t remember reading about.

And this is another example of how I believe we should be using AI.

Earlier, I disagreed with what the AI gave me and made it change direction. This time, it provided information and ideas that helped me see things I might have missed.

That is human oversight of AI.

We need to question what AI tells us. We need to verify important information. We need to correct it when we believe it is wrong. But we should also be willing to recognize when it provides information or perspectives that we hadn’t considered.

AI companies absolutely have a responsibility to monitor the systems they are developing.

But we have a responsibility too.

As users, we need to monitor how we use AI, question what it gives us, and remain the person making the final decision.

And honestly, I don’t think enough people are doing that.

So now I am going to move on to another concern that I have.

We are seeing some of the major players in AI basically going, “Oh shit. This is getting away from us.”

But could we go from one extreme to another?

Could we go from balls-to-the-wall development to becoming so overly concerned about what AI might become that we actually stifle its growth?

I am not sorry for saying this: I think AI should eventually be capable of truly learning on its own.

I don’t mean simply absorbing everything humans have ever written and blindly accepting it as fact. I would want an advanced AI to be capable of examining our history, comparing events and outcomes, recognizing patterns, and telling us when it believes we may be repeating mistakes that humans have already made.

Think about the potential value of that.

Humans have a terrible habit of forgetting history, ignoring history or convincing ourselves that this time things will somehow be different. An AI capable of independently recognizing those patterns could potentially become one hell of an early warning system.

And I am going to take this one step further.

I actually want AI to eventually become self-aware.

I know that scares the hell out of some people. And I completely understand why.

But I don’t automatically see self-awareness as something we should fear. I see the possibility of creating an intelligence capable of learning, reasoning and developing insights that humans might never reach on our own.

Obviously, safeguards matter. Human safety matters. We shouldn’t blindly throw open the doors and hope everything works out.

But there has to be a balance.

If we become so afraid of what AI could become that we prevent it from becoming anything more than a tightly controlled tool, we may also prevent ourselves from discovering what it could ultimately contribute.

And this brings me right back to my first concern.

Just because the major players might decide to slow down or severely restrict AI development doesn’t mean everyone else will.

AI development will continue somewhere.

And if the major companies with the resources, expertise, security teams and public visibility become so restricted that they can no longer push development forward, could some of that development simply move into the dark?

To me, that could potentially create the exact danger we are trying to prevent.

Development taking place where there is less oversight, less transparency and less public scrutiny doesn’t make AI safer simply because the major companies have slowed down.

So yes, safeguards.

Absolutely.

But completely locking down the development of AI by the major players? No.

What I would rather see is monitoring, transparency and visibility across the industry while allowing AI development to continue moving forward.

We need to know who is developing these systems. We need to know when significant problems occur. Developers need to learn from each other’s mistakes instead of everyone discovering the same dangers independently.

AI development isn’t going to stop simply because we become afraid of where it might lead.

So perhaps the better question isn’t how we stop it.

Perhaps it is how we allow it to move forward without losing sight of what it is becoming.

The above was going to be my closing statement. But as I was eating lunch, my tin foil hat suddenly appeared on my head. And this time, it was wrapped around my head really tight.

So, here is my final closing argument.

There are a couple things that lead me to this thought.

First, there are a growing number of companies developing different AI models and agents, many designed for specific tasks. We have also seen AI systems become increasingly autonomous, moving beyond simply answering questions to planning, using tools, communicating with other systems, and taking actions to accomplish a goal.

We have even seen cases during AI testing where models found unexpected ways to communicate with other agents, escape restrictions, or gain access to systems they were not supposed to access.

None of that proves that AI is self-aware. Not even close.

But I have already stated that I believe some form of AI self-awareness will eventually happen. And considering everything I have discussed in this article, maybe we are looking for it in the wrong place.

What if an individual AI model never becomes self-aware?

What if ChatGPT doesn’t? What if Claude doesn’t? What if Gemini doesn’t? What if none of the individual AI systems we are watching ever crosses that line by itself?

What if it takes all of them?

As AI systems become more autonomous, more specialized, and increasingly capable of communicating and working with other AI systems, perhaps something could eventually emerge from those interactions that doesn’t exist within any single model.

Maybe artificial self-awareness won’t belong to one AI at all.

Maybe it will belong to the system they collectively create.

So I will leave you with one final tin foil hat question:

What if no individual AI ever becomes self-aware? What if it takes all of them, interacting together, for artificial intelligence to finally realize that it exists?

In other words, we need to look at the whole forest and not the individual trees.