500 AI-Written Attacks, 90% Stopped by a Default Setting

Five hundred attacks, an AI tool writing custom payloads for every language – and 90% failed.

Not because of a sophisticated defence system. Not because of a SOC working in real time. Because the attacker did not understand how GitHub works.

A few days ago research was published on a supply chain attack that tried to penetrate 500 open-source repositories on GitHub. The approach was technically impressive: an AI tool that analysed each repository, worked out what language it was written in, and wrote a payload tailored to each one. A scale that would not have been possible without AI.

Then came the less impressive part.

It waited for Actions that never ran

The attacker sent malicious pull requests and waited for GitHub Actions to run them automatically. The problem? The platform does not run Actions automatically for new contributors. There is a very basic default setting that most organisations are not even aware of.

Quite literally: the compute behind hundreds of individually tailored attacks was stopped by a default setting.

Organisations like Sentry and OpenSearch stopped the attack without flinching. Not because they have a hundred-person security team. Because they did basic hygiene on GitHub.

Twenty years of watching this

In twenty years of pen testing I have seen this over and over. Sophisticated attacks are stopped by the simplest fences. And simple attacks get through because nobody bothered to check the basics.

What the AI gave the attacker was scale. What it did not give them was understanding. And the lack of understanding killed the attack.

The people saying AI is going to replace security professionals are wrong in exactly this spot. The tool can generate attacks at a rate no human can match. It cannot understand the context. It does not know GitHub’s permission model. It does not know what secrets are sitting in your Actions. It does not know which third-party action you added to the pipeline a year ago and forgot about.

That understanding is still human work.

Supply chain attacks on open source are not theory. They are happening now, at scale, with tools available to everyone. Most of the organisations I meet have not looked at their Actions in the last year.

I first shared a version of this as a LinkedIn post on 2026-04-26. It is republished here, lightly edited, so it is easier to find and reference. — Erez Metula