Meta Crashed My Server to Train Their AI 110 People Said It Happened to Them Too

The Post That Stopped 110 Engineers in Their Tracks

On 18 May 2026, a developer named Lucie posted this on Mastodon: "our gitlab was brought down by a DoS attack by Meta today, with 299991 requests from meta-externalagent/1.1."

Within hours the post had 110 boosts and 154 favourites. Not because it was surprising. Because it was familiar.

The replies came in from across the Fediverse. A developer running Forgejo on a Raspberry Pi described Meta's crawler looping through every file in every commit of their blog history with 70 concurrent connections, still going a day later when they finally got access to the server to block it. A self-hosted Phorge instance owner described the same pattern. Someone who manages hosting for clients said it had taken down customer websites on at least two separate occasions. A developer in Germany described blocking Meta's crawlers and watching their traffic costs drop immediately.

One reply put it plainly: "This is in fact a DoS attack by Meta."

That is Lucie's characterisation. That is the community's characterisation. The logs support it. The pattern is documented. What is missing from almost every conversation about this incident is the specific, actionable, fifteen-minute fix that makes the discussion academic rather than urgent.

That is what this piece is about.

What meta-externalagent/1.1 Actually Is

Meta documents their web crawlers in their developer documentation. The user agent string meta-externalagent/1.1 is listed there, along with a description of what it does. It is Meta's AI training crawler, distinct from their standard search and sharing crawlers. Its purpose is to collect web content for training large language models including Llama.

Unlike search engine crawlers, which are designed to be good citizens of the web, respecting robots.txt, honouring crawl delays, backing off when servers show signs of load, Meta's AI training crawler operates under different constraints. Multiple developers in the Mastodon thread and in other public discussions have noted that it does not honour standard backoff signals, rotates IP addresses making IP-based blocking difficult, and sends requests at volumes that bear no relationship to the size or importance of the target server.

A server running a private GitLab instance for a small team receives the same volume of requests as a major commercial platform because the crawler is not calibrated to the target. It is calibrated to the collection objective. The collection objective is as much content as possible, as fast as possible, for training data.

The server that Lucie runs is the production infrastructure for MNT Research, an open hardware laptop company with 13,000 Mastodon followers and a real product shipping to real customers. It is not a test environment. It is not idle capacity. When it went down, real work stopped.

Why the Standard Defences Do Not Work

The instinct when a crawler causes problems is to add it to robots.txt. robots.txt is an instruction that well-behaved crawlers follow voluntarily. There is no technical enforcement mechanism. A crawler that is designed to collect training data at maximum speed has no technical obligation to read robots.txt or to honour what it says. Meta's crawler documentation states that their crawlers do respect robots.txt. The community reports tell a more complicated story.

The second instinct is to block by IP address. Meta's crawler rotates across a large pool of IP addresses. Blocking one or two ranges is not a complete solution because the next request arrives from a different address in the same pool. You can maintain a blocklist of known Meta crawler IP ranges, the 57.141.20.0/24 block that Lucie's logs showed, but this requires ongoing maintenance and is always reactive rather than preventive.

The third instinct is to wait for the platform to fix it. That conversation has been happening in developer communities for years. The crawling behaviour has not changed in response to community reports.

The solution that actually works is architectural. It does not depend on the crawler's cooperation. It does not depend on IP blocklists that require constant updating. It operates at the infrastructure layer, before the request ever reaches the application server, and it works regardless of what IP address the request arrives from.

The Fix

A reverse proxy sitting between the public internet and the application server reads every incoming request before passing it through. When a request arrives carrying the meta-externalagent user agent header, the proxy applies a rate limit. Any requests above that rate limit are dropped immediately at the edge, returning a 429 status code, before they consume any resource on the application server behind it.

# nginx.conf: Stop Meta crawler flooding your server
# Add to your http block:

limit_req_zone $http_user_agent zone=meta_crawler:10m rate=5r/s;

# Add to your server block:

server {
  location / {
    if ($http_user_agent ~* "meta-externalagent") {
      limit_req zone=meta_crawler burst=5 nodelay;
    }
  }
}

# Result: 5 requests per second pass through
# Everything above that returns 429 instantly
# Your server never sees the flood

At five requests per second, a legitimate crawl of a small server's public content completes in a reasonable time. At 299,991 requests in a concentrated window, the proxy drops 299,986 of them before they touch the application. The server processes a normal load. The team has a normal morning.

For those running Cloudflare in front of their infrastructure, the free tier includes bot management rules that can be configured to rate-limit by user agent string without writing any nginx configuration at all. The protection available at zero cost is sufficient to prevent what happened to Lucie's server.

The fix existed before the incident. It will exist after the next one. The gap between the fix and the servers that need it is awareness, not complexity.

Why This Matters Beyond the Technical Fix

The Mastodon thread that Lucie's post generated is a useful document of who is affected by this pattern. It is not only developers running personal projects. It is small businesses running self-hosted infrastructure because they made a deliberate decision to own their stack rather than rent it from a platform. The MNT Research GitLab instance is production infrastructure for a real commercial operation. The customer websites that went down were serving real customers.

The specific cost when this happens to a business running paid advertising is different from the cost to a personal project. A company spending money on Google Ads or Meta Ads to bring visitors to their website has a continuous financial obligation running regardless of whether the website is accessible. When the server goes down under a crawler flood during business hours, every paid click that arrives during the outage sees an error page. The ad spend continues. The return goes to zero. The Technical Tax in that scenario is not the slow drip of a six-second load time. It is the full cost of every click that landed while the server was under the wave.

The businesses most exposed to this are the ones running application servers directly exposed to the public internet without a proxy layer in front of them, because nobody told them they needed one and the default configuration of most hosting environments does not include one. The managed host keeps the server running under normal conditions. It does not protect against an ungoverned crawler sending three hundred thousand requests in a session.

The Auditor's Take

The community response to Lucie's post was immediate and consistent. People wanted Meta held accountable. They wanted a lawsuit. They wanted consequences. Those conversations are legitimate and the frustration behind them is entirely justified.

What a performance engineer does while that conversation is happening is deploy the proxy configuration, set the rate limit, and make the conversation academic. The crawler will keep running. The IPs will keep rotating. The training data collection objective will not change because of a Mastodon thread, however widely it is boosted.

The reverse proxy does not require Meta's cooperation. It does not wait for a regulatory response. It does not depend on a lawsuit outcome. It reads the user agent header, applies the rule, and drops the flood in milliseconds. The server stays online. The team keeps working. The business keeps converting the paid traffic it is spending money to attract.

110 people boosted the post. The fix takes fifteen minutes. Both of those things are true simultaneously and neither one cancels the other out.

The original incident was reported publicly by Lucie, CEO of MNT Research, on Mastodon on 18 May 2026. All technical details are drawn from the public thread, Meta's published developer documentation, and standard nginx configuration practice. The diagnosis is always free.