Residential Proxies: The New Scraping Armada

AI scraping has escalated. In early 2025, LWN reported on the scourge. Over a year later, the problem is worse. Sites are hammered by shadowy actors using residential proxy networks — millions of unique IP addresses, each hitting a site only a few times, then gone. They fake user-agent fields, skip images and CSS, making detection nearly useless.

These proxies come from compromised devices: malware-infected PCs, poorly secured media streaming boxes, and even "free" VPN services like Bright Data that route traffic through users' devices without their informed consent. Google took down IPIDEA in early 2025, correlating with a significant traffic drop at LWN. But peace was short-lived. In July 2025, Google and the FBI dismantled NetNut, another residential proxy network. Yet attacks resumed.

Who's Behind the Scrapers?

Three types of operators run these networks:

  1. Criminal gangs using malware to build botnets.
  2. Shady companies like Bright Data, offering "ethically sourced" IPs via SDKs or VPNs, often pretending GDPR compliance.
  3. Frontier AI companies (e.g., OpenAI, Google) that scrape with clear user-agent strings, respect robots.txt, and don't use residential proxies — but their sheer volume still strains sites.

The real mystery: who pays the residential proxy networks? It's not the frontier model companies (as far as LWN knows). But undercover models — corporate, government, criminal — are likely customers. These tools are weapons in an AI arms race, and the open web is collateral damage.

Defending the Open Web

Sites are fighting back. Anubis (proof-of-work) is widespread. Others use CAPTCHAs, login gates, paywalls, or data poisoning (e.g., iocaine). LWN has implemented aggressive optimizations and defensive measures (details kept secret for obvious reasons). They avoided Anubis because it annoys users and scrapers with millions of machines can easily solve PoW. They also avoid allowlisting dominant search engines to prevent further monopoly entrenchment.

During attacks, LWN's response time is often better than during calm periods — defensive measures kick in when needed. Anonymous readers may encounter delays; logged-in users don't.

The Arms Race Continues

Google's takedowns provide temporary relief, but new networks emerge. The industry enabling this — from malware authors to VPN providers to AI companies — shows little regard for independent sites. Until ethical standards are enforced, site operators must keep fighting.

Technical Takeaways for Developers

  • Rate limiting alone is insufficient when traffic comes from millions of unique IPs.
  • Client-side challenges (CAPTCHAs, PoW) can be bypassed by distributed botnets.
  • Server-side optimization (caching, minimizing expensive ops) helps absorb load.
  • Monitoring for anomalies (no image/CSS requests, fake user-agents) can flag bots early.
  • Consider data poisoning tools like iocaine to contaminate scraped data.
  • Avoid vendor lock-in from anti-bot services that may entrench monopolies.

Conclusion

The scraping war is far from over. LWN's experience shows that even aggressive defenses only buy time. The open web's survival depends on broader industry accountability. Until then, developers must harden their sites and stay vigilant.