open enough to be crushed

open enough to be crushed

Problem statement

Open repositories — university and national research archives, preprint servers, cultural-heritage collections — were built to be maximally accessible: no login, full text, machine-readable, harvestable. That design succeeded, and it is now the attack surface. In COAR's April 2025 survey of 66 repositories worldwide, over 90% reported encountering aggressive bots, usually more than once a week; 80% had experienced service disruptions; about a third had at least one outage of 2–24 hours and about a fifth an outage lasting several days. Wikimedia reports the same shape of problem in its own numbers: "Since January 2024, we have seen the bandwidth used for downloading multimedia content grow by 50%," and "at least 65% of this resource-consuming traffic we get for the website is coming from bots, a disproportionate amount given the overall pageviews from bots are about 35% of the total" — because crawlers read the obscure pages that caches were never designed to hold. The unsolved problem is not "stop the bots." It is that every defence a repository can deploy today is a blunt instrument that also removes access for the humans and benign machines the repository exists to serve.

Why this matters

This is the load-bearing infrastructure of open science and public knowledge, run mostly by small library IT teams. The COAR survey found repositories now spend on average about three days a month on bot mitigation — roughly 10% of a full-time employee, with some reporting "0.5 FTE on this issue per week" and one "20% of a full time engineer," plus an energy cost from servers running hot. The failure mode that matters is not downtime but quiet enclosure: as one respondent put it, "at times we restricted our sites to national IPs, and received messages from global researchers unable to access our open resources." If the rational response to crawler load is IP-range blocking, CAPTCHAs, logins and country-level restrictions, then the global repository network converts itself into a set of gated silos — and the people cut off first are researchers in under-resourced institutions and countries, the exact constituency open access was built for. COAR frames the stake precisely: bot pressure "could potentially result in repositories limiting access to their resources for both human and machine users — leading to a situation where the value of the global repository network is substantially diminished."

What’s been tried and why it hasn’t worked

COAR's respondents document a full ladder of countermeasures and the specific way each one fails. robots.txt is the protocol designed for exactly this and is widely deployed, but "there is no inherent mechanism for enforcing them, and less scrupulous systems may simply ignore them" — it works on well-behaved crawlers, which are not the problem. IP blocking and firewalls (Fail2ban, mod_evasive, blocking whole cloud provider ranges, in some cases whole countries) stop traffic but cannot distinguish a scraper renting an address from a researcher behind the same provider. Rate limiting helps until crawlers distribute across addresses. Whitelisting friendly bots inverts the problem into an unbounded maintenance job — "it requires resources on our side to maintain/manage a whitelist." CDNs and CAPTCHA challenges are the most effective and the most damaging: as COAR observes, "such interventions block all benign systems, while only blocking some (perhaps most) malign systems," which is why legitimate archiving and discovery services get caught. Honeypots such as Cloudflare's AI Labyrinth waste crawler resources but do nothing for the load already arriving. Two structural facts explain why none of this converges. First, the cost asymmetry: a crawler's cheapest request (a deep, uncached, dynamically generated page or a query) is the repository's most expensive, so defences that count requests mis-measure the actual load. Second, identity: there is no reliable way for a server to know what is on the other end of a connection, so every mitigation is a proxy — IP, user-agent, behaviour — and every proxy has false positives that land on humans. Respondents describe the result as a treadmill: "each measure was implemented after the last measures became ineffective," and "at present, it's a cat-and-mouse game."

What would unlock progress

The community has already named the success criterion, and it is not blocking: "Success would mean allowing legitimate AI harvesting traffic without service degradation." That reframes the problem from access control to supply design — if bulk consumers were offered a cheap, complete, well-advertised path (dumps, bulk APIs, delta feeds, an object store), crawling the human-facing interface would become the expensive option rather than the default, which is how the load could be removed instead of fought. The complementary unlock is machine-readable, enforceable expression of terms plus verifiable agent identity — early candidates include licence declarations attached to robots.txt and cryptographically signed agent identities — so that a repository can differentiate rather than deny. Adjacent precedents are strong: OAI-PMH and ResourceSync solved bulk metadata harvesting decades ago by giving harvesters a better door, and CDNs solved flash-crowd load by making the cheap path the obvious one.

Entry points for student teams

A team could instrument a real repository (many run DSpace, EPrints or InvenioRDM) and build a cost-aware traffic classifier: score requests by the server resources they consume rather than by count, and measure how much load could be shed by serving bulk consumers a dump instead of blocking them — and, critically, measure the human false-positive rate of each mitigation, a number nobody currently reports. A protocol team could prototype and evaluate a signed-agent-identity or machine-readable licence handshake against realistic crawler behaviour. A measurement team could replicate the COAR survey for a different sector — municipal archives, museum collections, government open-data portals — where no baseline exists. A design team could work the user-facing side: what does a CAPTCHA-free proof of humanity look like for a blind researcher on a slow connection? Relevant skills: systems and web engineering, network measurement, applied statistics, HCI and accessibility, information policy.