digital · education · family: success's shadow
open enough to be crushed
Open research repositories are being knocked offline by AI crawlers, and every defence that works also blocks their readers
Problem statement
Open repositories were built to be as accessible as possible: their underlying mission, in COAR's words, "is to provide access to their collections so they are reused and repurposed for the good of scholarship and society." That design succeeded, and it is now the attack surface. In COAR's April 2025 survey of 66 repositories worldwide, over 90% reported encountering aggressive bots, usually more than once a week; 80% had experienced service disruptions; about a third had at least one outage of 2–24 hours and about a fifth an outage lasting several days. Wikimedia reports the same shape of problem in its own numbers: "Since January 2024, we have seen the bandwidth used for downloading multimedia content grow by 50%," and "at least 65% of this resource-consuming traffic we get for the website is coming from bots, a disproportionate amount given the overall pageviews from bots are about 35% of the total" — because crawlers "bulk read" larger numbers of pages, including less popular ones, which are more likely to have to be served from the core datacenter rather than a regional cache and so cost more to serve. The unsolved problem is not "stop the bots." It is that every defence a repository can deploy today is a blunt instrument that also removes access for the humans and benign machines the repository exists to serve.
Why this matters
This is the load-bearing infrastructure of open science and public knowledge. The COAR survey found repositories now spend on average about three days a month on bot mitigation — roughly 10% of a full-time employee, with some reporting "0.5 FTE on this issue per week" and one "20% of a full time engineer," plus an energy cost from servers running hot. The failure mode that matters is not downtime but quiet enclosure: as one respondent put it, "at times we restricted our sites to national IPs, and received messages from global researchers unable to access our open resources." If the rational response to crawler load is IP-range blocking, CAPTCHAs and country-level restrictions, then the global repository network converts itself into a set of gated silos. COAR frames the stake precisely: bot pressure "could potentially result in repositories limiting access to their resources for both human and machine users — leading to a situation where the value of the global repository network is substantially diminished."
What’s been tried and why it hasn’t worked
COAR's respondents document a full ladder of countermeasures and the limits of each. robots.txt is the protocol designed for exactly this and is widely deployed, but "there is no inherent mechanism for enforcing them, and less scrupulous systems may simply ignore them" — it works on well-behaved crawlers, which are not the problem. IP blocking and firewalls (Fail2ban, mod_evasive, blocking whole cloud provider ranges, in some cases whole countries) stop traffic but catch others with it: one respondent notes that blocking a provider's range carries a "slight risk could block other services." Rate limiting caps the number of requests per second accepted from any one IP address. Whitelisting friendly bots adds a maintenance job — "it requires resources on our side to maintain/manage a whitelist." CDNs and CAPTCHA challenges block without discrimination: as COAR observes, "such interventions block all benign systems, while only blocking some (perhaps most) malign systems." Honeypots such as Cloudflare's AI Labyrinth, mentioned by one respondent, waste the crawler's time and bandwidth. Two things in the sources help explain why none of this converges. First, cost: Wikimedia reports that requests for less popular pages are much more expensive to serve than the cached pages human readers concentrate on, and one COAR respondent disabled queries because bot activity on them was more likely to bring down the site. Second, identity: COAR says there is no way to be 100% certain of the purpose of these bots, and many respondents were aware that their measures were blocking legitimate users and friendly networks. Respondents describe the result as a treadmill: "each measure was implemented after the last measures became ineffective," and "at present, it's a cat-and-mouse game."
What would unlock progress
One COAR respondent has already named the success criterion, and it is not blocking: "Success would mean allowing legitimate AI harvesting traffic without service degradation." That reframes the problem from access control to supply design — if bulk consumers were offered a cheap, complete, well-advertised path (dumps, bulk APIs, delta feeds, an object store), crawling the human-facing interface would become the expensive option rather than the default, which is how the load could be removed instead of fought. Wikimedia poses the same question for itself: "How might we funnel developers and reusers into preferred, supported channels of access?" The complementary unlock would be machine-readable, enforceable expression of terms plus verifiable agent identity, so that a repository can differentiate rather than deny.
Entry points for student teams
A team could instrument a real repository (DSpace is the one platform the COAR report names) and build a cost-aware traffic classifier: score requests by the server resources they consume rather than by count, and measure how much load could be shed by serving bulk consumers a dump instead of blocking them — and, critically, measure the human false-positive rate of each mitigation, a number the COAR survey does not report. A protocol team could prototype and evaluate a signed-agent-identity or machine-readable licence handshake against realistic crawler behaviour. A measurement team could replicate the COAR survey for a different sector — municipal archives, museum collections, government open-data portals — which the COAR survey did not cover. A design team could work the user-facing side: what does a CAPTCHA-free proof of humanity look like for a blind researcher on a slow connection? Relevant skills: systems and web engineering, network measurement, applied statistics, HCI and accessibility, information policy.
Genes
Structural cousins: stuck the same way, in other fields
Sources
Shearer K, Walk P, "The impact of AI bots and crawlers on open repositories: Results of a COAR survey, April 2025," Confederation of Open Access Repositories, 3 June 2025, accessed 2026-08-18; Mueller B, Danis C, Lavagetto G (Wikimedia Foundation), "How crawlers impact the operations of the Wikimedia projects," Diff, 1 April 2025, accessed 2026-08-18 go to source 1 ↗ go to source 2 ↗
Working notes
Notes made while writing and checking this brief, shown as written. They use working shorthand.
The COAR survey report (9 pp.) was downloaded and read in full on 2026-08-18; all percentages, staffing figures and quoted respondent statements are verbatim from it (n=66: 22 Canada/US, 22 Europe, 9 Latin America, 6 Asia, 4 Australasia, 2 Africa, 1 unknown). The Wikimedia figures are verbatim quotes from the Foundation's own engineering post. COAR is the international association of repository operators and the survey is of its members, which makes this practitioner discourse rather than journalism; it is graded tier 2 as an association survey rather than a peer-reviewed or agency source.
Related collection briefs: `digital-ml-benchmark-dataset-integrity-erosion` and `digital-synthetic-content-provenance-verification` concern the quality and provenance of AI training material; this brief is the infrastructure-load and access-enclosure problem on the supply side, and does not overlap them.
Source type: Self-articulated (repository operators surveying themselves; platform operator publishing its own traffic data).
Verified at intake 2026-08-18: gate (net) + adversarial source check + contested-tag second coding. Verifier: COAR 8-page PDF and the Wikimedia Diff post fetched; every percentage, staffing figure and quotation confirmed verbatim.
Reconciliation 2026-10-01: Fact repair after reading the whole write-up against both of its sources. Both were fetched directly on 2026-10-01 and read in full: the COAR survey report (https://coar-repositories.org/wp-content/uploads/2025/06/Report-of-the-COAR-Survey-on-AI-Bots-June-2025-1.pdf, 9 pages) and the Wikimedia Diff post (https://diff.wikimedia.org/2025/04/01/how-crawlers-impact-the-operations-of-the-wikimedia-projects/). (1) The Source line gave the author of the Diff post as "Wikimedia Foundation." The page's byline is Birgit Mueller, Chris Danis and Giuseppe Lavagetto, each of the Wikimedia Foundation; the Source line now names them. (2) The opening sentence listed kinds of repository (university and national archives, preprint servers, cultural-heritage collections) and design features (no login, full text, machine-readable, harvestable). Neither source gives these lists; the sentence now quotes COAR's statement of the repositories' mission. (3) The text said crawlers read "the obscure pages that caches were never designed to hold." The Diff post says crawlers "bulk read" larger numbers of pages, including less popular ones, which are more likely to be served from the core datacenter and so cost more; the sentence now says that (the second reader adjusted it so that it does not say crawlers bulk read only the less popular pages). (4) "Run mostly by small library IT teams" is not in either source and was removed. (5) The statement that the people cut off first are researchers in under-resourced institutions and countries is not in either source and was removed, as was "logins" in the list of responses (the COAR report does not mention logins). (6) The text said IP blocking "cannot distinguish a scraper renting an address from a researcher behind the same provider." The report says only that one respondent noted a "slight risk could block other services"; the sentence now quotes that. (7) "Rate limiting helps until crawlers distribute across addresses" is not in the report, which only describes what rate limiting does; the sentence now gives that description, and the paragraph's opening clause, which promised "the specific way each one fails," now says "the limits of each." (8) Whitelisting was called "an unbounded maintenance job"; the report has one respondent saying it requires resources to maintain, and the text now says it adds a maintenance job. (9) CDNs and CAPTCHAs were called "the most effective and the most damaging," and the text said this is "why legitimate archiving and discovery services get caught." The report ranks no measure and names no such services; both phrases were removed and the COAR quotation kept, introduced with "block without discrimination," which is what the quoted sentence says (the first reader's "cut the widest" was itself a ranking). (10) The text said honeypots "do nothing for the load already arriving." The report says only that the AI Labyrinth, mentioned by one respondent, wastes the crawler's time and bandwidth; the sentence now says that. (11) The "two structural facts" (that defences counting requests mis-measure load, and that every mitigation is a proxy with false positives) were the write-up's own reasoning stated as fact. The passage now gives what the sources say: the Diff post on the higher cost of serving less popular pages, one COAR respondent disabling queries, COAR's statement that there is no way to be 100% certain of the purpose of the bots, and its finding that many respondents knew they were blocking legitimate users. (12) "The community has already named the success criterion" became "One COAR respondent"; the quoted sentence is one respondent's. (13) Removed because neither source contains them: licence declarations attached to robots.txt and signed agent identities as "early candidates," and the sentence on OAI-PMH, ResourceSync and CDNs as precedents. A sentence quoting the Diff post's question about funnelling reusers into "preferred, supported channels of access" was added in their place. (14) In the entry points, "many run DSpace, EPrints or InvenioRDM" became a statement that DSpace is the one platform the report names; "a number nobody currently reports" and "where no baseline exists" now say only that the COAR survey does not report that number or cover those sectors. (15) The notes above said the COAR report is 8 pages; the file has 9. They also said both sources date the change to January 2024 or later; the Diff post does, and the COAR report says "the last two years or so." Both were corrected in place. The intake line above still says "8-page PDF" and is left as a record. Confirmed in the sources and left as written: the COAR authors, title and date; 66 responses and their regional split; over 90% encountering aggressive bots, usually more than once a week; 80% with service disruptions; about one-third with an outage of 2 to 24 hours and about one-fifth with one lasting several days; about three days a month, or about 10% of a full-time employee; and every remaining quotation from the COAR report and the Diff post (the 50% bandwidth growth since January 2024, the 65% and 35% bot shares, the 0.5 FTE and 20% engineer statements, the national-IPs statement, the robots.txt, whitelist, CDN, "cat-and-mouse" and "Success would mean" passages, and COAR's sentence on the diminished value of the network). Not checked, because they are the write-up's own proposals and not claims about a source: the suggested bulk-access paths (dumps, bulk APIs, delta feeds, an object store) and the student project ideas. No title changed. Genome tags untouched. A second reader went over these repairs on the same day, reading both sources in full again, and confirmed every quotation in the write-up against its source; items (3), (7) and (9) carry that reader's rewordings, and every other repair stands as the first reader made it.