55 min read

Public Corpora Are Turning From Default to Exception

Why public corpora are becoming the exception in enterprise AI: dated evidence on robots.txt restrictions, licensing, disclosure duties, plus an acceptance checklist and governance tiers for public, licensed and internal data.

Public Corpora Are Turning From Default to Exception

Introduction

For roughly a decade, "start with a public corpus" was the unexamined first move in enterprise data and AI programs. C4, RefinedWeb, Dolma and their Common Crawl-derived cousins were treated as a free, effectively unlimited baseline: assemble the data, filter it, train or fine-tune, and move on. That default is now collapsing, and the collapse is measurable rather than a matter of taste.

The stakes have shifted for the people who sign off on these programs — the CTO who owns the platform, the data architect who assembles the pipeline, the procurement lead who signs the vendor contract. Continuing to treat public data as a cost-free baseline now carries three distinct costs: legal exposure when a corpus traces to sources that were never licensed, silent quality decay as the most valuable domains restrict access, and re-platforming later when a snapshot you validated is no longer the snapshot you can legally use.

This piece works in a fixed order. It defines what a public corpus actually is and separates access from permission. It lays out dated, attributable evidence that the shift is structural. It then gives you a decision framework for when a public corpus is still the right call, an acceptance checklist a data team can apply before ingestion, a tiered governance model for mixing public, licensed and internal data, and the failure modes to design against. Vendor performance figures are not treated as fact anywhere in the argument; where a number is vendor-reported, it is labeled as such.

What "Public Corpus" Actually Means — and What It Doesn't

A public corpus is a curated, assembled collection of data whose defining property is that it was compiled for downstream reuse. C4, RefinedWeb and Dolma are the canonical examples: each is a filtered derivative of Common Crawl snapshots, assembled with a documented pipeline and released for others to train on. The assembly is the point. Someone made curation choices — which domains to keep, which languages to subsample, which boilerplate to strip — and published the result as a reusable artifact.

"Publicly accessible" is a weaker and different property. A page reachable without a login is not thereby licensed for assembly into a training corpus. Common Crawl's own description of its crawler makes the distinction concrete: CCBot collects data from "publicly accessible web pages," does not go behind paywalls, does not log in to any websites, and does not employ any method designed to evade access restrictions. It also honors robots.txt exclusions and complies with good-faith takedown requests. Access, in other words, is engineered to be conservative at the source — and it is still not the same thing as permission for every downstream use.

Common Crawl's terms of use reinforce the gap. The foundation grants a limited, non-assignable, non-transferable, non-sublicensable, non-exclusive license to access and use its service and crawled content, reserves all other rights, and states that it may remove crawled content at any time. A team can therefore satisfy "it was public" and still fail "it was a corpus we were entitled to use for this purpose." That distinction is the vocabulary the rest of this article relies on.

The Evidence That This Is a Shift, Not a Preference

The claim that this is a shift rather than a preference rests on dated signals pointing in one direction. Four kinds of evidence matter: measured consent signals, measured crawler restrictions, regulation, and enforcement.

Fully restricted tokens by robots.txt, C4 / RefinedWeb / Dolma (whole corpus vs. head distribution) Whole-corpus robots.txt restriction rose from about 1% of tokens in mid-2023 to 5–7% in April 2024, with a forecast of a further 2–4 percentage points by April 2025. The head distribution (HeadAll) rose from under 3% a year earlier to 20–33% in April 2024. Values are measured percentages of tokens except the April 2025 point, which is a model forecast. Fully restricted tokens by robots.txt, C4 / RefinedWeb / Dolma (whole corpus vs. head distribution). line data: Mid-2023, whole corpus (measured) 1; April 2024, whole corpus (measured, low end) 5; April 2024, whole corpus (measured, high end) 7; April 2025, whole corpus (SARIMA forecast, low end) 7; April 2025, whole corpus (SARIMA forecast, high end) 11; One year before April 2024, HeadAll (measured, under 3%) 3; April 2024, HeadAll (measured, low end) 20; April 2024, HeadAll (measured, high end) 33.Source: Consent in Crisis: The Rapid Decline of the AI Data Commons (Longpre et al., NeurIPS 2024 Datasets & Benchmarks) 2024-07. Fully restricted tokens byrobots.txt, C4 / RefinedWeb/ Dolma (whole corpus vs.head distribution) 0 12.5 25 37.5 50 1 Mid-2023,wholecorpus... 5 April2024,whole... 7 April2024,whole... 7 April2025,whole... 11 April2025,whole... 3 One yearbeforeApril... 20 April2024,HeadAll... 33 April2024,HeadAll... Source: Consent in Crisis: The Rapid Decline of theAI Data Commons (Longpre et al., NeurIPS 2024Datasets & Benchmarks) (2024-07)

Robots.txt restriction rates for the three Common Crawl-derived corpora audited by Consent in Crisis (Longpre et al., NeurIPS 2024). The whole-corpus series measures the share of tokens from domains that fully restrict one or more AI crawlers; the head-distribution series (HeadAll, the union of the top-2k domains per corpus) measures the same restriction on the most actively maintained, critical sources. The April 2025 figure is a SARIMA forecast of an additional absolute 2–4% of tokens, not an observation. Terms-of-service restrictions are a separate measure and are not plotted here.

Measured consent signals. The Consent in Crisis audit, published in the NeurIPS 2024 Datasets and Benchmarks Track, conducted a longitudinal audit of the consent protocols for the web domains underlying C4, RefinedWeb and Dolma. The authors examined 14,000 web domains, pulling historical versions of each site's homepage, robots.txt and terms of service from the Wayback Machine at monthly intervals from January 2016 to April 2024. Their headline finding: across the entire corpora, roughly 1% of tokens were restricted in mid-2023, compared with 5–7% of tokens in April 2024. In C4 specifically, about 5% or more of all tokens — or 28% or more of the most actively maintained, critical sources — became fully restricted from use, and 45% of C4 was restricted under terms of service. The authors also found general inconsistencies between what sites express in their terms of service and what they express in robots.txt. Their own framing is careful: these are codified preferences, and whether they are respected or enforced is a separate question. They also make no assertion that the prior absence of a robots.txt or restriction implies consent.

Measured crawler restrictions. A 2025 study by Bouchaud and Ramaciotti, submitted in October 2025, analyzed crawler restrictions across the top one million most-visited websites since 2023. A quarter of the top thousand websites restrict AI crawlers, falling to about one-tenth across the broader top million. Content type matters: 34.2% of news outlets disallow OpenAI's GPTBot, rising to 55% for outlets with high factual reporting. The authors' inference — that heterogeneous blocking patterns may skew training datasets toward low-quality or polarized content — is their interpretation of the pattern, not a measured downstream outcome. The measured finding is which sources block, and how unevenly.

Regulation. EU AI Act Article 53 requires providers of general-purpose AI models to draw up and make publicly available a sufficiently detailed summary about the content used for training, according to a template provided by the AI Office. The European Commission published that template on 24 July 2025, describing it as a common minimal baseline for the information to be made public. In the United States, California's training-data transparency law — signed in 2024 — requires developers of generative AI systems to publish a summary of the datasets used to train their models, including sources or owners and whether datasets were purchased or licensed. It is a disclosure law, not a prohibition: it does not establish a specific enforcement mechanism or penalty structure, and it exempts internal systems available only within a corporate family.

Enforcement. In August 2023, the Danish anti-piracy group Rights Alliance sent a takedown notice that took the Books3 dataset — nearly 200,000 books, about 37GB, sourced from a pirate site — offline from its original host, though copies persisted elsewhere. In June 2025, Judge Alsup ruled that Anthropic's use of books to train its models was fair use, but that building a central library from pirated copies was not. Days later, Judge Chhabria ruled for Meta on fair use while introducing a market-dilution theory of harm that the court acknowledged was not found in earlier cases. These are district-court decisions on specific facts, not settled law — and the two holdings on sourcing point in opposite directions.

The through-line is what distinguishes a shift from a preference: every dated signal runs toward more restriction and more disclosure obligation. None of the signals runs the other way.

When a Public Corpus Is Still the Right Call

None of the above makes public corpora disqualifying. It makes them a decision that requires justification. Four tests determine whether the exception is the right call.

The task tolerates the corpus's known limits. The corpus's measured composition and freshness must match what you are doing with it. A corpus assembled from a 2019 crawl is a poor fit for a task that depends on current events, and a corpus dominated by news and forums is a poor fit for a narrow technical domain.

The snapshot's license and consent status is documented and compatible. You need the specific snapshot's provenance and consent record, not the corpus's general reputation, and the record must cover your intended use — including redistribution if a derivative leaves your organization.

The use is bounded. Benchmarking, prototyping and narrow domain work sit differently from building the foundation of a product that carries disclosure obligations. The narrower the use, the easier the justification.

You can absorb re-validation. If the source's terms change, you need the budget and process to re-check rather than discover the change downstream.

When all four hold, use the corpus and record why. When any one fails, the default has flipped: the burden is on the corpus, not on the alternative.

An Acceptance Checklist Before a Corpus Enters the Pipeline

The checklist below converts the argument into pass/fail gates. Each item names the evidence required; a gate without evidence is a fail.

  • Provenance. Can you name the crawl dates, the source domains and the assembly method for this specific snapshot? The Consent in Crisis authors deliberately looked beneath datasets to the domains they were derived from; your record should be able to do the same for your copy.

  • Consent status. What do robots.txt and terms of service say for the domains in this snapshot, and are they consistent with each other? The audit found systematic inconsistency between the two signals, so a corpus that passes one check may fail the other.

  • Contamination. Has the corpus been checked against your evaluation sets? The C4 contamination analysis found that the percentage of GLUE inputs appearing verbatim in C4.en varies widely, from less than 2% to over 50%; the provable-contamination method shows contamination can be flagged without access to pretraining data or model weights. Treat this as a measurable gate, not a worry.

  • Composition and freshness. Does the corpus's measured composition match the task, and how stale is it relative to your use? Record the answer with the snapshot, not in someone's memory.

  • Redistribution rights. If the corpus or a derivative leaves your organization, does the license permit it? Common Crawl's terms, for example, grant a limited, non-sublicensable license and reserve all other rights.

  • Re-validation trigger. What change in source terms or regulation forces a re-check? Name the trigger and the owner before ingestion, not after.

Governing the Public / Licensed / Internal Mix

The binary "public versus private" is the wrong control structure. What follows is a proposed governance framework — a governance choice, not a legal safe harbor, and the legal determination belongs to your counsel.

The model has four tiers, each with its own provenance record, review cadence and permitted uses.

Public with documented consent. Web-derived data whose snapshot-level provenance and consent status are recorded and consistent with the intended use. Review cadence: on every new snapshot and on any change to source terms. Permitted uses: those the record supports, including redistribution only where the license allows it.

Licensed. Data acquired under a contract that names the permitted uses. Review cadence: at contract renewal and on any scope change. Permitted uses: exactly what the contract says, with the contract reference attached to the dataset.

Internal first-party. Data your organization generated or collected directly. Review cadence: on schema or collection-policy change. Permitted uses: internal analytics and model development, subject to your own privacy and retention rules.

Synthetic or derived. Outputs generated from the tiers above. Review cadence: whenever the upstream tier changes. Permitted uses: inherit the most restrictive upstream tier unless counsel confirms otherwise.

The tiers connect to the disclosure obligations described earlier. A tier that feeds a model subject to Article 53 or California-style disclosure needs a record that is summary-ready — sources or owners, whether data was purchased or licensed, and enough detail to populate a public summary without a scramble. Building that record at ingestion is cheap; reconstructing it after a model ships is not.

Failure Modes When Public Is the Default

Five failure modes recur when public is the default. Each is stated with the control that prevents it; where a mode is a reasoned risk rather than a documented incident, it is labeled as such.

Silent composition drift. The corpus changes under you between snapshots, so the data you validated is not the data you are running. Control: pin the snapshot by crawl date and record its composition at ingestion.

Consent-signal mismatch. robots.txt says one thing and the terms of service say another, and your pipeline checks only one. Control: check both signals per snapshot and record any inconsistency as a finding, not a footnote.

Benchmark contamination. Evaluation results are inflated because test inputs appear verbatim in the training corpus. Control: run a contamination check against your evaluation sets before the corpus enters the pipeline, and re-run it when the evaluation set changes.

License laundering. A dataset labeled "public" traces back to pirated sources — the pattern the Books3 takedown exposed. Control: require provenance that reaches the original source, not just the aggregator's label.

Disclosure gaps discovered after shipping. A model subject to a training-content summary obligation ships without a record detailed enough to populate the summary. Control: make the summary-ready record a gate at ingestion, before the model is trained.

A sixth risk is reasoned rather than documented: as restrictions concentrate on the most actively maintained and monetized domains, a corpus that respects them loses access to exactly the content that made it valuable. The audit's own framing supports treating this as a live risk — it notes that if restrictions are respected, they rapidly bias the diversity, freshness and scaling laws of general-purpose AI systems. The control is the same composition check, run on a schedule rather than once.

Conclusion

Public corpora are becoming the exception because the evidence, the law and the enforcement now point the same way. Measured consent signals moved from roughly 1% to 5–7% of tokens restricted in a single year; crawler restrictions concentrate on the most valuable sources; disclosure obligations now have a published template in the EU and a state-law analogue in California; and enforcement has taken datasets offline and split fair use from pirated sourcing in the same month. The teams that treat "public" as a decision to be justified rather than a default to be assumed will avoid the re-platforming and disclosure costs the rest of the field is walking into. Build the provenance record and the acceptance checklist now, before the next snapshot or the next regulation forces the question.

FAQ

Doesn't this only affect foundation-model training, not enterprise analytics? No. The same provenance and consent questions apply to any pipeline that ingests assembled web data, including enrichment and retrieval workloads. What differs by use is the disclosure obligation, not the underlying permission question.

Isn't robots.txt voluntary, so restrictions don't really bind? Compliance is good-faith and voluntary, and the Consent in Crisis authors separate codified preference from enforcement. The practical risk is legal and reputational rather than technical — which is why the checklist treats consent signals as evidence to record, not as a technical block.

If fair use won for Anthropic and Meta, why worry? Those were district-court rulings on specific facts. In the Anthropic case, fair use for training and liability for building a library from pirated copies were decided differently in the same order, and the Meta ruling rests on a market-dilution theory the court itself described as new. Treat the law as unsettled and your provenance record as the thing you control.

What if I've already built on a public corpus? Audit provenance and consent status first, then decide whether to re-source or document. Do not assume the snapshot you have is the one you validated — check the crawl dates and the current terms for its domains before you rely on it again.