1h 11m read

Open Weights Give You Weights, Not Provenance

Open-weight downloads give you files and a licence, not a verified record of origin. A practical guide to licence obligations, checkpoint verification, disclosure gaps and data lineage for enterprise AI deployments.

Open Weights Give You Weights, Not Provenance

An open-weight download gives you a set of files and a licence. It does not give you a verified account of where the model came from, what it was trained on, or whether the bytes you received are the bytes the publisher released. Those are three separate questions, and the download answers none of them.

For a CTO, data architect or procurement lead, that distinction has a practical consequence: provenance is a control you build around the artefact, not a property you inherit with it. The licence tells you what you may do. The repository tells you what was published. Neither tells you what you are running.

This article works through the chain in order. It starts with what is actually in the download, then separates open weights from open source AI, then reads the licence obligations that bind a deployment, then shows how to verify a checkpoint, then sets out what training-data and evaluation documentation can and cannot establish. From there it assigns accountability for failures, connects model provenance to the lineage of the enterprise data the model queries, and closes with a procurement and verification checklist you can run before signing or deploying.

What you actually download: files, not a pedigree

A model repository is a bundle, not an object. The typical contents are weight files, configuration files, tokenizer files and a model card. The weights are the parameters. The configuration and tokenizer files determine how those parameters are interpreted and how text is converted to and from token IDs. The model card is documentation the publisher wrote. It is useful, and it is not a verified record of anything.

The weight format carries its own risk. PyTorch's long-standing serialization uses pickle, and Hugging Face's own pickle-scanning guidance states plainly that dangerous arbitrary code execution attacks can be perpetrated when you load a pickle file. Deserializing a pickle can execute code. The Hub scans pickle files and displays the imports it finds, but it does not block the download; the same guidance carries a disclaimer that scanning is not 100% foolproof and that checking whether something is safe remains the user's responsibility.

This is not hypothetical. JFrog's security research documented a PyTorch model on the Hub with a payload injected through pickle's __reduce__ method that opened a reverse shell to an external host, and reported roughly 100 models with real, harmful payloads at the time of its analysis, excluding false positives. Treat that as a point-in-time research finding, not a current rate. The mechanism is what matters: a weight file can be code.

Safetensors was built to remove that vector. The PyTorch project page for safetensors describes it as a secure, fast format designed to replace pickle that prevents arbitrary code execution during deserialization by only allowing numerical tensor data. That is a real improvement in one dimension. It says nothing about whether the tensors are any good, or whether they are the tensors the publisher trained.

So the download presents two independent verification problems. Safety: can loading this file execute something I did not intend? Identity: is this the artefact the publisher released? A clean scan answers neither completely, and answering one does not answer the other.

"Open weights" is not "open source" — and procurement needs the distinction

The Open Source Initiative draws the line explicitly. Its explainer on open weights states that open weights alone fall short of open source because they do not provide the underlying training process, code, or comprehensive data details required for full-fledged use, study, modification, and sharing.

The Open Source AI Definition 1.0 sets out what closes that gap. It requires the four freedoms — to use, study, modify and share — and it names three components that must accompany them:

  • Data Information: sufficiently detailed information about the data used to train the system for a skilled person to build a substantially equivalent system, including a complete description of all data used (including unshareable data), its provenance, scope and characteristics, how it was obtained and selected, labelling procedures, and processing and filtering methods; a listing of publicly available training data and where to obtain it; and a listing of third-party training data and where to obtain it, including for a fee.

  • Code: the complete source code used to train and run the system, under an OSI-approved licence.

  • Parameters: the weights and other configuration settings, made available under OSI-approved terms.

The definition is deliberately indifferent to what you call the offering. The OSI FAQ confirms that it makes no distinction between an AI system, a model, or weights and parameters: the requirements for the preferred form for making modifications are the same. Calling a weights release "open source" does not lower the bar.

The FAQ also addresses the obvious objection about training data. The definition does not require releasing the data itself; it requires Data Information, including a detailed description of data that cannot be shared, such as PII, so that another party can construct a dataset with identical structure. That is a disclosure standard, and it is a demanding one.

A genuinely open release shows what meeting it looks like. Ai2's OLMo 2 release ships weights, data, code, recipes, intermediate checkpoints and instruction-tuned models. That is the shape the definition describes.

For procurement, the decision rule is to classify each candidate into one of three buckets and record the classification with the evidence behind it: open weights only, open source AI against a named definition, or neither. Record which definition the vendor is claiming against, because "open" on a slide is not a claim anyone can test.

Which licence obligations actually bind your deployment

Licence terms vary enough across releases that a generic "open licence" assumption will misstate your obligations. Two primary texts make the point.

Count of recorded obligation types by licence (three-licence sample) Grouped bar chart counting how many of five obligation categories each of three licences records: scale threshold, attribution requirement, derivative naming requirement, incorporated use policy, and termination on breach. Apache-2.0 records 1 of 5 (attribution/notice preservation only); Llama 4 Community License records 5 of 5; Gemma Terms of Use records 3 of 5 (attribution, incorporated use policy, termination). Counts are a summary of the licence texts, not a measure of obligation severity. Count of recorded obligation types by licence (three-licence sample). grouped bar data: Scale threshold: Apache-2.0 0, Llama 4 Community 1, Gemma Terms of Use 0; Attribution: Apache-2.0 1, Llama 4 Community 1, Gemma Terms of Use 1; Derivative naming: Apache-2.0 0, Llama 4 Community 1, Gemma Terms of Use 0; Use policy incorporated: Apache-2.0 0, Llama 4 Community 1, Gemma Terms of Use 1; Termination on breach: Apache-2.0 0, Llama 4 Community 1, Gemma Terms of Use 1.Source: Llama 4 Community License Agreement; Gemma Terms of Use 2025-04-05. Count of recorded obligationtypes by licence(three-licence sample) Apache-2.0 Llama 4Community Gemma Terms ofUse 0 0.25 0.5 0.75 1 0 1 0 Scalethreshold 1 1 1 Attribution 0 1 0 Derivativenaming 0 1 1 Use policyincorporated 0 1 1 Terminationon breach Source: Llama 4 Community License Agreement; GemmaTerms of Use (2025-04-05)

Three-licence sample, not a survey. Terms as retrieved on 2026-10-30 from the publishers' own pages; obligations are summarised and the licence text governs. Apache-2.0 is included as a reference point for a permissive licence with no scale threshold or model-specific use policy.

The Llama 4 Community License imposes specific duties on distribution. If you distribute the materials or a product containing them, you must provide a copy of the agreement and prominently display "Built with Llama" on a related website, user interface, blog post, about page or product documentation. If you use the materials or their outputs to create, train, fine-tune or otherwise improve a distributed model, the model name must begin with "Llama." You must retain an attribution notice in a "Notice" text file in distributed copies. Your use must comply with an acceptable use policy incorporated by reference into the agreement. And there is a commercial trigger: if, on the release date, the monthly active users of your products or services exceeded 700 million in the preceding calendar month, you must request a separate licence from Meta and are not authorized to exercise the rights until it is granted. The agreement also disclaims warranties and places responsibility for determining the appropriateness of use and redistribution on you.

The Gemma Terms of Use take a different shape. They incorporate a prohibited use policy, and Google reserves the right to restrict, remotely or otherwise, usage of the Gemma Services to the maximum extent permitted by law. That reservation is a term of the agreement, and it belongs in your risk register rather than in a footnote.

Regulation adds a layer. Article 53(2) of the EU AI Act exempts providers of models released under a free and open-source licence that allows access, usage, modification and distribution, with parameters, architecture information and usage information made publicly available, from the obligations in Article 53(1)(a) and (b). The exception explicitly does not apply to general-purpose AI models with systemic risks. A permissive-sounding licence does not by itself settle your regulatory position.

The practical instrument is a licence-obligation register, one row per model, with the trigger condition recorded next to each obligation: does it fire on distribution, on creating a derivative, on a user-count threshold, on incorporating an acceptable-use policy, or on a remote-restriction clause? Map each trigger to your actual deployment. A model you never redistribute and never expose as a service may trigger fewer obligations than one embedded in a product — but the licence text governs, not the assumption.

Verifying the checkpoint is the one the publisher released

Verification is a procedure, and the hub gives you the mechanics for it.

Pin a revision. Downloading from the default branch means the artefact can change under you. The Hub CLI documentation shows the --revision option accepting a commit hash, branch name or tag, and the API reference shows resolve_revision returning a concrete commit hash that can then be passed to every subsequent download so all files come from the same commit. Unit 42's analysis of model namespace reuse recommends exactly this: pinning to a specific commit so the model stays in an expected state and its behaviour does not change unexpectedly.

Verify cached checksums. The CLI exposes hf cache verify to verify cached file checksums. Note the limitation recorded in huggingface_hub issue #2364: checksums are computed in the cache directory only under specific conditions, and the maintainers note that downloaded file size is already checked to catch network truncation. A checksum check is a stronger integrity control than a size check, and you should know which one your pipeline actually performs.

Check commit signing. The GPG signing documentation defines three states: Verified means the commit is signed and the signature is verified against a public key uploaded to the signer's account; Unverified means it is signed but the signature could not be verified; No signing status means the commit is not signed at all. A verified commit tells you who signed it. It does not tell you the weights are safe, well-trained, or even that the signer trained them.

Keep the three controls distinct in what they prove. A checksum confirms the bytes match a recorded hash. A signature confirms who authored the commit. Neither confirms the model is fit for your purpose. And the scanner is best-effort: Hugging Face's pickle guidance says so directly.

The intake gate that follows from this is short. Pin the revision. Verify the hash. Prefer safetensors and treat any pickle file as executable code requiring the same review you would give a dependency. Record the revision in the deployment manifest so that the model version in production is a fact you can look up rather than a memory.

What you can and cannot establish about training data and evaluation

The OSI's Data Information requirement is a standard for what a release should disclose. It is not a description of common practice, and many releases do not meet it.

NIST's Generative AI Profile (NIST AI 600-1) records the consequence: most model developers do not disclose specific data sources on which models were trained, limiting user awareness of whether personally identifiable information was trained on and, if so, how it was collected. That is a statement about disclosure practice, and it means an undisclosed training corpus is a normal condition you have to manage, not an anomaly you can escalate away.

Evaluation documentation has a parallel problem. The OLMo 2 authors argue that it should be standard practice for developers to keep a subset of evaluation tasks unseen, and they encourage other open-weight developers to state clearly which tasks were used as reference during model development. They also concede the limit of their own approach: they have no guarantee that tasks they consider unseen during OLMo 2's development are not part of the development set of other models they compare against. Benchmark contamination is not something a single publisher can rule out for the whole field.

The protocol for a documentation gap is to record it rather than resolve it by assumption. For each candidate model, log what is disclosed about training data, what is disclosed about evaluation tasks and which were used as reference, and mark everything else as undisclosed. Treat undisclosed training data as an open risk with an owner and a review date. A model card that is silent on data sources has told you something — that the information is not there — and your record should say so in those terms.

Who is accountable when the model is wrong against your governed data

NIST AI 600-1 describes confabulation as a natural result of the way generative models are designed: they generate outputs that approximate the statistical distribution of their training data, which can produce factually inaccurate or internally inconsistent content. The risk materializes when users believe false content, often because of the confident nature of the response, and act on it. The profile flags this as especially important when generative AI is integrated into consequential decision-making.

Nothing in a model licence changes that dynamic. A licence grants rights and imposes use restrictions. It does not transfer accountability for outputs in your context. The deploying organization decides to use the output, and the deploying organization owns that decision. The Llama 4 licence makes the allocation explicit in its warranty disclaimer: outputs are provided as is, and you are solely responsible for determining the appropriateness of using or redistributing the materials and assume the risks.

Accountability becomes tractable when you separate two provenance chains. Model provenance is where the weights came from: revision, hash, licence, documented training scope. Data provenance is where the answer's inputs came from: source system, transformation history, metric definition, timestamp. A wrong answer is traceable only through the second chain. NIST's suggested actions point the same way: establish known assumptions and practices for determining data origin and content lineage (MP-2.1-001), document sources and types of training data and their origins (MG-3.2-003), and verify that fine-tuning or retrieval-augmented generation data is grounded (MS-2.5-005).

An accountability map assigns each failure mode to the party who can actually control it:

  • Bad weights — the model produces poor outputs on your task regardless of input. The publisher controls training; you control selection and evaluation. Your control is pre-deployment testing against your own task, not the model card.

  • Bad retrieval — the model answers correctly from the wrong document, or from a stale or unauthorized one. You control the retrieval layer, its access controls and its freshness.

  • Bad metric definition — the model computes a number correctly against a definition your business does not use. You control the semantic layer and the metric definitions.

The publisher's name on the licence does not move any of these rows.

Model provenance is only half the chain: lineage and metric definition

An open-weight model queried against governed enterprise data produces an answer whose trustworthiness depends on two independent provenance chains. The model's chain is revision, hash, licence and documented training scope. The data's chain is source system, transformation history, metric definition and timestamp. Verifying the first does not verify the second, and a verified checkpoint does not make a query against an undefined metric correct.

This is where most enterprise deployments actually fail. The model is pinned and the hash is recorded, and the answer to "what was revenue last quarter" still depends on which of three revenue definitions the semantic layer applied, whether the underlying table was the restated version, and whether the query hit yesterday's partition. None of that is in the model repository, and none of it can be supplied by the publisher.

The data-side chain is something the deploying team has to own. That is the case for running the model against a governed data layer rather than an ungoverned copy: the lakehouse holds the ingestion history, the transformation lineage and the metric definitions that let you trace a wrong answer back to its cause. Singdata's platform is positioned around that layer — a lakehouse that governs ingestion through to lineage, with business semantics and agent access in the same architecture. The evaluation criteria a buyer should test are concrete: can you trace a given answer back to its source tables and the transformation steps applied; can you see which metric definition was used and when it changed; can you bind a specific model revision to a specific governed dataset so that the pairing is recorded rather than assumed. Those are testable questions, and they are the ones that determine whether the second chain exists at all.

A procurement and verification checklist

The checklist below is an acceptance gate to run before signing or deploying. Each item produces a record; an item you cannot complete is itself a finding.

  1. Classify the release. Against the OSI definition, record whether the candidate is open weights only, open source AI, or neither. Record the definition the vendor claims.

  2. Read the primary licence text. Not the model card summary, not the marketing page. The agreement itself, including incorporated policies.

  3. Build a licence-obligation register. One row per obligation with its trigger condition: distribution, derivative creation, naming, attribution file, user-count threshold, acceptable-use incorporation, remote restriction.

  4. Pin a revision. Resolve to a commit hash and record it in the deployment manifest. Do not track the default branch in production.

  5. Verify integrity and signing. Run the cache checksum verification your tooling supports, and record the commit signing status. Note what each control proves and what it does not.

  6. Prefer safetensors. Treat any pickle-based weight file as executable code subject to dependency review.

  7. Document the disclosure gap. Record what the publisher discloses about training data and evaluation tasks, and mark what is undisclosed as an open risk with an owner.

  8. Map accountability. Assign bad weights, bad retrieval and bad metric definition to the party who controls each, and record the control that party operates.

  9. Bind model to data. Record which governed dataset and which metric definitions the pinned model revision is authorized to query.

This is a starting gate, not a certification. It establishes that you checked specific things and recorded the result. It does not establish that the model is safe, accurate or fit for purpose — those are claims your own evaluation has to support.

The strongest counterargument: isn't this just vendor due diligence?

The fair objection is that procurement and security review already cover this ground. There is real overlap: both activities ask who the supplier is, what they claim, and what controls they operate.

The difference is the object under review. Vendor due diligence assesses a supplier's claims and controls before you buy. Model provenance verifies an artefact you now possess and can inspect byte for byte. The evidence is different — hashes, signatures, licence texts, revision pins rather than questionnaires and certifications — and the failure mode is different. A vendor review catches a misrepresented capability. It does not catch a silent substitution on the default branch, an unsigned commit, or a pickle file that executes on load.

The honest limitation is on the other side. For many releases the training-data and evaluation record simply is not available, and no amount of verification will produce it. The response is to record the gap, assign it an owner and manage it as a risk. A checklist that pretends to close that gap is worse than one that names it.

Which returns to the thesis. Open weights give you weights. Provenance is the record you build around them: the revision you pinned, the hash you verified, the licence obligations you registered, the disclosures you logged and the gaps you accepted with your eyes open. Treat the checklist above as the first version of your own intake standard, and revise it as your deployments and the available documentation change.

Frequently asked questions

Does safetensors make a model safe? No. It removes the deserialization code-execution vector by allowing only numerical tensor data. It does not address a model that is badly trained, biased, or deliberately poisoned. Safety of the artefact and safety of the model's behaviour are separate questions.

Does a verified commit prove the publisher trained the weights? No. A Verified status means the commit is signed and the signature was checked against a public key uploaded to the signer's account. It proves who signed the commit. It says nothing about the training process or the origin of the parameters.

Does the OSI definition require releasing the training data? No. It requires Data Information sufficient for a skilled person to build a substantially equivalent system, including a detailed description of data that cannot be shared, such as PII, so that an equivalent dataset can be constructed. The data itself need not be redistributed.

Does the EU AI Act's open-source exception apply to every open-weight model? No. Article 53(2) exempts providers of models released under a free and open-source licence meeting the stated conditions from the obligations in Article 53(1)(a) and (b), and the exception does not apply to general-purpose AI models with systemic risks.

Does an acceptable-use policy bind a deployment that never redistributes the model? The licence text governs. Where the acceptable-use policy is incorporated by reference into the agreement, as it is in the Llama 4 licence, it is a term of the agreement and applies according to its own wording. Read the specific clause rather than assuming either that it binds or that it does not.