Step 1: Content-address blobs
Content-address blobs
Versioning data and models well starts with one idea: address everything by the hash of its content. Then identical data is stored once, and a version id is just the hash, reproducible from the bytes alone with no central counter to trust.
store.py gives you the store: new_store(), get(store, id), manifest_bytes(tree),
read_manifest(store, id) and a deterministic train(...). A store holds blobs, runs, tags and
pins.
Write two functions in versioning.py:
blob_id(data): the content id of some bytes, their sha256 hex digest.put(store, data): store the bytes under their content id and return the id. Putting the same bytes twice keeps a single copy.
Run it: the same content gets the same id, and the store dedupes it.
versioning.py, the file you edit78 lines
"""Your version store. You content-address data and models, snapshot a dataset as a tree, record which data
and code produced each model, tell whether a run reproduces, verify the store's integrity, and garbage
collect what no tagged release still needs."""
import hashlib
import json
import store as S
# ---------- Step 1: content addressing ----------
def blob_id(data):
"""The content id of some bytes: their sha256 hex digest."""
# TODO (Step 1): sha256 hex digest of the bytes.
raise NotImplementedError("Step 1: write blob_id()")
def put(store, data):
"""Store bytes under their content id and return the id. Storing the same bytes twice keeps one copy."""
# TODO (Step 1): store data under blob_id(data) and return the id; the same bytes overwrite
# the same key, so they are kept once.
raise NotImplementedError("Step 1: write put()")
# ---------- Step 2: snapshot a dataset ----------
def snapshot(store, files):
"""Store every file's bytes, build a {path: blob_id} tree, store that tree too, and return its id. Two
datasets that share a file share its blob; two identical datasets get the same snapshot id."""
raise NotImplementedError("snapshot() arrives in Step 2")
def diff(store, snap_a, snap_b):
"""What changed from snapshot a to b: {"added", "removed", "changed"} lists of paths. A path is changed
when it exists in both but points at different blobs."""
raise NotImplementedError("diff() arrives in Step 2")
# ---------- Step 3: record lineage ----------
def record_run(store, name, version, data_snap, code_id, params, model_bytes):
"""Store the model and record the run that made it: its name, version, the data snapshot, the code id
and params, and the model's id. Returns the run record."""
raise NotImplementedError("record_run() arrives in Step 3")
def provenance(store, model_id):
"""The inputs that produced a model: {"data_snap", "code_id", "params"} of the run whose model_id this
is, or None if no run made it."""
raise NotImplementedError("provenance() arrives in Step 3")
# ---------- Step 4: reproducibility and integrity ----------
def run_fingerprint(run):
"""A hash of a run's INPUTS only (data snapshot, code id, params). Two runs reproduce each other when
their fingerprints match, whatever their name or version."""
raise NotImplementedError("run_fingerprint() arrives in Step 4")
def verify_store(store):
"""The ids of any blobs whose stored bytes no longer hash to their id: corruption or tampering. An
intact store returns an empty list."""
raise NotImplementedError("verify_store() arrives in Step 4")
# ---------- Step 5: garbage collection ----------
def reachable(store):
"""Every blob id still needed by a tagged release: for each tagged run, its model, its code, its data
snapshot and every file blob inside that snapshot, plus anything pinned directly."""
raise NotImplementedError("reachable() arrives in Step 5")
def gc(store):
"""Delete blobs no tagged release needs, and return the ids removed. Reachable and pinned blobs stay,
so any tagged model remains fully materializable afterwards."""
raise NotImplementedError("gc() arrives in Step 5")Provided for you:
store.pytry_it.py