Ground truth for models that predict.
Prediction markets are the rare dataset where beliefs are timestamped, priced, and then graded by reality. That makes this archive unusually good raw material for forecasting evals, calibration research and agent training — and its certification model makes the pipelines reproducible.
Forecasting evals from resolved markets
Outcomes are a RECONSTRUCTION you control, not a shipped label: the archive carries the window bounds, the aligned oracle stream that settles Up/Down, and both order books — so you derive the outcome from start and end price under the venue’s resolution rules, and can audit every input. Build evaluation sets where the model sees everything before a cutoff and is scored against the reconstructed result, with the cutoff enforceable to the millisecond.
import pyarrow.parquet as pq
# Books before the cutoff = what was knowable; the outcome is reconstructed
# from the settling oracle stream under the venue's resolution rules.
books = pq.read_table("updown_book_snapshot_100ms.parquet").to_pandas()
known = books[books.bucket_ms <= CUTOFF_MS] # leak-free context
oracle = pq.read_table("chainlink_price_100ms.parquet").to_pandas()
start = oracle[oracle.bucket_ms == WINDOW_OPEN_MS].price.iloc[0]
end = oracle[oracle.bucket_ms == WINDOW_CLOSE_MS].price.iloc[0]
label = int(end > start) # reconstructed Up/Down — check venue tie rulesCalibration against the crowd
Market prices are crowd probabilities with money behind them. Score a model’s stated probabilities against market prices at the same instant, then against realized outcomes — the gap between the three is the most honest calibration curve a forecaster can get.
# model_p: your model's probability at time t # market_p: best bid/ask midpoint at the same 100ms bucket mid = (row.bid_prices[0] + row.ask_prices[0]) / 2 brier_model = (model_p - outcome) ** 2 brier_market = (mid - outcome) ** 2 # the bar to beat
Agent tool-use over the API
The API is four GET endpoints with bearer auth — a natural tool surface for an agent loop. An agent can list availability, pull a day, and verify the hash before reasoning over it, so every conclusion is pinned to certified bytes.
TOOLS = [
{"name": "availability", "url": f"{API}/v1/availability?source={{source}}"},
{"name": "download",
"url": f"{API}/v1/download?source={{source}}&channel={{ch}}&scope={{scope}}&day={{day}}"},
]
# Agent loop: check availability -> download day -> sha256 == manifest -> analyzeLeak-free by construction
Every aligned row carries freshness_state, source_lag_ms and receive_lag_ms, and per-dataset timestamp walls audit the stamps (one disclosed venue exception: Predict.Fun’s clock runs up to ~7.4s ahead of true receive time — both stamps are stored honestly). Training sets built on these fields contain exactly what a live system could have known — lookahead is caught in data, not in code review.
usable = books[
(books.freshness_state == "fresh") &
(books.receive_lag_ms < 200) # only what arrived in time
]Reproducible pipelines
Every file has a SHA256 manifest, so a training run can pin its inputs the way it pins package versions. Re-running the pipeline months later either uses byte-identical data or fails loudly — no silent dataset drift under your fine-tune.
assert sha256(open("t.parquet","rb").read()).hexdigest() == manifest["sha256"]
# pin it in your run config:
run_config["dataset_fingerprint"] = manifest["sha256"]Snippets use the real schemas — column names match the dataset docs exactly. Files are standard Parquet; anything that reads Arrow reads the archive.
Building something on top of this?
Evals, agents, fine-tunes, benchmarks — tell us what you're training and we'll help scope the coverage. Certified history restarts at the ten-source inception day.