- rm-apiCLI
- CSV ·
JSON · JSONLFormats - incremental SQLiteSync
- frozen datasetResult
A backtest you cannot re-run is a story, not a result. The fix is a local, frozen dataset: pull the history once, write it to disk, and run every experiment against the same bytes. The rm-api CLI exists for exactly this, bulk exports and an incremental sync that turn the historical snapshot archive into a reproducible file you control.
Querying the API live is perfect for monitoring and one-off lookups. Research wants something different: a fixed snapshot of history that does not move under your feet between runs. Export once to a local file and your dataset is frozen, the same input every notebook, every backtest, every reviewer sees.
Why a frozen local dataset matters
Reproducible by construction
A file on disk is identical on every run. Two backtests over the same export are comparing logic, not chasing a moving dataset.
- Same bytes each run
- Logic vs data isolated
- Reviewable results
Fast local iteration
No round-trip per query. Once the data is local, you iterate at disk speed and never touch a rate limit mid-experiment.
- No per-call latency
- No rate-limit stalls
- Tight feedback loop
Columnar, analysis-ready
Flat tabular exports load straight into dataframes, the same uniform snapshot schema across every market and category.
- One schema
- Drops into pandas
- No bespoke parsing
What the rm-api CLI gives you
The CLI wraps the same read-only endpoints you would call by hand, but optimised for volume. It does bulk export to flat formats, CSV, JSON and JSONL, and it does an incremental SQLite sync that keeps a local database current without re-downloading what you already hold.
- Bulk export, pull a market and window straight to CSV for spreadsheets and dataframes, or JSON / JSONL for streaming row-by-row into a pipeline.
- Incremental SQLite sync, maintain a local database that only fetches new snapshots since the last run, so a daily top-up stays small.
- Same fields throughout, every export carries the snapshot schema, so a script written for crypto runs unchanged on any category you have access to.
- Read-only, always, the CLI pulls data; it never places an order or touches Polymarket itself.
CSV, JSONL, or a synced database?
- 1Reach for CSV when the destination is a dataframe or a spreadsheet and the window fits comfortably in memory, it is the shortest path into pandas.
- 2Reach for JSONL when you want to stream large pulls line by line, processing each snapshot without loading the whole file at once.
- 3Reach for the incremental SQLite sync when you are maintaining a living dataset, a local store you top up regularly and query repeatedly without re-pulling history.
- 4Whichever you pick, snapshot the export alongside your code so a result can always be traced back to the exact data that produced it.
From export to notebook to ClickHouse
A CSV or JSONL export drops straight into a pandas dataframe, and from there into whatever notebook or backtest you are running. When a single file stops being enough, many markets, long horizons, repeated heavy scans, the same snapshot schema loads into ClickHouse, the columnar store our own pipeline uses. Nothing about the data shape changes; you are just moving the same rows into an engine built to scan billions of them.
- CSVStraight into pandas and spreadsheets
- JSON ·
JSONLStream large pulls row by row - SQLite syncIncremental local store, topped up over time
- ClickHouseSame schema when you outgrow a single file
The method is free; volume is what you buy
The export workflow, the schema and the CLI are the same on every tier, that is the generous part. What the paid tiers add is convenience and scale: more credits for the large paginated pulls a deep export needs, higher rate limits to move volume faster, and the broader categories beyond crypto. The Free tier already exports all crypto at full depth across unlimited history; you upgrade for throughput and breadth, not for a secret feature.
One caveat to carry into the dataframe: a snapshot records resting size and the touch, not executed volume. A deep level in your export is intent to trade, which can be pulled before anyone hits it, so treat depth as the book’s standing offer, not as filled trades, when your backtest reasons about liquidity.
Freeze the data once and every experiment after it is honest by construction. The export is not a convenience, it is what makes the result reproducible.
Build your research dataset
The historical guide covers the export endpoints; quant research shows the ClickHouse path when one file is no longer enough.
Frequently asked questions
What formats can the rm-api CLI export to?
It does bulk export to CSV, JSON and JSONL, and it maintains an incremental SQLite sync. CSV is the shortest path into a dataframe or spreadsheet, JSONL streams large pulls row by row, and the SQLite sync keeps a local database current by fetching only new snapshots since the last run.
Why export to a local file instead of querying the API live?
A frozen local dataset is reproducible by construction, the same bytes on every run, so a backtest compares your logic rather than chasing a dataset that moves between runs. It also iterates at disk speed with no per-call latency or rate-limit stalls mid-experiment. Querying live stays ideal for monitoring and one-off lookups.
How do I scale beyond a single exported file?
Load the export into ClickHouse, the same columnar store our own pipeline uses. The snapshot schema is identical, so the data shape does not change; you are just moving the same rows into an engine built to scan billions of them when many markets or long horizons outgrow one file.
Does the Free tier let me export data?
Yes. The export workflow, schema and CLI are the same on every tier, and Free already exports all crypto at full depth across unlimited history. What paid tiers add is volume and breadth, more credits for large paginated pulls, higher rate limits, and the non-crypto categories, not a hidden export feature.