- Local SQLiteStorage
- IncrementalSync
- NoneServer needed
- Plain SQLQuery
You do not need a data warehouse to keep your own copy of the order book. The rm-api CLI syncs snapshots into a local SQLite file, incrementally, so it only fetches what is new, giving you a queryable archive you fully control, with nothing more than a single file on disk.
A local archive is the right call when you want repeatable analysis: the data does not move under you between runs, queries are instant, and you can work offline. SQLite is ideal for this, it is just a file, it speaks ordinary SQL, and every analysis language can read it.
Why keep a local copy at all
Reproducible
Your backtest reads the same rows every run. No moving target, no surprise gaps from a live pull mid-analysis.
- Frozen dataset
- Instant local queries
- Works offline
Incremental
Sync only pulls snapshots newer than what you already have, so a daily top-up is fast and cheap on credits.
- Fetches only new rows
- Cheap to keep current
- Resumable
Queryable
It is plain SQL. Filter by market, time, or spread; join across markets; or point pandas straight at the file.
- Standard SQL
- Reads into pandas
- No server to run
A file you control
Once a window is synced, the archive is yours to keep, back it up, version it, ship it to a colleague. Reproducibility is what makes a backtest credible, and a local snapshot store is the simplest way to get it.
When the archive outgrows a single machine, the same snapshots are the ones our own pipeline stores in ClickHouse, so the schema you learn locally is the schema you scale into.
How an incremental sync works
The sync keeps a high-water mark, the timestamp of the newest snapshot you already hold for each market and side. On the next run it asks only for rows after that mark, appends them, and advances the watermark. That is what makes a daily top-up cheap: you never re-pay credits for data you have already stored, and an interrupted run simply resumes from where it stopped.
- 1Choose the markets and time window you want to keep locally.
- 2Run the first sync to seed the SQLite file with the initial window.
- 3Schedule a recurring top-up, daily is common, to pull only what is new since last time.
- 4Query the file with plain SQL, or point pandas or R straight at it for analysis.
What lands in each row
- best_bid, best_ask, mid_price, spread, the derived prices, ready to filter and chart.
- bid_depth_total, ask_depth_total, depth pre-summed, so liquidity queries are trivial in SQL.
- bids and asks arrays, the full ladder, stored when you sync with the book included.
- timestamp, event_timestamp, capture_timestamp, sequence_number, for time indexing, latency checks, and gap detection.
- crypto_price and crypto_price_age_ms, the spot reference and how fresh it was, on each crypto snapshot.
- Local SQLiteSingle file, no server
- IncrementalOnly fetches new rows
- Plain SQLUniversally readable
- 700M +
Snapshots in the full archive
Reproducibility is what separates a credible backtest from a lucky one. A frozen local file is the simplest way to get it.
Start your archive
The docs cover the CLI and the snapshot schema; the quant research page covers heavier storage and gap-detection workflows.
Frequently asked questions
Do I need a database server to store Polymarket data locally?
No. The rm-api CLI writes to a local SQLite file, a single file on disk that needs no server and speaks ordinary SQL. It is ideal for local development and small-to-medium research archives.
Does the sync re-download everything each time?
No. The sync is incremental, it only fetches snapshots newer than what you already have stored, so keeping the archive current is fast and light on your credit allowance.
Can I read the SQLite archive from pandas or R?
Yes. SQLite is universally supported, point pandas, R, a notebook, or any SQL client at the file and query it directly. You can also export to CSV or JSONL from the CLI if you prefer flat files.
When should I move from SQLite to something heavier like ClickHouse?
SQLite comfortably handles local development and small-to-medium research archives. When you are scanning hundreds of millions of rows across many markets, or sharing one store across a team, a columnar store like ClickHouse pays off, and since it is the same snapshot schema our own pipeline uses, the queries you wrote locally carry over.