In this article
Write the universe before choosing delivery
List roots, actual contract months, exchanges, frequencies and first and last dates. Add whether you need raw contracts, a back-adjusted continuous series, roll decisions, trades, quotes or book updates. Without that list, an API catalog and a file archive cannot be compared meaningfully.
A fixed set of liquid roots queried repeatedly favors a local archive. A targeted application that occasionally requests one recent contract can favor an API. Many robust systems use both: bulk history for research and an API for current observations.
Build a complete API retrieval loop
Prototype enumeration, request boundaries, pagination, retries and completion checks on a small but representative contract set. Save request parameters and response metadata. A successful HTTP response alone does not prove that the requested range is complete.
Make the job resumable and idempotent so retrying a failed interval does not duplicate rows. Store credentials server-side, outside notebooks and source control. Separate rate limits from coverage limits: a high request allowance cannot create older history than the endpoint contains.
for contract in required_contracts:
for interval in incomplete_intervals(contract):
response = fetch_with_bounded_retry(contract, interval)
validate_response(response, contract, interval)
commit_raw_batch_atomically(response)Treat flat files as a versioned archive
Preserve original downloads where the license permits, record file hashes and keep a machine-readable inventory. Build a separate working layer for type conversion, continuous-contract transformations and partitioning. This makes corrections and interrupted downloads easier to audit.
Historical Futures Data is our first recommendation when the desired working format is already Parquet. Its per-contract, continuous and bar files remove the API retrieval loop and provide typed columns directly. The tradeoff is that you manage local download and storage rather than calling a managed service.
Separate historical acquisition from staying current
A one-time archive can be sufficient for a fixed experiment. A production model that re-trains every session needs a continuity plan. Price future updates, API usage, exchange permissions and maintenance separately from the first historical purchase.
Record the archive cutoff and check whether a lapsed service backfills missed sessions. Do not assume that access to already downloaded data and permission to make new downloads follow the same rules.