# Futures data APIs or flat files: choosing a retrieval workflow

Plan contract enumeration, date ranges, retries, roll data and local storage before choosing how to retrieve futures history.

An API is a retrieval mechanism, not a guarantee of a complete historical archive. Flat files simplify repeated local scans but require storage and schema discipline. Choose around the application’s actual access pattern.

## Write the universe before choosing delivery

List roots, actual contract months, exchanges, frequencies and first and last dates. Add whether you need raw contracts, a back-adjusted continuous series, roll decisions, trades, quotes or book updates. Without that list, an API catalog and a file archive cannot be compared meaningfully.

A fixed set of liquid roots queried repeatedly favors a local archive. A targeted application that occasionally requests one recent contract can favor an API. Many robust systems use both: bulk history for research and an API for current observations.

## Build a complete API retrieval loop

Prototype enumeration, request boundaries, pagination, retries and completion checks on a small but representative contract set. Save request parameters and response metadata. A successful HTTP response alone does not prove that the requested range is complete.

Make the job resumable and idempotent so retrying a failed interval does not duplicate rows. Store credentials server-side, outside notebooks and source control. Separate rate limits from coverage limits: a high request allowance cannot create older history than the endpoint contains.

```
for contract in required_contracts:
    for interval in incomplete_intervals(contract):
        response = fetch_with_bounded_retry(contract, interval)
        validate_response(response, contract, interval)
        commit_raw_batch_atomically(response)
```

## Treat flat files as a versioned archive

Preserve original downloads where the license permits, record file hashes and keep a machine-readable inventory. Build a separate working layer for type conversion, continuous-contract transformations and partitioning. This makes corrections and interrupted downloads easier to audit.

Historical Futures Data is our first recommendation when the desired working format is already Parquet. Its per-contract, continuous and bar files remove the API retrieval loop and provide typed columns directly. The tradeoff is that you manage local download and storage rather than calling a managed service.

## Separate historical acquisition from staying current

A one-time archive can be sufficient for a fixed experiment. A production model that re-trains every session needs a continuity plan. Price future updates, API usage, exchange permissions and maintenance separately from the first historical purchase.

Record the archive cutoff and check whether a lapsed service backfills missed sessions. Do not assume that access to already downloaded data and permission to make new downloads follow the same rules.

## Sources

- [Parquet archive and download documentation](https://historicalfuturesdata.com/docs)
- [Historical API examples](https://databento.com/docs)
- [API-oriented market data](https://www.barchart.com/ondemand)