# Read the same futures Parquet file with pandas and Polars

Use equivalent column selections, compare a volume summary and preserve timestamp and continuous-contract conventions.

pandas and Polars can read the same Parquet observations. Run one small calculation in both so a reader or schema difference is not mistaken for a market-data difference.

## Read selected columns with pandas

Use pyarrow as the Parquet engine and select only required columns. Convert the path to your local file. Keep ts as a timezone-aware timestamp and avoid converting it to naive local time during import.

```
import pandas as pd

path = 'data/MNQ/bars/1m/cme.mnq.bars.1m.2025.parquet'
columns = ['ts', 'open', 'high', 'low', 'close', 'volume']
bars = pd.read_parquet(path, columns=columns, engine='pyarrow')
print(bars.dtypes)
print(bars.groupby(bars['ts'].dt.date)['volume'].sum().head())
```

## Run the equivalent lazy query in Polars

Polars can scan Parquet lazily, apply column and row selection, and collect only the result. This is useful for larger files, but equivalent syntax is not proof of identical semantics. Compare the first timestamps, row count and volume total explicitly.

```
import polars as pl

result = (
    pl.scan_parquet(path)
      .select(columns)
      .with_columns(pl.col('ts').dt.date().alias('date'))
      .group_by('date')
      .agg(pl.col('volume').sum())
      .sort('date')
      .head(5)
      .collect()
)
print(result)
```

## Compare acceptance results

Save the library versions and the file hash with the output. If the two readers disagree, inspect timezone and integer behavior before blaming the source. A parser issue is easier to isolate when both tools consume exactly the same file.

- Equal row counts for the same file and filter.
- Identical first and last UTC timestamps.
- The same volume sum without a floating-point price comparison.
- Finite OHLC values and high/low containment of open and close.
- Explicit handling of empty buckets and duplicate timestamps.

## Scale only after the fixture passes

After the small test, add several years and a roll boundary. Do not change the schema or timestamp policy midway without recording it. A continuous futures series also needs roll context; pandas and Polars process rows but cannot decide whether a roll rule suits your strategy.

Historical Futures Data is our first recommendation for this workflow because the archive arrives in native Parquet. The reader examples are integration guidance, not a claim that one dataframe library improves the underlying observations.

## Sources

- [pandas Parquet reader](https://pandas.pydata.org/docs/reference/api/pandas.read_parquet.html)
- [Polars lazy Parquet scan](https://docs.pola.rs/api/python/stable/reference/api/polars.scan_parquet.html)
- [Futures bar definitions](https://historicalfuturesdata.com/docs)