In a breakthrough for data engineers working with SQL Server, the mssql-python driver now supports fetching data directly into Apache Arrow structures, eliminating the need for Python object creation during data transfer. The feature, contributed by community developer Felix Graßl, promises dramatic speed improvements and reduced memory consumption for users of Polars, Pandas, and DuckDB.
"Fetching a million rows used to mean a million Python objects and GC allocations," said Sumit Sarabhai, a reviewer of the feature. "Now the entire fetch loop runs in C++ and writes directly into Arrow buffers."
Background
Apache Arrow defines a columnar in-memory format that enables zero-copy data exchange between different programming languages. Its Arrow C Data Interface provides a stable ABI that allows database drivers and data analysis libraries to share memory without serialization.
Previously, fetching data from SQL Server required converting each row into Python objects, creating overhead that slowed down workflows, especially for temporal types like DATETIME. The new Arrow support bypasses this entirely.
What This Means
For data pipelines, this translates into four key benefits: speed, lower memory usage, seamless interoperability, and reduced garbage collection pressure. A Polars pipeline reading from mssql-python never needs to materialize intermediate Python objects at any stage.
"This is a game-changer for anyone processing large SQL Server datasets," said Graßl. "Users can now leverage Arrow-native libraries like Polars and DuckDB without the usual memory overhead."
Key Technical Concepts
- API vs ABI: API is a source-code contract; ABI is a binary-level contract enabling direct data exchange without serialization.
- Arrow C Data Interface: The ABI specification that makes zero-copy language interoperability possible.
Performance Impacts
The columnar fetch path avoids Python object creation per row, which should make fetching faster for many SQL Server types, especially temporal types like DATETIME and DATETIMEOFFSET. Memory usage drops dramatically: a column of one million integers becomes a single contiguous C array instead of a million Python objects.
Subsequent operations like filters, joins, and aggregations also work in-place on the same Arrow buffers. This eliminates the need for any Python-side conversions or serialization between libraries.
For users, the most immediate benefit is the ability to process large datasets without hitting memory limits or CPU bottlenecks from garbage collection. The feature is available now in the latest version of mssql-python.